US20230269368A1 - Supporting multiple partition sizes using a unified pixel input data interface for fetching reference pixels in video encoders - Google Patents
Supporting multiple partition sizes using a unified pixel input data interface for fetching reference pixels in video encoders Download PDFInfo
- Publication number
- US20230269368A1 US20230269368A1 US18/106,330 US202318106330A US2023269368A1 US 20230269368 A1 US20230269368 A1 US 20230269368A1 US 202318106330 A US202318106330 A US 202318106330A US 2023269368 A1 US2023269368 A1 US 2023269368A1
- Authority
- US
- United States
- Prior art keywords
- video
- pixel
- block
- frame
- reference pixel
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000005192 partition Methods 0.000 title description 12
- 230000004044 response Effects 0.000 claims abstract description 12
- 230000005055 memory storage Effects 0.000 claims abstract description 9
- 238000000034 method Methods 0.000 claims description 28
- 230000008569 process Effects 0.000 description 18
- 239000013598 vector Substances 0.000 description 11
- 230000006835 compression Effects 0.000 description 5
- 238000007906 compression Methods 0.000 description 5
- 230000008901 benefit Effects 0.000 description 3
- 238000005457 optimization Methods 0.000 description 3
- 238000013139 quantization Methods 0.000 description 3
- 241000278713 Theora Species 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 101100498818 Arabidopsis thaliana DDR4 gene Proteins 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 238000013213 extrapolation Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000013442 quality metrics Methods 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- 239000004557 technical material Substances 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
- H04N19/43—Hardware specially adapted for motion estimation or compensation
- H04N19/433—Hardware specially adapted for motion estimation or compensation characterised by techniques for memory access
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/103—Selection of coding mode or of prediction mode
- H04N19/105—Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/119—Adaptive subdivision aspects, e.g. subdivision of a picture into rectangular or non-rectangular coding blocks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/132—Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
- H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
- H04N19/51—Motion estimation or motion compensation
- H04N19/523—Motion estimation or motion compensation with sub-pixel accuracy
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
- H04N19/51—Motion estimation or motion compensation
- H04N19/55—Motion estimation with spatial constraints, e.g. at image or region borders
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
- H04N19/51—Motion estimation or motion compensation
- H04N19/563—Motion estimation with padding, i.e. with filling of non-object values in an arbitrarily shaped picture block or region for estimation purposes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
- H04N19/61—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding
Definitions
- a video coding format is a content representation format for storage or transmission of digital video content (such as in a data file or bitstream) It typically uses a standardized video compression algorithm.
- video coding formats include H.262 (MPEG-2 Part 2), MPEG-4 Part 2, H.264 (MPEG-4 Part 10), HEVC (H.265), Theora, Real Video RV40, VP9, and AV1.
- a video codec is a device or software that provides encoding and decoding for digital video. Most codecs are typically implementations of video coding formats.
- Some websites may have billions of users and each user may upload or download one or more videos each day.
- the website may store the video in one or more different video coding formats, each being compatible with or more efficient for a certain set of applications, hardware, or platforms. Therefore, higher video compression rates are desirable.
- AV1 offers up to 50% more compression compared to its predecessor.
- higher compression rates come higher computational complexity; therefore, improved hardware architecture and techniques in video coding would be desirable.
- FIG. 1 illustrates a block diagram of an embodiment of a video encoder 100 .
- FIG. 2 illustrates an embodiment of a system 200 for storing and providing video pixel data.
- FIG. 3 illustrates an exemplary process 300 for storing and providing video pixel data.
- FIG. 4 A illustrates an example of a reference pixel block that has at least a portion that is outside a frame of a video.
- FIG. 4 B illustrates an example of adding padding pixel data to portion 406 of the reference pixel block 408 of the video that is inside the video frame to form the reference pixel block 408 , which is larger in size than portion 406 .
- FIG. 5 illustrates that the padding pixel value at a particular pixel location is based on the pixel value of a closest pixel within the portion of the reference pixel block of the video inside the frame of the video.
- FIG. 6 A illustrates that a 32 ⁇ 32 reference pixel block 602 may be divided into two smaller block units of 16 ⁇ 32 reference pixel blocks ( 602 A and 602 B).
- FIG. 6 B illustrates that with an 8-tap interpolation, the left block unit of 16 ⁇ 32 pixel block unit 602 A together with the extra pixels for interpolation become a 24 ⁇ 40 pixel transfer block 604 A that is transferred by reference pixel cache controller 212 via the 24 pixel wide pixel data interface.
- FIG. 6 C illustrates that with an 8-tap interpolation, the right block unit of 16 x 32 pixel block unit 602 B together with the extra pixels for interpolation become a 24 ⁇ 40 pixel transfer block 604 B that is transferred by reference pixel cache controller 212 via the 24 pixel wide pixel data interface.
- FIG. 7 illustrates a table 700 of examples of transferring reference pixel data from a DRAM memory to a fractional motion estimation (FME) module via a DMA controller using a unified pixel data interface
- FME fractional motion estimation
- FIG. 8 illustrates one exemplary set of parameters that are specified by the client in table 800 .
- FIG. 9 illustrates a table 900 showing an example of transferring reference pixel data from a DRAM memory to a motion compensation module via a DMA controller using a unified pixel data interface.
- the invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor.
- these implementations, or any other form that the invention may take, may be referred to as techniques.
- the order of the steps of disclosed processes may be altered within the scope of the invention.
- a component such as a processor or a memory described as being configured to perform a task may be implemented as a general component that is temporarily configured to perform the task at a given time or a specific component that is manufactured to perform the task.
- the term ‘processor’ refers to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions.
- FIG. 1 illustrates a block diagram of an embodiment of a video encoder 100 .
- video encoder 100 supports the video coding format AV1 (Alliance for Open Media Video 1).
- video encoder 100 may also support other video coding formats as well, such as H.262 (MPEG-2 Part 2), MPEG-4 Part 2, H.264 (MPEG-4 Part 10), HEVC (H.265), Theora, RealVideo RV40, and VP9.
- Video encoder 100 includes many modules. Some of the main modules of video encoder 100 are shown in FIG. 1 . As shown in FIG. 1 , video encoder 100 includes a direct memory access (DMA) controller 114 for transferring video data. Video encoder 100 also includes an AMBA (Advanced Microcontroller Bus Architecture) to CSR (control and status register) module 116 . Other main modules include a motion estimation module 102 , a mode decision module 104 , a decoder prediction module 106 , a central controller 108 , a decoder residue module 110 , and a filter 112 .
- DMA direct memory access
- AMBA Advanced Microcontroller Bus Architecture
- CSR control and status register
- Other main modules include a motion estimation module 102 , a mode decision module 104 , a decoder prediction module 106 , a central controller 108 , a decoder residue module 110 , and a filter 112 .
- Video encoder 100 includes a central controller module 108 that controls the different modules of video encoder 100 , including motion estimation module 102 , mode decision module 104 , decoder prediction module 106 , decoder residue module 110 , filter 112 , and DMA controller 114 .
- Video encoder 100 includes a motion estimation module 102 .
- Motion estimation module 102 includes an integer motion estimation (IME) module 118 and a fractional motion estimation (FME) module 120 .
- Motion estimation module 102 determines motion vectors that describe the transformation from one image to another, for example, from one frame to an adjacent frame.
- a motion vector is a two-dimensional vector used for inter-frame prediction; it refers the current frame to the reference frame, and its coordinate values provide the coordinate offsets from a location in the current frame to a location in the reference frame.
- Motion estimation module 102 estimates the best motion vector, which may be used for inter prediction in mode decision module 104 .
- An inter coded frame is divided into blocks, e.g., prediction units or partitions within a macroblock.
- the encoder will try to find a block similar to the one it is encoding on a previously encoded frame, referred to as a reference frame. This process is done by a block matching algorithm. If the encoder succeeds on its search, the block could be encoded by a vector, known as a motion vector, which points to the position of the matching block at the reference frame. The process of motion vector determination is called motion estimation.
- Video encoder 100 includes a mode decision module 104 .
- the main components of mode decision module 104 include an inter prediction module 122 , an intra prediction module 128 , a motion vector prediction module 124 , a rate-distortion optimization (RDO) module 130 , and a decision module 126 .
- Mode decision module 104 detects one prediction mode among a number of candidate inter prediction modes and intra prediction modes that gives the best results for encoding a block of video.
- Intra prediction is the process of deriving the prediction value for the current sample using previously decoded sample values in the same decoded frame. Intra prediction exploits spatial redundancy, i.e., correlation among pixels within one frame, by calculating prediction values through extrapolation from already coded pixels for effective delta coding. Inter prediction is the process of deriving the prediction value for the current frame using previously encoded reference frames. Inter prediction exploits temporal redundancy.
- Rate-distortion optimization is the optimization of the amount of distortion (loss of video quality) against the amount of data required to encode the video, i.e., the rate.
- RDO module 130 provides a video quality metric that measures both the deviation from the source material and the bit cost for each possible decision outcome. Both inter prediction and intra prediction have different candidate prediction modes, and inter prediction and intra prediction that are performed under different prediction modes may result in final pixels requiring different rates and having different amounts of distortion and other costs.
- different prediction modes may use different block sizes for prediction.
- some parts of the image there may be a large region that can all be predicted at the same time (e.g., a still background image), while in other parts there may be some fine details that are changing (e.g., in a talking head) and a smaller block size would be appropriate. Therefore, some video coding formats provide the ability to vary the block size to handle a range of prediction sizes.
- the decoder decodes each image in units of superblocks (e.g., 128 ⁇ 128 or 64 ⁇ 64 pixel superblocks). Each superblock has a partition that specifies how it is to be encoded. Superblocks may be divided into smaller blocks according to different partitioning patterns. This allows superblocks to be divided into partitions as small as 4 ⁇ 4 pixels.
- different prediction modes may use different settings in inter prediction and intra prediction.
- inter prediction modes there are different inter prediction modes corresponding to using different reference frames, which have different motion vectors.
- the intra prediction modes depend on the neighboring pixels, and AV1 uses eight main directional modes, and each allows a supplementary signal to tune the prediction angle in units of 3°.
- the modes include DC, Vertical, Horizontal, TM (True Motion), Horizontal Up, Left Diagonal, Vertical Right, Vertical Left, Right Diagonal, and Horizontal Down.
- RDO module 130 receives the output of inter prediction module 122 corresponding to each of the inter prediction modes and determines their corresponding amounts of distortion and rates, which are sent to decision module 126 . Similarly, RDO module 130 receives the output of intra prediction module 128 corresponding to each of the intra prediction modes and determines their corresponding amounts of distortion and rates, which are also sent to decision module 126 .
- inter prediction module 122 or intra prediction module 128 predicts the pixels, and the residual data (i.e., the differences between the original pixels and the predicted pixels) may be sent to RDO module 130 , such that RDO module 130 may determine the corresponding amount of distortion and rate.
- RDO module 130 may estimate the amounts of distortion and rates corresponding to each prediction mode by estimating the final results after additional processing steps (e.g., applying transforms and quantization) are performed on the outputs of inter prediction module 122 and intra prediction module 128 .
- Decision module 126 evaluates the cost corresponding to each inter prediction mode and intra prediction mode.
- the cost is based at least in part on the amount of distortion and the rate associated with the particular prediction mode.
- the rate includes different components, including the coefficient rate, mode rate, partition rate, and token cost/probability. Other additional costs may include the cost of sending a motion vector in the bit stream.
- Decision module 126 selects the best inter prediction mode that has the lowest overall cost among all the inter prediction modes.
- decision module 126 selects the best intra prediction mode that has the lowest overall cost among all the intra prediction modes. Decision module 126 then selects the best prediction mode (intra or inter) that has the lowest overall cost among all the prediction modes. The selected prediction mode is the best mode detected by mode decision module 104 .
- Decoder prediction module 106 includes an inter prediction module 132 , an intra prediction module 134 , and a reconstruction module 136 . If the selected mode is an inter prediction mode, then the inter prediction module 132 is used to do the inter prediction, whereas if the selected mode is an intra prediction mode, then the intra prediction module 134 is used to do the intra prediction. Decoder residue module 110 includes a transform and quantization module (T/Q) 138 and an inverse quantization and inverse transform module (IQ/IT) 140 .
- T/Q transform and quantization module
- IQ/IT inverse quantization and inverse transform module
- Inter prediction is a key technique for achieving significant compression gains in video encoding.
- Inter prediction is the process of deriving the prediction values for the current frame using previously decoded reference frames.
- these reference frames are stored in DRAM (Dynamic Random Access Memory), and a fetch engine, such as a DMA (Direct Memory Access), is used to copy the reference frames to a local cache.
- DRAM Dynamic Random Access Memory
- a fetch engine such as a DMA (Direct Memory Access)
- Multiple clients may request the reference frame data from the cache at various stages of the encoding process.
- Different video encoding formats may support different sets of partition shapes. For example, H.264 (MPEG-4 Part 10) and VP9 each supports multiple partition shapes. Therefore, a unified pixel data interface that may be used by the clients to fetch the data efficiently from the cache would be desirable.
- a system for storing and providing video pixel data for video encoding comprises a memory storage and a cache storage.
- the system further comprises a controller.
- the controller is configured to receive a request that causes transferring of a reference pixel block of a video, wherein the video is being encoded using the reference pixel block.
- the controller is configured to determine whether the reference pixel block has at least a portion that is outside a frame of the video.
- the controller In response to the determination of the reference pixel block having at least a portion outside the frame of the video, the controller is configured to cause a portion of the reference pixel block of the video inside the frame of the video to be fetched from a memory storage and stored in a cache storage, pad a remaining missing portion of the reference pixel block of the video outside the frame of the video with padding pixel data to form the reference pixel block, and transfer the reference pixel block in response to the request.
- FIG. 2 illustrates an embodiment of a system 200 for storing and providing video pixel data.
- System 200 includes a direct memory access (DMA) controller 208 for transferring video data from one or more DRAM memory modules 214 to different modules of a video encoder.
- the different modules of the video encoder may include a motion estimation module 202 , a mode decision module 204 , and a motion compensation module 206 .
- These modules of the video encoder are clients of the DMA controller 208 , and these modules may request DMA controller 208 to transfer video data from DRAM memory 214 at various stages of the encoding process.
- DMA controller 208 includes a reference pixel cache controller 212 and a local cache 210 .
- DRAM memory 214 may be any Double Data Rate Synchronous Dynamic Random-Access Memory (DDR SDRAM) memory, which is a double data rate (DDR) synchronous dynamic random-access memory (SDRAM) class of memory integrated circuits, such as a DDR4 or DDR5 SDRAM memory.
- DDR SDRAM Double Data Rate Synchronous Dynamic Random-Access Memory
- DDR double data rate
- SDRAM synchronous dynamic random-access memory
- FIG. 3 illustrates an exemplary process 300 for storing and providing video pixel data.
- process 300 may be performed by DMA controller 208 .
- a request that causes transferring of a reference pixel block of a video is received, wherein the video is being encoded using the reference pixel block.
- the reference pixel block is associated with a previously decoded reference frame that is used for inter prediction.
- a read request (indicated in FIG. 2 as rd_req) may be sent by a client of the DMA controller 208 to the DMA controller 208 .
- the client may be one of the modules of the video encoder, including motion estimation module 202 , mode decision module 204 , and motion compensation module 206 .
- the read request is received by the reference pixel cache controller 212 of DMA controller 208 .
- FIG. 4 A illustrates an example of a reference pixel block that has at least a portion that is outside a frame of a video.
- Source block 404 is a block of pixels in a current video frame 402 .
- source block 404 may be a 16 ⁇ 16 pixel block.
- the reference pixel block 408 that is needed for inter prediction is 24 ⁇ 24 pixels in size.
- a motion vector 410 corresponding to source block 404 and the reference pixel block 408 is shown connecting the two blocks.
- a portion 406 of the reference pixel block 408 is inside its video frame, whereas the remaining portion of the reference pixel block 408 is outside its video frame. Since the reference pixel block has at least a portion that is outside a frame of the video, process 300 proceeds to step 306 ; otherwise, process 300 proceeds to step 305 and exits.
- a portion of the reference pixel block of the video inside the frame of the video is caused to be fetched from a memory storage and stored in a cache storage.
- portion 406 of the reference pixel block 408 that is inside the video frame is fetched from DDR memory 214 and stored in local cache 210 .
- a remaining portion of the reference pixel block of the video outside the frame of the video is padded with padding pixel data to form the reference pixel block.
- FIG. 4 B illustrates an example of adding padding pixel data to portion 406 of the reference pixel block 408 that is inside the video frame to form the reference pixel block 408 , which is larger in size than portion 406 .
- the formed reference pixel block 408 may be stored in a local buffer in reference pixel cache controller 212 .
- the missing pixels are padded by reference pixel cache controller 212 before the reference pixel block 408 is sent back to the client.
- the key advantage is that the padding logic is shared by all the clients.
- the padding pixel values are determined based on the positions of the missing pixels relative to the portion of the reference pixel block of the video inside the frame of the video.
- FIG. 5 illustrates that the padding pixel value at a particular pixel location is based on the pixel value of a closest pixel within the portion of the reference pixel block of the video inside the frame of the video.
- the missing pixel is padded with the pixel value of a pixel that is in the portion of the reference pixel block inside the frame and is directly below it and on the first line of the frame.
- a missing pixel 502 above the video frame is padded with the pixel value of the pixel 504 that is in portion 406 and directly below missing pixel 502 and on the first top line of portion 406 .
- the missing pixels replicate the first line pixels.
- the missing pixel is padded with the pixel value of a pixel that is in the portion of the reference pixel block inside the frame and is the last right-most pixel in that portion on the same horizontal line.
- a missing pixel 506 to the right of the video frame is padded with the pixel value of the pixel 508 that is in portion 406 and is the last right-most pixel on the same horizontal line.
- the missing pixels replicate the last pixel of the corresponding line.
- the missing pixel is padded with the pixel value of a pixel that is in the portion of the reference pixel block inside the frame and is directly above it and on the last line of the frame. In other words, for bottom padding, the missing pixels replicate the last line pixels.
- the missing pixel is padded with the pixel value of a pixel that is in the portion of the reference pixel block inside the frame and is the first left-most pixel in that portion on the same horizontal line. In other words, for left padding, the missing pixels replicate the first pixel of the corresponding line.
- the missing pixel is padded with the pixel value of a pixel that is in the portion of the reference pixel block inside the frame and is the last right-most pixel in that portion on the first line of the frame.
- a missing pixel 512 that is above and to the right of the video frame is padded with the pixel value of the pixel 510 that is in portion 406 and is the last right-most pixel on the first line of the frame.
- the reference pixel block is provided to the client in response to the client’s request.
- the reference pixel block may be sent from the reference pixel cache controller 212 to the client (indicated as rd_data in FIG. 2 ), which is one of the modules of the video encoder, including motion estimation module 202 , mode decision module 204 , and motion compensation module 206 .
- multiple clients may request the reference frame data from the cache at various stages of the encoding process.
- Different video encoding formats may support different sets of partition shapes. For example, H.264 (MPEG-4 Part 10) and VP9 each supports multiple partition shapes. Therefore, a unified pixel data interface to fetch the data efficiently from the cache is used. As will be described in greater detail below, the width of the unified pixel data interface is selected to optimally support different partition shapes and different video encoding formats.
- FME fractional motion estimation
- a pixel data interface of 24 ⁇ 1 per line in size is selected to support different partition shapes.
- interpolation requires extra samples in addition to the original block of the pixels to be read from DDR memory 214 via DMA controller 208 .
- H.264 uses a 6-tap interpolation, which requires six additional pixels.
- a block of pixels that needs to be transferred (e.g., a prediction unit) is divided into smaller block units that are 16 samples wide, such that a smaller block unit together with the additional samples required for interpolation still fit into the 24 pixel wide pixel data interface.
- a prediction unit has a width that is a multiple of 16 pixels, then the prediction unit is divided into multiple block units.
- Each block unit is 16 pixels in width, and each block unit corresponds to one transfer block, and wherein each line of the transfer block includes 16 pixels of the corresponding unit and additional pixels for interpolation.
- a prediction unit has a width that is a multiple of 8 pixels
- Each block unit is 8 pixels in width, and each block unit corresponds to one transfer block, and wherein each line of the transfer block includes 8 pixels of the corresponding unit and additional pixels for interpolation.
- more cycles are needed to finish the entire super block; however, less area is needed as fewer pixels per clock are processed.
- Each block unit is 32 pixels in width, and each block unit corresponds to one transfer block, and wherein each line of the transfer block includes 32 pixels of the corresponding unit and additional pixels for interpolation. In this embodiment, fewer cycles are needed to finish the entire super block; however, more area is needed as more pixels per clock are processed.
- the transferring of the reference pixel block is divided into a plurality of transfer blocks, wherein each transfer block comprises a width and a height, and wherein the width is a predetermined block unit pixel width plus additional pixels for interpolation.
- FIGS. 6 A, 6 B, and 6 C illustrate an example of transferring reference pixel data from a DRAM memory to a fractional motion estimation (FME) module via a DMA controller using a unified pixel data interface.
- FIG. 6 A illustrates that a 32 ⁇ 32 reference pixel block 602 may be divided into two smaller block units of 16 ⁇ 32 reference pixel blocks ( 602 A and 602 B).
- FIG. 6 B illustrates that with an 8-tap interpolation, the left block unit of 16 ⁇ 32 pixel block unit 602 A together with the extra pixels for interpolation become a 24 ⁇ 40 pixel transfer block 604 A that is transferred by reference pixel cache controller 212 via the 24 pixel wide pixel data interface.
- FIG. 6 A illustrates that a 32 ⁇ 32 reference pixel block 602 may be divided into two smaller block units of 16 ⁇ 32 reference pixel blocks ( 602 A and 602 B).
- FIG. 6 B illustrates that with an 8-tap interpolation, the left block unit of 16
- FIG. 6 C illustrates that with an 8-tap interpolation, the right block unit of 16 ⁇ 32 pixel block unit 602 B together with the extra pixels for interpolation become a 24 ⁇ 40 pixel transfer block 604 B that is transferred by reference pixel cache controller 212 via the 24 pixel wide pixel data interface. Therefore, an original 32 ⁇ 32 reference pixel block is split into two 24 ⁇ 40 pixel block transfers.
- FIG. 7 illustrates a table 700 of examples of transferring reference pixel data from a DRAM memory to a fractional motion estimation (FME) module via a DMA controller using a unified pixel data interface.
- FME fractional motion estimation
- a 16 ⁇ 16 original reference pixel block does not need to be divided into smaller block units.
- the 16 ⁇ 16 pixel block together with the extra pixels for interpolation become a 24 ⁇ 24 pixel transfer block that is transferred by reference pixel cache controller 212 via the 24 pixel wide pixel data interface. Therefore, an original 16 ⁇ 16 reference pixel block requires one 24 ⁇ 24 pixel block transfer.
- a 16 ⁇ 32 original reference pixel block does not need to be divided into smaller block units.
- the 16 ⁇ 32 pixel block together with the extra pixels for interpolation become a 24 ⁇ 40 pixel transfer block that is transferred by reference pixel cache controller 212 via the 24 pixel wide pixel data interface. Therefore, an original 16 ⁇ 32 reference pixel block requires one 24 ⁇ 40 pixel block transfer.
- a 32 ⁇ 16 original reference pixel block may be divided into two smaller block units of 16 ⁇ 16 reference pixel blocks.
- one 16 ⁇ 16 pixel block unit together with the extra pixels for interpolation become a 24 ⁇ 24 pixel transfer block that is transferred by reference pixel cache controller 212 via the 24 pixel wide pixel data interface.
- the second 16 ⁇ 16 pixel block unit together with the extra pixels for interpolation become a 24 ⁇ 24 pixel transfer block that is transferred by reference pixel cache controller 212 via the 24 pixel wide pixel data interface. Therefore, an original 32 ⁇ 16 reference pixel block is split into two 24 ⁇ 24 pixel block transfers.
- Row 4 of table 700 is the same as the example shown in FIGS. 6 A, 6 B, and 6 C .
- the 32 ⁇ 32 pixel block together with the extra pixels for interpolation become a 40 ⁇ 40 pixel block.
- the original 32 ⁇ 32 reference pixel block is split into two 24 ⁇ 40 pixel block transfers.
- a 32 ⁇ 64 original reference pixel block may be divided into two smaller 16 ⁇ 64 reference pixel block units.
- one 16 ⁇ 64 pixel block unit together with the extra pixels for interpolation become a 24 ⁇ 72 pixel transfer block that is transferred by reference pixel cache controller 212 via the 24 pixel wide pixel data interface.
- the second 16 ⁇ 64 pixel block unit together with the extra pixels for interpolation become a 24 ⁇ 72 pixel transfer block that is transferred by reference pixel cache controller 212 via the 24 pixel wide pixel data interface. Therefore, an original 32 ⁇ 64 reference pixel block is split into two 24 ⁇ 72 pixel block transfers.
- a 64 ⁇ 32 original reference pixel block may be divided into four smaller 16 ⁇ 32 reference pixel block units.
- each 16 ⁇ 32 pixel block unit together with the extra pixels for interpolation become a 24 ⁇ 40 pixel transfer block that is transferred by reference pixel cache controller 212 via the 24 pixel wide pixel data interface. Therefore, an original 64 ⁇ 32 reference pixel block is split into four 24 ⁇ 40 pixel block transfers.
- a 64 ⁇ 64 original reference pixel block may be divided into four smaller 16 ⁇ 64 reference pixel block units.
- each 16 ⁇ 64 pixel block unit together with the extra pixels for interpolation become a 24 ⁇ 72 pixel transfer block that is transferred by reference pixel cache controller 212 via the 24 pixel wide pixel data interface. Therefore, an original 64 ⁇ 64 reference pixel block is split into four 24 ⁇ 72 pixel block transfers.
- a client may specify in a read request (e.g., rd_req in FIG. 2 ) the size of the block that is needed to be transferred (e.g., the block’s width and height), the start location, and the horizontal/vertical block split information.
- FIG. 8 illustrates one exemplary set of parameters that are specified by the client in table 800 .
- the splitting of a reference pixel block into multiple transfer blocks may be handled by the reference pixel cache controller 212 and the data is returned in multiple transfer cycles as shown in table 700 .
- Reference pixel blocks are split horizontally as downstream engines require data to be processed in a vertical raster scan order.
- One advantage of splitting a block of pixel data (e.g., 72 ⁇ 72) to multiple overlapping sub blocks (e.g., 24 ⁇ 72 sub blocks) is that the client can process each sub block independently and efficiently, in a pipeline or in parallel, while storing the data only once in a central location.
- FIG. 9 illustrates a table 900 showing an example of transferring reference pixel data from a DRAM memory to a motion compensation module via a DMA controller using a unified pixel data interface.
- the PU size is 64 ⁇ 64, but the TU size is 32 ⁇ 32.
- the 64 ⁇ 64 block is processed as eight 16 ⁇ 32 blocks.
- a block of pixels that needs to be transferred (e.g., a prediction unit) is divided into smaller block units that are 16 samples wide, such that a smaller block unit together with the additional samples required for interpolation still fit into the 24 pixel wide pixel data interface.
- a prediction unit has a width that is a multiple of 16 pixels, then the prediction unit is divided horizontally into multiple block units.
- the prediction unit has a pixel height that is a multiple of the TU height (e.g., 32 pixels), then the prediction unit is further divided vertically.
- Each block unit is 16 pixels in width and 32 pixels in height, and each block unit corresponds to one transfer block.
- Each line of the transfer block includes 16 pixels of the corresponding unit and an additional eight pixels for interpolation.
- Each column of the transfer block includes 32 pixels of the corresponding unit and eight additional pixels for interpolation. Therefore, each of the eight transfer blocks is 24 ⁇ 40 pixels in size.
- a client may specify in a read request (e.g., rd_req in FIG. 2 ) whether the reference pixel blocks need to be divided vertically.
- the splitting of a reference pixel block into multiple transfer blocks may be handled by the reference pixel cache controller 212 and the data is returned in multiple cycles.
- One advantage of dividing the reference pixel block vertically is that it is more area efficient for vertical filtering kernels.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
A system for storing and providing video pixel data for video encoding is disclosed. The system comprises a memory storage and a cache storage. The system further comprises a controller. The controller is configured to receive a request that causes transferring of a reference pixel block of a video, wherein the video is being encoded using the reference pixel block. The controller is configured to determine whether the reference pixel block has at least a portion that is outside a frame of the video. In response to the determination of the reference pixel block having at least a portion outside the frame of the video, the controller is configured to cause a portion of the reference pixel block of the video inside the frame of the video to be fetched from a memory storage and stored in a cache storage, pad a remaining missing portion of the reference pixel block of the video outside the frame of the video with padding pixel data to form the reference pixel block, and transfer the reference pixel block in response to the request.
Description
- This application is a continuation of U.S. Pat. Application No. 17/322,668 entitled SUPPORTING MULTIPLE PARTITION SIZES USING A UNIFIED PIXEL INPUT DATA INTERFACE FOR FETCHING REFERENCE PIXELS IN VIDEO ENCODERS filed May 17, 2021 which is incorporated herein by reference for all purposes.
- A video coding format is a content representation format for storage or transmission of digital video content (such as in a data file or bitstream) It typically uses a standardized video compression algorithm. Examples of video coding formats include H.262 (MPEG-2 Part 2), MPEG-4
Part 2, H.264 (MPEG-4 Part 10), HEVC (H.265), Theora, Real Video RV40, VP9, and AV1. A video codec is a device or software that provides encoding and decoding for digital video. Most codecs are typically implementations of video coding formats. - Recently, there has been an explosive growth of video usage on the Internet. Some websites (e.g., social media websites or video sharing websites) may have billions of users and each user may upload or download one or more videos each day. When a user uploads a video from a user device onto a website, the website may store the video in one or more different video coding formats, each being compatible with or more efficient for a certain set of applications, hardware, or platforms. Therefore, higher video compression rates are desirable. For example, AV1 offers up to 50% more compression compared to its predecessor. However, with higher compression rates come higher computational complexity; therefore, improved hardware architecture and techniques in video coding would be desirable.
- Various embodiments of the invention are disclosed in the following detailed description and the accompanying drawings.
-
FIG. 1 illustrates a block diagram of an embodiment of avideo encoder 100. -
FIG. 2 illustrates an embodiment of asystem 200 for storing and providing video pixel data. -
FIG. 3 illustrates anexemplary process 300 for storing and providing video pixel data. -
FIG. 4A illustrates an example of a reference pixel block that has at least a portion that is outside a frame of a video. -
FIG. 4B illustrates an example of adding padding pixel data toportion 406 of thereference pixel block 408 of the video that is inside the video frame to form thereference pixel block 408, which is larger in size thanportion 406. -
FIG. 5 illustrates that the padding pixel value at a particular pixel location is based on the pixel value of a closest pixel within the portion of the reference pixel block of the video inside the frame of the video. -
FIG. 6A illustrates that a 32 × 32reference pixel block 602 may be divided into two smaller block units of 16 × 32 reference pixel blocks (602A and 602B). -
FIG. 6B illustrates that with an 8-tap interpolation, the left block unit of 16 × 32pixel block unit 602A together with the extra pixels for interpolation become a 24 × 40pixel transfer block 604A that is transferred by referencepixel cache controller 212 via the 24 pixel wide pixel data interface. -
FIG. 6C illustrates that with an 8-tap interpolation, the right block unit of 16 x 32pixel block unit 602B together with the extra pixels for interpolation become a 24 × 40pixel transfer block 604B that is transferred by referencepixel cache controller 212 via the 24 pixel wide pixel data interface. -
FIG. 7 illustrates a table 700 of examples of transferring reference pixel data from a DRAM memory to a fractional motion estimation (FME) module via a DMA controller using a unified pixel data interface -
FIG. 8 illustrates one exemplary set of parameters that are specified by the client in table 800. -
FIG. 9 illustrates a table 900 showing an example of transferring reference pixel data from a DRAM memory to a motion compensation module via a DMA controller using a unified pixel data interface. - The invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor. In this specification, these implementations, or any other form that the invention may take, may be referred to as techniques. In general, the order of the steps of disclosed processes may be altered within the scope of the invention. Unless stated otherwise, a component such as a processor or a memory described as being configured to perform a task may be implemented as a general component that is temporarily configured to perform the task at a given time or a specific component that is manufactured to perform the task. As used herein, the term ‘processor’ refers to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions.
- A detailed description of one or more embodiments of the invention is provided below along with accompanying figures that illustrate the principles of the invention. The invention is described in connection with such embodiments, but the invention is not limited to any embodiment. The scope of the invention is limited only by the claims and the invention encompasses numerous alternatives, modifications and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the invention. These details are provided for the purpose of example and the invention may be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the invention has not been described in detail so that the invention is not unnecessarily obscured.
-
FIG. 1 illustrates a block diagram of an embodiment of avideo encoder 100. For example,video encoder 100 supports the video coding format AV1 (Alliance for Open Media Video 1). However,video encoder 100 may also support other video coding formats as well, such as H.262 (MPEG-2 Part 2), MPEG-4Part 2, H.264 (MPEG-4 Part 10), HEVC (H.265), Theora, RealVideo RV40, and VP9. -
Video encoder 100 includes many modules. Some of the main modules ofvideo encoder 100 are shown inFIG. 1 . As shown inFIG. 1 ,video encoder 100 includes a direct memory access (DMA)controller 114 for transferring video data.Video encoder 100 also includes an AMBA (Advanced Microcontroller Bus Architecture) to CSR (control and status register)module 116. Other main modules include amotion estimation module 102, amode decision module 104, adecoder prediction module 106, acentral controller 108, adecoder residue module 110, and afilter 112. -
Video encoder 100 includes acentral controller module 108 that controls the different modules ofvideo encoder 100, includingmotion estimation module 102,mode decision module 104,decoder prediction module 106,decoder residue module 110,filter 112, andDMA controller 114. -
Video encoder 100 includes amotion estimation module 102.Motion estimation module 102 includes an integer motion estimation (IME)module 118 and a fractional motion estimation (FME)module 120.Motion estimation module 102 determines motion vectors that describe the transformation from one image to another, for example, from one frame to an adjacent frame. A motion vector is a two-dimensional vector used for inter-frame prediction; it refers the current frame to the reference frame, and its coordinate values provide the coordinate offsets from a location in the current frame to a location in the reference frame.Motion estimation module 102 estimates the best motion vector, which may be used for inter prediction inmode decision module 104. An inter coded frame is divided into blocks, e.g., prediction units or partitions within a macroblock. Instead of directly encoding the raw pixel values for each block, the encoder will try to find a block similar to the one it is encoding on a previously encoded frame, referred to as a reference frame. This process is done by a block matching algorithm. If the encoder succeeds on its search, the block could be encoded by a vector, known as a motion vector, which points to the position of the matching block at the reference frame. The process of motion vector determination is called motion estimation. -
Video encoder 100 includes amode decision module 104. The main components ofmode decision module 104 include aninter prediction module 122, anintra prediction module 128, a motionvector prediction module 124, a rate-distortion optimization (RDO)module 130, and adecision module 126.Mode decision module 104 detects one prediction mode among a number of candidate inter prediction modes and intra prediction modes that gives the best results for encoding a block of video. - Intra prediction is the process of deriving the prediction value for the current sample using previously decoded sample values in the same decoded frame. Intra prediction exploits spatial redundancy, i.e., correlation among pixels within one frame, by calculating prediction values through extrapolation from already coded pixels for effective delta coding. Inter prediction is the process of deriving the prediction value for the current frame using previously encoded reference frames. Inter prediction exploits temporal redundancy.
- Rate-distortion optimization (RDO) is the optimization of the amount of distortion (loss of video quality) against the amount of data required to encode the video, i.e., the rate.
RDO module 130 provides a video quality metric that measures both the deviation from the source material and the bit cost for each possible decision outcome. Both inter prediction and intra prediction have different candidate prediction modes, and inter prediction and intra prediction that are performed under different prediction modes may result in final pixels requiring different rates and having different amounts of distortion and other costs. - For example, different prediction modes may use different block sizes for prediction. In some parts of the image there may be a large region that can all be predicted at the same time (e.g., a still background image), while in other parts there may be some fine details that are changing (e.g., in a talking head) and a smaller block size would be appropriate. Therefore, some video coding formats provide the ability to vary the block size to handle a range of prediction sizes. The decoder decodes each image in units of superblocks (e.g., 128 × 128 or 64 × 64 pixel superblocks). Each superblock has a partition that specifies how it is to be encoded. Superblocks may be divided into smaller blocks according to different partitioning patterns. This allows superblocks to be divided into partitions as small as 4 × 4 pixels.
- Besides using different block sizes for prediction, different prediction modes may use different settings in inter prediction and intra prediction. For example, there are different inter prediction modes corresponding to using different reference frames, which have different motion vectors. For intra prediction, the intra prediction modes depend on the neighboring pixels, and AV1 uses eight main directional modes, and each allows a supplementary signal to tune the prediction angle in units of 3°. In VP9, the modes include DC, Vertical, Horizontal, TM (True Motion), Horizontal Up, Left Diagonal, Vertical Right, Vertical Left, Right Diagonal, and Horizontal Down.
-
RDO module 130 receives the output ofinter prediction module 122 corresponding to each of the inter prediction modes and determines their corresponding amounts of distortion and rates, which are sent todecision module 126. Similarly,RDO module 130 receives the output ofintra prediction module 128 corresponding to each of the intra prediction modes and determines their corresponding amounts of distortion and rates, which are also sent todecision module 126. - In some embodiments, for each prediction mode,
inter prediction module 122 orintra prediction module 128 predicts the pixels, and the residual data (i.e., the differences between the original pixels and the predicted pixels) may be sent toRDO module 130, such thatRDO module 130 may determine the corresponding amount of distortion and rate. For example,RDO module 130 may estimate the amounts of distortion and rates corresponding to each prediction mode by estimating the final results after additional processing steps (e.g., applying transforms and quantization) are performed on the outputs ofinter prediction module 122 andintra prediction module 128. -
Decision module 126 evaluates the cost corresponding to each inter prediction mode and intra prediction mode. The cost is based at least in part on the amount of distortion and the rate associated with the particular prediction mode. In some embodiments, the cost (also referred to as rate distortion cost, or RD Cost) may be a linear combination of the amount of distortion and the rate associated with the particular prediction mode; for example, RD Cost = distortion + λ*rate, where λ is a Lagrangian multiplier. The rate includes different components, including the coefficient rate, mode rate, partition rate, and token cost/probability. Other additional costs may include the cost of sending a motion vector in the bit stream.Decision module 126 selects the best inter prediction mode that has the lowest overall cost among all the inter prediction modes. In addition,decision module 126 selects the best intra prediction mode that has the lowest overall cost among all the intra prediction modes.Decision module 126 then selects the best prediction mode (intra or inter) that has the lowest overall cost among all the prediction modes. The selected prediction mode is the best mode detected bymode decision module 104. - After the best prediction mode is selected by
mode decision module 104, the selected best prediction mode is sent tocentral controller 108.Central controller 108 controlsdecoder prediction module 106,decoder residue module 110, and filter 112 to perform a number of steps using the mode selected bymode decision module 104. This generates the inputs to an entropy coder that generates the final bitstream.Decoder prediction module 106 includes aninter prediction module 132, anintra prediction module 134, and areconstruction module 136. If the selected mode is an inter prediction mode, then theinter prediction module 132 is used to do the inter prediction, whereas if the selected mode is an intra prediction mode, then theintra prediction module 134 is used to do the intra prediction.Decoder residue module 110 includes a transform and quantization module (T/Q) 138 and an inverse quantization and inverse transform module (IQ/IT) 140. - Inter prediction is a key technique for achieving significant compression gains in video encoding. Inter prediction is the process of deriving the prediction values for the current frame using previously decoded reference frames. Typically, these reference frames are stored in DRAM (Dynamic Random Access Memory), and a fetch engine, such as a DMA (Direct Memory Access), is used to copy the reference frames to a local cache. Multiple clients may request the reference frame data from the cache at various stages of the encoding process. Different video encoding formats may support different sets of partition shapes. For example, H.264 (MPEG-4 Part 10) and VP9 each supports multiple partition shapes. Therefore, a unified pixel data interface that may be used by the clients to fetch the data efficiently from the cache would be desirable.
- A system for storing and providing video pixel data for video encoding is disclosed. The system comprises a memory storage and a cache storage. The system further comprises a controller. The controller is configured to receive a request that causes transferring of a reference pixel block of a video, wherein the video is being encoded using the reference pixel block. The controller is configured to determine whether the reference pixel block has at least a portion that is outside a frame of the video. In response to the determination of the reference pixel block having at least a portion outside the frame of the video, the controller is configured to cause a portion of the reference pixel block of the video inside the frame of the video to be fetched from a memory storage and stored in a cache storage, pad a remaining missing portion of the reference pixel block of the video outside the frame of the video with padding pixel data to form the reference pixel block, and transfer the reference pixel block in response to the request.
-
FIG. 2 illustrates an embodiment of asystem 200 for storing and providing video pixel data.System 200 includes a direct memory access (DMA)controller 208 for transferring video data from one or moreDRAM memory modules 214 to different modules of a video encoder. The different modules of the video encoder may include amotion estimation module 202, amode decision module 204, and amotion compensation module 206. These modules of the video encoder are clients of theDMA controller 208, and these modules may requestDMA controller 208 to transfer video data fromDRAM memory 214 at various stages of the encoding process.DMA controller 208 includes a referencepixel cache controller 212 and alocal cache 210.DRAM memory 214 may be any Double Data Rate Synchronous Dynamic Random-Access Memory (DDR SDRAM) memory, which is a double data rate (DDR) synchronous dynamic random-access memory (SDRAM) class of memory integrated circuits, such as a DDR4 or DDR5 SDRAM memory. -
FIG. 3 illustrates anexemplary process 300 for storing and providing video pixel data. In some embodiments,process 300 may be performed byDMA controller 208. - At
step 302, a request that causes transferring of a reference pixel block of a video is received, wherein the video is being encoded using the reference pixel block. The reference pixel block is associated with a previously decoded reference frame that is used for inter prediction. As shown inFIG. 2 , a read request (indicated inFIG. 2 as rd_req) may be sent by a client of theDMA controller 208 to theDMA controller 208. The client may be one of the modules of the video encoder, includingmotion estimation module 202,mode decision module 204, andmotion compensation module 206. The read request is received by the referencepixel cache controller 212 ofDMA controller 208. - At
step 304, whether the reference pixel block has at least a portion that is outside a frame of the video is determined.FIG. 4A illustrates an example of a reference pixel block that has at least a portion that is outside a frame of a video.Source block 404 is a block of pixels in acurrent video frame 402. For example, source block 404 may be a 16 × 16 pixel block. Thereference pixel block 408 that is needed for inter prediction is 24 × 24 pixels in size. Amotion vector 410 corresponding to source block 404 and thereference pixel block 408 is shown connecting the two blocks. In this example, aportion 406 of thereference pixel block 408 is inside its video frame, whereas the remaining portion of thereference pixel block 408 is outside its video frame. Since the reference pixel block has at least a portion that is outside a frame of the video,process 300 proceeds to step 306; otherwise,process 300 proceeds to step 305 and exits. - At
step 306, in response to the determination of the reference pixel block having at least a portion outside the frame of the video, a portion of the reference pixel block of the video inside the frame of the video is caused to be fetched from a memory storage and stored in a cache storage. For example,portion 406 of thereference pixel block 408 that is inside the video frame is fetched fromDDR memory 214 and stored inlocal cache 210. - At
step 308, a remaining portion of the reference pixel block of the video outside the frame of the video is padded with padding pixel data to form the reference pixel block.FIG. 4B illustrates an example of adding padding pixel data toportion 406 of thereference pixel block 408 that is inside the video frame to form thereference pixel block 408, which is larger in size thanportion 406. In some embodiments, the formedreference pixel block 408 may be stored in a local buffer in referencepixel cache controller 212. The missing pixels are padded by referencepixel cache controller 212 before thereference pixel block 408 is sent back to the client. The key advantage is that the padding logic is shared by all the clients. - The padding pixel values are determined based on the positions of the missing pixels relative to the portion of the reference pixel block of the video inside the frame of the video.
FIG. 5 illustrates that the padding pixel value at a particular pixel location is based on the pixel value of a closest pixel within the portion of the reference pixel block of the video inside the frame of the video. - If a missing pixel is above the video frame, then the missing pixel is padded with the pixel value of a pixel that is in the portion of the reference pixel block inside the frame and is directly below it and on the first line of the frame. For example, with reference to
FIG. 5 , a missingpixel 502 above the video frame is padded with the pixel value of thepixel 504 that is inportion 406 and directly below missingpixel 502 and on the first top line ofportion 406. In other words, for top padding, the missing pixels replicate the first line pixels. - If a missing pixel is to the right of the video frame, then the missing pixel is padded with the pixel value of a pixel that is in the portion of the reference pixel block inside the frame and is the last right-most pixel in that portion on the same horizontal line. For example, with reference to
FIG. 5 , a missingpixel 506 to the right of the video frame is padded with the pixel value of thepixel 508 that is inportion 406 and is the last right-most pixel on the same horizontal line. In other words, for right padding, the missing pixels replicate the last pixel of the corresponding line. - If a missing pixel is below the video frame, then the missing pixel is padded with the pixel value of a pixel that is in the portion of the reference pixel block inside the frame and is directly above it and on the last line of the frame. In other words, for bottom padding, the missing pixels replicate the last line pixels.
- If a missing pixel is to the left of the video frame, then the missing pixel is padded with the pixel value of a pixel that is in the portion of the reference pixel block inside the frame and is the first left-most pixel in that portion on the same horizontal line. In other words, for left padding, the missing pixels replicate the first pixel of the corresponding line.
- If a missing pixel is above and to the right of the video frame, then the missing pixel is padded with the pixel value of a pixel that is in the portion of the reference pixel block inside the frame and is the last right-most pixel in that portion on the first line of the frame. For example, with reference to
FIG. 5 , a missingpixel 512 that is above and to the right of the video frame is padded with the pixel value of thepixel 510 that is inportion 406 and is the last right-most pixel on the first line of the frame. - At
step 310, the reference pixel block is provided to the client in response to the client’s request. For example, as shown inFIG. 2 , the reference pixel block may be sent from the referencepixel cache controller 212 to the client (indicated as rd_data inFIG. 2 ), which is one of the modules of the video encoder, includingmotion estimation module 202,mode decision module 204, andmotion compensation module 206. - As discussed above, multiple clients may request the reference frame data from the cache at various stages of the encoding process. Different video encoding formats may support different sets of partition shapes. For example, H.264 (MPEG-4 Part 10) and VP9 each supports multiple partition shapes. Therefore, a unified pixel data interface to fetch the data efficiently from the cache is used. As will be described in greater detail below, the width of the unified pixel data interface is selected to optimally support different partition shapes and different video encoding formats.
- One of the clients that may request the reference frame data from the DMA controller is the fractional motion estimation (FME) module. A pixel data interface of 24 × 1 per line in size is selected to support different partition shapes. When processing a block of reference frame pixels, interpolation requires extra samples in addition to the original block of the pixels to be read from
DDR memory 214 viaDMA controller 208. H.264 uses a 6-tap interpolation, which requires six additional pixels. VP9 uses an 8-tap interpolation, which requires eight additional pixels. Therefore, to process a 16 pixel wide block of pixels, up to eight additional pixels may be needed per line. In other words, 16 + 8 = 24 pixels may be needed. Therefore, the pixel data interface is selected to be 24 pixels in width. And a block of pixels that needs to be transferred (e.g., a prediction unit) is divided into smaller block units that are 16 samples wide, such that a smaller block unit together with the additional samples required for interpolation still fit into the 24 pixel wide pixel data interface. In other words, if a prediction unit has a width that is a multiple of 16 pixels, then the prediction unit is divided into multiple block units. Each block unit is 16 pixels in width, and each block unit corresponds to one transfer block, and wherein each line of the transfer block includes 16 pixels of the corresponding unit and additional pixels for interpolation. - Other pixel widths may be used depending on the system requirements. For example, an 8 pixel wide block of pixels may be processed per clock, and up to eight additional pixels may be needed per line. In other words, 8 + 8 = 16 pixels may be needed. Therefore, the pixel data interface is selected to be 16 pixels in width. And a block of pixels that needs to be transferred (e.g., a prediction unit) is divided into smaller block units that are 8 samples wide, such that a smaller block unit together with the additional samples required for interpolation still fit into the 16 pixel wide pixel data interface. In other words, if a prediction unit has a width that is a multiple of 8 pixels, then the prediction unit is divided into multiple block units. Each block unit is 8 pixels in width, and each block unit corresponds to one transfer block, and wherein each line of the transfer block includes 8 pixels of the corresponding unit and additional pixels for interpolation. In this embodiment, more cycles are needed to finish the entire super block; however, less area is needed as fewer pixels per clock are processed.
- In another example, a 32 pixel wide block of pixels may be processed per clock, and up to eight additional pixels may be needed per line. In other words, 32 + 8 = 40 pixels may be needed. Therefore, the pixel data interface is selected to be 40 pixels in width. And a block of pixels that needs to be transferred (e.g., a prediction unit) is divided into smaller block units that are 32 samples wide, such that a smaller block unit together with the additional samples required for interpolation still fit into the 40 pixel wide pixel data interface. In other words, if a prediction unit has a width that is a multiple of 32 pixels, then the prediction unit is divided into multiple block units. Each block unit is 32 pixels in width, and each block unit corresponds to one transfer block, and wherein each line of the transfer block includes 32 pixels of the corresponding unit and additional pixels for interpolation. In this embodiment, fewer cycles are needed to finish the entire super block; however, more area is needed as more pixels per clock are processed.
- As shown in the above examples, the transferring of the reference pixel block is divided into a plurality of transfer blocks, wherein each transfer block comprises a width and a height, and wherein the width is a predetermined block unit pixel width plus additional pixels for interpolation. For example, the predetermined block unit pixel width is 16 and the addition pixels for interpolation is 8, and the width is 16 + 8 = 24 pixels.
-
FIGS. 6A, 6B, and 6C illustrate an example of transferring reference pixel data from a DRAM memory to a fractional motion estimation (FME) module via a DMA controller using a unified pixel data interface.FIG. 6A illustrates that a 32 × 32reference pixel block 602 may be divided into two smaller block units of 16 × 32 reference pixel blocks (602A and 602B).FIG. 6B illustrates that with an 8-tap interpolation, the left block unit of 16 × 32pixel block unit 602A together with the extra pixels for interpolation become a 24 × 40pixel transfer block 604A that is transferred by referencepixel cache controller 212 via the 24 pixel wide pixel data interface.FIG. 6C illustrates that with an 8-tap interpolation, the right block unit of 16 × 32pixel block unit 602B together with the extra pixels for interpolation become a 24 × 40pixel transfer block 604B that is transferred by referencepixel cache controller 212 via the 24 pixel wide pixel data interface. Therefore, an original 32 × 32 reference pixel block is split into two 24 × 40 pixel block transfers. -
FIG. 7 illustrates a table 700 of examples of transferring reference pixel data from a DRAM memory to a fractional motion estimation (FME) module via a DMA controller using a unified pixel data interface. - As shown in
row 1 of table 700, a 16 × 16 original reference pixel block does not need to be divided into smaller block units. With an 8-tap interpolation, the 16 × 16 pixel block together with the extra pixels for interpolation become a 24 × 24 pixel transfer block that is transferred by referencepixel cache controller 212 via the 24 pixel wide pixel data interface. Therefore, an original 16 × 16 reference pixel block requires one 24 × 24 pixel block transfer. - As shown in
row 2 of table 700, a 16 × 32 original reference pixel block does not need to be divided into smaller block units. With an 8-tap interpolation, the 16 × 32 pixel block together with the extra pixels for interpolation become a 24 × 40 pixel transfer block that is transferred by referencepixel cache controller 212 via the 24 pixel wide pixel data interface. Therefore, an original 16 × 32 reference pixel block requires one 24 × 40 pixel block transfer. - As shown in
row 3 of table 700, a 32 × 16 original reference pixel block may be divided into two smaller block units of 16 × 16 reference pixel blocks. With an 8-tap interpolation, one 16 × 16 pixel block unit together with the extra pixels for interpolation become a 24 × 24 pixel transfer block that is transferred by referencepixel cache controller 212 via the 24 pixel wide pixel data interface. With an 8-tap interpolation, the second 16 × 16 pixel block unit together with the extra pixels for interpolation become a 24 × 24 pixel transfer block that is transferred by referencepixel cache controller 212 via the 24 pixel wide pixel data interface. Therefore, an original 32 × 16 reference pixel block is split into two 24 × 24 pixel block transfers. -
Row 4 of table 700 is the same as the example shown inFIGS. 6A, 6B, and 6C . The 32 × 32 pixel block together with the extra pixels for interpolation become a 40 × 40 pixel block. The original 32 × 32 reference pixel block is split into two 24 × 40 pixel block transfers. - As shown in
row 5 of table 700, a 32 × 64 original reference pixel block may be divided into two smaller 16 × 64 reference pixel block units. With an 8-tap interpolation, one 16 × 64 pixel block unit together with the extra pixels for interpolation become a 24 × 72 pixel transfer block that is transferred by referencepixel cache controller 212 via the 24 pixel wide pixel data interface. With an 8-tap interpolation, the second 16 × 64 pixel block unit together with the extra pixels for interpolation become a 24 × 72 pixel transfer block that is transferred by referencepixel cache controller 212 via the 24 pixel wide pixel data interface. Therefore, an original 32 × 64 reference pixel block is split into two 24 × 72 pixel block transfers. - As shown in
row 6 of table 700, a 64 × 32 original reference pixel block may be divided into four smaller 16 × 32 reference pixel block units. With an 8-tap interpolation, each 16 × 32 pixel block unit together with the extra pixels for interpolation become a 24 × 40 pixel transfer block that is transferred by referencepixel cache controller 212 via the 24 pixel wide pixel data interface. Therefore, an original 64 × 32 reference pixel block is split into four 24 × 40 pixel block transfers. - As shown in
row 7 of table 700, a 64 × 64 original reference pixel block may be divided into four smaller 16 × 64 reference pixel block units. With an 8-tap interpolation, each 16 × 64 pixel block unit together with the extra pixels for interpolation become a 24 × 72 pixel transfer block that is transferred by referencepixel cache controller 212 via the 24 pixel wide pixel data interface. Therefore, an original 64 × 64 reference pixel block is split into four 24 × 72 pixel block transfers. - In some embodiments, a client may specify in a read request (e.g., rd_req in
FIG. 2 ) the size of the block that is needed to be transferred (e.g., the block’s width and height), the start location, and the horizontal/vertical block split information.FIG. 8 illustrates one exemplary set of parameters that are specified by the client in table 800. In some embodiments, the splitting of a reference pixel block into multiple transfer blocks may be handled by the referencepixel cache controller 212 and the data is returned in multiple transfer cycles as shown in table 700. Reference pixel blocks are split horizontally as downstream engines require data to be processed in a vertical raster scan order. One advantage of splitting a block of pixel data (e.g., 72 × 72) to multiple overlapping sub blocks (e.g., 24 × 72 sub blocks) is that the client can process each sub block independently and efficiently, in a pipeline or in parallel, while storing the data only once in a central location. - One of the clients that may request the reference frame data from the DMA controller is
motion compensation module 206. Similar to the fractional motion estimation client mentioned above,motion compensation module 206 requires extra pixels for interpolation. One difference betweenmotion compensation module 206 andmotion estimation module 202 is that data for the former is processed in transform units (TUs). Therefore, a block of pixels may be split vertically as well depending on the TU size.FIG. 9 illustrates a table 900 showing an example of transferring reference pixel data from a DRAM memory to a motion compensation module via a DMA controller using a unified pixel data interface. The PU size is 64 × 64, but the TU size is 32 × 32. The 64 × 64 block is processed as eight 16 × 32 blocks. A block of pixels that needs to be transferred (e.g., a prediction unit) is divided into smaller block units that are 16 samples wide, such that a smaller block unit together with the additional samples required for interpolation still fit into the 24 pixel wide pixel data interface. In other words, if a prediction unit has a width that is a multiple of 16 pixels, then the prediction unit is divided horizontally into multiple block units. If the prediction unit has a pixel height that is a multiple of the TU height (e.g., 32 pixels), then the prediction unit is further divided vertically. Each block unit is 16 pixels in width and 32 pixels in height, and each block unit corresponds to one transfer block. Each line of the transfer block includes 16 pixels of the corresponding unit and an additional eight pixels for interpolation. Each column of the transfer block includes 32 pixels of the corresponding unit and eight additional pixels for interpolation. Therefore, each of the eight transfer blocks is 24 × 40 pixels in size. - In some embodiments, a client may specify in a read request (e.g., rd_req in
FIG. 2 ) whether the reference pixel blocks need to be divided vertically. In some embodiments, the splitting of a reference pixel block into multiple transfer blocks may be handled by the referencepixel cache controller 212 and the data is returned in multiple cycles. One advantage of dividing the reference pixel block vertically is that it is more area efficient for vertical filtering kernels. - Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and not restrictive.
Claims (20)
1. A system, comprising:
a memory storage;
a cache storage;
a controller configured to:
receive a request that causes transferring of a reference pixel block of a video, wherein the video is being encoded using the system;
determine whether the reference pixel block has at least a portion that is outside a frame of the video; and
in response to the determination of the reference pixel block having at least the portion that is outside the frame of the video:
cause a portion of the reference pixel block of the video inside the frame of the video to be fetched from the memory storage and stored in the cache storage;
pad a remaining missing portion of the reference pixel block of the video outside the frame of the video with padding pixel data to form the reference pixel block; and
transfer the reference pixel block in response to the request, wherein the transferring of the reference pixel block is divided into a plurality of transfer blocks, wherein each transfer block comprises a width and a height, and wherein the width is a predetermined block unit pixel width plus additional pixels for interpolation, and wherein in the event a prediction unit has a width that is a multiple of the predetermined block unit pixel width, the prediction unit is divided horizontally into a plurality of block units, wherein each block unit has the predetermined block unit pixel width, and wherein each block unit corresponds to one of the transfer blocks, and wherein each line of a transfer block includes the pixels in a corresponding line of the corresponding block unit and the additional pixels for interpolation.
2. The system of claim 1 , wherein values of the padding pixel data are determined based on positions of the remaining missing portion relative to the portion of the reference pixel block of the video inside the frame of the video.
3. The system of claim 2 , wherein a value of the padding pixel data at a particular pixel location is based on a pixel value of a closest pixel within the portion of the reference pixel block of the video inside the frame of the video.
4. The system of claim 3 , wherein a pixel location that is above the frame of the video is padded with a value of a pixel that is in the portion of the reference pixel block of the video inside the frame of the video and directly below the pixel location and on a first line of the portion of the reference pixel block of the video inside the frame of the video.
5. The system of claim 3 , wherein a pixel location that is to the right of the frame of the video is padded with a value of a pixel that is in the portion of the reference pixel block of the video inside the frame of the video and is a last right-most pixel on a same horizontal line.
6. The system of claim 3 , wherein a pixel location that is below the frame of the video is padded with a value of a pixel that is in the portion of the reference pixel block of the video inside the frame of the video and directly above the pixel location and on a last line of the portion of the reference pixel block of the video inside the frame of the video.
7. The system of claim 3 , wherein a pixel location that is to the left of the frame of the video is padded with a value of a pixel that is in the portion of the reference pixel block of the video inside the frame of the video and is a first left-most pixel on a same horizontal line.
8. The system of claim 1 , wherein in the event a prediction unit has a width that is a multiple of the predetermined block unit pixel width and a height that is a multiple of a transform unit height, the prediction unit is divided horizontally and vertically into a plurality of block units, and wherein each block unit has the predetermined block unit pixel width and having the transform unit height, and wherein each block unit corresponds to one of the transfer blocks, and wherein each line of a transfer block includes the pixels in a corresponding line of the corresponding block unit and the additional pixels for interpolation, and wherein each column of the transfer block includes a number of pixels of the corresponding block unit and additional pixels for interpolation, wherein the number of pixels equals the transform unit height.
9. A method, comprising:
receiving by a controller a request that causes transferring of a reference pixel block of a video;
determining by the controller whether the reference pixel block has at least a portion that is outside a frame of the video; and
in response to the determination of the reference pixel block having at least the portion that is outside the frame of the video:
causing by the controller a portion of the reference pixel block of the video inside the frame of the video to be fetched from a memory storage and stored in a cache storage;
padding by the controller a remaining missing portion of the reference pixel block of the video outside the frame of the video with padding pixel data to form the reference pixel block; and
transferring by the controller the reference pixel block in response to the request, wherein the transferring of the reference pixel block is divided into a plurality of transfer blocks, wherein each transfer block comprises a width and a height, and wherein the width is a predetermined block unit pixel width plus additional pixels for interpolation, and wherein in the event a prediction unit has a width that is a multiple of the predetermined block unit pixel width, the prediction unit is divided horizontally into a plurality of block units, wherein each block unit has the predetermined block unit pixel width, and wherein each block unit corresponds to one of the transfer blocks, and wherein each line of a transfer block includes the pixels in a corresponding line of the corresponding block unit and the additional pixels for interpolation.
10. The method of claim 9 , wherein values of the padding pixel data are determined based on positions of the remaining missing portion relative to the portion of the reference pixel block of the video inside the frame of the video.
11. The method of claim 10 , wherein a value of the padding pixel data at a particular pixel location is based on a pixel value of a closest pixel within the portion of the reference pixel block of the video inside the frame of the video.
12. The method of claim 11 , wherein a pixel location that is above the frame of the video is padded with a value of a pixel that is in the portion of the reference pixel block of the video inside the frame of the video and directly below the pixel location and on a first line of the portion of the reference pixel block of the video inside the frame of the video.
13. The method of claim 11 , wherein a pixel location that is to the right of the frame of the video is padded with a value of a pixel that is in the portion of the reference pixel block of the video inside the frame of the video and is a last right-most pixel on a same horizontal line.
14. The method of claim 11 , wherein a pixel location that is below the frame of the video is padded with a value of a pixel that is in the portion of the reference pixel block of the video inside the frame of the video and directly above the pixel location and on a last line of the portion of the reference pixel block of the video inside the frame of the video.
15. The method of claim 11 , wherein a pixel location that is to the left of the frame of the video is padded with a value of a pixel that is in the portion of the reference pixel block of the video inside the frame of the video and is a first left-most pixel on a same horizontal line.
16. A system, comprising:
a controller with a processor configured to:
receive a request that causes transferring of a reference pixel block of a video, wherein the video is being encoded using the system;
determine whether the reference pixel block has at least a portion that is outside a frame of the video; and
in response to the determination of the reference pixel block having at least the portion that is outside the frame of the video:
cause a portion of the reference pixel block of the video inside the frame of the video to be fetched from memory storage and stored in cache storage;
pad a remaining missing portion of the reference pixel block of the video outside the frame of the video with padding pixel data to form the reference pixel block; and
transfer the reference pixel block in response to the request, wherein the transferring of the reference pixel block is divided into a plurality of transfer blocks, wherein each transfer block comprises a width and a height, and wherein the width is a predetermined block unit pixel width plus additional pixels for interpolation, and wherein in the event a prediction unit has a width that is a multiple of the predetermined block unit pixel width, the prediction unit is divided horizontally into a plurality of block units, wherein each block unit has the predetermined block unit pixel width, and wherein each block unit corresponds to one of the transfer blocks, and wherein each line of a transfer block includes the pixels in a corresponding line of the corresponding block unit and the additional pixels for interpolation; and
a memory coupled to the processor and configured to provide the processor with instructions.
17. The system of claim 16 , wherein values of the padding pixel data are determined based on positions of the remaining missing portion relative to the portion of the reference pixel block of the video inside the frame of the video.
18. The system of claim 17 , wherein a value of the padding pixel data at a particular pixel location is based on a pixel value of a closest pixel within the portion of the reference pixel block of the video inside the frame of the video.
19. The system of claim 18 , wherein a pixel location that is above the frame of the video is padded with a value of a pixel that is in the portion of the reference pixel block of the video inside the frame of the video and directly below the pixel location and on a first line of the portion of the reference pixel block of the video inside the frame of the video.
20. The system of claim 18 , wherein a pixel location that is to the right of the frame of the video is padded with a value of a pixel that is in the portion of the reference pixel block of the video inside the frame of the video and is a last right-most pixel on a same horizontal line.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US18/106,330 US20230269368A1 (en) | 2021-05-17 | 2023-02-06 | Supporting multiple partition sizes using a unified pixel input data interface for fetching reference pixels in video encoders |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/322,668 US11622106B2 (en) | 2021-05-17 | 2021-05-17 | Supporting multiple partition sizes using a unified pixel input data interface for fetching reference pixels in video encoders |
US18/106,330 US20230269368A1 (en) | 2021-05-17 | 2023-02-06 | Supporting multiple partition sizes using a unified pixel input data interface for fetching reference pixels in video encoders |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/322,668 Continuation US11622106B2 (en) | 2021-05-17 | 2021-05-17 | Supporting multiple partition sizes using a unified pixel input data interface for fetching reference pixels in video encoders |
Publications (1)
Publication Number | Publication Date |
---|---|
US20230269368A1 true US20230269368A1 (en) | 2023-08-24 |
Family
ID=82020005
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/322,668 Active US11622106B2 (en) | 2021-05-17 | 2021-05-17 | Supporting multiple partition sizes using a unified pixel input data interface for fetching reference pixels in video encoders |
US18/106,330 Pending US20230269368A1 (en) | 2021-05-17 | 2023-02-06 | Supporting multiple partition sizes using a unified pixel input data interface for fetching reference pixels in video encoders |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/322,668 Active US11622106B2 (en) | 2021-05-17 | 2021-05-17 | Supporting multiple partition sizes using a unified pixel input data interface for fetching reference pixels in video encoders |
Country Status (2)
Country | Link |
---|---|
US (2) | US11622106B2 (en) |
WO (1) | WO2022245678A1 (en) |
Family Cites Families (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6891893B2 (en) * | 2000-04-21 | 2005-05-10 | Microsoft Corp. | Extensible multimedia application program interface and related methods |
US8559514B2 (en) | 2006-07-27 | 2013-10-15 | Qualcomm Incorporated | Efficient fetching for motion compensation video decoding process |
JP2009272724A (en) * | 2008-04-30 | 2009-11-19 | Panasonic Corp | Video coding-decoding device |
US20110122950A1 (en) * | 2009-11-26 | 2011-05-26 | Ji Tianying | Video decoder and method for motion compensation for out-of-boundary pixels |
US9380314B2 (en) * | 2010-12-20 | 2016-06-28 | Texas Instruments Incorporated | Pixel retrieval for frame reconstruction |
US20120314767A1 (en) * | 2011-06-13 | 2012-12-13 | Qualcomm Incorporated | Border pixel padding for intra prediction in video coding |
US9883203B2 (en) * | 2011-11-18 | 2018-01-30 | Qualcomm Incorporated | Adaptive overlapped block motion compensation |
US9325991B2 (en) * | 2012-04-11 | 2016-04-26 | Qualcomm Incorporated | Motion vector rounding |
US9510008B2 (en) | 2013-11-25 | 2016-11-29 | Entropic Communications, Llc | Video decoder memory bandwidth compression |
CN107798382B (en) * | 2017-11-21 | 2020-09-01 | 南京地平线机器人技术有限公司 | Method and apparatus for adapting feature data in convolutional neural networks |
US10986349B2 (en) * | 2017-12-29 | 2021-04-20 | Microsoft Technology Licensing, Llc | Constraints on locations of reference blocks for intra block copy prediction |
WO2020113156A1 (en) * | 2018-11-30 | 2020-06-04 | Tencent America LLC | Method and apparatus for video coding |
WO2020156535A1 (en) * | 2019-02-01 | 2020-08-06 | Beijing Bytedance Network Technology Co., Ltd. | Interactions between in-loop reshaping and block differential pulse coded modulation |
CA3127136A1 (en) * | 2019-02-01 | 2020-08-06 | Beijing Bytedance Network Technology Co., Ltd. | Signaling of in-loop reshaping information using parameter sets |
US11706411B2 (en) * | 2020-09-02 | 2023-07-18 | Tencent America LLC | Content adaptive segmented prediction |
-
2021
- 2021-05-17 US US17/322,668 patent/US11622106B2/en active Active
-
2022
- 2022-05-14 WO PCT/US2022/029345 patent/WO2022245678A1/en active Application Filing
-
2023
- 2023-02-06 US US18/106,330 patent/US20230269368A1/en active Pending
Also Published As
Publication number | Publication date |
---|---|
US20220368887A1 (en) | 2022-11-17 |
WO2022245678A1 (en) | 2022-11-24 |
US11622106B2 (en) | 2023-04-04 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10051273B2 (en) | Video decoder and video decoding method | |
US9872018B2 (en) | Random access point (RAP) formation using intra refreshing technique in video coding | |
Chen et al. | Analysis and design of macroblock pipelining for H. 264/AVC VLSI architecture | |
KR100772379B1 (en) | External memory device, method for storing image date thereof, apparatus for processing image using the same | |
US9380314B2 (en) | Pixel retrieval for frame reconstruction | |
CN105282558A (en) | Prediction method, coding method, decoding method and device thereof of pixels in frame | |
US8594189B1 (en) | Apparatus and method for coding video using consistent regions and resolution scaling | |
US20160050431A1 (en) | Method and system for organizing pixel information in memory | |
CN108810549B (en) | Low-power-consumption-oriented streaming media playing method | |
US8503534B2 (en) | Multi-bus architecture for a video codec | |
WO2004102971A1 (en) | Video processing device with low memory bandwidth requirements | |
US9300975B2 (en) | Concurrent access shared buffer in a video encoder | |
US20220337837A1 (en) | Architecture for rate estimation in video coding | |
USRE47004E1 (en) | Moving image coding device and method | |
US11622106B2 (en) | Supporting multiple partition sizes using a unified pixel input data interface for fetching reference pixels in video encoders | |
US20100014597A1 (en) | Efficient apparatus for fast video edge filtering | |
US11606568B1 (en) | End-of-block detection of quantized transform coefficient matrices in video encoding | |
US20100246682A1 (en) | Scaled motion search section with downscaling and method for use therewith | |
US20230345021A1 (en) | Efficient storage of data for a multi-stage two-dimensional transform | |
US11909993B1 (en) | Fractional motion estimation engine with parallel code unit pipelines | |
US11425393B1 (en) | Hardware optimization of rate calculation in rate distortion optimization for video encoding | |
US9094686B2 (en) | Systems and methods for faster throughput for compressed video data decoding | |
US20130170565A1 (en) | Motion Estimation Complexity Reduction | |
US11750800B1 (en) | Encoder and associated signal processing method | |
Shen et al. | Memory bandwidth reduction for video decoders based on data arrangements |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |