WO2022062719A1 - Video processing method, computer-readable storage medium, and electronic device - Google Patents
Video processing method, computer-readable storage medium, and electronic device Download PDFInfo
- Publication number
- WO2022062719A1 WO2022062719A1 PCT/CN2021/111123 CN2021111123W WO2022062719A1 WO 2022062719 A1 WO2022062719 A1 WO 2022062719A1 CN 2021111123 W CN2021111123 W CN 2021111123W WO 2022062719 A1 WO2022062719 A1 WO 2022062719A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- video
- video processing
- processed
- processing unit
- frame
- Prior art date
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
- H04N19/436—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation using parallelised computational arrangements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
- H04N19/423—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation characterised by memory arrangements
Definitions
- the present invention relates to the technical field of video processing, and in particular, to a video processing method, a computer-readable storage medium, and an electronic device.
- Patent Publication No. CN104952037A proposes an FPGA-based JPEG image file decompression and scaling process acceleration method and system implementation, the main purpose of which is to provide high-speed image scaling processing for background images, and the image scaling needs to be decompressed first . Therefore, its system design focuses on the image processing acceleration of PC or server plus FPGA platform. This architecture is not suitable for embedded video processing acceleration or frame rate improvement.
- the memory access structure is simple and cannot break through the memory bandwidth limitation. The matching problem of the number of cores described, the access memory bandwidth bottleneck of 5 or more cores is not fully considered, so it is only a simple architecture design.
- One aspect of the present application provides a video processing method, comprising the steps of:
- S1 from a plurality of parallel video processing units, select the first video processing unit in an idle state to perform video processing on the video frame to be processed;
- Another aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, the processor implements the above video processing method when executing the computer program .
- a video unit selection module configured to select a first video processing unit in an idle state from a plurality of parallel video processing units to perform video processing on the video frame to be processed;
- a video frame processing module configured to write the to-be-processed video frame into the first HBM storage slice corresponding to the first video processing unit, configure and start the first video processing unit, and the first video processing unit.
- a video processing unit for reading and processing the to-be-processed video frame from the first HBM storage slice, and then writing the processed video frame into the first HBM storage slice;
- FIG. 1 is a schematic flowchart of a video processing method according to an embodiment of the present invention.
- FIG. 2 is a schematic diagram of an overall framework of a video processing method according to an embodiment of the present invention.
- FIG. 5 is an overview diagram of an application case of the video processing method according to an embodiment of the present invention.
- FIG. 7 is a schematic timing diagram of a video processing unit involved in an embodiment of the present invention.
- FIG. 8 is a schematic time sequence diagram of a video amplification processing functional module involved in an embodiment of the present invention.
- the video processing method includes the steps:
- S1 from a plurality of parallel video processing units, select the first video processing unit in an idle state to perform video processing on the video frame to be processed;
- the MPSoC control unit of the FPGA performs the above steps.
- the MPSoC control unit allocates a video processing unit in an idle state for each video frame to be processed for video processing. Therefore, the processing speed of a video frame to be processed can be processed according to the speed of the video frame.
- the required frame rate to configure the number of video units, the increase in the number of video processing units means faster parallel processing, which means high data throughput.
- the beneficial effects of the present invention are: when processing video with high resolution and high frame rate, video processing is performed on the video frame to be processed by a plurality of parallel video processing units, thereby improving the processing speed of the video processing;
- the writing and reading of the video frames to be processed are carried out through the high-bandwidth memory HBM built in the FPGA, so as to improve the data throughput and meet the storage bandwidth requirements;
- the present invention can realize the video processing through the FPGA, so that it has a wide range of functions. Usability, that is, the present invention can satisfy universality, storage bandwidth requirements and processing speed requirements at the same time when processing high-resolution and high-frame-rate videos.
- the video processing unit includes a control and state monitoring register
- Configuring and starting the first video processing unit in the step S2 specifically includes the following steps:
- starting the first video processing unit in the step S2 specifically includes the following steps:
- the two threads are implemented based on the query mechanism.
- the first thread continuously confirms the input status of the video frame to be processed during execution. Because the video stream has a fixed input and output resolution, it can The data is measured to obtain a preset threshold. When the preset threshold is reached, the flag bit can be set in advance, and the operation of writing the video processing unit register in the first thread is started to start the video processing function module in advance, thereby effectively reducing the video output. time delay.
- reading and outputting the processed video frame from the first HBM memory slice specifically includes the following steps:
- a burst transmission is initiated to the AXI4 Slave port of the first HBM memory slice through the AXI4 Master port to read and output the processed video frame from the first HBM memory slice.
- the HBM storage slice includes an input buffer area and an output buffer area, the to-be-processed video frame is stored in the input buffer area, and the processed video frame is stored in the output buffer area.
- the MPSoC control unit of the FPGA implements high-performance address mapping communication through the AXI4 port.
- the AXI4 port allows a maximum of 256 rounds of data burst transmission. Therefore, the video processing function module can also be transferred from the HBM memory chip through the AXI4 port.
- the input buffer area reads the video frames to be processed, and at the same time writes the processed video frames into the output video buffer area.
- Another embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the video processing methods described in the above embodiments.
- another embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor implements the computer program when executing the computer program.
- the video processing method and corresponding circuit, device, storage medium and electronic device of the present application are mainly used in application scenarios for processing any video, such as video decompression and scaling processing, etc.
- application scenarios for processing any video such as video decompression and scaling processing, etc.
- the following description is combined with specific application scenarios:
- the first embodiment of the present invention is:
- FPGA Programmable Large Scale Logic Array
- ARM general-purpose embedded processor CPU
- HBM High bandwidth Memory, high bandwidth memory
- Video Process video processing unit
- MPSoC FPGA Programmable FPGA with multi-core ARM processor
- AXI_lite lightweight ARM bus for register access
- AXI4 address mapping ARM bus, used to access large-capacity memory
- the FPGA includes the MPSoC control unit, the video processing unit and the HBM storage device.
- the video processing unit needs to be developed and implemented in the FPGA.
- Both MPSoC and HBM are units already included and existing in the FPGA.
- the number of video processing units is configured according to actual video processing requirements. The increase in the number of video processing units means the speed of parallel processing, which means high data throughput.
- the HBM storage device provides high-bandwidth access for the input video frame buffer and the output video frame buffer of the corresponding video processing unit.
- the MPSoC control unit is a control core that coordinates the efficient parallel operation of multiple video processing units, that is, the MPSoC control unit executes the steps of this embodiment.
- the MPSoC control unit is provided with a video processing main program, and the main video processing main program is There are two tasks, one is the video processing unit configuration, and the other is the task scheduling of the video processing unit.
- this is a list of defined standard general-purpose control and status monitoring registers, including a start register for running control, a frame number register for recording the sequence number of the current frame, and a frame number register for recording the input video frame resolution
- the relevant frame serial number registers are described as follows: the video frame to be processed as the input frame and the processed video frame as the output frame have corresponding frame serial numbers.
- each video processing unit has a frame
- the serial number register when writing the processing video frame, sort the processing video frame, and then write the serial number value into the frame serial number register, so that the video frame processor can be set according to the principle of the video processing unit after the processing is completed.
- the thread 2 queries the output it will be arranged and reported according to the frame number.
- MPSoC for the rear stage or video output unit to read in sequence.
- AXI4 Another interface is AXI4, as shown in Figure 2, through this interface, the video processing unit can read video data frames from the input video frame buffer in the HBM storage area, and write the processed video frames into the output video buffer at the same time.
- a corresponding HBM memory slice is allocated to each video processing unit.
- Each HBM memory slice includes an input buffer area and an output buffer area. Video frames to be processed are stored in the input buffer area, and processed video frames are stored in the output buffer area.
- the size of an independent HBM memory slice is 256MB, which is enough for two 8K video frame buffers. In this way, each video processing unit can only access the HBM storage slices designated for itself, and there are up to 32 such HBM storage slices in this embodiment, that is, there are also 32 corresponding video processing units.
- each video processing unit corresponds to its own address partition, and there is no cross access to each other.
- the AXI bus interface of the MPSoC control unit needs to use cross access, which means that all HBM memory slices can be accessed.
- Each memory slice of HBM is divided into I and O (input buffer and output buffer).
- the input buffer and output buffer can be further divided.
- Each HBM memory slice can also be set with at least two input buffer areas of equal capacity and at least two output buffer areas. When the output pauses for a long time to read the area, the system can switch the The input buffer and output buffer areas complete the input processing and output of the next round of video frames to improve the utilization of the video processing unit.
- the video processing unit Assuming that the video source is from MPSoC, such as a video decoder, it solves continuous video frames, the output resolution is 4K (3840*2160), and the frame rate is 120.
- the video processing unit namely the Video Process in Figure 2
- the video processing unit has video amplification. Function processing unit, therefore, in this embodiment, the video processing unit can be regarded as a video amplification processing unit, that is, corresponding to the video amplification processing unit shown in Figure 6, other units remain unchanged, first we determine the number of Video Process , which only needs to determine the time consumption of a single upscaled video processing unit.
- the value marked as enable is filled into the enable register to enable the first video processing unit.
- An electronic device 1 comprising a memory 3, a processor 2 and a computer program stored in the memory 3 and running on the processor 2, the processor 2 implements the video processing of the first or second embodiment when executing the computer program method.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computing Systems (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
- Television Systems (AREA)
Abstract
Provided are a video processing method, computer-readable storage medium, and electronic device, the video processing method comprising the steps: selecting a first video processing unit in an idle state from among a plurality of parallel video processing units to perform video processing on a video frame to be processed; writing the video frame to be processed to a first high-bandwidth memory (HBM) chip, and configuring and starting a first video processing unit, the first video processing unit being used for reading and processing the video frame to be processed from the first HBM memory chip, the processed video frame then being written to the first HBM memory chip; after the processing of the video frame to be processed by the first video processing unit is complete, reading and outputting the processed video frame from the first HBM memory chip; then, processing each video frame in the video as described above, to complete video processing.
Description
本申请要求于2020年9月28日提交的申请号为202011039945.6、名称为“视频处理方法、计算机可读存储介质及电子设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。This application claims the priority of the Chinese Patent Application No. 202011039945.6 and entitled "Video Processing Method, Computer-readable Storage Medium and Electronic Device" filed on September 28, 2020, the entire contents of which are incorporated herein by reference middle.
本发明涉及视频处理技术领域，特别涉及一种视频处理方法、计算机可读存储介质及电子设备。The present invention relates to the technical field of video processing, and in particular, to a video processing method, a computer-readable storage medium, and an electronic device.
随着人们对于视频清晰度的要求不断地增高，传统的2K、4K分辨率已经不能满足用户的日常需求，人们开始追求8K分辨率的视频，这就要求视频处理器也能够处理8K视频。由于8K视频的处理量是传统4K视频的4倍，若8K分辨率的视频同时要求120fps及以上的帧率的话，对于现有的主流视频处理器来说，无法同时满足存储带宽和处理速度上的需求。As people's requirements for video resolution continue to increase, the traditional 2K and 4K resolutions can no longer meet the daily needs of users, and people begin to pursue 8K resolution videos, which requires video processors to also be able to handle 8K videos. Since the processing capacity of 8K video is 4 times that of traditional 4K video, if the 8K resolution video requires a frame rate of 120fps and above at the same time, the existing mainstream video processors cannot meet the storage bandwidth and processing speed at the same time. demand.
如专利公开号CN104952037A提出了一种基于FPGA的JPEG图片文件的解压缩和缩放处理的加速的方法和***实现，其主要目的是为后台图片提供高速的图片缩放处理，对图片缩放首先要解压缩。所以其***设计偏重于PC或者服务器加FPGA平台的图片处理加速，这个架构不适合嵌入式视频处理加速或者提高帧率，另外存储器访问结构简单，无法突破内存带宽限制，也没有充分考虑专利中所描述的内核数目的匹配问题，5个甚至多个内核的访问内存带宽瓶颈问题没有充分考虑，因而只是简单的架构设计。专利公开号CN204761566U提出了一种基于FPGA和DSP(Digital Signal Process，数字信号处理单元)的实时图像处理实现方法，FPGA的作用是缓存预处理并分发图片给DSP单元，其图像处理的核心是DSP而不是FPGA，因此具有一定的局限性。For example, Patent Publication No. CN104952037A proposes an FPGA-based JPEG image file decompression and scaling process acceleration method and system implementation, the main purpose of which is to provide high-speed image scaling processing for background images, and the image scaling needs to be decompressed first . Therefore, its system design focuses on the image processing acceleration of PC or server plus FPGA platform. This architecture is not suitable for embedded video processing acceleration or frame rate improvement. In addition, the memory access structure is simple and cannot break through the memory bandwidth limitation. The matching problem of the number of cores described, the access memory bandwidth bottleneck of 5 or more cores is not fully considered, so it is only a simple architecture design. Patent publication number CN204761566U proposes a real-time image processing implementation method based on FPGA and DSP (Digital Signal Process, digital signal processing unit). The function of FPGA is to cache preprocessing and distribute pictures to DSP units. The core of its image processing is DSP rather than an FPGA, so it has certain limitations.
即现有缺乏一种视频处理方法，在处理高分辨率和高帧率的视频时，能够同时满足泛用性、存储带宽需求和处理速度需求。That is, there is currently a lack of a video processing method that can simultaneously meet the requirements of versatility, storage bandwidth and processing speed when processing high-resolution and high-frame-rate videos.
发明内容SUMMARY OF THE INVENTION
本申请一方面提供一种视频处理方法，包括步骤：One aspect of the present application provides a video processing method, comprising the steps of:
S1、从多个并行的视频处理单元中选取处于闲置状态的第一视频处理单元来对待处理视频帧进行视频处理；S1, from a plurality of parallel video processing units, select the first video processing unit in an idle state to perform video processing on the video frame to be processed;
S2、将所述待处理视频帧写入到所述第一视频处理单元所对应连接的第一HBM存储片中，配置并启动所述第一视频处理单元，所述第一视频处理单元用于从所述第一HBM存储片中读取并处理所述待处理视频帧，之后将处理后的已处理视频帧写入到所述第一HBM存储片中；S2. Write the to-be-processed video frame into the first HBM storage slice corresponding to the first video processing unit, configure and start the first video processing unit, and the first video processing unit is used for Read and process the to-be-processed video frame from the first HBM storage slice, and then write the processed video frame into the first HBM storage slice;
S3、判断所述第一视频处理单元对所述待处理视频帧的处理是否完成，若是，则从所述第一HBM存储片中读取并输出所述已处理视频帧；S3, determine whether the processing of the to-be-processed video frame by the first video processing unit is completed, and if so, read and output the processed video frame from the first HBM storage slice;
S4、将视频里的每一视频帧作为所述待处理视频帧，重复步骤S1-S3，完成视频处理。S4. Use each video frame in the video as the to-be-processed video frame, and repeat steps S1-S3 to complete the video processing.
本申请另一方面提供一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现如上所述的视频处理方法。Another aspect of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the above-mentioned video processing method.
本申请再一方面提供一种电子设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述计算机程序时实现如上所述的视频处理方法。Another aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, the processor implements the above video processing method when executing the computer program .
本申请再一方面提供一种视频处理装置，包括：Another aspect of the present application provides a video processing device, comprising:
视频单元选择模块，配置为从多个并行的视频处理单元中选取处于闲置状态的第一视频处理单元来对待处理视频帧进行视频处理；A video unit selection module, configured to select a first video processing unit in an idle state from a plurality of parallel video processing units to perform video processing on the video frame to be processed;
视频帧处理模块，配置为将所述待处理视频帧写入到所述第一视频处理单元所对应连接的第一HBM存储片中，配置并启动所述第一视频处理单元，所述第一视频处理单元用于从所述第一HBM存储片中读取并处理所述待处理视频帧，之后将处理后的已处理视频帧写入到所述第一HBM存储片中；以及A video frame processing module, configured to write the to-be-processed video frame into the first HBM storage slice corresponding to the first video processing unit, configure and start the first video processing unit, and the first video processing unit. a video processing unit for reading and processing the to-be-processed video frame from the first HBM storage slice, and then writing the processed video frame into the first HBM storage slice; and
判断模块，配置为判断所述第一视频处理单元对所述待处理视频帧的处理是否完成，若是，则从所述第一HBM存储片中读取并输出所述已处理视频帧。A judging module configured to judge whether the processing of the to-be-processed video frame by the first video processing unit is completed, and if so, read and output the processed video frame from the first HBM memory slice.
本发明的各个实施例的细节将在下面的附图和描述中进行说明。根据说明书、附图以及权利要求书的记载，本领域技术人员将容易理解本发明的其它特征、解决的问题以及有益效果。The details of various embodiments of the invention are set forth in the accompanying drawings and the description below. Those skilled in the art will easily understand other features, problems to be solved, and beneficial effects of the present invention from the description, drawings, and claims.
为了更好地描述和说明本申请的实施例，可参考一幅或多幅附图，但用于描述附图的附加细节或示例不应当被认为是对本申请的发明创造、目前所描述的实施例或优选方式中任何一者的范围的限制。In order to better describe and illustrate the embodiments of the present application, reference may be made to one or more drawings, but the additional details or examples used to describe the drawings should not be considered as invention-creations, presently described implementations of the present application A limitation of the scope of any one of the examples or preferred modes.
图1为本发明实施例的视频处理方法的流程示意图；1 is a schematic flowchart of a video processing method according to an embodiment of the present invention;
图2为本发明实施例的视频处理方法的整体框架示意图；2 is a schematic diagram of an overall framework of a video processing method according to an embodiment of the present invention;
图3为本发明实施例涉及的MPSoC主程序的视频处理线程示意图；3 is a schematic diagram of a video processing thread of an MPSoC main program involved in an embodiment of the present invention;
图4为本发明实施例涉及的视频处理单元的处理流程图；FIG. 4 is a processing flowchart of a video processing unit involved in an embodiment of the present invention;
图5为本发明实施例的视频处理方法的应用案例概况图；5 is an overview diagram of an application case of the video processing method according to an embodiment of the present invention;
图6为本发明实施例涉及的视频处理单元的框架示意图；6 is a schematic diagram of a framework of a video processing unit involved in an embodiment of the present invention;
图7为本发明实施例涉及的视频处理单元的时序示意图；7 is a schematic timing diagram of a video processing unit involved in an embodiment of the present invention;
图8为本发明实施例涉及的视频放大处理功能模块的时序示意图；8 is a schematic time sequence diagram of a video amplification processing functional module involved in an embodiment of the present invention;
图9为本发明实施例的一种***的结构示意图。FIG. 9 is a schematic structural diagram of a system according to an embodiment of the present invention.
为详细说明本发明的技术内容、所实现目的及效果，以下结合实施方式并配合附图予以说明。In order to describe in detail the technical content, achieved objects and effects of the present invention, the following descriptions are given with reference to the embodiments and the accompanying drawings.
请参照图1至图8，视频处理方法，包括步骤：Please refer to FIG. 1 to FIG. 8 , the video processing method includes the steps:
S1、从多个并行的视频处理单元中选取处于闲置状态的第一视频处理单元来对待处理视频帧进行视频处理；S1, from a plurality of parallel video processing units, select the first video processing unit in an idle state to perform video processing on the video frame to be processed;
S2、将所述待处理视频帧写入到所述第一视频处理单元所对应连接的第一HBM存储片中，配置并启动所述第一视频处理单元，所述第一视频处理单元用于从所述第一HBM存储片中读取并处理所述待处理视频帧，之后将处理后的已处理视频帧写入到所述第一HBM存储片中；S2. Write the to-be-processed video frame into the first HBM storage slice corresponding to the first video processing unit, configure and start the first video processing unit, and the first video processing unit is used for Read and process the to-be-processed video frame from the first HBM storage slice, and then write the processed video frame into the first HBM storage slice;
S3、判断所述第一视频处理单元对所述待处理视频帧的处理是否完成，若是，则从所述第一HBM存储片中读取并输出所述已处理视频帧；S3, determine whether the processing of the to-be-processed video frame by the first video processing unit is completed, and if so, read and output the processed video frame from the first HBM storage slice;
S4、将视频里的每一视频帧作为所述待处理视频帧，重复步骤S1-S3，完成视频处理。S4. Use each video frame in the video as the to-be-processed video frame, and repeat steps S1-S3 to complete the video processing.
其中，FPGA的MPSoC控制单元来执行上述步骤，MPSoC控制单元对于每一个待处理视频帧，都分配一个处于闲置状态的视频处理单元进行视频处理，由此，可以根据一个待处理视频帧的处理速度和所需要的帧率来配置视频单元的数目，视频处理单元的数目增大意味着并行处理的速度加快，也就意味着高的数据吞吐率。Among them, the MPSoC control unit of the FPGA performs the above steps. The MPSoC control unit allocates a video processing unit in an idle state for each video frame to be processed for video processing. Therefore, the processing speed of a video frame to be processed can be processed according to the speed of the video frame. And the required frame rate to configure the number of video units, the increase in the number of video processing units means faster parallel processing, which means high data throughput.
从上述描述可知，本发明的有益效果在于：在处理高分辨率和高帧率的视频时，通过多个并行的视频处理单元来对待处理视频帧进行视频处理，从而提高视频处理的处理速度；通过FPGA中自带的高带宽存储器HBM来进行待处理视频帧的写入和读取，从而实现数据吞吐率的提升，以满足存储带宽需求；本发明通过FPGA即可实现视频处理，从而具有泛用性，即本发明在处理高分辨率和高帧率的视频时，能够同时满足泛用性、存储带宽需求和处理速度需求。As can be seen from the above description, the beneficial effects of the present invention are: when processing video with high resolution and high frame rate, video processing is performed on the video frame to be processed by a plurality of parallel video processing units, thereby improving the processing speed of the video processing; The writing and reading of the video frames to be processed are carried out through the high-bandwidth memory HBM built in the FPGA, so as to improve the data throughput and meet the storage bandwidth requirements; the present invention can realize the video processing through the FPGA, so that it has a wide range of functions. Usability, that is, the present invention can satisfy universality, storage bandwidth requirements and processing speed requirements at the same time when processing high-resolution and high-frame-rate videos.
进一步地，所述视频处理单元包括控制和状态监控寄存器；Further, the video processing unit includes a control and state monitoring register;
所述步骤S2中配置并启动所述第一视频处理单元具体包括以下步骤：Configuring and starting the first video processing unit in the step S2 specifically includes the following steps:
填充所述第一视频处理单元的所述控制和状态监控寄存器，以启动所述第一视频处理单元；Filling the control and status monitoring registers of the first video processing unit to start the first video processing unit;
所述步骤S3中在得到并输出已处理视频帧之后还包括以下步骤：In the step S3, the following steps are further included after obtaining and outputting the processed video frame:
更改所述第一视频处理单元的所述控制和状态监控寄存器中的数值，以将所述第一视频处理单元更改为闲置状态。changing the value in the control and status monitoring register of the first video processing unit to change the first video processing unit to an idle state.
进一步地，所述控制和状态监控寄存器包括启动寄存器、帧序号寄存器、输入视频分辨率寄存器、输出视频分辨率寄存器以及处理完成寄存器，所述填充所述第一视频处理单元的所述控制和状态监控寄存器具体包括以下步骤：Further, the control and status monitoring registers include a startup register, a frame serial number register, an input video resolution register, an output video resolution register, and a processing completion register, and the control and status of the first video processing unit are filled. The monitoring register specifically includes the following steps:
将所述待处理视频帧在所述视频中的帧序号填入至所述帧序号寄存器；Filling the frame sequence number of the to-be-processed video frame in the video into the frame sequence number register;
将所述待处理视频帧的分辨率和所述已处理视频帧的分辨率分别填入所述输入视频分辨率寄存器和输出视频分辨率寄存器；Filling the input video resolution register and the output video resolution register with the resolution of the video frame to be processed and the resolution of the processed video frame respectively;
将标记为启动的数值填入至所述启动寄存器。The start register is filled with the value marked as start.
从上述描述可知，通过对视频处理单元的控制和状态监控寄存器进行数据填充，来控制和状态监控视频处理单元的工作状态，在完成数据处理后，设置视频处理单元为闲置状态，以继续分配给其他视频帧进行处理，从而使得视频处理单元能见缝插针的处理视频帧，并依次有序的输出视频帧，所有视频处理单元之间形成流水线操作，以实现数据吞吐率的提升。It can be seen from the above description that the control and status monitoring registers of the video processing unit are filled with data to control and monitor the working status of the video processing unit. Other video frames are processed, so that the video processing unit can process the video frames and output video frames in sequence, and a pipeline operation is formed between all video processing units to improve the data throughput rate.
进一步地，预设包括多个第一子线程的第一线程和包括多个第二子线程的第二线程；Further, preset a first thread including a plurality of first sub-threads and a second thread including a plurality of second sub-threads;
基于查询机制的多个所述第一子线程按照所述步骤S1和所述步骤S2对多个所述待处理视频帧进行并行处理；A plurality of the first sub-threads based on the query mechanism perform parallel processing on a plurality of the to-be-processed video frames according to the step S1 and the step S2;
基于查询机制的多个所述第二子线程按照所述步骤S3对多个所述待处理视频帧进行并行处理。The multiple second sub-threads based on the query mechanism perform parallel processing on the multiple to-be-processed video frames according to the step S3.
进一步地，所述步骤S2中启动所述第一视频处理单元具体包括以下步骤：Further, starting the first video processing unit in the step S2 specifically includes the following steps:
由所述第一子线程对所述待处理视频帧的写入进行实时监测，当所述待处理帧的写入数据还剩余预设阈值时，启动所述第一视频处理单元。The writing of the to-be-processed video frame is monitored in real time by the first sub-thread, and the first video processing unit is started when the write data of the to-be-processed frame still has a preset threshold value.
从上述描述可知，两个线程都是基于查询机制实现，其中，第一线程在执行时不断的确认待处理视频帧的输入状态，因为视频流是固定的输入输出分辨率，可以对视频输入的数据进行计量，得到一个预设阈值，到了预设阈值就可以提前置位标志位，启动第一线程里面的写视频处理单元寄存器操作，以提前开始启动视频处理功能模块，从而有效降低视频输出的时延。It can be seen from the above description that the two threads are implemented based on the query mechanism. The first thread continuously confirms the input status of the video frame to be processed during execution. Because the video stream has a fixed input and output resolution, it can The data is measured to obtain a preset threshold. When the preset threshold is reached, the flag bit can be set in advance, and the operation of writing the video processing unit register in the first thread is started to start the video processing function module in advance, thereby effectively reducing the video output. time delay.
进一步地，所述步骤S2中将待处理视频帧写入到所述第一视频处理单元所对应连接的第一HBM存储片中具体包括以下步骤：Further, writing the to-be-processed video frame into the first HBM storage slice corresponding to the first video processing unit in the step S2 specifically includes the following steps:
将所述待处理视频帧放入到AXI4总线，通过AXI4 Master端口向所述第一HBM存储片的AXI4 Slave端口发起突发传输，以将所述待处理视频帧写入到所述第一视频处理单元所对应连接的第一HBM存储片中；Put the to-be-processed video frame into the AXI4 bus, and initiate burst transmission to the AXI4 Slave port of the first HBM memory slice through the AXI4 Master port to write the to-be-processed video frame to the first video in the first HBM memory slice corresponding to the processing unit;
所述步骤S3中从所述第一HBM存储片中读取并输出所述已处理视频帧具体包括以下步骤：In the step S3, reading and outputting the processed video frame from the first HBM memory slice specifically includes the following steps:
通过AXI4 Master端口向所述第一HBM存储片的AXI4 Slave端口发起突发 传输，以从所述第一HBM存储片中读取并输出所述已处理视频帧。A burst transmission is initiated to the AXI4 Slave port of the first HBM memory slice through the AXI4 Master port to read and output the processed video frame from the first HBM memory slice.
进一步地，所述HBM存储片包括输入缓存区域和输出缓存区域，所述待处理视频帧存储到所述输入缓存区域，所述已处理视频帧存储到所述输出缓存区域。Further, the HBM storage slice includes an input buffer area and an output buffer area, the to-be-processed video frame is stored in the input buffer area, and the processed video frame is stored in the output buffer area.
从上述描述可知，FPGA的MPSoC控制单元通过AXI4端口以实现高性能地址映射通信，AXI4端口允许最大256轮的数据突发传输，由此视频处理功能模块也通过可以AXI4端口从HBM存储片中的输入缓存区读取待处理视频帧，同时把已处理视频帧写入输出视频缓存区域。It can be seen from the above description that the MPSoC control unit of the FPGA implements high-performance address mapping communication through the AXI4 port. The AXI4 port allows a maximum of 256 rounds of data burst transmission. Therefore, the video processing function module can also be transferred from the HBM memory chip through the AXI4 port. The input buffer area reads the video frames to be processed, and at the same time writes the processed video frames into the output video buffer area.
进一步地，若所述待处理视频帧的分辨率为4K且所述已处理视频帧的分辨率为8K时，所述视频处理单元和对应的HBM存储片均为8个。Further, if the resolution of the to-be-processed video frame is 4K and the resolution of the processed video frame is 8K, both the video processing unit and the corresponding HBM storage slices are 8.
从上述描述可知，单个视频处理单元处理一帧4K放大到8K需要时间为60ms，这60ms包括从内存读取视频的时间以及视频写入内存的时间，如果在只有一个视频处理单元的情况下，帧率只能实现1000ms/60ms＝16.67帧，而本发明采用了8个视频处理单元和对应的HBM存储片，由此，帧率最高可到达133.33>120帧，由此，可以在实现8K分辨率的同时使得帧数可以达到120帧，从而达到高分辨率和高帧率的视频需求。It can be seen from the above description that it takes 60ms for a single video processing unit to process a frame of 4K to 8K. This 60ms includes the time to read the video from the memory and the time to write the video to the memory. If there is only one video processing unit, The frame rate can only achieve 1000ms/60ms=16.67 frames, and the present invention adopts 8 video processing units and corresponding HBM memory slices, thus, the frame rate can reach up to 133.33>120 frames, thus, it can achieve 8K resolution At the same time, the number of frames can reach 120 frames, so as to meet the video requirements of high resolution and high frame rate.
本发明另一实施方式提供了一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现如上各实施例所述的视频处理方法。Another embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the video processing methods described in the above embodiments.
请参照图9，本发明另一实施方式提供了一种电子设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述计算机程序时实现如上各实施例所述的视频处理方法。Referring to FIG. 9 , another embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor implements the computer program when executing the computer program. The video processing methods described in the above embodiments.
其中，关于上述两种实施方式中的计算机程序所包含的视频处理方法的具体实现过程和对应效果，可以参照前面实施方式的视频处理方法中的相关描述。For the specific implementation process and corresponding effects of the video processing methods included in the computer programs in the above two embodiments, reference may be made to the relevant descriptions in the video processing methods in the previous embodiments.
本申请的视频处理方法和对应的电路、装置、存储介质及电子设备主要应用于对任何视频进行处理的应用场景，比如视频解压缩和缩放处理等，以下结合具体的应用场景进行说明：The video processing method and corresponding circuit, device, storage medium and electronic device of the present application are mainly used in application scenarios for processing any video, such as video decompression and scaling processing, etc. The following description is combined with specific application scenarios:
请参照图1至图8，本发明的实施例一为：Please refer to FIG. 1 to FIG. 8 , the first embodiment of the present invention is:
为了便于理解本实施例，对本实施例涉及到的英文进行简单说明：In order to facilitate the understanding of this embodiment, the English involved in this embodiment is briefly described:
FPGA：可编程大规模逻辑阵列FPGA: Programmable Large Scale Logic Array
ARM：通用嵌入式处理器CPU；ARM: general-purpose embedded processor CPU;
HBM：High bandwidth Memory，高带宽存储器；HBM: High bandwidth Memory, high bandwidth memory;
Video Process：视频处理单元；Video Process: video processing unit;
MPSoC FPGA：可编程带多核ARM处理器的FPGA；MPSoC FPGA: Programmable FPGA with multi-core ARM processor;
M:master，主控端；M:master, the main control terminal;
S:slave，从端；S:slave, from the end;
AXI_lite:轻量级ARM总线，用于寄存器访问；AXI_lite: lightweight ARM bus for register access;
AXI4:地址映射ARM总线、用于访问大容量内存；AXI4: address mapping ARM bus, used to access large-capacity memory;
I:Input buffer，输入缓存；I:Input buffer, input buffer;
O:Output buffer，输出缓存；O:Output buffer, output buffer;
Burst传输:突发传输，单次读写请求可以连续多达256个连续读写动作。Burst transfer: Burst transfer, a single read and write request can have up to 256 consecutive read and write actions.
由此，本实施例提供的一种视频处理方法，包括步骤：Thus, a video processing method provided by this embodiment includes the steps:
S1、从多个并行的视频处理单元中选取处于闲置状态的第一视频处理单元来对待处理视频帧进行视频处理；S1, from a plurality of parallel video processing units, select the first video processing unit in an idle state to perform video processing on the video frame to be processed;
其中，FPGA包括MPSoC控制单元、视频处理单元和HBM存储器件，视频处理单元需要在FPGA中开发实现，MPSoC和HBM都是FPGA中已经包含并存在的单元。视频处理单元的数目根据实际的视频处理需求进行配置，视频处理单元的数目的增大意味着并行处理的速度，也就意味着高的数据吞吐率。其中，HBM存储器件提供高带宽存取，用于对应视频处理单元的输入视频帧缓存和输出视频帧缓存。其中，MPSoC控制单元是协调多个视频处理单元高效率并行工作的控制核心，即MPSoC控制单元来执行本实施例的步骤，MPSoC控制单元内设置有视频处理主程序，该视频处理主程序的主要任务有两个，一个任务是视频处理单元配置，另外一个任务是视频处理单元的任务调度。Among them, the FPGA includes the MPSoC control unit, the video processing unit and the HBM storage device. The video processing unit needs to be developed and implemented in the FPGA. Both MPSoC and HBM are units already included and existing in the FPGA. The number of video processing units is configured according to actual video processing requirements. The increase in the number of video processing units means the speed of parallel processing, which means high data throughput. Among them, the HBM storage device provides high-bandwidth access for the input video frame buffer and the output video frame buffer of the corresponding video processing unit. The MPSoC control unit is a control core that coordinates the efficient parallel operation of multiple video processing units, that is, the MPSoC control unit executes the steps of this embodiment. The MPSoC control unit is provided with a video processing main program, and the main video processing main program is There are two tasks, one is the video processing unit configuration, and the other is the task scheduling of the video processing unit.
对于视频处理单元配置这个任务，如图6所示，每一个视频处理单元配置都是在FPGA中根据视频处理功能需求实现的功能模块，再配上都需要包含的两个固定接口单元，一个接口是AXI_Lite从接口，它是一个轻量级的地址映射单次传输接口，占用很少的逻辑单元，主要是用于视频处理单元控制和状态监 控寄存器的读写，视频处理单元的控制和状态监控寄存器是为了控制视频处理功能模块的运行控制和状态监控。For the task of video processing unit configuration, as shown in Figure 6, each video processing unit configuration is a functional module implemented in the FPGA according to the video processing function requirements, coupled with two fixed interface units that need to be included, one interface It is the AXI_Lite slave interface, which is a lightweight address mapping single transfer interface, occupying very few logic units, mainly used for reading and writing the video processing unit control and status monitoring registers, and the control and status monitoring of the video processing unit. The register is used to control the operation control and status monitoring of the video processing function module.
表1.控制和状态监控寄存器列表Table 1. List of Control and Status Monitoring Registers
如表1所示，这是一个定义的标准的通用控制和状态监控寄存器列表，包括用于运行控制的启动寄存器、用于记录当前帧的序号的帧序号寄存器、用于记录输入视频帧分辨率的输入视频分辨率寄存器、用于记录输出视频帧分辨率的输出视频分辨率寄存器、用于标记当前帧是否处理完毕处理完成寄存器以及用于标记当前帧处理是否出现错误的错误指示寄存器。As shown in Table 1, this is a list of defined standard general-purpose control and status monitoring registers, including a start register for running control, a frame number register for recording the sequence number of the current frame, and a frame number register for recording the input video frame resolution The input video resolution register, the output video resolution register for recording the resolution of the output video frame, the processing completion register for marking whether the current frame has been processed, and the error indicating register for marking whether there is an error in the processing of the current frame.
其中，有关帧序号寄存器说明如下：待处理视频帧作为输入帧以及已处理视频帧作为输出帧都有对应的帧序号，为了防止帧率提升或者降低的情况，每个视频处理单元都有一个帧序号寄存器，当写入处理视频帧的时候，为这个处理视频帧进行排序，然后把序号值写入这个帧序号寄存器，这样视频帧处理器在处理结束后，就可以根据视频处理单元的原理设置输出帧序号到输出帧寄存器，如果帧率不变就按照等值设置，如果帧率扩大一倍，就编号输出两个对应的帧序号.线程二在查询输出的时候，就按帧序号排列上报给MPSoC，供后级或视频输出单元按顺序读取。Among them, the relevant frame serial number registers are described as follows: the video frame to be processed as the input frame and the processed video frame as the output frame have corresponding frame serial numbers. In order to prevent the frame rate from increasing or decreasing, each video processing unit has a frame The serial number register, when writing the processing video frame, sort the processing video frame, and then write the serial number value into the frame serial number register, so that the video frame processor can be set according to the principle of the video processing unit after the processing is completed. Output the frame number to the output frame register. If the frame rate remains unchanged, set it according to the same value. If the frame rate is doubled, the number will output two corresponding frame numbers. When the thread 2 queries the output, it will be arranged and reported according to the frame number. To MPSoC, for the rear stage or video output unit to read in sequence.
另外一个接口是AXI4，如图2所示，通过这个接口，视频处理单元可以从HBM的存储区中的输入视频帧缓存区读取视频数据帧，同时把处理后的视频帧写入输出视频缓存区。其中，为每个视频处理单元分配了对应的HBM存储片，每一个HBM存储片包括输入缓存区域和输出缓存区域，待处理视频帧存储到输入缓存区域，已处理视频帧存储到输出缓存区域。一个独立的HBM存储片的容量大小是256MB，足够两个8K视频帧缓存。这样每个视频处理单元只能够访问给自己划定的HBM存储片，这样的HBM存储片在本实施例中多达32个，即对应的视频处理单元也有32个。Another interface is AXI4, as shown in Figure 2, through this interface, the video processing unit can read video data frames from the input video frame buffer in the HBM storage area, and write the processed video frames into the output video buffer at the same time. Area. A corresponding HBM memory slice is allocated to each video processing unit. Each HBM memory slice includes an input buffer area and an output buffer area. Video frames to be processed are stored in the input buffer area, and processed video frames are stored in the output buffer area. The size of an independent HBM memory slice is 256MB, which is enough for two 8K video frame buffers. In this way, each video processing unit can only access the HBM storage slices designated for itself, and there are up to 32 such HBM storage slices in this embodiment, that is, there are also 32 corresponding video processing units.
其中，针对HBM的访问机制设计说明如下:每一个视频处理单元都单独对应各自的地址分区，相互没有交叉访问。而MPSoC控制单元的AXI总线接口需要使用交叉访问，也就是说可以访问所有的HBM存储片，HBM的每一个存储片分为I和O(input buffer和output buffer)。input buffer和output buffer可以再支持划分，每个HBM存储片还可以设置至少两个等容量input buffer区域和至少两个output buffer区域，当输出暂停较长时间读取区域的时候，***可以切换下input buffer和output buffer区域，完成下一轮视频帧的输入处理和输出，以提高视频处理单元的利用率，Among them, the design description of the access mechanism for HBM is as follows: each video processing unit corresponds to its own address partition, and there is no cross access to each other. The AXI bus interface of the MPSoC control unit needs to use cross access, which means that all HBM memory slices can be accessed. Each memory slice of HBM is divided into I and O (input buffer and output buffer). The input buffer and output buffer can be further divided. Each HBM memory slice can also be set with at least two input buffer areas of equal capacity and at least two output buffer areas. When the output pauses for a long time to read the area, the system can switch the The input buffer and output buffer areas complete the input processing and output of the next round of video frames to improve the utilization of the video processing unit.
对于视频处理单元的任务调度这个任务，视频处理主程序通过建立两个线程，分别为包括多个第一子线程的第一线程和包括多个第二子线程的第二线程。在本实施例，即对应图3所示，第一线程为视频数据填充线程，主要完成视频帧的input buffer输入，以及对应的视频处理单元的寄存器配置和启动；第二线程就是视频处理单元监控线程，负责检测视频处理单元的工作状态，比如是否正常完成或是否发生错误，并报告视频处理主程序。其中每个子线程的数目根据输入输出视频帧率需要动态增加或减少。For the task scheduling task of the video processing unit, the video processing main program establishes two threads, respectively a first thread including multiple first sub-threads and a second thread including multiple second sub-threads. In this embodiment, corresponding to Fig. 3, the first thread is the video data filling thread, which mainly completes the input buffer input of the video frame, and the register configuration and startup of the corresponding video processing unit; the second thread is the video processing unit monitoring. The thread is responsible for detecting the working status of the video processing unit, such as whether it is completed normally or whether an error occurs, and reports the video processing main program. The number of each sub-thread is dynamically increased or decreased according to the input and output video frame rates.
由此，在本实施例中，以视频放大处理为例子进行说明。Therefore, in the present embodiment, the video enlargement processing will be described as an example.
假设视频源是来自MPSoC，比如视频解码器，解出连续的视频帧，输出分辨率为4K(3840*2160)，帧率120，这个时候的视频处理单元即图2中Video Process就是具有视频放大功能的处理单元，因此，在本实施例中，可将视频处理单元视为视频放大处理单元，即对应图6所示的视频放大处理单元，其他单元保持不变，首先我们确定Video Process的数目，这只需要确定单个放大视频处理单元的时间消耗即可。Assuming that the video source is from MPSoC, such as a video decoder, it solves continuous video frames, the output resolution is 4K (3840*2160), and the frame rate is 120. At this time, the video processing unit, namely the Video Process in Figure 2, has video amplification. Function processing unit, therefore, in this embodiment, the video processing unit can be regarded as a video amplification processing unit, that is, corresponding to the video amplification processing unit shown in Figure 6, other units remain unchanged, first we determine the number of Video Process , which only needs to determine the time consumption of a single upscaled video processing unit.
由于4K的视频帧放大到8K的视频帧，单个视频处理单元处理一帧4K放大到8K需要时间为60ms，这60ms包括从内存读取视频的时间以及写入内存视频的时间。那么如果在只有一个视频处理单元的情况下，帧率只能实现1000ms/60ms＝16.67帧，如果***要求8K分辨率的情况下实现120帧，这种情况下就要考虑采用本文设计的视频处理***，并行的视频处理单元的数目为120帧/16.67帧＝8(向上取整)。即本实施例中视频处理单元和对应的HBM存储片 均为8个。Since a 4K video frame is enlarged to an 8K video frame, it takes 60ms for a single video processing unit to process a frame of 4K to 8K. This 60ms includes the time to read the video from the memory and the time to write the video into the memory. Then if there is only one video processing unit, the frame rate can only achieve 1000ms/60ms=16.67 frames, and if the system requires 8K resolution to achieve 120 frames, in this case, the video processing designed in this paper should be considered. In the system, the number of parallel video processing units is 120 frames/16.67 frames=8 (rounded up). That is, in this embodiment, there are 8 video processing units and corresponding HBM memory slices.
由此，设置两个线程启动，之后在步骤S1中，MPSoC控制单元检测MPSoC视频解码器是否有视频帧解码输出，若有，则存在待处理视频帧需要放大处理，此时MPSoC控制单元检测处于闲置状态的视频处理单元，比如第一视频处理单元闲置，就分配给第一视频处理单元。Therefore, two threads are set to start, and then in step S1, the MPSoC control unit detects whether the MPSoC video decoder has a video frame decoding output. If so, there are video frames to be processed that need to be amplified. A video processing unit in an idle state, such as the first video processing unit being idle, is allocated to the first video processing unit.
S2、将待处理视频帧写入到第一视频处理单元所对应连接的第一HBM存储片中，配置并启动第一视频处理单元，第一视频处理单元用于从第一HBM存储片中读取并处理待处理视频帧，之后将处理后的已处理视频帧写入到第一HBM存储片中；S2. Write the video frame to be processed into the first HBM storage slice corresponding to the first video processing unit, configure and start the first video processing unit, and the first video processing unit is used to read from the first HBM storage slice Get and process the video frame to be processed, and then write the processed video frame into the first HBM storage slice;
如图5所示，在本步骤中，把输出的待处理视频帧写入到视频处理单元对应的HBM存储片的输入缓存区域，即本实施例中为写入到第一HBM存储片中，具体包括以下步骤：As shown in FIG. 5 , in this step, the output to-be-processed video frame is written into the input buffer area of the HBM storage slice corresponding to the video processing unit, that is, in this embodiment, it is written into the first HBM storage slice, Specifically include the following steps:
将待处理视频帧放入到AXI4总线，通过AXI4 Master端口向第一HBM存储片的AXI4 Slave端口发起突发传输，以将待处理视频帧写入到第一视频处理单元所对应连接的第一HBM存储片中；Put the to-be-processed video frame into the AXI4 bus, and initiate a burst transmission to the AXI4 Slave port of the first HBM memory slice through the AXI4 Master port to write the to-be-processed video frame to the first video processing unit corresponding to the first video processing unit. In HBM memory slice;
在写入完成后，填充第一视频处理单元的控制和状态监控寄存器，以启动第一视频处理单元，具体包括以下步骤：After the writing is completed, the control and state monitoring registers of the first video processing unit are filled to start the first video processing unit, which specifically includes the following steps:
将待处理视频帧在视频中的帧序号填入至帧序号寄存器；Fill the frame serial number of the video frame to be processed in the video into the frame serial number register;
将待处理视频帧的分辨率和已处理视频帧的分辨率分别填入输入视频分辨率寄存器和输出视频分辨率寄存器；Fill the input video resolution register and the output video resolution register with the resolution of the video frame to be processed and the resolution of the processed video frame respectively;
将标记为启动的数值填入至启动寄存器，以启动第一视频处理单元。The value marked as enable is filled into the enable register to enable the first video processing unit.
之后的步骤由视频处理单元进行视频处理，为了便于理解本发明，本实施例以图8为例对视频处理单元的处理作以下举例说明：The following steps are performed by the video processing unit. In order to facilitate understanding of the present invention, the present embodiment uses FIG. 8 as an example to illustrate the processing of the video processing unit as follows:
视频处理单元在本实施例即为视频放大处理单元，它包括双线性放大处理模块、FIFO缓存、位宽转换模块和AXI4突发传输读写控制模块。MPSoC控制单元启动视频处理单元后，双线性放大处理模块开始请求数据，请求命令最终传递到AXI4突发传输读控制模块，它最终会从HBM存储片中读取待处理视频帧的数据，双线性放大处理模块得到需要的数据后，经过双线性放大处理，就 把数据输出到AXI4突发写传输控制模块，最终写入HBM存储片中。由于数据的读写是突发传输，所以都需要一个FIFO缓存突发传输数据，另外双线性放大处理是像素级处理，所以需要把256bit(HBM的AXI4总线接口数据位宽)转换成36bit(一个像素的数据位宽)。The video processing unit is a video amplification processing unit in this embodiment, which includes a bilinear amplification processing module, a FIFO buffer, a bit width conversion module and an AXI4 burst transmission read-write control module. After the MPSoC control unit starts the video processing unit, the bilinear amplification processing module starts to request data, and the request command is finally transmitted to the AXI4 burst transmission read control module, which will eventually read the data of the video frame to be processed from the HBM memory slice. After the linear amplification processing module obtains the required data, after bilinear amplification processing, the data is output to the AXI4 burst write transmission control module, and finally written into the HBM memory chip. Since the reading and writing of data is burst transmission, a FIFO buffer is required for burst transmission of data. In addition, the bilinear amplification processing is pixel-level processing, so it is necessary to convert 256bit (HBM's AXI4 bus interface data bit width) into 36bit ( data bit width of one pixel).
S3、判断第一视频处理单元对待处理视频帧的处理是否完成，若是，则从第一HBM存储片中读取并输出已处理视频帧；S3, determine whether the processing of the video frame to be processed by the first video processing unit is completed, and if so, read and output the processed video frame from the first HBM storage slice;
由此，基于查询机制的第二子线程实时监测这个视频处理单元是否完成所述待处理视频的放大，若判断已处理完成，则通过AXI4 Master端口向第一HBM存储片的AXI4 Slave端口发起突发传输，以从第一HBM存储片中读取并输出已处理视频帧，之后更改第一视频处理单元的控制和状态监控寄存器中的数值，以将第一视频处理单元更改为闲置状态。Thus, the second sub-thread based on the query mechanism monitors in real time whether the video processing unit completes the amplification of the to-be-processed video, and if it is judged that the processing has been completed, a burst is initiated to the AXI4 Slave port of the first HBM memory slice through the AXI4 Master port. A transmission is sent to read and output the processed video frame from the first HBM memory slice, and then the value in the control and status monitoring register of the first video processing unit is changed to change the first video processing unit to the idle state.
S4、将视频里的每一视频帧作为待处理视频帧，重复步骤S1-S3，完成视频处理。S4. Take each video frame in the video as a to-be-processed video frame, and repeat steps S1-S3 to complete the video processing.
由此，如果当第一视频处理单元在进行放大处理时，视频里的下一视频帧解码出来，则需要另外一个第一子线程和另外一个第二子线程，另外一个第一子线程分配另外一个视频处理单元，比如第二视频处理单元和对应的第二HBM存储片来进行同样的操作，而另外一个第二子线程同样也是并行处理。Therefore, if the next video frame in the video is decoded when the first video processing unit is performing the enlargement process, another first sub-thread and another second sub-thread are required, and another first sub-thread allocates another A video processing unit, such as the second video processing unit and the corresponding second HBM memory slice, perform the same operation, and another second sub-thread also performs parallel processing.
这样视频处理主程序的两个线程如图4所示，不断的按照上述步骤把视频里的每一视频帧依次写入到HBM，视频处理单元见缝插针的处理视频帧，并依次有序的输出视频帧，视频处理单元之间形成流水线操作。In this way, the two threads of the video processing main program are shown in Figure 4, and continuously write each video frame in the video to the HBM according to the above steps. The video processing unit processes the video frames and outputs the video in sequence. Frame, pipeline operation is formed between video processing units.
由上可知：It can be seen from the above:
1、从带宽需求上来看，假设每一个视频帧的色深为12bit，则4K的待处理视频帧的输入信号带宽:3840*2160*36*120fps＝33.38Gbps，8K的已处理视频帧的读出信号带宽:7680*4320*36*120fps＝133.49Gbps，则存储器件输入输出总带宽:133.49Gbps+33.38Gbps＝166.87Gbps，每秒钟接近167Gbps的数据吞吐率，如果拿普通的顶级DDR4-4266的带宽:4266M*64＝266.625Gbps，虽然理论上够用，但是考虑到有效率问题，也是比较紧张的，而本实施例中使用的HBM存储器件，其带宽为460G*8＝3680Gbps，由此可以看出HBM的巨大的带宽优势， 也为我们做更高视频吞吐率的视频处理提供了物理基础。1. From the perspective of bandwidth requirements, assuming that the color depth of each video frame is 12bit, the input signal bandwidth of the 4K video frame to be processed: 3840*2160*36*120fps=33.38Gbps, and the reading of the 8K processed video frame Output signal bandwidth: 7680*4320*36*120fps=133.49Gbps, then the total input and output bandwidth of the storage device: 133.49Gbps+33.38Gbps=166.87Gbps, the data throughput rate per second is close to 167Gbps, if you take the ordinary top DDR4-4266 The bandwidth: 4266M*64=266.625Gbps, although it is sufficient in theory, it is also relatively tight considering the efficiency issue, and the bandwidth of the HBM storage device used in this embodiment is 460G*8=3680Gbps, thus It can be seen that the huge bandwidth advantage of HBM also provides a physical basis for us to do video processing with higher video throughput.
2、从数据吞吐率来看，4K的待处理视频帧和8K的已处理视频帧在数据量上相差四倍，即输出数据吞吐率比输入数据吞吐率提高了4倍。2. From the perspective of data throughput rate, the data volume of 4K unprocessed video frames and 8K processed video frames differs by four times, that is, the output data throughput rate is four times higher than the input data throughput rate.
3、从处理速度上来看，如图7所示，在第一个待处理视频帧输入60S之后才输出，之后以120fps的时间间隔1000/120＝8.33ms源源不断的输出8K的视频帧，即原先一个视频处理单元处理一个视频帧是60ms，而本实施例相当于是8.33ms，则处理速度提高了7.2倍。3. From the perspective of processing speed, as shown in Figure 7, the first video frame to be processed is output after 60S input, and then 8K video frames are continuously output at 120fps time interval 1000/120=8.33ms, that is Originally, it takes 60ms for one video processing unit to process one video frame, but this embodiment is equivalent to 8.33ms, and the processing speed is increased by 7.2 times.
当然，本实施例以4K放大到8K进行说明，但对于不同高分辨率、不同高帧率的要求以及不同的视频处理操作来说，均能达到更大的数据吞吐率和更快的处理速度，因此，上述情况应当属于本发明的等同实施例。Of course, this embodiment is explained by zooming in from 4K to 8K, but for different high resolutions, different high frame rate requirements, and different video processing operations, a larger data throughput rate and faster processing speed can be achieved. , therefore, the above-mentioned situations should belong to the equivalent embodiments of the present invention.
请参照图1至图8，本发明的实施例二为：Please refer to FIG. 1 to FIG. 8 , the second embodiment of the present invention is:
本实施例提供的一种视频处理方法，在上述实施例一的基础上，步骤S2中启动第一视频处理单元具体包括以下步骤：In the video processing method provided by this embodiment, on the basis of the above-mentioned first embodiment, starting the first video processing unit in step S2 specifically includes the following steps:
由第一子线程对待处理视频帧的写入进行实时监测，当待处理帧的写入数据还剩余预设阈值时，启动第一视频处理单元。The writing of the to-be-processed video frame is monitored in real time by the first sub-thread, and the first video processing unit is started when the written data of the to-be-processed frame still has a preset threshold value.
其中，在进行数据写入时，数据位宽为256bit，时钟400MHz，写入一个4K的待处理视频帧理论上大致需要3840*2160*36/256/400M＝29.16us，读出一个8K的已处理视频帧大致就需要4倍时间，都是微秒级别，而在本实施例中，预设阈值可以为1/8，即29.16-29.16/8＝25.515us，就启动第一视频处理单元，这样后续的3.645us用于配置响应，以进一步缩小视频处理时间，提高处理速度。Among them, when writing data, the data bit width is 256bit, and the clock is 400MHz. In theory, writing a 4K video frame to be processed requires approximately 3840*2160*36/256/400M=29.16us, and reading an 8K video frame It takes roughly 4 times as long to process a video frame, all of which are at the microsecond level. In this embodiment, the preset threshold can be 1/8, that is, 29.16-29.16/8=25.515us, and the first video processing unit is activated. In this way, the subsequent 3.645us is used to configure the response to further reduce the video processing time and improve the processing speed.
本发明的实施例三为：The third embodiment of the present invention is:
本发明另一实施方式提供了一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现上述实施例一或实施例二的视频处理方法。Another embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the video processing method of the first embodiment or the second embodiment.
请参照图9，本发明的实施例四为：Please refer to FIG. 9 , the fourth embodiment of the present invention is:
一种电子设备1，包括存储器3、处理器2及存储在存储器3上并可在处理器2上运行的计算机程序，处理器2执行计算机程序时实现上述实施例一或实施例二的视频处理方法。An electronic device 1, comprising a memory 3, a processor 2 and a computer program stored in the memory 3 and running on the processor 2, the processor 2 implements the video processing of the first or second embodiment when executing the computer program method.
综上所述，本发明提供的一种视频处理方法、计算机可读存储介质及电子设备，在处理高分辨率和高帧率的视频时，通过多个并行的视频处理单元来对待处理视频帧进行视频处理，从而提高视频处理的处理速度；通过FPGA中自带的高带宽存储器HBM来进行待处理视频帧的写入和读取，从而实现数据吞吐率的提升，以满足存储带宽需求；本发明通过FPGA即可实现视频处理，结合FPGA可配置的特点，可以灵活的根据需求动态增加或减少视频处理单元的数目，以及更换视频处理单元的种类，而没有任何成本代价的产生，从而具有泛用性，即本发明在处理高分辨率和高帧率的视频时，能够同时满足泛用性、存储带宽需求和处理速度需求。To sum up, the present invention provides a video processing method, a computer-readable storage medium and an electronic device. When processing high-resolution and high-frame-rate videos, multiple parallel video processing units are used to process video frames to be processed. Perform video processing to improve the processing speed of video processing; write and read video frames to be processed through the built-in high-bandwidth memory HBM in the FPGA, thereby improving data throughput to meet storage bandwidth requirements; this The invention can realize video processing through FPGA. Combined with the configurable characteristics of FPGA, the number of video processing units can be dynamically increased or decreased according to requirements, and the types of video processing units can be changed without any cost. Usability, that is, the present invention can satisfy universality, storage bandwidth requirement and processing speed requirement at the same time when processing high resolution and high frame rate video.
以上所述仅为本发明的实施例，并非因此限制本发明的专利范围，凡是利用本发明说明书及附图内容所作的等同变换，或直接或间接运用在相关的技术领域，均同理包括在本发明的专利保护范围内。The above descriptions are only examples of the present invention, and are not intended to limit the scope of the patent of the present invention. Any equivalent transformations made by using the contents of the description and drawings of the present invention, or directly or indirectly applied in related technical fields, are similarly included in the within the scope of patent protection of the present invention.
Claims (11)
- 一种视频处理方法，包括步骤：A video processing method, comprising the steps of:S1、从多个并行的视频处理单元中选取处于闲置状态的第一视频处理单元来对待处理视频帧进行视频处理；S1, from a plurality of parallel video processing units, select the first video processing unit in an idle state to perform video processing on the video frame to be processed;S2、将所述待处理视频帧写入到所述第一视频处理单元所对应连接的第一HBM存储片中，配置并启动所述第一视频处理单元，所述第一视频处理单元用于从所述第一HBM存储片中读取并处理所述待处理视频帧，之后将处理后的已处理视频帧写入到所述第一HBM存储片中；S2. Write the to-be-processed video frame into the first HBM storage slice corresponding to the first video processing unit, configure and start the first video processing unit, and the first video processing unit is used for Read and process the to-be-processed video frame from the first HBM storage slice, and then write the processed video frame into the first HBM storage slice;S3、判断所述第一视频处理单元对所述待处理视频帧的处理是否完成，若是，则从所述第一HBM存储片中读取并输出所述已处理视频帧；S3, determine whether the processing of the to-be-processed video frame by the first video processing unit is completed, and if so, read and output the processed video frame from the first HBM storage slice;S4、将视频里的每一视频帧作为所述待处理视频帧，重复步骤S1-S3，完成视频处理。S4. Use each video frame in the video as the to-be-processed video frame, and repeat steps S1-S3 to complete the video processing.
- 根据权利要求1所述的视频处理方法，其中，所述视频处理单元包括控制和状态监控寄存器；The video processing method of claim 1, wherein the video processing unit comprises a control and status monitoring register;所述步骤S2中配置并启动所述第一视频处理单元具体包括以下步骤：Configuring and starting the first video processing unit in the step S2 specifically includes the following steps:填充所述第一视频处理单元的所述控制和状态监控寄存器，以启动所述第一视频处理单元；Filling the control and status monitoring registers of the first video processing unit to start the first video processing unit;所述步骤S3中在得到并输出已处理视频帧之后还包括以下步骤：In the step S3, the following steps are further included after obtaining and outputting the processed video frame:更改所述第一视频处理单元的所述控制和状态监控寄存器中的数值，以将所述第一视频处理单元更改为闲置状态。changing the value in the control and status monitoring register of the first video processing unit to change the first video processing unit to an idle state.
- 根据权利要求2所述的视频处理方法，其中，所述控制和状态监控寄存器包括启动寄存器、帧序号寄存器、输入视频分辨率寄存器、输出视频分辨率寄存器以及处理完成寄存器，所述填充所述第一视频处理单元的所述控制和状态监控寄存器具体包括以下步骤：The video processing method according to claim 2, wherein the control and status monitoring registers include a start register, a frame sequence number register, an input video resolution register, an output video resolution register, and a processing completion register, and the filling in the first The control and state monitoring registers of a video processing unit specifically include the following steps:将所述待处理视频帧在所述视频中的帧序号填入至所述帧序号寄存器；Filling the frame sequence number of the to-be-processed video frame in the video into the frame sequence number register;将所述待处理视频帧的分辨率和所述已处理视频帧的分辨率分别填入所述输入视频分辨率寄存器和输出视频分辨率寄存器；Filling the input video resolution register and the output video resolution register with the resolution of the video frame to be processed and the resolution of the processed video frame respectively;将标记为启动的数值填入至所述启动寄存器。The start register is filled with the value marked as start.
- 根据权利要求1所述的视频处理方法，其中，预设包括多个第一子线程的第一线程和包括多个第二子线程的第二线程；The video processing method according to claim 1, wherein a first thread including a plurality of first sub-threads and a second thread including a plurality of second sub-threads are preset;基于查询机制的多个所述第一子线程按照所述步骤S1和所述步骤S2对多个所述待处理视频帧进行并行处理；A plurality of the first sub-threads based on the query mechanism perform parallel processing on a plurality of the to-be-processed video frames according to the step S1 and the step S2;基于查询机制的多个所述第二子线程按照所述步骤S3对多个所述待处理视频帧进行并行处理。The multiple second sub-threads based on the query mechanism perform parallel processing on the multiple to-be-processed video frames according to the step S3.
- 根据权利要求4所述的视频处理方法，其中，所述步骤S2中启动所述第一视频处理单元具体包括以下步骤：The video processing method according to claim 4, wherein in the step S2, starting the first video processing unit specifically includes the following steps:由所述第一子线程对所述待处理视频帧的写入进行实时监测，当所述待处理帧的写入数据还剩余预设阈值时，启动所述第一视频处理单元。The writing of the to-be-processed video frame is monitored in real time by the first sub-thread, and the first video processing unit is started when the write data of the to-be-processed frame still has a preset threshold value.
- 根据权利要求1所述的视频处理方法，其中，所述步骤S2中将待处理视频帧写入到所述第一视频处理单元所对应连接的第一HBM存储片中具体包括以下步骤：The video processing method according to claim 1, wherein in the step S2, writing the to-be-processed video frame into the first HBM storage slice correspondingly connected to the first video processing unit specifically includes the following steps:将所述待处理视频帧放入到AXI4总线，通过AXI4 Master端口向所述第一HBM存储片的AXI4 Slave端口发起突发传输，以将所述待处理视频帧写入到所述第一视频处理单元所对应连接的第一HBM存储片中；Put the to-be-processed video frame into the AXI4 bus, and initiate burst transmission to the AXI4 Slave port of the first HBM memory slice through the AXI4 Master port to write the to-be-processed video frame to the first video in the first HBM memory slice corresponding to the processing unit;所述步骤S3中从所述第一HBM存储片中读取并输出所述已处理视频帧具体包括以下步骤：In the step S3, reading and outputting the processed video frame from the first HBM memory slice specifically includes the following steps:通过AXI4 Master端口向所述第一HBM存储片的AXI4 Slave端口发起突发传输，以从所述第一HBM存储片中读取并输出所述已处理视频帧。A burst transmission is initiated to the AXI4 Slave port of the first HBM memory slice through the AXI4 Master port to read and output the processed video frame from the first HBM memory slice.
- 根据权利要求1至6任一所述的视频处理方法，其中，所述HBM存储片包括输入缓存区域和输出缓存区域，所述待处理视频帧存储到所述输入缓存区域，所述已处理视频帧存储到所述输出缓存区域。The video processing method according to any one of claims 1 to 6, wherein the HBM storage slice includes an input buffer area and an output buffer area, the to-be-processed video frame is stored in the input buffer area, and the processed video frame is stored in the input buffer area. Frames are stored into the output buffer area.
- 根据权利要求1至6任一所述的视频处理方法，其中，若所述待处理视频帧的分辨率为4K且所述已处理视频帧的分辨率为8K时，所述视频处理单元和对应的HBM存储片均为8个。The video processing method according to any one of claims 1 to 6, wherein, if the resolution of the video frame to be processed is 4K and the resolution of the processed video frame is 8K, the video processing unit and the corresponding There are 8 HBM memory slices.
- 一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被 处理器执行时实现如权利要求1-8任意一项所述的视频处理方法。A computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the video processing method according to any one of claims 1-8.
- 一种电子设备，包括存储器和处理器，所述存储器上存储可在所述处理器上运行的计算机程序，其特征在于，所述处理器执行所述计算机程序时实现如权利要求1-8任意一项所述的视频处理方法。An electronic device, comprising a memory and a processor, the memory stores a computer program that can run on the processor, characterized in that, when the processor executes the computer program, any one of claims 1-8 is implemented. The video processing method described in one item.
- 一种视频处理装置，包括：A video processing device, comprising:视频单元选择模块，配置为从多个并行的视频处理单元中选取处于闲置状态的第一视频处理单元来对待处理视频帧进行视频处理；A video unit selection module, configured to select a first video processing unit in an idle state from a plurality of parallel video processing units to perform video processing on the video frame to be processed;视频帧处理模块，配置为将所述待处理视频帧写入到所述第一视频处理单元所对应连接的第一HBM存储片中，配置并启动所述第一视频处理单元，所述第一视频处理单元用于从所述第一HBM存储片中读取并处理所述待处理视频帧，之后将处理后的已处理视频帧写入到所述第一HBM存储片中；以及A video frame processing module, configured to write the to-be-processed video frame into the first HBM storage slice corresponding to the first video processing unit, configure and start the first video processing unit, and the first video processing unit. a video processing unit for reading and processing the to-be-processed video frame from the first HBM storage slice, and then writing the processed video frame into the first HBM storage slice; and判断模块，配置为判断所述第一视频处理单元对所述待处理视频帧的处理是否完成，若是，则从所述第一HBM存储片中读取并输出所述已处理视频帧。A judging module configured to judge whether the processing of the to-be-processed video frame by the first video processing unit is completed, and if so, read and output the processed video frame from the first HBM memory slice.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011039945.6A CN112235579B (en) | 2020-09-28 | 2020-09-28 | Video processing method, computer-readable storage medium and electronic device |
| CN202011039945.6 | 2020-09-28 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2022062719A1 true WO2022062719A1 (en) | 2022-03-31 |
Family
ID=74119420
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/CN2021/111123 WO2022062719A1 (en) | 2020-09-28 | 2021-08-06 | Video processing method, computer-readable storage medium, and electronic device |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN112235579B (en) |
| WO (1) | WO2022062719A1 (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115022671A (en) * | 2022-06-02 | 2022-09-06 | 智道网联科技(北京)有限公司 | Multi-process video output method, cloud terminal, electronic device and storage medium |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112235579B (en) * | 2020-09-28 | 2022-09-06 | 深圳市洲明科技股份有限公司 | Video processing method, computer-readable storage medium and electronic device |
| CN113457160B (en) * | 2021-07-15 | 2024-02-09 | 腾讯科技（深圳）有限公司 | Data processing method, device, electronic equipment and computer readable storage medium |
| CN114268793B (en) * | 2021-12-21 | 2024-04-16 | 北京达佳互联信息技术有限公司 | Coding method and device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6188381B1 (en) * | 1997-09-08 | 2001-02-13 | Sarnoff Corporation | Modular parallel-pipelined vision system for real-time video processing |
| CN204761566U (en) * | 2015-06-30 | 2015-11-11 | 无锡市同威软件有限公司 | Image processing accelerating system based on FPGA |
| WO2020073193A1 (en) * | 2018-10-09 | 2020-04-16 | 深圳鲲云信息科技有限公司 | Data processing method and related product |
| CN112235579A (en) * | 2020-09-28 | 2021-01-15 | 深圳市洲明科技股份有限公司 | Video processing method, computer-readable storage medium and electronic device |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101882302B (en) * | 2010-06-02 | 2012-05-30 | 北京理工大学 | Motion blur image restoration system based on multi-core |
| US10555035B2 (en) * | 2017-06-09 | 2020-02-04 | Disney Enterprises, Inc. | High-speed parallel engine for processing file-based high-resolution images |
| CN109062858A (en) * | 2018-08-01 | 2018-12-21 | 郑州云海信息技术有限公司 | A kind of FPGA accelerator card based on Xilinx XCVU37P chip |
| CN111316643A (en) * | 2019-03-29 | 2020-06-19 | 深圳市大疆创新科技有限公司 | Video coding method, device and movable platform |
| CN110381322B (en) * | 2019-07-15 | 2023-03-14 | 腾讯科技（深圳）有限公司 | Video stream decoding method and device, terminal equipment and storage medium |
| CN111314741B (en) * | 2020-05-15 | 2021-01-05 | 腾讯科技（深圳）有限公司 | Video super-resolution processing method and device, electronic equipment and storage medium |
-
2020
- 2020-09-28 CN CN202011039945.6A patent/CN112235579B/en active Active
-
2021
- 2021-08-06 WO PCT/CN2021/111123 patent/WO2022062719A1/en active Application Filing
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6188381B1 (en) * | 1997-09-08 | 2001-02-13 | Sarnoff Corporation | Modular parallel-pipelined vision system for real-time video processing |
| CN204761566U (en) * | 2015-06-30 | 2015-11-11 | 无锡市同威软件有限公司 | Image processing accelerating system based on FPGA |
| WO2020073193A1 (en) * | 2018-10-09 | 2020-04-16 | 深圳鲲云信息科技有限公司 | Data processing method and related product |
| CN112235579A (en) * | 2020-09-28 | 2021-01-15 | 深圳市洲明科技股份有限公司 | Video processing method, computer-readable storage medium and electronic device |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115022671A (en) * | 2022-06-02 | 2022-09-06 | 智道网联科技(北京)有限公司 | Multi-process video output method, cloud terminal, electronic device and storage medium |
| CN115022671B (en) * | 2022-06-02 | 2024-03-01 | 智道网联科技(北京)有限公司 | Multi-process video output method, cloud terminal, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112235579A (en) | 2021-01-15 |
| CN112235579B (en) | 2022-09-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2022062719A1 (en) | Video processing method, computer-readable storage medium, and electronic device | |
| CN100336075C (en) | Appts. method and system with graphics-rendering engine having time allocator | |
| CN107527317B (en) | Data transmission system based on image processing | |
| WO2023134128A1 (en) | Video compression processing method, device, and medium | |
| JP2006099774A (en) | Data processing method and device, processing system, computer processing system, and computer network | |
| JP2018534607A (en) | Efficient display processing using prefetch | |
| WO2022161227A1 (en) | Image processing method and apparatus, and image processing chip and electronic device | |
| US20150234632A1 (en) | Multi-processor video processing system and video image synchronous transmission and display method therein | |
| CN114501024B (en) | Video compression system, method, computer readable storage medium and server | |
| WO2023197507A1 (en) | Video data processing method, system, and apparatus, and computer readable storage medium | |
| US9153212B2 (en) | Compressed frame writeback and read for display in idle screen on case | |
| WO2024074012A1 (en) | Video transmission control method, apparatus and device, and nonvolatile readable storage medium | |
| CN109451317A (en) | A kind of image compression system and method based on FPGA | |
| JP4798849B2 (en) | Improved graphics engine master mode operation | |
| US20140085320A1 (en) | Efficient processing of access requests for a shared resource | |
| US9646563B2 (en) | Managing back pressure during compressed frame writeback for idle screens | |
| US9117299B2 (en) | Inverse request aggregation | |
| WO2023045168A1 (en) | Image processing method and apparatus, and related assembly | |
| WO2022142546A1 (en) | Data driving method and device for sparse rendering, and storage medium | |
| US10546558B2 (en) | Request aggregation with opportunism | |
| US9196187B2 (en) | System and method of reducing power using a display inactive indication | |
| JP2003177958A (en) | Specialized memory device | |
| CN116166185A (en) | Caching method, image transmission method, electronic device and storage medium | |
| KR102645239B1 (en) | GPU kernel optimization with SIMO approach for downscaling using GPU cache | |
| WO2021136433A1 (en) | Electronic device and computer system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21871096 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 21871096 Country of ref document: EP Kind code of ref document: A1 |