CN115499559A

CN115499559A - Image processing apparatus and method, processing chip, and electronic device

Info

Publication number: CN115499559A
Application number: CN202110683166.8A
Authority: CN
Inventors: 刘君
Original assignee: Zeku Technology Shanghai Corp Ltd
Current assignee: Zeku Technology Shanghai Corp Ltd
Priority date: 2021-06-18
Filing date: 2021-06-18
Publication date: 2022-12-20
Also published as: WO2022262386A1

Abstract

The embodiment of the application discloses an image processing device and method, a processing chip and electronic equipment. The image processing apparatus includes: the device comprises a delay processing module and a noise reduction processing module; the delay processing module is used for searching and matching a first feature point data window in a first image block of an obtained reference frame in an obtained second image block of a current frame to output motion adjustment information to the noise reduction processing module, and performing feature point detection on the second image block in parallel to obtain a second feature point data window, wherein the reference frame is used for representing an image frame positioned in front of the current frame in time sequence; and the noise reduction processing module is used for carrying out block rotation and alignment processing on the reference frame by utilizing the motion adjustment information to obtain an aligned frame, and carrying out noise reduction filtering processing on the current frame and/or the aligned frame to output a target frame, thereby realizing the reduction of processing time delay in the video noise reduction process.

Description

Image processing apparatus and method, processing chip, and electronic device

Technical Field

The present application relates to the field of computer technologies, and in particular, to an image processing apparatus and method, a processing chip, and an electronic device.

Background

Currently, in the video noise reduction technology, fusion noise reduction filtering is performed through correlation of image frames in a video stream in a time domain. Meanwhile, in order to overcome the non-stationary characteristic of the video stream in the time domain caused by the motion of an object, the corresponding pixels of the current pixel in the previous frame and the next frame are found through motion estimation and motion compensation, so that the filtering efficiency is ensured by maintaining the stationarity in the time domain. However, in the actual video denoising process, there is usually a certain processing delay problem between each processing step, so how to reduce the processing delay in the video denoising process and further improve the processing efficiency needs further analysis.

Disclosure of Invention

The embodiment of the application provides an image processing device and method, a processing chip and electronic equipment, and aims to reduce processing time delay in a video noise reduction process and improve processing efficiency through parallel processing of search matching and feature point detection.

In a first aspect, an embodiment of the present application provides an image processing apparatus, including: the device comprises a delay processing module and a noise reduction processing module; wherein,

the delay processing module is used for searching and matching a first feature point data window in a first image block of an acquired reference frame in an acquired second image block of a current frame to output motion adjustment information to the noise reduction processing module, and performing feature point detection on the second image block in parallel to obtain a second feature point data window, wherein the reference frame is used for representing an image frame positioned in front of the current frame in time sequence;

the noise reduction processing module is configured to perform block rotation and alignment processing on the reference frame by using the motion adjustment information to obtain an aligned frame, and perform noise reduction filtering processing on the current frame and/or the aligned frame to output a target frame.

It can be seen that, first, the delay processing module obtains a first feature point data window and a current frame in a first image block of a reference frame, that is, directly obtains the first feature point data window (without performing feature point detection) and the current frame; secondly, the delay processing module executes search matching of a first feature point data window of the reference frame in a second image block of the current frame and feature point detection of the second image block of the current frame in parallel; thirdly, the delay processing module outputs motion adjustment information to the noise reduction processing module; and finally, the noise reduction processing model carries out rotation alignment on the reference frame by utilizing the motion adjustment information to obtain an alignment frame, and carries out noise reduction filtering processing on the current frame and/or the alignment frame to output a target frame, so that parallel processing of searching matching and feature point detection is realized by directly obtaining the first feature point data window and the current frame, processing time delay in the video noise reduction process is reduced by parallel processing, and processing efficiency is improved.

In a second aspect, an embodiment of the present application provides an image processing method, including:

acquiring a first feature point data window and a current frame in a first image block of a reference frame, wherein the reference frame is used for representing an image frame positioned in front of the current frame in time sequence;

searching and matching the first characteristic point data window in a second image block of the current frame to obtain motion adjustment information, and performing characteristic point detection on the second image block in parallel to obtain a second characteristic point data window;

performing block rotation and alignment processing on the reference frame by using the motion adjustment information to obtain an aligned frame;

and carrying out noise reduction filtering processing on the current frame and/or the aligned frame to obtain a target frame.

It can be seen that, first, a first feature point data window (i.e. feature point detection does not need to be performed) and a current frame in a first image block of a reference frame are directly obtained; secondly, searching and matching a first feature point data window of the reference frame in a second image block of the current frame and detecting feature points of the second image block of the current frame in parallel; and finally, performing rotation alignment on the reference frame by utilizing the motion adjustment information to obtain an aligned frame, and performing noise reduction filtering processing on the current frame and/or the aligned frame to output a target frame, so that parallel processing of searching matching and feature point detection is realized by directly obtaining the first feature point data window and the current frame, processing time delay in the video noise reduction process is reduced by parallel processing, and processing efficiency is improved.

In a third aspect, an embodiment of the present application provides a processing chip including the image processing apparatus in the first aspect of the embodiment of the present application.

Therefore, the image processing device in the processing chip directly obtains the first feature point data window and the current frame to realize parallel processing of searching matching and feature point detection, and further realizes reduction of processing time delay in the video noise reduction process through the parallel processing, and improves processing efficiency.

In a fourth aspect, an embodiment of the present application provides an electronic device, which includes a memory, a communication interface, and the image processing apparatus in the first aspect of the embodiment of the present application.

Therefore, the image processing device in the electronic device directly obtains the first feature point data window and the current frame to realize parallel processing of search matching and feature point detection, and further realizes reduction of processing time delay in the video noise reduction process through the parallel processing, and improves processing efficiency.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. It is to be expressly understood that the drawings described below are only illustrative of some embodiments of the invention. It is also possible for a person skilled in the art to derive other figures from these figures without inventive effort.

Fig. 1 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application;

fig. 2 is a schematic flow chart of video denoising provided by an embodiment of the present application;

fig. 3 is a schematic diagram illustrating feature point detection performed on an image block of a current frame according to an embodiment of the present application;

FIG. 4 is a schematic diagram illustrating search matching between a current frame and a reference frame according to an embodiment of the present disclosure;

fig. 5 is a schematic structural diagram of relative displacement from a feature point data window of a current frame to a target feature point data window according to an embodiment of the present application;

FIG. 6 is a schematic structural diagram of another image processing apparatus provided in an embodiment of the present application;

FIG. 7 is a schematic structural diagram of another image processing apparatus according to an embodiment of the present application;

FIG. 8 is a schematic structural diagram of another image processing apparatus provided in an embodiment of the present application;

fig. 9 is a schematic flow chart of another video denoising provided by an embodiment of the present application;

fig. 10 is a schematic flowchart of an image processing method according to an embodiment of the present application;

FIG. 11 is a schematic structural diagram of a processing chip according to an embodiment of the present disclosure;

fig. 12 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

In order to make the technical solutions of the present application better understood by those skilled in the art, the technical solutions in the embodiments of the present application are described below clearly and completely in combination with the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only some embodiments of the present application, and not all embodiments. All other embodiments obtained by a person skilled in the art without making any inventive step on the basis of the description of the embodiments of the present application are intended to be covered by the present application.

The terms "first," "second," and the like in the description and claims of the present application and in the above-described drawings are used for distinguishing between different objects and not for describing a particular order. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, software, product, or apparatus that comprises a list of steps or elements is not limited to those listed but may include other steps or elements not listed or inherent to such process, method, product, or apparatus.

In the embodiments of the present application, the term "connected" should be interpreted broadly unless explicitly stated or limited otherwise. For example, "connected" may be fixed, coupled, or detachable; may be directly connected, indirectly connected through an intermediary, or in spaced contact; but may be a physical connection, an electrical connection, or the like.

Reference in the embodiments of the present application to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.

Before describing the technical solutions of the embodiments of the present application, the following description will be made on related concepts that may be related to the present application.

(1) Video noise reduction

Since an image capturing apparatus, such as a complementary metal-oxide-semiconductor (CMOS) sensor, a charge-coupled device (CCD) sensor, etc., will be affected by noise during data acquisition, so that random noise exists in a video stream, it is necessary to remove the noise by using an image noise reduction technology.

The video denoising method can be divided into a space domain, a frequency domain, a wavelet domain, a time-space domain and the like according to different processing domains. Where the noise reduction methods between different processing domains overlap, or where one noise reduction method involves multiple processing domains. For example, a noise reduction method using frequency domain filtering may be used in a noise reduction method using time domain or time-space domain, that is, after all or a part of image frames in a video stream are transformed into a frequency domain by fourier transform, noise reduction processing is performed using time domain filtering or time-space domain filtering.

The modes of spatial filtering, frequency domain filtering, wavelet domain filtering and the like in video denoising are consistent with the modes in image denoising, and only the modes are expanded to process multi-frame images, and meanwhile, more redundant information in video signals is utilized for optimization, so that a better denoising effect is achieved. The spatial filtering is to directly perform algebraic operation of pixel values on each frame of image in a video stream, and only consider the correlation of the image frame on a spatial domain. The frequency domain filtering is to transform the image frames in the video stream into the frequency domain through fourier transform, and then attenuate the frequencies representing noise to retain the original information in the video stream to the maximum extent. The wavelet domain filtering is to convert the image frames in the video stream into a time-frequency domain and then perform denoising. The time domain filtering is to perform noise reduction filtering by considering the correlation of image frames in a video stream in a time dimension, has simple operation and high efficiency, and does not introduce the phenomenon of spatial blurring.

(2) Three-dimensional digital Noise Reduction (3-dimensional digital Noise Reduction)

The 3DNR is a noise reduction method combining spatial filtering and temporal filtering, and the general idea is to detect motion vectors among multiple frames of images in a video stream through motion estimation, perform alignment processing according to the motion vectors, and finally perform fusion noise reduction filtering on the multiple frames of images after the alignment processing, thereby outputting filtered images.

Currently, in video noise reduction technology, fusion noise reduction filtering is performed by correlation of image frames in a video stream in a time domain. Meanwhile, in order to overcome the non-stationary characteristic of the video stream in the time domain caused by the motion of an object, the corresponding pixels of the current pixel in the front frame and the rear frame are found through motion estimation and motion compensation, so that the filtering efficiency is ensured by maintaining the stationarity in the time domain. However, in the actual video denoising process, there is usually a certain processing delay between each processing step, so how to reduce the processing delay in the video denoising process and further improve the processing efficiency needs further analysis.

In conjunction with the above description, the image processing apparatus according to the embodiment of the present application will be described below with reference to fig. 1. Fig. 1 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application. The image processing apparatus 10 includes a delay processing module 100 and a noise reduction processing module 200. The delay processing module 100 is connected to the noise reduction processing module 200. Wherein,

the delay processing module 100 is configured to search and match a first feature point data window in a first image block of an acquired reference frame in an acquired second image block of a current frame to output motion adjustment information to the noise reduction processing module, and perform feature point detection on the second image block in parallel to obtain a second feature point data window, where the reference frame is used to represent an image frame located before the current frame in time sequence.

And a noise reduction processing module 200, configured to perform block rotation and alignment processing on the reference frame by using the motion adjustment information to obtain an aligned frame, and perform noise reduction filtering processing on the current frame and/or the aligned frame to output a target frame.

First, the image processing apparatus 10 according to the embodiment of the present application includes hardware and/or software modules for performing the respective functions in order to implement the functions of temporal noise reduction and low processing latency. Those skilled in the art will readily appreciate that the functions of the various blocks described in connection with the embodiments provided herein may be implemented in hardware or a combination of hardware and computer software. Whether a module is implemented as hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Those skilled in the art may call a specific program or algorithm through a processor (such as a CPU, etc.) to implement the functions of the modules described in the embodiments of the present application, but such implementation should not be considered as beyond the scope of the present application. In addition, in the embodiment of the present application, the image processing apparatus 10 may be divided into functional blocks, for example, each functional block may be divided according to the function, or two or more functions may be integrated into one functional block. The integrated functional module can be realized in a hardware form or a software form. Meanwhile, the division of the functional modules in the embodiment of the present application is schematic, and is only a logic function division, and there may be another division manner in actual implementation, which is not particularly limited.

Secondly, in the video denoising method shown in fig. 2, a reference frame and a current frame to be denoised are obtained from a video stream, and the current frame is uploaded to a bus for storage, wherein the reference frame is a previous frame or a next frame of the current frame; carrying out smooth filtering on the current frame, and carrying out feature point detection on the current frame after smooth filtering to obtain a feature point data window; searching and matching the characteristic point data window of the current frame to a reference frame to obtain a motion vector, namely a motion matching result, and calculating motion adjustment information through the motion matching result, wherein the motion adjustment information is used for rotationally aligning the reference frame; reading a current frame and a reference frame from a bus, and performing rotation alignment processing on the reference frame through the motion adjustment information; and performing 2-frame fusion noise reduction processing on the current frame and the reference frame after the rotation alignment to obtain a target frame, thereby realizing time domain noise reduction processing on the current frame.

Then, for the current frame, feature point detection may be performed, where the current frame may be subjected to image blocking to obtain N × N image blocks with equal size and non-overlapping, where a value of N may be 2, 4, 8, or 16, and then feature point detection is performed on each image block to obtain multiple feature point data windows. For example, referring to fig. 3, a current frame is divided into 4 × 4 image blocks, and then feature point detection is performed on each image block to obtain a plurality of feature point data windows.

For searching and matching the feature point data window of the current frame to the reference frame to obtain the optimal motion vector, a data window similar to the feature point data window of the current frame may be searched and matched in the reference frame to obtain a target feature point data window in the reference frame, as shown in fig. 4, an area 410 in fig. 4 represents a search and matching range, and then the relative displacement from the feature point data window of the current frame to the target feature point data window is calculated to obtain the motion vector, as shown in fig. 5.

Finally, in the feature point detection process shown in fig. 3, feature point detection is usually performed on 4 image blocks in the first row block of the current frame (i.e., 1/4 frame of the current frame) after being partitioned, and then feature point detection is performed on 4 image blocks in the second row block of the current frame, and so on. Therefore, a 1/N frame processing delay is usually introduced in the feature point detection process, i.e. a 1/4 frame processing delay in FIG. 3. Meanwhile, in the search matching process shown in fig. 4, the feature point data window of 1/4 frame of the current frame (i.e. the current first line block) is usually searched and matched in 1/4 frame of the reference frame (i.e. the first line block of the reference frame), and so on. Therefore, a processing delay of 1/N frame is also introduced in the process of searching for the matching.

In addition, in the fusion noise reduction process of the current frame and the reference frame after the rotation alignment, the current frame of 1 frame and the reference frame of 1 frame still need to be completely read from the bus. Therefore, a processing delay of 1 frame is also introduced in the fusion noise reduction processing. It can be seen that there is usually a large processing delay in the actual video denoising processing step.

In summary, in order to reduce the processing delay, the following specifically describes the technical solution of the embodiment of the present application.

Specifically, the current frame may be used to represent a frame of image to be denoised currently in the video stream.

In particular, the reference frame may be used to represent an image frame temporally preceding the current frame. For example, the reference frame may be a previous frame image of the current frame.

Specifically, the search matching may be understood as determining a search range in the image block of the current frame through a search matching algorithm, finding a most approximate (or matched) data window in the search range through the first feature point data window, and finally obtaining motion adjustment information through a Motion Vector (MV) from the first feature point data window to the most approximate data window. The search matching algorithm may include a full search matching algorithm, a three-step search algorithm, a diamond search algorithm, a four-step search algorithm, a continuous filtering algorithm, a multi-layer filtering algorithm, a partial distortion removal algorithm, and the like.

In particular, the motion adjustment information may be used to perform block rotation and alignment processing on the reference frame.

Specifically, the feature point detection may include hash corner detection (Harris corner detection). Wherein a corner in an image frame may have the following characteristics: an intersection between the contours; for the same scene, the scene still has stable property even if the visual angle changes; the pixel points in the area near the corner point have large changes in the gradient direction or the gradient amplitude.

It should be noted that, the hash corner detection may be performed by moving on the image block of the current frame through a local window, and determining whether a pixel value in the local window changes greatly. Whether the pixel value has a large change can be described by an image gradient. The larger the image gradient, the larger the change in the pixel value in the local window (or the larger the change rate of the gray scale value). Therefore, if there is a large variation in the gray values (in the image gradient) in the local window, there is a corner in the region where the local window is located.

It can be understood that, if the hash corner detection is adopted to perform the feature point detection on the image block of the current frame to obtain the feature point data window, the feature point data window may be used to represent a corner in the image block, or the feature point data window may be used to represent an image area and the like composed of the corner in the image block and pixel points around the corner.

Specifically, the first image block may be used to represent at least one image block of a plurality of image blocks of the reference frame that are equal in size and do not overlap with each other.

It can be understood that, in the embodiment of the present application, image blocking processing may be performed on an obtained reference frame to obtain N × N image blocks that are equal in size and do not overlap with each other, where a value of N may be 2, 4, 8, or 16, and feature point detection may be performed on each image block to obtain a plurality of feature point data windows, and finally the plurality of feature point data windows are stored, so that it is beneficial to directly obtain a feature point data window of the reference frame in subsequent search matching without performing feature point detection, and further, processing delay in a video denoising process is reduced.

Specifically, the second image block may be used to represent at least one image block of a plurality of image blocks of the current frame that are equal in size and do not overlap with each other.

It can be understood that, in the embodiment of the present application, image blocking processing may be performed on an obtained current frame to obtain N × N image blocks having equal size and no overlap, where a value of N may be 2, 4, 8, or 16.

Specifically, the position of the first image block in the reference frame has a corresponding relationship with the position of the second image block in the current frame.

It is understood that, if the first image block is an image block in the first row block of the reference frame, the second image block is an image block in the first row block of the current frame, and the rest of the same principles are known.

It can be seen that, compared with the distributed processing in which feature point detection is performed on the current frame first and then search matching is performed on the feature point data window of the current frame to the reference frame in fig. 2, in the embodiment of the present application, the delay processing module 100 directly obtains the first feature point data window (i.e., feature point detection is not required) of the reference frame and the current frame, performs search matching of the first feature point data window in the second image block of the current frame and feature point detection of the second image block of the current frame in parallel, and outputs motion adjustment information to the noise reduction processing module 200 through the delay processing module 100, so that parallel processing of search matching and feature point detection is realized by directly obtaining the first feature point data window and the current frame, and further, processing delay in the video noise reduction process is reduced through parallel processing, and processing efficiency is improved. Meanwhile, the second characteristic point data window of the current frame is directly uploaded to avoid uploading and storing the rest redundant data in the current frame so as to reduce the bandwidth of data uploading.

In conjunction with the above description, the following embodiments of the present application will specifically describe each module in the image processing apparatus 10 with reference to fig. 6.

In one possible example, the delay processing module 100 specifically includes a feature point detection module 130, a motion estimation module 110, and a motion adjustment module 120; wherein,

the feature point detecting module 130 is configured to perform feature point detection on the second image block to obtain a second feature point data window.

And the motion estimation module 110 is configured to search and match the first feature point data window in the second image block to obtain a motion vector, and output the motion vector to the motion adjustment module.

The motion adjustment module 120 is configured to perform data fitting processing on the input motion vector to obtain a homography matrix, use the homography matrix as motion adjustment information, and output the motion adjustment information to the noise reduction processing module 200.

It should be noted that, first, in the embodiment of the present application, the first feature point data window of the reference frame and the current frame may be directly obtained by the delay processing module 100, and search matching of the first feature point data window in the current frame and feature point detection of the current frame are executed in parallel, so that parallel processing of search matching and feature point detection is realized by directly obtaining the first feature point data window and the current frame, and further, processing delay in a video denoising process is reduced by parallel processing.

Secondly, the feature point detection in the embodiment of the present application includes hash corner detection, so that the feature point detection module 130 can detect corners in the image frame of the current frame.

Thirdly, motion estimation is one of the important components of the video denoising technology, and a Motion Vector (MV) of a pixel point is obtained by calculating the relative motion offset of the pixel point in two adjacent frame images. The motion estimation algorithm may include a block matching algorithm, a pixel method, a feature method, a phase method, and the like. Thus, search matching of the reference frame and the current frame to obtain the motion vector can be achieved by the motion estimation module 110. The motion estimation module 110 may determine a search range in the image block of the current frame through a search and matching algorithm, and then find the most approximate (or matching) data window within the search range. The search matching algorithm may include a full search matching algorithm, a three-step search algorithm, a diamond search algorithm, a four-step search algorithm, a continuous culling algorithm, a multi-layer culling algorithm, a partial distortion removal algorithm, and the like.

Finally, the data fitting process of the motion vector in the embodiment of the present application may include a least square method, and the motion vector is subjected to data fitting process by the least square method to obtain a homography matrix (homography matrix). The homography transformation can be used for describing the position mapping relation of the object between the world coordinate system and the pixel coordinate system, and the corresponding homography transformation is called as a homography matrix. Therefore, the calculation of the motion vector to obtain the motion adjustment information for performing the block rotation and alignment process on the reference frame is performed by the motion adjustment module 120.

Specifically, the motion estimation module 110 is specifically configured to: calculating the sum of absolute errors between the gray value of each pixel point in the first characteristic point data window and the gray value of each pixel point in the data window in the searching range of the second image block to obtain a first parameter value; taking a data window corresponding to the minimum value in the first parameter values as a target data window; and calculating the relative displacement from the first characteristic point data window to the target data window to obtain a motion vector, and outputting the motion vector.

It should be noted that, in the embodiment of the present application, it is considered that a minimum Sum of Absolute Differences (SAD) criterion is adopted in the first feature point data window of the reference frame to search and match in the second image block of the current frame, and a data window corresponding to the minimum value in the SAD values (i.e. the first parameter values) of the first feature point data window is found. At this time, the data window is the data window corresponding to the first feature point data window that is the best match, i.e., the target data window. Therefore, the first feature point data window and the target data window form relative positions in the time domain to obtain a motion vector, so that motion adjustment information for performing block rotation and alignment processing on the reference frame is constructed by the motion vector.

In one possible example, the delay processing module 100 further comprises a smoothing filter module 140. Wherein,

the smoothing filter module 140 may be configured to perform smoothing filtering on the current frame and output the current frame after the smoothing filtering.

It should be noted that the smoothing filter module 140 can be used to reduce noise and artifacts in the current frame. The smoothing filter module 140 may include an average filter, a square frame filter, a median filter, a gaussian filter, a bilateral filter, etc.

In one possible example, the delay processing module 100 further includes an image blocking module 150. Wherein,

the image blocking module 150 may be configured to perform image blocking on the current frame and output the current frame after the image blocking.

It should be noted that, in the embodiment of the present application, the image partitioning module 150 may partition an image frame (including a current frame or a reference frame) of an acquired video stream into N × N image blocks with the same size and without overlapping, so as to facilitate subsequent direct feature point detection on the image blocks, and improve the feature point detection efficiency on the image frame.

In one possible example, the noise reduction processing module 200 specifically includes a rotational alignment module 210, a filter decision module 220, and a fusion filter module 230. Wherein,

the rotation alignment module 210 may be configured to call the motion adjustment information to perform block rotation and alignment processing on the read reference frame to obtain an aligned frame, and output the aligned frame.

The filtering decision module 220 is configured to perform average absolute error calculation on the read gray value of each pixel in the third image block of the current frame and the gray value of each pixel in the fourth image block of the input alignment frame to obtain a second parameter value; and determining noise reduction filtering strategy information for the current frame and the aligned frame according to a comparison result between the second parameter value and a preset threshold, and outputting the noise reduction filtering strategy information to the fusion filtering module 230, where the noise reduction filtering strategy information includes time domain filtering or spatial domain filtering.

The fusion filtering module 230 may be configured to perform fusion noise reduction filtering processing on the read current frame and/or the input aligned frame according to the input noise reduction filtering policy information to output the target frame.

It should be noted that, first, in order to reduce a residual value between each pixel in the image block of the current frame and each pixel in the image block of the reference frame, the embodiment of the present application considers that the block rotation and alignment processing is performed on the reference frame by the rotation alignment module 210.

Secondly, in order to overcome the phenomenon of 'tailing' of a fast moving object easily caused by time domain filtering, the embodiment of the application considers that the motion intensity is detected by mean absolute error (MAD) algorithm, so that for objects with different motion intensities, different filtering intensities are adopted by a filter, the phenomenon of 'tailing' caused by the fast moving object is avoided, and the image filtering effect is improved. The filtering decision module 220 calculates a gray value of each pixel in the third image block of the current frame and a gray value of each pixel in the fourth image block of the alignment frame through the MAD algorithm to obtain an MAD value (i.e., a second parameter value).

Thirdly, the embodiment of the application tests a large number of image sequences to determine an empirical value for measuring the motion intensity, namely a preset threshold, and then determines the motion intensity of the third image block of the current frame on the motion track represented by the motion vector according to the comparison result between the second parameter value and the preset threshold.

If the comparison result between the second parameter value and the preset threshold is that the second parameter value is greater than the preset threshold, it indicates that the third image block has a strong motion intensity on the motion track. At this time, the filtering strength of the filter may be reduced (or it is determined that the noise reduction filtering strategy information is spatial filtering), so that the original information of the filtered third image block is kept as much as possible, and the tailing phenomenon is avoided.

If the comparison result between the second parameter value and the preset threshold is that the second parameter value is smaller than the preset threshold, it indicates that the third image block has a weaker motion intensity on the motion track, i.e., a smooth and slow motion. For example, background pictures in a video stream typically do not move. At this time, the filtering strength of the filter may be increased (or the noise reduction filtering strategy information is determined to be time-domain filtering), so as to effectively remove noise. Meanwhile, the low motion intensity of the third image block cannot cause trailing, and better filtering effect is guaranteed.

Finally, in order to implement denoising filtering processing on the current frame, the fusion filtering module 230 of the embodiment of the present application performs fusion denoising filtering processing on the current frame and/or the aligned frame according to time domain filtering or spatial domain filtering.

Specifically, the spatial filtering may include mean filtering, median filtering, gaussian filtering, or bilateral filtering. Therefore, the current frame is subjected to noise reduction filtering according to spatial filtering to output a target frame, so that an image block with stronger motion intensity on a motion track is filtered through the spatial filtering, original information in the image block is reserved as much as possible, and the trailing phenomenon is avoided.

In particular, the temporal filtering may comprise linear weighted filtering. Therefore, the target frame output by performing the fusion noise reduction filtering on the current frame and the aligned frame according to the linear weighted filtering satisfies the following formula:

F _out ＝ω*F _t +(1-ω)*F _t-1 ；

wherein, F _out Each pixel value, F, in an image block representing a target frame _t Representing the values of the pixels in the image block of the current frame, F _t-1 Each pixel value in the image block of the aligned frame is represented, and ω represents a preset weight. In addition, the filtering strength of the filter can be dynamically adjusted by presetting the weight ω.

In summary, an image block with a weak motion intensity on a motion trajectory is filtered through linear weighted filtering, so that noise in the image block is effectively removed.

Specifically, the third image block may be used to represent at least one image block of a plurality of image blocks of the current frame, which have equal size and are non-overlapping with each other. It is understood that the filtering decision module 220 of the embodiment of the present application may perform image blocking processing on the read current frame to obtain N × N image blocks with equal size and without overlapping.

Specifically, the fourth image block may be used to represent at least one image block of a plurality of image blocks of the aligned frame that are equal in size and do not overlap with each other. It is understood that the filtering decision module 220 of the embodiment of the present application may perform image blocking on the input aligned frames to obtain N × N image blocks with equal size and without overlapping, where N may be 2, 4, 8, or 16.

Specifically, the position of the third image block in the current frame and the position of the fourth image block in the alignment frame have a corresponding relationship. It is to be understood that, if the third image block is an image block in the first row block of the current frame, the second image block is an image block in the first row block of the aligned frame, and the rest of the same principles are known.

In one possible example, the image processing apparatus 10 further includes a storage module. The storage module may be configured to store at least one of the current frame, the reference frame, the first feature point data window, and the second feature point data window.

In particular, the memory module may include a double data rate DDR memory.

The DDR memory may include a double data rate synchronous dynamic random access memory (DDR SDRAM).

It should be noted that, the delay processing module 100 in the embodiment of the present application may directly obtain the first feature point data window and the current frame in the image block of the reference frame from the storage module, and then upload the current second feature point data window to the storage module for storage, so as to subsequently and directly read the second feature point data window from the storage module without performing feature point detection again to perform search matching in parallel processing, thereby reducing the processing delay in the video denoising process. Meanwhile, the parallel processing of searching matching and feature point detection is realized by directly obtaining the first feature point data window and the current frame, so that the processing time delay in the video noise reduction process is reduced through the parallel processing, and the processing efficiency is improved. In addition, the second characteristic point data window of the current frame is uploaded to avoid uploading and storing the rest redundant data in the current frame so as to reduce the bandwidth of data uploading.

In summary, the flow of image processing according to the embodiment of the present application will be described below by way of example. The method comprises the steps of obtaining a characteristic point data window of a reference frame and a current frame to be denoised, wherein the reference frame is a previous frame of the current frame, and the current frame is subjected to smooth filtering; performing parallel detection of the feature points of the current frame after smooth filtering and searching and matching of a feature point data window of a reference frame in an image block of the current frame after smooth filtering; obtaining an optimal motion vector, namely a motion matching result, through searching and matching from a feature point data window of a reference frame to a current frame, and calculating motion adjustment information through the motion matching result, wherein the motion adjustment information is used for carrying out rotation alignment on the reference frame; acquiring a current frame and a reference frame, and performing rotation alignment processing on the reference frame through motion adjustment information; the current frame and the reference frame after the rotation alignment are subjected to noise reduction filtering to obtain a target frame, so that parallel processing of searching matching and feature point detection is realized by directly obtaining a feature point data window of the reference frame and the current frame, processing time delay in a video noise reduction process is reduced by the parallel processing, and processing efficiency is improved.

In accordance with the above embodiments, the following describes an image processing apparatus according to an embodiment of the present application, with reference to fig. 7. Fig. 7 is a schematic structural diagram of another image processing apparatus according to an embodiment of the present application. The image processing apparatus 70 includes a delay processing module 710, a storage module 720, and a noise reduction processing module 730. The delay processing module 710 is connected to the storage module 720, the noise reduction processing module 730 is connected to the storage module 720, and the delay processing module 710 is connected to the noise reduction processing module 730. Wherein,

a delay processing module 710, configured to read a first feature point data window in a first image block of a current frame and a reference frame from the storage module 720, where the reference frame is used to represent an image frame temporally located before the current frame; searching and matching the first characteristic point data window in a second image block of the current frame to output motion adjustment information, and performing characteristic point detection on the second image block in parallel to obtain a second characteristic point data window; and uploading the second feature point data window to the storage module 720.

A noise reduction processing module 730, configured to read the current frame and the reference frame from the storage module 720; calling the motion adjustment information output by the delay processing module 710 to perform block rotation and alignment processing on the reference frame to obtain an aligned frame; and performing noise reduction filtering processing on the current frame and/or the aligned frame to output a target frame.

It should be noted that, in order to implement the functions of temporal noise reduction and low processing latency, the image processing apparatus 70 according to the embodiment of the present application includes hardware and/or software modules for performing the respective functions. Those skilled in the art will readily appreciate that the functions of the various modules described in connection with the embodiments provided herein may be implemented in hardware or a combination of hardware and computer software. Whether a module is executed as hardware or computer software driven hardware depends upon the particular application and design constraints imposed on the solution. Those skilled in the art may call a specific program or algorithm through a processor (such as a CPU, etc.) to implement the functions of the modules described in the embodiments of the present application, but such implementation should not be considered as beyond the scope of the present application. In addition, in the embodiment of the present application, the image processing apparatus 70 may be divided into functional blocks, for example, each functional block may be divided according to the function, or two or more functions may be integrated into one functional block. The integrated functional module can be realized in a hardware form or a software form. Meanwhile, the division of the functional modules in the embodiment of the present application is schematic, and is only a logic function division, and there may be another division manner in actual implementation, which is not particularly limited.

Compared with the distributed processing of firstly performing feature point detection on the current frame and then performing search matching on the feature point data window of the current frame to the reference frame, in the embodiment of the application, firstly, the delay processing module 710 directly reads the first feature point data window and the current frame in the first image block of the reference frame from the storage module 720, that is, directly obtains the first feature point data window (without performing feature point detection) and the current frame; secondly, the delay processing module executes search matching of a first feature point data window of the reference frame in a second image block of the current frame and feature point detection of the second image block of the current frame in parallel; thirdly, the delay processing module 710 outputs the motion adjustment information matching result to the denoising processing module 730 and uploads the second feature point data window to the storage module 720 for storage; and finally, the noise reduction processing model performs rotation alignment on the reference frame by utilizing the motion adjustment information to obtain an aligned frame, and performs noise reduction filtering processing on the current frame and/or the aligned frame to output a target frame, so that parallel processing of searching matching and feature point detection is realized by directly obtaining the first feature point data window and the current frame, processing time delay in the video noise reduction process is reduced by parallel processing, and processing efficiency is improved. In addition, the second characteristic point data window of the current frame is uploaded to the storage module 720, so that the remaining redundant data in the current frame is prevented from being uploaded and stored, and the bandwidth of data uploading is reduced.

Next, the present embodiment will specifically describe each block in the image processing apparatus 70 with reference to fig. 8.

In one possible example, the delay processing module 710 may specifically include a parallel processing module 711 and a data window upload module 712. Wherein,

the parallel processing module 711 may be configured to search and match the first feature point data window of the parameter frame read from the storage module 720 in the second image block to output the motion adjustment information, and perform feature point detection on the second image block of the current frame read from the storage module 720 in parallel to obtain a second feature point data window.

A data window uploading module 712, which may be configured to upload the second feature point data window to the storage module 720.

It should be noted that, in the embodiment of the present application, the parallel processing module 711 may execute search matching of the feature point data window of the reference frame in the current frame and feature point detection of the current frame in parallel, so as to implement reduction of processing delay in the video denoising process through parallel processing of the search matching and the feature point detection.

In addition, the feature point data window of the current frame is uploaded to the storage module for storage through the data window uploading module 712, and other redundant data features of the current frame are not uploaded for storage, so that the redundant data in the current frame is prevented from being uploaded for storage to reduce the bandwidth of data uploading, and the feature point data window of the current frame is directly read from the storage module without performing feature point detection when searching and matching are performed on the next frame image of the current frame in the follow-up process, so that the processing time delay in the video denoising process is reduced, and the processing efficiency is improved.

In one possible example, the parallel processing module 711 specifically includes a feature point detection module 7111, a motion estimation module 7113, and a motion adjustment module 7115. Wherein,

the feature point detecting module 7111 may be configured to perform feature point detection on the read second image block to obtain a second feature point data window.

The motion estimation module 7113 may be configured to search and match the read first feature point data window in the second image block to obtain a motion vector, and output the motion vector to the motion adjustment module 7115.

The motion adjustment module 7115 may be configured to perform data fitting processing on the input motion vector to obtain a homography matrix, use the homography matrix as motion adjustment information, and output the motion adjustment information to the noise reduction processing module 730.

It should be noted that, first, the feature point detection in the embodiment of the present application includes hash corner detection, so that the feature point detection module 7111 may detect corners in an image frame of a current frame.

Secondly, motion estimation is one of the important components of the video noise reduction technology, and a Motion Vector (MV) of a pixel point is obtained by calculating the relative motion offset of the pixel point in two adjacent frames of images. The motion estimation algorithm may include a block matching algorithm, a pixel method, a feature method, a phase method, and the like. Thus, search matching of the reference frame and the current frame to obtain a running vector may be achieved by motion estimation module 7113. The motion estimation module 7113 may determine a search range in the image block of the current frame by a search and matching algorithm, and then find the most approximate (or matching) data window in the search range. The search matching algorithm may include a full search matching algorithm, a three-step search algorithm, a diamond search algorithm, a four-step search algorithm, a continuous culling algorithm, a multi-layer culling algorithm, a partial distortion removal algorithm, and the like.

Finally, the data fitting process of the motion vector in the embodiment of the present application may include a least square method, and the motion vector is subjected to data fitting process by the least square method to obtain a homography matrix (homography matrix). The homography transformation can be used for describing the position mapping relation of the object between the world coordinate system and the pixel coordinate system, and the corresponding homography transformation is called as a homography matrix. Therefore, the calculation of motion vectors for motion adjustment information for block rotation and alignment processing of reference frames is achieved by the motion adjustment module 7115.

Specifically, the motion estimation module 7113 is specifically configured to: calculating the absolute error sum of the gray value of each pixel point in the read first characteristic point data window and the gray value of each pixel point in the data window in the searching range of the second image block to obtain a first parameter value; taking a data window corresponding to the minimum value in the first parameter values as a target data window; and calculating the relative displacement from the first characteristic point data window to the target data window to obtain a motion vector, and outputting the motion vector.

In one possible example, the delay processing module 710 further includes a smoothing filter module 713. Wherein,

the smoothing filter module 713 may be configured to perform smoothing filtering processing on the read current frame, and output the current frame after the smoothing filtering processing.

It is noted that the smoothing filter module 713 may be used to reduce noise and artifacts in the current frame. The smoothing filter module 713 may include an averaging filter, a square frame filter, a median filter, a gaussian filter, a bilateral filter, etc.

In one possible example, the delay processing module 710 further includes an image blocking module 714. Wherein,

the image blocking module 714 may be configured to perform image blocking on the current frame, and output the current frame after the image blocking.

It should be noted that in the embodiment of the present application, the image partitioning module 714 may divide the image frame (including the current frame or the reference frame) of the video stream read from the storage module 720 into N × N image blocks with the same size and without overlapping with each other, so as to facilitate the subsequent direct feature point detection on the image blocks, and improve the feature point detection efficiency on the image frame.

In one possible example, the noise reduction processing module 730 specifically includes a rotational alignment module 731, a filter decision module 732, and a fusion filter module 733. Wherein,

the rotation alignment module 731 may be configured to call the motion adjustment information to perform block rotation and alignment processing on the read reference frame to obtain an aligned frame, and output the aligned frame.

The filtering decision module 732 is configured to perform average absolute error calculation on the read gray value of each pixel in the third image block of the current frame and the gray value of each pixel in the fourth image block of the input alignment frame to obtain a second parameter value; and determining noise reduction filtering strategy information for the current frame and the aligned frame according to a comparison result between the second parameter value and a preset threshold, and outputting the noise reduction filtering strategy information to the fusion filtering module 733, wherein the noise reduction filtering strategy information includes time domain filtering or spatial domain filtering.

The fusion filtering module 733, configured to perform fusion noise reduction filtering processing on the read current frame and/or the input aligned frame according to the input noise reduction filtering policy information to output a target frame.

It should be noted that, first, in order to reduce a residual value between each pixel in the image block of the current frame and each pixel in the image block of the reference frame, the embodiment of the present application considers that the block rotation and alignment process is performed on the reference frame by the rotation alignment module 731.

Secondly, in order to overcome the phenomenon of 'tailing' of a fast moving object easily caused by time domain filtering, the embodiment of the application considers that the motion intensity is detected by mean absolute error (MAD) algorithm, so that for objects with different motion intensities, different filtering intensities are adopted by a filter, the phenomenon of 'tailing' caused by the fast moving object is avoided, and the image filtering effect is improved. The filtering decision module 732 calculates a gray value of each pixel in the third image block of the current frame and a gray value of each pixel in the fourth image block of the alignment frame by using an MAD algorithm to obtain an MAD value (i.e., a second parameter value).

Thirdly, the embodiment of the application tests a large number of image sequences to determine an empirical value for measuring the motion intensity, namely a preset threshold, and then determines the motion intensity of the third image block of the current frame on the motion track represented by the motion vector according to the comparison result between the second parameter value and the preset threshold. If the comparison result between the second parameter value and the preset threshold is that the second parameter value is greater than the preset threshold, it indicates that the third image block has stronger motion intensity on the motion track. At this time, the filtering strength of the filter may be reduced (or it is determined that the noise reduction filtering strategy information is spatial filtering), so that the original information of the filtered third image block is kept as much as possible, and the tailing phenomenon is avoided. If the comparison result between the second parameter value and the preset threshold is that the second parameter value is smaller than the preset threshold, it indicates that the third image block has a weaker motion intensity on the motion track, i.e., a smooth and slow motion. For example, background pictures in a video stream typically do not move. At this time, the filtering strength of the filter may be increased (or the noise reduction filtering policy information is determined to be time-domain filtering), so as to effectively remove the noise. Meanwhile, the low motion intensity of the third image block cannot cause the trailing phenomenon, and the better filtering effect is favorably ensured.

Finally, in order to perform denoising filtering processing on the current frame, the fusion filtering module 733 according to the embodiment of the present application performs fusion denoising filtering processing on the current frame and/or the aligned frame according to time-domain filtering or spatial filtering. The spatial filtering may include mean filtering, median filtering, gaussian filtering, bilateral filtering, or the like. Therefore, the current frame is subjected to noise reduction filtering according to spatial filtering to output a target frame, so that an image block with stronger motion intensity on a motion track is filtered through the spatial filtering, original information in the image block is kept as much as possible, and the trailing phenomenon is avoided.

Additionally, the temporal filtering may include linear weighted filtering. Therefore, the fusion noise reduction filtering of the current frame and the aligned frame according to the linear weighted filtering to output the target frame satisfies the following formula:

F _out ＝ω*F _t +(1-ω)*F _t-1 ；

In summary, an image block with a weaker motion intensity on a motion track is filtered through linear weighted filtering, so that noise in the image block is effectively removed.

Specifically, the third image block may be used to represent at least one image block of a plurality of image blocks of the current frame, which have equal size and are non-overlapping with each other. It is understood that the filtering decision module 732 in this embodiment of the present application may perform image blocking on the read current frame to obtain N × N image blocks with equal size and without overlapping.

Specifically, the fourth image block may be used to represent at least one image block of a plurality of image blocks of the aligned frame, which have equal size and do not overlap with each other. It is to be understood that the filtering decision module 732 in this embodiment of the present application may perform image blocking on the input aligned frames to obtain N × N image blocks with equal size and without overlapping, where N may be 2, 4, 8, or 16.

Specifically, the position of the third image block in the current frame has a corresponding relationship with the position of the fourth image block in the alignment frame. It is to be understood that, if the third image block is an image block in the first row block of the current frame, the second image block is an image block in the first row block of the aligned frame, and the rest of the same principles are known.

In one possible example, memory module 720 may include a double data rate DDR memory.

In particular, the DDR memory may include a double data rate synchronous dynamic random access memory (DDR SDRAM).

It should be noted that the present application focuses on the description of various embodiments. Therefore, the specific implementation functions of each module in the image processing apparatus 70 can refer to the relevant description in the embodiment described in fig. 1, and are not repeated here.

In summary, the flow of image processing in the embodiment of the present application is described below as an example, as shown in fig. 9. In fig. 9, the feature point data window of the reference frame and the current frame to be denoised are directly read from the DDR, the reference frame is the previous frame of the current frame, and the current frame is subjected to smooth filtering; parallelly executing the feature point detection of the current frame after smooth filtering and the searching and matching of a feature point data window of a reference frame in an image block of the current frame after smooth filtering; obtaining an optimal motion vector, namely a motion matching result, through searching and matching from a feature point data window of a reference frame to a current frame, and calculating motion adjustment information through the motion matching result, wherein the motion adjustment information is used for performing rotary alignment on the reference frame; uploading the characteristic point data window of the current frame obtained by the characteristic point detection of the current frame to a DDR for storage; reading a current frame and a reference frame from the DDR, and performing rotation alignment processing on the reference frame through motion adjustment information; the current frame and the reference frame after the rotation alignment are subjected to fusion noise reduction filtering to obtain a target frame, so that parallel processing of searching matching and feature point detection is realized by directly obtaining a feature point data window of the reference frame and the current frame, processing time delay in a video noise reduction process is reduced by parallel processing, and processing efficiency is improved. In addition, the feature point data window of the current frame is only directly uploaded, so that the residual redundant data in the current frame is prevented from being uploaded and stored, and the bandwidth of data uploading is reduced.

It can be seen that, in the embodiment of the present application, first, a first feature point data window in a first image block of a current frame and a reference frame is directly read from a storage module through a delay processing module; secondly, performing feature point detection of a second image block of the current frame and searching and matching of a first feature point data window of the reference frame in the second image block of the current frame in parallel; thirdly, outputting a motion adjustment information matching result to the noise reduction processing module through the delay processing module, and uploading the second characteristic point data window to the storage module for storage; and finally, rotationally aligning the reference frame through a noise reduction processing model to obtain an aligned frame, and performing noise reduction filtering processing on the current frame and/or the aligned frame to output a target frame, so that processing time delay in the video noise reduction process is reduced through parallel processing of searching matching and feature point detection, the processing efficiency is improved, and the condition that other redundant data in the current frame are uploaded and stored to reduce the bandwidth of data uploading is avoided by only uploading a feature point data window of the current frame.

In accordance with the above embodiments, please refer to fig. 10 for describing the image processing method according to the embodiments of the present application.

S1002, a first feature point data window in a first image block of a reference frame and a current frame are obtained.

Wherein the reference frame is used to represent an image frame temporally preceding the current frame.

Specifically, the current frame may be used to represent a frame of image in the video stream that is currently to be denoised.

Specifically, the reference frame may be a previous frame image of the current frame.

S1004, searching and matching the first feature point data window in a second image block of the current frame to obtain motion adjustment information, and performing feature point detection on the second image block in parallel to obtain a second feature point data window.

Specifically, the search matching may be understood as that a search range is determined in the image block of the current frame through a search matching algorithm, a most approximate (or matched) data window is found in the search range through the first feature point data window, and finally, the Motion Vector (MV) from the first feature point data window to the most approximate data window is used to obtain the motion adjustment information. The search matching algorithm may include a full search matching algorithm, a three-step search algorithm, a diamond search algorithm, a four-step search algorithm, a continuous filtering algorithm, a multi-layer filtering algorithm, a partial distortion removal algorithm, and the like.

It should be noted that, if the hash corner detection is adopted to perform the feature point detection on the image block of the current frame to obtain the feature point data window, the feature point data window may be used to represent the corner point in the image block, or the feature point data window may be used to represent an image area and the like composed of the corner point in the image block and pixel points around the corner point.

And S1006, performing block rotation and alignment processing on the reference frame by using the motion adjustment information to obtain an aligned frame.

It should be noted that, in order to reduce a residual value between each pixel in the image block of the current frame and each pixel in the image block of the reference frame, the embodiment of the present application considers that the block rotation and alignment processing is performed on the reference frame through the motion adjustment information.

And S1008, carrying out noise reduction filtering processing on the current frame and/or the alignment frame to obtain a target frame.

It can be seen that, in the embodiment of the present application, first, a first feature point data window and a current frame in a first image block of a reference frame are obtained; secondly, searching and matching a first feature point data window of the reference frame in a second image block of the current frame and detecting feature points of the second image block of the current frame in parallel; and finally, performing block rotation and alignment on the reference frame by utilizing the motion adjustment information to obtain an aligned frame, and performing noise reduction filtering processing on the current frame and/or the aligned frame to output a target frame, so that parallel processing of searching matching and feature point detection is realized by directly obtaining a first feature point data window and the current frame of the reference frame, processing time delay in the video noise reduction process is reduced by parallel processing, and processing efficiency is improved.

In combination with the above description, the following specifically describes the technical solutions of the embodiments of the present application.

Specifically, the searching and matching the first feature point data window in the second image block in S1004 to obtain the motion adjustment information may include: searching and matching the first feature point data window in the second image block to obtain a motion vector; and performing data fitting processing on the motion vector to obtain a homography matrix, and adjusting the motion of the homography matrix by using the homography matrix.

Further, search matching the first feature point data window in the second image block to obtain a motion vector may include: calculating the sum of absolute errors between the gray value of each pixel point in the first characteristic point data window and the gray value of each pixel point in the data window in the searching range of the second image block to obtain a first parameter value; taking a data window corresponding to the minimum value in the first parameter values as a target data window; and calculating the relative displacement from the first characteristic point data window to the target data window to obtain a motion vector.

Specifically, the performing noise reduction filtering processing on the current frame and/or the aligned frame in S1008 to obtain the target frame may include: acquiring a third image block of the current frame and a fourth image block of the alignment frame; carrying out average absolute error calculation on the gray value of each pixel point in the third image block and the gray value of each pixel point in the fourth image block to obtain a second parameter value; determining noise reduction filtering strategy information aiming at the current frame and the aligned frame according to a comparison result between the second parameter value and a preset threshold value, wherein the noise reduction filtering strategy information comprises time domain filtering or space domain filtering; and carrying out noise reduction filtering processing on the current frame and/or the aligned frame according to the noise reduction filtering strategy information to obtain a target frame.

Specifically, after the step of obtaining the first feature point data window in the first image block of the reference frame and the current frame in S1002, the method further includes: and uploading the second characteristic point data window to a bus for storage.

The bus may include DDR memory, among other things.

It should be noted that, in the embodiment of the present application, only the feature point data window of the current frame is uploaded to the bus for storage, and the other redundant data features of the current frame are not uploaded for storage, which is not only beneficial to avoiding uploading and storing the redundant data in the current frame to reduce the bandwidth of data uploading, but also beneficial to directly reading the feature point data window of the current frame from the storage module without performing feature point detection when searching and matching are performed on the next frame image of the current frame in the following process, thereby realizing reduction of processing delay in the video denoising process, and improving processing efficiency.

Specifically, after the step of obtaining the first feature point data window in the first image block of the reference frame and the current frame in S1002, the method further includes: and performing smooth filtering processing on the current frame, and outputting the current frame after the smooth filtering processing.

Specifically, after the first feature point data window in the first image block of the reference frame and the current frame are obtained in S1002, the method further includes: and performing blocking processing on the current frame to obtain at least one image block, wherein the at least one image block comprises a second image block.

It should be noted that the present application focuses on the description of various embodiments. Therefore, the technical solution in the embodiment shown in fig. 10, which is not described in detail, may refer to the related description in the embodiment shown in fig. 1 or 7, and is not described in detail again.

In combination with the above description, an embodiment of the present application further provides a processing chip, where the processing chip includes the image processing apparatus described above.

It should be noted that the description of the embodiments in this application has all its emphasis. Therefore, for specific implementation functions of each module in the image processing apparatus of the processing chip, reference may be made to the relevant description in the above embodiments, and details are not repeated here.

In addition, the image processing device in the processing chip executes search matching and feature point detection in parallel and uploads only the feature point data window of the current frame, so that processing time delay in the video noise reduction process is reduced, processing efficiency is improved, and uploading and storing of the rest redundant data in the current frame are avoided to reduce data uploading bandwidth.

For example, please refer to fig. 11, wherein fig. 11 is a schematic structural diagram of a processing chip according to an embodiment of the present disclosure. The processing chip 1100 includes an image processing apparatus 1110 and a communication bus for connecting the image processing apparatus 1110. In particular, the processing chip 1100 may be a processor. The processor may include a Central Processing Unit (CPU), an Application Processor (AP), a modem processor, a Graphic Processing Unit (GPU), an Image Signal Processor (ISP), a video codec, a Digital Signal Processor (DSP), a Field Programmable Gate Array (FPGA), an application-specific integrated circuit (ASIC), a baseband processor, and/or a neural Network Processor (NPU), and the like. The different processors may be independent devices, or may be integrated in the same processor.

Further, the processing chip 1100 may also include a memory for storing instructions and data. In some embodiments, the memory in the processing chip 1100 may be a cache memory. The memory may hold instructions or data that have just been used or recycled by the processor. If the processing chip 1100 needs to reuse the instructions or data, it can be called directly from the memory, thereby avoiding repeated accesses and reducing the latency of the processing chip 1100 to improve system efficiency.

Further, the processing chip 1100 may further include one or more interfaces. The interface may include an integrated circuit (I2C) interface, an integrated circuit built-in audio (I2S) interface, a Pulse Code Modulation (PCM) interface, a universal asynchronous receiver/transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a general-purpose-input/output (GPIO) interface, a Subscriber Identity Module (SIM) interface, and/or a Universal Serial Bus (USB) interface, etc.

In combination with the foregoing description, an embodiment of the present application further provides an electronic device, where the electronic device includes a memory, a communication interface, and the above-mentioned image processing apparatus.

It should be noted that the present application focuses on the description of various embodiments. Therefore, for specific implementation functions of each module in the image processing apparatus of the electronic device, reference may be made to the relevant description in the foregoing embodiments, and details are not repeated here.

In addition, the image processing device in the electronic equipment executes search matching and feature point detection in parallel and uploads only the feature point data window of the current frame, so that the processing time delay in the video noise reduction process is reduced, the processing efficiency is improved, and the situation that the rest redundant data in the current frame are uploaded and stored to reduce the data uploading bandwidth is avoided.

For example, please refer to fig. 12, and fig. 12 is a schematic structural diagram of an electronic device according to an embodiment of the present application. The electronic apparatus 1200 includes a memory 1201, a communication interface 1202, an image processing apparatus 1203, and a communication bus for connecting the memory 1201, the communication interface 1202, and the image processing apparatus 1203.

Specifically, the electronic device 1200 according to the embodiment of the present application may be a handheld device, a vehicle-mounted device, a wearable device, an Augmented Reality (AR) device, a Virtual Reality (VR) device, or other devices connected to a wireless modem, or may be various User Equipments (UEs), terminal devices (terminal devices), mobile phones (smart phones), smart screens, smart tvs, smart watches, laptops, stations (STAs), access Points (APs), mobile Stations (MSs), personal Digital Assistants (PDAs), personal Computers (PCs), or relay devices.

Specifically, the image processing device 1203 may be a processor. The processor may include a CPU, GPU, ISP, DSP, FPGA, ASIC, baseband processor and/or NPU, among others. When the processor is a CPU, the CPU may be a single-core CPU or a multi-core CPU.

Specifically, the memory 1201 may include a double data rate synchronous dynamic random access memory (DDR SDRAM), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), and/or a portable read-only memory (CD-ROM), and the memory 1201 is used to store related instructions and data.

In particular, the communication interface 1202 is used to receive and transmit data, signals or instructions or the like.

For the sake of simplicity, the above embodiments are described as a series of combinations of operations. It will be appreciated by those of skill in the art that the present application is not limited by the order of acts described, as some steps in the embodiments of the present application may occur in other orders or concurrently. Moreover, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that acts and modules referred to are not necessarily required to implement the embodiments of the application.

In the foregoing embodiments, the description of each embodiment in the present application has an emphasis, and reference may be made to relevant descriptions of other embodiments for parts that are not described in detail in a certain embodiment.

In the several embodiments provided in the present application, it should be understood by those skilled in the art that the described apparatus can be implemented in other ways. It will be appreciated that the above described apparatus embodiments are merely illustrative. For example, the division of the modules is only one logical function division, and actually, there may be another division manner. That is, multiple modules may be combined or integrated into a single module, and some features may be omitted or not implemented. In addition, the connections between the modules shown or discussed may be indirect coupling connections, direct coupling connections, or communication connections, may be through some interfaces, and may also be electrical or in other forms.

The above modules, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. It is to be understood that the technical solutions of the present application (some of which contribute to the prior art or all or part of which) may be embodied in the form of a computer-readable storage medium. The computer readable storage medium may be stored in a memory and includes instructions for causing a computer device (e.g., a personal computer, an electronic device, or a network device) to perform all or part of the steps of the embodiments of the present application. The computer-readable storage medium may be stored in various memories such as a usb disk, a ROM, a RAM, a portable hard disk, a magnetic disk, or an optical disk.

While the embodiments of the present application have been described in detail, it should be understood by those skilled in the art that the embodiments of the present application are only used for assisting understanding of the core concept of the technical solutions of the present application, and therefore, the embodiments of the present application may be changed in terms of the specific implementation and the application scope. The contents described in the present specification should not be construed as limiting the scope of the present application. In addition, any modification, equivalent replacement, improvement and the like made on the basis of the technical solutions of the embodiments of the present application should be included in the protection scope of the embodiments of the present application.

Claims

1. An image processing apparatus characterized by comprising: the device comprises a delay processing module and a noise reduction processing module; wherein,

2. The apparatus of claim 1, wherein the delay processing module comprises: the device comprises a characteristic point detection module, a motion estimation module and a motion adjustment module; wherein,

the feature point detection module is used for performing feature point detection on the second image block to obtain a second feature point data window;

the motion estimation module is used for searching and matching the first characteristic point data window in the second image block to obtain a motion vector and outputting the motion vector to the motion adjustment module;

the motion adjustment module is configured to perform data fitting processing on the input motion vector to obtain a homography matrix, use the homography matrix as the motion adjustment information, and output the motion adjustment information to the noise reduction processing module.

3. The apparatus of claim 2, wherein the motion estimation module is configured to:

calculating the sum of absolute errors between the gray value of each pixel point in the first characteristic point data window and the gray value of each pixel point in the data window in the searching range of the second image block to obtain a first parameter value;

taking a data window corresponding to the minimum value in the first parameter values as a target data window;

and calculating the relative displacement from the first characteristic point data window to the target data window to obtain the motion vector, and outputting the motion vector.

4. The apparatus of claim 2, wherein the delay processing module further comprises an image blocking module; wherein,

and the image blocking module is used for carrying out image blocking processing on the current frame and outputting the current frame subjected to image blocking processing.

5. The apparatus of claim 1, wherein the noise reduction processing module comprises: the device comprises a rotation alignment module, a filtering decision module and a fusion filtering module; wherein,

the rotation alignment module is configured to call the motion adjustment information to perform block rotation and alignment processing on the read reference frame to obtain an aligned frame, and output the aligned frame;

the filtering decision module is used for calculating the average absolute error of the read gray value of each pixel point in the third image block of the current frame and the input gray value of each pixel point in the fourth image block of the alignment frame to obtain a second parameter value; determining denoising filtering strategy information aiming at the current frame and the aligned frame according to a comparison result between the second parameter value and a preset threshold value, and outputting the denoising filtering strategy information to the fusion filtering module, wherein the denoising filtering strategy information comprises time domain filtering or space domain filtering;

and the fusion filtering module is used for performing noise reduction filtering processing on the read current frame and/or the input alignment frame according to the input noise reduction filtering strategy information to output the target frame.

6. The apparatus of any one of claims 1-5, further comprising: a storage module;

the storage module is configured to store at least one of the current frame, the reference frame, the first feature point data window, and the second feature point data window.

7. The apparatus of claim 6, wherein the memory module comprises a Double Data Rate (DDR) memory.

8. An image processing method, comprising:

9. The method of claim 8, wherein said search matching the first window of feature point data within the second block of images to obtain motion adjustment information comprises:

searching and matching the first characteristic point data window in the second image block to obtain a motion vector;

and performing data fitting processing on the motion vector to obtain a homography matrix, and taking the homography matrix as the motion adjustment information.

10. The method of claim 9, wherein said search matching said first window of feature point data within said second block of images to obtain motion vectors comprises:

and calculating the relative displacement from the first characteristic point data window to the target data window to obtain the motion vector.

11. The method according to claim 8, wherein the performing noise reduction filtering processing on the current frame and/or the aligned frame to obtain a target frame comprises:

acquiring a third image block of the current frame and a fourth image block of the aligned frame;

carrying out average absolute error calculation on the gray value of each pixel point in the third image block and the gray value of each pixel point in the fourth image block to obtain a second parameter value;

determining denoising filtering strategy information aiming at the current frame and the aligned frame according to a comparison result between the second parameter value and a preset threshold value, wherein the denoising filtering strategy information comprises time domain filtering or space domain filtering;

and carrying out noise reduction filtering processing on the current frame and/or the aligned frame according to the noise reduction filtering strategy information to obtain the target frame.

12. The method according to any of claims 8-11, wherein after performing feature point detection on the second image block in parallel to obtain a second feature point data window, the method further comprises:

and uploading the second characteristic point data window to a bus for storage.

13. The method according to any of claims 8-12, wherein after obtaining the first feature point data window in the first image block of the reference frame and the current frame, the method further comprises:

and performing blocking processing on the current frame to obtain at least one image block, wherein the at least one image block comprises the second image block.

14. A processing chip characterized by comprising the image processing apparatus according to any one of claims 1 to 7.

15. An electronic device characterized by comprising a memory, a communication interface and an image processing apparatus according to any one of claims 1 to 7.