CN112241982A - Image processing method and device and machine-readable storage medium - Google Patents
Image processing method and device and machine-readable storage medium Download PDFInfo
- Publication number
- CN112241982A CN112241982A CN201910651713.7A CN201910651713A CN112241982A CN 112241982 A CN112241982 A CN 112241982A CN 201910651713 A CN201910651713 A CN 201910651713A CN 112241982 A CN112241982 A CN 112241982A
- Authority
- CN
- China
- Prior art keywords
- image
- roi
- frame
- video frame
- monitoring video
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 15
- 238000012545 processing Methods 0.000 claims abstract description 106
- 238000012544 monitoring process Methods 0.000 claims abstract description 74
- 238000001514 detection method Methods 0.000 claims abstract description 29
- 230000005540 biological transmission Effects 0.000 claims abstract description 26
- 238000000034 method Methods 0.000 claims abstract description 25
- 230000002708 enhancing effect Effects 0.000 claims abstract description 3
- 238000013528 artificial neural network Methods 0.000 claims description 9
- 230000000295 complement effect Effects 0.000 claims description 8
- 238000005457 optimization Methods 0.000 abstract description 12
- 230000000694 effects Effects 0.000 abstract description 4
- 230000009466 transformation Effects 0.000 description 16
- 238000010586 diagram Methods 0.000 description 9
- 230000006835 compression Effects 0.000 description 5
- 238000007906 compression Methods 0.000 description 5
- 238000011176 pooling Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 238000013519 translation Methods 0.000 description 5
- 238000013527 convolutional neural network Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000004913 activation Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 241001465754 Metazoa Species 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000003595 spectral effect Effects 0.000 description 2
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 1
- 230000003139 buffering effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000009432 framing Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 238000013441 quality evaluation Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/90—Dynamic range modification of images or parts thereof
- G06T5/94—Dynamic range modification of images or parts thereof based on local image properties, e.g. for local contrast enhancement
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N17/00—Diagnosis, testing or measuring for television systems or their details
- H04N17/002—Diagnosis, testing or measuring for television systems or their details for television cameras
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20092—Interactive image processing based on input by user
- G06T2207/20104—Interactive definition of region of interest [ROI]
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
The application provides an image processing method, an image processing device and a machine-readable storage medium, wherein the method comprises the following steps: carrying out target detection and tracking on the monitoring video frames to determine the position information of the same target in each monitoring video frame; determining a region of interest (ROI) corresponding to the same target in each monitoring video frame according to the position information of the target in each monitoring video frame; enhancing the image sequence of the ROI area based on the quality score of the ROI area in each monitoring video frame; and carrying out coding transmission on the image sequence after the enhancement processing. The method can improve the effect of image quality optimization.
Description
Technical Field
The present application relates to the field of video monitoring, and in particular, to an image processing method and apparatus, and a machine-readable storage medium.
Background
In recent years, with the continuous development of science and technology, more and more intelligent cameras are applied to monitoring scenes to complete specific tasks. For example, the face snapshot machine can detect the face in the monitoring picture in real time and transmit the detected face image to the server for face recognition; the traffic snapshot machine can help the traffic management department to quickly and effectively monitor illegal vehicles and snapshot and evidence the illegal vehicles.
However, due to the influence of factors such as illumination, camera erection height, and target motion, the quality of the region of interest (ROI) in the finally obtained target image generally has a large difference, and some images have good quality, but there may be poor quality conditions such as blur, insufficient brightness, insufficient contrast, and the like.
In view of the above situation, the current solution is to perform optimization processing on decoded image data at the back end, improve the brightness and contour details of the ROI in the monitored image, and control the color of the ROI, thereby achieving the purpose of improving the image definition of the ROI.
Practice shows that the information when the ROI is processed by the solution is the decoded image data after network transmission, and the information is seriously lost due to coding compression at the moment, so that the improvement of subsequent processing is limited; in addition, the current solution only processes the traffic vehicle information, and the applicable scenarios are limited.
Disclosure of Invention
In view of the above, the present application provides an image processing method and an apparatus thereof.
Specifically, the method is realized through the following technical scheme:
according to a first aspect of the embodiments of the present application, there is provided an image processing method applied to a video monitoring front-end device, the method including:
carrying out target detection and tracking on the monitoring video frames to determine the position information of the same target in each monitoring video frame;
determining the corresponding ROI (region of interest) of the target in each monitoring video frame according to the position information of the same target in each monitoring video frame;
intercepting the ROI area from the cached monitoring video frame in the original format according to the quality score of the ROI area in each monitoring video frame, and performing enhancement processing on the image sequence of the intercepted ROI area in the original format;
and carrying out coding transmission on the image sequence after the enhancement processing.
According to a second aspect of the embodiments of the present application, there is provided an image processing apparatus applied to a video surveillance front-end device, the apparatus including:
the target detection unit is used for carrying out target detection and tracking on the monitoring video frames so as to determine the position information of the same target in each monitoring video frame;
the determining unit is used for determining a region of interest (ROI) corresponding to the target in each monitoring video frame according to the position information of the same target in each monitoring video frame;
the intercepting unit is used for intercepting the ROI area from the cached monitoring video frame in the original format according to the quality score of the ROI area in each monitoring video frame;
the enhancement processing unit is used for carrying out enhancement processing on the intercepted image sequence of the ROI in the original format;
and the transmission unit is used for coding and transmitting the image sequence after the enhancement processing.
According to a third aspect of embodiments of the present application, there is provided an image processing apparatus comprising a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor, the processor being caused by the machine-executable instructions to perform the above-mentioned image processing method.
According to a fourth aspect of embodiments of the present application, there is provided a machine-readable storage medium storing machine-executable instructions that, when invoked and executed by a processor, cause the processor to perform the above-described image processing method.
According to the image processing method, target detection and tracking are carried out on monitoring video frames through video monitoring front-end equipment, so that the position information of the same target in each monitoring video frame is determined, the ROI (region of interest) region of the target corresponding to each monitoring video frame is determined according to the position information of the same target in each monitoring video frame, further, the ROI region is intercepted from the cached monitoring video frame in the original format according to the quality score of the ROI region in each monitoring video frame, the intercepted image sequence of the ROI region in the original format is subjected to enhancement processing, and the image sequence subjected to enhancement processing is subjected to coding transmission, so that the influence of information loss in the compression transmission process on image quality optimization is avoided, and the image quality optimization effect is improved; in addition, the image quality optimization is no longer limited to vehicles, and the applicable scenes of the scheme are expanded.
Drawings
FIG. 1 is a flow chart illustrating a method of image processing according to an exemplary embodiment of the present application;
FIG. 2 is a schematic diagram illustrating a specific application scenario according to an exemplary embodiment of the present application;
FIG. 3A is a block diagram illustrating a first processing module according to an exemplary embodiment of the present application;
FIG. 3B is a schematic diagram of a second sub-processing unit according to an exemplary embodiment of the present application;
FIG. 3C is a block diagram of a second processing module according to an exemplary embodiment of the present application;
FIG. 4 is a schematic diagram of an image processing apparatus according to an exemplary embodiment of the present application;
fig. 5 is a schematic diagram of a hardware structure of an image processing apparatus according to an exemplary embodiment of the present application.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
In order to make those skilled in the art better understand the technical solutions provided by the embodiments of the present application, the following is a brief description of the original format described in the present application.
Raw data format refers to a raw data format in which the image sensor converts the captured light source signals into digital signals, the raw data containing sensing data from one or more spectral bands.
Illustratively, the raw data may include sensed data sampled for optical signals in the spectral band having wavelengths in the range of 380nm to 780nm, and/or 780nm to 2500 nm.
For example, an RGB sensor senses a resulting RAW (unprocessed) image signal.
Illustratively, the imaging device collects a light source signal, converts the collected light source signal into an analog signal, converts the analog signal into a digital signal, inputs the digital signal into a processing chip for processing (the processing may include bit width clipping, image processing, encoding and decoding processing, and the like), obtains data in a second data format (the original format may be referred to as a first data format), and transmits the data in the second data format to a display device for display or other devices for processing.
Therefore, the image in the original format is the image when the sensor converts the acquired light source information into the digital signal, the image is not processed by the processing chip, the bit width is high, and the image contains more abundant image information compared with the image in the second data format which is subjected to bit width cutting, image processing and encoding and decoding processing.
In order to make the aforementioned objects, features and advantages of the embodiments of the present application more comprehensible, embodiments of the present application are described in detail below with reference to the accompanying drawings.
Referring to fig. 1, a schematic flowchart of an image processing method provided in an embodiment of the present application is shown, where the image processing method may be applied to a video monitoring front-end device, such as an IPC (Internet Protocol Camera), and as shown in fig. 1, the image processing method may include the following steps:
for convenience of description and understanding, the following description will be made taking the execution subject of steps S100 to S130 as an example of IPC.
Step S100, carrying out target detection and tracking on the monitoring video frames to determine the position information of the same target in each monitoring video frame.
In the embodiment of the application, when the IPC acquires the surveillance video, the IPC may perform target detection on the surveillance video frames, and track the target after the target is detected, so as to determine the position information of the same target in each surveillance video frame.
Illustratively, the targets may include, but are not limited to, pedestrians, vehicles, animals, license plates, and the like.
In an example, the performing target detection and tracking on the surveillance video frames to determine the position information of the same target in each surveillance video frame may include:
and based on the neural network, carrying out target detection and tracking on the monitoring video frames so as to determine the position information of the same target in each monitoring video frame.
For example, in order to implement target detection and tracking of the surveillance video frames, a neural network for target detection and tracking may be trained through a training sample subjected to target labeling, and target detection and tracking are performed on the surveillance video frames by using the trained neural network, so as to determine position information of the same target in each surveillance video frame.
Step S110, according to the position information of the same target in each monitoring video frame, determining the ROI area of the corresponding interesting area of the target in each monitoring video frame.
In the embodiment of the application, for any object, the IPC may determine the corresponding ROI area of the object in each monitored video frame according to the position information of the object in each monitored video frame.
For example, for any surveillance video frame, the target framing area in the surveillance video frame may be determined as the corresponding ROI area of the target in the surveillance video frame.
And step S120, intercepting the ROI area from the cached monitoring video frame in the original format according to the quality score of the ROI area in each monitoring video frame, and performing enhancement processing on the image sequence of the ROI area in the original format.
In the embodiment of the application, when the video monitoring front-end equipment performs target detection and tracking on the monitoring video frame, the monitoring video frame in the original format can be cached.
For any object, when the IPC determines the ROI area of the object in each monitored video frame, the IPC may intercept the ROI area from the cached monitored video frame in the original format according to the quality score of the ROI area in each monitored video frame, and perform enhancement processing on the image sequence of the intercepted ROI area in the original format.
In a possible implementation manner, the intercepting the ROI from the buffered original format surveillance video frame according to the quality score of the ROI in each surveillance video frame may include:
determining a quality score of an ROI area in each monitoring video frame;
and intercepting the ROI area with the quality score higher than a preset score threshold value from the cached monitoring video frame in the original format based on the quality score of the ROI area in each monitoring video frame.
For example, for any object, when the IPC determines the ROI region of the object in each monitored video frame, the quality score can be performed on the ROI region in each monitored video frame.
For example, the quality of the ROI region may be scored according to target imaging quality (e.g., face sharpness, face angle, etc.) in the ROI region, or the quality of the ROI region may be scored according to quality of an image corresponding to the ROI region (e.g., image sharpness, brightness, contrast, etc.), which is not described herein in this embodiment.
In one example, a quality score for a ROI region in each surveillance video frame may be determined based on a neural network.
When the IPC determines the quality score of the ROI region in each monitored video frame, the monitored video frames with the quality score of the ROI region not higher than a preset score threshold (which may be set according to an actual application scene) are removed from the cached monitored video frames in the original format based on the quality score of the ROI region in each monitored video frame, and the monitored video frames with the quality score of the ROI region higher than the preset score threshold are subjected to ROI region interception to obtain an image sequence of the ROI region in the original format, and the intercepted image sequence of the ROI region in the original format is subjected to enhancement processing.
In an example, the performing enhancement processing on the image sequence of the intercepted ROI region in the original format may include:
selecting a reference frame from a plurality of frame images in an input image sequence of an ROI (region of interest) in an original format;
aligning the plurality of frame image pairs based on the reference frame;
and performing enhancement processing on the multi-frame images based on complementary information between the aligned multi-frame images.
Illustratively, when the image sequence of the ROI in the original format needs to be enhanced, a plurality of frames of images (the specific number of frames may be set according to the actual scene) may be selected from the image sequence of the ROI in the original format each time as the input of the enhancement process, and one frame of image may be selected from the input plurality of frames of images as the reference frame.
For example, the middle frame or the last frame in the multi-frame image may be selected as the reference frame.
For example, assuming that the number of the multi-frame images is 3, the 2 nd frame or the 3 rd frame may be selected as the reference frame.
After the reference frame is selected, the multiple frames of images may be aligned based on the reference frame, and the multiple frames of images may be enhanced based on complementary information between the aligned multiple frames of images.
And step S130, coding and transmitting the image sequence after the enhancement processing.
In the embodiment of the application, after the IPC performs enhancement processing on the image sequence of the corresponding ROI region according to the manner described in the above steps, the image sequence after enhancement processing may be encoded and compressed, and the encoded and compressed code stream is transmitted to the back-end device through the network.
For example, after the IPC performs the enhancement processing on the image sequence of the corresponding ROI region in the manner described in the above steps, a part or all of the image sequence may be selected from the image sequence after the enhancement processing, compressed and encoded, and transmitted to the backend device through the network.
In an example, the encoding and transmitting the image sequence after the enhancement processing may include:
and for the same target, selecting a frame of image from the image sequence after enhancement processing for coding transmission.
For any object, for example, after the IPC performs enhancement processing on the image sequence of the ROI corresponding to the object in the manner described in the above steps, a frame of image may be selected from the image sequence after enhancement processing for encoding and transmission.
For example, the IPC may determine the quality score of each enhanced image frame, and select the image frame with the highest quality score for encoding transmission.
It can be seen that, in the method flow shown in fig. 1, the video monitoring front-end device performs image quality optimization on the monitored video frame for the target in the monitored video, thereby avoiding the influence of information loss in the compression transmission process on the image quality optimization and improving the effect of the image quality optimization; in addition, the image quality optimization is no longer limited to vehicles, and the applicable scenes of the scheme are expanded.
In order to enable those skilled in the art to better understand the technical solutions provided in the embodiments of the present application, the following describes the technical solutions provided in the embodiments of the present application with reference to specific application scenarios.
Referring to fig. 2, a schematic structural diagram of a specific application scenario provided in the embodiment of the present application is shown in fig. 2, in the application scenario, a video monitoring front-end device may include a first processing module, an ROI region extraction module, a second processing module, and a third processing module.
The image processing flow of the present application will be described below with reference to the functions of the respective modules.
A first processing module
The first processing module is used for detecting the target and tracking the target after the target is detected.
For example, the first processing module may detect a target such as a pedestrian, a vehicle, an animal, or a license plate, perform accurate positioning after detecting the target, and output target position information after positioning is completed; and then, tracking the target around the target according to the positioned target position information, and determining the position information of the target in each frame of image.
For example, the first processing module may buffer the image sequence during the target detection, such as buffering the image sequence in a first data format (e.g., RGB (red green blue) format or YUV (brightness, chrominance, saturation) format).
In one example, referring to fig. 3A, the first processing module may include: the system comprises a first sub-processing unit, a second sub-processing unit and a third sub-processing unit.
The first sub-processing unit is used for carrying out target detection so as to determine target position information in each frame of image;
the second sub-processing unit is used for tracking the target so as to determine the position information of the same target in each frame of image;
the third sub-processing unit is used for evaluating the image quality of the target area and outputting an evaluation score (namely the quality score).
For example, the first sub-processing unit may be implemented by a neural network, and directly outputs the target coordinates. As shown in fig. 3B, the neural network for implementing the first sub-processing unit may include a convolutional layer (Conv), a pooling layer (Pool), a full connection layer (FC layer), and a Bounding Box Regression (BBR).
For example, the operation of a convolutional layer may be represented by the following formula:
YCi(I)＝g(Wi*YCi-1(I)+Bi)
wherein, YCi(I) Is the output of the current convolutional layer, YCi-1(I) For the input of the current convolution layer, denotes the convolution operation, WiAnd BiWeight coefficients and offset coefficients of the convolution filter of the current convolution layer, respectively, g () represents an activation function, when the activation function is ReLU, g (x) is max (0, x).
The pooling layer is a special down-sampling layer, i.e. the feature map obtained by convolution is reduced, the size of a reduction window is NxN, and when the maximum pooling is used, the maximum value is obtained from the NxN window to be used as the value of the corresponding point of the latest image, and the specific formula is as follows:
YPj(I)＝maxpool(YPj-1(I))
wherein, YCi(I) For the input of the jth pooling layer, YPj(I) Is the output of the jth pooling layer.
The full connection layer FC can be regarded as a convolution layer with a filter window of 1 × 1, and is implemented similarly to convolution filtering, where the expression is as follows:
wherein, FkI(I) For input to the kth fully-connected layer, YFk(I) Is the output of the kth fully-connected layer, R, C is FkI(I) Width and height of (W)ijAnd BijThe connection weights and offsets, respectively, for the fully-connected layer, g () represents the activation function.
The frame regression BBR is to find a relation such that the window P output by the full link layer is mapped to obtain a window G' closer to the real window G. The regression is typically performed by a translation or scaling transformation of the window P.
Let the coordinate of the window P of the full link layer output be (x)1，x2，y1，y2) Transformed coordinate (x) after window3，x4，y3，y4) If the translation is converted into translation transformation, the translation scale is (Δ x, Δ y), and the coordinate relationship before and after translation is as follows:
x3＝x1+Δx
x4＝x2+Δx
y3＝y1+Δy
y4＝y2+Δy
if the transformation is scaling transformation, the scaling in direction X, Y is dx and dy, and the coordinate relationship before and after transformation is:
x4-x3＝(x2-x1)*dx
y4-y3＝(y2-y1)*dy
the second sub-processing unit is used for positioning a target in the current frame according to the target position information of the current image frame detected by the first sub-processing unit, comparing the similarity of the target of the current image frame with the target of the previous image frame, if the similarity is higher than a set threshold, the target is considered to be the same target, and updating the target position information, so that target tracking is realized.
The third sub-processing unit is used for evaluating the image quality of the target area of the current image frame to obtain a quality score.
Taking the image quality evaluation of the target area by the average brightness as an example, the average brightness of the target area can be calculated as follows:
wherein L ismRepresents the average luminance of the target area, n represents the total number of pixels of the target area, and R represents the target area.
The calculation formula of the quality score of the target area is as follows:
S＝100-|80-Lm|
the higher the quality score, the better the representative image quality. .
ROI area intercepting module
The ROI area intercepting module is used for intercepting an image of an ROI area (namely the target area) in a multi-frame image (an original image) based on the position information of the target.
Illustratively, the ROI region truncation module may truncate an ROI region image from the cached image sequence in the first data format based on the position information of the target.
When the ROI area image interception is carried out on the original image, the original image with the quality score of the target area not higher than a preset score threshold value is removed, the original image with the quality score of the target area higher than the preset score threshold value is reserved, and the ROI area image interception is carried out.
Third, second processing module
The second processing module is used for aligning the multi-frame ROI area images and enhancing the multi-frame ROI area images according to complementary information among the multi-frame ROI area images.
In one example, as shown in fig. 3C, the second processing module includes: a fourth sub-processing unit and a fifth sub-processing unit.
The fourth sub-processing unit is used for selecting one frame from the input multi-frame ROI area images as a reference frame, and carrying out position transformation on other frames based on the reference frame to enable the other frames to be aligned with the reference frame;
and the fifth sub-processing unit is used for performing enhancement processing by using complementary information between the multiple frames of ROI area images according to the aligned multiple frames of ROI area images.
In one example, after performing the enhancement processing on the input multiple frames of ROI area images in the above manner, one frame of enhanced ROI area image (which may be a reference frame enhanced by using complementary information between the multiple frames of ROI area images) may be output.
For example, the fourth sub-processing unit may or may not be implemented by a convolutional neural network.
For example, if the fourth sub-processing unit is implemented by a non-convolutional neural network, feature points of a target image (such as any frame of ROI region image except a reference frame in the aforementioned multiple frames of ROI region images) and a reference image (i.e., the aforementioned reference frame) may be extracted by using a feature point detection method such as Scale-invariant feature transform (SIFT), Harris corner detection, and the like; then calculating according to the position information among the characteristic points to obtain image transformation parameters; and finally, converting the multi-frame image to obtain aligned multi-frame image data.
Methods of image transformation may include, but are not limited to, projective transformation, affine transformation, rotational-translational transformation, and the like.
Taking the example of implementing image transformation by rotational-translational transformation, the image transformation formula is as follows:
wherein x and y represent the transformed coordinates of each pixel point of the image, w and z represent the coordinates of each pixel point of the image, and alpha and dhor、dverAnd parameters representing the image transformation obtained by the transformation parameter calculation module.
Illustratively, the fifth sub-processing unit is configured to enhance the images of the multiple frames of ROI regions, and the enhancement may be implemented by a convolutional neural network or by a non-convolutional neural network, which is not described herein again in this embodiment of the present application.
Fourth and third processing module
And the third processing module is used for carrying out compression coding on the ROI area image after enhancement processing and transmitting the ROI area image to the back-end equipment through a network.
In the embodiment of the application, the video monitoring front-end equipment is used for carrying out target detection and tracking on monitoring video frames to determine the position information of the same target in each monitoring video frame, and determining the ROI (region of interest) corresponding to the target in each monitoring video frame according to the position information of the same target in each monitoring video frame, and further intercepting the ROI from the cached monitoring video frame in the original format according to the quality score of the ROI in each monitoring video frame, carrying out enhancement processing on the image sequence of the intercepted ROI in the original format, and carrying out coding transmission on the image sequence after enhancement processing, so that the influence of information loss in the compression transmission process on image quality optimization is avoided, and the image quality optimization effect is improved; in addition, the image quality optimization is no longer limited to vehicles, and the applicable scenes of the scheme are expanded.
The methods provided herein are described above. The following describes the apparatus provided in the present application:
referring to fig. 4, a schematic structural diagram of an image processing apparatus according to an embodiment of the present disclosure is shown in fig. 4, where the image processing apparatus may include:
a target detection unit 410, configured to perform target detection and tracking on the monitored video frames to determine position information of the same target in each monitored video frame;
a determining unit 420, configured to determine, according to position information of the same target in each monitored video frame, a region of interest ROI corresponding to the target in each monitored video frame;
an intercepting unit 430, configured to intercept an ROI from the cached surveillance video frame in the original format according to the quality score of the ROI in each surveillance video frame;
an enhancement processing unit 440, configured to perform enhancement processing on the captured image sequence of the ROI in the original format;
a transmitting unit 450, configured to perform encoding transmission on the image sequence after the enhancement processing.
In an optional implementation manner, the target detection unit 410 is specifically configured to perform target detection and tracking on the surveillance video frames based on a neural network, so as to determine position information of the same target in each surveillance video frame.
In an optional embodiment, the determining unit 420 is further configured to determine a quality score of a ROI region in each of the monitored video frames;
the intercepting unit 430 is specifically configured to intercept, from the cached monitoring video frames in the original format, the ROI area with the quality score higher than the preset score threshold based on the quality score of the ROI area in each monitoring video frame.
In an optional implementation manner, the enhancement processing unit 440 is specifically configured to select a reference frame from a plurality of frame images in an input image sequence of the ROI region in the original format; aligning the plurality of image pairs based on the reference frame; and performing enhancement processing on the multi-frame images based on the complementary information between the aligned multi-frame images.
In an alternative embodiment, the transmission unit 450 is specifically configured to select, for the same target, a part or all of image frames from the enhanced image sequence for encoding transmission.
In an optional implementation manner, the transmission unit 450 is specifically configured to select a frame of image with the highest quality score from the image sequence after the enhancement processing for encoding and transmitting.
Fig. 5 is a schematic diagram of a hardware structure of an image processing apparatus according to an embodiment of the present disclosure. The image processing apparatus may include a processor 501, a machine-readable storage medium 502 storing machine-executable instructions. The processor 501 and the machine-readable storage medium 502 may communicate via a system bus 503. Also, the processor 501 may perform the image processing method described above by reading and executing machine-executable instructions in the machine-readable storage medium 502 corresponding to the image processing logic.
The machine-readable storage medium 502 referred to herein may be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, and the like. For example, the machine-readable storage medium may be: a RAM (random Access Memory), a volatile Memory, a non-volatile Memory, a flash Memory, a storage drive (e.g., a hard drive), a solid state drive, any type of storage disk (e.g., an optical disk, a dvd, etc.), or similar storage medium, or a combination thereof.
Embodiments of the present application also provide a machine-readable storage medium, such as machine-readable storage medium 502 in fig. 5, comprising machine-executable instructions that are executable by processor 501 in an image processing apparatus to implement the image processing method described above.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the scope of protection of the present application.
Claims (14)
1. An image processing method is applied to video monitoring front-end equipment, and the method comprises the following steps:
carrying out target detection and tracking on the monitoring video frames to determine the position information of the same target in each monitoring video frame;
determining a region of interest (ROI) corresponding to the same target in each monitoring video frame according to the position information of the target in each monitoring video frame;
intercepting the ROI area from the cached monitoring video frame in the original format according to the quality score of the ROI area in each monitoring video frame, and performing enhancement processing on the image sequence of the intercepted ROI area in the original format;
and carrying out coding transmission on the image sequence after the enhancement processing.
2. The method of claim 1, wherein the performing object detection and tracking on the surveillance video frames to determine the position information of the same object in each surveillance video frame comprises:
and based on a neural network, carrying out target detection and tracking on the monitoring video frames so as to determine the position information of the same target in each monitoring video frame.
3. The method of claim 1, wherein the step of intercepting the ROI from the buffered surveillance video frames in the original format according to the quality score of the ROI in each surveillance video frame comprises:
determining a quality score of the ROI area in each monitoring video frame;
and intercepting the ROI area with the quality score higher than a preset score threshold value from the cached monitoring video frame in the original format based on the quality score of the ROI area in each monitoring video frame.
4. The method according to claim 3, wherein the enhancing the image sequence of the intercepted ROI area in original format comprises:
selecting a reference frame from a plurality of frame images in the input image sequence of the ROI in the original format;
aligning the plurality of image pairs based on the reference frame;
and performing enhancement processing on the multi-frame images based on the complementary information between the aligned multi-frame images.
5. The method according to claim 1, wherein said encoding transmission of the image sequence after enhancement processing comprises:
and for the same target, selecting part or all image frames from the image sequence after the enhancement processing for coding transmission.
6. The method of claim 5, wherein the selecting a portion of the image frame from the enhanced image sequence for encoded transmission comprises:
and selecting a frame of image with the highest quality score from the image sequence after the enhancement processing for coding and transmission.
7. An image processing apparatus, applied to a video surveillance front-end device, the apparatus comprising:
the target detection unit is used for carrying out target detection and tracking on the monitoring video frames so as to determine the position information of the same target in each monitoring video frame;
the determining unit is used for determining a region of interest (ROI) corresponding to the target in each monitoring video frame according to the position information of the same target in each monitoring video frame;
the intercepting unit is used for intercepting the ROI area from the cached monitoring video frame in the original format according to the quality score of the ROI area in each monitoring video frame;
the enhancement processing unit is used for carrying out enhancement processing on the intercepted image sequence of the ROI in the original format;
and the transmission unit is used for coding and transmitting the image sequence after the enhancement processing.
8. The apparatus of claim 7,
the target detection unit is specifically configured to perform target detection and tracking on the surveillance video frames based on a neural network, so as to determine position information of the same target in each surveillance video frame.
9. The apparatus of claim 7,
the determining unit is further configured to determine a quality score of the ROI region in each of the monitored video frames;
the intercepting unit is specifically configured to intercept, from the cached monitoring video frames in the original format, the ROI region whose quality score is higher than a preset score threshold based on the quality score of the ROI region in each monitoring video frame.
10. The apparatus of claim 9,
the enhancement processing unit is specifically used for selecting a reference frame from a plurality of frame images in an input image sequence of an ROI (region of interest) in an original format; aligning the plurality of image pairs based on the reference frame; and performing enhancement processing on the multi-frame images based on the complementary information between the aligned multi-frame images.
11. The apparatus of claim 7,
the transmission unit is specifically configured to select, for the same target, a part or all of the image frames from the image sequence after the enhancement processing for encoding transmission.
12. The apparatus of claim 11,
the transmission unit is specifically configured to select a frame of image with the highest quality score from the image sequence after the enhancement processing for encoding and transmission.
13. An image processing apparatus comprising a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor, the processor being caused by the machine-executable instructions to perform the method of any one of claims 1 to 6.
14. A machine-readable storage medium having stored thereon machine-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method of any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910651713.7A CN112241982A (en) | 2019-07-18 | 2019-07-18 | Image processing method and device and machine-readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910651713.7A CN112241982A (en) | 2019-07-18 | 2019-07-18 | Image processing method and device and machine-readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112241982A true CN112241982A (en) | 2021-01-19 |
Family
ID=74167920
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910651713.7A Pending CN112241982A (en) | 2019-07-18 | 2019-07-18 | Image processing method and device and machine-readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112241982A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114827567A (en) * | 2022-03-23 | 2022-07-29 | 阿里巴巴（中国）有限公司 | Video quality analysis method, apparatus and readable medium |
| CN116112645A (en) * | 2023-04-11 | 2023-05-12 | 重庆华悦生态环境工程研究院有限公司深圳分公司 | Multi-image transmission method and device for reservoir environment |
| WO2023179692A1 (en) * | 2022-03-25 | 2023-09-28 | 影石创新科技股份有限公司 | Motion video generation method and apparatus, terminal device, and storage medium |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110135151A1 (en) * | 2009-12-07 | 2011-06-09 | Samsung Electronics Co., Ltd. | Method and apparatus for selectively supporting raw format in digital image processor |
| CN104125470A (en) * | 2014-08-07 | 2014-10-29 | 成都瑞博慧窗信息技术有限公司 | Video data transmission method |
| CN106203497A (en) * | 2016-07-01 | 2016-12-07 | 浙江工业大学 | A kind of finger vena area-of-interest method for screening images based on image quality evaluation |
| CN106557767A (en) * | 2016-11-15 | 2017-04-05 | 北京唯迈医疗设备有限公司 | A kind of method of ROI region in determination interventional imaging |
| CN107124612A (en) * | 2017-04-26 | 2017-09-01 | 东北大学 | The method for compressing high spectrum image perceived based on distributed compression |
| US20180181833A1 (en) * | 2014-08-25 | 2018-06-28 | Agency For Science, Technology And Research | Methods and systems for assessing retinal images, and obtaining information from retinal images |
| CN109218695A (en) * | 2017-06-30 | 2019-01-15 | 中国电信股份有限公司 | Video image enhancing method, device, analysis system and storage medium |
| CN109242811A (en) * | 2018-08-16 | 2019-01-18 | 广州视源电子科技股份有限公司 | A kind of image alignment method and device thereof, computer readable storage medium and computer equipment |
| US20190050999A1 (en) * | 2017-08-14 | 2019-02-14 | Siemens Healthcare Gmbh | Dilated Fully Convolutional Network for Multi-Agent 2D/3D Medical Image Registration |
-
2019
- 2019-07-18 CN CN201910651713.7A patent/CN112241982A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110135151A1 (en) * | 2009-12-07 | 2011-06-09 | Samsung Electronics Co., Ltd. | Method and apparatus for selectively supporting raw format in digital image processor |
| CN104125470A (en) * | 2014-08-07 | 2014-10-29 | 成都瑞博慧窗信息技术有限公司 | Video data transmission method |
| US20180181833A1 (en) * | 2014-08-25 | 2018-06-28 | Agency For Science, Technology And Research | Methods and systems for assessing retinal images, and obtaining information from retinal images |
| CN106203497A (en) * | 2016-07-01 | 2016-12-07 | 浙江工业大学 | A kind of finger vena area-of-interest method for screening images based on image quality evaluation |
| CN106557767A (en) * | 2016-11-15 | 2017-04-05 | 北京唯迈医疗设备有限公司 | A kind of method of ROI region in determination interventional imaging |
| CN107124612A (en) * | 2017-04-26 | 2017-09-01 | 东北大学 | The method for compressing high spectrum image perceived based on distributed compression |
| CN109218695A (en) * | 2017-06-30 | 2019-01-15 | 中国电信股份有限公司 | Video image enhancing method, device, analysis system and storage medium |
| US20190050999A1 (en) * | 2017-08-14 | 2019-02-14 | Siemens Healthcare Gmbh | Dilated Fully Convolutional Network for Multi-Agent 2D/3D Medical Image Registration |
| CN109242811A (en) * | 2018-08-16 | 2019-01-18 | 广州视源电子科技股份有限公司 | A kind of image alignment method and device thereof, computer readable storage medium and computer equipment |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114827567A (en) * | 2022-03-23 | 2022-07-29 | 阿里巴巴（中国）有限公司 | Video quality analysis method, apparatus and readable medium |
| CN114827567B (en) * | 2022-03-23 | 2024-05-28 | 阿里巴巴（中国）有限公司 | Video quality analysis method, apparatus and readable medium |
| WO2023179692A1 (en) * | 2022-03-25 | 2023-09-28 | 影石创新科技股份有限公司 | Motion video generation method and apparatus, terminal device, and storage medium |
| CN116112645A (en) * | 2023-04-11 | 2023-05-12 | 重庆华悦生态环境工程研究院有限公司深圳分公司 | Multi-image transmission method and device for reservoir environment |
| CN116112645B (en) * | 2023-04-11 | 2023-11-21 | 重庆华悦生态环境工程研究院有限公司深圳分公司 | Multi-image transmission method and device for reservoir environment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11790504B2 (en) | Monitoring method and apparatus | |
| US20160358338A1 (en) | Image recognition apparatus, image recognition method, and recording medium | |
| JP2023526207A (en) | Maintaining a constant size of the target object in the frame | |
| TW202101371A (en) | Method and apparatus for processing video stream | |
| JP7077395B2 (en) | Multiplexed high dynamic range image | |
| US8582915B2 (en) | Image enhancement for challenging lighting conditions | |
| JP5478047B2 (en) | Video data compression pre-processing method, video data compression method and video data compression system using the same | |
| US8922674B2 (en) | Method and system for facilitating color balance synchronization between a plurality of video cameras and for obtaining object tracking between two or more video cameras | |
| US20110164789A1 (en) | Detection of vehicles in images of a night time scene | |
| CN112241982A (en) | Image processing method and device and machine-readable storage medium | |
| JP2007058634A (en) | Image processing method and image processor, digital camera equipment, and recording medium with image processing program stored thereon | |
| US8798369B2 (en) | Apparatus and method for estimating the number of objects included in an image | |
| US20230127009A1 (en) | Joint objects image signal processing in temporal domain | |
| CN111179302B (en) | Moving target detection method and device, storage medium and terminal equipment | |
| US20110085026A1 (en) | Detection method and detection system of moving object | |
| CN115661720A (en) | Target tracking and identifying method and system for shielded vehicle | |
| WO2019228450A1 (en) | Image processing method, device, and equipment, and readable medium | |
| CN112581481B (en) | Image processing method and device, electronic equipment and computer readable storage medium | |
| CN112241735A (en) | Image processing method, device and system | |
| KR101726692B1 (en) | Apparatus and method for extracting object | |
| TWI676965B (en) | Object image recognition system and object image recognition method | |
| CN111062272A (en) | Image processing and pedestrian identification method and device based on color recovery and readable storage medium | |
| EP4332910A1 (en) | Behavior detection method, electronic device, and computer readable storage medium | |
| CN116403200A (en) | License plate real-time identification system based on hardware acceleration | |
| US10140503B2 (en) | Subject tracking apparatus, control method, image processing apparatus, and image pickup apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |