CN115035466A

CN115035466A - Infrared panoramic radar system for safety monitoring

Info

Publication number: CN115035466A
Application number: CN202210591644.7A
Authority: CN
Inventors: 朱正; 董官清; 孙贝贝; 闫志坚; 曲悠扬; 李玉祥; 史金辉; 刘昕阳; 倪方淇; 刁伟建; 王晓峰
Original assignee: Harbin Engineering University
Current assignee: Harbin Engineering University
Priority date: 2022-05-27
Filing date: 2022-05-27
Publication date: 2022-09-09

Abstract

The invention belongs to the technical field of infrared photoelectric technology detection, and particularly relates to an infrared panoramic radar system for safety monitoring. The invention comprises an image acquisition module, an image processing module and an upper computer display and early warning module; the image acquisition module acquires images in a mode of combining a rotating holder and an infrared detector, so that the size of each image is the same, the bottom ends of the images are positioned on the same horizontal plane, and the subsequent splicing complexity is reduced; the image processing module synthesizes the plurality of infrared images with narrow view field and high spatial resolution into an infrared panoramic image with wide view field and high spatial resolution, and integrates a plurality of image information into one image, thereby being convenient for system monitoring. The invention adopts an infrared thermal imaging mode, uses a high-sensitivity infrared detector to be matched with a high-precision servo motor, and simultaneously combines panoramic image splicing and a target identification technology based on deep learning to realize real-time monitoring and effective tracking identification of infrared targets in the peripheral range.

Description

Infrared panoramic radar system for safety monitoring

Technical Field

The invention belongs to the technical field of infrared photoelectric technology detection, and particularly relates to an infrared panoramic radar system for safety monitoring.

Background

Today, with rapid development of science and technology, the security field gradually faces huge challenges, and various countermeasures aiming at the traditional security equipment are developed, so that related security situations are seriously influenced, and serious security accidents are caused.

In terms of security monitoring means for important areas, no matter domestic and foreign countries, existing measures mostly depend on equipment such as radars and the like to guarantee the safety of the areas, but the traditional devices can only effectively monitor large-scale targets, and the monitoring of small-scale targets such as suspicious personnel and the like is difficult to achieve. In contrast, people have the advantages of small size and low noise intensity, which makes it difficult for traditional radars to find relevant traces, and poses great challenges to the overall security situation.

In the face of this current situation, a conventional solution is to arrange a large number of monitoring cameras with fixed viewing angles in a monitoring area, and once the monitoring range is too large, a large number of monitoring devices need to be installed in the whole area to realize the overall monitoring of the whole area, which may bring expensive system cost and heavy installation, maintenance, and inspection tour.

Disclosure of Invention

The invention aims to provide an infrared panoramic radar system for safety monitoring.

An infrared panoramic radar system for safety monitoring comprises an image acquisition module, an image processing module and an upper computer display and early warning module; the image acquisition module comprises an infrared detector 201, the infrared detector 201 is installed on a rotating holder 202, the infrared detector 201 shoots a surrounding 360-degree environment through the rotating holder 202, and meanwhile, the bottommost end of each picture shot by the infrared detector 201 is ensured to be positioned on the same horizontal line; the image acquisition module continuously acquires surrounding infrared information through the infrared detector 201 to obtain scattered infrared images, and transmits the infrared images to the image processing module according to a shooting sequence; the image processing module is used for preprocessing the acquired infrared image, splicing the image, carrying out target identification on the spliced image and transmitting the image subjected to target identification to the upper computer display and early warning module in real time; and the upper computer display and early warning module displays the received image subjected to target identification processing in real time, judges whether to send an alarm or not according to the target identification result, and positions the position of the invading target.

Further, the method for preprocessing the acquired infrared image by the image processing module specifically comprises the following steps:

step 2.1: reading an original infrared image;

step 2.2: performing smooth filtering on the original infrared image by adopting a low-pass Gaussian filter kernel;

let r be [ s ] ² +t ² ] ^1/2 To obtain

Obtaining Gaussian kernels with different sizes by adjusting the size of the variable r, adjusting the size of the standard deviation sigma to adjust the image processing effect, and finally obtaining a denoised image with the best smooth effect;

step 2.3: sharpening the image by using a second derivative Laplacian operator;

for image f (x, y) is defined as:

in the x direction have

In the y direction have

The discrete laplacian of the two variables is:

obtaining a Laplacian image with enhanced feature details by processing the image by utilizing a Laplacian kernel;

laplacian is a derivative operator, so that a sharp gray transition in an image can be highlighted, a slowly changing gray area is not emphasized, and the background feature can be restored by adding the laplacian image to the original image, and meanwhile, the sharpening effect of laplacian is retained, specifically:

where f (x, y) and g (x, y) are the input image and the sharpened image, respectively.

Further, the method for image stitching after preprocessing the acquired infrared image by the image processing module specifically comprises the following steps:

step 3.1: using ROI algorithm to select the area of the preprocessed image, extracting the overlapping area of each acquired image, and using the adjacent overlapping areas as a group of binding images;

step 3.2: extracting feature points of the infrared image of the selected area by adopting an SIFT algorithm;

step 3.3: screening out correct feature matching pairs by using an RANSAC algorithm;

step 3.4: performing similarity calculation on the infrared images to be spliced by adopting a self-adaptive similarity calculation method based on the infrared images after the characteristic matching so as to determine the splicing sequence of the infrared images;

step 3.5: and fusing the images by adopting a weighted image fusion algorithm according to the effective characteristic matching pairs to realize the splicing of the infrared panoramic images.

Further, the step 3.2 specifically includes:

the first stage of the SIFT algorithm is to search stable characteristics by using a scale space function to realize the search of the position with unchanged scale change in an image, the image is represented as a parameter cluster of the smoothed image by the scale space, the purpose is to simulate the detail loss when the scale of the image is reduced, the smooth parameter is controlled to be called as a scale parameter, a Gaussian kernel is used for realizing the smoothing in the SIFT, and the scale parameter is a standard deviation;

the scale space L (x, y, σ) of a grayscale image f (x, y) is the convolution of f (x, y) with a variable-scale gaussian kernel G (x, y, σ):

L(x，y，σ)＝G(x，y，σ)★f(x，y)

where the scale is controlled by a parameter σ, G (x, y, σ) is of the form:

the input image f (x, y) is sequentially subjected to standard deviation sigma, k ² σ，k ³ σ，...，k ^M Performing Gaussian kernel convolution of sigma to generate a series of Gaussian filtering images divided by a constant factor k;

SIFT subdivides a scale space into octaves, each octave corresponds to doubling of sigma, a first image in a second octave is obtained by firstly sampling an original image downwards, namely sampling every other row and column, and then smoothing the first image by using a kernel, wherein the standard deviation of the kernel is 2 times of that in the first octave, and in subsequent processing of each octave, the first image in a new octave is formed in the following mode;

down-sampling the original image for enough times to make the image size be half of the previous octave;

smoothing the down-sampled image with a new standard deviation that is 2 times the standard deviation of the previous octave;

searching the position of the initial key point in the scale space: firstly, detecting an extreme value of the Gaussian difference of two adjacent scale space images in an octave, and then convolving an input image corresponding to the octave, wherein the expression is as follows:

D(x，y，σ)＝[G(x，y，kσ)-G(x，y，σ)]★f(x，y)＝L(x，y，kσ)-L(x，y，σ)

at each position in the D (x, y, σ) image, comparing the position pixel value with its 8 neighboring pixel values in the current image and its 9 neighboring pixel values in the upper and lower images, and if the value of the position is the maximum or minimum value in the range, selecting the position as the extreme point;

interpolation operation is carried out on the value of D (x, y, sigma) through Taylor series expansion, and the precision of the position of the key point is improved; deleting key points with low contrast and poor positioning; calculating the magnitude and direction of each keypoint using the formula obtained using the histogram-based steps associated with these formulas;

M(x，y)＝[(L(x+1，y)-L(x-1，y)) ² +(L(x，y+1)-L(x，y-1)) ² ] ^1/2

θ(x，y)＝arctan[(L(x，y+1)-L(x，y-1))/(L(x+1，y)-L(x-1，y))]

a descriptor is computed around a local region of each distinct keypoint, while the changes to scale, direction, illumination and viewpoint of the image are as invariant as possible, and used to identify matches between local regions in different images.

Further, the step 3.4 specifically includes:

extracting feature points of the two images by using an SIFT algorithm, respectively recording the feature points of the overlapped area of the two images as m and n, then performing feature matching to obtain matched feature point pairs, recording the matched feature point pairs as k, and substituting the parameters m, n and k into a similarity formula:

the greater the value of the similarity S, the closer the two images are;

supposing that X infrared images to be spliced are provided, firstly selecting an image A from the images to be spliced, calculating the similarity S of the image and all other infrared images by taking the image A as a reference, sequencing the similarities, selecting an image B with the highest similarity S from the images, and splicing the two images; and then, taking the image B as a reference, calculating the similarity between the image B and the rest images, selecting the image C with the highest similarity for splicing, and so on until all the images determine the splicing sequence.

Further, the method for the image processing module to perform target identification on the spliced image specifically comprises the following steps:

firstly, converting an infrared panoramic image of a single channel into an RGB color image of three channels, wherein the length and the width of an input picture which can be processed by a Yolov3 model are the same, and if the picture is directly adjusted to a required size, the problem of distortion is easy to occur, so the picture needs to be filled before the size is adjusted; for pictures with a length greater than a width, if the pictures are to be adjusted to be pictures with the same length and width without distortion, gray bars need to be added in the width direction; similarly, for the picture with the width larger than the length, the gray strips are required to be added in the length direction; after filling, the length and the width of the picture are scaled in the same proportion and adjusted to the required size;

inputting a Yolov3 model to identify and predict the target after the image data is preprocessed; the Yolov3 model comprises three parts, namely a trunk feature extraction network Darknet53, an enhanced feature extraction network FPN and a classifier Yolo Head; the trunk feature extraction network Darknet53 is used for extracting features of the input infrared panoramic image; the reinforced feature extraction network FPN performs feature fusion on the three effective feature layers, and further extracts features through multiple convolution processing and up-sampling operation and transmits the features to the next part; the classifier Yolo Head classifies and predicts the images according to the extracted image features, each feature layer of the Yolo Head divides the images into grids corresponding to the length and the width of the images, then three prior frames are established by taking each grid point as a center, and objects in the frames are identified and detected; after obtaining the prior frame and the detection result, decoding the prior frame to combine the prior frame with the original image; after the prior frame selection and decoding operation is carried out on the infrared panoramic image, a final prediction result and a target position are determined according to score sorting and non-maximum suppression, a prediction frame with a score meeting the requirement is found out according to the size of a threshold value, and the prediction frame with a confidence coefficient smaller than the threshold value is removed, so that the number of frames can be greatly reduced, and the next screening is facilitated; and then, further screening by combining the predicted category and the overlap ratio of the predicted frames, and selecting the predicted frame with the highest confidence coefficient as a final result for a plurality of frames with the same category and the overlap ratio reaching a certain value.

The invention has the beneficial effects that:

the invention provides an infrared panoramic radar system for safety monitoring, which comprises an image acquisition module, an image processing module and an upper computer display and early warning module, wherein the image acquisition module is used for acquiring images; the image acquisition module acquires images in a mode of combining a rotating holder and an infrared detector, so that the size of each image is the same, the bottom ends of the images are positioned on the same horizontal plane, and the subsequent splicing complexity is reduced; the image processing module synthesizes the plurality of infrared images with narrow view field and high spatial resolution into an infrared panoramic image with wide view field and high spatial resolution, and integrates a plurality of image information into one image, thereby being convenient for system monitoring. The invention adopts an infrared thermal imaging mode, uses a high-sensitivity infrared detector to be matched with a high-precision servo motor, and simultaneously combines panoramic image splicing and a target identification technology based on deep learning to realize real-time monitoring and effective tracking identification of infrared targets in the peripheral range.

Drawings

Fig. 1 is a schematic structural diagram of an infrared panoramic radar system for security surveillance according to an embodiment of the present invention.

Fig. 2 is a schematic diagram of an apparatus of an image acquisition module of an infrared panoramic radar system in an embodiment of the present invention.

Fig. 3 is a schematic flowchart of infrared image stitching in the image processing module in the embodiment of the present invention.

Fig. 4 is a schematic diagram of a flow of preprocessing an infrared raw image according to an embodiment of the present invention.

Fig. 5 is a diagram of various common laplacian kernels in image sharpening according to an embodiment of the present invention.

FIG. 6 is a diagram illustrating the extraction of the overlapping area of adjacent images according to an embodiment of the present invention.

Fig. 7 is a feature point extraction flow chart based on the SIFT algorithm in the embodiment of the present invention.

FIG. 8 is a Gaussian pyramid representation of a scale space image according to an embodiment of the invention.

Fig. 9 is a schematic diagram of a difference gaussian pyramid of a scale space image according to an embodiment of the present invention.

Fig. 10 is a schematic diagram of an infrared panoramic image filling operation in the embodiment of the present invention.

FIG. 11 is a schematic diagram of a gradient descent process of the Yolov3 model in the embodiment of the present invention.

Fig. 12 is a schematic structural diagram of a Yolov3 network model in the embodiment of the present invention.

FIG. 13 is a diagram of the structure of the residual convolution in Darknet53 according to an embodiment of the present invention.

Fig. 14 is a schematic diagram of an a priori block decoding process in the embodiment of the present invention.

Detailed Description

The invention is further described below with reference to the accompanying drawings.

The invention provides an infrared panoramic radar system for safety monitoring, which adopts an infrared thermal imaging mode, uses a high-sensitivity infrared detector to cooperate with a high-precision servo motor, and simultaneously combines panoramic image splicing and a target identification technology based on deep learning to realize real-time monitoring and effective tracking identification of infrared targets in a peripheral range.

An infrared panoramic radar system for safety monitoring mainly comprises an image acquisition module, an image processing module and an upper computer display and early warning module;

the function of image acquisition module is control infrared detector, realizes its shooting to the surrounding environment, gathers external environment image, and concrete step is:

the infrared detector is arranged on the rotating holder, and the rotating holder is used as a base of the whole radar system, so that the infrared detector can shoot a surrounding 360-degree environment, and the bottommost end of each picture shot by the detector is in the same horizontal line;

the horizontal field of view of a single image of the infrared detector is 120 degrees, and the single rotation angle of the lens is 90 degrees, so that a 15-degree overlapping area is ensured between two adjacent images, and the feature matching between the subsequent adjacent images is facilitated;

further, the collected infrared image is uploaded to the image processing module in a mode of combining optical fiber transmission and wireless transmission, and the image processing module performs related processing operation on the acquired image, and the specific process is as follows:

preprocessing the acquired infrared image: according to the image receiving sequence, carrying out image smoothing on the image to remove the noise mixed in the image, adopting a 3 multiplied by 3 or 5 multiplied by 5 Gaussian convolution kernel to remove the image noise, and selecting according to the specific noise condition;

further, enhancing edge features in the image by adopting a Laplace differential operator to obtain a Laplace sharpened image;

further, the original image and the laplacian image are superposed, so that the laplacian processing effect is enhanced, and the original background information can be retained;

further, after the image preprocessing is finished, the image processing module performs image splicing on the infrared image after the preprocessing is finished, and the specific steps are as follows:

selecting an overlapping area of each image by using an ROI algorithm;

further, a Scale-invariant feature transform (SIFT) algorithm is adopted to extract feature points and match features in the selected infrared image overlapping area, and a RANSAC algorithm is adopted to screen the subsequent feature matching pairs, wherein the method comprises the following steps:

performing rough extraction on the infrared image features of the ROI selected region to obtain feature points of the image, and performing feature matching on the same feature points in different images to form a primary feature matching pair;

screening the primary feature matching pairs by adopting an RANSAC algorithm, eliminating wrong feature matching pairs and reserving effective feature matching pairs;

further, an image fusion algorithm is utilized, according to effective feature matching pairs reserved after the RANSAC algorithm is screened, similarity calculation is carried out on infrared images to be spliced by adopting a self-adaptive similarity calculation method, the splicing sequence of the infrared images is determined, and the images are fused by adopting a weighted image fusion algorithm according to the preset splicing sequence, so that the infrared panoramic images are spliced;

furthermore, a histogram equalization method is adopted, the contrast of the image is increased, the spliced effect is improved, and finally the infrared panoramic image is formed;

further, the infrared panoramic image which is spliced well is uploaded to a target recognition module, an infrared target in the infrared panoramic image is recognized by using a target recognition algorithm, and the specific steps comprise:

converting the single-channel gray-scale panoramic image into a three-channel RGB format which can be processed by the deep learning model;

adjusting the width-height ratio of the image by adopting a padding mode, and then scaling the image into a specified size;

further, carrying out normalization processing on the color panoramic image with the adjusted size;

further, a Yolov3 model is used for carrying out target framing and identification on the panoramic image, and decoding operation is carried out on the prediction frame so as to enable the prediction frame to be combined with the original image;

screening a prediction box according to the score sorting and the non-maximum suppression to finally determine the type, confidence and position of the target;

further, the final image processing result is transmitted to an upper computer display and early warning module in real time, and subsequent operations are performed, wherein the specific process comprises the following steps:

after the image processing module processes the 360-degree infrared panoramic image once, the image after target recognition is transmitted to a rear upper computer display and early warning module in real time;

further, the upper computer display module displays the received panoramic image subjected to the target identification processing in real time, particularly, the latest received image covers the previously received image, and the images are stored in the storage device according to the receiving sequence;

further, according to the result of target identification, the early warning module controls the sending of an alarm to remind security personnel of the invasion of the suspicious target;

furthermore, security personnel can accurately position the target position through the image, the determination of the invading target position is completed, and the overall safety of the area is ensured.

Example 1:

the embodiment of the invention provides an infrared panoramic radar system for safety monitoring, which utilizes the principle of infrared thermal imaging and is matched with panoramic image splicing and target identification technologies to realize the detection of infrared targets in a detection range, thereby being a supplement to the current security system. Specifically, in the image acquisition module, the infrared information around the camera is continuously acquired by rotating the holder and matching the infrared detector, so that the dispersed infrared image is obtained. The obtained infrared images are transmitted to an image processing module according to a shooting sequence, image splicing is carried out according to an adjacent sequence after preprocessing, target identification is carried out on the spliced images, the identified images are transmitted to an upper computer display and early warning module, and a corresponding scheme is made according to an image identification result.

Fig. 1 is a schematic structural diagram of an infrared panoramic radar system for security monitoring according to an embodiment of the present invention. As shown in fig. 1, the whole system mainly comprises an image acquisition module, an image processing module, an upper computer display and early warning module and a power module. ZYNQ is selected as a core processor of the system, the ZYNQ sends a configuration instruction to the infrared detector through a serial port, the image acquisition module is driven to acquire images, the acquired infrared images are transmitted to the image processing module in a mode of mainly using wired transmission and assisting in wireless transmission, and image splicing and target identification are carried out in the image processing module. Through the transmission mode combining wired and wireless, the continuity of image transmission can be guaranteed to the maximum extent, and the influence on the normal use of the system after the wired line is damaged is prevented. And then, the processing result is sent to an upper computer display and early warning module for display, and the early warning system carries out alarm judgment according to the monitoring result to remind security personnel of the invasion of outsiders. The power module provides sufficient electric quantity for the whole system, and normal operation of the system is guaranteed.

Fig. 2 is a schematic device diagram of an image acquisition module of an infrared panoramic radar system according to an embodiment of the present invention. As shown in fig. 2, an infrared detector 201 is mounted on a rotating pan/tilt head 202 and rotates in a designated direction 206. When the imager rotates by a fixed angle, the current field of view is photographed, and finally a dispersed image 203, a dispersed image 204 and the like are formed. The size of the pixels of each image generated by the same infrared detector is the same, so that the consistency of subsequent operation is ensured, and in addition, the bottommost end of each image is on the same horizontal plane in a rotary shooting mode, so that the complexity of the subsequent operation is reduced.

All the images generated by the infrared detector are arranged in sequence, a certain overlapping area 205 exists between every two adjacent images, and then the images are spliced by operating the overlapping area 205. Compared with the

images

203 and 204, the images spliced by the overlapping area 205 comprise more scenes or larger scenes, the shooting visual angle is expanded by splicing a plurality of images to obtain a panoramic image, the information content in the images is improved, and the monitoring of the whole area is facilitated.

Fig. 3 is a schematic flow chart of infrared image stitching in the image processing module provided in the embodiment of the present invention, which specifically includes the following steps:

step 301: receiving an initial raw infrared image acquired by an image acquisition device.

Step 302: and performing image preprocessing operation on the original infrared image received in the step 301.

In addition, the infrared detector is imaged by temperature difference, the contrast of the image is low, and the detail expression capability is poor, so that the preprocessing of the infrared image is mainly divided into two parts, namely image denoising and image enhancement. The specific operation steps are shown in fig. 4:

step 401: the obtained original infrared image is read.

Step 402: and due to the consideration of quality and efficiency, the read image is subjected to smoothing filtering by adopting a low-pass Gaussian filter kernel. Gaussian core

Is the only separable circularly symmetric kernel, which not only can be compared with the box filter with the computational advantage, but also has many other useful properties suitable for image processing, and is convenient for subsequent processing.

Let r be [ s ] ² +t ² ] ^1/2 Can obtain

And obtaining Gaussian kernels with different sizes by adjusting the size of the variable r, and adjusting the image processing effect by adjusting the size of the standard deviation sigma to finally obtain a denoised image with the best smooth effect.

Step 403: sharpening the image with the second derivative laplacian, defined for image f (x, y) as:

in the x direction have

In the y direction have

The discrete laplacian of the two variables is:

this formula can be implemented by performing a convolution operation with the 501 kernel in fig. 5, and another laplacian kernel is shown in fig. 5.

Through processing of the image by utilizing the Laplace kernel, the Laplace image with enhanced feature details can be obtained, and preparation is made for subsequent processing.

Step 404: laplacian is a derivative operator and therefore highlights sharp gray transitions in the image and de-emphasizes slowly changing gray regions. This tends to produce images with gray-scale edge lines and other discontinuities that are superimposed on a dark featureless background. The background characteristics can be restored by adding the laplacian image and the original image, and meanwhile, the sharpening effect of the laplacian is kept.

The basic method of sharpening an image using laplacian is

Where f (x, y) and g (x, y) are the input image and the sharpened image, respectively. Note that if the laplace core in fig. 5-501 or 502 is used, c is-1; if the laplace kernel of fig. 5-503 or 504 is used, then c is 1.

Step 405: and outputting the preprocessed image to wait for subsequent processing.

Step 303: and performing region selection on the preprocessed image by using an ROI algorithm. The overlapping regions (the left and right edge regions of the image) of the acquired images are extracted, as shown in fig. 6, the adjacent overlapping regions are used as a group of binding images, and then each group of binding images is processed.

By extracting the image in different areas, the pixel size of the image to be processed can be greatly reduced, the time required by subsequent image processing is further reduced, and the efficiency is improved.

Step 304: after the selected area is extracted, extracting feature points of the infrared image of the selected area by adopting an SIFT algorithm, wherein FIG. 7 is a flow chart of the SIFT algorithm, and the specific steps are as follows:

the first stage of the SIFT algorithm is to search stable features by using a scale space function to search for the position with unchanged scale change in an image, the image is represented as a parameter cluster of the smoothed image by the scale space, the purpose is to simulate the detail loss when the scale of the image is reduced, the smooth parameter is controlled to be called as a scale parameter, a Gaussian kernel is used for realizing the smoothing in the SIFT, and the scale parameter is the standard deviation.

The scale space L (x, y, σ) of the grayscale image f (x, y) is the convolution of f with a variable-scale gaussian kernel G (x, y, σ):

L(x，y，σ)＝G(x，y，σ)★f(x，y)

where the mesoscale is controlled by a parameter σ, G is of the form:

the input image f (x, y) is sequentially subjected to standard deviation sigma, k ² σ，k ³ The gaussian kernel convolution of σ, …, produces a stack of gaussian filtered images divided by a constant factor k.

SIFT subdivides the scale space into octaves, and fig. 8 is a schematic diagram of a gaussian pyramid of the scale space image, each octave corresponding to a doubling of σ. The first image in the second octave is obtained by first down-sampling the original image, i.e. every other row and column, and then smoothing it with a kernel whose standard deviation is 2 times the standard deviation used in the first octave. In the subsequent octave processing, the first image of the new octave is formed as follows:

smoothing the down-sampled image with a new standard deviation 2 times the standard deviation of the previous octave;

searching the position of the initial key point in the scale space: firstly, detecting an extreme value of the Gaussian difference of two spatial images with adjacent scales in an octave, and then performing convolution on an input image corresponding to the octave, wherein the expression is as follows:

fig. 9 is a schematic diagram of a difference gaussian pyramid of a scale space image.

At each location in the D (x, y, σ) image, the location pixel value is compared to its 8 neighboring pixel values in the current image and its 9 neighboring pixel values in the upper and lower images, and if the value of the location is the maximum or minimum value in the range, the location is selected as the extreme point.

Interpolation operation is carried out on the value of D (x, y, sigma) through Taylor series expansion, and the precision of the position of the key point is improved; the key points of low contrast and poor positioning are deleted.

In use mode

M(x，y)＝[(L(x+1，y)-L(x-1，y)) ² +(L(x，y+1)-L(x，y-1)) ² ] ^1/2

And

θ(x，y)＝arctan[(L(x，y+1)-L(x，y-1))/(L(x+1，y)-L(x-1，y))]

the magnitude and direction of each keypoint is calculated using histogram-based steps associated with these equations.

Step 305: and screening out correct feature matching pairs by using an RANSAC algorithm. SIFT is a descriptor with strong robustness, which can generate less error matching compared with other matching algorithms, but still has wrong corresponding points, so that RANSAC algorithm is needed to eliminate the error matching points from the feature descriptors generated by SIFT algorithm.

And extracting 5-10 pairs of matching points from the obtained pairs of matching points, calculating a transformation matrix, and calculating mapping errors for all the matching points. And finally, recalculating the homography matrix H according to the maximum statistic point number set.

After the homography matrix between the images is estimated by using the RANSAC algorithm, all the images are integrated on a common image plane.

Step 306: and performing similarity calculation on the infrared images to be spliced by adopting a self-adaptive similarity calculation method based on the infrared images after the characteristic matching so as to determine the splicing sequence of the infrared images.

Each infrared image is output sequentially through an infrared imager, but in the processing process, the phenomenon of picture sequence errors possibly occurs due to the reason that the processing speed of each image in the previous period is different and the like, so the splicing sequence of the infrared images is determined by performing similarity calculation on the infrared images to be spliced by adopting a self-adaptive similarity calculation method, and the method comprises the following specific steps of:

extracting feature points of two images by using an SIFT algorithm, respectively recording the feature points of an overlapped area of the two images as m (m is 1, 2, 3.) and n (n is 1, 2, 3.)), then performing feature matching to obtain matched feature point pairs, recording the matched feature point pairs as k (k is 1, 2, 3.)), and substituting parameters m, n and k into a similarity formula:

the greater the value of the similarity S, the closer the two images are.

At present, suppose that there are X infrared images to be stitched, one image a is selected from the images to be stitched, the similarity S between the image a and all other infrared images is calculated by taking the image a as a reference, the similarity ranking is performed, an image B with the highest similarity S is selected from the images, and the two images are stitched. And then, taking the image B as a reference, calculating the similarity between the image B and the rest images, selecting the image C with the highest similarity for splicing, and so on until all the images determine the splicing sequence.

Step 307: and fusing the images by adopting a weighted image fusion algorithm according to the effective characteristic matching pair to realize the splicing of the infrared panoramic image.

After the images are spliced, the infrared panoramic image is sent to a target recognition module, and a target recognition algorithm is called to recognize the infrared target in the images. The Yolov3 model often achieves good effect on color image (three channels) training and prediction, so that the infrared panoramic image of a single channel is firstly converted into an RGB color image of three channels. Since the length and width of the input picture that can be processed by the Yolov3 model are the same, the picture is easy to be distorted if the picture is directly adjusted to the required size, and therefore the picture needs to be filled before the size is adjusted.

Fig. 10 is a schematic diagram of an infrared panoramic image filling operation performed before predicting an infrared panoramic image according to an embodiment of the present invention. For a picture with a length greater than a width, if the picture is adjusted to be a picture with the same length and width without distortion, gray bars need to be added in the width direction; similarly, for a picture with a width greater than the length, a gray bar needs to be added in the length direction. The infrared panoramic image used by the invention belongs to the former, so that gray strips need to be added in the width direction, and then the length and the width of the image are scaled in the same proportion to be adjusted to the required size.

Fig. 11 is a schematic diagram of a Yolov3 model gradient descent process provided by the embodiment of the present invention. It can be seen that if the gradient descent algorithm is performed on a cost function such as the left graph, a very small learning rate must be used, requiring multiple iterations until a minimum is found. However, if the function is a more circular spherical contour like the right image, the gradient descent algorithm can more directly find the minimum value no matter where the function starts, so that a larger step size can be used in the gradient descent algorithm, and the network convergence speed is greatly increased. In view of this, before the picture is input into the neural network model, normalization processing needs to be performed on the image data, so that the speed of network training and prediction is increased.

After image data is preprocessed, the image data can be input into a Yolov3 model for target recognition and prediction. Fig. 12 is a schematic structural diagram of a Yolov3 network model provided in an embodiment of the present invention. The whole Yolov3 model comprises three parts, namely a trunk feature extraction network Darknet53, an enhanced feature extraction network FPN and a classifier Yolo Head. The method for identifying and detecting the target of the input infrared panoramic image specifically comprises the following steps:

the trunk feature extraction network Darknet53 performs feature extraction on the input infrared panoramic image. This section uses the Residual network Residual, the Residual convolution structure is shown in fig. 13. The stem part is a convolution of 1 × 1 and a convolution of 3 × 3; the residual edges are directly added with the output of the trunk part without any processing. The whole trunk feature extraction network is formed by residual convolution, and the times of residual structure repetition are represented by the multiplied numbers of Resblock _ body in FIG. 12. The extracted features of the part are called feature layers, and the last three feature layers are selected to construct the next part of network;

and the reinforced feature extraction network FPN performs feature fusion on the three effective feature layers. Further extracting features through multiple convolution processing and up-sampling operation and transmitting the features to the next part;

the classifier Yolo Head performs classification prediction on the image according to the extracted image features. The shape of the output layer in fig. 12 indicates that the resized infrared panoramic image is divided into meshes of 13 × 13, 26 × 26, and 52 × 52 specifications, 255 is the product of the number of channels and the a priori frame data. The parameters can be changed according to requirements in actual application.

Each feature layer of the Yolo Head divides the picture into grids corresponding to the length and the width of the picture, then three prior frames are established by taking each grid point as the center, and objects in the frames are identified and detected. And after the prior frame and the detection result are obtained, decoding the prior frame to combine the prior frame with the original image. Fig. 14 is a schematic diagram of an a priori block decoding process provided in an embodiment of the present invention. In practical application, after the prior frame decoding operation, the frame selection area can better conform to a real picture.

After the prior framing and decoding operations are performed on the infrared panoramic image, a final prediction result and a target position are determined according to score sorting and non-maximum suppression, and the method specifically comprises the following steps:

finding out the prediction frames with scores meeting the requirements according to the size of the threshold value, and removing the prediction frames with confidence degrees smaller than the threshold value, so that the number of the frames can be greatly reduced, and the next screening is facilitated;

and then, further screening by combining the predicted category and the overlap ratio of the predicted frames, and selecting the predicted frame with the highest confidence coefficient as a final result for a plurality of frames with the same category and the overlap ratio reaching a certain value. Therefore, the target identification and detection of the infrared panoramic image are completed.

And after the target identification is finished, sending the obtained result to an upper computer display and early warning module by an image processing module. The final image processing result is displayed on the display module, and particularly, the latest received image overwrites the previously received image, and the images are stored in the storage device according to the receiving sequence. Meanwhile, the early warning module can detect abnormal infrared targets in the image, when the abnormal targets are detected, the early warning module can trigger the alarm switch, the alarm can give an alarm to remind security personnel of invasion of outsiders, the security personnel can accurately position the target position through monitoring images, and prevention is timely made to protect the safety of the reef.

Compared with the prior art, the invention has the beneficial effects that:

1. the infrared panoramic radar system for safety monitoring synthesizes a plurality of infrared images with narrow view field and high spatial resolution into a wide view field and high spatial resolution infrared panoramic image, and integrates a plurality of image information into one image, thereby being convenient for system monitoring.

2. According to the infrared panoramic radar system for safety monitoring, the images are acquired in a mode of combining the rotating holder and the infrared detector, so that the size of each image is the same, the bottom ends of the images are positioned on the same horizontal plane, and the subsequent splicing complexity is reduced.

3. The infrared panoramic radar system for safety monitoring adopts a combined image transmission mode of taking wired transmission as main transmission and taking wireless transmission as standby transmission, thereby not only ensuring the real-time property and the accuracy of image transmission, but also ensuring that the acquired images can be transmitted in time after the wired transmission line is damaged.

4. According to the infrared panoramic radar system for safety monitoring, the edge characteristics in the image are enhanced by adopting the Laplace differential operator, the Laplace sharpening processing effect is protected in a mode of overlapping the original image and the Laplace sharpened image, meanwhile, background information is reserved, and the overall detail characteristics of the image are enhanced.

5. According to the infrared panoramic radar system for safety monitoring, the images are divided into regions, the ROI algorithm is adopted to automatically select the overlapping region between the adjacent images, irrelevant parts in the images are removed, the data volume of image feature point extraction is reduced, the calculation complexity is reduced, and the real-time performance of the system is improved.

6. The infrared panoramic radar system for safety monitoring combines the SIFT algorithm and the RANSAC algorithm, and improves the accuracy of feature matching.

7. According to the infrared panoramic radar system for safety monitoring, the similarity calculation is carried out on the infrared images to be spliced by adopting a self-adaptive similarity calculation method based on the infrared images subjected to feature matching screening, the splicing sequence of the infrared images is determined, and the panoramic images are spliced according to the splicing sequence, so that the splicing efficiency and accuracy are greatly improved.

8. The infrared panoramic radar system for safety monitoring adopts a histogram equalization method, increases the contrast of images, eliminates the phenomenon of uneven image brightness caused by external factors such as illumination and the like, and improves the effect after splicing.

9. The infrared panoramic radar system for safety monitoring adopts a padding mode to perform zooming after filling, thereby ensuring no distortion of image proportion and ensuring the accuracy of target identification and detection.

10. The infrared panoramic radar system for safety monitoring adopts the Yolov3 model to identify and detect the target of the infrared panoramic image, and can identify, classify and count the target in the image efficiently, accurately and stably in real time.

11. The infrared panoramic radar system for safety monitoring adopts score screening and non-maximum suppression to finally determine the information and the position of the prediction frame, and improves the accuracy of target identification and detection in the infrared panoramic image.

12. According to the infrared panoramic radar system for safety monitoring, the number of suspicious infrared targets in the final image is judged to make a corresponding alarm level, so that security personnel are reminded to make preparations in time.

The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An infrared panoramic radar system for security surveillance, characterized by: the system comprises an image acquisition module, an image processing module and an upper computer display and early warning module; the image acquisition module comprises an infrared detector (201), the infrared detector (201) is installed on a rotating tripod head (202), the infrared detector (201) is used for shooting the surrounding 360-degree environment through the rotating tripod head (202), and meanwhile, the bottommost end of each picture shot by the infrared detector (201) is ensured to be positioned on the same horizontal line; the image acquisition module continuously acquires surrounding infrared information through an infrared detector (201) to obtain scattered infrared images, and transmits the infrared images to the image processing module according to a shooting sequence; the image processing module is used for preprocessing the acquired infrared image, splicing the image, carrying out target identification on the spliced image and transmitting the image subjected to target identification to the upper computer display and early warning module in real time; and the upper computer display and early warning module displays the received image subjected to target identification processing in real time, judges whether to give an alarm or not according to the target identification result, and positions the position of the invading target.

2. An infrared panoramic radar system for security surveillance as claimed in claim 1, characterized in that: the method for preprocessing the acquired infrared image by the image processing module specifically comprises the following steps:

step 2.1: reading an original infrared image;

let r be [ s ] ² +t ² ] ^1/2 To obtain

Obtaining Gaussian kernels with different sizes by adjusting the size of the variable r, adjusting the size of the standard deviation sigma to adjust the image processing effect, and finally obtaining a denoised image with the best smoothing effect;

step 2.3: sharpening the image by using a second derivative Laplacian operator;

for image f (x, y) is defined as:

in the x direction have

In the y direction have

The discrete laplacian of the two variables is:

3. An infrared panoramic radar system for security surveillance, as claimed in claim 1, wherein: the method for image splicing after preprocessing the acquired infrared image by the image processing module specifically comprises the following steps:

step 3.1: using an ROI algorithm to perform region selection on the preprocessed images, extracting overlapping regions of the acquired images, and using the adjacent overlapping regions as a group of binding images;

step 3.3: screening out correct characteristic matching pairs by using an RANSAC algorithm;

4. An infrared panoramic radar system for security surveillance as claimed in claim 3, characterized in that: the step 3.2 is specifically as follows:

L(x,y,σ)＝G(x,y,σ)★f(x,y)

where the scale is controlled by a parameter σ, G (x, y, σ) is of the form:

the input image f (x, y) is sequentially subjected to standard deviation sigma, k ² σ，k ³ σ,…,k ^M Performing Gaussian kernel convolution of sigma to generate a series of Gaussian filtering images divided by a constant factor k;

down-sampling the original image for enough times to make the size of the image half of the previous octave;

D(x,y,σ)＝[G(x,y,kσ)-G(x,y,σ)]★f(x,y)＝L(x,y,kσ)-L(x,y,σ)

M(x,y)＝[(L(x+1,y)-L(x-1,y)) ² +(L(x,y+1)-L(x,y-1)) ² ] ^1/2

θ(x,y)＝arctan[(L(x,y+1)-L(x,y-1))/(L(x+1,y)-L(x-1,y))]

5. An infrared panoramic radar system for security surveillance as claimed in claim 3, wherein: the step 3.4 is specifically as follows:

extracting feature points of the two images by using an SIFT algorithm, respectively recording the feature points of the overlapping area of the two images as m and n, then performing feature matching to obtain matched feature point pairs, recording the matched feature point pairs as k, and substituting the parameters m, n and k into a similarity formula:

the greater the value of the similarity S, the closer the two images are;

6. An infrared panoramic radar system for security surveillance, as claimed in claim 1, wherein: the method for the image processing module to perform target identification on the spliced image specifically comprises the following steps:

firstly, converting an infrared panoramic image of a single channel into an RGB color image of three channels, wherein the length and the width of an input picture which can be processed by a Yolov3 model are the same, and if the picture is directly adjusted to a required size, the problem of distortion is easy to occur, so the picture needs to be filled before the size is adjusted; for pictures with a length greater than a width, if the pictures are to be adjusted to be pictures with the same length and width without distortion, gray bars need to be added in the width direction; similarly, for the picture with the width larger than the length, gray strips need to be added in the length direction; after filling, the length and the width of the picture are scaled in the same proportion and adjusted to the required size;