CN114693547A - Radio frequency image enhancement method and radio frequency image identification method based on image super-resolution - Google Patents
Radio frequency image enhancement method and radio frequency image identification method based on image super-resolution Download PDFInfo
- Publication number
- CN114693547A CN114693547A CN202210204332.6A CN202210204332A CN114693547A CN 114693547 A CN114693547 A CN 114693547A CN 202210204332 A CN202210204332 A CN 202210204332A CN 114693547 A CN114693547 A CN 114693547A
- Authority
- CN
- China
- Prior art keywords
- resolution
- image
- radio frequency
- frequency image
- low
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 69
- 238000013507 mapping Methods 0.000 claims abstract description 28
- 238000012549 training Methods 0.000 claims description 34
- 238000005070 sampling Methods 0.000 claims description 33
- 230000002708 enhancing effect Effects 0.000 claims description 5
- 239000000126 substance Substances 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 claims description 2
- 238000005259 measurement Methods 0.000 claims description 2
- 238000013528 artificial neural network Methods 0.000 abstract description 5
- 238000013527 convolutional neural network Methods 0.000 description 17
- 230000003321 amplification Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 238000003199 nucleic acid amplification method Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 238000012545 processing Methods 0.000 description 5
- 238000012360 testing method Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000004913 activation Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 230000008447 perception Effects 0.000 description 2
- 238000000926 separation method Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- QVFWZNCVPCJQOP-UHFFFAOYSA-N chloralodol Chemical compound CC(O)(C)CC(C)OC(O)C(Cl)(Cl)Cl QVFWZNCVPCJQOP-UHFFFAOYSA-N 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10032—Satellite or aerial image; Remote sensing
- G06T2207/10044—Radar image
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Image Processing (AREA)
Abstract
The invention provides a radio frequency image enhancement method and a radio frequency image identification method based on image super-resolution. The radio frequency image enhancement method comprises the following steps: acquiring a high-resolution radio-frequency image and a low-resolution radio-frequency image corresponding to the high-resolution radio-frequency image; learning the mapping relation between the low-resolution radio frequency image and the high-resolution radio frequency image by using an image super-resolution network; and constructing an enhanced radio frequency image data set based on the low-resolution radio frequency image based on the mapping relation between the low-resolution radio frequency image and the high-resolution radio frequency image. The method can reconstruct the radio frequency image superior to an interpolation method by utilizing the learning capability of the neural network, thereby improving the precision of the radio frequency image identification task.
Description
Technical Field
The invention relates to the technical field of radio frequency image perception, in particular to a radio frequency image enhancement method and a radio frequency image identification method based on image super-resolution.
Background
In the field of wireless sensing, various wireless devices (radar, WiFi, LoRa and the like) are used for detecting the influence of a target on a wireless signal, and the behavior and action characteristics of the target can be inferred by analyzing the influenced wireless signal. Small wireless sensing devices (radar and WiFi) cannot transmit and store data with large size, radio frequency images acquired by the small wireless sensing devices are possibly small in size and low in information quantity, and the problem of low identification precision can be caused when the small wireless sensing devices are directly used for identification tasks. Because deep learning shows great advantages in image recognition tasks, more and more wireless sensing tasks convert influenced wireless signals into images, namely radio frequency images, and then the radio frequency images are recognized by using a neural network, so that the performance of the method is superior to that of the traditional method. In the field of wireless sensing, devices such as radar and WiFi are all small devices, which limits transmission and storage of large data, and a common solution is to compress a radio frequency image in a sensing task so as to facilitate transmission and storage. The compression of the radio frequency image inevitably causes the loss of effective information, thereby influencing the accuracy of subsequent image identification. In the task of image recognition based on deep learning, it is common practice to perform amplification preprocessing on an image, for example, to perform amplification processing on an image with a smaller size by using a Bicubic interpolation method. However, the Bicubic interpolation method essentially refers to the features of a few data points around the estimation point, which does not make up for the information amount in the image compression process, and is an amplification method based on the image information, and the information amount of the image is not obviously improved.
Disclosure of Invention
The invention provides a radio frequency image enhancement method and a radio frequency image identification method based on image super-resolution. The problem that the recognition accuracy of the low-resolution image or the Bicubic image is low in the recognition task is solved. Specifically, a mapping relation between a low-resolution image and a high-resolution image is learned through an image super-resolution network, the low-resolution image related to a training data set is subjected to super-resolution reconstruction through the mapping relation, and then the reconstructed radio-frequency image is subjected to classification and identification through a convolutional neural network.
The technical means adopted by the invention are as follows:
a radio frequency image enhancement method based on image super resolution comprises the following steps:
acquiring a high-resolution radio-frequency image and a low-resolution radio-frequency image corresponding to the high-resolution radio-frequency image;
learning the mapping relation between the low-resolution radio frequency image and the high-resolution radio frequency image by using an image super-resolution network;
and constructing an enhanced radio frequency image data set based on the low-resolution radio frequency image based on the mapping relation between the low-resolution radio frequency image and the high-resolution radio frequency image.
Further, acquiring a high resolution radio frequency image and a low resolution radio frequency image corresponding to the high resolution radio frequency image includes:
measuring a high-resolution radio frequency image HR through a radar, wherein the size of the high-resolution radio frequency image HR is h x h;
and performing n-time down-sampling on the basis of the high-resolution radio frequency image HR through Bicubic to obtain a low-resolution radio frequency image LR, wherein the size of the low-resolution radio frequency image LR is (h/n) × (h/n).
And acquiring a hyper-resolution image SR through n times of hyper-resolution reconstruction based on the low-resolution radio frequency image LR.
Further, acquiring a high resolution radio frequency image and a low resolution radio frequency image corresponding to the high resolution radio frequency image, further comprising:
after a high-resolution radio frequency image HR, a low-resolution radio frequency image LR and a hyper-resolution image SR are obtained, an image reconstruction measurement index PSNR is obtained according to the following calculation:
wherein the content of the first and second substances,
the MSE represents the mean square error of the difference value of corresponding pixels of two images, and the length and the width of the image are assumed to be h.
Further, learning the mapping relationship between the low-resolution radio frequency image and the high-resolution radio frequency image by using an image super-resolution network, wherein the learning comprises the following steps:
performing n-time upsampling on the low-resolution radio frequency image LR through Bicubic to obtain a sampling image LR ═ ═ c;
constructing a training data set based on the sampling image LR ≠ and the high-resolution radio frequency image HR, training an image super-resolution network model by taking the sampling image LR ≠ as a training sample and taking the high-resolution radio frequency image HR as a label, wherein the super-resolution network model is used for learning a mapping relation between the sampling image LR ℃ and the high-resolution radio frequency truth image HR.
Further, constructing a high-quality radio frequency image data set based on the low-resolution radio frequency image based on the mapping relation between the low-resolution radio frequency image and the high-resolution radio frequency image comprises the following steps:
the method comprises the steps of carrying out Bicubic interpolation on a low-resolution radio frequency image LR sample to be enhanced to obtain a sampling image LR ≠ to be enhanced, and inputting the sampling image LR ℃ to be enhanced into a trained image super-resolution network model to obtain a reconstructed enhanced image SR.
The invention also discloses a radio frequency image identification method, which comprises the following steps: and constructing an identification task based on the radio frequency image, wherein the identification task is realized based on the enhanced radio frequency image data set constructed by the radio frequency image enhancement method.
Compared with the prior art, the invention has the following advantages:
the invention learns a mapping relation between a low-resolution radio frequency image and a high-resolution radio frequency image by utilizing an image super-resolution network, so that the network has the capability of converting the low-resolution image related to a training set into the high-resolution image. By utilizing the conversion capability, all low-resolution data can be subjected to super-resolution processing through a trained network, and a super-resolution data set with better quality is constructed. The identification task based on the hyper-differential data set has higher identification precision.
Based on the reason, the invention can be widely popularized in the field of radio frequency image perception.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a flow chart of a radio frequency image enhancement method based on image super-resolution according to the present invention.
FIG. 2 is a schematic diagram of the radio frequency image enhancement method based on image super-resolution according to the present invention.
Fig. 3 is a network structure diagram of the SRCNN _ RES in the embodiment.
FIG. 4 is a diagram of a convolutional neural network structure in an embodiment.
FIG. 5 is a Bicubic and SRCNN _ RES reconstructed image in an embodiment, in which (a) is an HR image; (b) is a Bic _4 image; (c) is an SR _4 image; (d) is a Bic _8 image; (e) is an SR _8 picture.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The image super-resolution method is a method of converting a low-resolution image into a high-resolution image using depth learning. The method utilizes the deep neural network to learn the mapping relation between the low-resolution images and the high-resolution images, and can perform the super-resolution on the low-resolution images related to the training images to obtain the high-resolution images by means of the mapping relation. The radio frequency image is amplified by using an image super-resolution method instead of a Bicubic method, so that the obtained radio frequency image has more information content, and further, the performance is improved in the identification task.
The key point of the image super-resolution research based on deep learning is the construction of a neural network model, and the quality of the network model determines the quality of a super-resolution reconstructed image. Classical image super-resolution networks are: the SRCNN network learns the mapping relation from the low-resolution image to the high-resolution image by utilizing a shallow convolutional neural network and performs the super-resolution processing on the low-resolution image by utilizing the mapping relation; the VDSR network constructs a deep convolutional neural network by utilizing cross connection, so that the network receptive field is improved, and the super-resolution performance is further improved; the EDSR network reduces the training overhead by removing the batch normalization layer, and constructs a deeper network by utilizing a residual error structure; the RCAN network, introduces an attention network, enabling the network to focus on useful features in the image.
The image super-resolution field utilizes PSNR and SSIM to measure the image reconstruction quality. PSNR is the ratio of the maximum power of the signal to the signal-to-noise power, with a larger PSNR indicating a better reconstructed image quality. SSIM is an index for measuring the similarity of two images, the value of SSIM is between 0 and 1, and the larger the value is, the smaller the image distortion is.
Based on the research and development background, the invention provides a radio frequency image enhancement method based on image super-resolution, which utilizes an image super-resolution network to learn a mapping relation between low-resolution and high-resolution radio frequency images, and utilizes the mapping relation to construct a high-quality radio frequency image data set based on the low-resolution radio frequency images, thereby improving the classification precision of an identification task based on the radio frequency images. As shown in fig. 1, the method specifically comprises the following steps:
s1, acquiring a high-resolution radio frequency image and a low-resolution radio frequency image corresponding to the high-resolution radio frequency image. The method comprises the following steps: measuring a high-resolution radio frequency image HR through a radar, wherein the size of the high-resolution radio frequency image HR is h x h; and performing n-time down-sampling on the basis of the high-resolution radio frequency image HR through Bicubic to obtain a low-resolution radio frequency image LR, wherein the size of the low-resolution radio frequency image LR is (h/n) × (h/n).
Specifically, for the HR image in the image super-resolution dataset, it is assumed that the radio frequency image measured by the radar is the HR image, and the size of the HR image is: h. For an LR image in an image super-resolution data set, assuming that the LR image is obtained by downsampling a certain multiple n by Bicubic interpolation, the size of the LR image is: (h/n) × (h/n), the relationship between HR and LR being:
LR＝BLn(HR)
wherein BLnRepresenting n-fold down-sampling of the HR image using Bicubic interpolation method.
For the super-resolution image SR obtained by utilizing image super-resolution, the mapping relation of the neural network learned between LR and HR is assumed to be FnThen the relationship between SR and LR is
SR＝Fn(LR)
Wherein, FnRepresenting an n-fold super-resolution reconstruction of the LR image.
Further, still include: after a to-be-enhanced sampling image LR ≠ ≠ to be obtained by performing Bicubic interpolation on a to-be-enhanced low-resolution radio-frequency image LR sample, calculating a PSNR (peak to noise ratio) value of the to-be-enhanced sampling image LR ≠ for evaluating the quality of the to-be-enhanced sampling image LR ≠ ℃; and inputting the to-be-enhanced sampling image LR ≈ to a trained image super-resolution network model to obtain a reconstructed enhanced image SR, and calculating a PSNR value of the reconstructed enhanced image SR for evaluating the quality of the reconstructed enhanced image SR. Specifically, it is assumed that there are LR, HR, and SR images. The image reconstruction metric PSNR can be calculated by the following formula
Wherein the content of the first and second substances,
the MSE represents the mean square error of the difference value of corresponding pixels of two images, and the length and the width of the image are assumed to be h.
And S2, learning the mapping relation between the low-resolution radio frequency image and the high-resolution radio frequency image by using an image super-resolution network. The method comprises the following steps: performing n-time upsampling on the low-resolution radio frequency image LR through Bicubic to obtain a sampling image LR ═ ═ c; constructing a training data set based on the sampling image LR ≠ and the high-resolution radio frequency image HR, training an image super-resolution network model by taking the sampling image LR ≠ as a training sample and taking the high-resolution radio frequency image HR as a label, wherein the super-resolution network model is used for learning a mapping relation between the sampling image LR ℃ and the high-resolution radio frequency truth image HR.
Preferably, the image super-resolution network training phase process is as follows:
1) constructing a training data set, assuming that radar acquisition data are HR images, performing n-time down-sampling on the HR images to obtain corresponding LR images, and performing n-time up-sampling on the LR images by using a Bicubic interpolation method to obtain LR ═ images with the same size as the HR images;
2) the super-resolution network is a recursive network, the input and output of the network are images, the data set does not need to be classified, and all data as a whole train the network model to have adaptability on all relevant data. To increase the amount of training data, LR-HR image blocks are extracted using a sliding window on the paired LR ↓andhr images. Wherein the LR part in the LR-HR image block is used as training data, and the HR part is used as a label of the training data;
3) training a super-resolution network model by using the constructed LR-HR data to enable the network to learn the mapping relation between the LR radio frequency image and the HR radio frequency image;
4) training a deep network based on a back propagation learning method, so that the deep network can continuously learn the high-frequency characteristics which do not exist in the LR ≠ radio-frequency image but exist in the HR radio-frequency image.
S3, constructing an enhanced radio frequency image data set based on the low-resolution radio frequency image based on the mapping relation between the low-resolution radio frequency image and the high-resolution radio frequency image. The method comprises the following steps: the method comprises the steps of carrying out Bicubic interpolation on a low-resolution radio frequency image LR sample to be enhanced to obtain a sampling image LR ≠ to be enhanced, and inputting the sampling image LR ℃ to be enhanced into a trained image super-resolution network model to obtain a reconstructed enhanced image SR.
Preferably, the process of reconstructing the image based on the image super-resolution network is as follows:
1) and carrying out n-time amplification processing on the LR radio frequency image to be amplified by using a Bicubic interpolation method to obtain an LR ≠ image. PSNR value of LR ≠ radio-frequency image reconstructed by Bicubic can be calculated, and quality of the reconstructed radio-frequency image is evaluated;
2) and sending the LR ↓imageinto a trained super-resolution network model, and outputting a corresponding SR image by a network. PSNR value of SR reconstruction radio frequency image can be calculated, and reconstruction image quality can be evaluated.
The working principle of the method for enhancing the radio frequency image by utilizing the image super-resolution is shown in figure 2, and the method is integrally divided into an off-line training stage and an on-line reconstruction stage. And (3) supposing that the radio frequency image obtained by the radar is a true value HR image, and performing n-time down sampling on the HR image to obtain an LR image. In the off-line training stage, LR ≠ as a training sample obtained by performing n-time Bicubic upsampling on an LR image, and an HR image is used as a label to construct a data set to train a super-resolution network model, so that the network can learn the mapping relation between the LR ≠ radio frequency image and the HR radio frequency image. In the on-line estimation stage, based on a mapping network model from an LR image to an HR image obtained by neural network training, LR ≈ r obtained by interpolating an LR sample Bicubic is input into a depth network model, and the network directly outputs a reconstructed SR image.
The following further describes the aspects and effects of the present invention based on specific application examples.
The purpose of this embodiment is to enhance the 24G radar radio frequency image based on the SRCNN super-resolution network, and the system configuration is as follows:
1) and constructing a radio frequency image based on the gesture action by using 24G millimeter wave radar. There are 8 types of gestures, each type has 120 radio frequency images, and the size of each radio frequency image is 96 × 96;
2) selecting 40 pieces of radio frequency images in each class as super-resolution network training data, and selecting 80 pieces of radio frequency images as test data of a cha super-resolution network model;
3) the magnification was 4 times and 8 times, respectively. When the magnification is 4, the size of the LR image is 24 × 24; the size of the LR image was 12 × 12 at 8 x magnification;
4) respectively amplifying the LR images with the sizes of 24 x 24 and 12 x 12 by 4 times and 8 times by using a Bicubic interpolation method to obtain LR ≠ images with the sizes of 96 x 96; synchronously cutting an LR-HR image pair on the LR ≠ and the HR image pair corresponding to the LR ═ 96 by using a sliding window with the window size smaller than or equal to 96 × 96, so as to realize data enhancement;
5) the super-division network model is constructed based on the SRCNN network, and a residual cascade structure is used for increasing the network depth and improving the network receptive field. The network structure is shown in fig. 3;
6) training a super-resolution network by using a large number of LR-HR blocks, wherein an LR part is used as a super-resolution network training sample, and an HR part is used as a label;
7) performing Bicubic amplification pretreatment on LR data needing hyperfraction enhancement to obtain LR ≠ ℃,. inputting the trained network model, outputting the SR image corresponding to the LR image as the output of the network, and calculating the PSNR of the SR radio-frequency image;
8) three data sets are provided, wherein one data set is a data set group constructed by HR radio frequency images, the other data set is a data set BIC _ LR constructed by LR ≠ radio frequency images, and the other data set is a data set SR _ SRCNN _ RES constructed by SR images;
and (3) training a gesture classification convolutional neural network by using three data sets of group, BIC _ LR and SR _ SRCNN _ RES under the conditions of 4 times and 8 times of over-differentiation respectively, and detecting classification recognition accuracy. The convolutional neural network structure is shown in fig. 4.
Task: and constructing an SR _ SRCNN _ RES data set based on the SRCNN _ RES network model, and testing the identification accuracy of the convolutional neural network on the data set.
In this embodiment, a 12-layer SRCNN _ RES hyper-division network is written by Python, and a network architecture is shown in fig. 3, where SRCNN _ RES is formed by SRCNN modules cascaded by using a residual structure, and each SRCNN has 3 layers; a 5-layer convolutional neural network is written by Python, and the network architecture is shown in fig. 4.
The SRCNN _ RES hyper-division network characteristics are as follows:
1) the super-division network is formed by stacking 4 SRCNN modules, and the modules are connected by using a cross-connection structure; the number of channels of each layer of each SRCNN module is 64, 32 and 32, and the sizes of convolution kernels of each layer are 9, 5 and 5;
2) the hyper-division network training adopts a random gradient descent learning algorithm of a self-adaptive learning rate;
3) the super-resolution network adopts a full convolution structure;
4) the nonlinear activation function of the hyper-division network adopts a ReLU function,
5) the cost function of the hyper-division network adopts the mean square error L _ MSE of corresponding pixels of the reconstructed SR radio frequency image and the real HR radio frequency image, and is shown as the following formula
Wherein HR isiAnd SRiAnd the value of the ith pixel point of the HR radio frequency image and the value of the ith pixel point of the SR radio frequency image subjected to the super-resolution reconstruction are represented.
The process of the SRCNN _ RES super-division network offline training stage is as follows:
1) the training data set is 24G millimeter wave radar data, 8 types of gesture actions are performed, 40 radio frequency images are arranged in each type, and the size of each image is 96 × 96. The radar acquires the radio frequency image as real HR data. Down-sampling the HR image by 4 times and 8 times to obtain LR _4 and LR _8, which have sizes of 24 × 24 and 12 × 12, respectively. Performing up-sampling on the obtained LR _4 and LR _8 by 4 times and 8 times by using a Bicubic interpolation method to obtain LR _4 ≠ and LR _8 ℃.;
2) constructing a group of data sets BIC _ LR _4 by the obtained LR _4 ≠ (sample) and the corresponding HR (label); constructing a set of data set BIC _ LR _8 from the obtained LR _8 ↓ (sample) and the corresponding HR (label);
3) respectively training a hyper-distributed network SRCNN _ RES by using a data set BIC _ LR _4 and a data set BIC _ LR _8 to obtain network models SRCNN _ RES _4 and SRCNN _ RES _ 8;
the online estimation stage process of the SRCNN _ RES hyper-divided network is as follows:
1) the test data set is the remaining 24G radar data, with 8 types of gesture motion, 80 rf images of each type, and 96 × 96 images. The radar acquires the radio frequency image as real HR data. Down-sampling the HR image by 4 times and 8 times to obtain LR _4 and LR _8, which have sizes of 24 × 24 and 12 × 12, respectively. Performing up-sampling on the obtained LR _4 and LR _8 by 4 times and 8 times by using a Bicubic interpolation method to obtain LR _4 ≠ and LR _8 ℃.;
2) and respectively inputting LR _4 ↓andLR _8 ×) into the trained network models SRCNN _ RES _4 and SRCNN _ RES _8 to obtain the super-resolution reconstruction radio frequency data sets SR _4 and SR _ 8.
The characteristics of the gesture classification convolutional neural network are as follows:
1) the convolutional neural network uses an improved LeNet network model, the first three layers are convolutional layers, and the second two layers are full-connection layers; the number of convolution kernels of the convolution layer is 8, 16 and 32 respectively, and the sizes of the convolution kernels are all 5 x 5;
2) the convolutional neural network adopts a random gradient descent learning algorithm of a self-adaptive learning rate;
3) the nonlinear activation function of the convolutional neural network employs a ReLU function,
4) the cost function of the convolutional neural network adopts the cross entropy L _ CE of the estimated gesture class and the real gesture class vector, which is shown in the following formula
Where N represents the gesture class, yiAnd
a vector representing the i-th type of real gesture and a vector of the estimated gesture.
The process of the offline training phase of the convolutional neural network is as follows:
1) three data sets, namely group real data, BIC _ LR interpolation reconstruction data and SR _ SRCNN _ RES hyper-resolution reconstruction data are obtained in the hyper-resolution stage. Wherein, BIC _ LR and SR _ SRCNN _ RES can be divided into: BIC _ LR _4, BIC _ LR _8, and SR _4, SR _ 8. There were 5 data sets. Each data set contains 8 types of gestures, each type containing 80 radio frequency images, each image being 96 x 96 in size;
2) for each type of data set, 60 pieces of gesture data in each type of data set are used as training data, a convolutional neural network is trained, radio frequency image data serves as an input sample, and the category to which the radio frequency image data belongs serves as a label. And obtaining the trained 5-type network model parameters.
The convolutional neural network online estimation process is as follows:
1) and for each type of data set, inputting the rest 20 pieces of data of each type of gestures as test data into the trained network model to obtain the classification and identification accuracy of the gestures.
The gesture recognition results of the 5 data sets constructed by utilizing the image super-resolution network and the convolutional neural network are shown in table 1, and the table shows the recognition accuracy conditions of the gesture data sets constructed by using the interpolation method and the image super-resolution method provided by the invention and the recognition accuracy under real data when the amplification factor is 4 times and 8 times. Meanwhile, in fig. 5, radio frequency images representing the same scene in various data sets are visualized, HR represents a real radio frequency image, Bic represents a Bicubic interpolation reconstructed radio frequency image, SR represents an SRCNN _ RES over-division reconstructed radio frequency image, and 4 and 8 represent magnification factors.
TABLE 1 gesture recognition results
| BIC_LR | SRCNN_RES | Ground | |
| 4 times magnification | 80.625％ | 84.375％ | 86.076％ |
| 8 times magnification | 78.375％ | 81.875％ | 86.076％ |
When the magnification is 4 times, the gesture recognition accuracy rates of the interpolation data set BIC _ LR, the super-resolution data set SRCNN _ RES and the real data set are 80.625%, 84.375% and 86.076% respectively; when the magnification is 8 times, the gesture recognition accuracy rates of the interpolation data set BIC _ LR, the super-resolution data set srncn _ RES, and the real data set are 78.375%, 81.875%, and 86.076%, respectively. The image super-resolution method provided by the invention is closer to real data in gesture recognition precision, and the recognition precision is superior to that of the traditional Bicubic interpolation method. Meanwhile, with the increase of the magnification, the identification precision of the data set reconstructed by the Bicubic interpolation and SRCNN _ RES super-separation method is reduced, but the super-separation method is always superior to the interpolation method. The results show that the method can acquire the information content contained in the data set, so as to construct the data set closer to real data.
It can be seen from fig. 5 that the super-resolution enhancement method proposed by the present invention can reconstruct a radio frequency image closer to real data. Even if the magnification is 8 times, the Bicubic reconstruction image is greatly distorted, and the image super-resolution enhancement method provided by the invention can reconstruct a radio frequency image which is relatively close to real data.
The invention also discloses a radio frequency image identification method, which comprises the following steps: and constructing an identification task based on the radio frequency image, wherein the identification task is realized based on the enhanced radio frequency image data set constructed by the radio frequency image enhancement method.
In the embodiments provided in the present application, it should be understood that the disclosed technical content can be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units may be a logical division, and in actual implementation, there may be another division, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on a plurality of units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (6)
1. A radio frequency image enhancement method based on image super resolution is characterized by comprising the following steps:
acquiring a high-resolution radio-frequency image and a low-resolution radio-frequency image corresponding to the high-resolution radio-frequency image;
learning the mapping relation between the low-resolution radio frequency image and the high-resolution radio frequency image by using an image super-resolution network;
and constructing an enhanced radio frequency image data set based on the low-resolution radio frequency image based on the mapping relation between the low-resolution radio frequency image and the high-resolution radio frequency image.
2. The method for enhancing the radio-frequency image based on the image super-resolution according to claim 1, wherein the obtaining of the high-resolution radio-frequency image and the low-resolution radio-frequency image corresponding to the high-resolution radio-frequency image comprises:
measuring a high-resolution radio frequency image HR through a radar, wherein the size of the high-resolution radio frequency image HR is h x h;
and performing n-time down-sampling on the basis of the high-resolution radio frequency image HR through Bicubic to obtain a low-resolution radio frequency image LR, wherein the size of the low-resolution radio frequency image LR is (h/n) × (h/n).
And acquiring a hyper-resolution image SR through n times of hyper-resolution reconstruction based on the low-resolution radio frequency image LR.
3. The method for enhancing radio frequency image based on image super resolution according to claim 2, wherein a high resolution radio frequency image and a low resolution radio frequency image corresponding to the high resolution radio frequency image are obtained, further comprising:
after a high-resolution radio frequency image HR, a low-resolution radio frequency image LR and a hyper-resolution image SR are obtained, an image reconstruction measurement index PSNR is obtained according to the following calculation:
wherein the content of the first and second substances,
MSE represents the maximum value of pixel values in an image, and the MSE represents the corresponding image of two imagesAnd (4) the mean square error of the pixel difference values assumes that the length and the width of the image are all h.
4. The method for enhancing radio-frequency images based on image super-resolution according to claim 1, wherein learning the mapping relationship between the low-resolution radio-frequency image and the high-resolution radio-frequency image by using an image super-resolution network comprises:
performing n-time upsampling on the low-resolution radio frequency image LR through Bicubic to obtain a sampling image LR ═ ═ c;
constructing a training data set based on the sampling image LR ≠ and the high-resolution radio frequency image HR, training an image super-resolution network model by taking the sampling image LR ≠ as a training sample and taking the high-resolution radio frequency image HR as a label, wherein the super-resolution network model is used for learning a mapping relation between the sampling image LR ℃ and the high-resolution radio frequency truth image HR.
5. The method for enhancing radio frequency image based on image super resolution according to claim 1, wherein constructing a high quality radio frequency image data set based on low resolution radio frequency image based on mapping relationship between the low resolution radio frequency image and high resolution radio frequency image comprises:
the method comprises the steps of carrying out Bicubic interpolation on a low-resolution radio frequency image LR sample to be enhanced to obtain a sampling image LR ≠ to be enhanced, and inputting the sampling image LR ℃ to be enhanced into a trained image super-resolution network model to obtain a reconstructed enhanced image SR.
6. A radio frequency image recognition method, comprising: constructing a radio frequency image based identification task based on an enhanced radio frequency image dataset constructed by the radio frequency image enhancement method of claim 1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210204332.6A CN114693547A (en) | 2022-03-03 | 2022-03-03 | Radio frequency image enhancement method and radio frequency image identification method based on image super-resolution |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210204332.6A CN114693547A (en) | 2022-03-03 | 2022-03-03 | Radio frequency image enhancement method and radio frequency image identification method based on image super-resolution |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114693547A true CN114693547A (en) | 2022-07-01 |
Family
ID=82137073
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210204332.6A Pending CN114693547A (en) | 2022-03-03 | 2022-03-03 | Radio frequency image enhancement method and radio frequency image identification method based on image super-resolution |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114693547A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117291802A (en) * | 2023-09-27 | 2023-12-26 | 华北电力大学（保定） | Image super-resolution reconstruction method and system based on composite network structure |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107680043A (en) * | 2017-09-29 | 2018-02-09 | 杭州电子科技大学 | Single image super-resolution output intent based on graph model |
| CN107958444A (en) * | 2017-12-28 | 2018-04-24 | 江西高创保安服务技术有限公司 | A kind of face super-resolution reconstruction method based on deep learning |
| CN109345476A (en) * | 2018-09-19 | 2019-02-15 | 南昌工程学院 | High spectrum image super resolution ratio reconstruction method and device based on depth residual error network |
| CN109903226A (en) * | 2019-01-30 | 2019-06-18 | 天津城建大学 | Image super-resolution rebuilding method based on symmetrical residual error convolutional neural networks |
| CN110310227A (en) * | 2019-06-27 | 2019-10-08 | 电子科技大学 | A kind of image super-resolution rebuilding method decomposed based on high and low frequency information |
| CN111383173A (en) * | 2020-02-12 | 2020-07-07 | 山东师范大学 | Baseline-based image super-resolution reconstruction method and system |
| WO2021056969A1 (en) * | 2019-09-29 | 2021-04-01 | 中国科学院长春光学精密机械与物理研究所 | Super-resolution image reconstruction method and device |
| CN113344779A (en) * | 2021-05-10 | 2021-09-03 | 山东师范大学 | SAR image super-resolution method and system based on cartoon texture decomposition and deep learning |
| CN113706386A (en) * | 2021-09-04 | 2021-11-26 | 大连钜智信息科技有限公司 | Super-resolution reconstruction method based on attention mechanism |
-
2022
- 2022-03-03 CN CN202210204332.6A patent/CN114693547A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107680043A (en) * | 2017-09-29 | 2018-02-09 | 杭州电子科技大学 | Single image super-resolution output intent based on graph model |
| CN107958444A (en) * | 2017-12-28 | 2018-04-24 | 江西高创保安服务技术有限公司 | A kind of face super-resolution reconstruction method based on deep learning |
| CN109345476A (en) * | 2018-09-19 | 2019-02-15 | 南昌工程学院 | High spectrum image super resolution ratio reconstruction method and device based on depth residual error network |
| CN109903226A (en) * | 2019-01-30 | 2019-06-18 | 天津城建大学 | Image super-resolution rebuilding method based on symmetrical residual error convolutional neural networks |
| CN110310227A (en) * | 2019-06-27 | 2019-10-08 | 电子科技大学 | A kind of image super-resolution rebuilding method decomposed based on high and low frequency information |
| WO2021056969A1 (en) * | 2019-09-29 | 2021-04-01 | 中国科学院长春光学精密机械与物理研究所 | Super-resolution image reconstruction method and device |
| CN111383173A (en) * | 2020-02-12 | 2020-07-07 | 山东师范大学 | Baseline-based image super-resolution reconstruction method and system |
| CN113344779A (en) * | 2021-05-10 | 2021-09-03 | 山东师范大学 | SAR image super-resolution method and system based on cartoon texture decomposition and deep learning |
| CN113706386A (en) * | 2021-09-04 | 2021-11-26 | 大连钜智信息科技有限公司 | Super-resolution reconstruction method based on attention mechanism |
Non-Patent Citations (4)
| Title |
|---|
| CHAO DONG ET AL.: "European Conference on Computer Vision", 《ECCV 2014》, pages 184 * |
| XIBIN SONG ET AL.: "Deep Depth Super-Resolution : Learning Depth Super-Resolution using Deep Convolutional Neural Network", 《HTTPS://ARXIV.ORG/ABS/1607.01977V1》, pages 1 - 13 * |
| 胡众义: "《现代航运与物流安全绿色智能技术研究丛书 内河航运图像和视频去雾算法》", 北京航空航天大学出版社, pages: 124 - 181 * |
| 苏锦程: "基于学习的红外云图超分辨率重建算法研究", 《中国优秀硕士学位论文全文数据库信息科技辑》, no. 5, pages 13 - 17 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117291802A (en) * | 2023-09-27 | 2023-12-26 | 华北电力大学（保定） | Image super-resolution reconstruction method and system based on composite network structure |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Tian et al. | Lightweight image super-resolution with enhanced CNN | |
| CN109886871B (en) | Image super-resolution method based on channel attention mechanism and multi-layer feature fusion | |
| Zhang et al. | CCR: Clustering and collaborative representation for fast single image super-resolution | |
| CN112037131A (en) | Single-image super-resolution reconstruction method based on generation countermeasure network | |
| Wei | Image super‐resolution reconstruction using the high‐order derivative interpolation associated with fractional filter functions | |
| CN112270644A (en) | Face super-resolution method based on spatial feature transformation and cross-scale feature integration | |
| Kuang et al. | Image super-resolution with densely connected convolutional networks | |
| CN110533591B (en) | Super-resolution image reconstruction method based on codec structure | |
| CN108921801B (en) | Method and apparatus for generating image | |
| CN114549308B (en) | Image super-resolution reconstruction method and system with large receptive field and oriented to perception | |
| CN112950480A (en) | Super-resolution reconstruction method integrating multiple receptive fields and dense residual attention | |
| CN111127316A (en) | Single face image super-resolution method and system based on SNGAN network | |
| CN115880158A (en) | Blind image super-resolution reconstruction method and system based on variational self-coding | |
| CN115829834A (en) | Image super-resolution reconstruction method based on half-coupling depth convolution dictionary learning | |
| CN114693547A (en) | Radio frequency image enhancement method and radio frequency image identification method based on image super-resolution | |
| Xiang et al. | Boosting high-level vision with joint compression artifacts reduction and super-resolution | |
| Tian et al. | Image compressed sensing using multi-scale residual generative adversarial network | |
| Deng et al. | Multiple frame splicing and degradation learning for hyperspectral imagery super-resolution | |
| Yu et al. | Scale-aware frequency attention network for super-resolution | |
| Zhou et al. | Super-resolution image visual quality assessment based on structure–texture features | |
| Yang et al. | Learning degradation-aware deep prior for hyperspectral image reconstruction | |
| CN111968073B (en) | No-reference image quality evaluation method based on texture information statistics | |
| CN112184552A (en) | Sub-pixel convolution image super-resolution method based on high-frequency feature learning | |
| CN115330901B (en) | Image reconstruction method and device based on compressed sensing network | |
| Cheng et al. | Adaptive feature denoising based deep convolutional network for single image super-resolution |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |