CN112686813A - Finger vein image restoration method based on partial convolution and mask updating - Google Patents

Finger vein image restoration method based on partial convolution and mask updating Download PDF

Info

Publication number
CN112686813A
CN112686813A CN202011528671.7A CN202011528671A CN112686813A CN 112686813 A CN112686813 A CN 112686813A CN 202011528671 A CN202011528671 A CN 202011528671A CN 112686813 A CN112686813 A CN 112686813A
Authority
CN
China
Prior art keywords
image
finger vein
repair
vein image
pixel
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202011528671.7A
Other languages
Chinese (zh)
Inventor
王丹丹
黄跃珍
魏东
杨芬
苏新铎
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
GRG Banking Equipment Co Ltd
Original Assignee
GRG Banking Equipment Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by GRG Banking Equipment Co Ltd filed Critical GRG Banking Equipment Co Ltd
Publication of CN112686813A publication Critical patent/CN112686813A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/77—Retouching; Inpainting; Scratch removal
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/14—Vascular patterns
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Biophysics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Multimedia (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Vascular Medicine (AREA)
  • Human Computer Interaction (AREA)
  • Collating Specific Patterns (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)

Abstract

The invention discloses a finger vein image restoration method based on partial convolution and mask updating, which comprises the following steps: detecting abnormal regions (noise regions) of the damaged finger vein image, and marking the abnormal regions; generating the damaged finger vein image with the abnormal region mark into a corresponding original mask image; inputting the damaged finger vein image and the original mask image into a pre-trained irregular hole image restoration model, wherein the irregular hole image restoration model performs feature coding and decoding operations on a non-abnormal area on the damaged finger vein image by adopting a partial convolution and mask updating mode according to the original mask image, and outputs a restoration image of the damaged finger vein image. The finger vein image repaired by the method has clear finger vein texture, natural transition of the edge of the original noise area, simple repairing process and obvious effect, and improves the success rate and the repairing speed of the finger vein image repair.

Description

Finger vein image restoration method based on partial convolution and mask updating
Technical Field
The invention relates to the technical field of biological identification, in particular to a finger vein image restoration method based on partial convolution and mask updating.
Background
With the development of artificial intelligence technology, people pay more and more attention to the protection of privacy information such as personal identity information. Finger vein recognition has the advantages of being difficult to forge and copy, high in safety and the like due to the fact that living body detection is achieved, and the finger vein recognition is widely applied to many scenes such as an access control system and a safe. The finger vein recognition mainly comprises four steps: collection, preprocessing, feature extraction and identification of finger vein images. The finger vein feature extraction is the most critical, and the identification performance of the finger vein identification system is directly influenced. In the image acquisition process, due to the influences of factors such as equipment and environment, noises such as finger chapping, peeling, dirty blocks on a mirror surface of acquisition equipment and the like may exist on the acquired finger vein image, so that the characteristics of the finger vein of the part are lost, and the identification performance of the finger vein identification system is seriously influenced.
At present, image restoration methods based on deep learning technology are adopted for restoring finger vein image noise areas. The existing image restoration method based on deep learning generally uses a standard convolution network, and performs feature convolution extraction on an effective region (non-noise region) and an ineffective region (noise region) as a whole, and then fills a pixel value of the effective region into a pixel missing region (ineffective region) to realize finger vein feature restoration of the noise region. However, the noise area repaired by the existing image repairing method based on deep learning usually has the problems of lack of finger vein texture features, unnatural edge transition of the noise area, excessively significant color contrast between the noise area and a normal area, and the like, and the problems need to be repaired for the second time in the later period, so that the process is complicated, and the repairing failure may occur.
Disclosure of Invention
The invention aims to provide a finger vein image restoration method based on partial convolution and mask updating.
In order to achieve the purpose, the invention adopts the following technical scheme:
the finger vein image restoration method based on partial convolution and mask updating is provided and comprises the following steps:
1) detecting abnormal regions of the damaged finger vein image, and marking the abnormal regions;
2) generating the damaged finger vein image with the abnormal region mark into a corresponding original mask image;
3) inputting the damaged finger vein image and the original mask image into a pre-trained irregular hole image restoration model, wherein the irregular hole image restoration model performs feature coding and decoding operations on a non-abnormal area on the damaged finger vein image by adopting a partial convolution and mask updating mode according to the original mask image, and outputs a restoration image of the damaged finger vein image.
Preferably, in the step 1), the step of detecting an abnormal region of the damaged finger vein image specifically includes:
1.1) carrying out image smoothing treatment on the damaged finger vein image to remove random noise and noise formed by fingerprints on the damaged finger vein image;
1.2) calculating the image gradient characteristics of the damaged finger vein image by using an edge detection operator, and extracting the edge information of the abnormal region to obtain an edge image corresponding to the abnormal region;
1.3) carrying out image binarization processing on the edge image;
1.4) carrying out image morphology processing on the binary image obtained by the calculation in the step 1.3);
1.5) calculating the ratio of the foreground pixel number of the binarized image after image morphology processing to the total pixel number of the binarized image, and judging whether the foreground of the binarized image is a finger vein feature abnormal area on the damaged finger vein image according to a threshold judgment method;
1.6) marking the foreground of the binary image which is judged to be abnormal in the finger vein characteristics as the abnormal area on the damaged finger vein image.
Preferably, in the step 1.4), the performing image morphology processing on the binarized image includes performing any one or more of expansion, corrosion, closing operation and connected domain hole filling on the binarized image.
Preferably, in the step 1.5), if a ratio of the number of foreground pixels in the binarized image to the total number of pixels of the binarized image is greater than 10%, the foreground of the binarized image is determined as the abnormal region where the finger vein features on the damaged finger vein image are abnormal.
Preferably, in the step 3), the partial convolution method adopted by the irregular hole image restoration model can be expressed by the following formula (1):
Figure BDA0002851571740000031
in formula (1), X represents a characteristic value of a current convolution window;
x' represents the result of the partial convolution;
w represents the weight of the convolution filter;
b is the bias corresponding to W;
m is a binary mask corresponding to the characteristic value X;
an indication of pixel-by-pixel multiplication;
Figure BDA0002851571740000032
represents a scaling;
sum (1) represents the number of pixels in the convolution window;
"1" in sum (1) denotes an all-1 matrix of the same size as M;
sum (m) represents the number of pixels of the non-noise region in the convolution window on the damaged finger vein image.
Preferably, (X | _ M) in formula (1) is calculated by the following formula (2):
Figure BDA0002851571740000041
in formula (2), (i, j) represents the pixel coordinate position in the current convolution window;
Figure BDA0002851571740000042
preferably, in the step 3), the mask updating method adopted by the irregular hole image repairing model can be expressed by the following formula (3):
Figure BDA0002851571740000043
in formula (3), m' represents the value of the binary mask corresponding to the updated eigenvalue X.
Preferably, the loss function L adopted by the irregular hole image restoration model is trainedtotalCan be expressed by the following formula (4):
Ltotal＝λ1Lhole+λ2Lvalid+λ3Lperceptual+λ4Lstyle+λ5Ltvformula (4)
In the formula (4), LholeThe repair loss of the vein missing region represents the difference between the original vein missing region on the repair map and the corresponding position region on the target image without missing;
Lvalidthe repair loss of the vein non-missing region represents the difference between the original non-vein missing region on the repair map and the corresponding position region on the target image;
Lperceptualrepresenting the difference between the convolution characteristic of a real image and the convolution characteristic of the finger vein image repaired by the model for perception loss;
Lstylerepresenting texture color difference between the repair map and the real image for style loss;
Ltvrepresenting the punishment of the smoothness of the finger vein image after restoration for the smoothness loss;
λ1、λ2、λ3、λ4、λ5are each Lhole、Lvalid、Lperceptual、Lstyle、LtvCorresponding loss weight.
Preferably, LholeCalculated by the following formula (5):
Figure BDA0002851571740000051
in the formula (5), IoutDamaged finger vein image I showing a missing region without repairinAn output repair map after model repair;
Igtis represented by the formula IinCorresponding target images without deficiency;
Figure BDA0002851571740000052
is represented bygtThe number of pixels in the array.
Preferably, LvalidCalculated by the following equation (6):
Figure BDA0002851571740000053
in the formula (6), IoutDamaged finger vein image I showing a missing region without repairinAn output repair map after model repair;
Igtis represented by the formula IinCorresponding target images without deficiency;
Figure BDA0002851571740000054
is represented bygtThe number of pixels of (a).
Preferably, LperceptualCalculated by the following equation (7):
Figure BDA0002851571740000055
in the formula (7), IoutDamaged finger vein image I showing a missing region without repairinAn output repair map after model repair;
Igtis represented by the formula IinCorresponding target images without deficiency;
Icompto repair the picture IoutSetting the pixel value of the pixel of the original non-missing region as IgtAn image obtained by adding pixel values corresponding to the pixel positions;
Figure BDA0002851571740000061
representing an image IgtA characteristic diagram of the p-th layer output after being activated by the loss network convolution;
Figure BDA0002851571740000062
representing an image IoutA characteristic diagram of the p-th layer output after being activated by the loss network convolution;
Figure BDA0002851571740000063
to represent
Figure BDA0002851571740000064
The number of the elements in (a) above,
Figure BDA0002851571740000065
the element of (1) is a feature map
Figure BDA0002851571740000066
Product of the dimensions.
Preferably, LstyleCalculated by the following equation (8):
Figure BDA0002851571740000067
in the formula (8), IoutDamaged finger vein image I showing a missing region without repairinAn output repair map after model repair;
Igtis represented by the formula IinCorresponding target images without deficiency;
Icompto repair the graph IoutSetting the pixel value of the pixel of the original non-missing region as IgtAn image obtained by adding pixel values corresponding to the pixel positions;
Figure BDA0002851571740000068
representing an image IgtA characteristic diagram of the p-th layer output after being activated by the loss network convolution;
Figure BDA0002851571740000069
representing an image IoutA characteristic diagram of the p-th layer output after being activated by the loss network convolution;
Figure BDA00028515717400000610
representing an image IcompA characteristic diagram of the p-th layer output after being activated by the loss network convolution;
cpcpis that
Figure BDA00028515717400000611
The size of the autocorrelation matrix;
Kp＝1/(HpWp Cp)；
(HpWp)×Cpis that
Figure BDA0002851571740000071
The size of the image of (a) is,
Figure BDA0002851571740000072
in (1)*Is IcompOr IgtOr Iout；
CpThe number of channels of the characteristic diagram representing the p-th layer output;
Hphigh of the characteristic diagram representing the p-th layer output;
Wpthe width of the profile representing the p-th layer output.
Preferably, LtvCalculated by the following equation (9):
Figure BDA0002851571740000073
in formula (9), IcompTo repair the graph IoutSetting the pixel value of the pixel of the original non-missing region as IgtAn image obtained by adding pixel values corresponding to the pixel positions;
Igtis represented by the formula IinCorresponding target images without deficiency;
Ioutdamaged finger vein image I showing a missing region without repairinAn output repair map after model repair;
Figure BDA0002851571740000074
representing an image IcompThe number of pixels above;
r represents an expansion region of a pixel;
i, j represents the coordinate position of the pixel;
Figure BDA0002851571740000075
is represented bycompA pixel value at coordinate location (i, j);
Figure BDA0002851571740000076
is shown and
Figure BDA0002851571740000077
pixel values of adjacent pixels in the same row and the next column;
Figure BDA0002851571740000078
is shown and
Figure BDA0002851571740000079
the pixel values of the next row of adjacent pixels in the same column.
According to the method, the damaged finger vein image is subjected to partial convolution according to the original mask image corresponding to the damaged finger vein image, the mask image is updated in the partial convolution process so as to eliminate the mask area on the damaged finger vein image, and then the finger vein image is restored through deconvolution and upsampling operations, so that the image restoration of the damaged finger vein image is completed. The finger vein image repaired by the method has clear finger vein texture, natural transition of the edge of the original noise area, simple repairing process and obvious effect, and improves the success rate and the repairing speed of the finger vein image repair.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the embodiments of the present invention will be briefly described below. It is obvious that the drawings described below are only some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort.
Fig. 1 is a step diagram of a finger vein image restoration method according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating abnormal region detection for a damaged finger vein image according to an embodiment of the present invention;
FIG. 3 is a schematic diagram illustrating the effect of detecting abnormal regions in an image of a damaged finger vein;
FIG. 4 is a network topology structure diagram of a neural network used for training an irregular hole image restoration model according to the present invention;
FIG. 5 is a schematic diagram of a network structure of a neural network used for training an irregular hole image restoration model according to the present invention;
fig. 6 is a schematic diagram of a mask image corresponding to an abnormal region on a damaged finger vein image.
Detailed Description
The technical scheme of the invention is further explained by the specific implementation mode in combination with the attached drawings.
Wherein the showings are for the purpose of illustration only and are shown by way of illustration only and not in actual form, and are not to be construed as limiting the present patent; to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of an actual product; it will be understood by those skilled in the art that certain well-known structures in the drawings and descriptions thereof may be omitted.
The same or similar reference numerals in the drawings of the embodiments of the present invention correspond to the same or similar components; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "inner", "outer", etc. are used for indicating the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, it is only for convenience of description and simplification of description, but it is not indicated or implied that the referred device or element must have a specific orientation, be constructed in a specific orientation and be operated, and therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and are not to be construed as limitations of the present patent, and the specific meanings of the terms may be understood by those skilled in the art according to specific situations.
In the description of the present invention, unless otherwise explicitly specified or limited, the term "connected" or the like, if appearing to indicate a connection relationship between the components, is to be understood broadly, for example, as being fixed or detachable or integral; can be mechanically or electrically connected; they may be directly connected or indirectly connected through intervening media, or may be connected through one or more other components or may be in an interactive relationship with one another. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art.
An embodiment of the present invention provides a finger vein image restoration method based on partial convolution and mask update, as shown in fig. 1, including:
step 1) carrying out abnormal region detection on the damaged finger vein image, and marking an abnormal region; the damaged finger vein image generally refers to a finger vein image with noise; the abnormal region refers to a noise region on the vein image.
Fig. 2 is a flowchart illustrating abnormal region detection on a damaged finger vein image according to an embodiment of the present invention, and as shown in fig. 2, the step of performing abnormal region detection on a damaged finger vein image according to the present invention specifically includes:
1.1) carrying out image smoothing treatment on the damaged finger vein image to remove random noise on the damaged finger vein image and noise formed by fingerprints;
1.2) calculating image gradient characteristics of the damaged finger vein image by using an edge detection operator, and extracting edge information of an abnormal area on the image to obtain an edge image corresponding to the abnormal area; for example, a Sobel edge detector may be used to highlight the sharp change in image intensity, so as to extract the edge of the abnormal region on the image;
1.3) carrying out image binarization processing on the edge image; for example, an adaptive threshold may be set through empirical summary, and if the pixel value in the edge image is greater than the adaptive threshold, the pixel value of the pixel is set to "0" and the pixel is represented as a pixel of a suspected noise region; otherwise, setting the value to be 0, and indicating that the pixel point is a pixel point of a non-noise area;
1.4) carrying out image morphology processing on the binary image obtained by the calculation in the step 1.3); the image morphological processing adopted by the invention comprises one or more of conventional image morphological processing methods such as expansion, corrosion, closed operation, connected domain hole filling and the like on the binary image;
1.5) calculating the ratio of the foreground pixel number of the binarized image after image morphology processing to the total pixel number of the binarized image, and judging whether the foreground of the binarized image is a finger vein characteristic abnormal area on a damaged finger vein image according to a threshold judgment method; in the invention, if the ratio of the foreground pixel number in the binary image to the total pixel number of the binary image is more than 10%, the foreground of the binary image is judged as an abnormal area with abnormal finger vein features on the damaged finger vein image, otherwise, the foreground area in the binary image is excluded as a noise area;
1.6) marking the foreground of the binary image which is judged to be abnormal in the finger vein characteristics as an abnormal area on the damaged finger vein image, namely a noise area on the damaged finger vein image.
Fig. 3 is a schematic diagram illustrating the effect of detecting abnormal regions in an image of a damaged finger vein. Fig. 3 (a) is a diagram showing an original damaged finger vein image; (b) the image is a binary image obtained by carrying out binarization processing on an edge image obtained by carrying out edge detection on the image (a); (c) the figure is an image obtained by performing image morphology processing on the figure (b); (d) the figure is a binarized mask image of the abnormal region finally determined.
Referring to fig. 1, the method for repairing a finger vein image according to an embodiment of the present invention further includes:
step 2), generating the damaged finger vein image with the abnormal region mark into a corresponding original mask image (see fig. 6 for a schematic diagram of the mask image); the specific method for generating the original mask image is not within the scope of the claimed invention, and therefore, will not be described in detail herein;
and 3) inputting the damaged finger vein image and the original mask image into a pre-trained irregular hole image restoration model, and performing feature coding and decoding operation on a non-abnormal area (effective area) on the damaged finger vein image by the irregular hole image restoration model in a partial convolution and mask updating mode according to the original mask image to output a restoration image of the damaged finger vein image.
The following specific description is given of the process of repairing the noise region on the damaged finger vein image by the irregular hole image repairing model in a partial convolution and mask updating manner:
the meaning of the partial convolution employed in the present invention will be explained first. The partial convolution adopted by the invention is different from the convolution operation of a common convolution neural network, the partial convolution uses a mask and a weighted convolution operation, and the pixel with the mask of '0' (the invention sets the pixel value in the detected abnormal area, namely the noise area, as '0') does not participate in the convolution operation. Since the abnormal region does not participate in the convolution operation, in the partial convolution process, the output size of the model needs to be adjusted after scaling the effective pixels (pixels in the non-noise region) by a certain proportion. The convolution result of the partial convolution employed in the present invention depends only on the pixels of the effective area (non-noise area).
The partial convolution method adopted by the irregular hole image restoration model provided by the invention can be expressed by the following formula (1):
Figure BDA0002851571740000121
in formula (1), X represents a characteristic value (pixel value) of the current convolution (sliding) window;
x' represents the result of the partial convolution;
w represents the weight of the convolution filter;
b is the bias corresponding to W;
m is a binary mask (represented by a two-dimensional vector) corresponding to the characteristic value X;
an indication of pixel-by-pixel multiplication;
Figure BDA0002851571740000122
represents a scaling;
sum (1) represents the number of pixels in the convolution window;
"1" in sum (1) denotes an all-1 matrix of the same size as M;
sum (m) represents the number of pixels of the non-noise region in the convolution window on the damaged finger vein image;
the English character "if" in formula (1) has the Chinese meaning of "if"; the English character "other" has the Chinese meaning of "other".
(X | _ M) in the formula (1) is calculated by the following formula (2):
Figure BDA0002851571740000123
in formula (2), (i, j) represents the pixel coordinate position in the current convolution window;
Figure BDA0002851571740000124
in the present invention, the noise area on the damaged finger vein image gradually shrinks as the partial convolution operation proceeds, so the mask image corresponding to the noise area needs to be updated on each convolution layer. If there are enough successive convolution layers, even the larger holes (noise regions) will gradually shrink with the depth of the partial convolution until they disappear, and finally only the response of the effective region will be left in the convolution signature. Therefore, after each partial convolution operation is performed, the value of the binary mask corresponding to the feature value X needs to be updated, and the update formula is expressed by the following formula (3):
Figure BDA0002851571740000131
in formula (3), m' represents the value of the binary mask corresponding to the updated eigenvalue X.
As can be seen from equation (3), as long as one pixel is valid in the mask region corresponding to the convolution kernel (convolution window), the mask corresponding to the convolution kernel is updated. Through multiple iterations, the masked area will be smaller and smaller, eventually completely disappearing.
FIG. 4 shows a network topology structure diagram of a neural network adopted by the irregular hole image inpainting model training method. FIG. 5 is a schematic diagram showing a network specific structure of a neural network adopted by the irregular hole image inpainting model training method. As shown in FIG. 4 and FIG. 5, the network structure of the neural network adopted by the irregular hole image restoration model trained by the invention is similar to the existing U-net semantic segmentation network. The U-net network is divided into an Encoder encoding module and a Decoder decoding module, and the 'Block' in FIG. 4 represents one or more layers of convolution characteristic diagrams. Skip-Link concatenates the Block profile of one coding module as input to the next layer of the decoding module. Feature transfer makes the finger vein image texture richer. The difference between the model training network provided by the invention and the existing U-net network is that the convolutional layer in the traditional U-net is replaced by a part of convolutional layer, namely, Block does not represent the convolutional characteristic diagram any more, but represents a part of convolutional characteristic diagram and a corresponding mask thereof. The nearest neighbor mode is used in the decoding module upsampling. The Skip-Link connects the partial convolution characteristic graph and the mask as the input of the next partial convolution, and the input of the last partial convolution layer comprises the original damaged finger vein image and the corresponding original mask image, so that the model can copy the pixels of the effective area.
Loss function L adopted by irregular hole image restoration model training methodtotalCan be expressed by the following formula (4):
Ltotal＝λ1Lhole+λ2Lvalid+λ3Lperceptual+λ4Lstyle+λ5Ltvformula (4)
In the formula (4), LholeA repair loss of a vein missing region (noise region) representing a difference between an original vein missing region (noise region on a damaged finger vein image before non-repair) on a repair map and a corresponding position region on a target image without missing;
Lvalida repair loss for a vein non-missing region (non-noise region) representing a difference between an original non-vein missing region (non-noise region on a damaged finger vein image before non-repair) on a repair map and a corresponding position region on a target image without missing;
Lperceptualrepresenting, for perceptual loss, a difference of a convolution feature of a real image and a convolution feature of the restoration map;
Lstylerepresenting texture color difference between the repair map and the real image for style loss;
Ltvrepresenting the punishment of the smoothness of the finger vein image after restoration for the smoothness loss;
λ1、λ2、λ3、λ4、λ5are each Lhole、Lvalid、Lperceptual、Lstyle、LtvCorresponding loss weight.
Specifically, the damaged finger vein image is represented as IinM is represented as image IinValue of binary mask, I, corresponding to characteristic value X in upper effective areaoutRepresented as image IinAnd (5) outputting the repair map after model prediction. I isgtIs shown as IinCorresponding to a target image without missing (noise-free). Based on this, LholeCan be calculated by the following formula (5):
Figure BDA0002851571740000141
in the formula (5), the first and second groups,
Figure BDA0002851571740000142
is represented bygtThe number of pixels in the array.
LvalidCalculated by the following equation (6):
Figure BDA0002851571740000143
loss of perception LperceptualCalculated by the following equation (7):
Figure BDA0002851571740000151
in the formula (7), IcompTo repair the picture IoutThe pixel value of the pixel of the original non-missing region (non-noise region) is directly set to IgtAn image obtained by adding pixel values corresponding to the pixel positions;
Figure BDA0002851571740000152
representing an image IgtA feature map output by the p-th layer after convolution activation of the loss network (for example, when the existing VGG-16 network is adopted as the loss network, any one of the first pooling layer pool1 to the third pooling layer pool3 in the VGG-16 network can be selected as the p-th layer);
Figure BDA0002851571740000153
representing an image IoutA characteristic diagram of the p-th layer output after being activated by the loss network convolution;
Figure BDA0002851571740000154
to represent
Figure BDA0002851571740000155
Number of elements of (1), element being a feature map
Figure BDA0002851571740000156
The product of dimensions, i.e. the number of signatures, the number of signature rows, the number of signature columns, is expressed by the formula (H)pWp)×CpExpression of CpThe number of channels of the characteristic diagram representing the p-th layer output; hpHigh of the characteristic diagram representing the p-th layer output; wpThe width of the profile representing the p-th layer output.
Here, it should be noted that the perceptual loss LperceptualIs an image Iout、IcompAnd IgtThe L1 distance after the image is mapped to the high-level feature space (the high-level features refer to the features generated by the image passing through a plurality of convolution layers of the loss network, and are compared with the pixel data or the low-level features of the image) through a pre-trained convolution network (also called a loss network, such as the conventional VGG-16 network) respectively reflects the similarity of the features obtained by the convolution of the real image and the features obtained by the convolution of the repaired image. The L1 for the two vectors is the sum of the absolute values of the elements of the two vectors subtracted. For example, vector X is (X1, X2, X3), vector Y is (Y1, Y2, Y3), and the L1 distance between X and Y is | | | X-Y | | survival1-x 1-y1| + | | x2-y2| + | | x3-y3 |. In the model training process, the high-level global information and the bottom-level detail information can be close to each other by considering the perception loss.
Stylistic losses are analogous to perceptual losses in computing image Iout、IcompAnd IgtThe loss generated when the autocorrelation matrix is calculated for each convolution feature map before the L1 distance in the high-level feature space reflects the difference in texture and color between the restored image and the original image.
Loss of style LstyleCan be calculated by the following equation (8):
Figure BDA0002851571740000161
in the formula (8), the first and second groups,
Figure BDA0002851571740000162
representing an image IcompA characteristic diagram of the p-th layer output after being activated by the loss network convolution;
cpcpis that
Figure BDA0002851571740000163
The size of the autocorrelation matrix;
Kp＝1/(HpWp Cp)；
(HpWp)×Cpis that
Figure BDA0002851571740000164
The image size of (a);
Cpthe number of channels of the characteristic diagram representing the p-th layer output;
Hphigh of the characteristic diagram representing the p-th layer output;
Wpthe width of the profile representing the p-th layer output.
LtvCalculated by the following equation (9):
Figure BDA0002851571740000165
in the formula (9), the reaction mixture,
Figure BDA0002851571740000166
representing an image IcompThe number of pixels above;
r represents an expansion region of a pixel;
i, j represents the coordinate position of the pixel;
Figure BDA0002851571740000171
is represented bycompA pixel value at coordinate location (i, j);
Figure BDA0002851571740000172
is shown and
Figure BDA0002851571740000173
pixel values of adjacent pixels in the same row and the next column;
Figure BDA0002851571740000174
is shown and
Figure BDA0002851571740000175
the pixel values of the next row of adjacent pixels in the same column.
It should be understood that the above-described embodiments are merely preferred embodiments of the invention and the technical principles applied thereto. It will be understood by those skilled in the art that various modifications, equivalents, changes, and the like can be made to the present invention. However, such variations are within the scope of the invention as long as they do not depart from the spirit of the invention. In addition, certain terms used in the specification and claims of the present application are not limiting, but are used merely for convenience of description.

Claims (13)

1. A finger vein image restoration method based on partial convolution and mask updating is characterized by comprising the following steps:
1) detecting abnormal regions of the damaged finger vein image, and marking the abnormal regions;
2) generating the damaged finger vein image with the abnormal region mark into a corresponding original mask image;
3) inputting the damaged finger vein image and the original mask image into a pre-trained irregular hole image restoration model, wherein the irregular hole image restoration model performs feature coding and decoding operations on a non-abnormal area on the damaged finger vein image by adopting a partial convolution and mask updating mode according to the original mask image, and outputs a restoration image of the damaged finger vein image.
2. The method for repairing a finger vein image according to claim 1, wherein the step of detecting an abnormal region in the damaged finger vein image in the step 1) specifically comprises:
1.1) carrying out image smoothing treatment on the damaged finger vein image to remove random noise and noise formed by fingerprints on the damaged finger vein image;
1.2) calculating the image gradient characteristics of the damaged finger vein image by using an edge detection operator, and extracting the edge information of the abnormal region to obtain an edge image corresponding to the abnormal region;
1.3) carrying out image binarization processing on the edge image;
1.4) carrying out image morphology processing on the binary image obtained by the calculation in the step 1.3);
1.5) calculating the ratio of the foreground pixel number of the binarized image after image morphology processing to the total pixel number of the binarized image, and judging whether the foreground of the binarized image is a finger vein feature abnormal area on the damaged finger vein image according to a threshold judgment method;
1.6) marking the foreground of the binary image which is judged to be abnormal in the finger vein characteristics as the abnormal area on the damaged finger vein image.
3. The method for repairing finger vein image according to claim 2, wherein in the step 1.4), the image morphology processing on the binarized image comprises any one or more of expansion, corrosion, closed operation and connected domain hole filling on the binarized image.
4. The method for repairing a finger vein image according to claim 2, wherein in step 1.5), if a ratio of a foreground pixel number in the binarized image to a total pixel number of the binarized image is greater than 10%, the foreground of the binarized image is determined as the abnormal region where the finger vein features on the damaged finger vein image are abnormal.
5. The finger vein image restoration method according to claim 1, wherein in the step 3), the partial convolution method adopted by the irregular hole image restoration model can be expressed by the following formula (1):
Figure FDA0002851571730000021
in formula (1), X represents a characteristic value of a current convolution window;
x' represents the result of the partial convolution;
w represents the weight of the convolution filter;
b is the bias corresponding to W;
m is a binary mask corresponding to the characteristic value X;
an indication of pixel-by-pixel multiplication;
Figure FDA0002851571730000022
represents a scaling;
sum (1) represents the number of pixels in the convolution window;
"1" in sum (1) denotes an all-1 matrix of the same size as M;
sum (m) represents the number of pixels of the non-noise region in the convolution window on the damaged finger vein image.
6. The finger vein image restoration method according to claim 5, wherein (X | _ M) in formula (1) is calculated by the following formula (2):
Figure FDA0002851571730000031
in formula (2), (i, j) represents the pixel coordinate position in the current convolution window;
Figure FDA0002851571730000032
7. the finger vein image restoration method according to claim 5, wherein in the step 3), the mask updating method adopted by the irregular hole image restoration model can be expressed by the following formula (3):
Figure FDA0002851571730000033
in formula (3), m' represents the value of the binary mask corresponding to the updated eigenvalue X.
8. The method for repairing finger vein image according to claim 5, wherein a loss function L is adopted for training the irregular hole image repairing modeltotalCan be expressed by the following formula (4):
Ltotal＝λ1Lhole+λ2Lvalid+λ3Lperceptual+λ4Lstyle+λ5Ltvformula (4)
In the formula (4), LholeThe repair loss of the vein missing region represents the difference between the original vein missing region on the repair map and the corresponding position region on the target image without missing;
Lvalidthe repair loss of the vein non-missing region represents the difference between the original non-vein missing region on the repair map and the corresponding position region on the target image;
Lperceptualrepresenting the difference between the convolution characteristic of a real image and the convolution characteristic of the finger vein image repaired by the model for perception loss;
Lstylerepresenting texture color difference between the repair map and the real image for style loss;
Ltvrepresenting the punishment of the smoothness of the finger vein image after restoration for the smoothness loss;
λ1、λ2、λ3、λ4、λ5are each Lhole、Lvalid、Lperceptual、Lstyle、LtvCorresponding loss weight.
9. The finger vein image restoration method according to claim 8, wherein L is LholeCalculated by the following formula (5):
Figure FDA0002851571730000041
in the formula (5), IoutDamaged finger vein image I showing a missing region without repairinAn output repair map after model repair;
Igtis represented by the formula IinCorresponding target images without deficiency;
Figure FDA0002851571730000044
is represented bygtThe number of pixels in the array.
10. The finger vein image restoration method according to claim 8, wherein L is LvalidCalculated by the following equation (6):
Figure FDA0002851571730000042
in the formula (6), IoutDamaged finger vein image I showing a missing region without repairinAn output repair map after model repair;
Igtis represented by the formula IinCorresponding target images without deficiency;
Figure FDA0002851571730000045
is represented bygtThe number of pixels of (a).
11. The finger vein image restoration method according to claim 8, wherein L is LperceptualCalculated by the following equation (7):
Figure FDA0002851571730000043
in the formula (7), IoutDamaged finger vein image I showing a missing region without repairinAn output repair map after model repair;
Igtis represented by the formula IinCorresponding target images without deficiency;
Icompto repair the picture IoutSetting the pixel value of the pixel of the original non-missing region as IgtAn image obtained by adding pixel values corresponding to the pixel positions;
Figure FDA0002851571730000051
representing an image IgtA characteristic diagram of the p-th layer output after being activated by the loss network convolution;
Figure FDA0002851571730000052
representing an image IoutA characteristic diagram of the p-th layer output after being activated by the loss network convolution;
Figure FDA0002851571730000053
to represent
Figure FDA0002851571730000054
The number of elements of (1), the elements being feature maps
Figure FDA0002851571730000055
Product of the dimensions.
12. The finger vein image restoration method according to claim 8, wherein L is LstyleCalculated by the following equation (8):
Figure FDA0002851571730000056
in the formula (8), IoutDamaged finger vein image I showing a missing region without repairinAn output repair map after model repair;
Igtis represented by the formula IinCorresponding target images without deficiency;
Icompto repair the graph IoutSetting the pixel value of the pixel of the original non-missing region as IgtAn image obtained by adding pixel values corresponding to the pixel positions;
Figure FDA0002851571730000057
representing an image IgtA characteristic diagram of the p-th layer output after being activated by the loss network convolution;
Figure FDA0002851571730000058
representing an image IoutA characteristic diagram of the p-th layer output after being activated by the loss network convolution;
Figure FDA00028515717300000510
representing an image IcompA characteristic diagram of the p-th layer output after being activated by the loss network convolution;
cpcp is
Figure FDA0002851571730000059
The size of the autocorrelation matrix;
Kp＝1/(HpWpCp)；
(HpWp)×Cpis that
Figure FDA0002851571730000061
The size of the image of (a) is,
Figure FDA0002851571730000062
in (1)*Is IcompOr IgtOr Iout；
CpThe number of channels of the characteristic diagram representing the p-th layer output;
Hphigh of the characteristic diagram representing the p-th layer output;
Wpthe width of the profile representing the p-th layer output.
13. The finger vein image restoration method according to claim 8, wherein L is LtvCalculated by the following equation (9):
Figure FDA0002851571730000063
in formula (9), IcompTo repair the graph IoutSetting the pixel value of the pixel of the original non-missing region as IgtAn image obtained by adding pixel values corresponding to the pixel positions;
Igtis represented by the formula IinCorresponding target images without deficiency;
Ioutdamaged finger vein image I showing a missing region without repairinAn output repair map after model repair;
Figure FDA0002851571730000067
representing an image IcompThe number of pixels above;
r represents an expansion region of a pixel;
i, j represents the coordinate position of the pixel;
Figure FDA0002851571730000064
is represented bycompA pixel value at coordinate location (i, j);
Figure FDA0002851571730000065
is shown and
Figure FDA0002851571730000066
pixel values of adjacent pixels in the same row and the next column;
Figure FDA0002851571730000071
is shown and
Figure FDA0002851571730000072
the pixel values of the next row of adjacent pixels in the same column.
CN202011528671.7A 2020-12-10 2020-12-22 Finger vein image restoration method based on partial convolution and mask updating Pending CN112686813A (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN2020114555078 2020-12-10
CN202011455507 2020-12-10

Publications (1)

Publication Number Publication Date
CN112686813A true CN112686813A (en) 2021-04-20

Family

ID=75450570

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011528671.7A Pending CN112686813A (en) 2020-12-10 2020-12-22 Finger vein image restoration method based on partial convolution and mask updating

Country Status (2)

Country Link
CN (1) CN112686813A (en)
WO (1) WO2022121031A1 (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113409214A (en) * 2021-06-22 2021-09-17 西安交通大学 Rapid repairing method for damaged fingerprint image
WO2022121031A1 (en) * 2020-12-10 2022-06-16 广州广电运通金融电子股份有限公司 Finger vein image restoration method based on partial convolution and mask updating
CN116879112A (en) * 2023-09-06 2023-10-13 国网湖北省电力有限公司超高压公司 Hydrophobicity grade judging method and system

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115272639B (en) * 2022-09-19 2022-12-23 武汉天际航信息科技股份有限公司 Method, apparatus and computer program product for repairing vehicle regions in an image
CN116452741B (en) * 2023-04-20 2024-03-01 北京百度网讯科技有限公司 Object reconstruction method, object reconstruction model training method, device and equipment
CN117196981B (en) * 2023-09-08 2024-04-26 兰州交通大学 Bidirectional information flow method based on texture and structure reconciliation
CN117372720B (en) * 2023-10-12 2024-04-26 南京航空航天大学 Unsupervised anomaly detection method based on multi-feature cross mask repair

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190295228A1 (en) * 2018-03-21 2019-09-26 Nvidia Corporation Image in-painting for irregular holes using partial convolutions
CN110348375A (en) * 2019-07-09 2019-10-18 华南理工大学 A kind of finger vena region of interest area detecting method neural network based
CN110443842A (en) * 2019-07-24 2019-11-12 大连理工大学 Depth map prediction technique based on visual angle fusion
CN111523411A (en) * 2020-04-10 2020-08-11 陕西师范大学 Synthetic aperture imaging method based on semantic patching
CN111784602A (en) * 2020-06-28 2020-10-16 江西理工大学 Method for generating countermeasure network for image restoration
CN111986107A (en) * 2020-08-05 2020-11-24 中国人民解放军战略支援部队信息工程大学 DSM local deletion repair method based on deep learning

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11049000B2 (en) * 2018-03-27 2021-06-29 International Business Machines Corporation Distributed state via cascades of tensor decompositions and neuron activation binding on neuromorphic hardware
CN110188614B (en) * 2019-04-30 2021-03-30 杭州电子科技大学 NLM filtering finger vein denoising method based on skin crack segmentation
CN112686813A (en) * 2020-12-10 2021-04-20 广州广电运通金融电子股份有限公司 Finger vein image restoration method based on partial convolution and mask updating

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190295228A1 (en) * 2018-03-21 2019-09-26 Nvidia Corporation Image in-painting for irregular holes using partial convolutions
CN110348375A (en) * 2019-07-09 2019-10-18 华南理工大学 A kind of finger vena region of interest area detecting method neural network based
CN110443842A (en) * 2019-07-24 2019-11-12 大连理工大学 Depth map prediction technique based on visual angle fusion
CN111523411A (en) * 2020-04-10 2020-08-11 陕西师范大学 Synthetic aperture imaging method based on semantic patching
CN111784602A (en) * 2020-06-28 2020-10-16 江西理工大学 Method for generating countermeasure network for image restoration
CN111986107A (en) * 2020-08-05 2020-11-24 中国人民解放军战略支援部队信息工程大学 DSM local deletion repair method based on deep learning

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022121031A1 (en) * 2020-12-10 2022-06-16 广州广电运通金融电子股份有限公司 Finger vein image restoration method based on partial convolution and mask updating
CN113409214A (en) * 2021-06-22 2021-09-17 西安交通大学 Rapid repairing method for damaged fingerprint image
CN116879112A (en) * 2023-09-06 2023-10-13 国网湖北省电力有限公司超高压公司 Hydrophobicity grade judging method and system

Also Published As

Publication number Publication date
WO2022121031A1 (en) 2022-06-16

Similar Documents

Publication Publication Date Title
CN112686813A (en) Finger vein image restoration method based on partial convolution and mask updating
Bunk et al. Detection and localization of image forgeries using resampling features and deep learning
CN110322495B (en) Scene text segmentation method based on weak supervised deep learning
US20230021661A1 (en) Forgery detection of face image
CN111553837B (en) Artistic text image generation method based on neural style migration
CN107516316B (en) Method for segmenting static human body image by introducing focusing mechanism into FCN
Choudhary et al. Signature Recognition & Verification System Using Back Propagation Neural Network
JP2021532453A (en) Extraction of fast and robust skin imprint markings using feedforward convolutional neural networks
Chakraborty PRNU-based image manipulation localization with discriminative random fields
Barni et al. Iris deidentification with high visual realism for privacy protection on websites and social networks
Jalilian et al. Persian sign language recognition using radial distance and Fourier transform
Gao A method for face image inpainting based on generative adversarial networks
CN113128364A (en) Fingerprint biological key generation method based on deep neural network coding
Liu et al. Palm-dorsa vein recognition based on independent principle component analysis
Chowdhury et al. Fuzzy rule based approach for face and facial feature extraction in biometric authentication
Bae et al. Fingerprint image denoising and inpainting using convolutional neural network
Zaidan et al. A novel hybrid module of skin detector using grouping histogram technique for Bayesian method and segment adjacent-nested technique for neural network
CN112084922B (en) Method for detecting crowd with abnormal behaviors based on gestures and facial expressions
Bappy et al. Detection and localization of image forgeries using resampling features and deep learning
CN112906527A (en) Finger vein biological key generation method based on deep neural network coding
Wyzykowski et al. A Universal Latent Fingerprint Enhancer Using Transformers
CN111738274B (en) Anti-attack camera source identification method based on local smooth projection
CN116912183B (en) Method and system for tampering and positioning depth repair image based on edge guiding and contrast loss
CN113283520B (en) Feature enhancement-based depth model privacy protection method and device for membership inference attack
Manjunatha et al. Tampering Detection and Segmentation Model for Multimedia Forensic

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20210420

RJ01 Rejection of invention patent application after publication