CN114596209A - Fingerprint image restoration method, system, equipment and storage medium - Google Patents
Fingerprint image restoration method, system, equipment and storage medium Download PDFInfo
- Publication number
- CN114596209A CN114596209A CN202011418497.0A CN202011418497A CN114596209A CN 114596209 A CN114596209 A CN 114596209A CN 202011418497 A CN202011418497 A CN 202011418497A CN 114596209 A CN114596209 A CN 114596209A
- Authority
- CN
- China
- Prior art keywords
- image
- fingerprint image
- edge
- fingerprint
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 60
- 238000003860 storage Methods 0.000 title claims abstract description 24
- 238000003708 edge detection Methods 0.000 claims abstract description 118
- 238000013136 deep learning model Methods 0.000 claims abstract description 40
- 238000012549 training Methods 0.000 claims description 15
- 238000013528 artificial neural network Methods 0.000 claims description 14
- 230000009469 supplementation Effects 0.000 claims description 11
- 230000001502 supplementing effect Effects 0.000 claims description 2
- 230000006870 function Effects 0.000 description 13
- 238000010586 diagram Methods 0.000 description 10
- 238000012545 processing Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 8
- 230000008439 repair process Effects 0.000 description 6
- 230000003042 antagnostic effect Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 230000000295 complement effect Effects 0.000 description 3
- 238000013527 convolutional neural network Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000003044 adaptive effect Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 238000005520 cutting process Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 1
- 238000013256 Gubra-Amylin NASH model Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000008034 disappearance Effects 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000009191 jumping Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012634 optical imaging Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/77—Retouching; Inpainting; Scratch removal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Collating Specific Patterns (AREA)
- Image Input (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a fingerprint image restoration method, a system, equipment and a storage medium, wherein the method comprises the following steps: carrying out edge detection on the fingerprint image to be repaired to obtain an edge detection image; and inputting the edge detection image into a deep learning model, wherein the deep learning model is used for completing the edge detection image so as to obtain a repaired fingerprint image. The invention adopts the deep learning model to carry out complete restoration on the fingerprint image, not only can realize the restoration of the fingerprint image with smaller information missing area, but also can be applied to the restoration of the fingerprint image with larger information missing area, and improves the restoration precision of the fingerprint image and the definition of the restored image.
Description
Technical Field
The present invention relates to the field of fingerprint image completion technologies, and in particular, to a method, a system, a device, and a storage medium for repairing a fingerprint image.
Background
The fingerprint identification technology is used as an organism characteristic identification technology and is more and more widely applied to production and life of people. At present, a plurality of fingerprint identification and matching algorithms exist, but the use of complete and clear fingerprints is an important precondition for identification in order to realize fingerprint feature point detection and fingerprint matching with higher precision. In practical applications, the acquired fingerprint image may be incomplete, and some areas inside the fingerprint image may have information missing, for example, as shown in fig. 1, lines are broken in the middle area to have information missing areas. Therefore, the fingerprint image needs to be repaired first and then used for subsequent fingerprint identification.
At present, the fingerprint repairing technology is basically used for repairing fingerprints with broken lines, however, the existing repairing method is generally only suitable for repairing fingerprint images with small distance of broken lines, namely small information missing areas.
Disclosure of Invention
In view of the problems in the prior art, an object of the present invention is to provide a method, a system, a device and a storage medium for repairing fingerprint images, which can realize high-precision repairing of fingerprint images with small or large missing areas.
The embodiment of the invention provides a fingerprint image restoration method, which comprises the following steps:
carrying out edge detection on the fingerprint image to be repaired to obtain an edge detection image;
and inputting the edge detection image into a deep learning model, wherein the deep learning model is used for completing the edge detection image so as to obtain a repaired fingerprint image.
In some embodiments, the fingerprint image to be repaired includes at least one fingerprint line, and performing edge detection on the fingerprint image to be repaired to obtain an edge detection image includes:
and detecting the edge of the fingerprint line, and representing the detected edge of the fingerprint line in the edge detection image by adopting a line.
In some embodiments, the deep learning model for complementing the edge detection image comprises:
and the deep learning model is used for performing edge completion and content completion on the edge detection image.
In some embodiments, the fingerprint image to be repaired includes an information known region and an information unknown region, the edge detection image includes edges of fingerprint lines corresponding to the information known region, and the edge completion includes: supplementing an edge corresponding to the information unknown region in the edge detection image; and/or
The content completion includes filling in pixel values between edges in the edge detection image.
In some embodiments, the edges of the fingerprint lines comprise the boundaries of the fingerprint lines in a direction perpendicular to their extension.
In some embodiments, the deep learning model comprises a generative confrontation network model.
In some embodiments, the generative confrontational network model includes a plurality of modules, each module including at least one pose residual neural network.
In some embodiments, the edge-detected image input to the generative confrontation network model is a grayscale image or a color image.
In some embodiments, the generative confrontation network model includes an edge generator network for edge-complementing the edge detection image and a complementing network for content-complementing an output image of the edge generator network.
In some embodiments, said inputting said edge detection image into said depth learning model comprises:
inputting the edge detection image and a mask image into the edge generator network, wherein the mask image is used for distinguishing an information known region and an information unknown region in the edge detection image, and the edge generator network is used for performing edge completion on the information unknown region of the edge detection image according to the edge detection image and the mask image and outputting an edge-completed fingerprint image;
inputting the edge-supplemented fingerprint image and the fingerprint image to be repaired into the supplementation network, wherein the supplementation network is used for performing content supplementation on the edge-supplemented fingerprint image according to the fingerprint image to be repaired and outputting the repaired fingerprint image.
In some embodiments, the inputting the edge detection image into the depth learning model comprises:
inputting the edge detection image into the edge generator network, wherein the edge generator network is used for performing edge completion on the edge detection image and outputting an edge completed fingerprint image;
inputting the edge-completed fingerprint image and the fingerprint image to be repaired into the completion network, wherein the completion network is used for completing the content of the edge-completed fingerprint image according to the fingerprint image to be repaired and outputting the repaired fingerprint image.
In some embodiments, the edge generator network includes a hole convolution layer and a residual network layer.
In some embodiments, the number of intervals of convolution kernels in the hole convolution layer of the edge generator network is 1.
In some embodiments, the edge generator network comprises a pose residual neural network comprising at least one residual network layer and at least one deconvolution layer.
In some embodiments, the completion network comprises a hole convolutional layer and a residual network layer, or the completion network comprises at least one residual network layer and at least one deconvolution layer.
In some embodiments, before performing the edge detection on the fingerprint image to be repaired, the method further includes segmenting the fingerprint image to be repaired to obtain a plurality of sub-fingerprint images to be repaired, and the fingerprint image repairing method further includes:
respectively carrying out edge detection on the plurality of sub-fingerprint images to be repaired to obtain a plurality of edge detection images;
inputting the plurality of edge detection images into the deep learning model, wherein the deep learning model is used for respectively completing the plurality of edge detection images to obtain a plurality of completed sub-fingerprint images;
and splicing the plurality of supplemented sub-fingerprint images to obtain a repaired fingerprint image.
In some embodiments, the deep learning model is a deep learning model that is trained in advance using training samples.
The embodiment of the invention also provides a fingerprint image restoration system, which is used for realizing the fingerprint image restoration method and comprises the following steps:
the edge detection module is used for carrying out edge detection on the fingerprint image to be repaired to obtain an edge detection image;
and the image repairing module comprises the deep learning model, and the deep learning model is used for completing the edge detection image so as to obtain the repaired fingerprint image.
An embodiment of the present invention further provides a fingerprint image restoration device, including:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the fingerprint image fixing method via execution of the executable instructions.
The embodiment of the invention also provides a computer readable storage medium for storing a program, and the program realizes the steps of the fingerprint image repairing method when being executed by a processor.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
The fingerprint image restoration method, the system, the equipment and the storage medium have the following beneficial effects:
the method adopts the deep learning model to repair the fingerprint image, can realize the repair of the fingerprint image with smaller information missing area, can also be applied to the repair of the fingerprint image with larger information missing area, and improves the repair precision of the fingerprint image and the definition of the repaired image, thereby leading the repaired fingerprint image to be better applied to the subsequent analysis processes such as fingerprint identification and the like.
Drawings
Other features, objects and advantages of the present invention will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, with reference to the accompanying drawings.
FIG. 1 is a schematic illustration of a fingerprint image having areas of missing information;
FIG. 2 is a flowchart of a fingerprint image inpainting method according to one embodiment of the invention;
FIG. 3 is a schematic diagram of a two-stage generative confrontation network model according to the embodiment of the invention shown in FIG. 2;
FIG. 4 is a schematic diagram of a fingerprint image processing process according to the embodiment of the invention shown in FIG. 2;
FIG. 5 is a schematic illustration of fingerprint lines and edges of fingerprint lines of the embodiment of the invention shown in FIGS. 3 and 4;
FIG. 6 is a schematic illustration of a fingerprint authenticity image according to the embodiment of the invention shown in FIGS. 3 and 4;
FIG. 7 is a schematic diagram of a two-stage generative confrontation network model according to another embodiment of the invention;
FIG. 8 is a schematic illustration of a fingerprint image processing process according to the embodiment of the invention shown in FIG. 7;
FIG. 9 is a schematic diagram of a generative confrontation network model according to yet another embodiment of the invention;
FIG. 10 is a schematic diagram of a fingerprint image restoration system according to an embodiment of the present invention;
FIG. 11 is a schematic structural diagram of a fingerprint image restoration device according to an embodiment of the present invention;
fig. 12 is a schematic structural diagram of a computer-readable storage medium according to an embodiment of the present invention.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repetitive description will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.
As shown in fig. 2, an embodiment of the present invention provides a fingerprint image repairing method, including the following steps:
s100: carrying out edge detection on the fingerprint image to be repaired to obtain an edge detection image;
s200: inputting the edge detection image into a deep learning model, wherein the deep learning model is used for completing the edge detection image;
s300: and acquiring an image output by the deep learning model as a repaired fingerprint image.
In some embodiments, the fingerprint image to be repaired includes at least one fingerprint line, and performing edge detection on the fingerprint image to be repaired in step S100 to obtain an edge detection image includes: and detecting the edge of the fingerprint line, and representing the detected edge of the fingerprint line in the edge detection image by adopting a line. Specifically, the edge of each fingerprint line of the fingerprint image to be repaired may be detected, where each fingerprint line is regarded as a pattern with a certain area, and the pattern of each fingerprint line includes its edge (i.e., outer contour) and a portion located within an area defined by its edge. Only the edges of the fingerprint lines are included in the edge detection image, and the parts in the area defined by the edges of the fingerprint lines are not included, so that the edge detection image can be understood as the 'skeleton' of the fingerprint image to be repaired.
In some embodiments, the deep learning model used to complement the edge detection image in step S200 comprises: and the deep learning model is used for performing edge completion and content completion on the edge detection image.
In some embodiments, the deep learning model includes a generative adaptive network GAN (generic adaptive networks) model, which has become a popular research direction in the artificial intelligence community, and the GAN is mainly composed of a generator and a discriminator, and is trained by means of antagonistic learning, so as to estimate potential distribution of data samples and generate new data samples. The fingerprint image restoration method adopts the generative confrontation network model to restore the fingerprint image, not only can realize the restoration of the fingerprint image with smaller information missing area, but also can be applied to the restoration of the fingerprint image with larger information missing area, and improves the restoration precision of the fingerprint image and the definition of the restored image.
In this embodiment, the generative confrontation network model is a two-stage generative confrontation network model, and the generative confrontation network model includes an edge generator network and a completion network, the edge generator network is used for performing edge completion on the edge detection image, and the completion network is used for performing content completion on an output image of the edge generator network. Wherein the edge generator can generate an illusion of the edges of the missing region of the image and then use this illusion edge as an a priori complement of the missing region by the complementing network. Therefore, repairing a fingerprint image includes two parts, edge generation (including edge detection and edge completion) using an edge generator network and content completion using a completion network. Specifically, a two-phase GAN is utilized, wherein the producers of the first phase GAN are edge producer networks and the producers of the second phase GAN are completion networks. During training the two-stage generative confrontation network model, discriminators can be designed in the two stages respectively. The training of the antagonistic learning is carried out through the discriminator of the first stage and the edge generator network of the first stage, and the training of the antagonistic learning is carried out through the discriminator of the second stage and the completion network of the second stage, so that the trained second-order generation type antagonistic network model is obtained.
Fig. 3 shows an alternative network structure of a two-stage generative confrontation network model in the embodiment shown in fig. 2. Wherein the input image used is a part of the entire fingerprint image. Namely, in the step S100: before the edge detection is performed on the fingerprint image to be repaired, the method can further comprise the step of segmenting the whole fingerprint image before the repair to obtain a plurality of sub-fingerprint images to be repaired. In step S100, edge detection may be performed on the sub-fingerprint images to be repaired, respectively, to obtain a plurality of edge detection images. In step S200, a plurality of edge detection images may be input into a two-stage generative confrontation network model for complementing the plurality of edge detection images, respectively. In step S300: and acquiring a plurality of complemented sub-fingerprint images output by the generative confrontation network model, and splicing the plurality of complemented sub-fingerprint images to obtain the whole repaired fingerprint image.
This is only an alternative embodiment. In other alternative embodiments, the whole fingerprint image may be directly input into the two-stage generation type confrontation network model, and the whole repaired fingerprint image is directly obtained, which also falls within the protection scope of the present invention.
In this embodiment, in the step S100, performing edge detection on the fingerprint image to be repaired includes the following steps:
as shown in fig. 3 and 4, in an embodiment, first, the fingerprint image a1 to be repaired is edge-detected, and an edge-detected image b1 is obtained. Specifically, Canny edge detection can be performed on the fingerprint image a1 to be repaired, as can be seen from comparison between a1 and b1 in fig. 4, when Canny edge detection is used, in the edge detection image, the detected edge of each fingerprint line a11 is represented by a line b11, where "edge" refers to the outer contour of each fingerprint line a11, and the skeleton of the fingerprint image a1 can be represented by using the line.
As shown in fig. 5, fig. 5 shows an arbitrary fingerprint line a11 in the fingerprint image a1 of fig. 4 and an edge line b11 corresponding to the fingerprint line a11 in the edge detection image b 1. In some embodiments, the edge line b11 may be one continuous complete line describing the outer contour of the fingerprint line a 11; in other embodiments, the edge line b11 may be two or more discontinuous lines describing the outer contour of the fingerprint line a11, and may include only a boundary in the direction perpendicular to the extending direction of the fingerprint line a11, or both a boundary in the direction perpendicular to the extending direction of the fingerprint line a11 and a boundary in the extending direction of the fingerprint line a11, for example. As in the previous embodiments, each fingerprint line is considered herein to be a pattern of areas, each fingerprint line pattern including its edges (i.e., outer contour) and portions within the area defined by its edges. Only the edges of the fingerprint lines are included in the edge detection image b1, and the portions within the area defined by the edges of the fingerprint lines are not included, so the edge detection image b1 can be understood as the "skeleton" of the fingerprint image a1 to be repaired.
Since fingerprint images contain less detail than images of other people, landscapes, etc. And the directionality of the fingerprint image is different from the directionality of the texture of other images because the directionality of the texture of other images has consistency, but the fingerprint image has consistency only in the local part of the fingerprint. Thus, fingerprint images are more difficult to repair than other images.
In this embodiment, the step S200: inputting a fingerprint image to be repaired and the edge detection image into a generating type confrontation network model, and comprising the following steps:
with combined reference to fig. 3 and 4, the edge detection image b1 and the mask image m are input to the edge generator network a, which is used to edge complement the edge detection image b 1. In some embodiments, the fingerprint image a1 to be repaired comprises an information known area a1x and an information unknown area a1y, specifically, the information known area a1x may be an area containing fingerprint lines, and the information unknown area a1y may be a blank area which does not contain fingerprint information; accordingly, the edge detection image b1 includes an edge corresponding to the fingerprint image in the information known region and an information unknown region. In the mask image m, areas corresponding to the known information areas and the unknown information areas of the fingerprint image a1 to be repaired are represented by different colors, and the edge generator network a can identify the known information areas and the unknown information areas in the edge detection image b1 according to the mask image m; for example, the information known region in the mask image m is represented as black, the information unknown region is represented as white, and the information unknown region is an information missing region. In some embodiments, the fingerprint image a1 to be repaired is an image collected based on the optical imaging principle of total reflection, and a circular area with a specific radius in the middle of the fingerprint image a1 to be repaired is an information missing area.
The edge generator network a performing edge completion on the edge detection image b1 includes: performing edge completion on the unknown information region in the edge detection image b1, and outputting an edge-completed fingerprint image c1, where the edge of the known information region in the edge-completed fingerprint image c1 may be the same as the edge of the known information region in the edge detection image b1, that is, only the edge of the information-missing region of the edge detection image b1 is completed by the edge completion process of the edge generator network a, so that it can be understood that the partially completed edge is an "imaginary edge", and prediction or completion is not needed for the known information region of the edge detection image b 1; inputting the edge completion fingerprint image C1 and the fingerprint image a1 to be repaired into a completion network C, and performing content completion in a skeleton to obtain a repaired fingerprint image d1, wherein an output image of the completion network C is an output image of the two-stage generation type confrontation network model. In some embodiments, the content completion includes filling in pixel values between two adjacent edges in the edge detection image, and in conjunction with the fingerprint image to be repaired a1, the completion network C may determine which edges on the edge-completed fingerprint image C1 need to be filled with content, or what content is to be filled between any two adjacent edges. In some embodiments, the content complementing comprises padding pixel values between edges of an information known area in the edge-complemented fingerprint image c1 and estimating pixel values between edges of an information unknown area in the edge-complemented fingerprint image c1 from the fingerprint image a1 to be restored. The pixel values filled between the edges of the information known area in the edge complementing fingerprint image c1 can be the same as the pixel values of the corresponding fingerprint line patterns of the information known area in the fingerprint image a1 to be repaired.
In some embodiments, the edge generator network a further comprises a discriminator D _ a in addition to the generator G _ a, where L in D _ a is a loss function for calculating a difference between the edge image of the real fingerprint and the result generated in the first stage generator G _ a in the training stage to serve as an update parameter in the first stage generator G _ a, so that the first stage generator G _ a has better edge completion capability, where the loss function of the first stage comprises LFMAnd Ladv1，LFMRepresenting the difference between the real fingerprint edge image and the output image obtained from the first stage generator at the feature level, Ladv1Representing the penalty of confrontation between the first stage generator G _ a and the discriminator D _ a. The completion network C comprises, in addition to the generator G _ C, a discriminator D _ C, where L in D _ a is a loss function for calculating the difference between the actual fingerprint image and the result generated in the second stage generator G _ C in the training stage to serve as an update of the parameters in the second stage generator G _ C, so that the second stage generator G _ C has better content completion capability. The loss function of the second stage includes Ladv2，Ladv2Representing the penalty of opposition between the second stage generator G _ C and the discriminator D _ C.
Fig. 6 is a schematic diagram of the fingerprint real images of the embodiments shown in fig. 3 and fig. 4, and it can be seen from comparison between the repaired fingerprint image d1 and the fingerprint image a1 to be repaired and the fingerprint real image of fig. 6 that the fingerprint image repairing method of the embodiment can be used to realize accurate repairing of fingerprint images with large information missing areas.
Consider that a fingerprint image is typically a monochrome image. In this embodiment, the fingerprint image to be repaired and the edge detection image, which are input into the two-stage generative countermeasure network model, are both single-channel grayscale images. In other alternative embodiments, the fingerprint image to be repaired and the edge detection image input into the two-stage generative countermeasure network model are both multi-channel images, such as RGB three-channel color images, and the like, which are within the protection scope of the present invention.
As shown in fig. 3, in this embodiment, the edge generator network adopts the structure of an edge generator network a, including a hole convolution layer and a residual error network layer. The hole Convolution (scaled Convolution) is to increase the reception field (reflection field) by injecting holes based on the Convolution map of the standard Convolution. Therefore, the hole convolution layer has a super-parameter (super-parameter) called the expansion rate (scaling rate) added to the standard convolution, and the super-parameter refers to the number of intervals of kernel. A Residual Network (Residual Network) is a convolutional neural Network, which is characterized by easy optimization and can improve accuracy by increasing a considerable depth. The residual blocks in the deep neural network are connected by jumping, so that the problem of gradient disappearance caused by increasing the depth in the deep neural network is solved.
Because the fingerprint image is single channel data, the fingerprint image possesses less information than other types of images. And for edge detection images the information is very dense. In this embodiment, the number of intervals of convolution kernels in the hole convolution layer of the edge generator network a is set to 1, so that more information is extracted from the input image for prediction of the fingerprint line.
As shown in fig. 3, in this embodiment, the completion network C includes a hole convolution layer and a residual network layer. I.e. the completing network C has the same network structure as the edge generator network a. The completion network C and the edge generator network a adopt different real images in the training process, that is, the real image used by the edge generator network a for calculating the loss function in the training process is a real fingerprint edge image including a complete fingerprint edge, and the real image used by the completion network C for calculating the loss function in the training process is a real fingerprint image including a complete fingerprint edge and content, so that the completion network C and the edge generator network a have different network parameters.
As shown in fig. 7, in the second embodiment of the present invention, another structure of a two-stage generative countermeasure network model is provided. The edge generator network adopts the structure of an edge generator network B and comprises at least one residual error network layer and at least one deconvolution layer. In this embodiment, the edge generator network B includes four residual network layers and four deconvolution layers. Deconvolution (Deconvolution), which may also be referred to as transposed convolution, may up-sample the feature image, increasing the dimensionality of the feature image.
Further, the edge generator network B may be a pose-residual neural network (tip-rest) including the at least one residual network layer and the at least one deconvolution layer.
In some embodiments, an edge output layer for supervision may be further added to the output of the residual network layer. Specifically, an edge output layer for supervision may be added to the output of each residual network layer, or only one, two, or three edge output layers for supervision may be added, all of which are within the protection scope of the present invention. In the embodiment, a supervision mechanism is added into the residual error network to realize supervised learning, so that the accuracy of the skeleton completion of the fingerprint image by the edge generator network B can be further improved.
Further, the edge generator network B may include at least one pose residual neural network.
In this embodiment, the completion network C also includes a network structure identical to the edge generator network a, i.e., a structure of a hole convolution layer plus a residual network layer. In another alternative embodiment, the completion network C may also be configured as the same network structure as the edge generator network B, i.e. the completion network C comprises at least one residual network layer and at least one deconvolution layer, and the structure of the pose residual neural network may be further adopted. In yet another alternative embodiment, the edge generator may adopt the structure of the edge generator network a, and the completion network C adopts the structure including at least one residual network layer and at least one deconvolution layer. The above embodiments are within the scope of the present invention.
As shown in fig. 7 and 8, in another embodiment, first, edge detection is performed on an input fingerprint image a2 to be repaired to obtain an edge detection image B2, and the edge detection image B2 is input into the edge generator network B to obtain an edge-completed fingerprint image c2 output by the edge generator network B. This embodiment differs from the embodiment shown in fig. 3 in that the edge generator network B can perform edge completion on the edge detection image B2 without a mask image, for example, the edge generator network B can perform edge completion only on an information unknown region on the edge detection image B2; inputting the edge-supplemented fingerprint image C2 and the fingerprint image a2 to be repaired into a supplementation network C, and performing content supplementation in the skeleton to obtain a repaired fingerprint image d 2. As can be seen from the comparison between the repaired fingerprint image d2 and the fingerprint image a2 to be repaired, and the real fingerprint image shown in fig. 6, the fingerprint image repairing method of the embodiment can realize accurate repairing of a fingerprint image having a large information missing area. In some embodiments, the edge generator network B further comprises a discriminator D _ B in addition to the generator G _ B, where L in D _ B is a loss function for calculating a difference between the edge image of the real fingerprint and the result generated in the first stage generator G _ B in the training stage to serve as an update parameter in the first stage generator G _ B, so that the first stage generator G _ B has better edge completion capability, where the loss function of the first stage comprises LFMAnd Ladv1，LFMRepresenting the difference, L, between the real fingerprint edge image and the output image obtained by the first stage generator G _ B at the feature leveladv1Representing the penalty of confrontation between the first stage generator G _ B and the discriminator D _ B. The completion network C comprises a generator G _ C and a discriminator D _ C, wherein L in D _ A is a loss function used for calculating the real fingerprint image in the training stage and generating in a second stage generator G _ CThe difference in the results of (a) may serve to update the parameters in the second stage generator G _ C so that the second stage generator G _ C has better content completion capabilities. The loss function of the second stage includes Ladv2，Ladv2Representing the penalty of opposition between the second stage generator G _ C and the discriminator D _ C.
In other alternative embodiments, the edge generator network and the completion network may also adopt other neural network structures, and are not limited to the various structures listed here, and all fall within the protection scope of the present invention.
Fig. 9 is a schematic structural diagram of a generative confrontation network model in a fingerprint image restoration method according to yet another embodiment of the present invention. The generative countermeasure network model adopts a network model with only one generator, and also belongs to the protection scope of the invention. In this embodiment, taking the structure of the generator including four residual error networks + four deconvolution networks as an example (similar to the edge generator network B in the embodiment shown in fig. 7), edge completion and content completion are simultaneously implemented by using one network structure, that is, after the fingerprint image a3 to be repaired is input, the repaired fingerprint image d3 is directly obtained. The generative confrontation network model may also include a discriminator for optimizing model parameters of the generator during training. The loss function in training includes LFMAnd Ladv，LFMRepresenting the difference between the actual fingerprint image and the output image obtained by the generator, LadvRepresenting the penalty of opposition between the generator and the arbiter. In other embodiments, the one generator may also adopt a network structure similar to the edge generator network a of the embodiment shown in fig. 3, and the edge completion and the content completion are simultaneously implemented by one network structure.
In other embodiments, the generative confrontation network model may include a plurality of modules, each module including at least one pose-residual neural network (pos-respet). The modules may also have other NETWORK structures of neural NETWORKs, for example, NETWORK structures of other convolutional neural NETWORKs, such as UNET, HOURGLASS NETWORK, and the like.
The generative confrontation network model in the embodiment of the invention is a GAN model which is trained in advance by adopting training samples.
As shown in fig. 10, an embodiment of the present invention further provides a fingerprint image repairing system, which is used for implementing the fingerprint image repairing method according to the foregoing embodiment of the present invention, and the system includes:
the edge detection module M100 is configured to perform edge detection on the fingerprint image to be repaired to obtain the edge detection image;
and the image repairing module M200 comprises the deep learning model, and the deep learning model is used for completing the edge detection image so as to obtain the repaired fingerprint image.
In the fingerprint image restoration system of the present invention, the functions of the modules can be implemented by using the specific implementation of the fingerprint image restoration method described above. Specifically, the edge detection module M100 may perform edge detection by using the specific implementation of step S100, for example, by using a Canny edge detection method. The image restoration module M200 may use the deep learning model for image restoration according to the above-mentioned specific implementation of step S200, for example, a generative confrontation network model, which may use a network model with one generator, or a two-stage generative confrontation network model. In the two-stage generation type countermeasure network model, the edge generator network may adopt the structure of the edge generator network a or the edge generator network B, the completion network may adopt the structure including the hole convolution layer and the residual error network layer, or may adopt a structure including a plurality of residual error network layers and a plurality of deconvolution layers similar to the edge generator network B.
Further, the fingerprint image restoration system may further include an image cutting module and an image stitching module, where before the edge detection module M100 performs edge detection on a fingerprint image to be restored, the image cutting module is configured to segment the whole fingerprint image before restoration to obtain a plurality of sub-fingerprint images to be restored. After the image repairing module M200 obtains the image output by the deep learning model as a repaired sub-fingerprint image, the image stitching module is configured to stitch a plurality of repaired sub-fingerprint images to obtain the whole repaired fingerprint image.
In one embodiment, the fingerprint image restoration system adopts a two-stage generation type confrontation network model to restore a fingerprint image, wherein the restoration comprises two parts of edge restoration by adopting an edge generator network and content restoration by adopting a restoration network, so that the restoration of the fingerprint image with a smaller information missing area can be realized, the restoration of the fingerprint image with a larger information missing area can also be applied, and the restoration precision of the fingerprint image and the definition of the restored image are improved. In other alternative embodiments, the deep learning model may be a network model with only one generator, and the fingerprint image is subjected to edge completion and content completion at the same time. Alternatively, the deep learning model may be a structure including a plurality of modules, each module including at least one pose residual neural network. The modules may also have the NETWORK structure of other neural NETWORKs, for example, the NETWORK structure of other convolutional neural NETWORKs, such as UNET, HOURGLASS NETWORK, etc.
The embodiment of the invention also provides fingerprint image restoration equipment, which comprises a processor; a memory having stored therein executable instructions of the processor; wherein the processor is configured to perform the steps of the fingerprint image fixing method via execution of the executable instructions.
As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or program product. Thus, various aspects of the invention may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" platform.
An electronic device 600 according to this embodiment of the invention is described below with reference to fig. 11. The electronic device 600 shown in fig. 11 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 11, the electronic device 600 is in the form of a general purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one memory unit 620, a bus 630 that couples various system components including the memory unit 620 and the processing unit 610, a display unit 640, and the like.
Wherein the storage unit stores program code executable by the processing unit 610 to cause the processing unit 610 to perform steps according to various exemplary embodiments of the present invention described in the fingerprint image restoration method section described above in this specification. For example, the processing unit 610 may perform the steps as shown in fig. 2.
The storage unit 620 may include readable media in the form of volatile memory units, such as a random access memory unit (RAM)6201 and/or a cache memory unit 6202, and may further include a read-only memory unit (ROM) 6203.
The memory unit 620 may also include a program/utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 600, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 600 to communicate with one or more other computing devices. Such communication may occur via an input/output (I/O) interface 650. Also, the electronic device 600 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via the network adapter 660. The network adapter 660 may communicate with the other modules of the electronic device 600 via the bus 630. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
The embodiment of the invention also provides a computer readable storage medium for storing a program, and the program realizes the steps of the fingerprint image repairing method when being executed by a processor. In some possible embodiments, aspects of the invention may also be implemented in the form of a program product comprising program code for causing a terminal device to perform the steps according to various exemplary embodiments of the invention described in the fingerprint image fixing method section above of this specification, when the program product is executed on the terminal device.
Referring to fig. 12, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be executed on a terminal device, such as a personal computer. However, the program product of the present invention is not limited in this regard and, in the present document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The computer readable storage medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable storage medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
In summary, by using the fingerprint image restoration method, system, device and storage medium of the present invention, the fingerprint image is restored using the deep learning network, and the restoration includes the edge generation and the restoration, which not only can realize the restoration of the fingerprint image with a smaller information missing area, but also can be applied to the restoration of the fingerprint image with a larger information missing area, and improves the restoration precision of the fingerprint image and the definition of the restored image, so that the restored fingerprint image can be better applied to the subsequent analysis processes such as fingerprint identification.
The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, several simple deductions or substitutions can be made without departing from the spirit of the invention, and all shall be considered as belonging to the protection scope of the invention.
Claims (20)
1. A fingerprint image restoration method is characterized by comprising the following steps:
carrying out edge detection on the fingerprint image to be repaired to obtain an edge detection image;
and inputting the edge detection image into a deep learning model, wherein the deep learning model is used for completing the edge detection image so as to obtain a repaired fingerprint image.
2. The fingerprint image restoration method according to claim 1, wherein the fingerprint image to be restored includes at least one fingerprint line, and performing edge detection on the fingerprint image to be restored to obtain an edge detection image includes:
and detecting the edge of the fingerprint line, and representing the detected edge of the fingerprint line in the edge detection image by adopting a line.
3. The fingerprint image restoration method according to claim 2, wherein the deep learning model is used for complementing the edge detection image and comprises:
and the deep learning model is used for performing edge completion and content completion on the edge detection image.
4. The fingerprint image restoration method according to claim 3, wherein the fingerprint image to be restored includes an information known area and an information unknown area, the edge detection image includes edges of fingerprint lines corresponding to the information known area, and the edge completion includes: supplementing an edge corresponding to the information unknown region in the edge detection image; and/or
The content completion includes filling in pixel values between edges in the edge detection image.
5. The fingerprint image restoration method according to claim 2 or 4, wherein the edges of the fingerprint lines include boundaries of the fingerprint lines in a direction perpendicular to the extending direction thereof.
6. The fingerprint image restoration method according to claim 4, wherein the deep learning model comprises a generative confrontation network model.
7. The fingerprint image repairing method according to claim 6, wherein the generative confrontation network model comprises a plurality of modules, each module comprising at least one pose residual neural network.
8. The fingerprint image restoration method according to claim 6, wherein the edge detection image inputted to the generative confrontation network model is a grayscale image or a color image.
9. The fingerprint image repairing method according to claim 6, wherein the generative confrontation network model comprises an edge generator network and a completion network, the edge generator network is used for performing edge completion on the edge detection image, and the completion network is used for performing content completion on an output image of the edge generator network.
10. The fingerprint image inpainting method of claim 9, wherein the inputting the edge detection image into the deep learning model comprises:
inputting the edge detection image and a mask image into the edge generator network, wherein the mask image is used for distinguishing an information known region and an information unknown region in the edge detection image, and the edge generator network is used for performing edge completion on the information unknown region of the edge detection image according to the edge detection image and the mask image and outputting an edge-completed fingerprint image;
inputting the edge-supplemented fingerprint image and the fingerprint image to be repaired into the supplementation network, wherein the supplementation network is used for performing content supplementation on the edge-supplemented fingerprint image according to the fingerprint image to be repaired and outputting the repaired fingerprint image.
11. The fingerprint image repairing method according to claim 9, wherein said inputting the edge detection image into the deep learning model comprises:
inputting the edge detection image into the edge generator network, wherein the edge generator network is used for performing edge completion on the edge detection image and outputting an edge completed fingerprint image;
inputting the edge-supplemented fingerprint image and the fingerprint image to be repaired into the supplementation network, wherein the supplementation network is used for performing content supplementation on the edge-supplemented fingerprint image according to the fingerprint image to be repaired and outputting the repaired fingerprint image.
12. The fingerprint image repairing method according to claim 9 or 10, wherein said edge generator network comprises a hole convolution layer and a residual network layer.
13. The fingerprint image inpainting method of claim 12, wherein the number of intervals of the convolution kernels in the hole convolution layer of the edge generator network is 1.
14. The fingerprint image inpainting method of claim 9 or 11, wherein the edge generator network comprises a pose residual neural network comprising at least one residual network layer and at least one deconvolution layer.
15. The fingerprint image repairing method according to claim 9, 10 or 11, wherein the completion network comprises a hole convolution layer and a residual network layer, or wherein the completion network comprises at least one residual network layer and at least one deconvolution layer.
16. The fingerprint image restoration method according to claim 1, further comprising, before performing edge detection on the fingerprint image to be restored, segmenting the fingerprint image to be restored to obtain a plurality of sub-fingerprint images to be restored, and further comprising:
respectively carrying out edge detection on the plurality of sub-fingerprint images to be repaired to obtain a plurality of edge detection images;
inputting the plurality of edge detection images into the deep learning model, wherein the deep learning model is used for respectively completing the plurality of edge detection images to obtain a plurality of completed sub-fingerprint images;
and splicing the plurality of supplemented sub-fingerprint images to obtain a repaired fingerprint image.
17. The fingerprint image restoration method according to claim 1, wherein the deep learning model is a deep learning model trained in advance by using training samples.
18. A fingerprint image restoration system for implementing the fingerprint image restoration method according to any one of claims 1 to 17, the system comprising:
the edge detection module is used for carrying out edge detection on the fingerprint image to be repaired to obtain an edge detection image;
and the image repairing module comprises the deep learning model, and the deep learning model is used for completing the edge detection image so as to obtain the repaired fingerprint image.
19. A fingerprint image restoration device characterized by comprising:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the fingerprint image fixing method of any one of claims 1 to 17 via execution of the executable instructions.
20. A computer-readable storage medium storing a program, wherein the program is configured to implement the steps of the fingerprint image restoration method according to any one of claims 1 to 17 when executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011418497.0A CN114596209A (en) | 2020-12-07 | 2020-12-07 | Fingerprint image restoration method, system, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011418497.0A CN114596209A (en) | 2020-12-07 | 2020-12-07 | Fingerprint image restoration method, system, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114596209A true CN114596209A (en) | 2022-06-07 |
Family
ID=81813362
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011418497.0A Pending CN114596209A (en) | 2020-12-07 | 2020-12-07 | Fingerprint image restoration method, system, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114596209A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117456571A (en) * | 2023-12-21 | 2024-01-26 | 荣耀终端有限公司 | Fingerprint identification method and electronic equipment |
-
2020
- 2020-12-07 CN CN202011418497.0A patent/CN114596209A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117456571A (en) * | 2023-12-21 | 2024-01-26 | 荣耀终端有限公司 | Fingerprint identification method and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111898696B (en) | Pseudo tag and tag prediction model generation method, device, medium and equipment | |
| CN112001914B (en) | Depth image complement method and device | |
| CN110188760B (en) | Image processing model training method, image processing method and electronic equipment | |
| CN110245579B (en) | People flow density prediction method and device, computer equipment and readable medium | |
| US20240029272A1 (en) | Matting network training method and matting method | |
| US10831976B1 (en) | Predicting local layout effects in circuit design patterns | |
| KR20170038622A (en) | Device and method to segment object from image | |
| JP2022177232A (en) | Method for processing image, method for recognizing text, and device for recognizing text | |
| US20230351558A1 (en) | Generating an inpainted image from a masked image using a patch-based encoder | |
| CN111067522A (en) | Brain addiction structural map assessment method and device | |
| CN115147598A (en) | Target detection segmentation method and device, intelligent terminal and storage medium | |
| CN111444807B (en) | Target detection method, device, electronic equipment and computer readable medium | |
| CN116363037A (en) | Multi-mode image fusion method, device and equipment | |
| CN108520532B (en) | Method and device for identifying motion direction of object in video | |
| CN114596209A (en) | Fingerprint image restoration method, system, equipment and storage medium | |
| CN113723515B (en) | Moire pattern recognition method, device, equipment and medium based on image recognition | |
| CN112950652B (en) | Robot and hand image segmentation method and device thereof | |
| CN112052863B (en) | Image detection method and device, computer storage medium and electronic equipment | |
| CN114741697A (en) | Malicious code classification method and device, electronic equipment and medium | |
| US11158059B1 (en) | Image reconstruction based on edge loss | |
| CN115131291A (en) | Object counting model training method, device, equipment and storage medium | |
| CN114549992A (en) | Cross-resolution building image extraction method and device | |
| CN114821216A (en) | Method for modeling and using picture descreening neural network model and related equipment | |
| CN113781653A (en) | Object model generation method and device, electronic equipment and storage medium | |
| CN114118127A (en) | Visual scene mark detection and identification method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |