CN110232337B - Chinese character image stroke extraction method and system based on full convolution neural network - Google Patents
Chinese character image stroke extraction method and system based on full convolution neural network Download PDFInfo
- Publication number
- CN110232337B CN110232337B CN201910454930.7A CN201910454930A CN110232337B CN 110232337 B CN110232337 B CN 110232337B CN 201910454930 A CN201910454930 A CN 201910454930A CN 110232337 B CN110232337 B CN 110232337B
- Authority
- CN
- China
- Prior art keywords
- stroke
- skeleton
- overlapped
- character image
- chinese character
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000605 extraction Methods 0.000 title claims abstract description 73
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 49
- 238000000034 method Methods 0.000 claims abstract description 75
- 238000012549 training Methods 0.000 claims abstract description 25
- 239000011159 matrix material Substances 0.000 claims description 23
- 238000010586 diagram Methods 0.000 claims description 17
- 238000004422 calculation algorithm Methods 0.000 claims description 11
- 230000005484 gravity Effects 0.000 claims description 11
- 230000008602 contraction Effects 0.000 claims description 5
- 238000005457 optimization Methods 0.000 claims description 3
- 238000011084 recovery Methods 0.000 claims 1
- 238000001308 synthesis method Methods 0.000 abstract description 2
- 230000008569 process Effects 0.000 description 9
- 238000013527 convolutional neural network Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 239000000284 extract Substances 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000012805 post-processing Methods 0.000 description 3
- 238000005070 sampling Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- 238000003064 k means clustering Methods 0.000 description 2
- 238000003909 pattern recognition Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000012916 structural analysis Methods 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 238000013441 quality evaluation Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/22—Character recognition characterised by the type of writing
- G06V30/226—Character recognition characterised by the type of writing of cursive writing
- G06V30/2268—Character recognition characterised by the type of writing of cursive writing using stroke segmentation
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Character Discrimination (AREA)
Abstract
The invention belongs to the field of computer vision and mode recognition, and particularly relates to a Chinese character image stroke extraction method and system based on a full convolution neural network, aiming at solving the problem of difficulty in extracting free-written handwritten character strokes. The method comprises the following steps: extracting the region of the acquired Chinese character image; performing skeletonization operation on the overlapped region and the non-overlapped region; calculating the consistency between any stroke segments in the skeletonized overlapped area; and connecting the stroke segments belonging to the same stroke in the overlapped area, and combining the directly connected stroke segments in the non-overlapped area into a complete skeleton-shaped stroke. On one hand, the invention can still realize the stroke extraction of the handwritten Chinese character under the condition that the strokes of the handwritten Chinese character which is freely written are overlapped, on the other hand, the invention adopts a character synthesis method to obtain the training sample and attaches different marking information of the training sample in different tasks, thereby greatly saving the labor cost.
Description
Technical Field
The invention belongs to the field of computer vision and pattern recognition, and particularly relates to a Chinese character image stroke extraction method and system based on a full convolution neural network.
Background
Stroke extraction of chinese character images plays an important role in structural analysis-based character recognition research and related applications. The single character classification of Chinese handwritten/printed characters based on deep learning technology has achieved quite high accuracy, but in many applications, people not only pay attention to the character type, but also pay attention to the problems of stroke interpretation, writing quality evaluation, shape beautification, font design and the like, and thus the strokes in the character image need to be segmented and extracted.
For the problem of stroke extraction of offline Chinese characters, the existing algorithms in the past mainly have two types: direct extraction methods and extraction methods based on character skeletons. The direct extraction method is mainly used for printing characters, and when a character image has smooth edges, simple stroke shapes, fixed stroke widths and clear relationships among strokes, the effect of the method is good, for example, researchers [1] such as Tseng, Chuang and the like summarize some general laws from character structures of various printing fonts, and perform stroke extraction through heuristic rules; cao, Tan and other [2] divide the printed characters into stroke segments (total 3 types) according to similar rules, and then screen and recombine the stroke segments into independent strokes; lee and Wu [3] represent the printed character image in the form of Graph (Graph), and infer the connection relation between stroke segments according to outline characteristics in the overlapped area of the strokes; chen et al [4] learn a two-dimensional manifold from a standard font and then use a template character corresponding to the manifold (in which the strokes have been extracted) to guide the stroke extraction of a real print sample. When an offline handwritten character image is processed, because the freehand handwritten character has higher diversity and complexity in the stroke shape and the relationship between strokes, it is difficult to achieve an ideal effect by directly extracting the strokes by using a heuristic rule. Therefore, most of the existing work of stroke extraction for offline handwritten characters is operated on a character skeleton, which simplifies the stroke extraction task at the connected region level into extraction at the line level [5 ]. When stroke extraction is carried out on a character skeleton, the rules adopted by most existing methods are similar to the relevant parts in the direct extraction method. The stroke extraction based on skeletonization faces the problems of skeleton distortion (especially stroke overlapping areas) and stroke segment connection ambiguity, and a method for solving the problems is not well solved so far.
In general, although researchers have proposed many methods for stroke extraction in chinese print/handwritten character images, major attention has been paid to more regular characters. For free-written handwritten characters, because the stroke shapes and positions are variable, and the condition of the stroke overlapping area is very complex, great challenges are brought to stroke extraction, and the conventional method does not give satisfactory results.
The following documents are background information related to the present invention:
[1]Lin Yu Tseng and Chen-Tsun Chuang."An efficient knowledge-based stroke extraction method for multi-font Chinese characters."Pattern Recognition,25(12):1445-1458,1992.
[2]Ruini Cao and Chew Lim Tan."A model of stroke extraction from Chinese character images."In.Proceedings of the 15th International Conference on Pattern Recognition,2000.
[3]Chungnan Lee and Bohom Wu."A Chinese-character-stroke-extraction algorithm based on contour information."Pattern Recognition,31(6):651-663,1998.
[4]Xudong Chen,Zhouhui Lian,Yingmin Tang,and Jianguo Xiao."An automatic stroke extraction method using manifold learning."In.Proceedings of Eurographics,2017.
[5]Cheng-Lin Liu,In-Jung Kim,and Jin H.Kim."Model-based stroke extraction and matching for handwritten Chinese character recognition."Pattern Recognition,34(12):2339-2352,2001.
[6]Tie-Qiang Wang and Cheng-Lin Liu,"Fully convolutional network based skeletonization for handwritten Chinese characters."AAAI Conference on Artificial Intelligence,2018.
[7]Byungsoo Kim,Oliver Wang,A.Cengiz
and Markus Gross."Semantic segmentation for line drawing vectorization using neural networks."In Proceedings of Eurographics,2017.
disclosure of Invention
In order to solve the above problems in the prior art, namely the problem that the strokes of the free-written handwritten characters are difficult to extract, the invention provides a Chinese character image stroke extraction method based on a full convolution neural network, and the extraction method comprises the following steps:
step S10, acquiring a Chinese character image as an input image;
step S20, extracting an overlapping region map of character strokes in the input image; the part of the input image, which is removed from the overlapped area, is a non-overlapped area map;
step S30, performing skeletonization operation on the overlapped region graph and the non-overlapped region graph respectively to obtain an overlapped region skeleton form stroke segment set and a non-overlapped region skeleton form stroke segment set;
step S40, calculating a consistency matrix between any two stroke segments based on the overlapping region skeleton form stroke segment set; two stroke segments with all elements in the consistency matrix being larger than or equal to a preset threshold belong to the same stroke;
step S50, connecting the stroke segments belonging to the same stroke in the overlapped region, and combining the stroke segments and the directly connected stroke segments in the non-overlapped region into a complete skeleton-shaped stroke.
In some preferred embodiments, step S10, "obtain chinese character image as input image", is performed by:
acquiring an acquired Chinese character image, removing the background of the acquired Chinese character image through an OTSU method-based global binarization algorithm or a local self-adaptive binarization algorithm to obtain a foreground image of the Chinese character image, and using the foreground image as an input image.
In some preferred embodiments, step S20 "extracts an overlap region map of character strokes in the input image; the part of the input image, which is removed from the overlapped area, is a non-overlapped area map, and the method comprises the following steps:
step S201, extracting the characteristics of the input image through an overlapped area extraction network contraction path based on the input image;
step S202, based on the characteristics of the input image, reversely generating through an expansion path symmetrical to the network contraction path extracted from the overlapped area, and acquiring a character stroke overlapped area diagram; the part of the input image, which is removed from the overlapped area, is a non-overlapped area map;
wherein the overlap region extraction network is a network constructed based on a full convolution neural network and used for extracting an overlap region map of character strokes in the input image.
In some preferred embodiments, step S40, "calculate a coherence matrix between any two stroke segments", is performed by:
step S401, selecting any two stroke segments in the skeleton form stroke segment set of the overlapped area, and respectively recording the two stroke segments as S1、S2；
Step S402, in the stroke segment S1、S2Uniformly selecting N points, and respectively recording the points as a set
Step S403, calculating the set by adopting a conditional full convolution network
And
obtaining NxN probabilities of any two points belonging to the same stroke to form a stroke segment S1、S2A coherence matrix between.
In some preferred embodiments, the skeleton extracts a training sample of the network, and the obtaining method is as follows:
step B10, using the stroke coordinate point sequence of the online handwritten character as the skeleton of the synthesized character image, and setting the stroke width;
step B20, expanding the skeleton of the synthesized character image into wide strokes according to the set stroke width based on the skeleton of the synthesized character image to obtain a synthesized character image; and the synthesized character image and the corresponding skeleton are training samples of the skeleton extraction network.
In some preferred embodiments, the method for extracting the training samples of the network in the overlapping area includes:
g10, obtaining a synthetic character image by adopting the method of the steps B10-B20 of the Chinese character image stroke extraction method based on the full convolution neural network;
step G20, calculating a stroke overlapping area corresponding to the synthesized character image based on the stroke coordinate point sequence information of the synthesized character image; and extracting training samples of a network for the overlapped area of the synthesized character image and the corresponding stroke.
In some preferred embodiments, after the step S30 of obtaining the skeletal form stroke segment set in the character stroke overlapping area and the skeletal form stroke segment set in the character stroke non-overlapping area, there is further provided an optimization step of the skeletal form stroke segment, and the method is as follows:
calculating the gravity center of each overlapped region in the overlapped region graph, acquiring all adjacent skeleton points of the region corresponding to the gravity center, and connecting the gravity center of the overlapped region and the adjacent skeleton points one by one in the skeleton graph to obtain an optimized overlapped region skeleton form stroke segment set;
and recalling skeleton pixel points by a clustering method based on the character stroke non-overlapping area to obtain an optimized skeleton form stroke segment set of the non-overlapping area.
In some preferred embodiments, after "connecting the stroke segments belonging to the same stroke in the overlapped region and combining the stroke segments directly connected in the non-overlapped region into a complete skeleton shape stroke" in step S50, there is further provided a step of recovering the original stroke shape, in which the method includes:
and associating the pixels in the input image with the complete skeleton form strokes to obtain Chinese character image strokes which are the original stroke forms of the input image.
On the other hand, the invention provides a Chinese character image stroke extraction system based on a full convolution neural network, which comprises an input module, an area extraction module, a skeletonization module, a stroke judgment module, a skeleton connection module and an output module;
the input module is configured to acquire and input a Chinese character image as an input image;
the region extraction module is configured to extract an overlapped region map of character strokes in the input image; the part of the input image, which is removed from the overlapped area, is a non-overlapped area map;
the skeletonization module is configured to perform skeletonization operation on the overlapped region graph and the non-overlapped region graph to obtain an overlapped region skeleton form stroke segment set and a non-overlapped region skeleton form stroke segment set;
the stroke judgment module is configured to calculate a consistency matrix between any two stroke segments based on the overlapped region skeleton form stroke segment set; two stroke segments with all elements in the consistency matrix being larger than or equal to a preset threshold belong to the same stroke;
the skeleton connecting module is configured to connect the stroke segments belonging to the same stroke in the overlapping region, and combine the stroke segments and the directly connected stroke segments in the non-overlapping region into a complete skeleton shape stroke;
and the output module is configured to output the acquired complete skeleton form strokes.
In a third aspect of the present invention, a storage device is provided, in which a plurality of programs are stored, the programs being suitable for being loaded and executed by a processor to implement the above-mentioned full convolution neural network-based chinese character image stroke extraction method.
In a fourth aspect of the present invention, a processing apparatus is provided, which includes a processor, a storage device; the processor is suitable for executing various programs; the storage device is suitable for storing a plurality of programs; the program is suitable for being loaded and executed by a processor to realize the Chinese character image stroke extraction method based on the full convolution neural network.
The invention has the beneficial effects that:
(1) the invention relates to a Chinese character image stroke extraction method based on a full convolution neural network, which adopts a conditional full convolution neural network to fully describe stroke shapes, positions and spatial relations among strokes in Chinese printing characters without an additional post-processing method.
(2) The invention relates to a Chinese character image stroke extraction method based on a full convolution neural network, which adopts a framework extraction module to extract strokes on a single-pixel-width framework aiming at the problem that the writing style and the stroke width of a Chinese handwritten character are variable, so that the operation not only maintains the structure of the handwritten character, but also can obviously save the calculated amount.
(3) The invention discloses a Chinese character image stroke extraction method based on a full convolution neural network, which detects stroke overlapping areas in a Chinese character image, independently processes the stroke overlapping areas to obtain all stroke segment sets converged in the same overlapping area, analyzes every two stroke segments in the sets, describes the relationship between the two stroke segments by using a stroke consistency matrix between the two stroke segments, judges that the two stroke segments are connected into a stroke, and avoids stroke extraction omission or redundancy of handwritten characters.
(4) The invention discloses a Chinese character image stroke extraction method based on a full convolution neural network, and provides a character image synthesis method for training, which can automatically generate millions of off-line Chinese handwritten character images and attach different labeling information of the off-line Chinese handwritten character images in different tasks, thereby greatly saving the labor cost.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:
FIG. 1 is a schematic flow chart of the Chinese character image stroke extraction method based on the full convolution neural network of the present invention;
FIG. 2 is a schematic structural diagram of a full convolution neural network used by an overlap area extraction network and a skeletonization network according to an embodiment of the method for extracting strokes of Chinese character images based on the full convolution neural network of the present invention;
FIG. 3 is a schematic diagram of training process of training data of a region extraction network and a skeleton extraction network according to an embodiment of the method for extracting strokes of Chinese character images based on a full convolution neural network;
FIG. 4 is a schematic diagram of a handwritten character skeletonization post-processing method according to an embodiment of the method for extracting strokes of Chinese character images based on a full convolution neural network of the present invention;
FIG. 5 is a schematic diagram of a conditional full convolution network training method according to an embodiment of the present invention;
FIG. 6 is an exemplary diagram of stroke extraction based on stroke coherence, according to an embodiment of the method for extracting strokes of Chinese character images based on full convolution neural network of the present invention.
Detailed Description
The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings.
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
The invention discloses a Chinese character image stroke extraction method based on a full convolution neural network, which comprises the following steps:
step S10, acquiring a Chinese character image as an input image;
step S20, extracting an overlapping region map of character strokes in the input image; the part of the input image, which is removed from the overlapped area, is a non-overlapped area map;
step S30, performing skeletonization operation on the overlapped region graph and the non-overlapped region graph to obtain an overlapped region skeleton form stroke segment set and a non-overlapped region skeleton form stroke segment set;
step S40, calculating a consistency matrix between any two stroke segments based on the overlapping region skeleton form stroke segment set; two stroke segments with all elements in the consistency matrix being larger than or equal to a preset threshold belong to the same stroke;
step S50, connecting the stroke segments belonging to the same stroke in the overlapped region, and combining the stroke segments and the directly connected stroke segments in the non-overlapped region into a complete skeleton-shaped stroke.
In order to more clearly describe the method for extracting strokes of a chinese character image based on a full convolution neural network of the present invention, each step in the embodiment of the method of the present invention is expanded and detailed below with reference to fig. 1.
The method for extracting strokes of Chinese character images based on the full convolution neural network comprises the steps of S10-S50, wherein the steps are described in detail as follows:
in step S10, a chinese character image is acquired as an input image.
Character recognition is an important branch of computer pattern recognition and is one of the most difficult problems in the field of recognition. Stroke extraction of chinese characters plays an important role in structural analysis-based character recognition research and related applications. The character recognition is divided into two categories, namely printed character recognition and handwritten character recognition, the printed character recognition has been greatly developed due to the fact that character formats are standard and strokes are clear, the handwritten characters are changed in character shapes and connected in handwriting, and the handwritten characters are difficult to recognize due to the fact that the handwritten characters are deformed and high in similarity among the handwritten characters.
In step S10, "obtain chinese character image as input image", the method is:
acquiring an acquired Chinese character image, removing the background of the acquired Chinese character image through an OTSU method-based global binarization algorithm or a local self-adaptive binarization algorithm to obtain a foreground image of the Chinese character image, and using the foreground image as an input image. The Chinese character image stroke extraction method based on the full convolution neural network can not only extract strokes of handwritten Chinese character images, but also has good effect on extracting strokes of printed Chinese character images.
The purpose of image binarization is to eliminate the influence of the gray value of a stroke pixel point on the extraction of subsequent strokes, a global binarization algorithm based on an OTSU method is generally applied to Chinese character images with uniform illumination, and a local self-adaptive binarization algorithm is generally applied to images with non-uniform illumination. There are many image binarization methods, and a suitable image binarization method can be selected by combining the characteristics of the image, which is not described in detail herein.
Step S20, extracting an overlapping region map of character strokes in the input image; the part of the input image, which is removed from the overlapped area, is a non-overlapped area map.
In the embodiment of the invention, the overlapped area extraction network constructed based on the full convolution neural network is adopted, and after the overlapped area of the input image is extracted, the area except the overlapped area in the input image is the non-overlapped area. The overlap area extraction network consists of two symmetrical parts: a contracted path and an expanded path. The contracted path is used to extract image features, which are corresponding (essential) features or characteristics of one class of objects that are different from other classes of objects, or a collection of these features and characteristics, which are data that can be extracted by measurement or processing. And reversely generating the expansion path based on the acquired features to acquire a character stroke overlapping area diagram.
Step S201, based on the input image, extracting features of the input image through a contracted path of an overlap area extraction network.
Step S202, based on the characteristics of the input image, reversely generating through an expansion path symmetrical to the network contraction path extracted from the overlapped area, and acquiring a character stroke overlapped area diagram; the part of the input image, which is removed from the overlapped area, is a non-overlapped area map.
Wherein the overlap region extraction network is a network constructed based on a full convolution neural network and used for extracting an overlap region map of character strokes in the input image.
Step S30, performing skeletonization operation on the overlapped region map and the non-overlapped region map to obtain an overlapped region skeleton form stroke segment set and a non-overlapped region skeleton form stroke segment set.
In the embodiment of the invention, the skeletonization operation of the character stroke overlapping region graph and the non-overlapping region graph is carried out by adopting a full convolution-based neural network.
Fig. 2 is a schematic diagram of a full convolutional neural network structure used by an overlap area extraction network according to an embodiment of the method for extracting strokes of a chinese character image based on a full convolutional neural network of the present invention, where a picture with characters at the top left in the diagram is an input picture of the network, a picture with two points at the top right in the diagram is an overlap area image output by the network, and the rest blocks represent convolutional calculation units.
The training of the fully convolutional neural network used by the overlap region extraction network and the skeleton extraction network requires millions of handwritten character sample data, each sample marks a stroke overlap region, and it is difficult to acquire such a number of training samples by acquiring handwritten character images and manually marking the samples. The invention synthesizes the character image by using the online handwritten character, because the online handwritten character has stroke track information (stroke coordinate point sequence), the stroke overlapping area is easy to calculate, the burden of manual marking is avoided, and a large amount of training samples can be obtained in a short time. The stroke point sequence of the online handwritten character is taken as a skeleton of the character image, each stroke is endowed with a stroke width, the skeleton is expanded into the stroke with the width, a synthesized character image consistent with the real character image is obtained, and the intersection of pixel points of different strokes is the stroke overlapping area.
The training sample of the skeleton extraction network is obtained by the following method:
step B10, using the stroke coordinate point sequence of the online handwritten character as the skeleton of the composite character image, and setting the stroke width.
Step B20, expanding the skeleton of the synthesized character image into wide strokes according to the set stroke width based on the skeleton of the synthesized character image to obtain a synthesized character image; and the synthesized character image and the corresponding skeleton are training samples of the skeleton extraction network.
The method for extracting the training sample of the network in the overlapped area comprises the following steps:
g10, obtaining a synthetic character image by adopting the method of the steps B10-B20 of the Chinese character image stroke extraction method based on the full convolution neural network;
step G20, calculating a stroke overlapping area corresponding to the synthesized character image based on the stroke coordinate point sequence information of the synthesized character image; and extracting training samples of a network for the overlapped area of the synthesized character image and the corresponding stroke.
Fig. 3 is a schematic diagram of training processes of a region extraction network and a skeleton extraction network training data according to an embodiment of the method for extracting strokes of a chinese character image based on a full convolution neural network of the present invention, wherein a module in a first column is input of a network training sample, a second column represents four groups of convolution units, each convolution unit in a third column, a fourth column, a fifth column and a sixth column is a conventional convolution branch in different scales, and a seventh column is an output result of network training. Because the convolution and pooling operation play a role together, the image size can be reduced by times, so that an up-sampling layer is needed to restore the image size, and the learnable up-sampling layer is adopted to replace bilinear interpolation up-sampling operation in the embodiment of the invention, so that the model can restore more image details. In the final multi-scale feature fusion stage, the convolution operation is used for directly fusing the final predicted image to obtain the final output image, and the operation can fully utilize more local information in a larger receptive field to deduce whether the central point of the current receptive field is judged to be a skeleton point.
In step S30, after the "overlapping region skeleton form stroke segment set and non-overlapping region skeleton form stroke segment set" are obtained, there is also provided an optimization step of skeleton form strokes, and the method includes:
calculating the gravity center of each overlapped region in the overlapped region graph, acquiring all adjacent skeleton points of the region corresponding to the gravity center, and connecting the gravity center of the overlapped region and the adjacent skeleton points one by one in the skeleton graph to obtain an optimized overlapped region skeleton form stroke segment set;
and recalling the skeleton pixel points by a clustering method based on the non-overlapping area to obtain an optimized skeleton form stroke segment set of the non-overlapping area.
As shown in fig. 4, which is a schematic diagram of a handwritten character skeletonization post-processing method according to an embodiment of the method for extracting strokes of a chinese character image based on a full convolution neural network of the present invention, K-means is a K-means clustering method, bwmorph _ thin @ matlab is a traditional thinning algorithm, and sigmoid represents a sigmoid function, which is also referred to as an S-type growth curve. Through K-means clustering, pixel points in the image are roughly divided into three categories of skeleton points, non-skeleton points and transition points between the skeleton points and the non-skeleton points. Through this operation, most skeleton points can be recalled, but a small number of non-skeleton redundant points are inevitably brought, for which, in one example of the present invention, these redundant points are deleted using a simple rule in a conventional Thinning algorithm, and a line in the skeleton map is guaranteed to be a single-pixel width. Then, for an overlapped region framed by a rectangle in the figure, the gravity center of the region is used to represent the region (i.e. the starting point of four arrows in the figure), and the region and the skeleton point generate four adjacent points (i.e. the end points of four arrows in the figure), so that an ideal skeletonization result can be obtained for the region by connecting the gravity center point and the four adjacent points in the skeleton diagram.
Step S40, calculating a consistency matrix between any two stroke segments based on the overlapping region skeleton form stroke segment set; and two stroke segments with all elements in the consistency matrix being larger than or equal to a preset threshold belong to the same stroke.
In the embodiment of the invention, a conditional full convolution neural network is adopted to calculate the consistency matrix between any two stroke segments. The conditional full convolution network is constructed based on a full convolution neural network and is trained by adopting a base point guided conditional method:
the input of the network is in a double-channel mode, the output is in a single-channel mode, the first channel of the input is a hand-written character image, the second channel is a mask of a base point, the mask keeps the same size with the character image, the value of the mask is 1 only at the coordinate of the base point of the search pixel, and the value of the other positions is 0. The mask acts as a conditional input to the full convolution network, the output of which is a complete stroke containing the base points in the mask. In the model of the invention, the number of VDSR units depends on two variables of the size of the convolution kernel and the size of the image, i.e. the VDSR units need to be stacked layer by layer until the receptive field of the model completely covers the whole picture.
As shown in fig. 5, which is a schematic diagram of a conditional full convolution network training method according to an embodiment of the method for extracting strokes of chinese character images based on a full convolution neural network of the present invention, the left side is a dual channel input of the network, the left side is a mask of handwritten character images and base points, the right side is a single channel output of the network, the left side is a complete stroke including the base points in the mask, the middle VDSR UNITS is a standard convolution unit, conv represents a convolution layer, ReLU represents an activation layer, and BatchNorm represents a batch normalization layer.
Step S401, selecting any two stroke segments in the skeleton form stroke segment set of the overlapped area, and respectively recording the two stroke segments as S1、S2。
Step S402, in the stroke segment S1、S2Uniformly selecting N points, and respectively recording the points as a set
Step S403, calculating the set by adopting a conditional full convolution network
And
obtaining NxN probabilities of any two points belonging to the same stroke to form a stroke segment S1、S2A coherence matrix between.
Calculating the probability that any two points in the two sets belong to the same stroke by the following method (1):
wherein p isnnIs composed of
And
probability of belonging to the same stroke, fvdsr() Representing a conditional full convolution network.
S1、S2The coherence matrix between them is shown in formula (2):
in one embodiment of the present invention, if all the values of the setting matrix are greater than or equal to the predetermined threshold value 0.5, the stroke segment S1And S2Belonging to the same stroke.
As shown in fig. 6, an exemplary graph of stroke extraction based on stroke coherence is shown as an embodiment of the method for extracting strokes of a chinese character image based on a full convolution neural network of the present invention, where the top left corner is a handwritten chinese character sample example, the top right corner is a skeleton diagram of the handwritten chinese character sample example, the points in the bottom left corner are selected point examples in the stroke segment in the overlap region, and the bottom right corner is the result output of the stroke extraction of the handwritten chinese character sample.
Step S50, connecting the stroke segments belonging to the same stroke in the overlapped region, and combining the stroke segments and the directly connected stroke segments in the non-overlapped region into a complete skeleton-shaped stroke.
In step S50, after "connecting the stroke segments belonging to the same stroke in the overlapped region and combining the stroke segments directly connected in the non-overlapped region into a complete skeleton form stroke", there is also provided a step of recovering the original stroke form, the method is:
and associating the pixels in the input image with the complete skeleton form strokes to obtain Chinese character image strokes which are the original stroke forms of the input image.
The system for extracting strokes of Chinese character images based on the full convolution neural network comprises an input module, an area extraction module, a skeletonization module, a stroke judgment module, a skeleton connection module and an output module, wherein the input module is used for inputting the strokes of Chinese characters;
the input module is configured to acquire and input a Chinese character image as an input image;
the region extraction module is configured to extract an overlapped region map of character strokes in the input image; the part of the input image, which is removed from the overlapped area, is a non-overlapped area map;
the skeletonization module is configured to perform skeletonization operation on the overlapped region graph and the non-overlapped region graph to obtain an overlapped region skeleton form stroke segment set and a non-overlapped region skeleton form stroke segment set;
the stroke judgment module is configured to calculate a consistency matrix between any two stroke segments based on the overlapped region skeleton form stroke segment set; two stroke segments with all elements in the consistency matrix being larger than or equal to a preset threshold belong to the same stroke;
the skeleton connecting module is configured to connect the stroke segments belonging to the same stroke in the overlapping region, and combine the stroke segments and the directly connected stroke segments in the non-overlapping region into a complete skeleton shape stroke;
and the output module is configured to output the acquired complete skeleton form strokes.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process and related description of the system described above may refer to the corresponding process in the foregoing method embodiments, and will not be described herein again.
It should be noted that, the system for extracting strokes of a chinese character image based on a full convolution neural network provided in the foregoing embodiment is only illustrated by the division of the functional modules, and in practical applications, the functions may be allocated to different functional modules according to needs, that is, the modules or steps in the embodiment of the present invention are further decomposed or combined, for example, the modules in the foregoing embodiment may be combined into one module, or may be further split into a plurality of sub-modules, so as to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the modules or steps, and are not to be construed as unduly limiting the present invention.
A storage device according to a third embodiment of the present invention stores therein a plurality of programs adapted to be loaded and executed by a processor to implement the above-described full convolutional neural network-based chinese character image stroke extraction method.
A processing apparatus according to a fourth embodiment of the present invention includes a processor, a storage device; a processor adapted to execute various programs; a storage device adapted to store a plurality of programs; the program is suitable for being loaded and executed by a processor to realize the Chinese character image stroke extraction method based on the full convolution neural network.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes and related descriptions of the storage device and the processing device described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
Those of skill in the art would appreciate that the various illustrative modules, method steps, and modules described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that programs corresponding to the software modules, method steps may be located in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. To clearly illustrate this interchangeability of electronic hardware and software, various illustrative components and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as electronic hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The terms "comprises," "comprising," or any other similar term are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
So far, the technical solutions of the present invention have been described in connection with the preferred embodiments shown in the drawings, but it is easily understood by those skilled in the art that the scope of the present invention is obviously not limited to these specific embodiments. Equivalent changes or substitutions of related technical features can be made by those skilled in the art without departing from the principle of the invention, and the technical scheme after the changes or substitutions can fall into the protection scope of the invention.
Claims (10)
1. A Chinese character image stroke extraction method based on a full convolution neural network is characterized by comprising the following steps:
step S10, acquiring a Chinese character image as an input image;
step S20, extracting an overlapping region map of character strokes in the input image; the part of the input image, which is removed from the overlapped area, is a non-overlapped area map;
step S30, performing skeletonization operation on the overlapped region graph and the non-overlapped region graph to obtain an overlapped region skeleton form stroke segment set and a non-overlapped region skeleton form stroke segment set;
step S40, calculating a consistency matrix between any two stroke segments based on the overlapping region skeleton form stroke segment set; two stroke segments with all elements in the consistency matrix being larger than or equal to a preset threshold belong to the same stroke;
step S50, connecting the stroke segments belonging to the same stroke in the overlapped region, and combining the stroke segments and the directly connected stroke segments in the non-overlapped region into a complete skeleton-shaped stroke.
2. The method for extracting strokes of Chinese character image based on full convolution neural network as claimed in claim 1, wherein in step S10 "obtaining Chinese character image as input image" comprises:
acquiring an acquired Chinese character image, removing the background of the acquired Chinese character image through an OTSU method-based global binarization algorithm or a local self-adaptive binarization algorithm to obtain a foreground image of the Chinese character image, and using the foreground image as an input image.
3. The method for extracting strokes of Chinese character image based on full convolution neural network as claimed in claim 1, wherein in step S20 "extracting the overlapped region map of character strokes in the input image; the part of the input image, which is removed from the overlapped area, is a non-overlapped area map, and the method comprises the following steps:
step S201, extracting the characteristics of the input image through an overlapped area extraction network contraction path based on the input image;
step S202, based on the characteristics of the input image, reversely generating an expansion path symmetrical to the network contraction path extracted from the overlapping area to obtain an overlapping area image; the part of the input image, which is removed from the overlapped area, is a non-overlapped area map;
wherein the overlap region extraction network is a network constructed based on a full convolution neural network and used for extracting an overlap region map of character strokes in the input image.
4. The method for extracting strokes of Chinese character image based on full convolution neural network as claimed in claim 1, wherein in step S40 "calculate the consistency matrix between any two stroke segments", the method comprises:
step S401, selecting any two stroke segments in the skeleton form stroke segment set of the overlapped area, and respectively recording the two stroke segments as
；
Step S402, in the stroke segment
Uniformly selecting N points, and respectively recording the points as a set
Step S403, calculating the set by adopting a conditional full convolution network
And
obtaining NxN probabilities of any two points belonging to the same stroke to form a stroke segment
A coherence matrix between.
5. The method for extracting strokes of Chinese character images based on full convolution neural network as claimed in claim 3, wherein in step S30, "skeletonizing the overlapped region map and the non-overlapped region map" is performed by:
performing skeletonization operation on the overlapped region graph and the non-overlapped region graph through a skeleton extraction network; the method for acquiring the training sample of the skeleton extraction network comprises the following steps:
step B10, using the stroke coordinate point sequence of the online handwritten character as the skeleton of the synthesized character image, and setting the stroke width;
step B20, expanding the skeleton of the synthesized character image into wide strokes according to the set stroke width based on the skeleton of the synthesized character image to obtain a synthesized character image; and the synthesized character image and the corresponding skeleton are training samples of the skeleton extraction network.
6. The method for extracting strokes of Chinese character images based on full convolution neural network as claimed in claim 5, wherein the method for extracting network of overlapped area comprises:
calculating a stroke overlapping area corresponding to the synthesized character image based on the stroke coordinate point sequence information of the synthesized character image; and extracting training samples of a network for the overlapped area of the synthesized character image and the corresponding stroke.
7. The method for extracting strokes of Chinese character images based on full convolution neural network as claimed in claim 1, wherein in step S30, after obtaining the overlapped region skeleton form stroke segment set and the non-overlapped region skeleton form stroke segment set, there is further provided an optimization step of skeleton form stroke segments, and the method is as follows:
calculating the gravity center of each overlapped region in the character stroke overlapped region, acquiring all adjacent skeleton points of the region corresponding to the gravity center, and connecting the gravity center of the overlapped region and the adjacent skeleton points one by one in a skeleton diagram to obtain an optimized overlapped region skeleton form stroke segment set;
and recalling skeleton pixel points by a clustering method based on the character stroke non-overlapping area to obtain optimized skeleton form strokes of the non-overlapping area.
8. The method for extracting strokes of Chinese character image based on full convolution neural network as claimed in any of claims 1-7, wherein there is further provided a step of original stroke shape recovery after "connecting the stroke segments belonging to the same stroke in the overlapped region and combining the stroke segments and the directly connected stroke segments in the non-overlapped region into a complete skeleton shape stroke" in step S50, and the method is as follows:
and associating the pixels in the acquired character image with the complete skeleton form strokes to obtain Chinese character image strokes which are the original stroke forms of the input image.
9. A Chinese character image stroke extraction system based on a full convolution neural network is characterized by comprising an input module, an area extraction module, a skeletonization module, a stroke judgment module, a skeleton connection module and an output module;
the input module is configured to acquire and input a Chinese character image as an input image;
the region extraction module is configured to extract an overlapped region map of character strokes in the input image; the part of the input image, which is removed from the overlapped area, is a non-overlapped area map;
the skeletonization module is configured to perform skeletonization operation on the character stroke overlapping region graph and the non-overlapping region graph to obtain an overlapping region skeleton form stroke segment set and a non-overlapping region skeleton form stroke segment set;
the stroke judgment module is configured to calculate a consistency matrix between any two stroke segments based on the overlapped region skeleton form stroke segment set; two stroke segments with all elements in the consistency matrix being larger than or equal to a preset threshold belong to the same stroke;
the skeleton connecting module is configured to connect the stroke segments belonging to the same stroke in the overlapping region, and combine the stroke segments and the directly connected stroke segments in the non-overlapping region into a complete skeleton shape stroke;
and the output module is configured to output the acquired complete skeleton form strokes.
10. A treatment apparatus comprises
A processor adapted to execute various programs; and
a storage device adapted to store a plurality of programs;
wherein the program is adapted to be loaded and executed by a processor to perform:
the method for extracting strokes of Chinese character image based on full convolution neural network as claimed in any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910454930.7A CN110232337B (en) | 2019-05-29 | 2019-05-29 | Chinese character image stroke extraction method and system based on full convolution neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910454930.7A CN110232337B (en) | 2019-05-29 | 2019-05-29 | Chinese character image stroke extraction method and system based on full convolution neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110232337A CN110232337A (en) | 2019-09-13 |
| CN110232337B true CN110232337B (en) | 2021-02-02 |
Family
ID=67858809
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910454930.7A Active CN110232337B (en) | 2019-05-29 | 2019-05-29 | Chinese character image stroke extraction method and system based on full convolution neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110232337B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110660019A (en) * | 2019-09-29 | 2020-01-07 | 华北电力大学 | Small data set simplified stroke generation method based on BPL |
| CN110969681B (en) * | 2019-11-29 | 2023-08-29 | 山东浪潮科学研究院有限公司 | Handwriting word generation method based on GAN network |
| CN112862025A (en) * | 2021-03-08 | 2021-05-28 | 成都字嗅科技有限公司 | Chinese character stroke filling method, system, terminal and medium based on computer |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1271140A (en) * | 1999-04-21 | 2000-10-25 | 中国科学院自动化研究所 | Handwriting identifying method based on grain analysis |
| CN101853126A (en) * | 2010-05-12 | 2010-10-06 | 中国科学院自动化研究所 | Real-time identification method for on-line handwriting sentences |
| CN105205448A (en) * | 2015-08-11 | 2015-12-30 | 中国科学院自动化研究所 | Character recognition model training method based on deep learning and recognition method thereof |
| CN108345833A (en) * | 2018-01-11 | 2018-07-31 | 深圳中兴网信科技有限公司 | The recognition methods of mathematical formulae and system and computer equipment |
| CN106446896B (en) * | 2015-08-04 | 2020-02-18 | 阿里巴巴集团控股有限公司 | Character segmentation method and device and electronic equipment |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9495620B2 (en) * | 2013-06-09 | 2016-11-15 | Apple Inc. | Multi-script handwriting recognition using a universal recognizer |
| CN108229397B (en) * | 2018-01-04 | 2020-08-18 | 华南理工大学 | Method for detecting text in image based on Faster R-CNN |
| CN108345850B (en) * | 2018-01-23 | 2021-06-01 | 哈尔滨工业大学 | Scene text detection method based on region classification of stroke feature transformation and deep learning of superpixel |
-
2019
- 2019-05-29 CN CN201910454930.7A patent/CN110232337B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1271140A (en) * | 1999-04-21 | 2000-10-25 | 中国科学院自动化研究所 | Handwriting identifying method based on grain analysis |
| CN101853126A (en) * | 2010-05-12 | 2010-10-06 | 中国科学院自动化研究所 | Real-time identification method for on-line handwriting sentences |
| CN106446896B (en) * | 2015-08-04 | 2020-02-18 | 阿里巴巴集团控股有限公司 | Character segmentation method and device and electronic equipment |
| CN105205448A (en) * | 2015-08-11 | 2015-12-30 | 中国科学院自动化研究所 | Character recognition model training method based on deep learning and recognition method thereof |
| CN108345833A (en) * | 2018-01-11 | 2018-07-31 | 深圳中兴网信科技有限公司 | The recognition methods of mathematical formulae and system and computer equipment |
Non-Patent Citations (2)
| Title |
|---|
| DeepAD: A Deep Learning Based Approach to Stroke-Level Abnormality Detection in Handwritten Chinese Character Recognition;Tie-Qiang Wang 等;《2018 IEEE International Conference on Data Mining (ICDM)》;20181231;1302-1307 * |
| Stroke Sequence-Dependent Deep Convolutional Neural;Baotian Hu 等;《arXiv:1610.04057v1》;20161213;1-31 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110232337A (en) | 2019-09-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108280397B (en) | Human body image hair detection method based on deep convolutional neural network | |
| CN110232337B (en) | Chinese character image stroke extraction method and system based on full convolution neural network | |
| WO2014205231A1 (en) | Deep learning framework for generic object detection | |
| Hobson et al. | HEp-2 staining pattern recognition at cell and specimen levels: datasets, algorithms and results | |
| Jiang et al. | DCT–CNN-based classification method for the Gongbi and Xieyi techniques of Chinese ink-wash paintings | |
| Yang et al. | Handwriting text recognition based on faster R-CNN | |
| CN110443235B (en) | Intelligent paper test paper total score identification method and system | |
| Kölsch et al. | Recognizing challenging handwritten annotations with fully convolutional networks | |
| CN109710788A (en) | Image pattern mark and management method and equipment | |
| CN113158895A (en) | Bill identification method and device, electronic equipment and storage medium | |
| Chen et al. | Page segmentation for historical handwritten document images using conditional random fields | |
| Dhali et al. | Binet: Degraded-manuscript binarization in diverse document textures and layouts using deep encoder-decoder networks | |
| CN115880704A (en) | Automatic case cataloging method, system, equipment and storage medium | |
| Suresh et al. | Telugu Optical Character Recognition Using Deep Learning | |
| Guo et al. | Decoupling semantic and edge representations for building footprint extraction from remote sensing images | |
| CN111414913B (en) | Character recognition method, recognition device and electronic equipment | |
| EP3686841B1 (en) | Image segmentation method and device | |
| Ke et al. | SRN: Side-output residual network for object reflection symmetry detection and beyond | |
| CN112801238B (en) | Image classification method and device, electronic equipment and storage medium | |
| CN114241486A (en) | Method for improving accuracy rate of identifying student information of test paper | |
| Khan et al. | Isolated handwritten pashto characters recognition using KNN classifier | |
| CN108334884B (en) | Handwritten document retrieval method based on machine learning | |
| CN111612045A (en) | Universal method for acquiring target detection data set | |
| Lai et al. | Pattern Recognition and Computer Vision: First Chinese Conference, PRCV 2018, Guangzhou, China, November 23-26, 2018, Proceedings, Part III | |
| Pal et al. | Handwritten Character Recognition |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |