LU502435B1 - Handwriting recognition method of digital writing by neurodegenerative patients - Google Patents

Abstract

This invention provides handwriting recognition method of digital writing by neurodegenerative patients, which comprises the following steps: asking users write with dot matrix pen to obtain written dynamic data information; preprocessing the obtained dynamic data information to generate a dynamic enhanced static image; extracting features; inputting the extracted features into a pre-trained handwriting recognition model; judging whether it is the handwriting of a neurodegenerative patient, if so, further judging the handwriting of the neurodegenerative patient; the analysis result is fed back to the user or the doctor. The data acquisition of the invention restores the usual writing state of paper and pen, conforms to the daily writing habit, and enables the patient's information to be collected in the daily writing state. Moreover, digital writing technology is more accurate for handwriting feature recognition.

Description

DESCRIPTION Handwriting recognition method of digital writing by neurodegenerative patients

TECHNICAL FIELD The invention belongs to the technical field of intelligence, particularly relates to handwriting recognition method of digital writing by neurodegenerative patients.

BACKGROUND Neurodegenerative diseases (NDs) affect millions of people all over the world. Neurodegenerative diseases include Alzheimer's disease (AD), Parkinson's disease (PD), Huntington's disease (HD), amyotrophic lateral sclerosis (ALS), etc, which are fatal and complex diseases with progressive development. It is expected that the incidence of these diseases will increase greatly in the next few decades. Unfortunately, neurodegenerative diseases have hidden onset, long cycle and few effective treatments, which will seriously affect the exercise ability and quality of life of patients. At present, there is no effective method to cure or reverse the disease's progress. Although the medical profession has made great progress in diagnosis, it is still difficult to find this disease at an early stage, because some common observable warning signals evolve silently and only appear 5 to 10 years after the onset of the disease. When obvious symptoms of the disease appear, the course of the disease often reaches the middle and late stages.

Up to now, the clinical diagnosis of such diseases is made by doctors, mainly using the corresponding medical scales (such as Alzheimer's Disease Rating Scale and Parkinson's Disease Rating Scale), which can be supported by imaging (such as magnetic resonance imaging), blood test and lumbar puncture (spinal puncture). The method, which has less harm to the body, low cost, flexible use, and is convenient for both early screening and tracking, feeding back patients’ condition after diagnosis, has not yet appeared.

Writing is a refined movement, It is also one of the abilities of NDs influence. Previous studies have shown that the handwriting function of neurodegenerative patients will deteriorate, which is reflected in spatial variability (accuracy, fluency of handwriting, etc.) and temporal variability (duration of exercise, acceleration time, etc.).

Nowadays, digital writing is more common, such as using dot matrix pen, tablet computer electromagnetic pen, etc, to get the user's handwriting, next, by extracting the handwriting characteristics of neurodegenerative patients, and classifying them, the handwriting of neurodegenerative patients can be identified (judging what kind of neurodegenerative diseases and the handwriting of patients with what degree of illness), which can assist in early diagnosis and timely intervention, and also help to track and feedback the characteristics of patients’ course of disease, and facilitate patients’ lives.

The technical scheme of the prior art 1 The existing disease identification method is mainly determined by the rating scale, whose results largely depend on doctors' explanations and experiences, and the method can be aided by imaging (such as magnetic resonance imaging), blood test and lumbar puncture (spinal puncture).

Disadvantages of prior art 1 The scoring system of the scale is based on the qualitative description of symptoms, corresponding to some corresponding scores, it is up to the doctor scores the patient's condition, thus It has certain subjectivity, moreover, the patient may have some resistance, and it is difficult to effectively track and feed back the course of the disease. And the use of imaging is expensive, and has a certain degree of impact on the body.

The technical solution of the prior art 2 An existing hand motor function analysis device based on handwritten character input belongs to the field of computer application and medical rehabilitation. The technical solution uses a pressure touch screen and an acceleration sensor to collect the hand motor function input signal of a user during daily mobile phone operation, and it inputs the signals into a pathological data model for hand motor function evaluation, and the pathological features are analyzed in real time to evaluate the user's hand motor function, which is then used as a diagnostic basis for nervous system diseases such as Parkinson's disease.

1. As the collection, capture and processing of written characters in the prior art need some tools, such as touch screens such as tablet computers, the cost of these tools is relatively unaffordable in remote areas and underdeveloped areas. The existing technology has high cost, which is not conducive to the classification of healthy individuals and sick people.

2. Using the touch screen to collect data does not conform to the user's writing habits, and can't restore the original user's real writing process.

SUMMARY The purpose of the present invention is to solve the above-mentioned defects in the prior art, and provide handwriting recognition method of digital writing by neurodegenerative patients.

This invention provides the following technical solutions: The pre-training and modeling method of the handwriting recognition and classification model comprises the following steps: S1, acquiring dynamic data information written by a user with a dot matrix pen; S2, preprocessing the obtained data information to generate a dynamically enhanced static image; S3, extracting features; S4, adding the extracted feature vectors and the corresponding handwriting category labels into the labeling sample set, wherein the handwriting category labels include what kind of neurodegenerative disease patients or non-neurodegenerative disease patients are, and the handwriting category labels used for pre-training handwriting recognition models are obtained by pre-diagnosis by doctors; S1-S4 are repeated, and samples of a plurality of users are added to the labeled sample set to obtain a sample database;

S5, dividing the labeled sample set into a training sample set and a testing sample set, training the model on the training sample set, and carrying out a model test on the testing sample set, wherein the test recognition accuracy reaches an applicable level, thus obtaining an available handwriting recognition and classification model.

According to the handwriting recognition training database, a convolutional neural network (CNN) is adopted for model training to obtain the handwriting recognition model.

Convolutional neural network is mainly composed of data input layer, convolution calculation layer, activation layer, pooling layer and full connection layer. In the data input layer, the original image is preprocessed, including de-averaging, normalization, etc. Selecting filter size in convolution calculation layer; In the active layer, the output result of convolution layer is mapped nonlinearly, the pooling layer is sandwiched between successive convolution layers , which is used to compress the amount of data and parameters and reduce over-fitting. The whole connection layer is used at the tail of convolutional neural network. Finally, outputting the last N-dimensional vector.

Handwriting recognition method of digital writing by neurodegenerative patients comprises the following steps: S101, letting user write with a dot matrix pen to obtain written dynamic data information: S102, preprocessing the obtained dynamic data information to generate a dynamic enhanced static image; S103, extracting features; S104, inputting the extracted features into a pre-trained handwriting recognition model; S105, judging whether it is the handwriting of a neurodegenerative patient, and if so, further judging what kind of handwriting of a neurodegenerative patient it is: S106. the analysis result is fed back to the user or the doctor.

> LU502435 Further, in S101, letting users use dot matrix pen to write on dot matrix paper. Paper dot matrix consists of some very small dots arranged according to special algorithm rules; the function of dot matrix is to provide a coordinate parameter information for the digital pen, so as to ensure that the digital pen can accurately record the written handwriting when writing on digital paper; according to the characteristics of dot matrix digital pen, when the nib is pressed down, the pressure sensor is triggered, and the built-in high-speed camera is started to take pictures of the dot matrix passed by the nib at a speed of hundreds of times per second, so as to obtain the real-time dynamic data information written by the user.

Further, the dynamic data information refers to the x,y coordinates of each point where the pen tip passes when the user writes with a dot matrix pen, the corresponding time stamp, the pressure value on the paper, the state of the pen and the number of strokes.

Further, the step S102 is specifically to preprocess the acquired dynamic information to further obtain the dynamic information in each coordinate, including the pressure, velocity, acceleration, etc. of each coordinate point, and its corresponding variance, standard deviation, maximum value, minimum value, etc, and use that mean to fill in the vacancy and outliers in the data. Z-score normalization (normalization) of data based on the mean and standard deviation of original data is used to eliminate the influence of different attributes of samples with different magnitudes.

Further, the preprocessed dynamic data information is reconstructed, and the points with dynamic data information are drawn in Cartesian coordinate system. The moving time in the air is different from the actual points on the paper, so as to generate a gray image, and the image is modified and improved according to the dynamic data information such as pressure, speed and acceleration.

Further, S103 also includes feature extraction by using the preprocessed convolutional neural network; before inputting the data into the convolutional neural network model, it is necessary to adjust the image size to the standard specification, which is typically 150*150 pixels, and input it to select a plurality of features with better final effect, thus obtaining a combined feature vector with a plurality of features.

Further, feature extraction means that in machine learning, pattern recognition and image processing, feature extraction starts with an initial set of measurement data, and establishes non-redundant derived values, namely features, in order to promote subsequent learning and generalization steps.

The beneficial effects of this invention is described as followings:

1. The data acquisition of the invention restores the usual paper-and-pencil writing state, which conforms to the daily writing habit, so that the patient's information can be collected in the daily writing state.

2. the invention is more effective to identify whether it is the handwriting of Alzheimer's patients according to the different handwriting characteristics of each patient (for example, the pressure of pen and the time spent in the air) by analyzing the handwriting of different neurodegenerative patients, the handwriting of Parkinson's patients can be well identified by measuring the total time and the jitter in writing. The handwriting of Huntington's patients can be identified through long strokes, larger and larger writing area, slow writing, etc. It is convenient to identify the handwriting of neurodegenerative patients through classification, thus facilitating the tracking and feedback of the course of disease.

BRIEF DESCRIPTION OF THE FIGURES Fig. 1 is a flow diagram of the present invention.

DETAILED DESCRIPTION OF THE INVENTION In order to make the purpose, technical scheme and advantages of the present invention clearer, the following technical scheme of the present invention is clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of them. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in the field without creative labor belong to the scope of the present invention.

According to the invention, firstly, the dot matrix pen and dot matrix paper are used to obtain the dynamic handwriting of the user, and the dot matrix pen is used to collect data, so that the collection cost is greatly reduced and the burden of the user is lightened.

Each sample point used has certain dynamic information, and the time and speed information are retained in the image, which together form the required image, making the image more distinctive.

Feature extraction is carried out, and finally, the selected features are input into the pre-trained handwriting recognition model to obtain the handwriting category of the user, which further improves the accuracy of judging the handwriting category of the user.

Compared with the existing technical scheme of assisting in identifying the disease stage, the invention is easier to measure remotely at home, handwriting information is not only easy to obtain, but also handwriting changes in different stages of illness are different.

Before the invention is used, sample modeling must be carried out.

Pre-training the handwriting recognition model, and the modeling method comprises the following steps: S1, acquiring dynamic data information written by a user with a dot matrix pen; S2, preprocessing the obtained data information to generate a dynamically enhanced static image; S3, extracting feature types; S4, adding the extracted feature vector and the corresponding handwriting category label into the labeling sample set, wherein the handwriting category label includes the kind of neurodegenerative disease patients or non-neurodegenerative disease patients, and these handwriting category labels used for handwriting recognition model pre-training are obtained by a doctor's pre-diagnosis, or the handwriting category labels are set according to a specific diagnosis scale for judging neurodegenerative patients.

For example, to obtain the handwriting category labels of patients with Alzheimer's disease, you should first obtain the standardized data of the user and the user's

© LU502435 corresponding current Mini-Mental State Examination (MMSE) score. The score ranges from O to 30 points, and the normal and abnormal decomposition values are related to the level of education; Education years> 6 years) group < 24 points. Below the cut-off value is cognitive deficit, and above is normal. If the score of users with secondary school education or above is less than or equal to 24 points, it means that the handwriting category label of the user should be Alzheimer's disease.

The handwriting categories of the invention are normal handwriting and handwriting of Alzheimer's patients. By analogy, a large number of training samples and corresponding handwriting category labels are obtained to form a patient handwriting recognition training database.

S1-S4 are repeated, and the samples of multiple users are added to the labeled sample set to obtain a sample database.

S5, dividing the labeled sample set into a training sample set and a testing sample set, training the model on the training sample set, and carrying out a model test on the testing sample set, wherein the test recognition accuracy reaches an applicable level, thus obtaining an available handwriting recognition and classification model.

According to the handwriting recognition training database, a convolutional neural network (CNN) is adopted for model training to obtain the handwriting recognition model; in the present invention, the types of handwriting need to be identified and classified, so it can be processed by CNN.

Convolutional neural network is mainly composed of data input layer, convolution calculation layer, activation layer, pooling layer and full connection layer. In the data input layer, the original image is preprocessed, including de-averaging, normalization, etc. Selecting filter size in convolution calculation layer; In the active layer, the output result of convolution layer is mapped nonlinearly, the pooling layer is sandwiched between successive convolution layers , which is used to compress the amount of data and parameters and reduce over-fitting. The whole connection layer is used at the tail of convolutional neural network. Finally, outputting the last N-dimensional vector.

As Figure 1 shown, this invention provides handwriting recognition method of digital writing by neurodegenerative patients, which comprises the following steps:

’ LU502435 S101, a user writes with a dot matrix pen to obtain written dynamic data information: For example, let users with Alzheimer's disease use dot matrix pen to write on dot matrix paper (the function of dot matrix is to provide a coordinate parameter information for digital pen, and ensure that the digital pen can accurately record the written handwriting when writing on digital paper). After processing by the processor, the acquired dynamic information (x,y coordinates of each point where the pen tip passes, the corresponding time stamp, the pressure value applied on the paper, the state of the pen and the number of strokes) is transmitted to the terminal, and the written handwriting is converted into a digital format that can be recognized by the computer, and the information can be simply input to the processing terminal by writing in daily life.

S102, preprocessing the obtained dynamic information to generate a dynamic enhanced static image; For example, the acquired writing dynamic information of Alzheimer's users is preprocessed to further obtain the dynamic information in each coordinate; dynamic information includes the pressure, velocity and acceleration of each coordinate point, and its corresponding variance, standard deviation, maximum value and minimum value, and use that mean to fill in the vacancy and outliers in the data. Z-score normalization (normalization) of data based on the mean and standard deviation of original data is used to eliminate the influence of different attributes of samples with different magnitudes.

The preprocessed dynamic information is reconstructed, and points with dynamic information are drawn in Cartesian coordinate system by normalizing the coordinates of the pen. The time of moving in the air and the actual point moving on the paper adopt different colors, so as to generate a grayscale image. According to the dynamic information such as pressure, speed and acceleration, the image is modified and improved. For example, the improved handwriting image shows that the color display is darker where the pressure increases, the dots are sparse where the writing speed is fast, and the dots are denser where the acceleration is slow.

S103, extracting features; using the formula of preprocessed Convolution Neural Networks for feature extraction, before inputting the data into the convolution neural network model, it is necessary to adjust the image size to a standard specification, such as 150*150 pixels, and then input it, and select a number of features with better final effect to obtain a combined feature vector with multiple features.

S104, inputting the extracted features into a pre-trained handwriting recognition model; S105, judging whether it is the handwriting of a neurodegenerative patient, if so, further judging what kind of neurodegenerative patient's handwriting is, because the handwriting changes in different stages of illness are different, so it can even be further judged what kind of diseased handwriting is; for example, the handwriting of patients with middle Alzheimer's disease stays in the air longer and the stroke length is longer than that of patients with early Alzheimer's disease.

S106, the analysis result is fed back to the user or the doctor.

Finally, it should be explained that the above embodiments are only used to illustrate the technical scheme of the present invention, but not to limit it; Although the invention has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that it is still possible to modify the technical solutions described in the foregoing embodiments, or to equivalently replace some technical features thereof; These modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of various embodiments of the present invention.

Claims (6)

1. Handwriting recognition method of digital writing by neurodegenerative patients is characterized in comprising the following steps: S101, letting user write with a dot matrix pen to obtain written dynamic data information: S102, preprocessing the obtained dynamic data information to generate a dynamic enhanced static image; S103, extracting features; S104, inputting the extracted features into a pre-trained handwriting recognition model; S105, judging whether it is the handwriting of a neurodegenerative patient, and if so, further judging what kind of handwriting of a neurodegenerative patient it is: S106; the analysis result is fed back to the user or the doctor.

2. Handwriting recognition method of digital writing by neurodegenerative patients, according to claim 1, is characterized in that it also includes pre-training of handwriting recognition model before recognition, which includes the following steps: S1, acquiring dynamic data information written by a user with a dot matrix pen; S2, preprocessing the obtained data information to generate a dynamically enhanced static image; S3, extracting features; S4, adding the extracted feature vectors and the corresponding handwriting category labels into the labeling sample set, wherein the handwriting category labels include what kind of neurodegenerative disease patients or non-neurodegenerative disease patients are, and the handwriting category labels used for pre-training handwriting recognition models are obtained by pre-diagnosis by doctors; S1-S4 are repeated, and samples of a plurality of users are added to the labeled sample set to obtain a sample database; S5, dividing the labeled sample set into a training sample set and a testing sample set, training the model on the training sample set, and carrying out a model test on the testing sample set, wherein the test recognition accuracy reaches an applicable level, thus obtaining an available handwriting recognition and classification model.

3. Handwriting recognition method of digital writing by neurodegenerative patients, according to claim 1, is characterized in that in S101, the dynamic data information refers to the x,y coordinates of each point where the pen tip passes when the user writes with a dot matrix pen, the corresponding time stamp, the pressure value on the paper, the state of the pen and the number of strokes.

4. Handwriting recognition method of digital writing by neurodegenerative patients, according to claim 1, is characterized in that the dynamic information obtained in S102 is preprocessed to further obtain the dynamic information in each coordinate, including the pressure, velocity and acceleration of each coordinate point, and its corresponding variance, standard deviation, maximum value and minimum value; the vacancy value and abnormal value in the data are filled by the mean value, and the data is standardized by the mean value and standard deviation based on the original data, so as to eliminate the influence of different attributes of samples with different magnitudes.

5. Handwriting recognition method of digital writing by neurodegenerative patients, according to claim 1, is characterized in that S102 also includes image reconstruction of pre-processed dynamic data information, drawing points with dynamic data information in Cartesian coordinate system, adopting different colors between the time of moving in the air and the actual points moving on paper, so as to generate a gray image, and modifying and improving the image according to the dynamic data information of pressure, speed and acceleration.

6. Handwriting recognition method of digital writing by neurodegenerative patients, according to claim 1, is characterized in that S103 also includes feature extraction by using the preprocessed convolutional neural network; before inputting the data into the convolutional neural network model, it is necessary to adjust the image size to the standard specification, which is typically 150*150 pixels, and input it to select a plurality of features with better final effect, thus obtaining a combined feature vector with a plurality of features.