CN108962397A - A kind of multichannel multitask the nervous system disease assistant diagnosis system based on pen and voice - Google Patents
A kind of multichannel multitask the nervous system disease assistant diagnosis system based on pen and voice Download PDFInfo
- Publication number
- CN108962397A CN108962397A CN201810576107.9A CN201810576107A CN108962397A CN 108962397 A CN108962397 A CN 108962397A CN 201810576107 A CN201810576107 A CN 201810576107A CN 108962397 A CN108962397 A CN 108962397A
- Authority
- CN
- China
- Prior art keywords
- channel
- feature
- voice
- user
- pen
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H70/00—ICT specially adapted for the handling or processing of medical references
- G16H70/20—ICT specially adapted for the handling or processing of medical references relating to practices or guidelines
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/40—Detecting, measuring or recording for evaluating the nervous system
- A61B5/4076—Diagnosing or monitoring particular conditions of the nervous system
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/50—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for simulation or modelling of medical disorders
Landscapes
- Health & Medical Sciences (AREA)
- Engineering & Computer Science (AREA)
- Medical Informatics (AREA)
- Public Health (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Primary Health Care (AREA)
- Life Sciences & Earth Sciences (AREA)
- Pathology (AREA)
- Epidemiology (AREA)
- Neurology (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Biophysics (AREA)
- Physiology (AREA)
- Physics & Mathematics (AREA)
- Neurosurgery (AREA)
- Bioethics (AREA)
- Heart & Thoracic Surgery (AREA)
- Molecular Biology (AREA)
- Surgery (AREA)
- Animal Behavior & Ethology (AREA)
- Veterinary Medicine (AREA)
- Medical Treatment And Welfare Office Work (AREA)
- Measuring And Recording Apparatus For Diagnosis (AREA)
Abstract
The invention belongs to digital medical fields, are related to a kind of multichannel multitask the nervous system disease assistant diagnosis system based on pen and voice.The system includes data acquisition unit, data pre-processing unit, multi-modal interaction feature acquiring unit and model construction and feedback unit.The system synchronizes task acquisition to the person's handwriting signal and voice signal of user, multi-channel feature is extracted in influence using the change of cerebral nerve state to cognition in sport, pronunciation, multitask coordinated processing etc., not only consider physiology characterization of the nervous system disease in an interaction channel, interactive voice channel single channel, the interchannels physiological properties such as related, mutual exclusion that channel is characterized to difference physiology channel when voice channel cooperation when multitask are considered simultaneously, and final training Decision Model Analysis determines whether user suffers from the nervous system disease.The present invention does not need any invasive treatment, can real-time, robust, accurate progress disease auxiliary diagnosis.
Description
Technical field
The invention belongs to digital medical fields, and in particular to a kind of multichannel multitask nervous system based on pen and voice
Disease assistant diagnosis system.
Background technique
With the variation of environment and the aging of population, more and more people suffer from the nervous system disease, strong influence
The work and life of user and its family.Clinically, mainly by Physician Global medical history, state of mind assessment, physique
The many indexs such as inspection determine whether user suffers from the nervous system disease, but since such disease incidence mechanism is complicated, check
Complex steps, therefore, it is difficult to obtain effective diagnosis scheme.The Main Physiological Characteristics of the nervous system disease can be reflected in user day
In normal human-computer interaction behavior, therefore as the continuous development of human-computer interaction technology and the mankind are for physiologic information and disease
What relationship recognized between characterization gradually gos deep into, and by obtaining and analyzing the physiologic information generated during daily interaction, facilitates
The auxiliary diagnosis and early warning of the nervous system disease.Compared with Artificial Diagnosis, by analysis physiologic information to nervous system disease
The sick method for carrying out computer-aided diagnosis is realized and carries out simple, quick, Noninvasive pathologic finding to user, reduces master
Influence of the sight factor to court verdict has very big researching value and development space.
In daily exchange, people can receive the information in multiple channels such as voice, writing, vision simultaneously, can also lead to simultaneously
The expression feedback that multiple channels carry out information is crossed, for people when carrying out multi-channel information processing, the information in different channels can be in brain
Different regions carries out cross processing, has certain flexibility ratio, but the behavior for handling multiple tasks simultaneously has dispersed brain
Attention leads to cost of processing information and receives the increase of cost, and the task switching of generation dissipates (switching costs) and appoints
Dissipation (mixing costs) is obscured in business, leads to the reduction of cognitive ability to a certain extent.Use with the nervous system disease
Family is usually expressed as the various problems such as verbal ability decline, dyskinesia and cognitive disorder.Utilize voice channel, channel
The method for carrying out diagnosis of nervous system diseases etc. the interactive information in single channel cannot comprehensively show the current physiology shape of user
State, while having ignored the integration of multichannel multiple tasks synchronizing information, influence of the cognitive load to user.Therefore how multi-pass is utilized
Human-computer interaction technology synchronous acquisition user in road produces in distinct interaction channel in interactive process that is daily while handling multiple tasks
Whether raw effective information suffers from the nervous system disease according to interchannel complementation redundancy, interactional relationship analysis user, can
The effective order of accuarcy for improving computer-aided diagnosis.
Voice channel can reflect the pronunciation state of user, and channel can reflect that user is different in terms of movement and cognition
Often.It studies on the basis of extracting the single channel interactive information in channel and voice channel respectively, is assisted by writing with pronunciation
The mode of work executes appointed task, the coordination energy of hand and linguistic function when executing multitask using channel is synchronous with voice channel
Power carries out multichannel multitask the nervous system disease auxiliary diagnosis to user, considers multi-job operation between user's interaction channel
The influence of information characteristic, whether diagnosis in real time suffers from the nervous system disease, to the Preliminary detection and prevention of the nervous system disease,
Have great importance.
Summary of the invention
The present invention is in view of the above-mentioned problems, propose a kind of based on the multichannel multitask the nervous system disease of pen and voice auxiliary
Diagnostic system.The system allows user to complete appointed task in such a way that a channel cooperates with voice channel, uses common touching
It touches screen and microphone and task acquisition is synchronized to the person's handwriting signal and voice signal of user, utilize the change of cerebral nerve state
Multi-channel feature is extracted in influence to cognition in sport, pronunciation, multitask coordinated processing etc., and training Decision Model Analysis determines
Whether user suffers from the nervous system disease.It is characterized in that being cooperateed in such a way that pen and voice combine to appointed task
Processing, the input signal of synchronous acquisition pen and voice channel not only consider the nervous system disease in an interaction channel, interactive voice
Physiology characterization in the single channel of channel, while considering difference physiology channel table when channel and voice channel cooperation when multitask
The interchannels physiological property such as correlation, mutual exclusion of sign obtains multi-modal interaction feature construction model, to user make it is real-time, accurate,
Objectively automatic diagnosis.
The technical solution adopted by the invention is as follows:
A kind of multichannel multitask diagnosis of nervous system diseases system based on pen and voice comprising:
Data acquisition unit is responsible for acquisition user information, medical history, scale assessment is carried out to user, is carried out to user
The synchronous task that digital pen interaction is interacted with pronunciation is tested, and saves the person's handwriting and audio file of user；
Data pre-processing unit, the data for being responsible for acquiring the data acquisition unit pre-process, and therefrom extract and use
Family information, medical history, scale result, pen and interactive voice data are stored in customer data base, diagnosis library, doctor respectively
Gain knowledge library, in case database；
Multi-modal interaction feature acquiring unit is responsible for being mentioned according to the pen and interactive voice data in the case database
The interaction feature of pen and voice channel and the multitask coordinated feature of interchannel are taken, and feature is merged and selected, is obtained
The input sample of training pathological model；
Model construction and feedback unit are responsible for obtaining pathology mould using the multi-modal interaction feature progress model training of input
Type, then by the pathological model and the dependency number in the customer data base, the diagnosis library, the medical knowledge base
Complex decision model is obtained according to Fusion training, the final differentiation result of the complex decision model is fed back into user.
Further, its personal characteristics is reacted in the age of the data acquisition unit records user, gender, education degree etc.
Information；Record physical signs and morbid state that user's history checks body；Due to different Scale and questionnaires weighting point not
Together, different state of mind Scale and questionnaires is completed under the guidance of medical practitioner.
Further, the data acquisition unit is when obtaining person's handwriting and audio task, to allow user that natural shape is presented
State, before user carries out official testing, it is desirable that user carries out an interactive operation in the case where comfortable sitting posture, in daily pronunciation
(deliberately not improve or lower one's voice) under state and carry out pronunciation operation, at the same keep between recording equipment and lip away from
From.Before recording, the coordination ability of synchronous acquisition data and voice data assessment hand and phonetic function is utilized to instruction manual
Purpose, it is synchronous to appointed task to execute the requirement write with pronunciation, carry out task demonstration.Here appointed task be corpus in
The simple text that machine occurs, such as " eat grape and do not spit Grape Skin, do not eat grape and spit Grape Skin ", " books are the mankind
Treasure-house of thought " etc..After user understands mission requirements, task test is carried out, is recognized as there is body during completion task
The case where knowing overload, then appropriate rest a period of time carries out task test again.If occurring in task test process and appointing
It is engaged in unrelated dialogue, then re-starts task test.
Further, when the data pre-processing unit is pre-processed, such as user information, medical history, scale result
Information change, be saved in corresponding database again, as unchanged, directly skip this step.
Further, the data pre-processing unit cleaning pen and interactive voice data reject empty, invalid pen interaction
Data and audio data；End-point detection is carried out to recording audio, intercepts effective audio；It will treated effective pen and voice
Duan Zuowei initial data is stored into case database.After obtaining effective data, feature extraction can be carried out, obtaining can be anti-
Using the multi-modal interaction feature of family the nervous system disease symptom.
Further, the multi-modal interaction feature acquiring unit obtains multi-modal interaction feature, comprising:
(1) characteristic extracting module is responsible for extracting a channel characteristics, and voice channel feature, a channel is cooperateed with voice channel
Feature；
(2) Fusion Features module and feature selection module are responsible for merging the feature of extraction, and dimensionality reduction obtains energy
The optimal subset of enough user's pathological characters of reflection comprehensively.
Further, the characteristic extracting module extracts voice channel feature according to pararthria and dysarthrosis.In language
In sound interaction channel, the nervous system disease acalaphasia, which is mainly reflected in, cannot issue normal sound (pararthria) and words and phrases
Read difficult (dysarthrosis).The influence of pararthria is mainly used to measure the ability of speaking substantially of people, is exactly the control to sound
Ability processed, tone caused by the ability of regulation and control for often showing as vocal cords is insufficient, making capacity is insufficient or the reasons such as irregular of vibration
Abnormal, loudness exception and dystimbria.The influence of dysarthrosis mainly measures the flexibility of the vocal organs of people and coordinates fortune
Kinetic force shows as significant speech sound whether can be issued.If people cannot control pronunciation appropriate in continuous pronunciation
The movement of organ, causes the motion range of target phoneme smaller, is confined near normal place, then reduces the flexibility of organ,
The exception for causing rhythm, rhythm etc. influences the clarity and fluency of voice.
Further, in interaction channel, exception of the patient with nervous system disease in movement and cognitive function is main
The state of wieling the pen and graphic plotting are embodied in as a result, therefore the characteristic extracting module is usually extracted from interactive task and wield the pen
Whether feature and cartographic features diagnosis user suffer from the nervous system disease.Motion feature can preferably reflect the movement function of user
Can, often show as the variation of the kinematic parameters such as the caused pen tip position of the problems such as hand trembles, pressure.And cartographic features then with
The cognitive function relationship at family is more close, often shows as the exception of deadline caused by user cognition problem, errors number increases
Etc..
Further, it is carried out in multitask coordinated operation in channel and voice channel, patient with nervous system disease is being remembered
Recall, recognize, the relevant issues of concertedness etc. dissipate (switching carrying out multitask switching and can generate more switchings
Costs dissipation (mixing costs) cost) is obscured with task, is mainly reflected in the complementation and redundancy of the feature of interchannel, because
, when pen and voice channel are performed simultaneously specified interactive task, the characteristic extracting module is according to extraction channel and voice for this
Interactive information correlation is stronger between channel cooperates with feature to reflect influence of the nervous system disease to factors such as cognitions.
Further, the Fusion Features module is the different physiological properties of reaction for extracting the characteristic extracting module
Feature is merged, and the feature that can react user's current state more comprehensively is obtained.In order to avoid extracting characteristic information
Redundancy, the feature selection module can be by having supervision or the selection of unsupervised feature selection approach to have the important of discrimination
Feature.The step of simplifying enough such as the feature of extraction, feature selecting can be omitted.
Further, the model construction includes: with feedback unit
(1) model construction module is responsible for obtaining pathological model and complex decision model using the method training of machine learning；
(2) decision feedback module is responsible for the result of decision using the method for multi-modal interaction shunting information optimization to user
It is fed back.
Further, the selection of classifier has large effect to final judgement, and certain classifiers are only special to part
Sign is sensitive, therefore the model construction module needs to select suitable classifier training pathological model, is accurately adjudicated knot
The case where fruit avoids over-fitting simultaneously, reaches the balance of generalization ability and power of test.The input of user's test data is trained
The pathological model arrived obtains pathological model diagnostic result.The model construction module is by pathological model diagnostic result and number of users
According to library, medical knowledge base, the related opinions such as diagnosis library combine and establish more attribute human-computer interaction complex decision models, obtain most
Whole complex decision as a result, utilize the stability of a variety of evaluation index computation models simultaneously.
Further, described in order to obtain user more really with deep experience after user obtains diagnostic result
The method that decision feedback module shunts optimization by multichannel, i.e. method of the microphone in conjunction with display screen are by final diagnosis knot
Fruit feeds back to user by pen and voice channel, by feeding back the performance for making user experience task.
Compared with prior art, the present invention has the advantage that as follows with good effect:
1. whether suffering from mind using pen multitasking form analysis user synchronous with voice The present invention gives a kind of
System through systemic disease, can be avoided single channel cannot comprehensively react the physiological status of user, while utilize at synchronous
Interchannel switching costs and influence of the mixing costs to interaction mode when managing multiple tasks, cotasking processing
To the correlation of different pass effects, increase the accuracy of building Model Diagnosis.
2. the present invention extracts three classes disease for the multitask coordinated property of channel, voice channel, channel and voice channel
Manage feature.Influence of the nervous system disease to voice is mainly made of two aspects, pararthria (dysphonia) and dysarthrosis
(dysarthria).Influence to channel is mainly by the state of wieling the pen (kinematic feature) and graphic plotting
It is formed in terms of the result two of (figurate feature).The influence of voice channel and the multitask coordinated perception in channel mainly by
The calculating such as correlation, the redundancy of interchannel composition, three category features are merged, and can be arrived and relatively be reacted nervous system disease comprehensively
The Multichannel fusion feature of disease.
3. the final diagnostic result that the present invention obtains complex decision carries out real-time diagnostic feedback.Utilize a variety of channels
Complex decision model is obtained with dimensional information Fusion training, comprehensively considers the physiology shape of user's history and multiple dimensions such as current
Accurate, diagnosis in real time and feedback can be obtained by simply interacting in state.
4. the present invention only needs common microphone and sum digit screen as information collection tool, any intrusion is not needed
Property treatment, it is thus only necessary to analyze the physical condition of the i.e. predictable user of user's current physiology data, realization is to the nervous system disease
Preliminary detection, improve the quality of life of user.
Detailed description of the invention
The desktop operation scenario schematic diagram that Fig. 1 is.
Fig. 2 is the composition frame diagram of the disease diagnosing system based on voice and pen.
Fig. 3 is the work flow diagram of the disease diagnosing system based on voice and pen.
Fig. 4 is the model construction schematic diagram of pathological model and complex decision model.
Specific embodiment
In order to make those skilled in the art better understand the present invention, further retouched in detail below in conjunction with example and attached drawing
The present invention is stated, but is not construed as limiting the invention.
The present invention can understand operation scenario and frame of the invention, the nervous system disease auxiliary diagnosis by such as Fig. 1,2
System is mainly by data acquisition unit, data pre-processing unit, multi-modal interaction feature acquiring unit, model construction and feedback
First four parts composition.When actual test, user need to only complete corresponding pen and interactive voice task, can pass through the nervous system
Disease assistant diagnosis system is predicted whether by computer assisted mode with the nervous system disease.The present embodiment be
Processing and training data under conditions of matlab7.10.0, weka, specifically as shown in figure 3, the workflow of the system is as follows:
1. the work of acquisition this part training data is the basis of experiment, specific tool and parameter are as follows:
(1) user information acquisition, medical history, scale check part, fill in information, the medical history of user first.Its
Middle scale inspection is carried out under the guidance of medical practitioner, and doctor provides phase reserved portion by the performance level of customer problem, note
Employ the scoring event at family.When patient understands difficulty or produce ambiguity to topic, doctor provides reasonable explanation and demonstration.
This, which is checked, supports user to carry out on computers, to the of less demanding of computer equipment, meets normal viewing, interaction.
Wherein, user information includes the multiple information such as gender, age, education level；Medical history is used by inquiry
The heredity medication history and history disease detection situation at family, record multinomial detection physical signs；User by with medical practitioner or warp
The people for crossing training exchanges the test topic for completing different scales, such as MMSE, MOCA scale, and is carried out according to performance level to result
Scoring.
(2) pen and voice collecting part carry out a data with flat-panel monitor and digital pen and acquire, and common microphone carries out
Voice collecting, using above equipment to specified test material complete as defined in writing task and bright reading task, write here with
Read aloud synchronous progress.With liquid crystal numerical digit screen Wacom Cintiq DTK-1300 when acquisition data, screen resolution is set as
1920*1080, subsidiary digital pen parameter Wacom KP-701E, sample rate 100Hz.Gloomy sea has been used when acquiring voice data
The external microphone wind of Sai Er, with microphone apart from being 10cm or so when user acquires voice, design parameter is monophonic, sample rate
For 44.1kHz.Testing material library is made of multiple sentences, and the test statement in a corpus occurs at random in when actual test.
2. data prediction.Effective information is extracted in the work of this part, is prepared for subsequent training data.
(1) main information in user information, medical history and scale score value is extracted, corresponding database is respectively stored into
In.
(2) data and audio data handle part, the work of this part remove it is bad, it is empty etc. not meet trained item
The handwriting data and voice data of part, clean the sample data in channel and voice channel, remove exceptional sample, extract
The effective interaction data and interactive voice data that user personality can be reacted, as initial data storage to case database
In.
3. multi-modal interaction feature.Effective Multichannel fusion data how are extracted, are had to the training of subsequent pathological model
Significant, specific research is as follows.
(1) feature extraction.The physiological status for how extracting validity feature characterization user has important meaning to the training of model
Justice.Feature is extracted here mainly by three aspects,
The feature of interaction channel is mainly manifested in motion feature and cartographic features, and motion feature is mainly with the sampling of stroke
Sequence often extracts the kinematic parameters such as position, pressure, the angle of person's handwriting as fundamental analysis unit, and utilizes a variety of analysis sides
The feature that method processing corresponding sports parameter obtains.Graphic feature is usually to be directed to particular task, generally using whole figure as
Object is analyzed, such as alphabet size, shape, departure degree feature are calculated.
Voice channel feature is mainly manifested in dysphonia and dysarthrosis, wherein dysphonia be mainly reflected in volume,
Hoarse sound, breathiness, trembles at coarse sound.Therefore can extract common feature including but not limited to Jitter, Shimmer,
HNR, NHR, RPDE etc..Dysarthrosis is mainly shown as cacoepy, stings the problems such as unclear word, tone allorhythmia.It can count
Calculate such as pronounce duration, the stability of pronunciation, voice rate, pronunciation rate, sentence map feature.
Channel and the multitask coordinated operation of voice channel can express in many aspects interactive influence, therefore can be never
A variety of collaboration features of same angle extraction pen and voice.It is including but not limited to handled in multiple interactive tasks at the same time, pen is logical
Road feature is expressed the meaning the correlation square of (the Pearson corresponding coefficient of such as every section of feature and corresponding response results) with voice channel feature
Battle array, alternative description of synchronism description and two kinds of behaviors expressed the meaning etc..
(2) Fusion Features and selection
Since the interactive information feature of single channel is difficult more comprehensively to express the ownness of user, by step (1)
Channel characteristics, voice channel feature, pen and the multitask coordinated feature of voice channel extracted are merged, and obtaining can be comprehensively anti-
Using the characteristic information of family current state.The method of fusion has very much, including but not limited to by a variety of extraction feature head and the tail phases
Traditional characteristic fusion method even.Multichannel fusion characteristic dimension is very likely greater than trained sample size, and excessively high dimension
The performance of classifier can be reduced, be able to reflect the optimal subset of user's pathological characters in order to obtain, using but be not limited to based on phase
The feature selection approach (CFS) of closing property, the feature that principal component analytical method (PCA) etc. merges voice with channel are selected
It selects, reduces Multichannel fusion intrinsic dimensionality and obtain multi-modal interaction feature, improve the efficiency and precision of pathological model training.
4. model construction and feedback.Suitable training method is selected, is actual measurement preparation model, and give appropriate
Feedback increases the experience of user.
The method training pathological model of a variety of machine learning can be used in the present invention, is with the training method of integrated study
Example is combined classifier using the method for improving prediction (stacking), uses the method for cross validation by multichannel first
Interaction feature set is divided into n equal portions, and n-1 is allocated as training data, remaining portion data progress test data, a variety of points of selection
Class device is used as " base " classifier, such as SVM, Bayesian network, and the prediction result that " base " classifier is obtained is as input feature vector
The disaggregated model of the training second layer, such as linear classifier, RF obtain the integrated study pathological model of voice and channel fusion
Diagnostic result.When data volume increases, the method that can attempt deep learning, it is accurate that training obtains, steady pathological model,
To whether effectively being analyzed with the nervous system disease.
Consider pathological model diagnostic result, the userspersonal information stored in customer data base, the state of an illness in medical knowledge base
The specification and standard of relevant evaluation parameter and the professional diagnosis in diagnosis library as a result, synthetic user historical information with work as
Preceding index, using but be not limited to the machine learning methods such as decision tree, hidden Markov training complex decision model, to user's mesh
Preceding physiological status obtains more comprehensive diagnosis, and model construction process is as shown in Figure 4.For the reliability for ensuring training pattern,
Trained model need to be evaluated and tested, be verification result validity, evaluation index including but not limited to accuracy rate, recall ratio,
Precision ratio, F value.System is after being calculated final evaluation result, by multi-modal interaction information privacy device to multi-pass
Road fusion human-computer interaction differentiations result by multichannel shunting optimize method pass through respectively channel and voice channel to
Family carries out Real-time Feedback.
The above embodiments are merely illustrative of the technical solutions of the present invention rather than is limited, the ordinary skill of this field
Personnel can modify to technical solution of the present invention when without departing from the spirit and scope of the present invention or equally replace
It changes, the scope of protection of the present invention shall be subject to the claims.
Claims (10)
1. a kind of multichannel multitask the nervous system disease assistant diagnosis system based on pen and voice characterized by comprising
Data acquisition unit is responsible for acquisition user information, medical history, scale assessment is carried out to user, carries out pen friendship to user
The synchronous task mutually interacted with pronunciation is tested, and saves the person's handwriting and audio file of user；
Data pre-processing unit, the data for being responsible for acquiring the data acquisition unit pre-process, and therefrom extract user's letter
Breath, medical history, scale result, pen and interactive voice data are stored in customer data base respectively, diagnosis library, medicine are known
In knowledge library, case database；
Multi-modal interaction feature acquiring unit is responsible for extracting pen according to the pen and interactive voice data in the case database
With the interaction feature of voice channel and the multitask coordinated feature of interchannel, and feature is merged and selected, is trained
The input sample of pathological model；
Model construction and feedback unit are responsible for obtaining pathological model using multi-modal interaction feature progress model training, then will
The pathological model merges instruction with the related data in the customer data base, the diagnosis library, the medical knowledge base
Complex decision model is got, the final differentiation result of the complex decision model is fed back into user.
2. the system as claimed in claim 1, which is characterized in that the user information of data acquisition unit acquisition includes
Gender, age, education level；The medical history includes the heredity medication history and history disease detection situation of user, and records
Multinomial detection physical signs；It is described that scale assessment is carried out to user, it is by user and medical practitioner or trained people
The test topic of different scales is completed in exchange, and is scored according to performance level result.
3. the system as claimed in claim 1, which is characterized in that the interactive device that the data acquisition unit uses is common
Numerical digit screen sum digit pen, sound pick-up outfit is common external microphone wind, it is desirable that subject synchronously completes under natural state
The interactive task of pen and voice pronounces while writing.
4. the system as claimed in claim 1, which is characterized in that according to actual needs, removal is empty for the data pre-processing unit
, ineligible handwriting data and voice data.
5. the system as claimed in claim 1, which is characterized in that the multi-modal interaction feature acquiring unit includes feature extraction
Module, the characteristic extracting module are characterized according to the physiology of channel and voice channel, extract the feature of single channel, while root
According to the influence of different inter-channel synchronizations processing multiple tasks collaborative perception, the multitask coordinated spy in channel and voice channel is extracted
Sign.
6. the system as claimed in claim 1, which is characterized in that the characteristic extracting module is according to pararthria and dysarthrosis
Voice channel feature is extracted, motion feature is extracted from interactive task and cartographic features are used as a channel characteristics, and extracts pen
Interactive information correlation is stronger between channel and voice channel cooperates with feature.
7. the system as claimed in claim 1, which is characterized in that the multi-modal interaction feature acquiring unit includes Fusion Features
Module, the Fusion Features module melt voice channel, channel, voice channel and the multitask coordinated feature in channel
Conjunction obtains Multichannel fusion feature, more comprehensively to show the physiological status of user in daily life.
8. the system as claimed in claim 1, which is characterized in that the multi-modal interaction feature acquiring unit includes feature selecting
Module, the feature selection module carry out dimensionality reduction to Multichannel fusion feature and obtain multi-modal interaction feature, different logical to reduce
Influence of the information redundancy to training effect between road.
9. the system as claimed in claim 1, which is characterized in that the model construction and feedback unit include model construction mould
Block, the model construction module obtain pathological model diagnostic result, and will be sick using multi-modal interaction feature training pathological model
Reason Model Diagnosis result is merged with the data in the customer data base, the diagnosis library, the medical knowledge base
Training complex decision model, obtains final court verdict.
10. the system as claimed in claim 1, which is characterized in that the model construction and feedback unit include decision feedback mould
Block, the decision feedback module, which shunts the complex decision result of the complex decision model by multi-channel information, to be optimized, with
Mode of the visual channel in conjunction with auditory channel carries out real-time diagnosis feedback to user.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810576107.9A CN108962397B (en) | 2018-06-06 | 2018-06-06 | Pen and voice-based cooperative task nervous system disease auxiliary diagnosis system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810576107.9A CN108962397B (en) | 2018-06-06 | 2018-06-06 | Pen and voice-based cooperative task nervous system disease auxiliary diagnosis system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108962397A true CN108962397A (en) | 2018-12-07 |
| CN108962397B CN108962397B (en) | 2022-07-15 |
Family
ID=64493586
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810576107.9A Active CN108962397B (en) | 2018-06-06 | 2018-06-06 | Pen and voice-based cooperative task nervous system disease auxiliary diagnosis system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108962397B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111354458A (en) * | 2018-12-20 | 2020-06-30 | 中国科学院软件研究所 | Touch interactive motion user feature extraction method based on general drawing task and auxiliary disease detection system |
| CN111683048A (en) * | 2020-05-06 | 2020-09-18 | 浙江大学 | Intrusion detection system based on multicycle model stacking |
| CN112294253A (en) * | 2019-07-26 | 2021-02-02 | 深圳百诺明医说科技有限公司 | Disease diagnosis system based on user voice change and household intelligent robot |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110119212A1 (en) * | 2008-02-20 | 2011-05-19 | Hubert De Bruin | Expert system for determining patient treatment response |
| CN103956171A (en) * | 2014-04-01 | 2014-07-30 | 中国科学院软件研究所 | Multi-channel mini-mental state examination system |
| CN106073706A (en) * | 2016-06-01 | 2016-11-09 | 中国科学院软件研究所 | A kind of customized information towards Mini-mental Status Examination and audio data analysis method and system |
| CN107273677A (en) * | 2017-06-08 | 2017-10-20 | 中国科学院软件研究所 | A kind of multi-channel nerve function quantitative evaluation system |
| CN107451385A (en) * | 2016-05-30 | 2017-12-08 | 中国科学院软件研究所 | A kind of the nervous system disease monitoring and early warning system based on daily necessities |
-
2018
- 2018-06-06 CN CN201810576107.9A patent/CN108962397B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110119212A1 (en) * | 2008-02-20 | 2011-05-19 | Hubert De Bruin | Expert system for determining patient treatment response |
| CN103956171A (en) * | 2014-04-01 | 2014-07-30 | 中国科学院软件研究所 | Multi-channel mini-mental state examination system |
| CN107451385A (en) * | 2016-05-30 | 2017-12-08 | 中国科学院软件研究所 | A kind of the nervous system disease monitoring and early warning system based on daily necessities |
| CN106073706A (en) * | 2016-06-01 | 2016-11-09 | 中国科学院软件研究所 | A kind of customized information towards Mini-mental Status Examination and audio data analysis method and system |
| CN107273677A (en) * | 2017-06-08 | 2017-10-20 | 中国科学院软件研究所 | A kind of multi-channel nerve function quantitative evaluation system |
Non-Patent Citations (4)
| Title |
|---|
| 吴亚飞: ""多任务学习和疾病分类中的稀疏优化问题研究"", 《中国优秀硕士学位论文全文数据库 基础科学辑》 * |
| 吴亚飞: ""多任务学习和疾病分类中的稀疏优化问题研究"", 《中国优秀硕士学位论文全文数据库 基础科学辑》, 15 March 2017 (2017-03-15), pages 1 * |
| 李洋等: ""云端融合的神经***疾病多通道辅助诊断研究"", 《中国科学：信息科学》 * |
| 李洋等: ""云端融合的神经***疾病多通道辅助诊断研究"", 《中国科学：信息科学》, vol. 47, no. 9, 20 September 2017 (2017-09-20), pages 1 - 16 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111354458A (en) * | 2018-12-20 | 2020-06-30 | 中国科学院软件研究所 | Touch interactive motion user feature extraction method based on general drawing task and auxiliary disease detection system |
| CN111354458B (en) * | 2018-12-20 | 2023-11-14 | 中国科学院软件研究所 | Touch interactive motion user feature extraction method and auxiliary disease detection system based on universal drawing task |
| CN112294253A (en) * | 2019-07-26 | 2021-02-02 | 深圳百诺明医说科技有限公司 | Disease diagnosis system based on user voice change and household intelligent robot |
| CN111683048A (en) * | 2020-05-06 | 2020-09-18 | 浙江大学 | Intrusion detection system based on multicycle model stacking |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108962397B (en) | 2022-07-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Chen et al. | Multimodal behavior and interaction as indicators of cognitive load | |
| Cohn et al. | Multimodal assessment of depression from behavioral signals | |
| Williamson et al. | Tracking depression severity from audio and video based on speech articulatory coordination | |
| Jiang et al. | Investigation of different speech types and emotions for detecting depression using different classifiers | |
| Muzammel et al. | AudVowelConsNet: A phoneme-level based deep CNN architecture for clinical depression diagnosis | |
| CN111461176B (en) | Multi-mode fusion method, device, medium and equipment based on normalized mutual information | |
| Alghowinem et al. | Detecting depression: a comparison between spontaneous and read speech | |
| US9737255B2 (en) | Measuring cognitive load | |
| CN106073706B (en) | A kind of customized information and audio data analysis method and system towards Mini-mental Status Examination | |
| Guohou et al. | What reveals about depression level? The role of multimodal features at the level of interview questions | |
| CN108962397A (en) | A kind of multichannel multitask the nervous system disease assistant diagnosis system based on pen and voice | |
| Yin et al. | Towards automatic cognitive load measurement from speech analysis | |
| Wang et al. | Depression speech recognition with a three-dimensional convolutional network | |
| Weiner et al. | Selecting features for automatic screening for dementia based on speech | |
| Tanaka et al. | Automatic detection of very early stage of dementia through multimodal interaction with computer avatars | |
| An et al. | Mental health detection from speech signal: A convolution neural networks approach | |
| Danner et al. | Advancing mental health diagnostics: GPT-based method for depression detection | |
| Zhang et al. | Adolescent depression detection model based on multimodal data of interview audio and text | |
| Alghowinem | From joyous to clinically depressed: Mood detection using multimodal analysis of a person's appearance and speech | |
| Böck et al. | Anticipating the user: acoustic disposition recognition in intelligent interactions | |
| Schipor et al. | Towards a multimodal emotion recognition framework to be integrated in a Computer Based Speech Therapy System | |
| Zhang et al. | A study of correlation between physiological process of articulation and emotions on Mandarin Chinese | |
| Wang et al. | Speech detection of depression based on multi-mlp | |
| Krause et al. | A method for transcribing the manual components of Cued Speech | |
| Lai et al. | ResNet for recognition of Qi-deficiency constitution and balanced constitution based on voice |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |