CN108345903B - A kind of multi-modal fusion image classification method based on mode distance restraint - Google Patents
A kind of multi-modal fusion image classification method based on mode distance restraint Download PDFInfo
- Publication number
- CN108345903B CN108345903B CN201810073841.3A CN201810073841A CN108345903B CN 108345903 B CN108345903 B CN 108345903B CN 201810073841 A CN201810073841 A CN 201810073841A CN 108345903 B CN108345903 B CN 108345903B
- Authority
- CN
- China
- Prior art keywords
- vector
- brain
- feature
- mode
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
- G06F18/2136—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods based on sparsity criteria, e.g. with an overcomplete basis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Magnetic Resonance Imaging Apparatus (AREA)
Abstract
The invention discloses a kind of multi-modal fusion image classification methods based on mode distance restraint, comprising the following steps: the first step, the rs-fMRI data and DTI data for obtaining multiple tested objects；Second step constructs brain function network characterization vector sum brain structural network feature vector for each tested object respectively；Third step carries out characteristic filter operation to the feature vector of both modalities which based on Kendall tau related coefficient and " overlapping " mode；4th step, the feature vector of same tested object both modalities which is added on K-support norm original base before the mapping after relative distance constraint, construct the objective function of multi-modal feature selection module, filter out the optimal characteristics vector of both modalities which；5th step is based on multi-kernel support vector machine model training classifier；The optimal characteristics vector of object both modalities which to be measured is inputted into trained classifier, predicts its class label.Classification accuracy of the present invention is high.
Description
Technical field
The present invention relates to technical field of image processing, more particularly to a kind of multi-modal fusion based on mode distance restraint
Image classification method.
Background technique
Nearly ten years with the progress of brain image technology, the research of brain science enters the period of a rapid development.Magnetic
Resonance image-forming is as a kind of noninvasive living body brain function detection technique, by its high-resolution, radiationless advantage, from 20
Since the 90's of century are born, the brain imaging technique being most widely used in brain science research has been rapidly become.Wherein magnetic resonance at
As technology includes structure Magnetic resonance imaging (sMRI, structural Magnetic Resonance Imaging), function
NMR imaging (fMRI, functional Magnetic Resonance Imaging) and diffusion tensor (DTI,
Diffusion Tensor Imaging) etc..Every kind of neuroimaging technology, which provides, portrays brain tissue different level, and
Each own advantage and disadvantage and applicable situation.Meanwhile multi-modal nuclear magnetic resonance image fusion treatment technology can integrate different modalities image
To the delineation information of brain tissue, systematically to explore the structure function of brain and studying the morbidity machine of Nervous and mental diseases
The great Neuscience problems such as reason provide completely new visual angle.
Human brain is described generally as the network organization an of economy and exquisiteness, most neuropsychiatric disease symptom
Related with the cranial nerve network imbalance of responsible mood or cognitive function, the imbalance of large-scale distributed neural network can be from brain function
It can be expressed in two levels of network and brain structural network.Meanwhile a large amount of neuroimaging researchs have proven to function and structure is special
The nicety of grading of image can be improved in the fusion of sign.The method for carrying out image classification currently based on multi-modal fusion is will not
A long feature vector is connected into the feature that Frequency extraction goes out and carries out subsequent analysis, is not accounted between different modalities data
Relationship, classification accuracy is to be improved.Divide therefore, it is necessary to provide the new image that carries out based on multi-modal fusion of one kind
The method of class.
Summary of the invention
A kind of method that technical problem solved by the invention is to provide multi-modal fusion based on mode distance restraint,
It establishes on the basis of being connected to the network Fusion Features based on brain structural network and brain function, makes full use of from different modalities mind
Complementary information through image data promotes classification accuracy.
Technical solution provided by the present invention are as follows:
A kind of multi-modal fusion image classification method based on mode distance restraint, comprising the following steps:
The first step obtains data, specifically: obtaining the rs-fMRI data and DTI data of multiple tested objects, and to it
It is pre-processed, obtains pretreated rs-fMRI data and pretreated DTI data；Wherein tested object includes known
The object to be measured of the sample object of class label and unknown class label；
Second step, the feature vector for constructing both modalities which respectively for each tested object, i.e. brain function network characterization to
Amount and brain structural network feature vector；
For each tested object, the method that constructs its brain function network characterization vector are as follows: first according to pretreatment after
Rs-fMRI data, the part that cerebellum is removed in its full brain is divided by 90 cortex and skin using automatic dissection tag template
Lower core rolls into a ball region, i.e. 90 brain areas；Then each of which brain area is defined as a node in its brain function network, it is each
A brain area is defined as the connection relationship in its brain function network respective nodes between the Pearson correlation coefficients of average time sequence
(calculating any brain area is the prior art to the Pearson correlation coefficients of average time sequence)；Again with each in its brain function network
Connection relationship between a node is matrix element, and one 90 × 90 tranquillization state brain function network symmetrical matrix is constructed for it；Most
90 diagonal elements on symmetrical matrix diagonal line are removed afterwards, extract all elements conduct of lower Delta Region in symmetrical matrix
Its brain function network characterization vector, vector dimension 4005；
For each tested object, the method that constructs its brain structural network feature vector are as follows: first according to pretreatment after
DTI data, the part that cerebellum is removed in its full brain is divided by 90 cortex and subcutaneous core using automatic dissection tag template
Group region, i.e. 90 brain areas；Each of which brain area is defined as a node in its brain structural network, by its each brain area pair
Between white matter fiber quantity be defined as the connection relationship in its brain structural network between respective nodes (according to FACT
(Deterministic fiber assignment with the continuous tracking) algorithm is traced back to any
The quantity of white matter fiber, the method are the prior art between node (i.e. brain area))；Again between each node in its brain structural network
Connection relationship be matrix element, construct one 90 × 90 tranquillization state brain structural network symmetrical matrix for it；Finally removal should
90 diagonal elements on symmetrical matrix diagonal line extract all elements of lower Delta Region in symmetrical matrix as its brain knot
Network forming network feature vector, vector dimension 4005；
Third step, characteristic filter operation, specifically: based on Kendall tau related coefficient and " overlapping " mode to subject
The feature vector of object both modalities which carries out characteristic filter operation, obtains the new feature vector of tested object both modalities which；
4th step, feature selecting operation, specifically: adding same tested object on K-support norm original base
The feature vector of both modalities which before the mapping after relative distance constraint, construct the objective function of multi-modal feature selection module,
The feature vector new to tested object both modalities which carries out feature selecting, to obtain the optimal spy of tested object both modalities which
Levy vector；
5th step, multi-mode classification analysis, specifically: the optimal characteristics vector based on training sample both modalities which is adopted
With multi-kernel support vector machine model training classifier；The optimal characteristics vector input of object both modalities which to be measured is trained
Classifier predicts its class label.
Further, in the third step characteristic filter operation specifically includes the following steps:
Step 3.1 obtains patient's group and Normal group sample data first, is used in combinationWithIndicate i-th of sample pair
As the jth dimensional feature in the e modal characteristics vector with h-th of sample object is (in corresponding brain function network or brain function network
Jth is between the connection relationship node), i.e. in the e modal characteristics vector of i-th of sample object and h-th of sample object
J tie up element, e=1, the 2, the 1st mode be brain function network, the 2nd mode be brain structural network, j=1,2 ..., 4005, yiAnd yh
Respectively indicate the class label of i-th of sample object and h-th of sample object, it is disease that class label, which is the 1 expression sample object,
People group sample object, class label be -1 indicate the sample object be Normal group sample object, further according to formula 1)
Calculate Kendall tau related coefficient:
Wherein:Indicate the Kendall tau related coefficient of the jth dimensional feature in e modal characteristics vector, m and n difference
The sample object number with Normal group is organized for patient,WithRespectively e mode reconciles to anharmonic couple
Number；We are not necessarily to relationship between two sample objects for considering same grouping,Therefore total sample object to quantity be m ×
n；Reconciliation pair is defined as:
Meanwhile anharmonic pair is defined as:
Wherein: sgn is sign function,WithAs meet formula 2) and formula 3) sample object to i-
The number of h；
Being positive indicates conspicuousness ratio of the jth dimensional feature in e modal characteristics vector in patient's group in normal control
Conspicuousness enhancing in group, andBe negative the conspicuousness for indicating the jth dimensional feature in e modal characteristics vector in patient's group
Weaken than the conspicuousness in Normal group.
Then rightJ=1,2 ..., 4005 are ranked up according to order of magnitude；It is right in e modal characteristics vector to choose
It answersMore than the characteristic dimension of given threshold.It is worth noting that, in above step, brain function network characterization and brain knot
Structure network characterization need to be separated and be operated.The threshold value is by user's sets itself, preferably threshold valueJ=1,2 ..., 4005
Average valueJ=1,2 ..., 4005 standard deviation；
Step 3.2, in order to ensure the corresponding relationship of brain network characterization, the spy that step 3.1 is selected with " overlapping " mode
Sign dimension is further screened, to guarantee that the connection relationship in brain function network or brain function network between any node pair is same
When occur or be not present in subsequent feature selection step；Screening technique are as follows: if the jth Wei Te in the 1st modal characteristics vector
Seeking peace, (jth is between the connection relationship and brain function net node i.e. in brain function network for jth dimensional feature in the 2nd modal characteristics vector
Jth is between the connection relationship node in network) it is selected simultaneously by step 3.1, then retain this feature dimension；Otherwise, if the 1st mode
In jth dimensional feature and the 2nd modal characteristics vector in feature vector in jth dimensional feature only one or none by step
3.1 selections, then filter out this feature dimension；
Step 3.3, feature vector (the brain function network characterization vector sum brain structure to each tested object both modalities which
Network characterization vector), only retain the characteristic dimension that filters out of step 3.2, it is (new to obtain the new feature vector of its both modalities which
Brain function network characterization vector or brain structural network feature vector).
Further, the principle that feature selecting operates in the 4th step are as follows:
(1) feature selecting based on k-support regularization term is carried out, needs to minimize mesh with k-support normal form
Scalar functions, calculation formula are expression formula 4):
Wherein:For the e modal characteristics vector matrix of S training sample, S
Referring respectively to training samples number with l, (training samples number S can be in step 3.1 patient and organize sample and normal right herein
According to a group sample, can also separately select sample) and the new feature vector of the obtained e mode of step 3.3 dimension；It indicates
I-th of training sample feature vector new by e mode that step 3.3 obtains；we∈Rl×1Represent e modal characteristics vector
Regression coefficient vector, for parameter to be optimized (the method for solving prior art of the parameter to be optimized)；Y=[y1,y2,…,yi,…,
yS]T∈RS×1For the class label vector of S training sample, all elements are all 1 or -1 in Y；F indicates Frobenius
Normal form；λ1The regularization parameter of the sparse degree of Controlling model, for empirical parameter (such as: 10 foldings can be being carried out in training set
Cross validation；The corresponding parameter of best classification results obtained in cross-validation is used for external cross validation
Test set is tested)；R is to meet expression formula 5 in { 0 ..., k-1 }) condition unique integral；K meets k < l；It is
Vector weIn the i-th big element；Expression formula 5) as follows:
(2) present invention carries out making full use of multi-modal benefit based on the feature selecting for improving K-support regularization term
Information is filled to select the optimal characteristics vector of each mode, adds same training sample on K-support norm original base
Brain function network and brain structural network feature vector before the mapping after relative distance constraint, its calculation formula is expression
Formula 6):
Wherein: D is relative distance constraint；WithThe new e mode that is obtained for i-th of training sample by third step and
T modal characteristics vector (i.e. brain function or brain structural network feature vector)；F indicates Frobenius normal form；
The objective function of multi-modal feature selection module is rewritten as expression formula 7 as a result):
Wherein: λ1> 0 and λ2> 0, λ1And λ2The reservation of Controlling model sparsity and different modalities feature vector relationship respectively
Degree.Above-mentioned objective function is solved, w is obtainede(method for solving of above-mentioned objective function is the prior art)；Determine weIn be greater than 0 yuan
The corresponding characteristic dimension of element selects the feature of these dimensions to constitute its e in the feature vector new from tested object e mode
Mode optimal characteristics vector.
Further, in the 5th step multi-mode classification analysis principle are as follows:
The multi-kernel support vector machine need to meet objective function, see expression formula 8):
Wherein: qeIndicate the hyperplane method vector of e-th of modal data；B indicates deviation；ξiIt indicates to measure error in data point
The non-negative slack variable of class；C indicate penalty factor, for weighing the weight of loss and class interval, be empirical parameter (such as:
10 folding cross validations can be being carried out in training set；The corresponding parameter of best classification results obtained in cross-validation
It is tested for the test set to external cross validation)；For non-linear transform function；For training sample xi
E mode optimal characteristics vector；βeIt indicates the weight factor of e modal characteristics vector and needs to meetWherein, exist
In the present invention, a total of both modalities which feature vector, i.e. G=2；
By expression formula 8) Lagrange duality transformation is carried out, its calculation formula is expression formulas 9):
βeBy user's self-setting, only need to guarantee
Wherein: aiAnd apRespectively training sample xiXpCorresponding Lagrange multiplier；yiAnd ypRespectively training sample
xiAnd xpClass label；It is training sample xiAnd xpThe core letter of e mode optimal characteristics vector
Number (Polynomial kernel function)；
For given object to be measured, its corresponding optimal characteristics vector is inputted into formula 10) in classifier, can obtain
To its class label:
Wherein: F (x) is the classification prediction result of object to be measured；Sgn () indicates sign function；It is trained sample
This xiWith the kernel function of object x e mode optimal characteristics vector to be measured；B is deviation, is obtained by training sample training.
The beneficial effects of the present invention are: the present invention is first respectively from rs-fMRI (resting-state functional
Magnetic resonance imaging, tranquillization state functional mri) and DTI data in extract brain function network characterization
And brain structural network feature, characteristic filter behaviour is then carried out to two category features using Kendall tau related coefficient respectively
Make；Secondly, making full use of multi-modal supplemental information to select the optimal feature subset of each mode, in K-support model
Added on number original bases same subject brain function network and brain structural network feature vector before the mapping after it is opposite away from
From constraint, to retain the relationship of different modalities characteristic, by keep from feature vector in brain function and structural network it
Between relationship, ensure that the sparsity of selection feature inside each mode；Finally, using multicore SVM for combining from different moulds
The feature that is selected in state and the prediction for carrying out image category label.Brain network characterization that the present invention selects at the same consider its
The correlation of brain function and brain structural level and disease, therefore the brain network characterization that the present invention is selected is as disease organism
The trustworthiness of label is higher, and the clinic study process for disclosing progression of disease has great importance.With it is previous
The feature that different modalities are extracted connects into the multi-modal fusion method phase that a long feature vector carries out subsequent analysis
Than the present invention considers the relationship between different modalities data and used this potential connection in feature selection step, mentions
Classification accuracy is risen；Using the present invention from selected optimal feature subset in brain function and two levels of structure as disease
The trustworthiness of the biomarker of disease is stronger.
Detailed description of the invention
Fig. 1 is that the present invention is based on the flow charts of the multi-modal fusion image classification method of mode distance restraint.
Specific embodiment
The embodiment of the present invention is described in detail with reference to the accompanying drawing, so that advantages and features of the invention can be easier to
In being readily appreciated by one skilled in the art, so as to make a clearer definition of the protection scope of the present invention.
The principle of the present invention is: extracted from rs-fMRI and DTI data respectively brain function and brain structural network feature to
Amount, makes full use of the interaction between two class mode, increases a new constraint on the basis of original K-support normal form and comes
The relationship for retaining different modalities characteristic, ensure that the sparsity of each modal characteristics with this.Finally, being used using multicore SVM
In the feature that selects from different modalities of combination and carry out the prediction of image category label.
Embodiment 1:
The invention discloses a kind of multi-modal fusion image classification methods based on mode distance restraint, as shown in Figure 1, institute
The method of stating includes the following steps:
The first step obtains data, specifically: obtaining the rs-fMRI data and DTI data of multiple tested objects, and carry out
Pretreatment, obtains pretreated rs-fMRI data and pretreated DTI data；
Second step, building brain function network characterization vector sum construct brain structural network feature vector, and Details as Follows:
Constructing brain function network characterization vector is constructed according to pretreated rs-fMRI data, specifically: being used
Automatic dissection tag template generates 90 cortex and subcutaneous core group region, and removes cerebellum part；It calculates any in each subject
Pearson correlation coefficients of the brain area to average time sequence；Node definition in brain function network is 90 cortex and subcutaneous
Core group region, the connection in brain function network are defined as any brain area to the Pearson correlation coefficients of average time sequence；To
The tranquillization state brain function network symmetrical matrix that one 90 × 90 is constructed for each subject, in removal symmetrical matrix diagonal line
On 90 diagonal elements after, extract all elements of lower Delta Region in symmetrical matrix as brain function network characterization vector；
Constructing brain structural network feature vector is constructed according to pretreated DTI data, specifically: using automatic dissection mark
It signs 90 cortex of template generation and subcutaneous core rolls into a ball region, and remove cerebellum part；Node definition in brain structural network is nine
Ten cortex and subcutaneous core roll into a ball region, and the connection in brain structural network is defined as according to FACT (Deterministic fiber
Assignment with the continuous tracking) white matter is fine between the arbitrary node (i.e. brain area) traced back to of algorithm
The quantity of dimension；To construct one 90 × 90 tranquillization state brain structural network symmetrical matrix for each subject, in removal pair
After claiming 90 diagonal elements on diagonal of a matrix, all elements of lower Delta Region in symmetrical matrix are extracted as brain structure
Network characterization vector；
Third step, characteristic filter operation, specifically: using Kendall tau related coefficient respectively come to brain function network
Feature vector and brain structural network feature vector carry out characteristic filter operation.
4th step, feature selecting operation, specifically: making full use of multi-modal supplemental information to select each mode
Optimal characteristics vector, brain function network and the brain structural network that same subject is added on K-support norm original base are special
Relative distance after levying vector before the mapping；
5th step, multi-mode classification analysis, specifically: the optimal characteristics vector based on training sample both modalities which is adopted
With multi-kernel support vector machine model training classifier；The optimal characteristics vector input of object both modalities which to be measured is trained
Classifier predicts its class label.
Details as Follows for preprocessing process in the present embodiment:
Rs-fMRI data are handled using SPM8 software and the tool box CONN, briefly, the pre-treatment step of image
Correction is moved including head, free-air correction, is registrated, normalized to the space MNI and space smoothing and handle, smoothing kernel FWHM=
8mm is rejected and is occurred the subject that the dynamic amplitude of head is greater than 2.5 degree greater than 2.5mm or rotation in any direction, by white matter, brain ridge
Liquid and head move coefficient and are considered as confounding factors；And use CompCor (component-based-noise-
Correction) method reduces the influence of the above non-nervous activity factor pair function NMR signal, followed by complete
Brain signal is returned to remove the negative correlation of a large amount of mistakes, and remaining time series carries out bandpass filtering (frequency 0.01-
0.08HZ) to reduce the influence of low-and high-frequency physiological noise；Finally, calculating seed region and other all voxel time serieses
Pearson correlation coefficient, and turned resulting related coefficient using fischer z-transform (Fisher z-transformation)
It is changed to and is just distributed very much.
DTI data are pre-processed and analyzed to be handled using the tool box PANDA.By by all diffusion-weighted figures
It is really corrected as being registrated to progress head movement and eddy loss on b=0 image.Then, using Stejskal and Tanner equation
Dispersion tensor element is calculated to obtain three characteristic values and feature vector.Then score anisotropy (FA) figure is generated.
According to data above, specific implementation the following steps are included:
Construct brain function network characterization vector the following steps are included:
1,90 cortex are generated using automatic dissection tag template (AAL) and subcutaneous core rolls into a ball region, and remove small brain
Point, using any brain area is calculated in each subject to the Pearson correlation coefficients of average time sequence, generate one 90 × 90 quiet
Cease state brain function network symmetrical matrix；
2, the node definition in brain function network is 90 cortex and subcutaneous core rolls into a ball region, the connection in brain function network
Any brain area is defined as to the Pearson correlation coefficients of average time sequence；
3,90 diagonal elements on symmetrical matrix diagonal line are removed；
4, extracted element is connected into the one-dimensional vector that a length is 4005 and (extracts lower trigonum in symmetrical matrix
The all elements in domain), as brain function network characterization vector.
Construct brain structural network feature vector the following steps are included:
1,90 cortex are generated using automatic dissection tag template (AAL) and subcutaneous core rolls into a ball region, and remove small brain
Point, the quantity of white matter fiber between any brain area pair is traced using FACT；
2, brain structural network interior joint is defined as 90 cortex and subcutaneous core group region, connects definition in brain function network
The quantity of white matter fiber between any brain area pair；
3,90 diagonal elements on symmetrical matrix diagonal line are removed；
4, extracted element is connected into the one-dimensional matrix that a length is 4005 and (extracts lower trigonum in symmetrical matrix
The all elements in domain), as brain function network characterization vector.
Two, characteristic filter process, specifically includes the following steps:
If patient's group has m and n sample with Normal group respectively.xijIt can indicate the brain function of i-th of j-th of sample
Energy network characterization or brain structural network feature, yiIndicate that (+1 is patient to the true tag data for needing to predict, -1 is normal right
According to).So i-th of brain function network characterization or the Kendall tau related coefficient of brain structural network feature are
Wherein:Indicate the Kendall tau related coefficient of the jth dimensional feature in e modal characteristics vector, m and n difference
The number of samples with Normal group is organized for patient,WithRespectively e mode reconciles to the number with anharmonic pair；I
Without considering relationship between two samples of same grouping,Therefore total sample logarithm amount is m × n；It reconciles to determining
Justice are as follows:
Meanwhile anharmonic pair is defined as:
Wherein: sgn is sign function,WithAs meet formula 2) and formula 3) sample to i-h's
Number；
Positive related coefficient ΓiIndicate that i-th of brain function network characterization or brain structural network feature compare in patient's grouping
It shows to significantly increase in normal control grouping, and a negative ΓiIndicate i-th of brain function network characterization or brain structure
Network characterization shows to be obviously reduced in patient is grouped.Then it is arranged according to the absolute value of Kendall tau related coefficient
Sequence, the brain function network characterization and brain structural network feature for selecting those to be more than a certain threshold value carry out next step operation.It is worth note
Meaning, in above step, brain function network characterization and brain structural network feature need to be separated and be operated.
Step 3.2, in order to ensure the corresponding relationship of brain network characterization, brain function network is filtered out with " overlapping " mode
Feature vector and brain structural network feature vector guarantee that brain function and brain the structural network connection between any brain area pair need
It comes across in subsequent feature selection step simultaneously.
Three, feature selecting operates, specifically includes the following steps:
Construct the objective function of multi-modal feature selection module are as follows:
Wherein:For the e modal characteristics vector matrix of S training sample, S
The dimension for the new e modal characteristics vector that training samples number and step 3.3 obtain is referred respectively to l；Indicate i-th
The new e modal characteristics vector that a training sample is obtained by step 3.3；we∈Rl×1Represent returning for e modal characteristics vector
Return coefficient vector, is parameter to be optimized；Y=[y1,y2,…,yi,…,yS]T∈RS×1For S training sample class label to
It measures, all elements are all 1 or -1 in Y；Subscript F indicates Frobenius normal form；λ1> 0 and λ2> 0, λ1And λ2Respectively control
The reservation degree of simulation sparsity and different modalities feature vector relationship is the regularization parameter of the sparse degree of Controlling model；r
To meet expression formula 5 in { 0 ..., k-1 }) condition unique integral；K meets k < l；It is vector weIn the i-th big member
Element；D is relative distance constraint, according to expression formula 6)；Expression formula 5) as follows:
Expression formula 5) as follows:
Wherein: D is relative distance constraint；WithThe new e mode that is obtained for i-th of training sample by third step and
T modal characteristics vector；
The objective function for solving multi-modal feature selection module, obtains we；Select weIn be greater than 0 element and correspond to the spy of dimension
Sign constitutes e mode optimal characteristics vector.
Four, multi-mode classification analysis
Step 5.1 is based on the following classifier of multi-kernel support vector machine model construction:
Wherein: F (x) is the class label of object x to be measured；Sgn () indicates sign function；yiFor training sample xiClass
Distinguishing label, aiRespectively training sample xiCorresponding Lagrange multiplier is parameter to be optimized；βeIndicate e modal characteristics to
It the weight factor of amount and needs to meetIt is training sample xiWith object x e mode optimal characteristics to be measured to
The kernel function of amount；B is deviation, is obtained by training sample training；
Step 5.2 passes through the following objective function of solution, obtains parameter aiValue:
Wherein:It is training sample xiAnd xpThe kernel function of e mode optimal characteristics vector；
Step 5.3, the parameter a for obtaining the objective function in solution procedure 5.2iValue substitute into step 5.1 in classification
Device；For given object to be measured, the optimal characteristics vector of its both modalities which is inputted into classifier, obtains its class label.Such as
Fruit F (x)=1, then object x to be measured is Disease, conversely, then object x to be measured is normal.
Claims (3)
1. a kind of multi-modal fusion image classification system based on mode distance restraint, which is characterized in that comprise the following modules:
Data acquisition module for obtaining the rs-fMRI data and DTI data of multiple tested objects, and pre-processes it,
Obtain pretreated rs-fMRI data and pretreated DTI data；Wherein tested object includes the sample of known class label
The object to be measured of this object and unknown class label；
Feature vector constructs module, for constructing the feature vector of both modalities which, i.e. brain function respectively for each tested object
Network characterization vector sum brain structural network feature vector；
For each tested object, the method that constructs its brain function network characterization vector are as follows: pretreated according to its first
The part that cerebellum is removed in its full brain is divided into 90 cortex and subcutaneous using automatic dissection tag template by rs-fMRI data
Core group region, i.e. 90 brain areas；Then each of which brain area is defined as a node in its brain function network, by its each brain
Area is defined as the connection relationship in its brain function network respective nodes between the Pearson correlation coefficients of average time sequence；Again with
Connection relationship in its brain function network between each node is matrix element, and one 90 × 90 tranquillization state brain function is constructed for it
Network symmetrical matrix；90 elements on the symmetrical matrix diagonal line are finally removed, lower Delta Region in symmetrical matrix is extracted
All elements are as its brain function network characterization vector, vector dimension 4005；
For each tested object, the method that constructs its brain structural network feature vector are as follows: pretreated according to its first
The part that cerebellum is removed in its full brain is divided into 90 cortex and subcutaneous core group using automatic dissection tag template by DTI data
Region, i.e. 90 brain areas；Each of which brain area is defined as a node in its brain structural network, it will be white between its each brain area pair
Matter fiber number is defined as the connection relationship in its brain structural network between respective nodes；Again with each node in its brain structural network
Between connection relationship be matrix element, construct one 90 × 90 tranquillization state brain structural network symmetrical matrix for it；Finally remove
90 elements on the symmetrical matrix diagonal line extract all elements of lower Delta Region in symmetrical matrix as its brain Structure Network
Network feature vector, vector dimension 4005；
Characteristic filter module, for being based on Kendall tau related coefficient and " overlapping " mode to tested object both modalities which
Feature vector carries out characteristic filter operation, obtains the new feature vector of tested object both modalities which；
Feature selection module, for adding the feature of same tested object both modalities which on K-support norm original base
Vector before the mapping after relative distance constraint, the objective function of multi-modal feature selection module is constructed, to two kinds of tested object
The new feature vector of mode carries out feature selecting, to obtain the optimal characteristics vector of tested object both modalities which；
Multi-mode categorization module, for the optimal characteristics vector based on training sample both modalities which, using multicore supporting vector
Machine model training classifier；The optimal characteristics vector of object both modalities which to be measured is inputted into trained classifier, predicts its class
Distinguishing label；
Characteristic filter module carry out characteristic filter operation specifically includes the following steps:
Step 3.1 obtains patient's group and Normal group sample data first, is used in combinationWithIndicate i-th of sample object and
Jth dimensional feature in h-th of sample object e modal characteristics vector, the i.e. e of i-th of sample object and h-th of sample object
Jth in modal characteristics vector ties up element, wherein e=1, and the 2, the 1st mode is brain function network, and the 2nd mode is brain structural network,
J=1,2 ..., 4005, yiAnd yhThe class label of i-th of sample object and h-th of sample object is respectively indicated, class label is
1 indicate the sample object be patient group sample object, class label be -1 indicate the sample object be Normal group sample
This object, further according to formula 1) calculate Kendall tau related coefficient:
Wherein:Indicate that the Kendall tau related coefficient of the jth dimensional feature in e modal characteristics vector, m and n are respectively disease
Sample object number in people's group and Normal group,WithRespectively e mode reconciles to the number with anharmonic pair；It reconciles
Pair is defined as:
Anharmonic pair is defined as:
Wherein: sgn is sign function, yi≠yh, yi≠yh；WithAs meet formula 2) and formula 3) sample object pair
The number of i-h；
Then rightJ=1,2 ..., 4005 are ranked up according to order of magnitude；It chooses corresponding in e modal characteristics vectorMore than the characteristic dimension of given threshold；
Step 3.2, in order to ensure the corresponding relationship of brain network characterization, step 3.1 is chosen with " overlapping " mode feature into
Row further screening, to guarantee connection relationship in brain function network or brain function network between any node pair appearance simultaneously or not
It appears in subsequent feature selection step；Screening technique are as follows: if jth dimensional feature and the 2nd mode in the 1st modal characteristics vector
Jth dimensional feature is selected by step 3.1 simultaneously in feature vector, then retains this feature dimension；Otherwise, if the 1st modal characteristics vector
In jth dimensional feature and the 2nd modal characteristics vector in jth dimensional feature only one or none selected by step 3.1, then
Filter out this feature dimension；
Step 3.3, to the feature vector of each tested object both modalities which, only retain the characteristic dimension that step 3.2 filters out,
Obtain the new feature vector of its both modalities which.
2. the multi-modal fusion image classification system according to claim 1 based on mode distance restraint, which is characterized in that
The objective function of the multi-modal feature selection module of feature selection module building are as follows:
Wherein:For the e modal characteristics vector matrix of S training sample, S and l divide
The dimension of the new feature vector of the e mode that training samples number and step 3.3 obtain is not referred to；Indicate i-th of training
The sample feature vector new by e mode that step 3.3 obtains；we∈Rl×1Represent the regression coefficient of e modal characteristics vector to
Amount is parameter to be optimized；Y=[y1,y2,…,yi,…,yS]T∈RS×1For the class label vector of S training sample, institute in Y
Some elements are all 1 or -1；Subscript F indicates Frobenius normal form；λ1> 0 and λ2> 0, λ1And λ2Respectively Controlling model is dilute
Dredge the regularization parameter of the reservation degree of property and different modalities feature vector relationship；R is to meet expression formula 5 in { 0 ..., k-1 })
Condition unique integral；K meets k < l；It is vector weIn the i-th big element；D is relative distance constraint, according to expression
Formula 6)；Expression formula 5) as follows:
Expression formula 5) as follows:
The objective function for solving multi-modal feature selection module, obtains we；Determine weIn be greater than the corresponding characteristic dimension of 0 element, from
The feature of these dimensions is selected to constitute its e mode optimal characteristics vector in the new feature vector of tested object e mode.
3. the multi-modal fusion image classification system according to claim 2 based on mode distance restraint, which is characterized in that
Multi-mode categorization module predict class label specifically includes the following steps:
Step 5.1 is based on the following classifier of multi-kernel support vector machine model construction:
Wherein: F (x) is the class label of object x to be measured；Sgn () indicates sign function；yiFor training sample xiClassification mark
Label, aiRespectively training sample xiCorresponding Lagrange multiplier is parameter to be optimized；βeIndicate the power of e modal characteristics vector
It repeated factor and needs to meet It is training sample xiWith the core of object x e mode optimal characteristics vector to be measured
Function；B is deviation, is obtained by training sample training；
Step 5.2 passes through the following objective function of solution, obtains parameter aiValue:
0≤ai≤ C, i=1,2 ..., S
Wherein: C is penalty factor,It is training sample xiAnd xpThe kernel function of e mode optimal characteristics vector；
Step 5.3, the parameter a for obtaining the objective function in solution procedure 5.2iValue substitute into step 5.1 in classifier；It is right
In given object to be measured, the optimal characteristics vector of its both modalities which is inputted into classifier, obtains its class label.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810073841.3A CN108345903B (en) | 2018-01-25 | 2018-01-25 | A kind of multi-modal fusion image classification method based on mode distance restraint |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810073841.3A CN108345903B (en) | 2018-01-25 | 2018-01-25 | A kind of multi-modal fusion image classification method based on mode distance restraint |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108345903A CN108345903A (en) | 2018-07-31 |
| CN108345903B true CN108345903B (en) | 2019-06-28 |
Family
ID=62961602
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810073841.3A Active CN108345903B (en) | 2018-01-25 | 2018-01-25 | A kind of multi-modal fusion image classification method based on mode distance restraint |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108345903B (en) |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109241903B (en) * | 2018-08-30 | 2023-08-29 | 平安科技（深圳）有限公司 | Sample data cleaning method, device, computer equipment and storage medium |
| CN109344889B (en) * | 2018-09-19 | 2021-01-29 | 深圳大学 | Brain disease classification apparatus, user terminal, and computer-readable storage medium |
| CN109754414A (en) * | 2018-12-27 | 2019-05-14 | 上海商汤智能科技有限公司 | Image processing method, device, electronic equipment and computer readable storage medium |
| CN109770932B (en) * | 2019-02-21 | 2022-04-29 | 河北工业大学 | Processing method of multi-modal brain nerve image features |
| CN110263791B (en) * | 2019-05-31 | 2021-11-09 | 北京京东智能城市大数据研究院 | Method and device for identifying functional area |
| CN110210403B (en) * | 2019-06-04 | 2022-10-14 | 电子科技大学 | SAR image target identification method based on feature construction |
| CN110298364B (en) * | 2019-06-27 | 2021-06-15 | 安徽师范大学 | Multi-threshold-value multi-task-based feature selection method for functional brain network |
| CN110473635B (en) * | 2019-08-14 | 2023-02-28 | 电子科技大学 | Analysis method of relation model of teenager brain structure network and brain function network |
| CN113616184B (en) * | 2021-06-30 | 2023-10-24 | 北京师范大学 | Brain network modeling and individual prediction method based on multi-mode magnetic resonance image |
| CN114242236A (en) * | 2021-12-18 | 2022-03-25 | 深圳先进技术研究院 | Structure-function brain network bidirectional mapping model construction method and brain network bidirectional mapping model |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103325119A (en) * | 2013-06-27 | 2013-09-25 | 中国科学院自动化研究所 | Default state brain network center node detecting method based on modality fusion |
| CN105046709B (en) * | 2015-07-14 | 2018-06-29 | 华南理工大学 | A kind of brain age analysis method based on nuclear magnetic resonance image |
| US20170270832A1 (en) * | 2016-03-15 | 2017-09-21 | Lauren Hill | Brain exhibit or model, and method of using or deploying same |
| CN105957047B (en) * | 2016-05-06 | 2019-03-08 | 中国科学院自动化研究所 | There is the multi-modal brain image fusing method of supervision |
| CN106667490B (en) * | 2017-01-09 | 2017-12-29 | 北京师范大学 | Relationship analysis method between the data of individual difference between a kind of tested object based on magnetic resonance brain image |
-
2018
- 2018-01-25 CN CN201810073841.3A patent/CN108345903B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN108345903A (en) | 2018-07-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108345903B (en) | A kind of multi-modal fusion image classification method based on mode distance restraint | |
| Ding et al. | Classification and quantification of neuronal fiber pathways using diffusion tensor MRI | |
| Ahmed et al. | Single volume image generator and deep learning-based ASD classification | |
| EP1946701B1 (en) | Brain function analysis method and brain function analysis program | |
| CN107944490B (en) | Image classification method based on semi-multimodal fusion feature reduction framework | |
| CN103800011B (en) | A kind of brain district based on functional mri effective connectivity analyzes system | |
| CN109528197A (en) | The individuation prediction technique and system of across the Species migration carry out mental disease of monkey-people based on brain function map | |
| JP2017076193A (en) | Brain activity analysis device, brain activity analysis method and brain activity analysis program | |
| Bassett et al. | Network methods to characterize brain structure and function | |
| CN109816630A (en) | FMRI visual coding model building method based on transfer learning | |
| US7349728B2 (en) | Evaluation of images of the brain obtained by means of functional magnetic resonance tomography | |
| de Beeck et al. | Introduction to human neuroimaging | |
| CN115359013A (en) | Brain age prediction method and system based on diffusion tensor imaging and convolutional neural network | |
| CN116807447A (en) | Brain age prediction modeling method, cognition improving method and system of dynamic brain network | |
| Hart et al. | Connections, tracts, fractals, and the rest: a working guide to network and connectivity studies in neurosurgery | |
| Vuksanović | Brain morphological and functional networks: Implications for neurodegeneration | |
| Ren et al. | Modeling hierarchical spatial and temporal patterns of naturalistic fMRI volume via volumetric deep belief network with neural architecture search | |
| Huntenburg | A core organizing axis of the human cerebral cortex | |
| Wein | Applications of spatio-temporal graph neural network models for brain connectivity analysis | |
| Yu et al. | Autism spectrum disorder identification with brain functional connectivity map extracted from EEG signals | |
| Qu | Graph Deep Learning for Multimodal Neuroimaging Study | |
| Bernhard et al. | Statistical parameter estimation and signal classification in cardiovascular diagnosis | |
| Ortiz-Rios et al. | Dynamic reconfiguration of macaque brain networks during free-viewing of natural scenes | |
| Bracamonte | MODELING OF PATIENT-SPECIFIC PERIAORTIC MECHANICS AND PULMONARY ARTERY HEMODYNAMICS BASED ON PHASE-CONTRAST MAGNETIC RESONANCE IMAGING SEQUENCES. | |
| He et al. | Research on computer aided diagnosis based on artificial intelligence |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |