CN116226712A - Online learner concentration monitoring method, system and readable storage medium - Google Patents
Online learner concentration monitoring method, system and readable storage medium Download PDFInfo
- Publication number
- CN116226712A CN116226712A CN202310194137.4A CN202310194137A CN116226712A CN 116226712 A CN116226712 A CN 116226712A CN 202310194137 A CN202310194137 A CN 202310194137A CN 116226712 A CN116226712 A CN 116226712A
- Authority
- CN
- China
- Prior art keywords
- learner
- brain
- image
- sight
- line
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/18—Eye characteristics, e.g. of the iris
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Ophthalmology & Optometry (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
Abstract
The embodiment of the application provides a method, a system and a readable storage medium for monitoring the concentration degree of an online learner, wherein the method comprises the steps of acquiring face monitoring images of learners at different moments and brain EEG signals; performing sight line estimation processing based on the face monitoring image, and determining the sight line angle change condition of the learner at different moments; performing signal change analysis processing based on the brain EEG signals, and determining the brain electrical signal change conditions of the brain EEG signals of learners at different moments; and (5) judging the concentration degree of the learner by combining the change condition of the angle of the line of sight and the change condition of the electroencephalogram. The implementation of the method can improve the concentration detection accuracy of online learners.
Description
Technical Field
The application relates to the technical field of intelligent human-computer interaction, in particular to an on-line learner concentration monitoring method, an on-line learner concentration monitoring system and a readable storage medium.
Background
With the development of education technology, consider the influence of environmental factors such as geographical position, time on-line teaching mode can be solved, the teaching mode has gradually developed the mode of on-line teaching from traditional off-line teaching before. The influence of epidemic situation is developed in a vigorous manner at one time, but the traditional online teaching is carried out by the teaching personnel, meanwhile, the concentration degree of the learner is required to be paid attention to, and the requirement of online teaching can not be met under the condition that the site condition is limited.
At present, an on-line teaching platform used in universities provides a concentration degree judging mode, such as a snapshot by utilizing a camera on intelligent equipment, and then the concentration degree of a learner is detected manually in the background. In addition, some important courses adopt a double-machine-position operation mode, when a computer screen is started, a student needs to set up another intelligent device at the rear, and the camera of the intelligent device is utilized to shoot simultaneously, and although the concentration degree of a detected learner carries out various settings on the concentration degree reduction of the learner, the concentration degree detection of the learner on line is difficult to obtain ideal results no matter the accuracy or the operability.
Disclosure of Invention
An object of the embodiments of the present application is to provide a method, a system and a readable storage medium for monitoring the concentration of an online learner, which can improve the accuracy of detecting the concentration of the online learner.
The embodiment of the application also provides an online learner concentration monitoring method, which comprises the following steps:
s1, acquiring face monitoring images of learners at different moments and brain EEG signals;
s2, performing sight line estimation processing based on the face monitoring image, and determining the sight line angle change conditions of learners at different moments;
s3, performing signal change analysis processing based on the brain EEG signals, and determining the brain electrical signal change conditions of the brain EEG signals of learners at different moments;
and S4, integrating the vision angle change condition and the electroencephalogram signal change condition, and judging the concentration degree of the learner.
In a second aspect, an embodiment of the present application further provides an on-line learner concentration monitoring system, where the system includes a data acquisition module, a human eye line of sight angle analysis module, an electroencephalogram signal analysis module, and a concentration judgment module, where:
the data acquisition module is used for acquiring face monitoring images of learners at different moments and brain EEG signals;
the human eye sight angle analysis module is used for carrying out sight estimation processing based on the human face monitoring image and determining the sight angle change condition of learners at different moments;
the electroencephalogram signal analysis module is used for carrying out signal change analysis processing based on the brain EEG signals and determining the electroencephalogram signal change conditions of the brain EEG signals of learners at different moments;
and the concentration degree judging module is used for synthesizing the vision angle change condition and the electroencephalogram signal change condition and judging the concentration degree of the learner.
In a third aspect, embodiments of the present application further provide a readable storage medium, where the readable storage medium includes an on-line learner concentration monitoring method program, where the on-line learner concentration monitoring method program, when executed by a processor, implements the steps of an on-line learner concentration monitoring method according to any one of the above.
As can be seen from the above, according to the on-line learner concentration monitoring method, system and readable storage medium provided in the embodiments of the present application, on one hand, line of sight estimation processing is performed based on the face monitoring image, so as to determine the apparent behavior of the change condition of the line of sight angle of the learner at different moments. On the other hand, based on the brain EEG signals, signal change analysis processing is carried out, the implicit behavior of the brain EEG signal change condition of the learner at different moments is determined, the online class learner concentration detection is carried out by combining the explicit behavior and the implicit behavior, and the learner with reduced concentration in the online class can be accurately found in real time, so that the learning efficiency of the learner can be effectively improved, the interaction frequency between the teacher and the learner can be increased, and the teaching quality is improved.
Additional features and advantages of the application will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the embodiments of the application. The objectives and other advantages of the application will be realized and attained by the structure particularly pointed out in the written description and claims thereof as well as the appended drawings.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of an on-line learner concentration monitoring method according to an embodiment of the present application;
fig. 2 is a schematic diagram of an actual application scenario of an on-line learner concentration monitoring method according to an embodiment of the present application;
FIG. 3 is a schematic diagram of a model for analyzing changes in angle of a learner's line of sight and changes in brain electrical signals;
FIG. 4 is a regularized two-dimensional schematic;
FIG. 5 is a flowchart illustrating an overall implementation of an on-line learner concentration monitoring method according to an embodiment of the present disclosure;
fig. 6 is a schematic structural diagram of an on-line learner concentration monitoring system according to an embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and completely with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. The components of the embodiments of the present application, which are generally described and illustrated in the figures herein, may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present application, as provided in the accompanying drawings, is not intended to limit the scope of the application, as claimed, but is merely representative of selected embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, are intended to be within the scope of the present application.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures. Meanwhile, in the description of the present application, the terms "first", "second", and the like are used only to distinguish the description, and are not to be construed as indicating or implying relative importance.
Referring to fig. 1, fig. 1 is a flowchart of an on-line learner concentration monitoring method according to some embodiments of the present application. The method comprises the following steps:
step S1, face monitoring images of learners at different moments and brain EEG signals are acquired.
And S2, performing sight line estimation processing based on the face monitoring image, and determining the sight line angle change condition of the learner at different moments.
And step S3, carrying out signal change analysis processing based on the brain EEG signals, and determining the brain electrical signal change conditions of the brain EEG signals of learners at different moments.
And S4, integrating the vision angle change condition and the electroencephalogram signal change condition, and judging the concentration degree of the learner.
It should be noted that, referring to fig. 2, in the actual application scenario of the above method, in the present embodiment, a head-mounted brain wave sensor is used to collect brain EEG signals of a learner, and a camera carried by an intelligent device is used to collect face monitoring images of the learner at different moments.
Considering that EEG is an important expression form of the input level of the learner, the true state that the learner with the concentration reduction tendency wants to hide can be effectively discovered by analyzing the apparent behavior of the change angle of the line of sight of the learner in the online listening class. In addition, the implicit action of the brain activity degree of the learner can be obtained through analyzing the EEG signal of the learner, and when the learner tends to have concentration degree reduced, the vision is erratic and stagnant, so that the EEG signal of the learner is insensitive, and the brain activity degree is not obvious.
In summary, in the present embodiment, the on-line learner concentration detection is performed by the explicit behavior of the change of the angle of the line of sight of the learner and the implicit behavior of the EEG signal of the learner, so as to achieve the effect of improving the accuracy of the on-line learner concentration detection.
As can be seen from the above, according to the method for monitoring the concentration of the online learner disclosed in the present application, on one hand, the line of sight estimation process is performed based on the face monitoring image, so as to determine the apparent behavior of the change condition of the line of sight angle of the learner at different moments. On the other hand, based on the brain EEG signals, signal change analysis processing is carried out, the implicit behavior of the brain EEG signal change condition of the learner at different moments is determined, the online class learner concentration detection is carried out by combining the explicit behavior and the implicit behavior, and the learner with reduced concentration in the online class can be accurately found in real time, so that the learning efficiency of the learner can be effectively improved, the interaction frequency between the teacher and the learner can be increased, and the teaching quality is improved.
In one embodiment, in step S2, the estimating the line of sight based on the face monitoring image, and determining the line of sight angle change of the learner at different moments includes:
step S21, based on the face monitoring image, combining a preset HRNETV2 feature extraction model to perform human eye feature extraction processing, and obtaining a human eye feature extraction diagram.
Specifically, the obtained human eye feature extraction map may further refer to fig. 3. The HRNETV2 feature extraction model is divided into 5 phases, namely Stem, stage1, stage2, stage3, stage4.
Specifically, when implemented, the input image will first be convolved twice using a 3 x 3 convolution operation with a step size of 2, such that the height (H) and width (W) of the image become H/4 and W/4 in size.
Then, 4 basic blocks or bottleblocks are used to perform feature extraction, and the obtained output is input to stage 1.
Then, a low resolution branch is generated, and each branch still uses 4 basic blocks or bottleblocks to perform feature extraction, and then repeated multi-scale fusion is performed to obtain a final output, wherein the final output is input into stage 2.
The processing steps in Stage2 and Stage4 are the same as those described above, and are not specifically described at present, and can be understood by referring to the above steps.
Finally, four branch results of the same size are connected, and the number of channels is converted into the number of classes of semantic segmentation by convolution of 1×1.
S22, inputting the human eye feature map into a random forest sight estimation model for sight estimation processing to obtain an initial sight angle change condition, wherein in the processing process, on one hand, expected calculation is carried out on the hidden variable set to maximize observed data, and on the other hand, an L2 regularization term is added
The problem of overfitting is avoided.
Specifically, the calculation of the hidden variable set will be described in detail in the following steps, and not described in detail at present.
In addition, the loss function L added to the L2 regularization term is:
wherein (1)>
For the L2 regularization term, λ is the regularization parameter. To prevent the parameter from being too large and achieve the purpose of limiting the number w, the current embodiment requires:
(C is the weight vector w and its transpose matrix). As can be seen from the regularized two-dimensional diagram shown in FIG. 4, when the arrow a and the arrow b are perpendicular to each other, this means +.>
No component is found on the tangent to W, at which time W stops moving to the optimal solution W in Is a position of (c).
Therefore, in the practical implementation, another consideration will be given
And the tangent to w remains vertical, i.e. +.>
Parallel to the direction of w, it is possible to obtain: />
The idea according to the optimization algorithm can be further determined: when the gradient is 0, the function takes the optimal value. At a known position
In one embodiment, λ=1.5 is taken, so that the problem that the random forest is prone to over-fitting can be solved.
Step S23, adopting DNet network based on differential convolution to perform accurate line-of-sight estimation, wherein in the estimation process, the difference d between the predicted test image I and the reference image F is combined p (I, J), and the true gaze value g gt (F) To predict the final gaze direction.
Specifically, accurate line-of-sight estimation using a dnat network based on differential convolution will be described in detail in the subsequent steps, and is not described in detail at present.
In one embodiment, an L2 regularization term and a DNet network based on differential convolution are fused to perform accurate sight estimation, so that the prediction accuracy of the sight direction is further ensured, and the accuracy and the robustness of the model on a sight estimation task are improved.
In one embodiment, in step S21, before performing the human eye feature extraction process, the method further includes:
step S211, performing graying processing on the face monitoring image to obtain a graying image.
Step S222, performing multi-scale transformation on the graying image to obtain a face image pyramid.
Step S223, performing interpolation processing on the face image pyramid by combining the following bicubic interpolation expression to obtain a corresponding interpolation image:
f(i+u，j+v)＝ABC；
where f (i, j) is the pixel gray value of the image, u is the fractional part of i, v is the fractional part of j, and S (x) is a preset base interpolation function.
It should be noted that, in order to fit the data, in the current step, the base interpolation function may be represented by S (w), where the formula is:
where a is a constant coefficient, typically taking a= -1.
And step S224, inputting the interpolation image into a P-Net network and an O-Net network which are connected in sequence, and obtaining a target face detection frame for positioning the face image after non-maximum value inhibition processing.
Specifically, when performing non-maximum suppression processing using a P-Net network, the candidate box is first corrected using a boundary regression vector. And combining the overlapped candidate frames by using a non-maximum suppression mechanism to obtain a plurality of coarse face detection frames.
Further, after obtaining a plurality of coarse face detection frames outputted from the P-Net network, the overlapping candidate frames are merged via the O-Net network using a non-maximum suppression mechanism. At this time, a plurality of more accurate face detection frames can be obtained.
In one embodiment, in step S22, the desired computation of the set of hidden changes to maximize the observed data is performed by the steps comprising:
step S221, the observed variable set, the hidden variable set and the model parameters are M, N and phi respectively, and the iterative optimization process is entered.
Step S222, when iterating to the t-th step, the model parameters are converted into phi t The probability distribution of the hidden variable set N at this time is p (N|M, phi) t )。
Step S223, let the optimization function be Q, wherein the expression of the optimization function Q is set as:
x is a set, Z is a hidden variable.
Step S224, based on the formula Φ t+1 ＝arg N maxQ(Φ|Φ t ) Finding parameters maximizes the expected likelihood, where Φ t+1 Is an estimate of the parameter.
In the above embodiments, the convergence of the EM algorithm (i.e., the application of the maximum expected algorithm) may ensure that the iteration approaches at least a local maximum, further facilitating the accuracy of line-of-sight estimation by accurately locating the data so that the observed data is maximized.
In one embodiment, in step S23, the accurate line-of-sight estimation using the dnat network based on differential convolution includes:
in step S231, a RELU function is selected as an activation function of the convolution layer and the full connection layer, and the formula is expressed as: f (x) =max (0, x), where x is input data and f (x) is output after RELU function processing.
Step S232, use d p (I, J) represents the difference in gaze predicted by the differential network, then the loss function L d The method comprises the following steps:
wherein D is k For a subset of training set D, only the image of one eye of the kth person is included, g gt (I) G is the true value of eye gaze gt (J) Is a predicted value of eye gaze.
Step S233, predicting the difference d between the test image I and the reference image F through the differential network p (I, J) and combining the real gaze valuesg gt (F) The final gaze direction is predicted by the following formula:
wherein D is c For a calibration set of reference images, w (·) represents weighting the importance of each prediction.
Specifically, after step S233, the obtained accurate gaze estimation result is classified, and converted into gaze estimation angles Pitch angle and Yaw angle, and the gaze change angle of the learner in the online listening class is analyzed through the Pitch angle and the Yaw angle.
In the present embodiment, a softmax function is applied at the output layer of the differential convolution network, and the gaze estimation result is converted into the gaze estimation angles Pitch angle and Yaw angle based on the softmax function.
In particular, the softmax function used at the time of implementation is expressed as follows:
wherein a is k For the kth input signal, n is the number of output nodes, a i Is the i-th input signal.
In one embodiment, in step S3, the performing signal change analysis based on the brain EEG signal includes:
and S31, adopting a DWT-PECSP algorithm to sequentially perform signal characteristic extraction and electroencephalogram signal classification and identification processing.
And step S32, performing power spectrum conversion on the brain wave signals subjected to classification and identification, and obtaining the power of the brain wave signals based on the obtained power spectrums corresponding to different brain wave types.
And step S33, observing the brain electric power of the learner, and representing the brain activity degree of the learner based on the brain electric power.
In particular, the present implementation may be further understood with reference to fig. 2, fig. 5, and subsequent implementation steps, which are not specifically described.
In one embodiment, in step S31, the signal feature extraction and the electroencephalogram classification and identification processing are sequentially performed by using a DWT-PECSP algorithm, including:
step S311, wavelet decomposition is carried out on the EEG signal subjected to threshold value preliminary screening, and the u-rhythm and beta-rhythm frequency band signals obtained by discrete wavelet transformation reconstruction are used as input vectors of CSP algorithm and are marked as X 1 ∈R N×T And X 2 ∈R N×T Wherein N is the number of channels, and T is the number of sampling points.
Specifically, in the present embodiment, the Mallat algorithm is used to perform wavelet decomposition on the EEG signal, and the calculation formula includes:
wherein l is the number of layers of wavelet decomposition, A l D as an approximation component of the last layer j The detail components corresponding to each layer.
Based on the above embodiment, it should be noted that when N<At T, X 1 And X 2 Can be expressed as:
step S312, find X 1 And X 2 Normalized covariance matrix R 1 And R is 2 Based on mean covariance
And->
And, obtaining the corresponding hybrid covariance R.
Specifically, the calculation formula of the hybrid covariance includes:
wherein R is 1 And R is 2 Can be obtained by the following meansThe formula is further calculated by:
step S313, carrying out feature decomposition based on the mixed covariance R, and determining a feature value lambda and a feature vector U.
Specifically, specific forms of feature decomposition include: r=uλu T 。
Step S314, based on the formula
Determining a whitening matrix P and based on the whitening matrix P to a covariance matrix R 1 And R is 2 Transformation and principal component decomposition are performed.
Specifically, the covariance matrix R is based on the whitening matrix P by using the following formula 1 And R is 2 Performing transformation and principal component decomposition:
wherein B is 1 Is S 1 Feature vector of B 2 Is S 2 Is described.
Step S315, based on the formula w=b T P determines the projection matrix W and transmits the signal X 1 And X 2 Filtering by a projection matrix W to obtain a matrix Z 1 And Z 2 。
Specifically, the matrix Z can be obtained by the following formula 1 And Z 2 ：
Z 1 ＝W*X 1 ，Z 2 ＝W*X 2 。
Step S316, electroencephalogram signal classification and identification are carried out by using a support vector machine.
Specifically, the support vector machine is a generalized linear classifier for binary classification of data according to a supervised learning mode, and the decision boundary is the maximum margin hyperplane for solving the learning sample.
In one embodiment, in step S32, the power of the electroencephalogram signal is obtained by the following equation:
wherein i is different types of brain waves, including delta wave, theta wave, alpha wave and beta wave; s is S i (t) power spectrums corresponding to different types of brain waves respectively; x (k) is the amplitude of 4 brain wave discrete points; n is the type number of brain waves.
Specifically, in the present embodiment, the brain wave signals with the classification and identification completed are subjected to power spectrum conversion. And then, recording the power spectrum of the electroencephalogram signal for one step to obtain the power of the electroencephalogram signal, wherein the expression is shown in the formula. In particular, during implementation, the brain electric power of a learner is observed, and the brain activity degree of the learner is represented by the brain electric power. Wherein, the beta wave is most sensitive to brain activity change, and the larger the beta wave power is, the more excited the brain of the learner is represented; conversely, if the β -wave power is smaller, the learner's brain is more suppressed.
Referring to fig. 6, the system for monitoring the concentration of an on-line learner disclosed in the present application includes a data acquisition module, a human eye sight angle analysis module, an electroencephalogram signal analysis module and a concentration judgment module, wherein:
the data acquisition module is used for acquiring face monitoring images of learners at different moments and brain EEG signals.
The human eye sight angle analysis module is used for carrying out sight estimation processing based on the human face monitoring image and determining the sight angle change condition of learners at different moments.
The electroencephalogram signal analysis module is used for carrying out signal change analysis processing based on the brain EEG signals and determining the electroencephalogram signal change conditions of the brain EEG signals of learners at different moments.
And the concentration degree judging module is used for synthesizing the vision angle change condition and the electroencephalogram signal change condition and judging the concentration degree of the learner.
In one embodiment, the modules in the system are further configured to perform the method of any of the alternative implementations of the above embodiments.
From the above, according to the on-line learner concentration monitoring system disclosed by the application, on one hand, the line of sight estimation processing is performed based on the face monitoring image, and the apparent behavior of the line of sight angle change condition of the learner at different moments is determined. On the other hand, based on the brain EEG signals, signal change analysis processing is carried out, the implicit behavior of the brain EEG signal change condition of the learner at different moments is determined, the online class learner concentration detection is carried out by combining the explicit behavior and the implicit behavior, and the learner with reduced concentration in the online class can be accurately found in real time, so that the learning efficiency of the learner can be effectively improved, the interaction frequency between the teacher and the learner can be increased, and the teaching quality is improved.
The present application provides a readable storage medium which, when executed by a processor, performs the method of any of the alternative implementations of the above embodiments. The storage medium may be implemented by any type of volatile or nonvolatile Memory device or combination thereof, such as static random access Memory (Static Random Access Memory, SRAM), electrically erasable Programmable Read-Only Memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), erasable Programmable Read-Only Memory (Erasable Programmable Read Only Memory, EPROM), programmable Read-Only Memory (PROM), read-Only Memory (ROM), magnetic Memory, flash Memory, magnetic disk, or optical disk.
On the other hand, the readable storage medium performs a line-of-sight estimating process based on the face monitoring image, and determines an explicit behavior of a change in the line-of-sight angle of the learner at different times. On the other hand, based on the brain EEG signals, signal change analysis processing is carried out, the implicit behavior of the brain EEG signal change condition of the learner at different moments is determined, the online class learner concentration detection is carried out by combining the explicit behavior and the implicit behavior, and the learner with reduced concentration in the online class can be accurately found in real time, so that the learning efficiency of the learner can be effectively improved, the interaction frequency between the teacher and the learner can be increased, and the teaching quality is improved.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. The above-described apparatus embodiments are merely illustrative, for example, the division of the units is merely a logical function division, and there may be other manners of division in actual implementation, and for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some communication interface, device or unit indirect coupling or communication connection, which may be in electrical, mechanical or other form.
Further, the units described as separate units may or may not be physically separate, and units displayed as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
Furthermore, functional modules in various embodiments of the present application may be integrated together to form a single portion, or each module may exist alone, or two or more modules may be integrated to form a single portion.
In this document, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions.
The foregoing is merely exemplary embodiments of the present application and is not intended to limit the scope of the present application, and various modifications and variations may be suggested to one skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.
Claims (10)
1. An on-line learner concentration monitoring method, comprising the steps of:
s1, acquiring face monitoring images of learners at different moments and brain EEG signals;
s2, performing sight line estimation processing based on the face monitoring image, and determining the sight line angle change conditions of learners at different moments;
s3, performing signal change analysis processing based on the brain EEG signals, and determining the brain electrical signal change conditions of the brain EEG signals of learners at different moments;
and S4, integrating the vision angle change condition and the electroencephalogram signal change condition, and judging the concentration degree of the learner.
2. The method according to claim 1, wherein in step S2, the estimating the line of sight based on the face monitoring image, and determining the change of the angle of the line of sight of the learner at different times, includes:
s21, performing human eye feature extraction processing by combining a preset HRNETV2 feature extraction model based on the human face monitoring image to obtain a human eye feature extraction diagram;
s22, inputting the human eye feature map into a random forest sight estimation model for sight estimation processing to obtain an initial sight angle change condition, wherein in the processing process, on one hand, expected calculation is carried out on the hidden variable set to maximize observed data, and on the other hand, an L2 regularization term is added
The problem of overfitting is avoided;
s23, adopting a DNet network based on differential convolution to perform accurate line-of-sight estimation, wherein in the estimation process, the difference d between the predicted test image I and the reference image F is combined p (I, J), andreal gaze value g gt (F) To predict the final gaze direction.
3. The method according to claim 2, characterized in that in step S21, before performing the human eye feature extraction process, the method further comprises:
s211, carrying out graying treatment on the face monitoring image to obtain a graying image;
s222, carrying out multi-scale transformation on the gray-scale image to obtain a face image pyramid;
s223, carrying out interpolation processing on the face image pyramid by combining the following bicubic interpolation expression to obtain a corresponding interpolation image:
f(i+u，j+v)＝ABC；
wherein f (i, j) is a pixel gray value of the image, u is a decimal part of i, v is a decimal part of j, and S (x) is a preset base interpolation function;
s224, inputting the interpolation image into a P-Net network and an O-Net network which are connected in sequence, and obtaining a target face detection frame for positioning the face image after non-maximum value inhibition processing.
4. The method according to claim 2, wherein in step S22, the expectation calculation is performed on the set of hidden changes so that the observed data is maximized by the steps of:
s221, setting an observed variable set, a hidden variable set and model parameters as M, N and phi respectively, and entering an iterative optimization process;
s222, when iterating to the t step, converting the model parameters into phi t The probability distribution of the hidden variable set N at this time is p (N|M, phi) t )；
S223, enabling the optimization function to be Q, wherein the expression of the optimization function Q is set as follows:
x is a set, Z is a hidden variable;
s224, based on formula phi t+1 ＝arg N maxQ(Φ|Φ t ) Finding parameters maximizes the expected likelihood, where Φ t+1 Is an estimate of the parameter.
5. The method according to claim 2, wherein in step S23, the accurate line-of-sight estimation using a dnat network based on differential convolution comprises:
s231, selecting RELU functions as activation functions of a convolution layer and a full connection layer, wherein the formula is expressed as follows: f (x) =max (0, x), where x is input data and f (x) is output after RELU function processing;
s232, use d p (I, J) represents the difference in gaze predicted by the differential network, then the loss function L d The method comprises the following steps:
wherein D is k For a subset of training set D, only the image of one eye of the kth person is included, g gt (I) To test the true value of eye gaze, g in an image gt (J) A predicted value for eye gaze;
s233, predicting the difference d between the test image I and the reference image F through a differential network p (I, J) and combining the real gaze value g gt (F) The final gaze direction is predicted by the following formula:
wherein g sm (I) To test the eye gaze flattening value in the image, D c For a calibration set of reference images, w (·) represents weighting the importance of each prediction.
6. The method according to claim 1, wherein in step S3, the signal change analysis process based on the brain EEG signal comprises:
s31, adopting a DWT-PECSP algorithm to sequentially perform signal feature extraction and electroencephalogram signal classification and identification processing;
s32, performing power spectrum conversion on brain wave signals subjected to classification and identification, and obtaining the power of the brain wave signals based on the obtained power spectrums corresponding to different brain wave types;
s33, observing the brain electric power of the learner, and representing the brain activity degree of the learner based on the brain electric power.
7. The method according to claim 6, wherein in step S31, the signal feature extraction and electroencephalogram classification recognition processing are sequentially performed by using a DWT-PECSP algorithm, including:
s311, performing wavelet decomposition on the EEG signal subjected to threshold primary screening, and taking u-rhythm and beta-rhythm frequency band signals obtained by discrete wavelet transformation reconstruction as input vectors of CSP algorithm, and marking as X 1 ∈R N×T And X 2 ∈R N×T Wherein N is the number of channels, and T is the number of sampling points;
s312, find X 1 And X 2 Normalized covariance matrix R 1 And R is 2 Based on mean covariance
And->
Obtaining a corresponding mixed covariance R;/>
s313, carrying out feature decomposition based on the mixed covariance R, and determining a feature value lambda and a feature vector U;
s314, based on formula
Determining a whitening matrix P and based on the whitening matrix P to a covariance matrix R 1 And R is 2 Performing transformation and principal component decomposition;
s315, based on formula w=b T P determines the projection matrix W and transmits the signal X 1 And X 2 Filtering by a projection matrix W to obtain a matrix Z 1 And Z 2 Wherein B is an orthogonal matrix;
s316, performing electroencephalogram signal classification and identification by using a support vector machine.
8. The method according to claim 6, wherein in step S32, the power of the electroencephalogram signal is obtained by the following equation:
wherein i is different types of brain waves, including delta wave, theta wave, alpha wave and beta wave; s is S i (t) power spectrums corresponding to different types of brain waves respectively; x (k) is the amplitude of the kth brain wave discrete point; n is the type number of brain waves.
9. An on-line learner concentration monitoring system is characterized by comprising a data acquisition module, a human eye sight angle analysis module, an electroencephalogram signal analysis module and a concentration judgment module, wherein:
the data acquisition module is used for acquiring face monitoring images of learners at different moments and brain EEG signals;
the human eye sight angle analysis module is used for carrying out sight estimation processing based on the human face monitoring image and determining the sight angle change condition of learners at different moments;
the electroencephalogram signal analysis module is used for carrying out signal change analysis processing based on the brain EEG signals and determining the electroencephalogram signal change conditions of the brain EEG signals of learners at different moments;
and the concentration degree judging module is used for synthesizing the vision angle change condition and the electroencephalogram signal change condition and judging the concentration degree of the learner.
10. A readable storage medium, characterized in that it comprises an on-line learner concentration monitoring method program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310194137.4A CN116226712A (en) | 2023-03-03 | 2023-03-03 | Online learner concentration monitoring method, system and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310194137.4A CN116226712A (en) | 2023-03-03 | 2023-03-03 | Online learner concentration monitoring method, system and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116226712A true CN116226712A (en) | 2023-06-06 |
Family
ID=86571055
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310194137.4A Pending CN116226712A (en) | 2023-03-03 | 2023-03-03 | Online learner concentration monitoring method, system and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116226712A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105893980A (en) * | 2016-04-26 | 2016-08-24 | 北京科技大学 | Method and system for evaluating concentration degree of attention |
| CN111796681A (en) * | 2020-07-07 | 2020-10-20 | 重庆邮电大学 | Self-adaptive sight estimation method and medium based on differential convolution in man-machine interaction |
| CN113326729A (en) * | 2021-04-16 | 2021-08-31 | 合肥工业大学 | Multi-mode classroom concentration detection method and device |
| CN113331839A (en) * | 2021-05-28 | 2021-09-03 | 武汉科技大学 | Network learning attention monitoring method and system based on multi-source information fusion |
| CN115607158A (en) * | 2022-11-10 | 2023-01-17 | 武汉东湖学院 | Learning fatigue state monitoring method and computer equipment |
-
2023
- 2023-03-03 CN CN202310194137.4A patent/CN116226712A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105893980A (en) * | 2016-04-26 | 2016-08-24 | 北京科技大学 | Method and system for evaluating concentration degree of attention |
| CN111796681A (en) * | 2020-07-07 | 2020-10-20 | 重庆邮电大学 | Self-adaptive sight estimation method and medium based on differential convolution in man-machine interaction |
| CN113326729A (en) * | 2021-04-16 | 2021-08-31 | 合肥工业大学 | Multi-mode classroom concentration detection method and device |
| CN113331839A (en) * | 2021-05-28 | 2021-09-03 | 武汉科技大学 | Network learning attention monitoring method and system based on multi-source information fusion |
| CN115607158A (en) * | 2022-11-10 | 2023-01-17 | 武汉东湖学院 | Learning fatigue state monitoring method and computer equipment |
Non-Patent Citations (3)
| Title |
|---|
| 杨永正等: "海底隧道纵坡变化对驾驶人脑电信号影响分析", 《深圳大学学报(理工版)》, pages 271 - 277 * |
| 陈旭: "基于差分卷积的视线估计关键技术研究", 《中国优秀硕士学位论文全文数据库信息科技辑》, pages 8 - 54 * |
| 龙佳伟等: "基于排列熵和CSP融合的脑电信号特征提取", 《计算机技术与发展》, pages 157 - 162 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110136103B (en) | Medical image interpretation method, device, computer equipment and storage medium | |
| CN111052146B (en) | System and method for active learning | |
| US10614337B2 (en) | Information processing apparatus and information processing method | |
| Chen et al. | Hyperspectral image classification using dictionary-based sparse representation | |
| Wu et al. | Learning-based super resolution using kernel partial least squares | |
| Guo et al. | A generalized and robust method towards practical gaze estimation on smart phone | |
| CN108573499A (en) | A kind of visual target tracking method based on dimension self-adaption and occlusion detection | |
| CN110598638A (en) | Model training method, face gender prediction method, device and storage medium | |
| CN109685830B (en) | Target tracking method, device and equipment and computer storage medium | |
| CN114549470B (en) | Hand bone critical area acquisition method based on convolutional neural network and multi-granularity attention | |
| CN108985161B (en) | Low-rank sparse representation image feature learning method based on Laplace regularization | |
| CN110781970A (en) | Method, device and equipment for generating classifier and storage medium | |
| CN111027582B (en) | Semi-supervised feature subspace learning method and device based on low-rank graph learning | |
| Zhou et al. | Adaptive weighted locality-constrained sparse coding for glaucoma diagnosis | |
| CN112884721B (en) | Abnormality detection method, abnormality detection system and computer-readable storage medium | |
| CN108921872B (en) | Robust visual target tracking method suitable for long-range tracking | |
| Li et al. | IST-TransNet: Infrared small target detection based on transformer network | |
| CN108960005B (en) | Method and system for establishing and displaying object visual label in intelligent visual Internet of things | |
| CN116758419A (en) | Multi-scale target detection method, device and equipment for remote sensing image | |
| Zhou et al. | A low-resolution image restoration classifier network to identify stored-grain insects from images of sticky boards | |
| CN116226712A (en) | Online learner concentration monitoring method, system and readable storage medium | |
| CN110751189B (en) | Ellipse detection method based on perception contrast and feature selection | |
| US11893784B2 (en) | Assessment of image quality for optical character recognition using machine learning | |
| CN115116117A (en) | Learning input data acquisition method based on multi-mode fusion network | |
| Bhavani | Automated Attendance System and Voice Assistance using Face Recognition |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |