WO2019028592A1 - 一种教学辅助方法及采用该方法的教学辅助*** - Google Patents

一种教学辅助方法及采用该方法的教学辅助*** Download PDF

Info

Publication number
WO2019028592A1
WO2019028592A1 PCT/CN2017/096217 CN2017096217W WO2019028592A1 WO 2019028592 A1 WO2019028592 A1 WO 2019028592A1 CN 2017096217 W CN2017096217 W CN 2017096217W WO 2019028592 A1 WO2019028592 A1 WO 2019028592A1
Authority
WO
WIPO (PCT)
Prior art keywords
student
module
image
classroom
teaching
Prior art date
Application number
PCT/CN2017/096217
Other languages
English (en)
French (fr)
Inventor
王书强
王永灿
杨岳
申妍燕
胡明辉
Original Assignee
中国科学院深圳先进技术研究院
深圳市思毕酷科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中国科学院深圳先进技术研究院, 深圳市思毕酷科技有限公司 filed Critical 中国科学院深圳先进技术研究院
Priority to PCT/CN2017/096217 priority Critical patent/WO2019028592A1/zh
Priority to US16/623,397 priority patent/US11270526B2/en
Priority to JP2020522765A priority patent/JP6892558B2/ja
Publication of WO2019028592A1 publication Critical patent/WO2019028592A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B5/00—Electrically-operated educational appliances
    • G09B5/02—Electrically-operated educational appliances with visual presentation of the material to be studied, e.g. using film strip
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/174—Facial expression recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B19/00—Teaching not covered by other main groups of this subclass

Definitions

  • the invention relates to the field of teaching assistance, in particular to a teaching assistant method and a teaching assistant system using the same.
  • the design teaching assistant system assists the instructors to smoothly carry out teaching activities.
  • the current teaching aid system research emphasizes functionality, mainly from providing students with an independent learning environment, providing students with sufficient learning resources and reducing the workload of teachers. Using different technical means, design intelligent assistant system to improve the teaching effect of teachers and the learning efficiency of students.
  • the Chinese invention patent with the publication number CN106097790A discloses a teaching aid device, which recognizes an image in a teaching activity through image recognition technology, and then determines whether the student has done something unrelated to the class in the class, and according to the recognition result. Inform the teacher to do the appropriate processing.
  • the prior art does not disclose a method and a process for the image recognition module to recognize an image, nor how the image recognition module thereof compares the existing image with the pre-stored image and judges the comparison result. According to the prior art solution, the technician cannot specifically implement the technical effect of assisting the teaching process. Therefore, the existing teaching aid methods are insufficient.
  • an object of the present invention is to provide a teaching assistant method having high image recognition accuracy and a teaching assistant system using the same.
  • a teaching assistant The method includes the following sequence steps:
  • the acquisition module collects the classroom image of the scene in real time and transmits it to the identification module;
  • the identification module analyzes the classroom image and determines a student whose behavior is abnormal in the classroom image
  • the prompting module notifies the instructing teacher of the recognition result of the identification module
  • the step s2 includes the following steps:
  • the recognition module uses the trained depth tensor column network model to perform behavior detection on students in the classroom image.
  • the step s2 further includes the following steps:
  • the recognition module uses the trained deep tensor column network model to perform expression recognition on students in the classroom image.
  • the step s22 specifically includes the following steps:
  • the step s1 includes the following steps:
  • the collecting module respectively installs an image collecting device in the left, middle and right areas in front of the classroom;
  • the image acquisition module takes the upper body image of all students in the class as the collection target.
  • the method further comprises the step of: s4.
  • the storage module synchronously archives the identification result.
  • the step s4 includes the following steps:
  • the identification result corresponding to each student is determined into a student electronic file according to the class;
  • the student's class state curve is drawn to facilitate the instructor to provide targeted counseling to the student in combination with the content of the teaching and the test results.
  • step s1 the following steps are further included:
  • the step q1 comprises the following steps:
  • the collection module captures the classroom image for a long time in the classroom and stores it;
  • the step q2 comprises the following steps:
  • the present invention further provides a teaching assistance system, which is provided with: an acquisition module, an identification module connected to the collection module, and a prompt module connected to the identification module;
  • the collecting module is configured to collect the classroom image of the scene in real time and transmit the image to the identification module;
  • the identification module is configured to analyze the classroom image and determine a student whose behavior is abnormal in the classroom image; the identification module includes:
  • a behavior detecting unit configured to perform behavior detection on a student in the classroom image by using a trained deep tensor column network model
  • the prompting module is configured to notify the instructor of the recognition result of the identification module.
  • a storage module connected to the identification module is further provided; the storage module is configured to synchronously archive the identification result and perform editing analysis;
  • the identification module further includes:
  • An expression recognition unit configured to perform expression recognition on a student in the classroom image by using a trained deep tensor column network model
  • the expression recognition unit includes a face detection subunit and a convolutional neural network classifier.
  • the teaching assistant method of the present invention can provide higher image recognition accuracy and reduce algorithm to hardware by using the trained deep tensor network model to perform behavior detection on students in the classroom image.
  • the requirements and the ability to use on embedded devices reduce the cost of teaching aids.
  • the present invention also uses the trained deep tensor network model to perform expression recognition on the students in the classroom image, so that the teaching assistant system can accurately recognize the abnormal behavior of the students during class.
  • the teaching aid system using this method also has the above advantages.
  • Figure 1 is a basic flow chart of a teaching aid method
  • FIG. 3 is a schematic diagram showing the architecture of a teaching assistant system using the teaching aid method of FIG. 1;
  • FIG. 4 is a schematic diagram of the complete architecture of the teaching aid system of FIG. 3;
  • FIG. 5 is a schematic diagram of a full-link weight matrix folding and merging into a third-order tensor
  • Figure 6 is a schematic diagram of the tensor column decomposition of the third-order tensor
  • Figure 7 is a schematic diagram of the tensor column decomposition
  • Figure 8 is a schematic exploded view of a tensor column of a matrix
  • FIG. 9 is a schematic diagram of a layout mode of an acquisition module
  • FIG. 10 is a schematic structural diagram of a deep tensor column network model used for behavior detection
  • FIG. 11 is a schematic diagram showing the structure of a deep tensor column network model used for expression recognition.
  • Embodiments of the present invention are further elaborated below with reference to FIGS. 1 through 11.
  • a teaching aid method as shown in FIG. 1 includes the following sequential steps:
  • the acquisition module collects the classroom image of the scene in real time and transmits it to the identification module.
  • the identification module analyzes the classroom image and determines a student whose behavior is abnormal in the classroom image.
  • the prompting module notifies the instructor of the recognition result of the identification module.
  • the step s2 includes the following steps:
  • the recognition module uses the trained depth tensor column network model to perform behavior detection on students in the classroom image.
  • the depth tensor network model used in step s21 is decomposed by the tensor column of the traditional fully connected layer matrix, which greatly compresses the parameter quantity of the tensor of the fully connected layer matrix, improves the efficiency of the algorithm, and reduces the algorithm.
  • the hardware requirements, the convenient system is deployed in the form of embedded devices, the use is more convenient and simple, and the cost can be reduced, which is beneficial to the large-scale promotion of the teaching assistant system.
  • step s2 further includes the following steps:
  • the recognition module uses the trained deep tensor column network model to perform expression recognition on students in the classroom image.
  • Step s22 resets the core algorithm of image recognition, and performs joint behavior on student pictures.
  • Detection and expression recognition enable deep tensor network models to achieve better recognition accuracy and efficiency.
  • the depth tensor network model reduces the model parameter quantity and improves the robustness of the system. It effectively improves the speed of the teaching assistant system to detect students' abnormal behavior and expression in real time.
  • step s22 specifically includes the following steps:
  • the recognition module is inconvenient to directly extract the expression feature. Therefore, the present invention realizes the recognition of the expression through step s221 and step s222. Firstly, each face area of each student is detected by the face detection sub-unit from the classroom picture collected by the image acquisition module, and then the convolved neural network classifier is used to perform facial expression recognition on each detected face area image block.
  • step s1 includes the following steps:
  • the acquisition module respectively installs an image acquisition device in the left, middle, and right areas in front of the classroom.
  • image capturing devices in two areas or a plurality of areas in front of the classroom in front of the classroom to prevent students from being occluded in a single direction.
  • most students in the normal teaching state basically do not have abnormal behaviors such as confusion, daze, boredom, etc., so the time interval of shooting can be set for each image acquisition device to reduce the sampling rate of the image. Save on processing and storage resources.
  • the image acquisition module takes the upper body image of all students in the class as the collection target.
  • the behavior and expression features of the students during the class can be extracted and recognized by the upper body image.
  • the upper body is the target and the targeted image features can be compared with the image regions.
  • the method further includes the following steps: s4.
  • the storage module synchronously archives the identification result.
  • the recognition result can be further analyzed and utilized from the overall direction.
  • the recognition results and analysis of the teaching effect and analysis of the student's learning curve the teaching activities can be carried out more specifically, so that the next teaching work can be more targeted, and the overall level and quality of teaching can be improved.
  • the step s4 includes the following steps:
  • the recognition result corresponding to each student is determined into a student electronic file according to the class.
  • the student's class state curve is drawn to facilitate the instructor to provide targeted counseling to the student in combination with the content of the teaching and the test results.
  • step s1 the following steps are further included:
  • it can be divided into constructing a data set for behavior detection and constructing a data set for expression recognition.
  • constructing a suitable data set is the basis for correctly detecting the abnormal behavior of students, and is directly related to the level of system recognition performance.
  • the abnormal behavior refers to all behaviors such as sleeping, talking, doing small actions, daze, etc. .
  • facial expressions such as concentration, interest, and thinking can also be extracted, and the deep tensor network model can be trained, so that the behavior of careful listening can also be recognized by the model.
  • a student expression data set for the classroom class In order to facilitate the teachers to understand the status of each class in the classroom more accurately and in real time, to meet the needs of students' facial expression recognition, we construct a student expression data set for the classroom class. From the collected classroom images, the student's facial expression picture blocks are intercepted, and the relevant expression labels corresponding to the seriousness of the lectures are given, such as concentration, interest, thinking, confusion, daze, boredom, and the like. It is convenient for the instructors to more easily grasp the status of each student's lectures and the situation and attitude of the students, and make real-time treatment and adjustment.
  • the training of behavior detection and expression recognition can be performed separately. The only difference is that they use different data sets for training.
  • the classroom student abnormal behavior recognition neural network model based on the "deep tensor column network”.
  • the multi-level convolutional layer is used to automatically learn and extract the behavioral characteristics of the students in the classroom picture.
  • the TT decomposition tensor column decomposition
  • tensor train decomposition is a tensor decomposition model in which each element of a tensor is represented by a product of several matrices.
  • I k represents the dimension of the kth order
  • tensor column is decomposed into:
  • the tensor column of the matrix is decomposed, assuming a matrix Choose a refactoring scheme, such as a refactoring scheme: After the reconstruction scheme is selected, the tensor column decomposition of the matrix first maps the matrix to the d-order tensor Tensor Perform tensor column decomposition, ie
  • the step q1 includes the following steps:
  • the acquisition module captures the classroom image for a long time in the classroom and stores it.
  • the image data with abnormal behavior may be relatively small, in order to avoid over-fitting of the model and enhance the anti-interference ability of the model to factors such as illumination changes, we make data enhancements to collect the anomalous behavior picture data of classroom students.
  • the picture is changed in contrast, RGB channel intensity, noise, etc., and the sample size and type of the picture data are increased.
  • step q2 includes the following steps:
  • the model structure of the deep tensor column network shown in FIG. 10 (here, only a 3-layer convolution is taken as an example); the steps of constructing the deep tensor column network model are:
  • the tensor A of S x S x m is outputted in the convolution layer of the last layer, that is, the original picture is divided into a grid of S x S, and each grid unit corresponds to As part of the original classroom picture, the picture features in each grid correspond to an m-dimensional vector of the tensor.
  • the improved full connection is a tensor column (TT) decomposition of the traditional fully connected layer matrix, thereby greatly compressing the parameters of the full connection layer, improving the efficiency of the algorithm, reducing the hardware requirements, and enabling use on embedded devices.
  • This teaching assistant system improves the speed of students in real-time detection of abnormal behaviors in the classroom, and facilitates the system to deploy in the form of embedded devices, which is more convenient and simple, and can reduce the cost, which is conducive to the large-scale promotion of the student recognition teaching aid system in this classroom abnormal behavior.
  • Tensor column decomposition is a tensor decomposition model in which each element of a tensor is represented by the product of several matrices.
  • the tensor column decomposition of the matrix needs to first select the reconstruction scheme. First, the matrix is mapped to the d-order tensor, and then the tensor column is decomposed.
  • the tensor column decomposition of the fully connected layer weight matrix is the following is a detailed explanation of the process (for convenience of explanation, we substitute some parameter examples, but the specific implementation is not limited to specific parameters)
  • the full connection layer weight matrix tensor column decomposition step is:
  • the virtual dimension corresponding to the row and column is merged, that is, the full connection weight matrix is reshaped into d-order tensor; according to the above example method, the weight matrix of the original 800x784 is reshaped to 700x32x28. Order tensor.
  • the parameter size of the re-decomposition scheme before and after the decomposition of several tensor columns is calculated as follows. It can be seen from the calculation results in the table that the parameter value of the fully connected layer weight parameter is reduced by hundreds of times after the tensor column decomposition, which can improve the efficiency of the algorithm and reduce the hardware requirements, and facilitate the teaching aid system in the embedded device.
  • the implementation of the above improve the detection of the real-time behavior of students in the classroom.
  • the loss function uses a square sum error loss function which includes 3 parts, a coordinate prediction function, a confidence prediction function containing the identification box of the student with abnormal behavior, and a confidence prediction function of the identification box of the student who does not contain the abnormal behavior.
  • x, y is the center position coordinate of the abnormal behavior student identification frame
  • w, h is the width and height of the abnormal behavior student identification frame.
  • the face detection network model is first trained, and the classroom detection of the face detection and behavior detection sub-module is abnormal.
  • the behavior detection model is similar.
  • the classroom abnormal behavior data set is changed into the adult face detection data set, and the image input model in the face detection data set is used to train, and the training process of the above-mentioned behavior detection sub-module 1-5 is repeated, so that the model is made. It can automatically learn facial features and automatically detect the student's face position from the classroom image.
  • CNN Convolutional Neural Network
  • the student's face picture block is continuously convolved to extract facial expression features.
  • the improved full link outputs the predicted face image of the student's face image.
  • the TT decomposition is also performed on the fully connected layer weight matrix. The specific process is described in detail in the behavior detection sub-module (4), and will not be described here.
  • the accuracy of the model learning is also included, as well as the steps of the model testing.
  • the above-mentioned trained network model parameters are imported into the depth tensor network of the behavior detection sub-module in the recognition module, and the classroom picture collected by the image acquisition module in real time is input, and the abnormal behavior in the picture is detected in real time. Students, if any, are identified and the recognition results are notified to the instructor by the prompt module and archived by the storage module for further analysis and mining of the data.
  • Whether the abnormal behavior is determined according to whether the probability of abnormal behavior given by the network model is greater than a given probability threshold.
  • the default probability threshold is given by a plurality of tests to give a reasonable value that is in line with the public's better balance sensitivity and accuracy. Appropriate adjustments can be made according to individual circumstances, so that the teaching aid system is more humane. During the test, the details can be adjusted according to the existing problems, so that the system can be optimally put into practical use.
  • the above-mentioned trained network model parameters are imported into the expression recognition sub-module in the recognition module, and the classroom image collected by the image acquisition module in real time is input, and firstly, the face detection network model detects all the people in the image. The face position, and then the detected face image block is simply processed, and then adjusted to a fixed size input expression recognition network model to identify the student's class expression.
  • the model automatically detects the face and recognizes the expression features, so that the model can be put into practical use, real-time detection and analysis of the expression information of the students in the classroom, combined with the results of the behavior detection module, so that the class teacher can understand the class of each student in the class more accurately and in real time.
  • the status allows the instructors to be more targeted and improve the quality and efficiency of teaching.
  • the present invention also provides a teaching assistant system, which is provided with: a set module, an identification module connected to the collection module, and a prompt module connected to the identification module.
  • the collecting module is configured to collect the classroom image of the scene in real time and transmit the image to the identification module.
  • the acquisition module collects the upper body picture of all students in the class.
  • the collection method is to install the image acquisition device on the left, middle and right tops of the front wall of the classroom to adjust the shooting angle to prevent occlusion and integrate multiple viewing angles, and set the time interval of each shooting of the image acquisition device to collect the collected time.
  • the image is processed into the size required by the recognition module and transmitted to the recognition module to provide data for classroom behavior recognition;
  • An identification module configured to analyze the classroom image, and determine a student whose behavior is abnormal in the classroom image; and specifically includes the following units:
  • a behavior detecting unit is configured to perform behavior detection on a student in the classroom image using a trained deep tensor column network model.
  • the purpose of the identification module is to identify the specific classroom behavior and expression of the students in the image acquisition module to determine whether the student is listening carefully and to understand the degree of acceptance of the content.
  • the behavior detection method is to collect the picture data of the students in the class, manually mark the pictures, and mark the abnormal students, that is, the students who are not seriously listening, including sleeping, talking, doing small actions, and being in a daze. Then use the constructed classroom behavior picture dataset to train the deep tensor network model, so that the recognition module can automatically learn the picture features and detect the abnormal behavior of the students in the picture.
  • the trained model is put into practical use, and the three images transmitted from the image acquisition module are acquired in real time (this patent uses three images as an example to illustrate, and under the condition of hardware device permission, multiple images can be collected in real time), respectively Detect abnormal behavior of students in the picture and frame behavioral abnormal students according to a given probability threshold.
  • the prompting module is configured to notify the instructor of the recognition result of the identification module.
  • the prompting module prompts the module to notify the instructor in a certain way in real time, and if the images of the three angles are not abnormal, the teacher does not notify, and the teacher can adjust the probability threshold to adjust the recognition sensitivity.
  • the teacher can know the class status of the class students and the acceptance of the content they are teaching in real time. Based on this, the teachers can focus on the questions or take corresponding countermeasures.
  • a storage module connected to the identification module is further provided, and the storage module is configured to synchronously archive the recognition result and perform editing and classification.
  • the storage module is the final result of all the identification of the system for the class, the student is a class, and is stored in the form of a student profile, and the school can make full use of these electronic files. Dig out useful information from it. On the one hand, it can analyze and evaluate the deficiencies in teaching according to the overall acceptance of students. On the other hand, it can analyze the learning curve of students and find the real reason for students' poor performance. Make up the leak.
  • the identification module further includes:
  • An expression recognition unit is configured to perform expression recognition on a student in the classroom image using the trained depth tensor column network model.
  • the expression recognition unit includes a face detection subunit and a convolutional neural network classifier.
  • the expression recognition sub-module method is similar to the behavior detection, except that the abnormal expression is marked for algorithm training.
  • two sub-modules are described by the same algorithm model and parallel recognition. However, by changing the loss function, the two tasks can be merged into the same model through the multi-task loss function, which is not elaborated here. It is also within the scope of this patent.
  • the solution claimed by the invention solves the technical problem of assisting the teacher to carry out the teaching activity by analyzing and processing the classroom image, and avoids the existing teaching equipment being too dependent on the external image recognition device, resulting in high hardware requirements and recognition. Accurate defects improve the efficiency of the teacher's teaching work.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Business, Economics & Management (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Molecular Biology (AREA)
  • Mathematical Physics (AREA)
  • Human Computer Interaction (AREA)
  • Educational Technology (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Educational Administration (AREA)
  • General Engineering & Computer Science (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • Image Analysis (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • User Interface Of Digital Computer (AREA)
  • Image Processing (AREA)

Abstract

一种教学辅助方法及采用该方法的教学辅助***，其中教学辅助方法通过采用训练好的深度张量列网络模型来对课堂图像中的学生进行行为检测，可以提供较高的图像识别精度和降低算法对硬件的要求，且能够在嵌入式设备上使用，降低了教学辅助方法的使用成本；同时采用该教学辅助方法的教学辅助***也具有同样的优点。

Description

一种教学辅助方法及采用该方法的教学辅助*** 技术领域
本发明涉及教学辅助领域，尤其涉及一种教学辅助方法及采用该方法的教学辅助***。
背景技术
在一般的教学活动中，由于上课的学生和授课老师的数量比例悬殊，授课老师在授课时没有太多的时间和精力通过观察每个学生的上课行为和表情来判断学生的学习状态。这就使得授课老师无法精确的了解每个学生的上课状态和对本次教授内容的被接受程度。很容易导致课堂上老师讲老师的，学生聊学生的，进而让整个教学活动被撕裂开来，也使得授课老师无法有的放矢的进行教学，严重的影响了教学质量和效率。所以，能够在学生上课时使用的教学辅助***历来是教育界所关注的重点问题。设计教学辅助***辅助授课教师顺利开展教学活动。目前的教学辅助***研究强调功能性，主要从为学生提供自主的学习环境，为学生提供充分的学习资源，减轻教师的工作量几个方面展开的。运用不同技术手段，设计智能化辅助***来提高教师的授课效果和学生的学习效率。
在现有技术中，公开号为CN106097790A的中国发明专利公开了一种教学辅助装置，通过图像识别技术识别教学活动中的图像，进而来判断学生上课是否做与上课无关的事情，并根据识别结果通知老师做相应处理。
由于该现有技术并未公开其图像识别模块识别图像的方法和过程，也没有公开其图像识别模块如何实现将现有图像与预存图像进行比对，并判断比对结果。技术人员根据该现有技术方案无法具体实现为教学过程进行辅助的技术效果。因此，现有的教学辅助方法存在不足。
发明内容
为了解决现有技术中存在的上述技术问题，本发明的目的在于提供一种具有较高的图像识别精度的教学辅助方法和采用该方法的教学辅助***。
为了解决上述技术问题，本发明所采用的技术方案为：一种教学辅助方 法，包括以下顺序步骤：
s1.采集模块实时采集现场的课堂图像，并传输给识别模块；
s2.所述识别模块对所述课堂图像进行分析，并判断所述课堂图像中行为异常的学生；
s3.提示模块将所述识别模块的识别结果通知授课教师；
所述步骤s2中包括以下步骤：
s21.所述识别模块使用训练好的深度张量列网络模型来对所述课堂图像中的学生进行行为检测。
优选的，所述步骤s2还包括以下步骤：
s22.所述识别模块使用训练好的深度张量列网络模型来对所述课堂图像中的学生进行表情识别。
优选的，所述步骤s22具体包括以下步骤：
s221.通过人脸检测子单元从所述采集模块采集到的所述课堂图像中识别出各学生的人脸区域；
s222.通过卷积神经网络分类器对检测到的所述人脸区域做表情识别。
优选的，步骤s1中包括以下步骤：
s11.所述采集模块在教室前方的左、中、右区域分别安装图像采集装置；
s12.所述图像采集模块以班级中所有学生上半身图像为采集目标。
优选的，还包括以下步骤：s4.存储模块同步存档所述识别结果。
优选的，所述步骤s4中包括以下步骤：
s41.将每个学生对应的所述识别结果按班级制定成学生电子档案；
s42.根据所述学生电子档案绘出学生上课状态曲线，用以便于授课教师结合当时教授的内容以及考试成绩对学生进行有针对性的辅导。
优选的，步骤s1之前还包括以下步骤：
q1.构建数据集；
q2.训练所述深度张量列网络模型。
优选的，所述步骤q1包括以下步骤：
q11.所述采集模块在教室长时间拍摄所述课堂图像并存储；
q12.选取存在异常的学生图片进行标注。
优选的，所述步骤q2包括以下步骤：
q21.通过神经网络模型的多层卷积层提取已标注的所述学生图片中的异常特征，所述异常特征与分解后的全连接层权重矩阵运算得到输出预测值；
q22.所述输出预测值与所述学生图片中的异常行为学生真实标注值的误差构成的损失函数；
q23.根据所述损失函数调整网络参数，得到训练好的深度张量列网络模型。
为了解决上述技术问题，本发明还提供一种教学辅助***，设置有：采集模块、与所述采集模块连接的识别模块、与所述识别模块连接的提示模块；
所述采集模块，用于实时采集现场的课堂图像并传输给识别模块；
所述识别模块，用于对所述课堂图像进行分析，并判断所述课堂图像中行为异常的学生；所述识别模块包括：
行为检测单元，用于使用训练好的深度张量列网络模型来对所述课堂图像中的学生进行行为检测；
所述提示模块，用于将所述识别模块的识别结果通知授课教师。
优选的，还设置有：与所述识别模块连接的存储模块；所述存储模块，用于同步存档所述识别结果并进行编辑分析；
所述识别模块还包括：
表情识别单元，用于使用训练好的深度张量列网络模型来对所述课堂图像中的学生进行表情识别；
所述表情识别单元包括人脸检测子单元和卷积神经网络分类器。
与现有技术相比，本发明的教学辅助方法，通过采用训练好的深度张量列网络模型来对所述课堂图像中的学生进行行为检测可以提供较高的图像识别精度和降低算法对硬件的要求，且能够在嵌入式设备上使用，降低了教学辅助方法的使用成本。
进一步的，本发明还采用训练好的深度张量列网络模型来对所述课堂图像中的学生进行表情识别，使得教学辅助***对学生上课时的异常行为识别精度更高。
采用该方法的教学辅助***，也同样具有上述优点。
附图说明
图1为一种教学辅助方法的基本流程图；
图2为一种教学辅助方法的详细流程图；
图3为采用图1教学辅助方法的教学辅助***架构示意图；
图4为图3教学辅助***的完整架构示意图；
图5为全链接权值矩阵折叠和融合为三阶张量示意图；
图6为三阶张量进行张量列分解示意图；
图7为张量列分解示意图；
图8为矩阵的张量列分解示意图；
图9为采集模块布设方式示意图；
图10为行为检测所采用的深度张量列网络模型结构示意图；
图11为表情识别所采用的深度张量列网络模型结构采示意图。
具体实施方式
以下参考附图1至附图11，对本发明的各实施例予以进一步地详尽阐述。
如附图1所示一种教学辅助方法，包括以下顺序步骤：
s1.采集模块实时采集现场的课堂图像，并传输给识别模块。
s2.所述识别模块对所述课堂图像进行分析，并判断所述课堂图像中行为异常的学生。
s3.提示模块将所述识别模块的识别结果通知授课教师。
所述步骤s2中包括以下步骤：
s21.所述识别模块使用训练好的深度张量列网络模型来对所述课堂图像中的学生进行行为检测。
具体的，步骤s21中采用的深度张量列网络模型通过对传统全连接层矩阵做张量列分解得来，极大压缩了全连接层矩阵张量的参数量，提高算法效率，降低了算法对硬件的要求，方便***以嵌入式设备形式部署，使用更加方便简单且能够降低成本，利于本教学辅助***的大规模推广。
如附图2所示，所述步骤s2还包括以下步骤：
s22.所述识别模块使用训练好的深度张量列网络模型来对所述课堂图像中的学生进行表情识别。
步骤s22重设了图像识别这一核心算法，通过联合对学生图片进行行为 检测和表情识别，使深度张量列网络模型获得更好的识别精度和效率。在深度张量列网络模型降低了模型参数量，提升了***的鲁棒性的基础上有效的提高了本教学辅助***实时检测课堂上学生异常行为与表情的速度。
在本实施例中，所述步骤s22具体包括以下步骤：
s221.通过人脸检测子单元从所述采集模块采集到的所述课堂图像中识别出各学生的人脸区域。
s222.通过卷积神经网络分类器对检测到的所述人脸区域做表情识别。
在具体操作中，由于人脸表情特征相对较细，识别模块不方便直接提取表情特征，因此，本发明通过步骤s221和步骤s222来实现表情的识别。首先通过人脸检测子单元从图像采集模块采集到的课堂图片中检测出各学生人脸区域，再通过卷积神经网络分类器对检测到的各人脸区域图像块做表情识别。
如附图9所示，步骤s1中包括以下步骤：
s11.所述采集模块在教室前方的左、中、右区域分别安装图像采集装置。
在其他实施例中，也可以采用在教室前方左、右两个区域或者多个区域安装图像采集装置，以防止单个方向拍摄容易有学生被遮挡。
同时，在优选的实施例中，正常授课状态下大部分学生基本不会出现困惑、发呆、厌烦等异常行为，故可以为每个图像采集装置设置拍摄的时间间隔，以降低图像的采样率，节省相应的处理和存储资源。
s12.所述图像采集模块以班级中所有学生上半身图像为采集目标。
在具体实施时，学生在上课时的行为和表情特征基本用上半身图像就可以提取和进行识别，以上半身为目标可以有针对性的拍摄特征比较富集的图像区域。
在本实施例中，还包括以下步骤：s4.存储模块同步存档所述识别结果。
在本实施例中，通过对识别结果进行同步存储，可以进一步从整体方向对识别结果进行分析利用。比如：根据识别结果分析和评估教学效果和分析学生的学***和质量。
优选的，所述步骤s4中包括以下步骤：
s41.将每个学生对应的所述识别结果按班级制定成学生电子档案。
有助于对每个学生上课状态的检测识别结果进行统计分析，以及主动的对学生在校的听课状态进行跟踪，避免仅靠学生的成绩来判断学生听课状态这种被动的方式迟滞性的弊端。
s42.根据所述学生电子档案绘出学生上课状态曲线，用以便于授课教师结合当时教授的内容以及考试成绩对学生进行有针对性的辅导。
同时，也可以将学生电子档案与教师的教学评估相结合，改进目前以学生考试成绩为课堂教学质量评估的主要参照过于片面的弊端。
在本实施例中，步骤s1之前还包括以下步骤：
q1.构建数据集。
在具体实施中，可以分为为行为检测构建数据集和为表情识别构建数据集。
具体的，在为行为检测构建数据集时，构建一个合适的数据集是能否正确检测出异常行为学生的基础，直接关系到***识别性能的高低。我们使用采集模块在多个教室长时间拍摄课堂上课情况，然后从中选取存在异常行为学生的图片进行标注，其中异常行为指一切表现为未认真听课的行为，如睡觉、说话、做小动作、发呆等。由于可能存在遮挡问题以及单视角的局限性，我们使用左、中、右三个视角的图像采集装置采集图片，分别标注。并对图片做简单处理，输入的固定尺寸以调整成适合模型，以方便在训练网络模型时使用。
在其他实施例中，也可以将专注、兴致盎然、思考等表情特征进行提取，并对深度张量列网络模型进行训练，使得认真听课的行为也可以通过该模型识别出来。
在为表情识别构建数据集时，由于此处我们做人脸表情识别分两步进行，首先做人脸检测，再做表情识别，我们需构建两个数据集，一个为课堂人脸检测数据集，另一个为课堂学生上课表情数据集。
人脸检测数据集
为能准确的从图像采集模块采集到的课堂图像中实时准确的检测出学生脸部，我们构建一个小型课堂人脸检测数据集。我们使用图像采集模块在多个教室长时间拍摄课堂上课情况采集到的课堂图像，对图片进行标注，给出图片中人脸位置，并对图片做简单处理，调整成适合模型输入的固定尺寸， 以方便在训练网络模型时使用。
学生上课表情数据集
为方便上课教师更加准确实时地了解课堂上每个学生的听课状态，满足学生上课表情识别需求，我们针对学生课堂听课这一场景，构建一学生上课表情数据集。从采集到的课堂图像中，截取出学生面部表情图片块，给出对应听课认真程度相关表情标签，如专注、兴致盎然、思考、困惑、发呆、厌烦等等。方便授课教师更加方便细致的掌握每位学生的听课状态和课程掌握情况与态度，做出实时的处理与调整。
q2.训练所述深度张量列网络模型。
在具体实施中，行为检测和表情识别的训练可以分开进行。其区别仅在于采用不同的数据集进行训练。
具体的，基于”深度张量列网络”的课堂学生异常行为识别神经网络模型。首先通过多层卷积层自动学习提取课堂图片中学生的行为特征，在使用学习到的课堂行为特征信息经TT分解(张量列分解)后的全连接层对学生课堂行为进行识别，检测出有异常课堂行为的学生。
如附图7所示，张量列分解(tensor train decomposition，TT-decomposition)是一种张量分解模型，将张量的每一个元素都用若干个矩阵的乘积表示。假设存在d阶张量
Figure PCTCN2017096217-appb-000001
(Ik表示第k阶的维数)，张量
Figure PCTCN2017096217-appb-000002
的张量列分解为：
Figure PCTCN2017096217-appb-000003
其中
Figure PCTCN2017096217-appb-000004
是张量
Figure PCTCN2017096217-appb-000005
第k阶级对应的核矩阵，规模为rk-1′rk，k＝1，2，...d，r0＝rd＝1；(rO，r1，Λrd)是d阶张量
Figure PCTCN2017096217-appb-000006
进行张量列时对应的TT-rank，实际上
Figure PCTCN2017096217-appb-000007
是规模为
Figure PCTCN2017096217-appb-000008
三阶张量，所以
Figure PCTCN2017096217-appb-000009
又叫核张量。
Figure PCTCN2017096217-appb-000010
如附图8所示，矩阵的张量列分解，假设矩阵
Figure PCTCN2017096217-appb-000011
选择重构方案，如重构方案：
Figure PCTCN2017096217-appb-000012
选定重构方案后，矩阵的张量列分解首先将矩阵映射到d阶张量
Figure PCTCN2017096217-appb-000013
再对张量
Figure PCTCN2017096217-appb-000014
进行张量列分解，即
Figure PCTCN2017096217-appb-000015
如附图2所示，所述步骤q1包括以下步骤：
q11.所述采集模块在教室长时间拍摄所述课堂图像并存储。
q12.选取存在异常的学生图片进行标注。
由于含有异常行为图像数据可能相对较少，为避免模型过拟合，并增强模型对光照变化等因素的抗干扰能力，我们对采集标注的课堂学生异常行为图片数据做数据增强。分别对图片进行改变对比度、RGB通道强度、加噪声等处理，增加图片数据样本量和种类。
在本实施例中，所述步骤q2包括以下步骤：
q21.通过神经网络模型的多层卷积层提取已标注的所述学生图片中的异常特征，异常特征与分解后的全连接层权重矩阵运算得到输出预测值。
如附图10所示的深度张量列网络的模型结构(此处仅以3层卷积为例说明)；构建该深度张量列网络模型的步骤为：
1.初始化网络模型参数。
2.将构建的课堂学生异常行为数据集中的图片输入到该模型进行训练。
3.图片经过不断卷积池化，在最后一层卷积层输出S x S x m的张量A，即把原图片划分出了一个S x S的网格，每个网格单元对应着原课堂图片的一部分，每个网格中的图片特征对应着该张量中的一个m维向量。
4.经过改进后的全链接，输出一个S x S x(5a)的张量，即每个网格单元对应的a个异常行为学生检测边界框坐标(x，y，w，h)与识别框中检测为异常行为学生的置信度。其中x和y为异常行为学生识别框中心点坐标，w和h分别为异常行为学生识别框的宽和高，并将坐标进行归一化，使其介于0到1之间。
其中改进后的全连接为对传统全连接层矩阵做张量列(TT)分解，从而极大压缩全连接层参数量，提高算法效率，降低对硬件的要求，使得能在嵌入式设备上使用。为本教学辅助***提高了实时检测课堂异常行为学生的速度，并方便***以嵌入式设备形式部署，更加方便简单且能够降低成本，利于本课堂异常行为学生识别教学辅助***的大规模推广。
张量列分解(TT分解)是一种张量分解模型，将张量的每一个元素都用若干个矩阵的乘积表示。矩阵的张量列分解需先选择重构方案，首先将矩阵映射到d阶张量，再对张量进行张量列分解。此处即是对全连接层权重矩阵做张量列分解，以下为对该过程的详细解释(为了方便说明，我们代入一些参数举例，但具体实现不局限于具体参数)
在本实施例中，全连接层权重矩阵张量列分解步骤为：
1.如附图5所示，将全连接权值矩阵的行和列均折叠为d个虚拟的维度；此处假设网络模型中S＝4，m＝50，n＝49，即图像采集模块采集到的课堂图像经逐层卷积池化后提取到4x4x50＝800个特征，下一隐层有4x4x49＝784个隐节点，则该全连接层权重参数为800x784的矩阵。为方便表示，取d＝3，将全连接权值矩阵的行和列均折叠为3个虚拟的维度，如附图所示。
2.如附图5所示，将行列对应的虚拟的维度进行融合，即将全连接权值矩阵重塑为d阶张量；按上述实例方法则原800x784的权重矩阵被重塑为了700x32x28的3阶张量。
3.如附图6所示，定义所述d阶张量的张量列秩r，其中rk表示原始张量除去前(k-1)阶效应后沿张量第k阶展开的矩阵的秩，其中r0＝rd＝1是约束条件；本文定义的张量列秩为3。
4.将所述d阶张量进行张量列分解得到全连接层权值矩阵的张量列分解表示，即
Figure PCTCN2017096217-appb-000016
其中
Figure PCTCN2017096217-appb-000017
是规模为
Figure PCTCN2017096217-appb-000018
三阶张量，Ik表示高阶张量第k阶的维数。在本实例中，即原700x32x28的3阶张量被分解为了1x700x3、3x32x3、3x28x1的3个核张量。全连接层权重由原来的627200个参数下降到了2472个参数。
为比较直观的表示TT分解(张量列分解)对全连接层权重参数量的压缩效果，现将几种重塑方案下张量列分解前后参数规模计算如下表。由表中计算结果可看出，全连接层权重参数经张量列分解后参数量下降了成百上千倍，能提高算法效率，降低对硬件的要求，方便本教学辅助***在嵌入式设备上的实现，提高检测课堂上学生异常行为的实时性。
Figure PCTCN2017096217-appb-000019
Figure PCTCN2017096217-appb-000020
q22.输出预测值与所述学生图片中的异常行为学生真实标注值的误差构成的损失函数；
q23.根据损失函数调整网络参数，得到训练好的深度张量列网络模型。
5.运用反向传播算法，根据输出的预测值与原图中的异常行为学生真实标注值间误差构成的损失函数L(此处损失函数采用平方和误差损失函数，在下文中具体介绍)，调整网络参数，至指定精度。然后保存网络参数。
损失函数使用平方和误差损失函数其中包括3部分，坐标预测函数，包含异常行为学生的识别框的置信度预测函数和不包含异常行为学生的识别框的置信度预测函数。
Figure PCTCN2017096217-appb-000021
其中，x，y是异常行为学生识别框的中心位置坐标，w，h是异常行为学生识别框的宽和高，
Figure PCTCN2017096217-appb-000022
为判断第i个网格中的第j个识别框是否负责检测，
Figure PCTCN2017096217-appb-000023
为判断是否有异常行为学生中心落入在网格i中，lcoord为坐标预测权重，lnoobj为不包含异常行为学生的识别框的置信度权重。
在优选的实施例中，如附图10、附图11所示，为表情识别训练时深度张量列网络模型时，首先训练人脸检测网络模型，人脸检测与行为检测子模块的课堂异常行为检测模型类似，将其中课堂异常行为数据集换成人脸检测数据集，使用人脸检测数据集中的图片输入模型训练，重复上述行为检测子模块中1-5的训练过程即可，使得模型能自动学习人脸特征，从课堂图像中自动检测出学生脸部位置。
其次训练课堂人脸面部表情识别时采用的卷积神经网络(CNN)分类器。 将前述构建的学生上课表情数据集中带表情标签的学生脸部图片块输入表情识别分类器，对表情识别网络模型进行训练。表情识别网络模型如附图11。
1.初始化表情识别网络模型参数。
2.将构建的学生上课表情数据集中带表情标签的学生脸部图片块输入到该模型进行训练。
3.学生脸部图片块经过不断卷积池化，提取面部表情特征。
4.经过改进后的全链接，输出预测的学生脸部图片块表情标签。此处也对全连接层权重矩阵做TT分解。具体过程在行为检测子模块中(4)中有详细介绍，此处不再赘述。
5.运用反向传播算法，根据输出的预测值与真实标注表情标签间误差构成的损失函数L，调整网络参数，至指定精度，然后保存网络参数。
在其他实施例中，还为检索模型学习的准确性，还包括模型测试的步骤。
在为行为检测进行测试时，将上述训练好的网络模型参数导入识别模块中行为检测子模块的深度张量列网络，输入由图像采集模块实时采集的课堂图片，实时检测图片中是否有异常行为学生，如果有则标出并将识别结果由提示模块通知授课教师，并由存储模块存档，以便后续对数据做进一步分析挖掘。是否为异常行为根据网络模型给出的异常行为概率是否大于给定的概率阈值确定，默认概率阈值通过多次测试给出一个合理的符合大众的能较好平衡灵敏度与准确度的值，老师后续可根据个人情况做适当调整，以使得本教学辅助***更为人性化。测试期间可根据存在的问题，在细节上做适当调整，以便使***达到最佳状态，然后投入实际使用。
在为表情识别进行测试时，将上述训练好的网络模型参数导入识别模块中表情识别子模块，输入由图像采集模块实时采集的课堂图片，首先由人脸检测网络模型检测出图片中的所有人脸位置，再将检测到的人脸图片块简单处理后，调整成固定大小输入表情识别网络模型识别学生上课表情。使得模型自动检测人脸并识别其表情特征，以便模型能投入实际使用，实时检测分析课堂上学生的表情信息，结合行为检测模块结果，方便上课教师更加准确实时地了解课堂上每个学生的上课状态，让授课教师更能有的放矢，提高教学质量和效率。
为了解决上述技术问题，本发明还提供一种教学辅助***，设置有：采 集模块、与所述采集模块连接的识别模块、与所述识别模块连接的提示模块。
所述采集模块，用于实时采集现场的课堂图像并传输给识别模块。
采集模块，如附图9所示，图像采集模块采集目标为班级所有学生上半身图片。采集方式是通过在教室前方墙壁的左、中、右顶端分别安装图像采集装置，调整好拍摄角度，以防止遮挡并综合多个视角，设置图像采集装置每次拍摄的时间间隔，把采集到的图片处理成识别模块所需大小后传输到识别模块，为进行课堂行为识别提供数据；
识别模块，用于对所述课堂图像进行分析，并判断所述课堂图像中行为异常的学生；其具体包括以下单元：
行为检测单元，用于使用训练好的深度张量列网络模型来对所述课堂图像中的学生进行行为检测。
具体的，识别模块的目的是识别图像采集模块中上课学生的具体课堂行为与表情来判断学生是否在认真听课，了解学生对授课内容的接受程度。行为检测方法是先采集课堂中上课学生图片数据，对图片做人工标注，标出其中异常学生，即未认真听课学生，具体包括睡觉、说话、做小动作、发呆等。再使用构建的课堂行为图片数据集训练深度张量列网络模型，使得识别模块能自动学习图片特征，检测出图片中的学生异常行为。最后将训练好的模型投入实际使用，实时获取图像采集模块中传输来的3张图像(本专利以三幅图像为例进行说明，硬件设备许可的条件下，可以实时采集多幅图像)，分别检测图片中的学生异常行为，并根据给定的概率阈值框出行为异常学生。
所述提示模块，用于将所述识别模块的识别结果通知授课教师。
提示模块，提示模块实时的将识别结果综合以某种方式通知授课教师，若3个角度的图像都无异常则不通知，教师可通过调节概率阈值以调节识别灵敏度。教师在接收到提示后可以实时了解课堂学生的听课状态和对其所教授的内容的接受程度，可以以此为基础对其中接受程度不是很好的同学重点提问或采取相应的对策。
优选的，还设置有：与所述识别模块连接的存储模块；所述存储模块，用于同步存档所述识别结果并进行编辑分类。
存储模块，存储模块是将该***的所有识别的最终结果以班级为目，学生为类，以学生个人档案的形式进行存储，学校可以充分利用这些电子档案， 从中挖掘出有用信息，一方面可以根据学生整体接受情况来分析和评估教学中的不足，另一方面可以分析学生的学习曲线，找到学生成绩不好的真正原因，可以有针对性地进行查缺补漏。
所述识别模块还包括：
表情识别单元，用于使用训练好的深度张量列网络模型来对所述课堂图像中的学生进行表情识别。
所述表情识别单元包括人脸检测子单元和卷积神经网络分类器。
表情识别子模块方法与行为检测类似，不同之处在于标注异常表情进行算法训练。本专利中以两子模块以相同算法模型分开并行识别方式进行描述，但也可以通过改变损失函数，通过多任务损失函数将两项任务融合到同一模型中识别，此处不做具体阐述，但亦在本专利保护范围之内。
本发明所要求保护的方案很好的解决了通过对课堂图像进行分析处理以辅助老师进行教学活动的技术问题，避免了现有的教学设备过于依赖外部的图像识别装置导致硬件要求高且识别不准确的缺陷，提升了老师教学工作的效率。
上述内容，仅为本发明的较佳实施例，并非用于限制本发明的实施方案，本领域普通技术人员根据本发明的主要构思和精神，可以十分方便地进行相应的变通或修改，故本发明的保护范围应以权利要求书所要求的保护范围为准。

Claims (11)

  1. 一种教学辅助方法，包括以下顺序步骤：
    s1.采集模块实时采集现场的课堂图像，并传输给识别模块；
    s2.所述识别模块对所述课堂图像进行分析，并判断所述课堂图像中行为异常的学生；
    s3.提示模块将所述识别模块的识别结果通知授课教师；
    其特征在于，所述步骤s2中包括以下步骤：
    s21.所述识别模块使用训练好的深度张量列网络模型来对所述课堂图像中的学生进行行为检测。
  2. 如权利要求1所述的一种教学辅助方法，其特征在于，所述步骤s2还包括以下步骤：
    s22.所述识别模块使用训练好的深度张量列网络模型来对所述课堂图像中的学生进行表情识别。
  3. 如权利要求2所述的一种教学辅助方法，其特征在于，所述步骤s22具体包括以下步骤：
    s221.通过人脸检测子单元从所述采集模块采集到的所述课堂图像中识别出各学生的人脸区域；
    s222.通过卷积神经网络分类器对检测到的所述人脸区域做表情识别。
  4. 如权利要求1所述的一种教学辅助方法，其特征在于，步骤s1中包括以下步骤：
    s11.所述采集模块在教室前方的左、中、右区域分别安装图像采集装置；
    s12.所述图像采集模块以班级中所有学生上半身图像为采集目标。
  5. 如权利要求1所述的一种教学辅助方法，其特征在于，还包括以下步骤：s4.存储模块同步存档所述识别结果。
  6. 如权利要求5所述的一种教学辅助方法，其特征在于，所述步骤s4中包括以下步骤：
    s41.将每个学生对应的所述识别结果按班级制定成学生电子档案；
    s42.根据所述学生电子档案绘出学生上课状态曲线，用以便于授课教师结合当时教授的内容以及考试成绩对学生进行有针对性的辅导。
  7. 如权利要求1所述的一种教学辅助方法，其特征在于，步骤s1之前 还包括以下步骤：
    q1.构建数据集；
    q2.训练所述深度张量列网络模型。
  8. 如权利要求7所述的一种教学辅助方法，其特征在于，所述步骤q1包括以下步骤：
    q11.所述采集模块在教室长时间拍摄所述课堂图像并存储；
    q12.选取存在异常的学生图片进行标注。
  9. 如权利要求8所述的一种教学辅助方法，其特征在于，所述步骤q2包括以下步骤：
    q21.通过神经网络模型的多层卷积层提取已标注的所述学生图片中的异常特征，所述异常特征与分解后的全连接层权重矩阵运算得到输出预测值；
    q22.所述输出预测值与所述学生图片中的异常行为学生真实标注值的误差构成的损失函数；
    q23.根据所述损失函数调整网络参数，得到训练好的深度张量列网络模型。
  10. 一种教学辅助***，其特征在于，设置有：采集模块、与所述采集模块连接的识别模块、与所述识别模块连接的提示模块；
    所述采集模块，用于实时采集现场的课堂图像并传输给识别模块；
    所述识别模块，用于对所述课堂图像进行分析，并判断所述课堂图像中行为异常的学生；所述识别模块包括：
    行为检测单元，用于使用训练好的深度张量列网络模型来对所述课堂图像中的学生进行行为检测；
    所述提示模块，用于将所述识别模块的识别结果通知授课教师。
  11. 如权利要求10所述的一种教学辅助***，其特征在于，还设置有：与所述识别模块连接的存储模块；所述存储模块，用于同步存档所述识别结果并进行编辑分析；
    所述识别模块还包括：
    表情识别单元，用于使用训练好的深度张量列网络模型来对所述课堂图像中的学生进行表情识别；
    所述表情识别单元包括人脸检测子单元和卷积神经网络分类器。
PCT/CN2017/096217 2017-08-07 2017-08-07 一种教学辅助方法及采用该方法的教学辅助*** WO2019028592A1 (zh)

Priority Applications (3)

Application Number Priority Date Filing Date Title
PCT/CN2017/096217 WO2019028592A1 (zh) 2017-08-07 2017-08-07 一种教学辅助方法及采用该方法的教学辅助***
US16/623,397 US11270526B2 (en) 2017-08-07 2017-08-07 Teaching assistance method and teaching assistance system using said method
JP2020522765A JP6892558B2 (ja) 2017-08-07 2017-08-07 教学補助方法およびその方法を採用する教学補助システム

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2017/096217 WO2019028592A1 (zh) 2017-08-07 2017-08-07 一种教学辅助方法及采用该方法的教学辅助***

Publications (1)

Publication Number Publication Date
WO2019028592A1 true WO2019028592A1 (zh) 2019-02-14

Family

ID=65273159

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/096217 WO2019028592A1 (zh) 2017-08-07 2017-08-07 一种教学辅助方法及采用该方法的教学辅助***

Country Status (3)

Country Link
US (1) US11270526B2 (zh)
JP (1) JP6892558B2 (zh)
WO (1) WO2019028592A1 (zh)

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110119702A (zh) * 2019-04-30 2019-08-13 西安理工大学 基于深度学习先验的人脸表情识别方法
CN110647807A (zh) * 2019-08-14 2020-01-03 中国平安人寿保险股份有限公司 异常行为确定方法、装置、计算机设备和存储介质
CN110728194A (zh) * 2019-09-16 2020-01-24 中国平安人寿保险股份有限公司 基于微表情和动作识别的智能培训方法、装置及存储介质
CN111027584A (zh) * 2019-10-23 2020-04-17 宋飞 一种课堂行为识别方法及设备
CN111079554A (zh) * 2019-11-25 2020-04-28 恒安嘉新（北京）科技股份公司 分析学生课堂表现的方法、装置、电子设备、及存储介质
CN111126302A (zh) * 2019-12-26 2020-05-08 上海金桥信息股份有限公司 一种移动终端的多面部识别方法
CN111950708A (zh) * 2020-08-11 2020-11-17 华中师范大学 一种发现大学生日常生活习惯的神经网络结构与方法
CN112396923A (zh) * 2020-11-25 2021-02-23 贵州轻工职业技术学院 一种市场营销的教学模拟***
CN112598550A (zh) * 2020-12-24 2021-04-02 苏州大学 基于行为分析的学生活动多维管理***及管理方法
CN113486700A (zh) * 2021-05-08 2021-10-08 北京科技大学 一种教学场景下基于注意力机制的面部表情分析方法
CN114140282A (zh) * 2021-11-19 2022-03-04 武汉东信同邦信息技术有限公司 一种基于深度学习的普教课堂答题快速评审方法及装置
CN117152688A (zh) * 2023-10-31 2023-12-01 江西拓世智能科技股份有限公司 一种基于人工智能的智慧课堂行为分析方法及***
CN117649702A (zh) * 2024-01-30 2024-03-05 山东诚海电子科技有限公司 一种基于人体姿态的教学辅助方法、设备及介质
CN113052041B (zh) * 2021-03-17 2024-04-30 广东骏杰科技有限公司 一种具备智能化教学***和监测方法
CN117972438A (zh) * 2024-03-31 2024-05-03 浪潮电子信息产业股份有限公司 一种数据处理方法、***、设备及存储介质

Families Citing this family (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11037330B2 (en) * 2017-04-08 2021-06-15 Intel Corporation Low rank matrix compression
CN111723617B (zh) * 2019-03-20 2023-10-27 顺丰科技有限公司 动作识别的方法、装置、设备及存储介质
CN112580910A (zh) * 2019-09-29 2021-03-30 鸿富锦精密电子(天津)有限公司 教学调查问卷分析方法、电子装置及存储介质
CN111476690B (zh) * 2020-05-15 2023-04-14 重庆医药高等专科学校 一种多模式家校沟通***及方法
CN111898473B (zh) * 2020-07-10 2023-09-01 华南农业大学 一种基于深度学习的司机状态实时监测方法
CN111967350A (zh) * 2020-07-30 2020-11-20 广州光大教育软件科技股份有限公司 远程课堂专注度分析方法、装置、计算机设备及存储介质
CN111857010A (zh) * 2020-07-31 2020-10-30 重庆电子工程职业学院 一种用于实训场所运维管控的辅助支持***
CN113239915B (zh) * 2021-07-13 2021-11-30 北京邮电大学 一种课堂行为的识别方法、装置、设备及存储介质
CN113569761B (zh) * 2021-07-30 2023-10-27 广西师范大学 一种基于深度学习的学生视点估计方法
CN113792626A (zh) * 2021-08-30 2021-12-14 华中师范大学 一种基于教师非言语行为的教学过程评价方法
CN114419861A (zh) * 2022-01-20 2022-04-29 牡丹江师范学院 一种基于互联网大学教学用英语智能教学辅助***
CN114898460B (zh) * 2022-04-22 2024-04-26 华中师范大学 一种基于图卷积神经网络的教师非言语行为检测方法
CN114898449B (zh) * 2022-07-13 2022-10-04 电子科技大学成都学院 一种基于大数据的外语教学辅助方法及装置
CN115909152B (zh) * 2022-11-16 2023-08-29 北京师范大学 一种基于群体行为的教学场景智能分析***
CN116682052A (zh) * 2023-04-20 2023-09-01 北京北水融智教育科技有限公司 一种基于云服务的教学服务平台
CN117711132A (zh) * 2024-02-05 2024-03-15 青岛培诺教育科技股份有限公司 一种学生安全监测方法、装置、电子设备及存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102945624A (zh) * 2012-11-14 2013-02-27 南京航空航天大学 基于云计算模型和表情信息反馈的智能化视频教学***
CN103559812A (zh) * 2013-11-07 2014-02-05 大连东方之星信息技术有限公司 一种教育督导评价报告生成***
US20150044657A1 (en) * 2013-08-07 2015-02-12 Xerox Corporation Video-based teacher assistance
CN106097790A (zh) * 2016-08-31 2016-11-09 王翠丽 一种教学辅助装置
CN106778745A (zh) * 2016-12-23 2017-05-31 深圳先进技术研究院 一种车牌识别方法及装置、用户设备
CN106778539A (zh) * 2016-11-25 2017-05-31 鲁东大学 教学效果信息获取方法及装置

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8491311B2 (en) * 2002-09-30 2013-07-23 Mind Research Institute System and method for analysis and feedback of student performance
US9730643B2 (en) * 2013-10-17 2017-08-15 Siemens Healthcare Gmbh Method and system for anatomical object detection using marginal space deep neural networks
US10460194B2 (en) * 2014-03-07 2019-10-29 Lior Wolf System and method for the detection and counting of repetitions of repetitive activity via a trained network
US9641585B2 (en) * 2015-06-08 2017-05-02 Cisco Technology, Inc. Automated video editing based on activity in video conference
US9792492B2 (en) * 2015-07-07 2017-10-17 Xerox Corporation Extracting gradient features from neural networks
US9916522B2 (en) * 2016-03-11 2018-03-13 Kabushiki Kaisha Toshiba Training constrained deconvolutional networks for road scene semantic segmentation
GB201613138D0 (en) * 2016-07-29 2016-09-14 Unifai Holdings Ltd Computer vision systems
US10748062B2 (en) * 2016-12-15 2020-08-18 WaveOne Inc. Deep learning based adaptive arithmetic coding and codelength regularization
US11301944B2 (en) * 2017-04-13 2022-04-12 International Business Machines Corporation Configuring classroom physical resources
US11120127B2 (en) * 2017-12-27 2021-09-14 Nec Corporation Reconstruction-based anomaly detection
WO2020019236A1 (en) * 2018-07-26 2020-01-30 Intel Corporation Loss-error-aware quantization of a low-bit neural network

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102945624A (zh) * 2012-11-14 2013-02-27 南京航空航天大学 基于云计算模型和表情信息反馈的智能化视频教学***
US20150044657A1 (en) * 2013-08-07 2015-02-12 Xerox Corporation Video-based teacher assistance
CN103559812A (zh) * 2013-11-07 2014-02-05 大连东方之星信息技术有限公司 一种教育督导评价报告生成***
CN106097790A (zh) * 2016-08-31 2016-11-09 王翠丽 一种教学辅助装置
CN106778539A (zh) * 2016-11-25 2017-05-31 鲁东大学 教学效果信息获取方法及装置
CN106778745A (zh) * 2016-12-23 2017-05-31 深圳先进技术研究院 一种车牌识别方法及装置、用户设备

Cited By (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110119702A (zh) * 2019-04-30 2019-08-13 西安理工大学 基于深度学习先验的人脸表情识别方法
CN110119702B (zh) * 2019-04-30 2022-12-06 西安理工大学 基于深度学习先验的人脸表情识别方法
CN110647807A (zh) * 2019-08-14 2020-01-03 中国平安人寿保险股份有限公司 异常行为确定方法、装置、计算机设备和存储介质
CN110728194A (zh) * 2019-09-16 2020-01-24 中国平安人寿保险股份有限公司 基于微表情和动作识别的智能培训方法、装置及存储介质
CN111027584A (zh) * 2019-10-23 2020-04-17 宋飞 一种课堂行为识别方法及设备
CN111079554A (zh) * 2019-11-25 2020-04-28 恒安嘉新（北京）科技股份公司 分析学生课堂表现的方法、装置、电子设备、及存储介质
CN111126302A (zh) * 2019-12-26 2020-05-08 上海金桥信息股份有限公司 一种移动终端的多面部识别方法
CN111950708A (zh) * 2020-08-11 2020-11-17 华中师范大学 一种发现大学生日常生活习惯的神经网络结构与方法
CN111950708B (zh) * 2020-08-11 2023-10-03 华中师范大学 一种发现大学生日常生活习惯的神经网络结构与方法
CN112396923A (zh) * 2020-11-25 2021-02-23 贵州轻工职业技术学院 一种市场营销的教学模拟***
CN112396923B (zh) * 2020-11-25 2023-09-19 贵州轻工职业技术学院 一种市场营销的教学模拟***
CN112598550A (zh) * 2020-12-24 2021-04-02 苏州大学 基于行为分析的学生活动多维管理***及管理方法
CN112598550B (zh) * 2020-12-24 2024-03-26 苏州大学 基于行为分析的学生活动多维管理***及管理方法
CN113052041B (zh) * 2021-03-17 2024-04-30 广东骏杰科技有限公司 一种具备智能化教学***和监测方法
CN113486700A (zh) * 2021-05-08 2021-10-08 北京科技大学 一种教学场景下基于注意力机制的面部表情分析方法
CN114140282A (zh) * 2021-11-19 2022-03-04 武汉东信同邦信息技术有限公司 一种基于深度学习的普教课堂答题快速评审方法及装置
CN117152688A (zh) * 2023-10-31 2023-12-01 江西拓世智能科技股份有限公司 一种基于人工智能的智慧课堂行为分析方法及***
CN117649702A (zh) * 2024-01-30 2024-03-05 山东诚海电子科技有限公司 一种基于人体姿态的教学辅助方法、设备及介质
CN117649702B (zh) * 2024-01-30 2024-05-07 山东诚海电子科技有限公司 一种基于人体姿态的教学辅助方法、设备及介质
CN117972438A (zh) * 2024-03-31 2024-05-03 浪潮电子信息产业股份有限公司 一种数据处理方法、***、设备及存储介质

Also Published As

Publication number Publication date
JP6892558B2 (ja) 2021-06-23
JP2020525965A (ja) 2020-08-27
US11270526B2 (en) 2022-03-08
US20200175264A1 (en) 2020-06-04

Similar Documents

Publication Publication Date Title
WO2019028592A1 (zh) 一种教学辅助方法及采用该方法的教学辅助***
CN108073888A (zh) 一种教学辅助方法及采用该方法的教学辅助***
CN109284737A (zh) 一种用于智慧教室的学生行为分析和识别***
CN108399376A (zh) 学生课堂学习兴趣智能分析方法及***
CN109117731A (zh) 一种课堂教学认知负荷测量***
CN110991381A (zh) 一种基于行为和语音智能识别的实时课堂学生状态分析与指示提醒***和方法
CN107491890A (zh) 一种可量化课堂教学质量评估***及方法
CN107316257A (zh) 一种基于课堂学生行为分析的教学质量评估方法及***
CN110889672A (zh) 一种基于深度学习的学生打卡及上课状态的检测***
WO2021047185A1 (zh) 基于人脸识别的监测方法、装置、存储介质及计算机设备
CN108171414A (zh) 教学质量评估***
CN109214664B (zh) 一种基于人工智能的情绪行为综合分析***
CN108182649A (zh) 用于教学质量评估的智能机器人
CN110619460A (zh) 基于深度学习目标检测的教室课堂质量评估***及方法
CN114898460B (zh) 一种基于图卷积神经网络的教师非言语行为检测方法
CN113705349A (zh) 一种基于视线估计神经网络的注意力量化分析方法及***
CN113361352A (zh) 基于行为识别的学生课堂行为分析的监控方法及***
CN116050892A (zh) 一种基于人工智能的智能教育评价监督方法
CN111523445A (zh) 一种基于改进的Openpose模型和面部微表情的考试行为检测方法
CN111178263A (zh) 一种实时表情分析方法及其装置
Yang et al. Student in-class behaviors detection and analysis system based on CBAM-YOLOv5
CN111199378B (zh) 学员管理方法、装置、电子设备及存储介质
CN113688789B (zh) 一种基于深度学习的在线学习投入度识别方法及***
Yang et al. Scb-dataset3: A benchmark for detecting student classroom behavior
CN111444877B (zh) 一种基于视频照片的教室人数识别方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17920650

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2020522765

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17920650

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 25/08/2020)

122 Ep: pct application non-entry in european phase

Ref document number: 17920650

Country of ref document: EP

Kind code of ref document: A1