CN108921037A - A kind of Emotion identification method based on BN-inception binary-flow network - Google Patents
A kind of Emotion identification method based on BN-inception binary-flow network Download PDFInfo
- Publication number
- CN108921037A CN108921037A CN201810579049.5A CN201810579049A CN108921037A CN 108921037 A CN108921037 A CN 108921037A CN 201810579049 A CN201810579049 A CN 201810579049A CN 108921037 A CN108921037 A CN 108921037A
- Authority
- CN
- China
- Prior art keywords
- network
- inception
- binary
- flow
- spp
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Biology (AREA)
- General Health & Medical Sciences (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The individual Emotion identification method based on posture information that the present invention provides a kind of relates generally to the mood that individual is judged using deep learning technique study individual posture.This method includes：It is firstly introduced into and the static state and behavioral characteristics of list entries is extracted by the study to original image and light stream image based on BN-inception binary-flow network model；Then spatial pyramid pond (Space Pyramid Pooling, SPP) is added on the basis of binary-flow network, image is made to input network with original size, to reduce because being influenced caused by deforming model performance.The invention firstly uses binary-flow networks to carry out the study of space-time characteristic to list entries, and introduces pyramid pond, to retain the raw information of video frame, enables network effectively to learn the feature of individual posture mood, obtains higher discrimination.
Description
Technical field
The present invention relates to the Emotion identification problems in deep learning field, are based on BN-inception more particularly, to one kind
The individual mood analysis method of+SPP binary-flow network.
Background technique
Mood is the feeling for combining people, a kind of state of thought and act, is played in the exchange of person to person important
Effect.Its emotional state can be judged generally according to people's facial expression, but in certain specific environment, such as monitoring view
In the presence of when blocking, we can not necessarily obtain clearly face facial expression for angle, face.In fact, people
True emotional not only relies on human face expression to be expressed, and individual limb action can also express certain emotional information.Therefore,
Research of the invention is concentrated mainly on the Emotion identification of the individual posture based on video.
Emotion identification is important research content and direction in computer vision field, the International Periodicals of current many authoritys
It is equipped with relevant theme and content with top-level meeting, and many external elite schools have also opened up relevant course.It is traditional based on view
The Emotion identification method of frequency relies primarily on the feature manually chosen, this method take time and effort and the model parameter that obtains it is extensive
Performance is poor, serves the limitation of Emotion identification.Deep learning is an important component of artificial intelligence field development,
The very popular research direction of artificial intelligence field is had become in recent years.It is in many fields (such as image recognition, speech recognition
Deng) very big breakthrough is all achieved, higher discrimination and generalization ability have especially been achieved in video analysis.Therefore originally
Patent utilizes advantage of the deep learning in video analysis, studies Emotion identification individual in video.
Just grown up based on the Emotion identification of posture information in recent years, relevant research is less, focuses primarily upon biography
The research of system algorithm.Li et al.[1]It proposes and a kind of raw skeleton coordinate and skeleton motion is utilized to carry out Activity recognition and classification；
Piana etc.[2]It is proposed a mood automatic identification model and system from all-around exercises, it is used to help autism children
Association identifies and is showed emotion by all-around exercises.Equally also someone by the motion feature of human body attitude and advanced kinematics
Geometrical characteristic is combined, and is carried out cluster and is classified.Crenn etc.[3]It is low that operation data etc. is obtained using the 3D frame sequence of people
Then feature decomposition is geometrical characteristic, motion feature and three kinds of Fourier's feature, calculates the member of these low-level features by layer feature
Feature (such as mean value, standard deviation) finally uses classifier by first tagsort.Deep learning regardless of recognition time still
It in accuracy, is compared with the traditional method and all improves a lot, but due to the shortage of mood data collection relevant to posture, adopt
It is also seldom that the individual Emotion identification correlative study based on posture information is carried out with deep learning.
Summary of the invention
The individual Emotion identification method based on posture that the object of the present invention is to provide a kind of, will be in deep learning and video
Human body attitude combines, and makes full use of the superiority of BN-inception+SPP network structure, while introducing binary-flow network structure
The individual Emotion identification based on video is carried out, effectively learns the emotional characteristics of individual posture, obtains higher discrimination.
For convenience of explanation, it is firstly introduced into following concept：
Optical flow method：It is a kind of expression way of simple and practical image motion, is normally defined in an image sequence
The apparent motion of brightness of image mode, i.e., the movement velocity of the point on space object surface is on the imaging plane of visual sensor
Expression.
Convolutional neural networks：A kind of multilayer feedforward neural network, every layer is made of multiple two-dimensional surfaces, the mind of each plane
It works independently through member, convolutional neural networks include convolutional layer and pond layer.
Double-current convolutional neural networks：For video behavioural characteristic extraction and design, network with single frames RGB original image and
The light stream image obtained based on video data is inputted respectively as two, is indicated with realizing the apparent information in object of action space
And the extraction of action process temporal aspect.
Spatial pyramid pond (Space Pyramid Pooling, SPP)：It is to be composed of multiple down-sampled layers, it
The fixed feature vector of a length can be converted to dividing to input feature vector figure from thick to thin, and characteristic pattern,
So SPP layers can extract various local messages.
The present invention specifically adopts the following technical scheme that：
Propose the individual Emotion identification method based on BN-inception+SPP binary-flow network, the main feature of this method
It is：
1. individual attitude data collection is divided into four mood classifications：Boring (bored), exciting (excited) is angry
(frantic), loosen (relaxed)；
2. spatial pyramid pond (Space is added before the full articulamentum of BN-inception binary-flow network
Pyramid Pooling, SPP), carry out the training of time-space network respectively to data set；
This method mainly includes the following steps that：
(1) by individual posture sequence data collection, four mood classifications are divided into：Boring, excitement is angry, loosens；
(2) the corresponding light stream image sequence of data set is generated using the optical flow algorithm of document [4], indicates the fortune of individual posture
Dynamic feature；
(3) raw data set and optical flow data collection are proportionally divided into training set, verifying collection and test set respectively；
(4) the double-current convolutional neural networks model based on BN-inception is introduced, and is added before its full articulamentum
SPP layers of optimization BN-inception network, using training set and verifying collection carry out time-space network training, using test set into
Row verifying；
(5) by based on BN-inception+SPP spatial flow and two channel network of time flow carry out average fusion, obtain
Accuracy ACC (Accuracy) and macro accuracy of the mean MAP (Macro Average Precision) on test set.
Detailed description of the invention
Fig. 1 is that the present invention is based on the individual Emotion identification general frame schematic diagrames of BN-inception+SPP binary-flow network.
Fig. 2-a~Fig. 2-b is the accuracy confusion matrix that the present invention obtains on test set when SPP layers are not added,
In, 2-a is the test matrix of spatial flow BN-inception network, and 2-b is the test square of time flow BN-inception network
Battle array.
Fig. 3-a~Fig. 3-b is the accuracy confusion matrix that the present invention obtains on test set when being added SPP layers, wherein
3-a is the test matrix of spatial flow BN-inception+SPP network, and 3-b is time flow BN-inception+SPP network
Test matrix.
Fig. 4 is that the spatial flow by based on BN-inception+SPP and two channel network of time flow of the invention averagely merge
Afterwards, the ACC and MAP on test set are obtained.
Specific embodiment
The present invention is described in further detail with reference to the accompanying drawings and embodiments, it is necessary to, it is noted that below
Embodiment is served only for that the present invention is described further, should not be understood as limiting the scope of the invention, fields
Personnel be skillful at according to foregoing invention content, some nonessential modifications and adaptations are made to the present invention and are embodied,
Protection scope of the present invention should be still fallen within.
In Fig. 1, a kind of individual Emotion identification method based on BN-inception+SPP binary-flow network, including following step
Suddenly：
(1) it first after obtaining the individual data items collection in public space, is generated using the optical flow algorithm of document [4] former
The light stream image sequence of beginning data set indicates the motion feature of individual posture；
(2) raw data set and obtained optical flow data collection are respectively divided into test set in proportion, verifying collects and training
Collect three parts, and gives corresponding mood classification；
(3) remove SPP layers shown in FIG. 1, the data of training set and verifying collection inputted time-space network respectively and learn,
Training pattern is obtained, test is carried out using the data of test set and verifies its effect；
(4) SPP layers are added, training set is inputted into time-space network with original size respectively and is learnt, training pattern is obtained,
Test, which is carried out, using the data of test set collection verifies its effect；
(5) it by the spatial flow based on BN-inception+SPP and after two channel network of time flow averagely merges, is surveyed
ACC and MAP on examination collection；
The present invention is separately trained the convolutional neural networks in two channels of spatial flow and time flow using Caffe, leads to
It crosses experiment the parameter of time flow and space flow network is arranged, as shown in table 1.Due to the individual posture mood number of foundation
It is less according to the sample size of collection, to prevent over-fitting, Dropout layers of side is added using data extending and in a network
Method.
The setting of 1 training parameter of table
Bibliography：
[1]Li C,Zhong Q,Xie D,et al.Skeleton-based Action Recognition with
Convolutional Neural Networks[J]. 2017:597-600.
[2]Piana S,StaglianòA,Odone F,et al.Adaptive Body Gesture
Representation for Automatic Emotion Recognition[J].ACM Transactions on
Interactive Intelligent Systems(TiiS),2016,6(1):6.
[3]Crenn A,Khan R A,Meyer A,et al.Body Expression Recognition from
Animated 3D Skeleton[C]// International Conference on 3D Imaging.IEEE,2017:1-
7.
[4]Brox T,Bruhn A,Papenberg N,et al.High Accuracy Optical Flow
Estimation Based on A Theory for Warping[C]//European Conference on Computer
Vision(ECCV),2004:25-36.
Claims (3)
1. a kind of individual Emotion identification method based on BN-inception+SPP binary-flow network, it is characterised in that：
A. individual attitude data collection is divided into four mood classifications：Boring (bored), exciting (excited) is angry
(frantic), loosen (relaxed)；
B. spatial pyramid pond (Space Pyramid is added before the full articulamentum of BN-inception binary-flow network
Pooling, SPP), carry out the training of time-space network respectively to data set；
This method mainly includes the following steps that：
(1) using the corresponding light stream image sequence of optical flow algorithm processing data set generation of document [1], the fortune of individual posture is indicated
Dynamic feature；
(2) data set is divided into training set, verifying collection and test set three parts, and gives the mood classification of each sequence；
(3) the double-current convolutional neural networks model based on BN-inception is introduced, and SPP layers are added before its full articulamentum
Optimize BN-inception network, the training of time-space network is carried out using training set and verifying collection, is verified using test set；
(4) by based on BN-inception+SPP spatial flow and two channel network of time flow carry out average fusion, tested
Accuracy ACC (Accuracy) and macro accuracy of the mean MAP (Macro Average Precision) on collection.
2. the individual Emotion identification method based on BN-inception+SPP binary-flow network as described in claim 1, feature
It is respectively to learn the space-time characteristic of data set using binary-flow network in step (3).
3. as described in claim 1 based on the Emotion identification method of BN-inception binary-flow network, it is characterised in that in step
Suddenly SPP layers are added in (3) first before the full articulamentum of BN-inception binary-flow network, so that training set is with original size
Network is inputted, avoids fixed input size bring motion information from losing, time-space network is then carried out respectively to data set again
Training.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810579049.5A CN108921037B (en) | 2018-06-07 | 2018-06-07 | Emotion recognition method based on BN-acceptance double-flow network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810579049.5A CN108921037B (en) | 2018-06-07 | 2018-06-07 | Emotion recognition method based on BN-acceptance double-flow network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108921037A true CN108921037A (en) | 2018-11-30 |
| CN108921037B CN108921037B (en) | 2022-06-03 |
Family
ID=64418934
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810579049.5A Active CN108921037B (en) | 2018-06-07 | 2018-06-07 | Emotion recognition method based on BN-acceptance double-flow network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108921037B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109766856A (en) * | 2019-01-16 | 2019-05-17 | 华南农业大学 | A kind of method of double fluid RGB-D Faster R-CNN identification milking sow posture |
| CN109814565A (en) * | 2019-01-30 | 2019-05-28 | 上海海事大学 | The unmanned boat intelligence navigation control method of space-time double fluid data-driven depth Q study |
| CN109886160A (en) * | 2019-01-30 | 2019-06-14 | 浙江工商大学 | It is a kind of it is non-limiting under the conditions of face identification method |
| CN110147729A (en) * | 2019-04-16 | 2019-08-20 | 深圳壹账通智能科技有限公司 | User emotion recognition methods, device, computer equipment and storage medium |
| CN110175596A (en) * | 2019-06-04 | 2019-08-27 | 重庆邮电大学 | The micro- Expression Recognition of collaborative virtual learning environment and exchange method based on double-current convolutional neural networks |
| CN110414561A (en) * | 2019-06-26 | 2019-11-05 | 武汉大学 | A kind of construction method of the natural scene data set suitable for machine vision |
| CN112131908A (en) * | 2019-06-24 | 2020-12-25 | 北京眼神智能科技有限公司 | Action identification method and device based on double-flow network, storage medium and equipment |
| WO2022037642A1 (en) * | 2020-08-19 | 2022-02-24 | 南京图格医疗科技有限公司 | Method for detecting and classifying lesion area in clinical image |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050265580A1 (en) * | 2004-05-27 | 2005-12-01 | Paul Antonucci | System and method for a motion visualizer |
| CN102663429A (en) * | 2012-04-11 | 2012-09-12 | 上海交通大学 | Method for motion pattern classification and action recognition of moving target |
| CN103544963A (en) * | 2013-11-07 | 2014-01-29 | 东南大学 | Voice emotion recognition method based on core semi-supervised discrimination and analysis |
| CN104732203A (en) * | 2015-03-05 | 2015-06-24 | 中国科学院软件研究所 | Emotion recognizing and tracking method based on video information |
| CN106295568A (en) * | 2016-08-11 | 2017-01-04 | 上海电力学院 | The mankind's naturalness emotion identification method combined based on expression and behavior bimodal |
| CN106897671A (en) * | 2017-01-19 | 2017-06-27 | 山东中磁视讯股份有限公司 | A kind of micro- expression recognition method encoded based on light stream and FisherVector |
| CN107368798A (en) * | 2017-07-07 | 2017-11-21 | 四川大学 | A kind of crowd's Emotion identification method based on deep learning |
| CN107491731A (en) * | 2017-07-17 | 2017-12-19 | 南京航空航天大学 | A kind of Ground moving target detection and recognition methods towards precision strike |
| CN107784114A (en) * | 2017-11-09 | 2018-03-09 | 广东欧珀移动通信有限公司 | Recommendation method, apparatus, terminal and the storage medium of facial expression image |
| CN107944442A (en) * | 2017-11-09 | 2018-04-20 | 北京智芯原动科技有限公司 | Based on the object test equipment and method for improving convolutional neural networks |
-
2018
- 2018-06-07 CN CN201810579049.5A patent/CN108921037B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050265580A1 (en) * | 2004-05-27 | 2005-12-01 | Paul Antonucci | System and method for a motion visualizer |
| CN102663429A (en) * | 2012-04-11 | 2012-09-12 | 上海交通大学 | Method for motion pattern classification and action recognition of moving target |
| CN103544963A (en) * | 2013-11-07 | 2014-01-29 | 东南大学 | Voice emotion recognition method based on core semi-supervised discrimination and analysis |
| CN104732203A (en) * | 2015-03-05 | 2015-06-24 | 中国科学院软件研究所 | Emotion recognizing and tracking method based on video information |
| CN106295568A (en) * | 2016-08-11 | 2017-01-04 | 上海电力学院 | The mankind's naturalness emotion identification method combined based on expression and behavior bimodal |
| CN106897671A (en) * | 2017-01-19 | 2017-06-27 | 山东中磁视讯股份有限公司 | A kind of micro- expression recognition method encoded based on light stream and FisherVector |
| CN107368798A (en) * | 2017-07-07 | 2017-11-21 | 四川大学 | A kind of crowd's Emotion identification method based on deep learning |
| CN107491731A (en) * | 2017-07-17 | 2017-12-19 | 南京航空航天大学 | A kind of Ground moving target detection and recognition methods towards precision strike |
| CN107784114A (en) * | 2017-11-09 | 2018-03-09 | 广东欧珀移动通信有限公司 | Recommendation method, apparatus, terminal and the storage medium of facial expression image |
| CN107944442A (en) * | 2017-11-09 | 2018-04-20 | 北京智芯原动科技有限公司 | Based on the object test equipment and method for improving convolutional neural networks |
Non-Patent Citations (2)
| Title |
|---|
| KAIMING HE ET AL: "Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition", 《IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE》 * |
| 陈胜娣 等: "基于改进的深度卷积神经网络的人体动作识别方法", 《计算机应用研究》 * |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109766856A (en) * | 2019-01-16 | 2019-05-17 | 华南农业大学 | A kind of method of double fluid RGB-D Faster R-CNN identification milking sow posture |
| CN109766856B (en) * | 2019-01-16 | 2022-11-15 | 华南农业大学 | Method for recognizing postures of lactating sows through double-current RGB-D Faster R-CNN |
| CN109814565A (en) * | 2019-01-30 | 2019-05-28 | 上海海事大学 | The unmanned boat intelligence navigation control method of space-time double fluid data-driven depth Q study |
| CN109886160A (en) * | 2019-01-30 | 2019-06-14 | 浙江工商大学 | It is a kind of it is non-limiting under the conditions of face identification method |
| CN110147729A (en) * | 2019-04-16 | 2019-08-20 | 深圳壹账通智能科技有限公司 | User emotion recognition methods, device, computer equipment and storage medium |
| CN110175596A (en) * | 2019-06-04 | 2019-08-27 | 重庆邮电大学 | The micro- Expression Recognition of collaborative virtual learning environment and exchange method based on double-current convolutional neural networks |
| CN112131908A (en) * | 2019-06-24 | 2020-12-25 | 北京眼神智能科技有限公司 | Action identification method and device based on double-flow network, storage medium and equipment |
| CN112131908B (en) * | 2019-06-24 | 2024-06-11 | 北京眼神智能科技有限公司 | Action recognition method, device, storage medium and equipment based on double-flow network |
| CN110414561A (en) * | 2019-06-26 | 2019-11-05 | 武汉大学 | A kind of construction method of the natural scene data set suitable for machine vision |
| WO2022037642A1 (en) * | 2020-08-19 | 2022-02-24 | 南京图格医疗科技有限公司 | Method for detecting and classifying lesion area in clinical image |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108921037B (en) | 2022-06-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108921037A (en) | A kind of Emotion identification method based on BN-inception binary-flow network | |
| CN107292813B (en) | A kind of multi-pose Face generation method based on generation confrontation network | |
| CN105005774B (en) | A kind of recognition methods of face kinship and device based on convolutional neural networks | |
| CN107423678A (en) | A kind of training method and face identification method of the convolutional neural networks for extracting feature | |
| CN108830252A (en) | A kind of convolutional neural networks human motion recognition method of amalgamation of global space-time characteristic | |
| CN109919031A (en) | A kind of Human bodys' response method based on deep neural network | |
| CN111274921B (en) | Method for recognizing human body behaviors by using gesture mask | |
| CN109508669A (en) | A kind of facial expression recognizing method based on production confrontation network | |
| CN109101865A (en) | A kind of recognition methods again of the pedestrian based on deep learning | |
| CN106570474A (en) | Micro expression recognition method based on 3D convolution neural network | |
| CN108052884A (en) | A kind of gesture identification method based on improvement residual error neutral net | |
| CN105160310A (en) | 3D (three-dimensional) convolutional neural network based human body behavior recognition method | |
| CN107092894A (en) | A kind of motor behavior recognition methods based on LSTM models | |
| CN109376720A (en) | Classification of motion method based on artis space-time simple cycle network and attention mechanism | |
| CN107392131A (en) | A kind of action identification method based on skeleton nodal distance | |
| CN110232361B (en) | Human behavior intention identification method and system based on three-dimensional residual dense network | |
| CN104063721B (en) | A kind of human behavior recognition methods learnt automatically based on semantic feature with screening | |
| CN108537181A (en) | A kind of gait recognition method based on the study of big spacing depth measure | |
| CN109858407A (en) | A kind of video behavior recognition methods based on much information stream feature and asynchronous fusion | |
| CN110119707A (en) | A kind of human motion recognition method | |
| CN114821640A (en) | Skeleton action identification method based on multi-stream multi-scale expansion space-time diagram convolution network | |
| CN113111857A (en) | Human body posture estimation method based on multi-mode information fusion | |
| CN110163567A (en) | Classroom roll calling system based on multitask concatenated convolutional neural network | |
| CN110046544A (en) | Digital gesture identification method based on convolutional neural networks | |
| CN110059593A (en) | A kind of human facial expression recognition method based on feedback convolutional neural networks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |