CN110781719A - Non-contact and contact cooperative mental state intelligent monitoring system - Google Patents
Non-contact and contact cooperative mental state intelligent monitoring system Download PDFInfo
- Publication number
- CN110781719A CN110781719A CN201910823581.1A CN201910823581A CN110781719A CN 110781719 A CN110781719 A CN 110781719A CN 201910823581 A CN201910823581 A CN 201910823581A CN 110781719 A CN110781719 A CN 110781719A
- Authority
- CN
- China
- Prior art keywords
- target individual
- features
- visible light
- infrared heat
- heat map
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
- A61B5/165—Evaluating the state of mind, e.g. depression, anxiety
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/30—Noise filtering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
- G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
- G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
- G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/56—Extraction of image or video features relating to colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/174—Facial expression recognition
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- General Engineering & Computer Science (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Molecular Biology (AREA)
- Human Computer Interaction (AREA)
- Evolutionary Biology (AREA)
- Psychiatry (AREA)
- Biophysics (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Developmental Disabilities (AREA)
- Psychology (AREA)
- Social Psychology (AREA)
- Pathology (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Surgery (AREA)
- Animal Behavior & Ethology (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- Hospice & Palliative Care (AREA)
- Educational Technology (AREA)
- Child & Adolescent Psychology (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Biodiversity & Conservation Biology (AREA)
Abstract
The invention provides a non-contact and contact cooperative mental state intelligent monitoring system, which is characterized in that visible light video, infrared heat map video and audio data of a monitored target individual are acquired based on a non-contact data acquisition module, and physiological information of the target individual is acquired by a contact data acquisition module; the data transmission module is used for transmitting the acquired data to the psychological state analysis module and the information display module; and a psychological state analysis module is set up to analyze and process the multi-channel data to obtain a psychological state characteristic value of the monitored target individual, and the characteristic value is sent to a long-term psychological database block for storage and sent to an information display module for real-time display. The system can completely and efficiently acquire the characteristic data for analyzing the psychological state of the target individual, so that the psychological state of the individual can be monitored in real time for a long time, and the psychological state monitoring effect can be accurately and efficiently improved.
Description
Technical Field
The invention relates to the field of psychological and data processing, in particular to a non-contact and contact cooperative psychological state intelligent monitoring system.
Background
Mental state is one of the basic forms of mental activities, and refers to the complete features of mental activities in a certain period of time, such as attention, fatigue, tension, relaxation, worry, joy, etc. It has the characteristics of both psychological process and individual psychological characteristics, and has both temporary and stable properties. The method is a medium link for connecting psychological process and individual psychological characteristics, and forms all backgrounds for developing psychological activities.
The psychological state has different expression forms in different psychological activity stages and activity fields, embodies the characteristics of complexity and comprehensiveness, and different standards are used for carrying out multi-directional investigation and research on the psychological state according to different fields and factors. If the influence of the psychological state on the activity effect is taken as a standard, the psychological state can be divided into an optimal psychological state, a general psychological state and a bad psychological state; according to the duration, the psychological states can be divided into relatively stable states with longer duration, such as attitudes, interests and moods, and psychological states with shorter duration, such as passion and stress states; the mental states can be divided into periodic mental states and aperiodic mental states according to whether they have significant periodicity or not. Obviously, the multi-standard division method can be used for carrying out multi-angle research on the psychological state, can also meet the requirements of various industries such as education departments, medical departments and the like, and has practical significance.
At present, an analysis method for individual psychological states generally collects data by using questionnaires, uploads the collected data to a cloud platform, and then obtains the data from the cloud platform for analysis so as to determine the psychological state characteristic information of a target individual. The psychological characteristic analysis method has the defects of low efficiency and poor accuracy in determining the psychological state characteristics of a certain individual due to the defects of the data acquired in the prior art.
Disclosure of Invention
Technical problem to be solved
Aiming at the defects of the prior art, the invention provides a non-contact and contact cooperative mental state intelligent monitoring system, which solves the defects that the prior art cannot efficiently acquire complete mental state analysis data, cannot monitor the mental state characteristics of an individual in real time for a long time and cannot efficiently and accurately determine the mental state characteristics of the individual.
(II) technical scheme
In order to achieve the purpose, the invention is realized by the following technical scheme:
the application provides a psychological state intelligent monitoring system that non-contact and contact are cooperative, includes:
the non-contact signal acquisition module is used for acquiring a visible light image of the target individual, an infrared heat map of the target individual and audio information of the target individual;
the contact signal acquisition module is used for acquiring physiological information of a target individual;
the data transmission module is used for transmitting the visible light image of the target individual, the infrared heat map of the target individual, the audio information of the target individual and the physiological information of the target individual to the psychological state analysis module and the information display module;
the mental state analysis module is used for extracting first image features in the visible light image and generating visible light image feature vectors based on the extracted image features; performing dimension reduction processing on the infrared heat map, extracting second image features of information obtained by the dimension reduction processing, and determining an infrared heat map feature vector based on the extracted second image features; extracting audio features in the audio information, determining emotion fluctuation features matched with the audio features, and generating emotion fluctuation feature vectors based on the emotion fluctuation features; extracting time-frequency characteristics in the physiological information, and generating a physiological time sequence characteristic vector based on the extracted time-frequency characteristics; determining a psychological state characteristic value of a target individual based on a visible light image characteristic vector, an infrared heat map characteristic vector, an emotion fluctuation characteristic vector, a physiological time sequence characteristic vector, a weight of the visible light image characteristic vector, a weight of the infrared heat map characteristic vector, a weight of the emotion fluctuation characteristic vector and a weight of the physiological time sequence characteristic vector, and sending the psychological state characteristic value of the target individual to a long-term psychological database block and an information display module;
the long-term psychological database is used for storing the psychological state characteristic value of the target individual;
the information display module is used for displaying the visible light image, the infrared heat image, the signal oscillogram of the audio information, the signal oscillogram of the physiological information and the psychological state characteristic value of the target individual.
In a possible implementation, the system further includes a storage module;
the data transmission module is also used for transmitting the visible light image of the target individual, the infrared heat map of the target individual, the audio information of the target individual and the physiological information of the target individual to the storage module;
the psychological state analysis module is also used for transmitting the psychological state characteristic value of the target individual to the storage module;
the storage module is used for storing the visible light image of the target individual, the infrared heat map of the target individual, the audio information of the target individual, the physiological information of the target individual and the psychological state characteristic value of the target individual.
In a possible implementation, the system further includes:
the remote service module is used for controlling the non-contact signal acquisition module to acquire a visible light image of the target individual, an infrared heat map of the target individual and audio information of the target individual; controlling a contact signal acquisition module to acquire physiological information of a target individual; the control data transmission module transmits the visible light image of the target individual, the infrared heat map of the target individual, the audio information of the target individual and the physiological information of the target individual to the remote service module; the image processing device is also used for extracting first image features in the visible light image and generating visible light image feature vectors based on the extracted image features; performing dimension reduction processing on the infrared heat map, extracting second image features of information obtained by the dimension reduction processing, and determining an infrared heat map feature vector based on the extracted second image features; extracting audio features in the audio information, determining emotion fluctuation features matched with the audio features, and generating emotion fluctuation feature vectors based on the emotion fluctuation features; extracting time-frequency characteristics in the physiological information, and generating a physiological time sequence characteristic vector based on the extracted time-frequency characteristics; based on the visible light image feature vector, the infrared heat map feature vector, the emotion fluctuation feature vector, the physiological time sequence feature vector, the weight of the visible light image feature vector, the weight of the infrared heat map feature vector, the weight of the emotion fluctuation feature vector and the weight of the physiological time sequence feature vector, the psychological state feature value of the target individual is determined, and the psychological state feature value is sent to the long-term database block and the information display module.
In a possible implementation, the system further includes:
the data preprocessing module is used for performing point operation processing, filtering processing and global optimization processing on the visible light image; preprocessing the infrared thermal image by utilizing a gray level transformation method and a denoising method of a wavelet packet threshold; and performing pre-weighting processing, low-pass filtering processing and framing processing on the audio information.
In one possible embodiment, the mental state analysis module, when extracting the first image feature in the visible light image, includes:
and extracting image features of a plurality of preset face feature points and a plurality of preset face motion areas in the visible light image to obtain the first image features.
In a possible implementation manner, when the mental state analysis module performs dimension reduction processing on the infrared heat map, and extracts a second image feature of information obtained by the dimension reduction processing, the mental state analysis module includes:
and performing dimension reduction processing on the infrared heat map by using a direct example checking method and a principal component analysis method based on the HSV color model, and extracting image features reduced into processed information to obtain the second image features.
In a possible implementation manner, the mental state analysis module, when extracting audio features in the audio information and determining emotional fluctuation features matching the audio features, includes:
extracting audio features in the audio information, and searching emotion fluctuation features matched with the audio features from a voice tone library; the tone library stores a plurality of voice features and emotion fluctuation features corresponding to each voice feature.
In a possible implementation, the mental state analysis module, when extracting time-frequency features in the physiological information, includes:
and performing time-frequency analysis processing on the physiological information based on Fourier transform and depth LSTM to obtain the video characteristics.
In one possible embodiment, the mental state analysis module, when determining the mental state feature value of the target individual based on the visible light image feature vector, the infrared heat map feature vector, the mood wave feature vector and the physiological time sequence feature vector, the weight of the visible light image feature vector, the weight of the infrared heat map feature vector, the weight of the mood wave feature vector and the weight of the physiological time sequence feature vector, includes:
and determining the psychological state characteristic value of the target individual by utilizing a convolutional neural network based on the visible light image characteristic vector, the infrared heat map characteristic vector, the emotion fluctuation characteristic vector, the physiological time sequence characteristic vector, the weight of the visible light image characteristic vector, the weight of the infrared heat map characteristic vector, the weight of the emotion fluctuation characteristic vector and the weight of the physiological time sequence characteristic vector.
In one possible implementation, the first image feature includes a head posture feature of the target individual, a micro-expression feature of the target individual, and an eye movement track feature of the target individual;
the second image characteristics comprise facial temperature characteristics, breathing frequency characteristics and heartbeat frequency characteristics of the target individual;
the audio features comprise sound wave frequency features and sound intensity features of the target individual;
the time-frequency characteristics comprise the skin electricity characteristics of the target individual, the pulse characteristics of the target individual and the blood sample characteristics of the target individual.
(III) advantageous effects
The invention provides a non-contact and contact cooperative mental state intelligent monitoring system. The method has the following beneficial effects:
the system utilizes a non-contact signal acquisition module to acquire a visible light image of a target individual, an infrared heat map of the target individual and audio information of the target individual; acquiring physiological information of a target individual by using a contact signal acquisition module; transmitting the visible light image of the target individual, the infrared heat map of the target individual, the audio information of the target individual and the physiological information of the target individual to a psychological state analysis module and an information display module by using a data transmission module; processing the visible light image of the target individual, the infrared heat map of the target individual, the audio information of the target individual and the physiological information of the target individual by using a psychological state analysis module to obtain a psychological state characteristic value of the target individual, and sending the psychological state characteristic value of the target individual to a long-term database block and an information display module; storing the psychological state characteristic value of the target individual by using a long-term mind database; and displaying the visible light image, the infrared heat image, the signal oscillogram of the audio information, the signal oscillogram of the physiological information and the psychological state characteristic value of the target individual by using the information display module. The system can be used for acquiring the data of the psychological state characteristics of the target individual, such as a visible light image, an infrared chart and the like, in real time, for a long time, completely and efficiently, so that the psychological state characteristics of a certain individual can be monitored in real time and for a long time, and high-accuracy psychological state characteristics can be determined efficiently.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1A is a schematic diagram illustrating a non-contact and contact-based intelligent mental state monitoring system according to an embodiment of the present invention;
fig. 1B schematically shows a schematic structural diagram of a contactless and contact-based cooperative mental state intelligent monitoring system according to another embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The scheme of mental state feature analysis in the prior art cannot efficiently acquire complete mental state analysis data, cannot monitor the mental state features of an individual in real time for a long time, and cannot efficiently and accurately determine the mental state features of the individual. In view of the above-mentioned drawbacks, the present application provides the following embodiments to solve the above-mentioned technical drawbacks in the prior art.
As shown in fig. 1A, a non-contact and contact-based mental state intelligent monitoring system provided for an embodiment of the present application includes:
and the non-contact signal acquisition module 110 is used for acquiring a visible light image of the target individual, an infrared heat map of the target individual and audio information of the target individual.
The target individual here is an individual whose mental state characteristic value is to be analyzed. The non-contact signal acquisition module can acquire a visible light image of a target individual by using a visible light camera, acquire an infrared heat map of the target individual by using an infrared camera and acquire audio information of the target individual by using a microphone. The visible light image may be used to determine information such as a head pose of the target individual, the infrared heat map may be used to determine information such as a face temperature of the target individual, and the audio information may be used to determine information such as a sound intensity of the target individual.
The contact signal acquisition module 120 is configured to acquire physiological information of the target individual.
The contact signal acquisition module can monitor physiological information of a target individual by using a finger-clip type instrument; the physiological information includes a target individual's skin electrical signal, blood oxygen signal, pulse signal, etc.
And the data transmission module 130 is used for transmitting the visible light image of the target individual, the infrared heat map of the target individual, the audio information of the target individual and the physiological information of the target individual to the mental state analysis module and the information display module.
Here, the data transmission module 130 may be a mobile operation platform.
The mental state analysis module 140 is configured to extract a first image feature in the visible light image, and generate a visible light image feature vector based on the extracted image feature; performing dimension reduction processing on the infrared heat map, extracting second image features of information obtained by the dimension reduction processing, and determining an infrared heat map feature vector based on the extracted second image features; extracting audio features in the audio information, determining emotion fluctuation features matched with the audio features, and generating emotion fluctuation feature vectors based on the emotion fluctuation features; extracting time-frequency characteristics in the physiological information, and generating a physiological time sequence characteristic vector based on the extracted time-frequency characteristics; based on the visible light image feature vector, the infrared heat map feature vector, the emotion fluctuation feature vector, the physiological time sequence feature vector, the weight of the visible light image feature vector, the weight of the infrared heat map feature vector, the weight of the emotion fluctuation feature vector and the weight of the physiological time sequence feature vector, the psychological state feature value of the target individual is determined, and the psychological state feature value of the target individual is sent to the long-term database block and the information display module.
Here, the mental state analysis module 140, when extracting the first image feature in the visible light image, includes:
and extracting image features of a plurality of preset face feature points and a plurality of preset face motion areas in the visible light image to obtain the first image features.
Here, when performing dimension reduction processing on the infrared heat map and extracting a second image feature of information obtained by the dimension reduction processing, the psychological state analysis module 140 includes:
and performing dimension reduction processing on the infrared heat map by using a direct example checking method and a principal component analysis method based on the HSV color model, and extracting image features reduced into processed information to obtain the second image features.
Here, the mental state analysis module 140, when extracting audio features from the audio information and determining emotional fluctuation features matching the audio features, includes:
extracting audio features in the audio information, and searching emotion fluctuation features matched with the audio features from a voice tone library; the tone library stores a plurality of voice features and emotion fluctuation features corresponding to each voice feature.
Here, the mental state analysis module 140, when extracting the time-frequency features in the physiological information, includes:
and performing time-frequency analysis processing on the physiological information based on Fourier transform and depth LSTM to obtain the video characteristics.
Here, the mental state analysis module 140, when determining the mental state feature value of the target individual based on the visible light image feature vector, the infrared heat map feature vector, the fluctuation of emotion feature vector, and the physiological time series feature vector, the weight of the visible light image feature vector, the weight of the infrared heat map feature vector, the weight of the fluctuation of emotion feature vector, and the weight of the physiological time series feature vector, includes:
and determining the psychological state characteristic value of the target individual by utilizing a convolutional neural network based on a time sequence based on the visible light image characteristic vector, the infrared heat map characteristic vector, the emotion fluctuation characteristic vector and the physiological time sequence characteristic vector, the weight of the visible light image characteristic vector, the weight of the infrared heat map characteristic vector, the weight of the emotion fluctuation characteristic vector and the weight of the physiological time sequence characteristic vector.
The first image characteristics comprise head posture characteristics of the target individual, micro-expression characteristics of the target individual and eye movement track characteristics of the target individual; the second image characteristics comprise facial temperature characteristics, breathing frequency characteristics and heartbeat frequency characteristics of the target individual; the audio features comprise sound wave frequency features and sound intensity features of the target individual; the time-frequency characteristics comprise the skin electricity characteristics of the target individual, the pulse characteristics of the target individual and the blood sample characteristics of the target individual.
And the long-term psychological database 150 is used for storing the psychological state characteristic values of the target individual.
The information display module 160 is configured to display a visible light image, an infrared heat map, a signal waveform map of audio information, a signal waveform map of physiological information, and a psychological state characteristic value of a target individual.
Here, the information display module may present the above information in the form of a graph using a PC high definition display screen.
In some embodiments, as shown in fig. 1B, the above-mentioned contactless and contact collaborative mental state intelligent monitoring system further includes a storage module 170. The data transmission module 130 is further configured to transmit the visible light image of the target individual, the infrared heat map of the target individual, the audio information of the target individual, and the physiological information of the target individual to the storage module 170; the mental state analysis module 140 is further configured to transmit the mental state feature values of the target individual to the storage module 170. The storage module 170 is configured to store the visible light image of the target individual, the infrared heat map of the target individual, the audio information of the target individual, the physiological information of the target individual, and the mental state characteristic value of the target individual. The storage module 170 may be a large-capacity storage device such as a hard disk.
In some embodiments, the above-mentioned contactless and contact collaborative mental state intelligent monitoring system further includes a remote service module 180, configured to control the contactless signal acquisition module to acquire a visible light image of the target individual, an infrared heat map of the target individual, and audio information of the target individual; controlling a contact signal acquisition module to acquire physiological information of a target individual; the control data transmission module transmits the visible light image of the target individual, the infrared heat map of the target individual, the audio information of the target individual and the physiological information of the target individual to the remote service module; the image processing device is also used for extracting first image features in the visible light image and generating visible light image feature vectors based on the extracted image features; performing dimension reduction processing on the infrared heat map, extracting second image features of information obtained by the dimension reduction processing, and determining an infrared heat map feature vector based on the extracted second image features; extracting audio features in the audio information, determining emotion fluctuation features matched with the audio features, and generating emotion fluctuation feature vectors based on the emotion fluctuation features; extracting time-frequency characteristics in the physiological information, and generating a physiological time sequence characteristic vector based on the extracted time-frequency characteristics; and determining a psychological state characteristic value of the target individual based on the visible light image characteristic vector, the infrared heat map characteristic vector, the emotion fluctuation characteristic vector, the physiological time sequence characteristic vector, the weight of the visible light image characteristic vector, the weight of the infrared heat map characteristic vector, the weight of the emotion fluctuation characteristic vector and the weight of the physiological time sequence characteristic vector, sending the psychological state characteristic value to a long-term database block for storage, and sending the psychological state characteristic value to an information display module for display.
The remote service module 180 may be used to remotely service, including online services, over the internet. Remote service refers to a mode of realizing real-time manual control between different regions (areas) by using communication means. The service mode has the characteristics of instantaneity, flexibility, humanization and the like, and the user can directly communicate with service implementing personnel for service.
In some embodiments, the above-mentioned non-contact and contact-type cooperative mental state intelligent monitoring system further includes a data preprocessing module 190, configured to perform point operation processing, filtering processing, and global optimization processing on the visible light image; preprocessing the infrared thermal image by utilizing a gray level transformation method and a denoising method of a wavelet packet threshold; and performing pre-weighting processing, low-pass filtering processing and framing processing on the audio information.
In the embodiment, the non-contact signal acquisition module and the contact signal acquisition module are used for monitoring the target individual for multiple times for a long time, namely, data acquisition is carried out, and the acquired data are subjected to integrated analysis, storage and display through the data transmission module, the remote service module, the psychological state analysis module, the storage module and the information display module, so that the psychological state characteristics of the target individual can be analyzed for multiple times for a long time in real time.
The system in the above embodiment combines a contact device with a non-contact device, specifically, trains a model by a neural network method, and performs optimization and parameter adjustment according to the acquired data. The mental state analysis module can obtain physiological and psychological signals of the model intelligently monitored target individuals by utilizing training, namely, the mental state of the target individuals is obtained through comprehensive analysis by giving different weights to information collected by the non-contact signal collection module and information collected by the contact signal collection module. And (4) training a model by the weight assignment method based on a convolutional neural network method, and performing optimization parameter adjustment according to the obtained data. In the embodiment, the psychological state characteristics of the target individual are monitored by the cooperation of the non-contact signal acquisition module and the contact signal acquisition module, compared with the conventional method for analyzing only psychological signals (namely, the visible light images, the infrared heat images and the audio information) or only physiological signals (namely, the physiological information), the psychological state characteristics of the target individual are obtained, the system is more persuasive, the psychological change of a monitored person can be monitored for a plurality of times for a long time, the accurate data analysis is facilitated, the operation is convenient, the efficiency is high, the evaluation accuracy can be effectively improved, the operation flow is simple and easy to understand, the operation of the equipment is simple, and the use is convenient.
The non-contact and contact cooperative mental state intelligent monitoring system acquires visible light video, infrared heat map video and audio data of a monitored target individual based on the non-contact data acquisition module, and acquires physiological information of the target individual by the contact data acquisition module; the data transmission module is used for transmitting the acquired data to the psychological state analysis module and the information display module; and a psychological state analysis module is set up to analyze and process the multi-channel data to obtain a psychological state characteristic value of the monitored target individual, and the characteristic value is sent to a long-term psychological database block for storage and sent to an information display module for real-time display. The system can completely and efficiently acquire the characteristic data for analyzing the psychological state of the target individual, so that the psychological state of the individual can be monitored in real time for a long time, and the psychological state monitoring effect can be accurately and efficiently improved.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A non-contact and contact cooperative mental state intelligent monitoring system is characterized by comprising:
the non-contact signal acquisition module is used for acquiring a visible light image of the target individual, an infrared heat map of the target individual and audio information of the target individual;
the contact signal acquisition module is used for acquiring physiological information of a target individual;
the data transmission module is used for transmitting the visible light image of the target individual, the infrared heat map of the target individual, the audio information of the target individual and the physiological information of the target individual to the psychological state analysis module and the information display module;
the mental state analysis module is used for extracting first image features in the visible light image and generating visible light image feature vectors based on the extracted image features; performing dimension reduction processing on the infrared heat map, extracting second image features of information obtained by the dimension reduction processing, and determining an infrared heat map feature vector based on the extracted second image features; extracting audio features in the audio information, determining emotion fluctuation features matched with the audio features, and generating emotion fluctuation feature vectors based on the emotion fluctuation features; extracting time-frequency characteristics in the physiological information, and generating a physiological time sequence characteristic vector based on the extracted time-frequency characteristics; determining a psychological state characteristic value of a target individual based on a visible light image characteristic vector, an infrared heat map characteristic vector, an emotion fluctuation characteristic vector, a physiological time sequence characteristic vector, a weight of the visible light image characteristic vector, a weight of the infrared heat map characteristic vector, a weight of the emotion fluctuation characteristic vector and a weight of the physiological time sequence characteristic vector, and sending the psychological state characteristic value of the target individual to a long-term psychological database block and an information display module;
the long-term psychological database is used for storing the psychological state characteristic value of the target individual;
and the information display module is used for displaying the visible light image, the infrared heat image, the signal oscillogram of the audio information, the signal oscillogram of the physiological information and the psychological state characteristic value of the target individual.
2. The system of claim 1, further comprising a storage module;
the data transmission module is also used for transmitting the visible light image of the target individual, the infrared heat map of the target individual, the audio information of the target individual and the physiological information of the target individual to the storage module;
the psychological state analysis module is also used for transmitting the psychological state characteristic value of the target individual to the storage module;
the storage module is used for storing the visible light image of the target individual, the infrared heat map of the target individual, the audio information of the target individual, the physiological information of the target individual and the psychological state characteristic value of the target individual.
3. The system of claim 1, further comprising:
the remote service module is used for controlling the non-contact signal acquisition module to acquire a visible light image of the target individual, an infrared heat map of the target individual and audio information of the target individual; controlling a contact signal acquisition module to acquire physiological information of a target individual; the control data transmission module transmits the visible light image of the target individual, the infrared heat map of the target individual, the audio information of the target individual and the physiological information of the target individual to the remote service module; the image processing device is also used for extracting first image features in the visible light image and generating visible light image feature vectors based on the extracted image features; performing dimension reduction processing on the infrared heat map, extracting second image features of information obtained by the dimension reduction processing, and determining an infrared heat map feature vector based on the extracted second image features; extracting audio features in the audio information, determining emotion fluctuation features matched with the audio features, and generating emotion fluctuation feature vectors based on the emotion fluctuation features; extracting time-frequency characteristics in the physiological information, and generating a physiological time sequence characteristic vector based on the extracted time-frequency characteristics; based on the visible light image feature vector, the infrared heat map feature vector, the emotion fluctuation feature vector, the physiological time sequence feature vector, the weight of the visible light image feature vector, the weight of the infrared heat map feature vector, the weight of the emotion fluctuation feature vector and the weight of the physiological time sequence feature vector, the psychological state feature value of the target individual is determined, and the psychological state feature value is sent to the long-term database block and the information display module.
4. The system of claim 1, further comprising:
the data preprocessing module is used for performing point operation processing, filtering processing and global optimization processing on the visible light image; preprocessing the infrared thermal image by utilizing a gray level transformation method and a denoising method of a wavelet packet threshold; and performing pre-weighting processing, low-pass filtering processing and framing processing on the audio information.
5. The system of claim 1, wherein the mental state analysis module, when extracting the first image feature in the visible light image, comprises:
and extracting image features of a plurality of preset face feature points and a plurality of preset face motion areas in the visible light image to obtain the first image features.
6. The system according to claim 1, wherein the mental state analysis module, when performing dimension reduction processing on the infrared heat map and extracting a second image feature of information obtained by the dimension reduction processing, comprises:
and performing dimension reduction processing on the infrared heat map by using a direct example checking method and a principal component analysis method based on the HSV color model, and extracting image features reduced into processed information to obtain the second image features.
7. The system according to claim 1, wherein the mental state analysis module, when extracting audio features from the audio information and determining emotional fluctuation features matching the audio features, comprises:
extracting audio features in the audio information, and searching emotion fluctuation features matched with the audio features from a voice tone library; the tone library stores a plurality of voice features and emotion fluctuation features corresponding to each voice feature.
8. The system according to claim 1, wherein the mental state analysis module, when extracting time-frequency features from the physiological information, comprises:
and performing time-frequency analysis processing on the physiological information based on Fourier transform and depth LSTM to obtain the video characteristics.
9. The system of claim 1, wherein the mental state analysis module, when determining the mental state feature value of the target individual based on the visible light image feature vector, the infrared heat map feature vector, the mood swing feature vector and the physiological timing feature vector, the weight of the visible light image feature vector, the weight of the infrared heat map feature vector, the weight of the mood swing feature vector and the weight of the physiological timing feature vector, comprises:
and determining the psychological state characteristic value of the target individual by utilizing a convolutional neural network based on the visible light image characteristic vector, the infrared heat map characteristic vector, the emotion fluctuation characteristic vector, the physiological time sequence characteristic vector, the weight of the visible light image characteristic vector, the weight of the infrared heat map characteristic vector, the weight of the emotion fluctuation characteristic vector and the weight of the physiological time sequence characteristic vector.
10. The system of claim 1, wherein the first image features comprise head pose features of the target individual, micro-expression features of the target individual, eye movement trajectory features of the target individual;
the second image characteristics comprise facial temperature characteristics, breathing frequency characteristics and heartbeat frequency characteristics of the target individual;
the audio features comprise sound wave frequency features and sound intensity features of the target individual;
the time-frequency characteristics comprise the skin electricity characteristics of the target individual, the pulse characteristics of the target individual and the blood sample characteristics of the target individual.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910823581.1A CN110781719A (en) | 2019-09-02 | 2019-09-02 | Non-contact and contact cooperative mental state intelligent monitoring system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910823581.1A CN110781719A (en) | 2019-09-02 | 2019-09-02 | Non-contact and contact cooperative mental state intelligent monitoring system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110781719A true CN110781719A (en) | 2020-02-11 |
Family
ID=69383354
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910823581.1A Pending CN110781719A (en) | 2019-09-02 | 2019-09-02 | Non-contact and contact cooperative mental state intelligent monitoring system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110781719A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112579744A (en) * | 2020-12-28 | 2021-03-30 | 北京智能工场科技有限公司 | Method for controlling risk in online psychological consultation |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106264568A (en) * | 2016-07-28 | 2017-01-04 | 深圳科思创动实业有限公司 | Contactless emotion detection method and device |
| CN107220591A (en) * | 2017-04-28 | 2017-09-29 | 哈尔滨工业大学深圳研究生院 | Multi-modal intelligent mood sensing system |
| CN108877840A (en) * | 2018-06-29 | 2018-11-23 | 重庆柚瓣家科技有限公司 | Emotion identification method and system based on nonlinear characteristic |
| CN108899050A (en) * | 2018-06-14 | 2018-11-27 | 南京云思创智信息科技有限公司 | Speech signal analysis subsystem based on multi-modal Emotion identification system |
| CN109447048A (en) * | 2018-12-25 | 2019-03-08 | 苏州闪驰数控***集成有限公司 | A kind of artificial intelligence early warning system |
| CN109528217A (en) * | 2018-10-16 | 2019-03-29 | 北京入思技术有限公司 | A kind of mood detection and method for early warning based on physiological vibrations analysis |
| CN109924977A (en) * | 2019-03-21 | 2019-06-25 | 西安交通大学 | A kind of surface electromyogram signal classification method based on CNN and LSTM |
| CN109993093A (en) * | 2019-03-25 | 2019-07-09 | 山东大学 | Road anger monitoring method, system, equipment and medium based on face and respiratory characteristic |
| CN110001652A (en) * | 2019-03-26 | 2019-07-12 | 深圳市科思创动科技有限公司 | Monitoring method, device and the terminal device of driver status |
-
2019
- 2019-09-02 CN CN201910823581.1A patent/CN110781719A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106264568A (en) * | 2016-07-28 | 2017-01-04 | 深圳科思创动实业有限公司 | Contactless emotion detection method and device |
| CN107220591A (en) * | 2017-04-28 | 2017-09-29 | 哈尔滨工业大学深圳研究生院 | Multi-modal intelligent mood sensing system |
| CN108899050A (en) * | 2018-06-14 | 2018-11-27 | 南京云思创智信息科技有限公司 | Speech signal analysis subsystem based on multi-modal Emotion identification system |
| CN108877840A (en) * | 2018-06-29 | 2018-11-23 | 重庆柚瓣家科技有限公司 | Emotion identification method and system based on nonlinear characteristic |
| CN109528217A (en) * | 2018-10-16 | 2019-03-29 | 北京入思技术有限公司 | A kind of mood detection and method for early warning based on physiological vibrations analysis |
| CN109447048A (en) * | 2018-12-25 | 2019-03-08 | 苏州闪驰数控***集成有限公司 | A kind of artificial intelligence early warning system |
| CN109924977A (en) * | 2019-03-21 | 2019-06-25 | 西安交通大学 | A kind of surface electromyogram signal classification method based on CNN and LSTM |
| CN109993093A (en) * | 2019-03-25 | 2019-07-09 | 山东大学 | Road anger monitoring method, system, equipment and medium based on face and respiratory characteristic |
| CN110001652A (en) * | 2019-03-26 | 2019-07-12 | 深圳市科思创动科技有限公司 | Monitoring method, device and the terminal device of driver status |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112579744A (en) * | 2020-12-28 | 2021-03-30 | 北京智能工场科技有限公司 | Method for controlling risk in online psychological consultation |
| CN112579744B (en) * | 2020-12-28 | 2024-03-26 | 北京智能工场科技有限公司 | Risk control method in online psychological consultation |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110916631B (en) | Student classroom learning state evaluation system based on wearable physiological signal monitoring | |
| CN111046823A (en) | Student classroom participation degree analysis system based on classroom video | |
| CN111544015B (en) | Cognitive power-based control work efficiency analysis method, device and system | |
| CN110598608B (en) | Non-contact and contact cooperative psychological and physiological state intelligent monitoring system | |
| CN110349667A (en) | The autism assessment system analyzed in conjunction with questionnaire and multi-modal normal form behavioral data | |
| CN111887867A (en) | Method and system for analyzing character formation based on expression recognition and psychological test | |
| CN111598451B (en) | Control work efficiency analysis method, device and system based on task execution capacity | |
| US20150305662A1 (en) | Remote assessment of emotional status | |
| CN110706786A (en) | Non-contact intelligent analysis and evaluation system for psychological parameters | |
| CN114648354A (en) | Advertisement evaluation method and system based on eye movement tracking and emotional state | |
| CN106667436A (en) | Sleep diagnosis method and system | |
| Bian et al. | An accurate lstm based video heart rate estimation method | |
| CN108983966B (en) | Criminal reconstruction assessment system and method based on virtual reality and eye movement technology | |
| US20220383896A1 (en) | System and method for collecting behavioural data to assist interpersonal interaction | |
| CN108109673A (en) | A kind of human body data measurin system and method | |
| CN110717461A (en) | Fatigue state identification method, device and equipment | |
| CN111276159A (en) | Infant pronunciation analysis method and server | |
| CN115862868A (en) | Psychological assessment system, psychological assessment platform, electronic device and storage medium | |
| CN115736922A (en) | Emotion normalization monitoring system and method based on trusted environment | |
| CN113764099A (en) | Psychological state analysis method, device, equipment and medium based on artificial intelligence | |
| CN110781719A (en) | Non-contact and contact cooperative mental state intelligent monitoring system | |
| CN110693508A (en) | Multi-channel cooperative psychophysiological active sensing method and service robot | |
| CN110598607B (en) | Non-contact and contact cooperative real-time emotion intelligent monitoring system | |
| CN105147249B (en) | The wearable or implantable devices evaluation system of one kind and method | |
| Majumder et al. | A smart cyber-human system to support mental well-being through social engagement |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200211 |