WO2020029406A1 - Procédé et dispositif de d'identification d'émotion de visage humain, dispositif informatique et support de stockage - Google Patents

Procédé et dispositif de d'identification d'émotion de visage humain, dispositif informatique et support de stockage Download PDF

Info

Publication number
WO2020029406A1
WO2020029406A1 PCT/CN2018/108251 CN2018108251W WO2020029406A1 WO 2020029406 A1 WO2020029406 A1 WO 2020029406A1 CN 2018108251 W CN2018108251 W CN 2018108251W WO 2020029406 A1 WO2020029406 A1 WO 2020029406A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
facial
emotion
training sample
feature vector
Prior art date
Application number
PCT/CN2018/108251
Other languages
English (en)
Chinese (zh)
Inventor
吴壮伟
Original Assignee
平安科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 平安科技（深圳）有限公司 filed Critical 平安科技（深圳）有限公司
Publication of WO2020029406A1 publication Critical patent/WO2020029406A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/174—Facial expression recognition

Definitions

  • the present application relates to the field of computer technology, and in particular, to a method, a device, a computer device, and a storage medium for facial emotion recognition.
  • facial expressions are an important carrier of human communication and an important way of non-verbal communication. It can express human emotions well.
  • human emotions affect human behavior to a certain extent. For example, when drivers are in negative emotions such as anger, sadness, and anxiety, it is easy to ignore the surrounding road conditions and reduce the speed of response to emergency things, resulting in a high incidence of traffic accidents. Based on this, the behavior of drivers and other personnel can be guided by identifying facial emotions. For example, when the driver's facial emotions are identified, if the driver is identified as having a negative emotion, the driver can be prompted to adjust his emotional state to avoid a traffic accident. Therefore, how to accurately recognize facial emotions has become an urgent technical problem.
  • the present application provides a facial emotion recognition method, device, computer equipment, and storage medium to accurately recognize facial emotions.
  • the present application provides a facial emotion recognition method, which includes: acquiring video images collected in real time; performing wavelet transformation on all frame images in the video image to obtain corresponding energy feature vectors; and obtaining standard energy Feature vector, and calculate a Euclidean distance value between each of the energy eigenvectors and the standard energy eigenvector according to an image difference calculation method; determine whether there is an Euclidean distance value that exceeds a preset threshold among the plurality of Euclidean distance values If there is a European-style distance value exceeding the preset threshold in the plurality of European-style distance values, an image corresponding to an energy feature vector exceeding the European-style distance value of the preset threshold is taken as a key frame image, wherein the key The number of frame images is at least one; obtaining a pre-stored emotion recognition model, and recognizing a face emotion in each of the key frame images based on the emotion recognition model; and according to all the faces in the key frame images Emotionally obtain a facial emotion corresponding to the video
  • the present application provides a facial emotion recognition device, which includes: an acquiring unit for acquiring a video image collected in real time; and a transform unit for performing wavelet transformation on all frame images in the video image to Obtain a corresponding energy feature vector; a distance calculation unit configured to obtain a standard energy feature vector, and calculate an Euclidean distance value between each of the energy feature vector and the standard energy feature vector according to an image difference calculation method; a distance judgment unit For determining whether there is an Euclidean distance value that exceeds a preset threshold among the plurality of Euclidean distance values; and a key frame acquisition unit, which is used for if there is an Euclidean distance value that exceeds the preset threshold among the plurality of Euclidean distance values Taking an image corresponding to an energy feature vector of a European distance value exceeding the preset threshold as a key frame image, wherein the number of the key frame images is at least one; an emotion recognition unit is configured to obtain a previously stored emotion recognition A model, and identifying facial emotions in
  • the present application further provides a computer device including a memory, a processor, and a computer program stored on the memory and executable on the processor.
  • the processor is implemented when the computer program is executed.
  • the facial emotion recognition method provided by the first aspect.
  • the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program, when executed by a processor, causes the processor to execute the first aspect.
  • the described facial emotion recognition method when executed by a processor, causes the processor to execute the first aspect.
  • the application provides a method, a device, a computer device, and a storage medium for facial emotion recognition. This method can accurately recognize facial emotions.
  • FIG. 1 is a schematic flowchart of a facial emotion recognition method according to an embodiment of the present application
  • FIGS. 2 to 6 are another schematic flowchart of a facial emotion recognition method provided by an embodiment of the present application.
  • FIG. 7 to 8 are specific schematic flowcharts of a facial emotion recognition method provided by an embodiment of the present application.
  • FIG. 9 is another schematic flowchart of a facial emotion recognition method according to an embodiment of the present application.
  • FIG. 10 is a schematic block diagram of a facial emotion recognition device according to an embodiment of the present application.
  • 11 to 15 are another schematic block diagrams of a facial emotion recognition device according to an embodiment of the present application.
  • FIG. 16 is a schematic block diagram of a computer device according to an embodiment of the present application.
  • FIG. 1 is a schematic flowchart of a facial emotion recognition method according to an embodiment of the present application.
  • the facial emotion recognition method can be applied to a facial emotion recognition system, and the facial emotion recognition system can be installed in a device with a camera function such as a mobile phone or a car.
  • the facial emotion recognition system can exist in the device as an independent system, or it can be embedded in other systems of the device.
  • a facial emotion recognition system can be embedded in a car driving system to identify the driver's emotions.
  • the facial emotion recognition system may be embedded in an application program of a mobile phone to assist the application program to realize a facial emotion recognition function.
  • the facial emotion recognition method includes steps S101 to S107.
  • the device where the facial emotion recognition system is located invokes a camera to perform real-time image acquisition of the user.
  • the device acquires video images collected within a certain period of time through a camera. For example, capture video images within 10 seconds of real-time capture. It can be understood that the video image will include multiple frames of images.
  • the facial emotion recognition method needs to use information such as a neutral expression image, a standard energy feature vector, and an emotion recognition model when performing facial emotion recognition
  • the user uses the facial emotion recognition system for facial emotion recognition.
  • the facial emotion recognition system also needs to perform the following operations:
  • FIG. 2 is another schematic flowchart of a facial emotion recognition method according to an embodiment of the present application. Prior to step S101, steps S101a, S101b, and S101c are also included.
  • S101b Perform wavelet transform on the neutral expression image to obtain a corresponding standard energy feature vector.
  • a neutral expression image and a standard energy feature vector need to be prepared in advance.
  • the neutral expression may be a facial expression of the user in a relatively stable mood.
  • the facial expressions commonly used by users when taking ID photos can be understood as neutral expressions.
  • the device may issue a voice prompt or a text prompt to prompt the user to make a neutral expression.
  • the image of the user's neutral expression is captured by the camera to obtain a neutral expression image.
  • neutral expression images can also be obtained in other ways.
  • a neutral expression image such as a photo of the user input is acquired.
  • the user transfers the image of the neutral expression taken in the past into the device where the facial emotion recognition system is located as a neutral expression image.
  • the identity information input by the user is obtained, and then the identity photo corresponding to the identity information is obtained from the background server as a neutral facial expression image.
  • the background server may be The background server of the vehicle system, the background server of mobile phone applications, the background server of the facial emotion recognition system, etc.
  • the background server can store the ID photos corresponding to the user's identity information, or call the third party after obtaining the identity information
  • the neutral expression image is subjected to wavelet transform using Gabor wavelet transform to obtain a corresponding standard energy feature vector, and the neutral expression image and the corresponding standard energy feature vector are stored for the convenience of the user.
  • the neutral expression image and the corresponding standard energy feature vector may be called for facial emotion recognition.
  • FIG. 3 is another schematic flowchart of a facial emotion recognition method according to an embodiment of the present application. Prior to step S101, steps S101d and S101e are also included.
  • S101d Obtain an emotional training sample image set, where the emotional training sample image set includes a plurality of emotional training sample images and an emotional label of a human face in the emotional training sample image.
  • S101e input the sentiment training sample image and the corresponding sentiment label into a convolutional neural network model for machine learning to obtain a sentiment recognition model, and store the sentiment recognition model.
  • an emotion recognition model needs to be prepared in advance.
  • the facial emotion recognition system needs to acquire a set of emotional training sample images.
  • the emotional training sample image set includes a large number of emotional training sample images and an emotional label of a human face corresponding to each emotional training sample image. It should be noted that the emotional labels of the faces in each of the emotional training sample images can be labeled manually, or can be labeled by other methods, which are not specifically limited here.
  • the emotional training sample image and the corresponding emotional labels of the human face are input to a convolutional neural network (English full name: Convolutional Neural Networks, CNN) model for machine learning to obtain emotional recognition Model, and then store the emotion recognition model in the device where the facial emotion recognition system is located, so as to facilitate the subsequent use of the facial emotion recognition system, the emotion recognition model can be called for emotion recognition.
  • a convolutional neural network English full name: Convolutional Neural Networks, CNN
  • the camera of the device where the facial emotion recognition system is located cannot perform a real-time image acquisition of the user, for example, the angle of the camera No, the user's face information is not collected in the video images collected in real time, or only half of the user's face information is collected.
  • Such video images will inevitably reduce the accuracy of facial emotion recognition during subsequent facial emotion recognition. . Therefore, in order to ensure that in the subsequent facial emotion recognition, a better video image of the face can be captured, and to improve the accuracy of subsequent facial emotion recognition, the camera needs to be calibrated before acquiring the real-time captured video image. work.
  • FIG. 4 is another schematic flowchart of a facial emotion recognition method according to an embodiment of the present application.
  • steps S101f, S101g, S101h and S101j are also included.
  • S101g Extract a preset number of frames as a calibration image from a plurality of frames of the calibration video image according to a preset rule.
  • S101h Based on a pre-stored face detection and recognition model, determine whether face information exists in the calibration image in each frame.
  • step S101j If there is no face information in the calibration image of at least one frame, issue a prompt message so that the user adjusts the angle of the camera according to the prompt information, and after adjusting the angle of the camera, return to step S101f, Until the face information exists in the calibration image of each frame.
  • the facial emotion recognition system needs to acquire a segment of a calibration video image collected in real time.
  • the calibration video image includes multiple frames of images. Then, according to a preset extraction rule, an image with a preset number of frames is extracted from a plurality of frames of the calibration video image as a calibration image.
  • the preset extraction rule may extract one image as a calibration image every 1 second.
  • the preset frame number can be set to 100, and the preset frame number can be set according to actual requirements.
  • the preset extraction rule may not be limited to the above-mentioned rules, and may be set according to actual requirements, and is not limited here.
  • step S101 can be performed, that is, real-time acquisition is performed. Steps to capture video images.
  • a prompt message can be sent by voice or display mode, so that the user can readjust the camera angle according to the prompt
  • step S101f that is, return to the step of obtaining a calibration video image acquired in real time, until the face information is present in each frame of the calibration image, thereby completing the camera Angle calibration.
  • FIG. 5 is another schematic flowchart of a facial emotion recognition method according to an embodiment of the present application. Prior to step S101, steps S101k, S101m, S101n, and S101p are also included.
  • S101k Obtain a training sample image set, where the training sample image set includes multiple training sample images and a face label used to characterize whether or not face information exists in the training sample image.
  • S101m Obtain a face Hal feature vector of the training sample image.
  • S101n input the face hal feature vector and face label corresponding to the training sample image into an Adaboost lifting model based on a decision tree model for training to obtain a face detection and recognition model.
  • a face detection recognition model needs to be prepared in advance, so as to be used when performing camera angle calibration.
  • a training sample image set is first obtained, where the training sample image set includes multiple training sample images, and a face label corresponding to each training sample image.
  • the face label is used to characterize whether there is face information in a corresponding face sample image.
  • the Hal feature extraction of the face is performed on each training sample image to obtain the Hal feature vector of the face corresponding to each training sample image.
  • the face hal feature vector corresponding to each training sample image and the corresponding face label are input to an Adaboost lifting model based on a decision tree model for training, and a face detection and recognition model can be obtained.
  • the face detection and recognition model is stored in the device where the facial emotion recognition system is located.
  • wavelet transform is required for all frame images in the video image to obtain the energy feature vector corresponding to each frame image.
  • the wavelet transform may be, for example, a Gabor wavelet transform.
  • the wavelet transform may also adopt other methods, which is not limited herein.
  • the standard energy feature vector is an energy feature vector obtained by performing wavelet transform on a neutral expression image of a user collected in advance.
  • the standard energy feature vector is stored in advance in a device where the facial emotion recognition system is located. Because the standard energy feature vector is stored in the device in advance, obtaining the standard energy feature vector is specifically to obtain the previously stored standard energy feature vector.
  • the Euclidean distance value between each energy feature vector and the standard energy feature vector in step S102 will be calculated according to the image difference calculation method.
  • the standard energy feature vector since the standard energy feature vector is stored in advance in the device where the facial emotion recognition system is located, the standard energy feature vector can be directly called in step S103, thereby reducing the Occupation of CPU resources of the device on which the facial emotion recognition system is located, reducing calculation time, etc.
  • the device on which the facial emotion recognition system is located may only store the neutral expression image in advance. In this way, when obtaining the standard energy feature vector in step S103, the previously stored neutral expression image is obtained first, and then Wavelet transform is performed on the neutral expression image to obtain a standard energy feature vector, and there is no limitation on the time for calculating the standard energy feature vector.
  • step S103 After calculating the Euclidean distance value between each energy feature vector and the standard energy feature vector in step S103, a plurality of Euclidean distance values are obtained, and then it is determined whether there is an Euclidean distance value that exceeds a preset threshold among the plurality of Euclidean distance values. If there is an Euclidean distance value exceeding a preset threshold, it indicates that the difference between the facial expression and the neutral expression in the video image is large, and step S105 is executed at this time.
  • FIG. 6 is another schematic flowchart of a facial emotion recognition method according to an embodiment of the present application.
  • step S104 determines that there is no Euclidean distance value exceeding a preset threshold.
  • step S108 will be executed, that is, the standard energy feature vector corresponding to The neutral expression image is used as the key frame image.
  • the subsequent steps S106 and S107 are performed.
  • the facial emotion corresponding to the video image may also be set as a neutral emotion to complete the recognition of the facial emotion.
  • an image corresponding to an energy feature vector exceeding the Euclidean distance value exceeding the preset threshold is used as a key frame image, where the The number of key frame images is at least one.
  • the number of Euclidean distance values exceeding a preset threshold may be one, or may be two or more. At this time, the number of key frame images is at least one.
  • the emotion recognition model is a model for recognizing facial emotions obtained by performing machine learning training in advance
  • the emotion recognition module may be, for example, a convolutional neural network model.
  • the device where the facial emotion recognition system is located first obtains the emotion recognition model, and then inputs the key frame image as an input value into the emotion recognition model.
  • the emotion recognition model performs emotion recognition on the key frame images to output each key frame image. Face emotions.
  • FIG. 7 is a specific schematic flowchart of a facial emotion recognition method according to an embodiment of the present application.
  • This step S106 includes steps S1061 to S1063.
  • Each of the key frame images is sequentially input as an input value into the emotion recognition model.
  • each key frame image is sequentially input as an input value into an emotion recognition model, and then the emotion recognition model outputs probability values of each key frame image on various preset emotions.
  • the multiple preset emotions include 7 preset emotions such as fear, anger, romance, disgust, joy, surprise, and neutrality.
  • the emotion recognition model will recognize the probability of the face emotion in each key frame image on these 7 preset expressions.
  • the emotion recognition model recognizes the face emotion in a key frame image in the above 7 preset expressions.
  • the emotional probabilities are 10%, 70%, 15%, 5%, 0%, 0%, and 0%.
  • the emotion corresponding to the larger probability value among the multiple probability values corresponding to each key frame image is used as the facial emotion in the key frame image.
  • 70% of the anger emotions with the largest probability value are used as the facial emotions corresponding to a certain key frame image.
  • FIG. 8 is a specific schematic flowchart of a facial emotion recognition method according to an embodiment of the present application.
  • This step S107 includes steps S1071 to S1072.
  • the facial emotion with a higher probability of occurrence is used as the facial emotion corresponding to the video image to complete the recognition of the facial emotion.
  • the number of key frame images is 10, and the facial emotion of 8 key frame images is identified by emotion recognition model as anger, the facial emotion of 1 key frame image is aversion, and the facial emotion of one key frame image is For fear, by performing probability statistics on the facial emotions of the 10 key frame images, it can be concluded that the probability of the appearance of angry facial emotions is 80%, the probability of the appearance of disgusted facial emotions is 10%, and the fear of facial emotions The probability of occurrence is 10%. In this way, the anger emotion with a high probability of occurrence can be used as the facial emotion corresponding to the entire video image, thereby completing the recognition of the facial emotion within the time period corresponding to the video image.
  • FIG. 9 is another schematic flowchart of a facial emotion recognition method according to an embodiment of the present application. After step S107, steps S109 to S112 are also included.
  • S111 Determine whether a probability of a facial emotion belonging to a preset emotion category exceeds a preset probability value.
  • the preset emotion category is the negative emotion category.
  • the facial emotions included in the negative emotion category are four types of fear, anger, sadness, and disgust.
  • the number of video images in the emotion list within two minutes is 100, then there will be 100 facial emotions, and then the probability of facial emotions belonging to the negative emotion category among these 100 facial emotions will be counted, such as The probability is 99%.
  • the probability of facial emotions belonging to the category of negative emotions exceeds the preset probability value of 80%, it means that the user has been in negative emotions within these 2 minutes.
  • the preset reminder method and pre-emption will be obtained.
  • Set prompt information and prompt the user with the preset prompt information according to the preset prompt mode.
  • the preset prompt mode may be, for example, a voice prompt mode, a text display mode, a voice prompt and vibration combination mode, and the like.
  • the preset prompt information may be, for example, "your current mood is low, please pay attention to driving safely” and the like.
  • the facial emotion recognition method in this embodiment can accurately recognize facial emotions.
  • An embodiment of the present application further provides a facial emotion recognition device, which is configured to execute any one of the foregoing facial emotion recognition methods.
  • a facial emotion recognition device which is configured to execute any one of the foregoing facial emotion recognition methods.
  • FIG. 10 is a schematic block diagram of a facial emotion recognition device according to an embodiment of the present application.
  • the facial emotion recognition device 300 may be installed in a device such as a car or a mobile phone.
  • the facial emotion recognition device 300 includes an acquisition unit 301, a transformation unit 302, a distance calculation unit 303, a distance judgment unit 304, a key frame acquisition unit 305, an emotion recognition unit 306 and an emotion acquisition unit 307.
  • the obtaining unit 301 is configured to obtain a video image collected in real time.
  • FIG. 11 is another schematic block diagram of a facial emotion recognition device according to an embodiment of the present application.
  • the facial emotion recognition apparatus 300 further includes a storage unit 308.
  • the obtaining unit 301 is further configured to obtain a neutral expression image.
  • the transform unit 302 is further configured to perform wavelet transform on the neutral expression image to obtain a corresponding standard energy feature vector.
  • the storage unit 308 is configured to store the neutral expression image and the standard energy feature vector.
  • FIG. 12 is another schematic block diagram of a facial emotion recognition device according to an embodiment of the present application.
  • the facial emotion recognition apparatus 300 further includes an emotion model training unit 309.
  • the obtaining unit 301 is further configured to obtain an emotional training sample image set, where the emotional training sample image set includes a plurality of emotional training sample images and an emotional label of a human face in the emotional training sample image.
  • An emotional model training unit 309 is configured to input the emotional training sample image and a corresponding emotional label into a convolutional neural network model and perform machine learning to obtain an emotional recognition model, and store the emotional recognition model.
  • FIG. 13 is another schematic block diagram of a facial emotion recognition device according to an embodiment of the present application.
  • the facial emotion recognition device 300 further includes an extraction unit 310, a face determination unit 311, and a prompting unit 312.
  • the obtaining unit 301 is further configured to obtain a calibration video image collected in real time.
  • the extraction unit 310 is configured to extract an image of a preset number of frames from a plurality of frames of the calibration video image according to a preset rule as a calibration image.
  • the face judging unit 311 is configured to determine whether face information exists in the calibration image in each frame based on a face detection and recognition model stored in advance.
  • the obtaining unit 301 is further configured to obtain real-time collected video images if face information exists in the calibration image in each frame.
  • a prompting unit 312 is configured to send a prompting message if the face information does not exist in at least one frame of the calibration image, so that the user can adjust the angle of the camera according to the prompting information, and obtain the unit after adjusting the angle of the camera.
  • Step 301 returns to performing the step of acquiring a calibration video image collected in real time, until face information exists in the calibration image in each frame.
  • FIG. 14 is another schematic block diagram of a facial emotion recognition device according to an embodiment of the present application.
  • the facial emotion recognition apparatus 300 further includes a vector acquisition unit 313 and a face model training unit 314.
  • the obtaining unit 301 is further configured to obtain a training sample image set, where the training sample image set includes a plurality of training sample images and a face label used to characterize whether there is face information in the training sample image.
  • a vector obtaining unit 313 is configured to obtain a face Hal feature vector of the training sample image.
  • a face model training unit 314, configured to input a face Hal feature vector and a face label corresponding to the training sample image into an Adaboost lifting model based on a decision tree model for training to obtain a face detection and recognition model, and The face detection recognition model is stored.
  • a transformation unit 302 is configured to perform wavelet transformation on all frame images in the video image to obtain corresponding energy feature vectors.
  • the distance calculation unit 303 is configured to obtain a standard energy feature vector, and calculate a Euclidean distance value between each of the energy feature vectors and the standard energy feature vector according to an image difference calculation method.
  • the distance judging unit 304 is configured to judge whether there is an Euclidean distance value among the plurality of Euclidean distance values that exceeds a preset threshold.
  • a key frame obtaining unit 305 is configured to use, as a key frame, an image corresponding to an energy feature vector of the Euclidean distance value exceeding the preset threshold value if there are Euclidean distance values exceeding the preset threshold value among the plurality of Euclidean distance values.
  • the key frame obtaining unit 305 is further configured to, if there is no Euclidean distance value exceeding the preset threshold value among the plurality of Euclidean distance values, convert the neutral expression image corresponding to the standard energy feature vector As the key frame image.
  • the emotion recognition unit 306 is configured to obtain a pre-stored emotion recognition model, and recognize a facial emotion in each of the key frame images based on the emotion recognition model.
  • the emotion recognition unit 306 is specifically configured to: sequentially input each of the key frame images as an input value into the emotion recognition model; and obtain each of the information output by the emotion recognition model.
  • the emotion acquiring unit 307 is configured to acquire the facial emotion corresponding to the video image according to the facial emotions in all the key frame images to complete the recognition of the facial emotion.
  • the emotion acquiring unit 307 is specifically configured to: perform probability statistics on the facial emotions in all the key frame images; and use the facial emotions with a higher probability of occurrence as the video image corresponding Facial emotions to complete recognition of facial emotions.
  • FIG. 15 is another schematic block diagram of a facial emotion recognition device according to an embodiment of the present application.
  • the facial emotion recognition device 300 further includes a recording unit 315, a statistics unit 316, a probability judgment unit 317, and an information prompting unit 318.
  • the recording unit 315 is configured to record a time period corresponding to the video image and a facial emotion corresponding to the video image into an emotion list.
  • the statistics unit 316 is configured to count, according to the emotion list, a probability of a facial emotion belonging to a preset emotional category among facial emotions corresponding to all the video images in a preset time period.
  • the probability judging unit 317 is configured to judge whether a probability of a facial emotion belonging to a preset emotion category exceeds a preset probability value.
  • An information prompting unit 318 is configured to obtain a preset prompting method and preset prompting information if the probability of a face emotion belonging to the preset emotion category exceeds the preset probability value, and provide the user with a preset prompting method according to the preset prompting method. Prompt the preset prompt information.
  • the facial emotion recognition device 300 in this embodiment can accurately recognize facial emotions.
  • the above-mentioned facial emotion recognition device can be implemented in the form of a computer program, which can be run on a computer device as shown in FIG. 16.
  • FIG. 16 is a schematic block diagram of a computer device according to an embodiment of the present application.
  • the computer device 500 may be a terminal such as a mobile phone, or may be a device used in a car.
  • the computer device 500 includes a processor 502, a memory, and a network interface 505 connected through a system bus 501.
  • the memory may include a non-volatile storage medium 503 and an internal memory 504.
  • the non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032.
  • the computer program 5032 includes program instructions. When the program instructions are executed, the processor 502 can execute a method for facial emotion recognition.
  • the processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500.
  • the internal memory 504 provides an environment for running a computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a method for facial emotion recognition.
  • the network interface 505 is used for network communication, such as sending assigned tasks.
  • 16 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer equipment 500 to which the solution of the present application is applied.
  • the specific computer equipment 500 may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.
  • the processor 502 is configured to run a computer program 5032 stored in a memory to achieve the following functions: acquiring video images collected in real time; performing wavelet transformation on all frame images in the video image to obtain corresponding energy characteristics Vector; obtaining a standard energy feature vector, and calculating an Euclidean distance value between each of the energy feature vectors and the standard energy feature vector according to an image difference calculation method; determining whether a plurality of the Euclidean distance values exceed a preset value Threshold Euclidean distance value; if there is an Euclidean distance value exceeding the preset threshold among the plurality of Euclidean distance values, an image corresponding to an energy feature vector exceeding the Euclidean distance value exceeding the preset threshold is taken as a key frame image, Wherein, the number of the key frame images is at least one; acquiring a pre-stored emotion recognition model, and recognizing a facial emotion in each of the key frame images based on the emotion recognition model; and according to all the key frames The facial emotion in the image acquires the facial emotion
  • the processor 502 before executing the real-time acquisition of the video image, the processor 502 also implements the following functions: acquiring a neutral expression image; performing wavelet transform on the neutral expression image to obtain a corresponding standard energy feature vector; and The neutral expression image and the standard energy feature vector are stored.
  • the processor 502 also implements the following function before acquiring video images acquired in real time: acquiring an emotional training sample image set, wherein the emotional training sample image set includes a plurality of emotional training sample images and the Emotion labels of human faces in the emotion training sample image; and inputting the emotion training sample images and corresponding emotion labels into a convolutional neural network model for machine learning to obtain an emotion recognition model, and storing the emotion recognition model.
  • the processor 502 also implements the following function before acquiring video images acquired in real time: acquiring a training sample image set, wherein the training sample image set includes multiple training sample images and is used to characterize the Whether there is a face label of the face information in the training sample image; obtaining the face Hal feature vector of the training sample image; and inputting the face Hal feature vector corresponding to the training sample image and the face label into a decision-based Training in the Adaboost lifting model of the tree model to obtain a face detection and recognition model; and storing the face detection and recognition model.
  • the processor 502 before executing the real-time video image acquisition, the processor 502 also implements the following functions: acquiring a real-time calibration video image; and extracting a pre-defined video from a plurality of frames of the calibration video image according to a preset rule.
  • the processor 502 when the processor 502 recognizes the facial emotion in each of the key frame images based on the emotion recognition model, the processor 502 specifically implements the following function: inputting each of the key frame images as an input value in sequence To the emotion recognition model; obtaining probability values of each of the key frame images output by the emotion recognition model on a plurality of preset emotions; and among the plurality of probability values corresponding to each of the key frame images The emotion corresponding to the larger probability value is used as the facial emotion in the key frame image.
  • the processor 502 when the processor 502 executes acquiring facial emotions corresponding to the video image according to the facial emotions in all the key frame images to complete the recognition of the facial emotions, it specifically implements the following functions: Probability statistics are performed on the facial emotions in the key frame image; and facial emotions with a higher probability of occurrence are used as facial emotions corresponding to the video image to complete the recognition of facial emotions.
  • the processor 502 may be a central processing unit, and the processor 502 may also be other general-purpose processors, digital signal processors, application specific integrated circuits, ready-made programmable gate arrays, or other programmable logic. Devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • a general-purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
  • a person of ordinary skill in the art can understand that all or part of the processes in the embodiment of the method for recognizing facial emotions described above can be performed by a computer program instructing related hardware.
  • the computer program may be stored in a computer-readable storage medium.
  • the computer program is executed by at least one processor in the computer system to implement the process steps of the embodiment including the facial emotion recognition method as described above.
  • the storage medium may be various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a magnetic disk, or an optical disk.
  • program codes such as a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a magnetic disk, or an optical disk.
  • each unit is only a logical function division, and there may be another division manner in actual implementation.
  • the steps in the method of the embodiment of the present application can be adjusted, combined, and deleted according to actual needs.
  • the units in the apparatus of the embodiment of the present application may be combined, divided, and deleted according to actual needs.
  • Each functional unit in each embodiment of the present application may be integrated into one processing unit, or each of the units may exist separately physically, or two or more units may be integrated into one unit.
  • the above integrated unit may be implemented in the form of hardware or in the form of software functional unit.
  • the integrated unit When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium.
  • the technical solution of this application is essentially a part that contributes to the existing technology, or all or part of the technical solution may be embodied in the form of a software product, which is stored in a storage medium Included are instructions for causing a computer device (which may be a personal computer, a terminal, or a network device, etc.) to perform all or part of the steps of the method described in the embodiments of the present application.

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

L'invention concerne un procédé et un dispositif d'identification d'émotion de visage humain, un dispositif informatique et un support de stockage. Le procédé comprend l'obtention d'un vecteur de caractéristique d'énergie de chaque trame d'une image vidéo ; le calcul d'une valeur de distance euclidienne entre chaque vecteur de caractéristique d'énergie et un vecteur de caractéristique d'énergie standard ; le filtrage d'une image de trame clé selon la valeur de distance euclidienne ; l'identification d'une émotion de visage humain dans chaque image de trame clé ; et l'obtention, en fonction de l'émotion de visage humain dans toutes les images de trame clé, d'une émotion de visage humain correspondant à l'image vidéo pour achever l'identification de l'émotion de visage humain.
PCT/CN2018/108251 2018-08-07 2018-09-28 Procédé et dispositif de d'identification d'émotion de visage humain, dispositif informatique et support de stockage WO2020029406A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810892915.6A CN109190487A (zh) 2018-08-07 2018-08-07 人脸情绪识别方法、装置、计算机设备及存储介质
CN201810892915.6 2018-08-07

Publications (1)

Publication Number Publication Date
WO2020029406A1 true WO2020029406A1 (fr) 2020-02-13

Family

ID=64921037

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/108251 WO2020029406A1 (fr) 2018-08-07 2018-09-28 Procédé et dispositif de d'identification d'émotion de visage humain, dispositif informatique et support de stockage

Country Status (2)

Country Link
CN (1) CN109190487A (fr)
WO (1) WO2020029406A1 (fr)

Cited By (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111627086A (zh) * 2020-06-03 2020-09-04 上海商汤智能科技有限公司 一种头像的展示方法、装置、计算机设备及存储介质
CN111860407A (zh) * 2020-07-29 2020-10-30 华侨大学 一种视频中人物的表情识别方法、装置、设备及存储介质
CN111950447A (zh) * 2020-08-11 2020-11-17 合肥工业大学 基于走路姿态的情绪识别方法和***、存储介质
CN112016437A (zh) * 2020-08-26 2020-12-01 中国科学院重庆绿色智能技术研究院 一种基于人脸视频关键帧的活体检测方法
CN112364736A (zh) * 2020-10-30 2021-02-12 深圳点猫科技有限公司 一种动态人脸表情识别方法、装置及设备
CN112418146A (zh) * 2020-12-02 2021-02-26 深圳市优必选科技股份有限公司 表情识别方法、装置、服务机器人和可读存储介质
CN112507824A (zh) * 2020-11-27 2021-03-16 长威信息科技发展股份有限公司 一种视频图像特征识别的方法及***
CN112507959A (zh) * 2020-12-21 2021-03-16 中国科学院心理研究所 一种基于视频中个体面部分析的情绪感知模型的建立方法
CN112686195A (zh) * 2021-01-07 2021-04-20 风变科技（深圳）有限公司 情绪识别方法、装置、计算机设备和存储介质
CN112699774A (zh) * 2020-12-28 2021-04-23 深延科技(北京)有限公司 视频中人物的情绪识别方法及装置、计算机设备及介质
CN112712022A (zh) * 2020-12-29 2021-04-27 华南理工大学 基于图像识别的压力检测方法、***、装置及存储介质
CN113434647A (zh) * 2021-06-18 2021-09-24 竹间智能科技（上海）有限公司 一种人机交互方法、***及存储介质
CN114943924A (zh) * 2022-06-21 2022-08-26 深圳大学 基于人脸表***的疼痛评估方法、***、设备及介质
CN114998440A (zh) * 2022-08-08 2022-09-02 广东数业智能科技有限公司 基于多模态的测评方法、装置、介质及设备
CN115019374A (zh) * 2022-07-18 2022-09-06 北京师范大学 基于人工智能的智慧课堂学生专注度低耗检测方法及***
CN115177252A (zh) * 2022-07-13 2022-10-14 承德石油高等专科学校 一种可便捷操作的智能养老服务心理疏导装置
CN116665281A (zh) * 2023-06-28 2023-08-29 湖南创星科技股份有限公司 一种基于医患交互的关键情绪提取方法
CN117312992A (zh) * 2023-11-30 2023-12-29 哈尔滨工业大学(深圳)(哈尔滨工业大学深圳科技创新研究院) 多视角人脸特征与音频特征融合的情绪识别方法及***

Families Citing this family (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109784277B (zh) * 2019-01-17 2023-04-28 南京大学 一种基于智能眼镜的情绪识别方法
CN109816893B (zh) * 2019-01-23 2022-11-04 深圳壹账通智能科技有限公司 信息发送方法、装置、服务器及存储介质
CN109934097A (zh) * 2019-01-23 2019-06-25 深圳市中银科技有限公司 一种基于人工智能的表情和心理健康管理***
CN109871807B (zh) * 2019-02-21 2023-02-10 百度在线网络技术（北京）有限公司 人脸图像处理方法和装置
CN109992505B (zh) * 2019-03-15 2024-07-02 平安科技（深圳）有限公司 应用程序测试方法、装置、计算机设备及存储介质
CN110047588A (zh) * 2019-03-18 2019-07-23 平安科技（深圳）有限公司 基于微表情的呼叫方法、装置、计算机设备及存储介质
CN110175526B (zh) * 2019-04-28 2024-06-21 平安科技（深圳）有限公司 狗情绪识别模型训练方法、装置、计算机设备及存储介质
CN110097004B (zh) * 2019-04-30 2022-03-29 北京字节跳动网络技术有限公司 面部表情识别方法和装置
CN110414323A (zh) * 2019-06-14 2019-11-05 平安科技（深圳）有限公司 情绪检测方法、装置、电子设备及存储介质
CN110399837B (zh) * 2019-07-25 2024-01-05 深圳智慧林网络科技有限公司 用户情绪识别方法、装置以及计算机可读存储介质
CN110751381A (zh) * 2019-09-30 2020-02-04 东南大学 一种路怒车辆风险评估和防控方法
CN110991427B (zh) * 2019-12-25 2023-07-14 北京百度网讯科技有限公司 用于视频的情绪识别方法、装置和计算机设备
CN111401198B (zh) * 2020-03-10 2024-04-23 广东九联科技股份有限公司 观众情绪识别方法、装置及***
CN111767779A (zh) * 2020-03-18 2020-10-13 北京沃东天骏信息技术有限公司 图像处理方法、装置、设备及计算机可读存储介质
CN111783587A (zh) * 2020-06-22 2020-10-16 腾讯数码（天津）有限公司 一种互动方法、装置和存储介质
CN111859025A (zh) * 2020-07-03 2020-10-30 广州华多网络科技有限公司 表情指令生成方法、装置、设备及存储介质
CN112541425A (zh) * 2020-12-10 2021-03-23 深圳地平线机器人科技有限公司 情绪检测方法、装置、介质及电子设备
TWI811605B (zh) * 2020-12-31 2023-08-11 宏碁股份有限公司 情緒指標預測方法與系統
CN114005153A (zh) * 2021-02-01 2022-02-01 南京云思创智信息科技有限公司 面貌多样性的个性化微表情实时识别方法
CN113076813B (zh) * 2021-03-12 2024-04-12 首都医科大学宣武医院 面具脸特征识别模型训练方法和装置
CN113128399B (zh) * 2021-04-19 2022-05-17 重庆大学 用于情感识别的语音图像关键帧提取方法
CN113505665B (zh) * 2021-06-28 2023-06-20 哈尔滨工业大学（深圳） 基于视频的学生在校情绪判读方法以及装置
CN114694234B (zh) * 2022-06-02 2023-02-03 杭州智诺科技股份有限公司 情绪识别方法、***、电子设备和存储介质
CN116343314B (zh) * 2023-05-30 2023-08-25 之江实验室 一种表情识别方法、装置、存储介质及电子设备

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104008391A (zh) * 2014-04-30 2014-08-27 首都医科大学 一种基于非线性降维的人脸微表情捕捉及识别方法
CN105354527A (zh) * 2014-08-20 2016-02-24 南京普爱射线影像设备有限公司 一种消极表情识别鼓励***
CN107403142A (zh) * 2017-07-05 2017-11-28 山东中磁视讯股份有限公司 一种微表情的检测方法
CN107665074A (zh) * 2017-10-18 2018-02-06 维沃移动通信有限公司 一种色温调节方法及移动终端

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104298682B (zh) * 2013-07-18 2018-02-23 广州华久信息科技有限公司 一种基于人脸表情图像的信息推荐效果的评价方法及手机
CN103870820A (zh) * 2014-04-04 2014-06-18 南京工程学院 极端光照人脸识别的光照归一化方法
CN106295566B (zh) * 2016-08-10 2019-07-09 北京小米移动软件有限公司 人脸表情识别方法及装置
CN106980811A (zh) * 2016-10-21 2017-07-25 商汤集团有限公司 人脸表情识别方法和人脸表情识别装置
CN106951856A (zh) * 2017-03-16 2017-07-14 腾讯科技（深圳）有限公司 表情包提取方法及装置
CN107358169A (zh) * 2017-06-21 2017-11-17 厦门中控智慧信息技术有限公司 一种人脸表情识别方法及人脸表情识别装置
CN107633203A (zh) * 2017-08-17 2018-01-26 平安科技（深圳）有限公司 面部情绪识别方法、装置及存储介质

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104008391A (zh) * 2014-04-30 2014-08-27 首都医科大学 一种基于非线性降维的人脸微表情捕捉及识别方法
CN105354527A (zh) * 2014-08-20 2016-02-24 南京普爱射线影像设备有限公司 一种消极表情识别鼓励***
CN107403142A (zh) * 2017-07-05 2017-11-28 山东中磁视讯股份有限公司 一种微表情的检测方法
CN107665074A (zh) * 2017-10-18 2018-02-06 维沃移动通信有限公司 一种色温调节方法及移动终端

Cited By (31)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111627086A (zh) * 2020-06-03 2020-09-04 上海商汤智能科技有限公司 一种头像的展示方法、装置、计算机设备及存储介质
CN111860407A (zh) * 2020-07-29 2020-10-30 华侨大学 一种视频中人物的表情识别方法、装置、设备及存储介质
CN111860407B (zh) * 2020-07-29 2023-04-25 华侨大学 一种视频中人物的表情识别方法、装置、设备及存储介质
CN111950447A (zh) * 2020-08-11 2020-11-17 合肥工业大学 基于走路姿态的情绪识别方法和***、存储介质
CN111950447B (zh) * 2020-08-11 2023-08-22 合肥工业大学 基于走路姿态的情绪识别方法和***、存储介质
CN112016437B (zh) * 2020-08-26 2023-02-10 中国科学院重庆绿色智能技术研究院 一种基于人脸视频关键帧的活体检测方法
CN112016437A (zh) * 2020-08-26 2020-12-01 中国科学院重庆绿色智能技术研究院 一种基于人脸视频关键帧的活体检测方法
CN112364736A (zh) * 2020-10-30 2021-02-12 深圳点猫科技有限公司 一种动态人脸表情识别方法、装置及设备
CN112507824A (zh) * 2020-11-27 2021-03-16 长威信息科技发展股份有限公司 一种视频图像特征识别的方法及***
CN112418146A (zh) * 2020-12-02 2021-02-26 深圳市优必选科技股份有限公司 表情识别方法、装置、服务机器人和可读存储介质
CN112418146B (zh) * 2020-12-02 2024-04-30 深圳市优必选科技股份有限公司 表情识别方法、装置、服务机器人和可读存储介质
CN112507959A (zh) * 2020-12-21 2021-03-16 中国科学院心理研究所 一种基于视频中个体面部分析的情绪感知模型的建立方法
CN112699774A (zh) * 2020-12-28 2021-04-23 深延科技(北京)有限公司 视频中人物的情绪识别方法及装置、计算机设备及介质
CN112699774B (zh) * 2020-12-28 2024-05-24 深延科技(北京)有限公司 视频中人物的情绪识别方法及装置、计算机设备及介质
CN112712022A (zh) * 2020-12-29 2021-04-27 华南理工大学 基于图像识别的压力检测方法、***、装置及存储介质
CN112712022B (zh) * 2020-12-29 2023-05-23 华南理工大学 基于图像识别的压力检测方法、***、装置及存储介质
CN112686195A (zh) * 2021-01-07 2021-04-20 风变科技（深圳）有限公司 情绪识别方法、装置、计算机设备和存储介质
CN113434647B (zh) * 2021-06-18 2024-01-12 竹间智能科技（上海）有限公司 一种人机交互方法、***及存储介质
CN113434647A (zh) * 2021-06-18 2021-09-24 竹间智能科技（上海）有限公司 一种人机交互方法、***及存储介质
CN114943924B (zh) * 2022-06-21 2024-05-14 深圳大学 基于人脸表***的疼痛评估方法、***、设备及介质
CN114943924A (zh) * 2022-06-21 2022-08-26 深圳大学 基于人脸表***的疼痛评估方法、***、设备及介质
CN115177252A (zh) * 2022-07-13 2022-10-14 承德石油高等专科学校 一种可便捷操作的智能养老服务心理疏导装置
CN115177252B (zh) * 2022-07-13 2024-01-12 承德石油高等专科学校 一种可便捷操作的智能养老服务心理疏导装置
CN115019374B (zh) * 2022-07-18 2022-10-11 北京师范大学 基于人工智能的智慧课堂学生专注度低耗检测方法及***
CN115019374A (zh) * 2022-07-18 2022-09-06 北京师范大学 基于人工智能的智慧课堂学生专注度低耗检测方法及***
CN114998440B (zh) * 2022-08-08 2022-11-11 广东数业智能科技有限公司 基于多模态的测评方法、装置、介质及设备
CN114998440A (zh) * 2022-08-08 2022-09-02 广东数业智能科技有限公司 基于多模态的测评方法、装置、介质及设备
CN116665281A (zh) * 2023-06-28 2023-08-29 湖南创星科技股份有限公司 一种基于医患交互的关键情绪提取方法
CN116665281B (zh) * 2023-06-28 2024-05-10 湖南创星科技股份有限公司 一种基于医患交互的关键情绪提取方法
CN117312992A (zh) * 2023-11-30 2023-12-29 哈尔滨工业大学(深圳)(哈尔滨工业大学深圳科技创新研究院) 多视角人脸特征与音频特征融合的情绪识别方法及***
CN117312992B (zh) * 2023-11-30 2024-03-12 哈尔滨工业大学(深圳)(哈尔滨工业大学深圳科技创新研究院) 多视角人脸特征与音频特征融合的情绪识别方法及***

Also Published As

Publication number Publication date
CN109190487A (zh) 2019-01-11

Similar Documents

Publication Publication Date Title
WO2020029406A1 (fr) Procédé et dispositif de d'identification d'émotion de visage humain, dispositif informatique et support de stockage
CN111488433B (zh) 一种适用于银行的提升现场体验感的人工智能交互***
WO2020098249A1 (fr) Dispositif électronique, procédé de recommandation de technique de conversation de réponse et support de stockage lisible par ordinateur
WO2020211388A1 (fr) Procédé et dispositif de prédiction de comportement employant un modèle de prédiction, appareil, et support de stockage
CN110147726B (zh) 业务质检方法和装置、存储介质及电子装置
US10462300B2 (en) Technologies for monitoring interaction between customers and agents using sentiment detection
WO2020248376A1 (fr) Procédé et appareil de détection d'émotion, dispositif électronique et support de stockage
US11138903B2 (en) Method, apparatus, device and system for sign language translation
CN110085225B (zh) 语音交互方法、装置、智能机器人及计算机可读存储介质
WO2020007129A1 (fr) Procédé et dispositif d'acquisition de contexte basés sur une interaction vocale
CN102890776B (zh) 通过面部表情调取表情图释的方法
CN106294774A (zh) 基于对话服务的用户个性化数据处理方法及装置
WO2020253128A1 (fr) Procédé, appareil, dispositif informatique et support d'informations associés à un service de communication basé sur la reconnaissance vocale
WO2019134580A1 (fr) Procédé et dispositif de gestion d'utilisateurs de jeu
JP2018169494A (ja) 発話意図推定装置および発話意図推定方法
CN113223560A (zh) 情绪识别方法、装置、设备及存储介质
WO2022257452A1 (fr) Procédé et appareil de réponse par mème, dispositif, et support de stockage
WO2021228148A1 (fr) Procédé d'extraction de caractéristiques destiné à protéger la confidentialité de données personnelles, procédé de formation de modèle et matériel
CN110879986A (zh) 人脸识别的方法、设备和计算机可读存储介质
CN112908325A (zh) 语音交互方法、装置、电子设备及存储介质
WO2021051602A1 (fr) Procédé et système de reconnaissance faciale à base de mot de passe lu avec les lèvres, dispositif, et support de stockage
CN113469023B (zh) 确定警觉度的方法、装置、设备和存储介质
US20210166685A1 (en) Speech processing apparatus and speech processing method
CN113128284A (zh) 一种多模态情感识别方法和装置
CN110503943B (zh) 一种语音交互方法以及语音交互***

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18929826

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18929826

Country of ref document: EP

Kind code of ref document: A1