CN111329498A - Multi-modal driver emotion auxiliary adjusting method - Google Patents
Multi-modal driver emotion auxiliary adjusting method Download PDFInfo
- Publication number
- CN111329498A CN111329498A CN202010157896.XA CN202010157896A CN111329498A CN 111329498 A CN111329498 A CN 111329498A CN 202010157896 A CN202010157896 A CN 202010157896A CN 111329498 A CN111329498 A CN 111329498A
- Authority
- CN
- China
- Prior art keywords
- driver
- road rage
- information
- emotion
- signal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/74—Details of notification to user or communication with user or patient ; user input means
- A61B5/7405—Details of notification to user or communication with user or patient ; user input means using sound
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0059—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
- A61B5/0077—Devices for viewing the surface of the body, e.g. camera, magnifying lens
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
- A61B5/18—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state for vehicle drivers or machine operators
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/48—Other medical applications
- A61B5/4803—Speech analysis specially adapted for diagnostic purposes
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/74—Details of notification to user or communication with user or patient ; user input means
- A61B5/746—Alarms related to a physiological condition, e.g. details of setting alarm thresholds or avoiding false alarms
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Veterinary Medicine (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Molecular Biology (AREA)
- Surgery (AREA)
- Animal Behavior & Ethology (AREA)
- Physics & Mathematics (AREA)
- Public Health (AREA)
- Pathology (AREA)
- Child & Adolescent Psychology (AREA)
- Developmental Disabilities (AREA)
- Educational Technology (AREA)
- Hospice & Palliative Care (AREA)
- Psychiatry (AREA)
- Psychology (AREA)
- Social Psychology (AREA)
- Physiology (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Traffic Control Systems (AREA)
Abstract
The invention discloses a multi-modal driver emotion auxiliary adjusting method, which relates to the field of artificial intelligence and machine learning, and comprises the steps of collecting multi-modal data information of a driver; according to the multi-modal data information, the emotional states of the drivers in the single modes are respectively identified, and whether the emotional state of the driver in each single mode is a road rage signal or not is judged; fusing judgment results of emotional states of a plurality of drivers in single mode, judging whether the fusion result is a total road rage signal, and grading the total road rage signal; and correspondingly outputting different voice reminding information according to the grade of the total road rage signal, and adjusting the emotion of the driver. The invention can monitor the emotion of the driver in real time, and can adopt related measures to assist in adjusting the emotion of the driver when the emotional state of the driver is abnormal, thereby reducing the occurrence of accidents and protecting the driving safely.
Description
Technical Field
The invention relates to the field of artificial intelligence and machine learning, in particular to a multi-modal driver emotion auxiliary adjusting method.
Background
The concept of road rage (Roadrage) was first derived from foreign psychology. The medical community classifies the road rage as paroxysmal rage disorder, which means that multiple anger fires explode, and the violent degree is called as a big feeling and an accident. For example, when a driver encounters traffic jam or other things affecting mood, the driver can show behaviors of operation modes such as anger, frequent whistling, frequent braking and the like.
Traffic accidents caused by road rage are rising every year, and the public traffic safety is seriously influenced. Although some driving assistance systems are already available in the market, the influence of the emotion of the driver on the driving safety is not considered, and an appropriate adjusting strategy is not matched according to the emotion recognition result.
Disclosure of Invention
In order to solve the problems in the background technology, the invention provides a multi-modal driver emotion auxiliary adjusting method, which collects data related to emotion judgment of a driver through various channels, performs emotion analysis and judgment through a deep learning method, and further adopts related measures to assist in adjusting the emotion of the driver, so that accidents are reduced.
The technical scheme of the invention is as follows:
a multi-modal driver emotion aid adjustment method comprises the following steps:
s1, collecting multi-modal data information of the driver; the multi-modal data information comprises image information, voice information and driving behavior information;
s2, according to the multi-modal data information, respectively identifying the emotional states of the drivers in a plurality of single modes, and judging whether the emotional state of each driver in the single mode is a road rage signal;
s3, fusing judgment results of emotional states of the drivers in the single modes, judging whether the fusion result is a general road rage signal, and grading the general road rage signal;
and S4, correspondingly outputting different voice reminding information according to the grade of the total road rage signal, and adjusting the emotion of the driver.
Preferably, the S2 includes image information identification, specifically:
the method comprises the steps that driver image information captured by a camera is disassembled into frames, the frames are respectively sent to an OpenCv face selector, and face images of drivers in a vehicle are cut out in real time;
and (3) converting the size of the facial image into 224 × 224 pixels, sending the facial image into a VGG16 deep convolution neural network model to extract features, and dividing the facial image into a road anger signal and a non-road anger signal.
Preferably, the S2 includes speech information recognition, specifically:
capturing the voice information of a driver in the vehicle in real time by adopting a VAD algorithm, and sending the voice information of the driver into an ASR model to convert the voice into character information;
judging whether the voice information of the driver contains a wake-up word or not; if the wake-up word is contained, sending the wake-up word into a voice interaction module to carry out human-computer interaction normally; if the awakening words are not contained, emotion analysis is carried out, and the voice information of the driver is divided into a road rage signal and a non-road rage signal.
Preferably, the S2 includes driving behavior information identification, specifically:
collecting driving behavior data of driver by film sensor mounted on steering wheel and brake
The collected driving behavior data of the driver are sent to a random forest model, and the driving behavior data of the driver are divided into a road rage signal and a non-road rage signal through the random forest model;
the driving behavior data of the driver comprise average starting hot-car time, on-site idling ratio, rapid accelerator pedaling frequency and rapid braking frequency of the driver within three days.
Preferably, in S3, if the emotional state of the driver in the single mode is a road rage signal, the emotional state is marked as "1"; the emotional state of the driver in each single mode is a road rage signal and is marked as '0'; the total road rage signal is graded, and the grading comprises the following steps:
when the emotional states of the drivers in the single modes are added to be 0, the driver is not angry;
when the emotional states of the drivers in the single modes are added to be 1, the driver is in mild road rage;
when the emotional states of the drivers in the single modes are added to be 2, the driver is moderate road rage;
and when the emotional states of the drivers in the single modes are added to be 3, the road is heavily irritated.
Compared with the prior art, the invention has the beneficial effects that: the invention can monitor the emotion of the driver in real time according to the image information, the voice information and the driving behavior information of the driver, and can adopt related voice reminding measures to assist in adjusting the emotion of the driver when the emotion state of the driver is abnormal, so that the occurrence of accidents is reduced, and the driving safety is protected.
Drawings
FIG. 1 is a flow chart of the present invention;
fig. 2 is a working principle diagram of the present invention.
Detailed Description
The technical solutions of the embodiments of the present invention are clearly and completely described below with reference to the drawings in the present invention, and it is obvious that the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, shall fall within the scope of protection of the present invention.
As shown in fig. 1, the invention provides a multimodal driver emotion auxiliary adjusting method, which includes:
s1, collecting multi-modal data information of the driver; the multi-modal data information comprises image information, voice information and driving behavior information;
s2, according to the multi-modal data information, respectively identifying the emotional states of the drivers in a plurality of single modes, and judging whether the emotional state of each driver in the single mode is a road rage signal;
s3, fusing judgment results of emotional states of the drivers in the single modes, judging whether the fusion result is a general road rage signal, and grading the general road rage signal;
and S4, correspondingly outputting different voice reminding information according to the grade of the total road rage signal, and adjusting the emotion of the driver.
Specifically, as shown in fig. 2, the emotional state data of the driver in each of the plurality of single modalities includes image information, voice information, and driving behavior information, respectively.
Image information identification, specifically: the method comprises the steps that driver image information captured by a camera is disassembled into frames, the frames are respectively sent to an OpenCv face selector, and face images of drivers in a vehicle are cut out in real time; and (3) converting the size of the facial image into 224 × 224 pixels, sending the facial image into a VGG16 deep convolution neural network model to extract features, and dividing the facial image into a road anger signal and a non-road anger signal.
The voice information recognition specifically comprises the following steps: capturing the voice information of a driver in the vehicle in real time by adopting a VAD algorithm, and sending the voice information of the driver into an ASR model to convert the voice into character information; judging whether the voice information of the driver contains a wake-up word or not; if the awakening words are contained, the voice interaction module is sent to carry out human-computer interaction normally. Such as voice short message sending, voice weather and road condition information inquiry, voice playing, song cutting and the like.
If the awakening words are not contained, emotion analysis is carried out, and the voice information of the driver is divided into a road rage signal and a non-road rage signal.
The driving behavior information identification specifically comprises the following steps: collecting driving behavior data of a driver through a film sensor arranged at a steering wheel and a brake; the collected driving behavior data of the driver are sent to a random forest model, and the driving behavior data of the driver are divided into a road rage signal and a non-road rage signal through the random forest model; the driving behavior data of the driver comprise average starting hot-car time, on-site idling ratio, rapid accelerator pedaling frequency and rapid braking frequency of the driver within three days.
Wherein, the emotional state of the driver in each single mode is a road rage signal, and is marked as '1'; the emotional state of the driver in each single mode is a road rage signal and is marked as '0'; the total road rage signal is graded, and the grading comprises the following steps:
(1) when the emotional states of the drivers in the single modes are added to be 0, the driver is not angry, namely the emotional state of the driver is 0, and the driver is not angry, the voice prompt is not carried out;
(2) when the emotional states of the drivers in the single modes are added to be 1, the driver is in mild road rage, namely the emotional state of the driver is 1, the level 1 prompt is activated, and meanwhile voice reminding information can be played, for example, whether you are not happy and a relaxing music bar is played. The soothing music in the song library may be collected by the driver in advance according to personal preferences or selected from selected songs.
(3) When the emotional states of the drivers in the single modes are added to be 2, the driver is moderate road rage, namely the emotional state of the driver is 2, 2-level prompt is activated, and meanwhile voice reminding information can be played, for example, "you are suspected of road rage now, mom wants to speak to you". And calling the voice recorded by the mother in advance from the voice library to prompt the driver.
(4) When the emotional states of the drivers in the single modes are added to be 3, the driver is severe road rage, namely the emotional state of the driver is 3, 3-level prompt is activated, and meanwhile voice reminding information can be played, for example, "dangerous now, your baby comes". And calling out the sound recorded by the child in advance from the voice library to prompt the driver.
Wherein, the level 2 and the level 3 can also be selected and set by the driver according to the importance degree in the mind.
The above disclosure is only for the preferred embodiments of the present invention, but the embodiments of the present invention are not limited thereto, and any variations that can be made by those skilled in the art are intended to fall within the scope of the present invention.
Claims (5)
1. A multi-modal driver emotion aid adjustment method is characterized by comprising the following steps:
s1, collecting multi-modal data information of the driver; the multi-modal data information comprises image information, voice information and driving behavior information;
s2, according to the multi-modal data information, respectively identifying the emotional states of the drivers in a plurality of single modes, and judging whether the emotional state of each driver in the single mode is a road rage signal;
s3, fusing judgment results of emotional states of the drivers in the single modes, judging whether the fusion result is a general road rage signal, and grading the general road rage signal;
and S4, correspondingly outputting different voice reminding information according to the grade of the total road rage signal, and adjusting the emotion of the driver.
2. The multi-modal driver emotion aid adjustment method as claimed in claim 1, wherein said S2 includes image information recognition, specifically:
the method comprises the steps that driver image information captured by a camera is disassembled into frames, the frames are respectively sent to an OpenCv face selector, and face images of drivers in a vehicle are cut out in real time;
and (3) converting the size of the facial image into 224 × 224 pixels, sending the facial image into a VGG16 deep convolution neural network model to extract features, and dividing the facial image into a road anger signal and a non-road anger signal.
3. The multi-modal driver emotion aid adjustment method as claimed in claim 1, wherein said S2 includes speech information recognition, specifically:
capturing the voice information of a driver in the vehicle in real time by adopting a VAD algorithm, and sending the voice information of the driver into an ASR model to convert the voice into character information;
judging whether the voice information of the driver contains a wake-up word or not; if the wake-up word is contained, sending the wake-up word into a voice interaction module to carry out human-computer interaction normally; if the awakening words are not contained, emotion analysis is carried out, and the voice information of the driver is divided into a road rage signal and a non-road rage signal.
4. The multi-modal driver emotion aid adjustment method as claimed in claim 1, wherein the S2 includes driving behavior information identification, specifically:
collecting driving behavior data of driver by film sensor mounted on steering wheel and brake
The collected driving behavior data of the driver are sent to a random forest model, and the driving behavior data of the driver are divided into a road rage signal and a non-road rage signal through the random forest model;
the driving behavior data of the driver comprise average starting hot-car time, on-site idling ratio, rapid accelerator pedaling frequency and rapid braking frequency of the driver within three days.
5. The multi-modal driver emotion aid adjusting method as claimed in claim 1, wherein in S3, each of the driver emotion states in the single mode is a road rage signal, which is recorded as "1"; the emotional state of the driver in each single mode is a road rage signal and is marked as '0'; the total road rage signal is graded, and the grading comprises the following steps:
when the emotional states of the drivers in the single modes are added to be 0, the driver is not angry;
when the emotional states of the drivers in the single modes are added to be 1, the driver is in mild road rage;
when the emotional states of the drivers in the single modes are added to be 2, the driver is moderate road rage;
and when the emotional states of the drivers in the single modes are added to be 3, the road is heavily irritated.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010157896.XA CN111329498A (en) | 2020-03-09 | 2020-03-09 | Multi-modal driver emotion auxiliary adjusting method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010157896.XA CN111329498A (en) | 2020-03-09 | 2020-03-09 | Multi-modal driver emotion auxiliary adjusting method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111329498A true CN111329498A (en) | 2020-06-26 |
Family
ID=71174294
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010157896.XA Pending CN111329498A (en) | 2020-03-09 | 2020-03-09 | Multi-modal driver emotion auxiliary adjusting method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111329498A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112078588A (en) * | 2020-08-11 | 2020-12-15 | 大众问问(北京)信息科技有限公司 | Vehicle control method and device and electronic equipment |
| CN112784695A (en) * | 2020-12-31 | 2021-05-11 | 南京视察者智能科技有限公司 | Driver abnormal state detection method based on image and voice recognition |
| CN113658580A (en) * | 2021-06-24 | 2021-11-16 | 大众问问(北京)信息科技有限公司 | Voice prompt method and device, computer equipment and storage medium |
| US20220355804A1 (en) * | 2021-05-05 | 2022-11-10 | Mitac Digital Technology Corporation | Method and system for enhancing driving safety |
| CN115359532A (en) * | 2022-08-23 | 2022-11-18 | 润芯微科技(江苏)有限公司 | Human face emotion capturing and outputting device based on 3D sensing |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100134302A1 (en) * | 2008-12-01 | 2010-06-03 | Electronics And Telecommunications Research Institute | System and method for controlling emotion of car driver |
| CN106114516A (en) * | 2016-08-31 | 2016-11-16 | 合肥工业大学 | The angry driver behavior modeling of a kind of drive automatically people's characteristic and tampering devic |
| CN106803423A (en) * | 2016-12-27 | 2017-06-06 | 智车优行科技（北京）有限公司 | Man-machine interaction sound control method, device and vehicle based on user emotion state |
| CN107235045A (en) * | 2017-06-29 | 2017-10-10 | 吉林大学 | Consider physiology and the vehicle-mounted identification interactive system of driver road anger state of manipulation information |
| CN107316436A (en) * | 2017-07-31 | 2017-11-03 | 努比亚技术有限公司 | Dangerous driving state processing method, electronic equipment and storage medium |
| CN108364373A (en) * | 2018-02-07 | 2018-08-03 | 安徽星网软件技术有限公司 | Vehicle behavior monitoring method and device |
| CN110751381A (en) * | 2019-09-30 | 2020-02-04 | 东南大学 | Road rage vehicle risk assessment and prevention and control method |
-
2020
- 2020-03-09 CN CN202010157896.XA patent/CN111329498A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100134302A1 (en) * | 2008-12-01 | 2010-06-03 | Electronics And Telecommunications Research Institute | System and method for controlling emotion of car driver |
| CN106114516A (en) * | 2016-08-31 | 2016-11-16 | 合肥工业大学 | The angry driver behavior modeling of a kind of drive automatically people's characteristic and tampering devic |
| CN106803423A (en) * | 2016-12-27 | 2017-06-06 | 智车优行科技（北京）有限公司 | Man-machine interaction sound control method, device and vehicle based on user emotion state |
| CN107235045A (en) * | 2017-06-29 | 2017-10-10 | 吉林大学 | Consider physiology and the vehicle-mounted identification interactive system of driver road anger state of manipulation information |
| CN107316436A (en) * | 2017-07-31 | 2017-11-03 | 努比亚技术有限公司 | Dangerous driving state processing method, electronic equipment and storage medium |
| CN108364373A (en) * | 2018-02-07 | 2018-08-03 | 安徽星网软件技术有限公司 | Vehicle behavior monitoring method and device |
| CN110751381A (en) * | 2019-09-30 | 2020-02-04 | 东南大学 | Road rage vehicle risk assessment and prevention and control method |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112078588A (en) * | 2020-08-11 | 2020-12-15 | 大众问问(北京)信息科技有限公司 | Vehicle control method and device and electronic equipment |
| CN112784695A (en) * | 2020-12-31 | 2021-05-11 | 南京视察者智能科技有限公司 | Driver abnormal state detection method based on image and voice recognition |
| CN112784695B (en) * | 2020-12-31 | 2024-02-20 | 南京视察者智能科技有限公司 | Method for detecting abnormal state of driver based on image and voice recognition |
| US20220355804A1 (en) * | 2021-05-05 | 2022-11-10 | Mitac Digital Technology Corporation | Method and system for enhancing driving safety |
| CN113658580A (en) * | 2021-06-24 | 2021-11-16 | 大众问问(北京)信息科技有限公司 | Voice prompt method and device, computer equipment and storage medium |
| CN115359532A (en) * | 2022-08-23 | 2022-11-18 | 润芯微科技(江苏)有限公司 | Human face emotion capturing and outputting device based on 3D sensing |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111329498A (en) | Multi-modal driver emotion auxiliary adjusting method | |
| CN106803423B (en) | Man-machine interaction voice control method and device based on user emotion state and vehicle | |
| CN112277955B (en) | Driving assistance method, device, equipment and storage medium | |
| CN106448221B (en) | Pushing method of online driving assistance information pushing system and application of pushing method | |
| CN112215097A (en) | Method for monitoring driving state of vehicle, vehicle and computer readable storage medium | |
| CN112785837A (en) | Method and device for recognizing emotion of user when driving vehicle, storage medium and terminal | |
| US20210362594A1 (en) | Car driver alcohol level and sleeping status detection and notification system | |
| CN112784695A (en) | Driver abnormal state detection method based on image and voice recognition | |
| CN109541600A (en) | A kind of heavy type commercial automobile safe and intelligent driving assistance system | |
| CN116729403A (en) | Emotion recognition reminding and control system and method in automobile driving process | |
| CN111904806A (en) | Blind guiding system | |
| CN206049658U (en) | Angry driver behavior modeling and tampering devic based on drive automatically people's characteristic | |
| Kamaruddin et al. | Driver behaviour state recognition based on speech | |
| JP2010271794A (en) | Driving behavior guiding system | |
| KR102270011B1 (en) | Deep learning-based autonomous vehicle visualize system for the visually impaired and method thereof | |
| Malta et al. | Multi-modal real-world driving data collection, transcription, and integration using Bayesian network | |
| KR102132058B1 (en) | Interactive voice communication system embedded in a car | |
| CN114655241A (en) | Automobile intelligent multimedia central control screen fatigue driving reminding method | |
| CN112773349A (en) | Method and device for constructing emotion log of user driving vehicle, storage medium and terminal | |
| CN111660927A (en) | Whistling method and system | |
| Kashevnik et al. | Driver intelligent support system in internet of transportation things: Smartphone-based approach | |
| JP2021114081A (en) | Drive recorder, recording method, and program | |
| JP3471042B2 (en) | Automotive electronics | |
| CN112706691A (en) | Vehicle reminding method and device | |
| US20230063708A1 (en) | Systems and methods for testing driver awareness within a vehicle and/or intervening during a possible crisis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |