CN108446645B - Vehicle-mounted face recognition method based on deep learning - Google Patents

Vehicle-mounted face recognition method based on deep learning Download PDF

Info

Publication number
CN108446645B
CN108446645B CN201810252216.5A CN201810252216A CN108446645B CN 108446645 B CN108446645 B CN 108446645B CN 201810252216 A CN201810252216 A CN 201810252216A CN 108446645 B CN108446645 B CN 108446645B
Authority
CN
China
Prior art keywords
driver
data set
control end
context vector
images
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201810252216.5A
Other languages
Chinese (zh)
Other versions
CN108446645A (en
Inventor
冀中
贺二路
庞彦伟
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tianjin University
Original Assignee
Tianjin University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tianjin University filed Critical Tianjin University
Priority to CN201810252216.5A priority Critical patent/CN108446645B/en
Publication of CN108446645A publication Critical patent/CN108446645A/en
Application granted granted Critical
Publication of CN108446645B publication Critical patent/CN108446645B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
    • G06V20/597—Recognising the driver's state or behaviour, e.g. attention or drowsiness
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Human Computer Interaction (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

A vehicle-mounted face recognition method based on deep learning comprises the following steps: acquiring an image and constructing a driver data set; constructing a model, including extracting visual features and semantic features and forming a feature processing model; training a feature processing model; optimizing the experimental result according to the test result, and transmitting the optimized feature processing model to the master control end; install warning light and camera on the car, reach total control end in real time with driver's operational aspect, total control end judges whether the driver has the driving of violating the rules according to the feature handling model after optimizing, and when the driving of violating the rules, total control end signals, arouses the warning light, reminds the driver civilized travel. The invention analyzes the face change in real time, extracts the face characteristics, detects whether the face has illegal operation, analyzes and compares the face with the data set, and automatically sends out an alarm once illegal behaviors are found, such as fatigue driving, answering a call, watching a mobile phone and the like, so as to timely stop the bad behaviors of a driver. The possibility of traffic accidents is reduced.

Description

Vehicle-mounted face recognition method based on deep learning
Technical Field
The invention relates to a face recognition method. In particular to a vehicle-mounted face recognition method based on deep learning.
Background
With the rapid development of information acquisition and information processing technologies, computer vision, that is, how to efficiently and accurately acquire relevant information from an environmental image or video by using a computer technology, and further analyze, judge and decide objects and phenomena occurring in an objective world, has become a very important research topic. With the rapid development of deep learning, computer vision develops rapidly in recent years, and the deep convolutional neural network plays an important role in research and development of computer vision.
With the rapid development of computer vision, the development of face recognition technology is good, and the application of face recognition is more and more extensive, and the face recognition is a biological recognition technology for recognizing based on facial features of a human face, and an image or video stream containing the human face is acquired by an image acquisition device such as a camera, and then face detection and face tracking are performed in the image, so that a series of related technologies related to the face of the detected human face are performed. The present face recognition technology mainly comprises (1) a face recognition method of geometric characteristics; (2) the face recognition method based on the characteristic face (PCA) (3) the neural network (4) the elastic image matching (5) the line segment Hausdorff distance (LHD) (6) and the Support Vector Machine (SVM) are adopted.
At present, traffic accidents frequently occur, besides natural reasons, a great part of the problems are the problems of drivers, mainly include fatigue driving, answering calls, distracting when driving and the like, and accidents caused by the reasons are countless.
Although the vehicle-mounted camera in China has certain development at present, the vehicle-mounted camera mainly focuses on the fields of vehicle speed testing, target tracking, pedestrian detection, obstacle detection and the like, no excessive research is carried out on safe and civilized driving of a driver, and a real-time and efficient judgment method for judging whether the driver violates driving is lacked in the past due to technical limitations. The face recognition technology based on deep learning can supervise and judge the real-time operation of the driver, and can effectively reduce traffic accidents caused by illegal operation of the driver.
Disclosure of Invention
The invention aims to solve the technical problem of providing a vehicle-mounted face recognition method based on deep learning, which can judge whether a driver has phenomena of fatigue driving and the like by analyzing face changes in real time and give prompt.
The technical scheme adopted by the invention is as follows: a vehicle-mounted face recognition method based on deep learning comprises the following steps:
1) acquiring an image and constructing a driver data set;
2) constructing a model, comprising:
(1) respectively extracting visual features and semantic features from the driver data set through a convolutional neural network and an LSTM network;
(2) inputting the extracted visual features and semantic features into an LSTM network with an attention mechanism to form a feature processing model;
3) training a feature processing model, wherein 60% of images in a driver data set are used for training, 20% of images are used for verification, and 20% of images are used for testing;
4) according to the test result, respectively aligning the parameters Wz、WrW, fine adjustment is carried out, an experimental result is optimized, and the optimized characteristic processing model is transmitted to a master control end;
5) install warning light and camera on the car, reach total control end in real time with driver's operational aspect, total control end judges whether the driver has the driving of violating the rules according to the feature handling model after optimizing, and when the driving of violating the rules, total control end signals, arouses the warning light, reminds the driver civilized travel.
The method comprises the steps of 1) utilizing a python-based network picture acquisition script to acquire different driver images through the Internet, making labels on the images, wherein the labels are marked with image contents in detail and then summarized to serve as a driver data set, and the image contents comprise: the driver drives normally, looks down at the mobile phone, watches everywhere, chats and drives tiredly.
The (1) in the step 2) comprises: extracting 14 multiplied by 512 dimensional visual characteristics of the driver data set by using conv5_3 layer of VGG-19 network in convolutional neural network to obtain characteristic vector aiGenerating a visual information context vector z by an attention mechanismvt(ii) a Semantic features of a driver data set are extracted through an LSTM network and a semantic context vector z is obtainedst。
The step (2) in the step 2) comprises the following steps:
(1) context vector z of visual informationvtAnd a semantic context vector zstForming a context vector z capable of more fully expressing image information by affine transformationt；
(2) The obtained context vector ztInput into LSTM network with attention mechanism, and analyze driver's behavior.
The characteristic processing model comprises the following steps:
et＝fatt(ai,ht-1)
Figure BDA0001608079300000021
Figure BDA0001608079300000022
zt＝σ(Wz·[ht-1,xt])
rt＝σ(Wr·[ht-1,xt])
Figure BDA0001608079300000023
Figure BDA0001608079300000024
wherein z isvtRepresenting a context vector of visual information, aiRepresenting the visual feature vector, αtRepresents a weight, ztRepresenting a context vector, xtInput representing the current time, htAnd ht-1Respectively representing the hidden layer states at the current time and the previous time,
Figure BDA0001608079300000025
for candidate states of the hidden layer at the current moment, Wz、WrAnd W is a parameter.
The vehicle-mounted face recognition method based on deep learning of the invention obtains real-time information of a driver through a vehicle-mounted camera in the driving process, analyzes face change in real time, extracts face characteristics, detects whether illegal operation exists through a convolutional neural network, analyzes and compares the face characteristics with a data set, and automatically gives an alarm once illegal behaviors are found, such as fatigue driving, answering a call, watching a mobile phone and the like, so as to timely stop bad behaviors of the driver. The possibility of traffic accidents is reduced.
Drawings
FIG. 1 is a flow chart of the vehicle-mounted face recognition method based on deep learning of the invention.
Detailed Description
The following describes the in-vehicle face recognition method based on deep learning in detail with reference to the embodiments and the accompanying drawings.
As shown in fig. 1, the vehicle-mounted face recognition method based on deep learning of the present invention includes the following steps:
1) acquiring images and constructing a driver data set, comprising:
acquiring different driver images through the Internet by using a python-based network image acquisition script, making labels on the images, wherein the labels are used for indicating image contents in detail and then summarizing the image contents to be used as a driver data set, and the image contents comprise: the driver drives normally, looks down at the mobile phone, watches everywhere, chats and drives tiredly.
2) Constructing a model, comprising:
(1) in order to better distinguish the behavior of the driver, the invention extracts visual features and semantic features. Respectively extracting visual features and semantic features from the driver data set through a convolutional neural network and an LSTM network; the method comprises the following steps: extracting 14 multiplied by 512 dimensional visual characteristics of the driver data set by using conv5_3 layer of VGG-19 network in convolutional neural network to obtain characteristic vector aiGenerating a visual information context vector z by an attention mechanismvt(ii) a Semantic features of a driver data set are extracted through an LSTM network and a semantic context vector z is obtainedst。
(2) Inputting the extracted visual features and semantic features into an LSTM network with an attention mechanism to form a feature processing model; the method comprises the following steps:
(2.1) context vector z of visual informationvtAnd a semantic context vector zstForming a context vector z capable of more fully expressing image information by affine transformationt；
(2.2) obtaining a context vector ztInput into LSTM network with attention mechanism, and analyze driver's behavior. For example: mouth shape changes during chatting, and eyes changes during fatigue.
LSTM (Long Short term) is a special RNN model that can learn long-term dependency information. LSTM can selectively remember or forget previous information by a forgetting gate. The Attention mechanism is a method of image processing that is currently very popular, and assigns Attention levels by calculating weights of relevant regions of an image. The characteristic processing model comprises the following steps:
et＝fatt(ai,ht-1)
Figure BDA0001608079300000031
Figure BDA0001608079300000032
zt＝σ(Wz·[ht-1,xt])
rt＝σ(Wr·[ht-1,xt])
Figure BDA0001608079300000033
Figure BDA0001608079300000034
wherein z isvtRepresenting a context vector of visual information, aiRepresenting the visual feature vector, αtRepresents a weight, ztRepresenting a context vector, xtInput representing the current time, htAnd ht-1Respectively representing the hidden layer states at the current time and the previous time,
Figure BDA0001608079300000041
for candidate states of the hidden layer at the current moment, Wz、WrAnd W is a parameter.
3) Training a feature processing model, wherein 60% of images in a driver data set are used for training, 20% of images are used for verification, and 20% of images are used for testing;
4) according to the test result, respectively aligning the parameters Wz、WrW, fine adjustment is carried out, an experimental result is optimized, and the optimized characteristic processing model is transmitted to a master control end;
5) install warning light and camera on the car, reach total control end in real time with driver's operational aspect, total control end judges whether the driver has the driving of violating the rules according to the feature handling model after optimizing, and when the driving of violating the rules, total control end signals, arouses the warning light, reminds the driver civilized travel.

Claims (2)

1. A vehicle-mounted face recognition method based on deep learning is characterized by comprising the following steps:
1) acquiring an image and constructing a driver data set;
2) constructing a model, comprising:
(1) respectively extracting visual features and semantic features from the driver data set through a convolutional neural network and an LSTM network; the method comprises the following steps: extracting 14 multiplied by 512 dimensional visual characteristics of the driver data set by using conv5_3 layer of VGG-19 network in convolutional neural network to obtain characteristic vector aiGenerating a visual information context vector z by an attention mechanismvt(ii) a Semantic features of a driver data set are extracted through an LSTM network and a semantic context vector z is obtainedst；
(2) Inputting the extracted visual features and semantic features into an LSTM network with an attention mechanism to form a feature processing model; the method comprises the following steps:
(2.1) context vector z of visual informationvtAnd a semantic context vector zstForming a context vector z capable of more fully expressing image information by affine transformationt；
(2.2) obtaining a context vector ztInputting the data into an LSTM network with an attention mechanism, and analyzing the behavior of a driver;
3) training a feature processing model, wherein 60% of images in a driver data set are used for training, 20% of images are used for verification, and 20% of images are used for testing; the characteristic processing model comprises the following steps:
et＝fatt(ai,ht-1)
Figure FDA0003115850960000011
Figure FDA0003115850960000012
zt＝σ(Wz·[ht-1,xt])
rt＝σ(Wr·[ht-1,xt])
Figure FDA0003115850960000013
Figure FDA0003115850960000014
wherein z isvtRepresenting a context vector of visual information, aiRepresenting the visual feature vector, αtRepresents a weight, ztRepresenting a context vector, xtInput representing the current time, htAnd ht-1Respectively representing the hidden layer states at the current time and the previous time,
Figure FDA0003115850960000015
for candidate states of the hidden layer at the current moment, Wz、WrW is a parameter;
4) according to the test result, respectively aligning the parameters Wz、WrW, fine adjustment is carried out, an experimental result is optimized, and the optimized characteristic processing model is transmitted to a master control end;
5) install warning light and camera on the car, reach total control end in real time with driver's operational aspect, total control end judges whether the driver has the driving of violating the rules according to the feature handling model after optimizing, and when the driving of violating the rules, total control end signals, arouses the warning light, reminds the driver civilized travel.
2. The deep learning-based vehicle-mounted face recognition method according to claim 1, wherein the step 1) comprises the steps of acquiring different driver images through the internet by using a python-based network picture acquisition script, labeling the images, and summarizing the images as a driver data set, wherein the image contents comprise: the driver drives normally, looks down at the mobile phone, watches everywhere, chats and drives tiredly.
CN201810252216.5A 2018-03-26 2018-03-26 Vehicle-mounted face recognition method based on deep learning Active CN108446645B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810252216.5A CN108446645B (en) 2018-03-26 2018-03-26 Vehicle-mounted face recognition method based on deep learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810252216.5A CN108446645B (en) 2018-03-26 2018-03-26 Vehicle-mounted face recognition method based on deep learning

Publications (2)

Publication Number Publication Date
CN108446645A CN108446645A (en) 2018-08-24
CN108446645B true CN108446645B (en) 2021-12-31

Family

ID=63197048

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810252216.5A Active CN108446645B (en) 2018-03-26 2018-03-26 Vehicle-mounted face recognition method based on deep learning

Country Status (1)

Country Link
CN (1) CN108446645B (en)

Families Citing this family (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109124625B (en) * 2018-09-04 2021-07-20 大连理工大学 Driver fatigue state level grading method
CN109614303A (en) * 2018-12-05 2019-04-12 国网北京市电力公司 A kind of violation information processing method and processing device
CN109886209A (en) * 2019-02-25 2019-06-14 成都旷视金智科技有限公司 Anomaly detection method and device, mobile unit
CN110059541A (en) * 2019-02-28 2019-07-26 长江大学 A kind of mobile phone usage behavior detection method and device in driving
CN110334614A (en) * 2019-06-19 2019-10-15 腾讯科技（深圳）有限公司 A kind of fatigue state method for early warning, device, equipment and storage medium
CN113378851A (en) * 2020-02-25 2021-09-10 阿里巴巴集团控股有限公司 Visual recognition method and device for image data, storage medium and processor
CN111738337B (en) * 2020-06-23 2022-03-25 吉林大学 Driver distraction state detection and identification method in mixed traffic environment
CN113688822A (en) * 2021-09-07 2021-11-23 河南工业大学 Time sequence attention mechanism scene image identification method
CN114267206A (en) * 2021-12-28 2022-04-01 上汽大众汽车有限公司 Security alarm method, security alarm device, security alarm system, and computer-readable storage medium
CN115071725A (en) * 2022-08-02 2022-09-20 广东车卫士信息科技有限公司 Driving behavior analysis method and device

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107038221A (en) * 2017-03-22 2017-08-11 杭州电子科技大学 A kind of video content description method guided based on semantic information
CN107330362A (en) * 2017-05-25 2017-11-07 北京大学 A kind of video classification methods based on space-time notice
CN107563498A (en) * 2017-09-08 2018-01-09 中国石油大学(华东) View-based access control model is combined the Image Description Methods and system of strategy with semantic notice
CN107608943A (en) * 2017-09-08 2018-01-19 中国石油大学(华东) Merge visual attention and the image method for generating captions and system of semantic notice

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10268671B2 (en) * 2015-12-31 2019-04-23 Google Llc Generating parse trees of text segments using neural networks

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107038221A (en) * 2017-03-22 2017-08-11 杭州电子科技大学 A kind of video content description method guided based on semantic information
CN107330362A (en) * 2017-05-25 2017-11-07 北京大学 A kind of video classification methods based on space-time notice
CN107563498A (en) * 2017-09-08 2018-01-09 中国石油大学(华东) View-based access control model is combined the Image Description Methods and system of strategy with semantic notice
CN107608943A (en) * 2017-09-08 2018-01-19 中国石油大学(华东) Merge visual attention and the image method for generating captions and system of semantic notice

Also Published As

Publication number Publication date
CN108446645A (en) 2018-08-24

Similar Documents

Publication Publication Date Title
CN108446645B (en) Vehicle-mounted face recognition method based on deep learning
CN109460704B (en) Fatigue detection method and system based on deep learning and computer equipment
Anagnostopoulos et al. A license plate-recognition algorithm for intelligent transportation system applications
CN111414813A (en) Dangerous driving behavior identification method, device, equipment and storage medium
Ou et al. Enhancing driver distraction recognition using generative adversarial networks
CN105286802B (en) Driver Fatigue Detection based on video information
KR102453627B1 (en) Deep Learning based Traffic Flow Analysis Method and System
Ragab et al. A visual-based driver distraction recognition and detection using random forest
CN108009548A (en) A kind of Intelligent road sign recognition methods and system
Potdar et al. A convolutional neural network based live object recognition system as blind aid
CN111178272A (en) Method, device and equipment for identifying driver behavior
CN111274980A (en) Small-size traffic sign identification method based on YOLOV3 and asymmetric convolution
CN108108651B (en) Method and system for detecting driver non-attentive driving based on video face analysis
CN114155512A (en) Fatigue detection method and system based on multi-feature fusion of 3D convolutional network
CN115376108A (en) Obstacle detection method and device in complex weather
CN114429126A (en) True and false message identification method based on reinforcement learning and affair knowledge graph
CN111626197B (en) Recognition method based on human behavior recognition network model
Vaca-Recalde et al. Driver monitoring system based on cnn models: An approach for attention level detection
CN110826396B (en) Method and device for detecting eye state in video
CN107392246A (en) A kind of background modeling method of feature based model to background model distance
CN116052189A (en) Text recognition method, system and storage medium
CN112329566A (en) Visual perception system for accurately perceiving head movements of motor vehicle driver
Senarathne et al. Blindaid-android-based mobile application guide for visually challenged people
Hu et al. Comprehensive driver state recognition based on deep learning and PERCLOS criterion
Feng et al. A distracted driving discrimination method based on the facial feature triangle and Bayesian network

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant