CN113569817B - Driver attention dispersion detection method based on image area positioning mechanism - Google Patents

Driver attention dispersion detection method based on image area positioning mechanism Download PDF

Info

Publication number
CN113569817B
CN113569817B CN202111110059.2A CN202111110059A CN113569817B CN 113569817 B CN113569817 B CN 113569817B CN 202111110059 A CN202111110059 A CN 202111110059A CN 113569817 B CN113569817 B CN 113569817B
Authority
CN
China
Prior art keywords
driver
image
state
neural network
key
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202111110059.2A
Other languages
Chinese (zh)
Other versions
CN113569817A (en
Inventor
赵磊
孙浩然
罗映
徐楠
***
闫法义
贝太学
李新海
张宗喜
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shandong Jianzhu University
Original Assignee
Shandong Jianzhu University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shandong Jianzhu University filed Critical Shandong Jianzhu University
Priority to CN202111110059.2A priority Critical patent/CN113569817B/en
Publication of CN113569817A publication Critical patent/CN113569817A/en
Application granted granted Critical
Publication of CN113569817B publication Critical patent/CN113569817B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Biophysics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Probability & Statistics with Applications (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Evolutionary Biology (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Image Analysis (AREA)

Abstract

A driver attention dispersion detection method based on an image area positioning mechanism is characterized in that areas needing attention under different behavior states of a driver in an image are obtained through a manual calibration method, the areas are combined with neural network activation mapping, a model optimization function based on area enhancement driving is established, the neural network is trained through the optimization function, so that a detection model can automatically obtain key areas in the image of the driver according to different behavior characteristics of the driver in the detection process, the problem of automatic extraction of key positions and features in a driver behavior detection method based on image information is solved, and the detection precision of the model is improved.

Description

Driver attention dispersion detection method based on image area positioning mechanism
Technical Field
The invention relates to the technical field of driver state recognition, in particular to a driver attention dispersion detection method based on an image area positioning mechanism.
Background
With the development of science and technology, intelligent electronic devices such as smart phones, tablet computers and vehicle information systems greatly improve the probability of driver distraction, easily generate potential safety hazards to cause traffic accidents and harm life and property safety. Statistically, nearly 125 million people die of traffic accidents each year. Nearly one fifth of accidents are caused by distractions of the driver. With the progress of artificial intelligence technology, the automatic driving technology is rapidly developed. However, current conditional autonomous driving systems still require a driver who is ready to take over in time. The american national traffic safety committee counted 37 car accidents in 18 months for the Uber autopilot test car between 2018 and 2019. Therefore, the accurate and effective driver distraction behavior detection system is designed to have important significance for improving traffic safety.
The driver distraction detection methods can be classified into the following three categories: based on the driver physiological information, the driving operation information, and the visual information. When the mental state of a driver changes, the physiological signal of the driver also changes, however, most of the physiological acquisition sensors need to be worn to the corresponding position of the body of the driver, and the driving experience is influenced. The driver state identification method based on the operation behaviors mainly utilizes the driver to acquire operation information of a steering wheel, an accelerator and a brake pedal, analyzes the driving behaviors of the driver in different states and presumes whether the driver is in a dangerous driving state. However, the recognition accuracy of the method is often influenced by the operating habits, skills, traffic road conditions and other factors of the driver. The vision-based detection method can non-invasively extract visual image information of the driver and is not affected by external interference. Therefore, the visual characteristics are the most widely used information in the driver distraction detection method. Vision-based detection methods can be divided into two categories: the first method directly classifies the original image to detect the state and behavior of the driver, and the method is often interfered by other factors in the image besides the driver in the image; the second method utilizes a target detection or image segmentation model to extract key areas or features such as hands, heads, upper bodies and the like from a driver image, and then inputs the extracted information into an identification model to obtain a detection result, however, the positioning of the areas or the features is often limited by the accuracy of an algorithm and is often subjected to false detection.
Disclosure of Invention
In order to overcome the defects of the technologies, the invention provides the driver attention dispersion detection method based on the image area positioning mechanism, solves the problem of automatic extraction of key positions and features in the driver behavior detection method based on the image information on the premise of not increasing the complexity of the model, and improves the detection precision of the model.
The technical scheme adopted by the invention for overcoming the technical problems is as follows:
a driver distraction detection method based on an image area positioning mechanism comprises the following steps:
a) acquiring visual images of different behaviors of a driver, and determining key areas needing attention in the different behavior states in an automatic positioning and manual adjusting mode according to the different behavior states of the driver in each visual image;
b) establishing a probability heat map of a key area in a visual image of a driver by using a Gaussian model, and establishing a driver behavior detection data set based on area positioning;
c) establishing a neural network model, constructing cost functions driven by class activation mapping and key region probability heat maps, and training a neural network by using the cost functions to obtain an optimized neural network model;
d) and installing a camera in the vehicle, acquiring a real-time image of the side part of the driver, inputting the image into the optimized neural network model, and extracting the output probability of the model to obtain the behavior state of the driver.
Further, a camera is installed in the vehicle in the step a), videos of different behaviors of the driver are collected through the camera, the videos are converted into visual images frame by frame, and the visual images are stored to obtain sample images.
Further, the behavior states of the driver in step a) are respectively defined as: a normal driving state, a state of using a smart phone or a tablet computer, a calling state, a conversation with a co-driver state, a drinking state and an operation center control electronic equipment state; when the driver is in a normal driving state, key areas needing attention are positioned on the hands and the upper arms of the driver in the visual image; when the driver is in a state of using the smart phone or the tablet personal computer, the key area needing attention is located in the mobile phone or the tablet personal computer of the hand of the driver in the visual image; when the driver is in a calling state, the key area needing attention is positioned at the mouth and the mobile phone position of the driver in the visual image; when the driver is in a conversation state with the co-driver, key areas needing attention are positioned at the mouth and the face of the driver in the visual image; when the driver is in a drinking state, the key area needing attention is positioned in a container held by the driver in the visual image; when the driver is in the state of operating the central control electronic equipment, the key area needing attention is positioned at the hand of the driver and the central control equipment in the visual image.
Further, the step of step a) comprises:
a-1) finding out the limb movement area of a driver in the process of executing different behaviors in a sample image, and establishing a key area based on different behavior states in the driver image;
a-2) based on the established key area, automatically acquiring the position information of an upper arm skeleton point and a head skeleton point of a driver in a sample image by a skeleton point positioning method, drawing a rectangular frame based on the upper arm skeleton point and the head skeleton point, wherein the skeleton point is positioned at the center of the rectangular frame, and obtaining the initial position of the key area of the image;
and a-3) manually correcting to obtain a final key area of the image according to the position and the size of the rectangular frame.
Further, step b) comprises the following steps:
b-1) based on the key region, by formula
Figure GDA0003358225130000021
Establishing a two-dimensional Gaussian model d, wherein Z is a normalization factor, sigma is a covariance matrix, T is transposition, p is a variable of the two-dimensional Gaussian model,
Figure GDA0003358225130000031
for key locations in the driver behavior image,
Figure GDA0003358225130000032
x is the abscissa of the key area, y is the ordinate of the key area, xmaxIs the maximum value of x, xminIs the minimum value of x, ymaxIs the maximum value of y, yminIs the minimum value of y;
b-2) converting the two-dimensional Gaussian model d into a two-dimensional image to obtain a probability heat map of a key area in the visual image of the driver;
b-3) traversing all image samples in the driver behavior detection data set based on the area positioning, and repeatedly executing the steps b-1) to b-2), and storing probability heat maps of key areas in the visual images of all drivers to obtain the driver behavior detection data set based on the area positioning.
Further, step c) comprises the steps of:
c-1) establishing a ResNeXt neural network model, and adopting a global average pooling layer on the top layer of the neural network;
c-2) establishing a SoftMax classifier at the top layer of the global pooling layer to output a driver behavior prediction probability value;
c-3) by the formula
Figure GDA0003358225130000033
Calculating the class activation mapping of each driver behavior state category output by the top layer of the neural network, wherein k is the number of top layer neurons, and W is the number of the top layer neuronsk,cIs a top-level weight parameter, fk(x, y) is a mapping value of a previous layer of the global mean pooling layer;
c-4) extracting the driver behavior prediction probability value and the class activation mapping of the neural network model through a formula
Figure GDA0003358225130000034
Calculating to obtain a region enhanced optimization function LEWherein F is a nonlinear transformation function, ACClass activation mapping for class C behavioral State class of neural networks, AC*Activating mappings for class C of the same class as the true behavioral State class, C*Being of the same class as the true category of behaviour, λFIs coefficient, < > is matrix Hadamard product, HC*Activating a mapping for a predefined class C;
c-5) by the formula L ═ LG+kELECalculating a cost function L, LGFor the cost function based on the driver state value,
Figure GDA0003358225130000035
kEis a coefficient of，
Figure GDA0003358225130000036
For the output values of the resenext neural network model,
Figure GDA0003358225130000037
is a calibration value;
c-6) training the ResNeXt neural network model through a cost function L until convergence, and establishing the hyper-parameters of the model through cross validation.
Further, a camera is arranged at the position of the roof above the right of the driver in the step d).
Further, step d) comprises the following steps:
d-1) reading the ResNeXt neural network model trained in c-6) as a detection model;
d-2) inputting each frame of image of the driver acquired by the camera into the detection model;
d-3) obtaining the prediction probability value in a SoftMax classifier at the top layer of the ResNeXt neural network model, and identifying the current behavior state of the driver.
The invention has the beneficial effects that: the method comprises the steps of obtaining regions needing attention of a driver in different behavior states in an image through a manual calibration method, combining the regions with neural network type activation mapping, establishing a model optimization function based on region enhancement driving, training a neural network through the optimization function, enabling a detection model to automatically obtain key regions in the image of the driver according to different behavior characteristics of the driver in the detection process, solving the problem of automatic extraction of key features and positions in a detection method based on visual features, and improving the identification precision of the model.
Drawings
FIG. 1 is a flow chart of a method of the present invention;
FIG. 2 is a flow chart of an embodiment of the present invention.
Detailed Description
The invention will be further explained with reference to fig. 1 and 2.
As shown in the attached drawings, a method for detecting distraction of a driver based on an image area positioning mechanism comprises the following steps:
a) the method comprises the steps of collecting visual images of different behaviors of a driver, and determining key areas needing attention in the different behavior states in an automatic positioning and manual adjusting mode according to different behavior states of the driver in each visual image.
b) And establishing a probability heat map of a key area in a visual image of the driver by using a Gaussian model, and establishing a driver behavior detection data set based on area positioning.
c) Establishing a neural network model, constructing cost functions driven by class activation mapping and key region probability heat maps, and training a neural network by using the cost functions to obtain the optimized neural network model.
d) And installing a camera in the vehicle, acquiring a real-time image of the side part of the driver, inputting the image into the optimized neural network model, and extracting the output probability of the model to obtain the behavior state of the driver.
As shown in the attached figure 2, the regions needing attention in different behavior states of the driver in the image are obtained through a manual calibration method, the regions are combined with neural network activation mapping, a model optimization function based on region enhancement driving is established, the neural network is trained through the optimization function, so that the detection model can automatically obtain key regions in the image of the driver according to different behavior characteristics of the driver in the detection process, the problem of automatic extraction of key features and positions in the detection method based on visual features is solved, and the identification precision of the model is improved.
Specifically, a camera is installed in the vehicle in the step a), videos of different behaviors of the driver are collected through the camera, the videos are converted into visual images frame by frame, and the visual images are stored to obtain sample images.
Specifically, the behavior states of the driver in step a) are respectively defined as: a normal driving state, a state of using a smart phone or a tablet computer, a calling state, a conversation with a co-driver state, a drinking state and an operation center control electronic equipment state; when the driver is in a normal driving state, key areas needing attention are positioned on the hands and the upper arms of the driver in the visual image; when the driver is in a state of using the smart phone or the tablet personal computer, the key area needing attention is located in the mobile phone or the tablet personal computer of the hand of the driver in the visual image; when the driver is in a calling state, the key area needing attention is positioned at the mouth and the mobile phone position of the driver in the visual image; when the driver is in a conversation state with the co-driver, key areas needing attention are positioned at the mouth and the face of the driver in the visual image; when the driver is in a drinking state, the key area needing attention is positioned in a container held by the driver in the visual image; when the driver is in the state of operating the central control electronic equipment, the key area needing attention is positioned at the hand of the driver and the central control equipment in the visual image.
Specifically, the step a) is as follows:
a-1) finding out the limb movement area of the driver in the process of executing different behaviors in the sample image, and establishing key areas based on different behavior states in the driver image.
a-2) based on the established key area, automatically obtaining the position information of the upper arm skeleton point and the head skeleton point of the driver in the sample image by a skeleton point positioning method, drawing a rectangular frame based on the upper arm skeleton point and the head skeleton point, and obtaining the initial position of the key area of the image, wherein the skeleton point is positioned at the center of the rectangular frame. Let the size of the image be
Figure GDA0003358225130000051
The dimensions of the initial rectangular box are set as follows: the height of a rectangular frame in the normal driving image is h/5, and the width of the rectangular frame in the normal driving image is b/4; the height of a rectangular frame of a driver in an image using a smart phone or a tablet computer is h/4, and the width of the rectangular frame is b/5; the height of a rectangular frame in a calling image of a driver is h/4, and the width of the rectangular frame is b/5; the height of the rectangular frame of the image of the driver in the conversation with the copilot is h/3, and the width of the rectangular frame of the driver in the image of the conversation with the copilot is b/5; the height of a rectangular frame of the driver in the drinking image is h/4, and the width of the rectangular frame is b/5; the rectangular frame in the driving operation central control electronic equipment image is h/2 in height and b/5 in width.
a-3) manually correcting to obtain a rectangular frame of which the image finally contains the key area according to the position and the size of the rectangular frame. Preferably, the ratio of the finally corrected rectangular frame in the key area is equal to or greater than 90% and the rectangular frame is equal to or less than 1/2 of the sample image. Based on the above principle, the size range of the final rectangular frame after manual correction is as follows: the range of the rectangular box height in the normal driving image is: h/9-h/7, and the wide range is b/6-b/3; the range of the height of the rectangular frame of the image of the driver using the smart phone or the tablet computer is as follows: h/6-h/2, and the wide range is b/6-b/3; the range of the height of the rectangular frame in the image of the driver making a call is: h/6-h/4, and the wide range is b/8-b/6; the range of the height of the rectangular frame in the image of the driver talking with the co-driver is: h/4-h/3, and the wide range is b/6-b/5; the range of the height of the rectangular frame of the driver in the drinking image is as follows: h/7-h/3, and the wide range is b/8-b/5; the range of the height of the rectangular frame in the driving operation central control electronic equipment image is as follows: h/3-h/2, and the wide range is b/6-b/4.
Specifically, the step b) comprises the following steps:
b-1) based on the key region, by formula
Figure GDA0003358225130000061
Establishing a two-dimensional Gaussian model d, wherein Z is a normalization factor, sigma is a covariance matrix, T is transposition, p is a variable of the two-dimensional Gaussian model,
Figure GDA0003358225130000062
for key locations in the driver behavior image,
Figure GDA0003358225130000063
x is the abscissa of the key area, y is the ordinate of the key area, xmaxIs the maximum value of x, xminIs the minimum value of x, ymaxIs the maximum value of y, yminIs the minimum value in y.
b-2) converting the two-dimensional Gaussian model d into a two-dimensional image to obtain a probability heat map of a key area in the visual image of the driver.
b-3) traversing all image samples in the driver behavior detection data set based on the area positioning, and repeatedly executing the steps b-1) to b-2), and storing probability heat maps of key areas in the visual images of all drivers to obtain the driver behavior detection data set based on the area positioning.
Specifically, the step c) comprises the following steps:
c-1) establishing a ResNeXt neural network model with the network layer number of 50, and adopting a global average pooling layer at the top layer of the neural network.
And c-2) establishing a SoftMax classifier at the top layer of the global pooling layer to output the predicted probability value of the driver behavior.
c-3) by the formula
Figure GDA0003358225130000064
Calculating the class activation mapping of each driver behavior state category output by the top layer of the neural network, wherein k is the number of top layer neurons, and W is the number of the top layer neuronsk,cIs a top-level weight parameter, fkAnd (x, y) is the mapping value of the previous layer of the global mean pooling layer.
c-4) extracting the driver behavior prediction probability value and the class activation mapping of the neural network model through a formula
Figure GDA0003358225130000065
Calculating to obtain a region enhanced optimization function LEWherein F is a nonlinear transformation function, ACClass activation mapping for class C behavioral State class of neural networks, AC*Activating mappings for class C of the same class as the true behavioral State class, C*Being of the same class as the true category of behaviour, λFIs coefficient, < > is matrix Hadamard product, HC*The mapping is activated for a predefined class C.
c-5) by the formula L ═ LG+kELECalculating a cost function L, LGFor a cost function based on the driver state value, the cost function for the conventional ResNeXt network is LG，
Figure GDA0003358225130000066
kEAs a function of the number of the coefficients,
Figure GDA0003358225130000067
for the output values of the resenext neural network model,
Figure GDA0003358225130000071
is a calibrated value.
c-6) training the ResNeXt neural network model through a cost function L until convergence, and establishing the hyper-parameters of the model through cross validation. The final defined hyper-parameters are mainly: the neural network trains the learning rate, the number of batch training samples (batch size), the coefficients in the loss function, the number of batch samples, the momentum parameter β of the momentum optimizer.
Preferably, in this patent, the learning rate r of neural network training is 0.001, and the number of batch training samples: batchsize 32, coefficient k in loss functionEThe momentum parameter β of the momentum optimizer is 0.9.
Preferably, a camera is installed at the position of the roof right above the driver in the step d).
Specifically, the step d) comprises the following steps:
d-1) reading the trained ResNeXt neural network model in c-6) as a detection model.
d-2) inputting each frame of image of the driver acquired by the camera into the detection model.
d-3) obtaining the prediction probability value in a SoftMax classifier at the top layer of the ResNeXt neural network model, and identifying the current behavior state of the driver.
In order to verify that the detection precision is improved by the driver distraction detection method based on the image area positioning mechanism, a driver behavior data set is constructed through a real-vehicle experiment, and the data set comprises 12688 driver images of 6 behaviors including a normal driving state, a state of using a smart phone or a tablet personal computer, a state of making a call, a state of talking with a co-driver, a water drinking state and a state of operating a central control electronic device, wherein 40 drivers, 10 females among 40 drivers and 30 males. If the data set is input into the ResNeXt model trained by the traditional training method, the model recognition accuracy is only 89.75% taking the ResNeXt model with 50 layers as an example. If the data set is input into the ResNeXt model trained by the invention, the model identification accuracy can reach 95.59% by taking a 50-layer ResNeXt model as an example.
In order to verify the accuracy of the driver distraction detection method based on the image area positioning mechanism in the patent, a driver behavior data set is constructed through a real-vehicle experiment, and the data set comprises 12688 driver images of 6 behaviors including a normal driving state, a state of using a smart phone or a tablet computer, a state of making a call, a state of talking with a copilot, a water drinking state and a state of operating central control electronic equipment, wherein 40 drivers, 10 females and 30 males among 40 drivers. The experiment takes a ResNeXt model with 50 layers as an example, training and verification are carried out by relying on the data set, the experimental result is shown in table 1, and C0-C5 in table 1 respectively represent 6 behaviors of a normal driving state, a state of using a smart phone or a tablet computer, a state of calling, a state of talking with a co-driver, a state of drinking water and a state of operating central control electronic equipment. Compared with the traditional training method, the recognition accuracy of the ResNeXt model can be effectively improved by the training method provided by the patent through experimental results.
TABLE 1
Figure GDA0003358225130000081
Finally, it should be noted that: although the present invention has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that changes may be made in the embodiments and/or equivalents thereof without departing from the spirit and scope of the invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (7)

1. A driver distraction detection method based on an image area positioning mechanism is characterized by comprising the following steps:
a) acquiring visual images of different behaviors of a driver, and determining key areas needing attention in the different behavior states in an automatic positioning and manual adjusting mode according to the different behavior states of the driver in each visual image;
b) establishing a probability heat map of a key area in a visual image of a driver by using a Gaussian model, and establishing a driver behavior detection data set based on area positioning;
c) establishing a neural network model, constructing cost functions driven by class activation mapping and key region probability heat maps, and training a neural network by using the cost functions to obtain an optimized neural network model;
d) installing a camera in the vehicle, acquiring a real-time image of the side part of the driver, inputting the image into the optimized neural network model, and extracting the output probability of the model to obtain the behavior state of the driver;
the step c) comprises the following steps:
c-1) establishing a ResNeXt neural network model, and adopting a global average pooling layer on the top layer of the neural network;
c-2) establishing a SoftMax classifier at the top layer of the global pooling layer to output a driver behavior prediction probability value;
c-3) by the formula
Figure FDA0003358225120000011
Calculating the class activation mapping of each driver behavior state category output by the top layer of the neural network, wherein k is the number of top layer neurons, and W is the number of the top layer neuronsk,cIs a top-level weight parameter, fk(x, y) is a mapping value of a previous layer of the global mean pooling layer;
c-4) extracting the driver behavior prediction probability value and the class activation mapping of the neural network model through a formula
Figure FDA0003358225120000012
Calculating to obtain a region enhanced optimization function LEWherein F is a nonlinear transformation function, ACClass activation mapping for a class C behavioral state class of the neural network,
Figure FDA0003358225120000013
as being of the same kind as the true behavioral state categoryClass C activation mapping of, C*Being of the same class as the true category of behaviour, λFA coefficient, an h is a matrix hadamard product,
Figure FDA0003358225120000017
activating a mapping for a predefined class C;
c-5) by the formula L ═ LG+kELECalculating a cost function L, LGFor the cost function based on the driver state value,
Figure FDA0003358225120000014
kEas a function of the number of the coefficients,
Figure FDA0003358225120000015
for the output values of the resenext neural network model,
Figure FDA0003358225120000016
is a calibration value;
c-6) training the ResNeXt neural network model through a cost function L until convergence, and establishing the hyper-parameters of the model through cross validation.
2. The method for detecting the distraction of the driver based on the image area localization mechanism according to claim 1, wherein: in the step a), a camera is installed in the vehicle, videos of different behaviors of a driver are collected through the camera, the videos are converted into visual images frame by frame, and the visual images are stored to obtain sample images.
3. The method for detecting the distraction of the driver based on the image area positioning mechanism according to claim 1, wherein the behavior states of the driver in the step a) are respectively defined as: a normal driving state, a state of using a smart phone or a tablet computer, a calling state, a conversation with a co-driver state, a drinking state and an operation center control electronic equipment state; when the driver is in a normal driving state, key areas needing attention are positioned on the hands and the upper arms of the driver in the visual image; when the driver is in a state of using the smart phone or the tablet personal computer, the key area needing attention is located in the mobile phone or the tablet personal computer of the hand of the driver in the visual image; when the driver is in a calling state, the key area needing attention is positioned at the mouth and the mobile phone position of the driver in the visual image; when the driver is in a conversation state with the co-driver, key areas needing attention are positioned at the mouth and the face of the driver in the visual image; when the driver is in a drinking state, the key area needing attention is positioned in a container held by the driver in the visual image; when the driver is in the state of operating the central control electronic equipment, the key area needing attention is positioned at the hand of the driver and the central control equipment in the visual image.
4. The method for detecting the distraction of the driver based on the image area positioning mechanism according to claim 3, wherein the step a) comprises:
a-1) finding out the limb movement area of a driver in the process of executing different behaviors in a sample image, and establishing a key area based on different behavior states in the driver image;
a-2) based on the established key area, automatically acquiring the position information of an upper arm skeleton point and a head skeleton point of a driver in a sample image by a skeleton point positioning method, drawing a rectangular frame based on the upper arm skeleton point and the head skeleton point, wherein the skeleton point is positioned at the center of the rectangular frame, and obtaining the initial position of the key area of the image;
a-3) manually correcting to obtain a rectangular frame of which the image finally contains the key area according to the position and the size of the rectangular frame.
5. The method for detecting the distraction of the driver based on the image area positioning mechanism according to claim 1, wherein the step b) comprises the following steps:
b-1) based on the key region, by formula
Figure FDA0003358225120000021
Establishing a two-dimensional Gaussian model d, wherein Z is a normalization factor and sigma is a covariance matrixThe matrix, T is the transpose, p is the variable of the two-dimensional Gaussian model,
Figure FDA0003358225120000022
for key locations in the driver behavior image,
Figure FDA0003358225120000023
x is the abscissa of the key area, y is the ordinate of the key area, xmaxIs the maximum value of x, xminIs the minimum value of x, ymaxIs the maximum value of y, yminIs the minimum value of y;
b-2) converting the two-dimensional Gaussian model d into a two-dimensional image to obtain a probability heat map of a key area in the visual image of the driver;
b-3) traversing all image samples in the driver behavior detection data set based on the area positioning, and repeatedly executing the steps b-1) to b-2), and storing probability heat maps of key areas in the visual images of all drivers to obtain the driver behavior detection data set based on the area positioning.
6. The method for detecting the distraction of the driver based on the image area localization mechanism according to claim 1, wherein: and d) mounting a camera at the position of the roof above the right of the driver.
7. The method for detecting the distraction of the driver based on the image area positioning mechanism according to claim 1, wherein the step d) comprises the following steps:
d-1) reading the ResNeXt neural network model trained in c-6) as a detection model;
d-2) inputting each frame of image of the driver acquired by the camera into the detection model;
d-3) obtaining the prediction probability value in a SoftMax classifier at the top layer of the ResNeXt neural network model, and identifying the current behavior state of the driver.
CN202111110059.2A 2021-09-23 2021-09-23 Driver attention dispersion detection method based on image area positioning mechanism Active CN113569817B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111110059.2A CN113569817B (en) 2021-09-23 2021-09-23 Driver attention dispersion detection method based on image area positioning mechanism

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111110059.2A CN113569817B (en) 2021-09-23 2021-09-23 Driver attention dispersion detection method based on image area positioning mechanism

Publications (2)

Publication Number Publication Date
CN113569817A CN113569817A (en) 2021-10-29
CN113569817B true CN113569817B (en) 2021-12-21

Family

ID=78173969

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111110059.2A Active CN113569817B (en) 2021-09-23 2021-09-23 Driver attention dispersion detection method based on image area positioning mechanism

Country Status (1)

Country Link
CN (1) CN113569817B (en)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109711463A (en) * 2018-12-25 2019-05-03 广东顺德西安交通大学研究院 Important object detection method based on attention
CN111914902A (en) * 2020-07-08 2020-11-10 南京航空航天大学 Traditional Chinese medicine identification and surface defect detection method based on deep neural network

Family Cites Families (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6186820B2 (en) * 2013-04-10 2017-08-30 株式会社デンソー Sleepiness prediction device and sleepiness prediction system
CN105809152B (en) * 2016-04-06 2019-05-21 清华大学 A kind of driver's cognition based on Multi-source Information Fusion is divert one's attention monitoring method
CN106778677A (en) * 2016-12-30 2017-05-31 东北农业大学 Feature based selection and driver's fatigue state recognition method and device of facial multizone combining classifiers
CN108227912B (en) * 2017-11-30 2021-05-11 北京市商汤科技开发有限公司 Device control method and apparatus, electronic device, computer storage medium
CN108960065B (en) * 2018-06-01 2020-11-17 浙江零跑科技有限公司 Driving behavior detection method based on vision
CN108985259B (en) * 2018-08-03 2022-03-18 百度在线网络技术（北京）有限公司 Human body action recognition method and device
CN109583338A (en) * 2018-11-19 2019-04-05 山东派蒙机电技术有限公司 Driver Vision decentralized detection method based on depth integration neural network
CN110119676B (en) * 2019-03-28 2023-02-03 广东工业大学 Driver fatigue detection method based on neural network
CN110298257B (en) * 2019-06-04 2023-08-01 东南大学 Driver behavior recognition method based on human body multi-part characteristics
WO2020122986A1 (en) * 2019-06-10 2020-06-18 Huawei Technologies Co.Ltd. Driver attention detection using heat maps
CN110288597B (en) * 2019-07-01 2021-04-02 哈尔滨工业大学 Attention mechanism-based wireless capsule endoscope video saliency detection method
CN110728185B (en) * 2019-09-10 2023-04-07 西安工业大学 Detection method for judging existence of handheld mobile phone conversation behavior of driver
CN113051958A (en) * 2019-12-26 2021-06-29 深圳市光鉴科技有限公司 Driver state detection method, system, device and medium based on deep learning
CN111563468B (en) * 2020-05-13 2023-04-07 电子科技大学 Driver abnormal behavior detection method based on attention of neural network
CN111860525B (en) * 2020-08-06 2022-10-21 宁夏宁电电力设计有限公司 Bottom-up optical character recognition method suitable for terminal block
CN112069988A (en) * 2020-09-04 2020-12-11 徐尔灵 Gun-ball linkage-based driver safe driving behavior detection method
CN112241679B (en) * 2020-09-14 2024-02-20 浙江理工大学 Automatic garbage classification method
CN112419670A (en) * 2020-09-15 2021-02-26 深圳市点创科技有限公司 Method, device and medium for detecting fatigue driving of driver by fusing key point positioning and image classification
CN112418261B (en) * 2020-09-17 2022-05-03 电子科技大学 Human body image multi-attribute classification method based on prior prototype attention mechanism
CN113313199A (en) * 2021-06-21 2021-08-27 北京工业大学 Brain CT medical report automatic generation method based on weak supervision attention

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109711463A (en) * 2018-12-25 2019-05-03 广东顺德西安交通大学研究院 Important object detection method based on attention
CN111914902A (en) * 2020-07-08 2020-11-10 南京航空航天大学 Traditional Chinese medicine identification and surface defect detection method based on deep neural network

Also Published As

Publication number Publication date
CN113569817A (en) 2021-10-29

Similar Documents

Publication Publication Date Title
CN111723748B (en) Infrared remote sensing image ship detection method
CN110059582B (en) Driver behavior identification method based on multi-scale attention convolution neural network
CN108875674B (en) Driver behavior identification method based on multi-column fusion convolutional neural network
CN110363182B (en) Deep learning-based lane line detection method
CN111460919B (en) Monocular vision road target detection and distance estimation method based on improved YOLOv3
CN107784587B (en) Driving behavior evaluation system
CN106407931B (en) A kind of depth convolutional neural networks moving vehicle detection method
CN106228125B (en) Method for detecting lane lines based on integrated study cascade classifier
CN101859382B (en) License plate detection and identification method based on maximum stable extremal region
CN107133974B (en) Gaussian Background models the vehicle type classification method combined with Recognition with Recurrent Neural Network
CN109345770A (en) A kind of child leaves in-vehicle alarm system and child leaves interior alarm method
CN109871799B (en) Method for detecting mobile phone playing behavior of driver based on deep learning
CN113065578B (en) Image visual semantic segmentation method based on double-path region attention coding and decoding
CN108694408B (en) Driving behavior recognition method based on deep sparse filtering convolutional neural network
CN108803623B (en) Method for personalized driving of automatic driving vehicle and system for legalization of driving
CN111753674A (en) Fatigue driving detection and identification method based on deep learning
CN111256693B (en) Pose change calculation method and vehicle-mounted terminal
CN110261329A (en) A kind of Minerals identification method based on full spectral coverage high-spectrum remote sensing data
CN111325146A (en) Truck type and axle type identification method and system
CN110363093A (en) A kind of driver&#39;s action identification method and device
CN208969813U (en) A kind of child leaves in-vehicle alarm system
CN112651326B (en) Driver hand detection method and system based on deep learning
CN113313320B (en) Vehicle track prediction method based on residual attention mechanism
CN112674998A (en) Blind person traffic intersection assisting method based on rapid deep neural network and mobile intelligent device
CN111738114A (en) Vehicle target detection method based on anchor-free accurate sampling remote sensing image

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant