CN117392642A - Driver multi-position detection method and device, electronic control unit and vehicle - Google Patents

Driver multi-position detection method and device, electronic control unit and vehicle Download PDF

Info

Publication number
CN117392642A
CN117392642A CN202210784656.1A CN202210784656A CN117392642A CN 117392642 A CN117392642 A CN 117392642A CN 202210784656 A CN202210784656 A CN 202210784656A CN 117392642 A CN117392642 A CN 117392642A
Authority
CN
China
Prior art keywords
face
driver
upper body
detection
area
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210784656.1A
Other languages
Chinese (zh)
Inventor
林枝叶
胡束芒
赵龙
陈现岭
贾澜鹏
王光甫
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Great Wall Motor Co Ltd
Original Assignee
Great Wall Motor Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Great Wall Motor Co Ltd filed Critical Great Wall Motor Co Ltd
Priority to CN202210784656.1A priority Critical patent/CN117392642A/en
Publication of CN117392642A publication Critical patent/CN117392642A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
    • G06V20/597—Recognising the driver's state or behaviour, e.g. attention or drowsiness
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/107—Static hand or arm
    • G06V40/11—Hand-related biometrics; Hand pose recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Databases & Information Systems (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Human Computer Interaction (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Image Analysis (AREA)

Abstract

The application is applicable to the technical field of intelligent traffic safety and provides a driver multi-position detection method, a device, an electronic control unit and a vehicle. The method comprises the following steps: acquiring a driver image; performing target position detection on the driver image, and determining a target area, wherein the target position detection comprises face detection and hand detection, and the target area comprises any one or more of the face area and the hand area; determining an upper body detection area according to the target area, and determining an upper body posture of the driver based on the upper body of the driver detected by the upper body detection area; and determining an optimal target area according to the upper body posture and the target area, and detecting a target part of the driver based on the optimal target area, wherein the target part comprises any one or more of a human face and a hand. The method and the device can avoid the problem of false detection or inaccurate detection of the face or the hand, and improve the detection precision of multi-part detection.

Description

Driver multi-position detection method and device, electronic control unit and vehicle
Technical Field
The application relates to the technical field of intelligent traffic safety, in particular to a driver multi-position detection method, a device, an electronic control unit and a vehicle.
Background
With the continuous development of automobile technology, automobile cabins are becoming more and more intelligent. A driver monitoring system (Driver Monitor System, DMS for short) may analyze driver attributes and behavior for fatigue, distraction, and dangerous behavior monitoring of the driver.
When the DMS monitors fatigue and distraction of a driver, it is necessary to detect a face of the driver to determine whether the driver is tired or distracted, and when the DMS monitors dangerous behavior of the driver, it is necessary to detect a hand of the driver to determine whether the driver has dangerous behavior, the face detection and the hand detection are simply referred to as multi-part detection.
However, due to the instability of the DMS detector, interference phenomenon between face detection and hand detection is easily caused, that is, the face detector detects the hand as a face, or the hand detector detects the face as a hand, resulting in a problem of false detection or inaccurate detection of the face or hand.
Disclosure of Invention
The embodiment of the application provides a driver multi-position detection method, a device, an electronic control unit and a vehicle, which can avoid the problem of false detection or inaccurate detection of a human face or a hand and improve the detection precision of multi-position detection.
The application is realized by the following technical scheme:
in a first aspect, an embodiment of the present application provides a method for detecting multiple parts of a driver, including:
acquiring a driver image;
performing target position detection on the driver image, and determining a target area, wherein the target position detection comprises face detection and hand detection, and the target area comprises any one or more of the face area and the hand area;
determining an upper body detection area according to the target area, and determining an upper body posture of the driver based on the upper body of the driver detected by the upper body detection area;
and determining an optimal target area according to the upper body posture and the target area, and detecting a target part of the driver based on the optimal target area, wherein the target part comprises any one or more of a human face and a hand.
In a possible implementation manner of the first aspect, determining the upper body detection region according to the target region includes:
if the face area exists, determining an upper body detection area according to the face area and a preset head-body proportional coefficient; if the face area does not exist, the driver image is set as the upper body detection area.
In a possible implementation manner of the first aspect, the DBFace network model is used to perform target location detection on the driver image, determine a target area, and perform head orientation detection on the driver image, determine a head orientation;
Determining an upper body detection area according to the face area and a preset head-body proportionality coefficient, including:
rotating the face area according to the orientation of the head so as to make the orientation of the head in the face area be positive;
and determining an upper body detection area according to the rotated face area and a preset head-body proportionality coefficient.
In a possible implementation manner of the first aspect, the DBFace network model performs head orientation detection on the driver image, and determines head orientation, including:
based on a multi-head attention mechanism, adding a head orientation detection branch after a feature extraction layer of the DBface network model, detecting the head orientation, and determining the head orientation; the head orientation includes forward, reverse, left and right.
In a possible implementation manner of the first aspect, determining the upper body detection area according to the rotated face area and the preset head-body proportionality coefficient includes:
determining the width and the height of the upper body detection area according to the width and the height of the rotated face area and a preset head-body proportionality coefficient;
the width and height of the upper body detection area are:
Width half-body ＝8×Width face +0.1×Width face
Height half-body ＝4×Height face +0.1×Height face
wherein Width half-body Height, the width of the upper body detection area half-body To the height of the upper body detection area, width face Height is the width of the rotated face region face Is the height of the rotated face region.
In one possible implementation manner of the first aspect, determining the upper body posture of the driver based on the upper body of the driver detected through the upper body detection area includes:
the method comprises the steps of detecting the upper body of a driver in an upper body detection area by adopting a Yolox-S network model;
when the upper body of the driver is detected, estimating the posture of the upper body of the driver by adopting an OpenPose network model, and determining an upper body skeleton point; the upper body skeleton point comprises a target position center point, and the target position center point comprises any one or more of a face center point and a hand center point.
In a possible implementation manner of the first aspect, determining an optimal target area according to the upper body posture and the target area, and detecting a target portion of the driver based on the optimal target area includes:
determining a face region center point according to the rotated face region, and determining a hand region center point according to the hand region;
determining an optimal face center point according to the face center point and the face region center point, and determining an optimal face region according to the optimal face center point and the face region;
And/or determining an optimal hand center point according to the hand center point and the hand region center point, and determining an optimal hand region according to the optimal hand center point and the hand region;
the face of the driver is detected based on the optimal face region and/or the hand of the driver is detected based on the optimal hand region.
In a possible implementation manner of the first aspect, the driver multi-site detection method further includes:
if the face area and the hand area are not detected, or the upper body is not detected, the driver image is re-acquired and detected.
In a second aspect, an embodiment of the present application provides a method for detecting abnormal driving behavior of a driver, including:
determining a plurality of sets of optimal target areas according to the driver multi-position detection method of any one of the first aspect, the optimal target areas including any one or more of an optimal face area and an optimal hand area;
judging whether the change trend of the face contour points included in the plurality of groups of optimal face areas accords with the fitting sample or not and whether the duration reaches the preset time or not by adopting a first deep learning model, and if so, outputting a fatigue driving signal and an alarm signal; calculating a face attitude angle according to the change trend of face contour points included in the multiple groups of optimal face areas, judging whether the face attitude angle exceeds a preset angle threshold, and if so, outputting a distraction driving signal and an alarm signal;
And/or judging whether the target object exists in the plurality of groups of optimal hand areas by adopting a second deep learning model, and judging the category of the target object when the target object exists; and outputting dangerous behavior categories and alarm signals according to the categories of the target objects.
In a third aspect, embodiments of the present application provide a driver multi-site detection apparatus, including:
the acquisition module is used for acquiring the driver image;
the detection module is used for detecting a target part of the driver image and determining a target area, wherein the target part detection comprises face detection and hand detection, and the target area comprises any one or more of the face area and the hand area;
a first determination module for determining an upper body detection area according to the target area and determining an upper body posture of the driver based on the upper body of the driver detected by the upper body detection area;
and the output module is used for determining an optimal target area according to the upper body gesture and the target area, detecting a target part of the driver based on the optimal target area, wherein the target part comprises any one or more of a human face and a hand.
In a fourth aspect, an embodiment of the present application provides a driver abnormal driving behavior detection apparatus, including:
A second determining module, configured to determine a plurality of sets of optimal target areas according to the driver multi-location detection method according to any one of the first aspects, where the optimal target areas include any one or more of an optimal face area and an optimal hand area;
the execution module is used for judging whether the change trend of the face contour points included in the plurality of groups of optimal face areas accords with the fitting sample and whether the duration reaches the preset time or not by adopting the first deep learning model, and outputting a fatigue driving signal and an alarm signal when the fitting sample accords with the duration reaching the preset time; calculating a face attitude angle according to the change trend of face contour points included in the multiple groups of optimal face areas, judging whether the face attitude angle exceeds a preset angle threshold, and outputting a distraction driving signal and an alarm signal when the face attitude angle exceeds the preset angle threshold; and/or judging whether the target object exists in the plurality of groups of optimal hand areas by adopting a second deep learning model, and judging the category of the target object when the target object exists; and outputting dangerous behavior categories and alarm signals according to the categories of the target objects.
In a fifth aspect, embodiments of the present application provide an electronic control unit, including a memory and a processor, where the memory stores a computer program executable on the processor, and the processor executes the computer program to implement the steps of the driver multi-location detection method according to any one of the first aspect and/or the steps of the driver abnormal driving behavior detection method according to the second aspect.
In a sixth aspect, embodiments of the present application provide a vehicle comprising an electronic control unit as described in the fifth aspect.
In a seventh aspect, embodiments of the present application provide a computer-readable storage medium storing a computer program, which when executed by a processor, implements the steps of the driver multi-site detection method according to any one of the first aspects, and/or the steps of the driver abnormal driving behavior detection method according to the second aspect.
In an eighth aspect, embodiments of the present application provide a computer program product, which when run on an electronic control unit, causes the electronic control unit to perform the steps of the driver multi-site detection method according to any one of the first aspects described above, and/or the steps of the driver abnormal driving behavior detection method according to the second aspect.
It will be appreciated that the advantages of the second to eighth aspects may be found in the relevant description of the first aspect, and are not repeated here.
According to the multi-position detection method, the device, the electronic control unit and the vehicle for the driver, the target position detection is carried out on the driver image by acquiring the driver image, the target area is determined, wherein the target position detection comprises face detection and hand detection, the target area comprises any one or more of the face area and the hand area, the upper body detection area is determined according to the target area, the upper body gesture of the driver is determined based on the upper body of the driver detected through the upper body detection area, the optimal target area is determined according to the upper body gesture and the target area, the target position of the driver is detected based on the optimal target area, the target position comprises any one or more of the face and the hand, the problem of false detection or inaccurate detection of the face or the hand can be avoided, and the detection precision of the multi-position detection is improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required for the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic illustration of false detection of a face and hands;
fig. 2 is a schematic application scenario of a driver multi-location detection method according to an embodiment of the present disclosure;
FIG. 3 is a flowchart of a method for detecting multiple parts of a driver according to an embodiment of the present disclosure;
FIG. 4 is a flowchart of a method for detecting multiple parts of a driver according to an embodiment of the present disclosure;
FIG. 5 is a schematic illustration of a head orientation provided in an embodiment of the present application;
FIG. 6 is a flowchart of a method for detecting multiple parts of a driver according to an embodiment of the present disclosure;
FIG. 7 is a flowchart of a method for detecting multiple parts of a driver according to an embodiment of the present disclosure;
FIG. 8 is a schematic illustration of upper body skeletal points provided in an embodiment of the present application;
FIG. 9 is a flowchart of a method for detecting multiple parts of a driver according to an embodiment of the present disclosure;
FIG. 10 is a flowchart of a method for detecting abnormal driving behavior of a driver according to an embodiment of the present disclosure;
FIG. 11 is a schematic structural view of a driver multi-location detection device according to an embodiment of the present disclosure;
fig. 12 is a schematic structural view of a driver abnormal driving behavior detection apparatus provided in an embodiment of the present application;
FIG. 13 is a schematic diagram of an electronic control unit according to an embodiment of the present disclosure;
fig. 14 is a schematic structural view of a vehicle according to an embodiment of the present application.
Detailed Description
The present application will be more clearly described with reference to the following specific examples. The following examples will assist those skilled in the art in further understanding the function of the present application, but are not intended to limit the present application in any way. It should be noted that variations and modifications could be made by those skilled in the art without departing from the spirit of the present application. These are all within the scope of the present application.
It should be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It should also be understood that the term "and/or" as used in this specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.
In the description of this application and the claims that follow, the terms "first," "second," "third," etc. are used merely to distinguish between descriptions and should not be construed to indicate or imply relative importance.
Reference in the specification to "one embodiment" or "some embodiments" or the like means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," and the like in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments" unless expressly specified otherwise. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless expressly specified otherwise.
Furthermore, references to "a plurality of" in the examples of this application should be interpreted as two or more.
With the continuous development of automobile technology, automobile cabins are becoming more and more intelligent. The driver plays an important role in the vehicle and is very important for the analysis of the attribute and the behavior of the driver. A driver monitoring system (Driver Monitor System, DMS) may analyze driver attributes and behavior to monitor whether the driver is tired, distracted, or in danger. Among them, dangerous behaviors such as smoking, making a call or eating, etc., affect the driver's attention. The DMS can perform fatigue monitoring, distraction monitoring and dangerous driving behavior monitoring on the driver, so that traffic accidents caused by bad driving behaviors of the driver are prevented.
Specifically, when the DMS monitors fatigue, the behavior of closing eyes and making yawns of the driver is sampled through the camera, and factors such as driving time and driving speed are combined to judge whether the driver is tired and the fatigue level. When the DMS monitors distraction, the sight line deviation, the face angle deviation and the like of the driver are sampled through the camera, whether the sight line or the face angle of the driver is deviated or not is judged according to the deviated angle threshold, timing is started from the triggering deviation threshold, and the distraction level is judged according to the time. When the DMS monitors dangerous driving behaviors, gestures of a driver are sampled through the camera, and whether the driver has behaviors such as smoking, calling or eating or the like is judged. And when the driver is monitored to have fatigue driving, distraction driving and dangerous driving behaviors, corresponding prompts such as buzzer alarms, voice alarms, tightening of safety belts, instrument alarms and the like are given.
From the foregoing, it is known that the DMS needs to perform face detection on the driver to determine whether the driver is tired or distracted when performing fatigue monitoring and distraction monitoring on the driver, and needs to perform hand detection on the driver to determine whether the driver has dangerous driving behavior when performing dangerous driving behavior monitoring on the driver, the face detection and the hand detection being abbreviated as multi-part detection. However, due to the instability of the DMS detector, interference phenomenon between face detection and hand detection is easily caused, that is, when a hand is close to a camera, or a face is close to a hand, a phenomenon that the face detector erroneously detects a hand as a face, or the hand detector erroneously detects a face as a hand (see fig. 1) easily occurs, resulting in a problem of erroneous detection or inaccurate detection of a face or a hand.
Based on the above problems, the inventor finds that the upper body skeleton point of the driver can be determined by detecting the upper body posture of the driver, and then the face detection and the hand detection are optimized according to the upper body skeleton point, namely the multi-part detection is optimized.
That is, in the embodiment of the present application, the target region is determined by acquiring the driver image and performing target location detection on the driver image, where the target location detection includes face detection and hand detection, the target region includes any one or more of the face region and the hand region, and the upper body detection region is determined according to the target region, the upper body posture of the driver is determined based on the upper body of the driver detected by the upper body detection region, the optimal target region is determined according to the upper body posture and the target region, and the target location of the driver is detected based on the optimal target region, and the target location includes any one or more of the face and the hand, so that the problem of false detection or inaccurate detection of the face or the hand can be avoided, and the detection accuracy of the multi-location detection is improved.
Fig. 2 is an application scenario schematic diagram of a driver multi-location detection method according to an embodiment of the present application. As shown in fig. 2, in an actual scene, the electronic control unit 10 acquires an image of the driver 30 acquired by the camera 20, performs target position detection on the above-described driver image, determines an upper body detection region according to the obtained target region, wherein the target position detection includes face detection and hand detection, the target region includes any one or more of a face region and a hand region, determines an upper body posture of the driver based on the upper body of the driver detected by the upper body detection region, finally determines an optimal target region according to the upper body posture, and detects a target position of the driver based on the optimal target region.
Fig. 3 is a flowchart of a driver multi-location detection method according to an embodiment of the present disclosure. As shown in fig. 3, the method in the embodiment of the present application may include:
step 101, a driver image is acquired.
Alternatively, the driver image may be acquired through a DMS camera during the driving process, where the DMS camera may be a near infrared camera, and the driver image acquired by the DMS camera may be in a YUV data format.
Step 102, detecting a target part of the driver image and determining a target area.
Optionally, the target portion detection includes face detection and hand detection, and the target region includes any one or more of a face region and a hand region. And carrying out face detection and hand detection on the acquired driver image by adopting a deep learning model. For example, in the embodiment of the present application, face detection and hand detection are performed on the driver image by using an Anchor-free-based DBFace network model, so as to determine any one or more of a face area and a hand area. Model training of the DBFace network model is required before face detection and hand detection of driver images are performed using the DBFace network model.
It should be noted that the DBFace network model is a face detection network model, so before hand detection is performed by using the DBFace network model, a class representing a hand needs to be added to the detection classification class, and then corresponding model training and subsequent hand detection are performed. The same network model is adopted to detect the face and the hands of the driver image, so that the detection speed can be improved. Optionally, other network models may be used to perform hand detection, so as to improve the accuracy of hand detection.
Optionally, face detection and hand detection can be performed on the driver image at the same time, face detection can be performed on the driver image only, hand detection can be performed on the driver image only, and specific conditions are determined according to detection requirements. For example, in practical applications, if fatigue monitoring, distraction monitoring, and dangerous driving behavior monitoring are required for the driver, face detection and hand detection are performed on the driver image in step 102. If only fatigue and/or distraction of the driver is required, then only the driver image may be face detected in step 102. If only dangerous behavior monitoring is required, then only the driver image may be hand detected in step 102.
For example, after face detection and hand detection are performed on the driver image, if no face area and no hand area are detected, the driver image is re-acquired and detected, i.e., step 101 and subsequent steps are re-performed.
Optionally, the face area may be indicated by a face detection frame, and the hand area may be indicated by a hand detection frame, where the face detection frame and the hand detection frame are square.
And step 103, determining an upper body detection area according to the target area.
In one possible implementation, referring to fig. 4, in step 103, specifically may include:
step 1031, if a face area exists, determining an upper body detection area according to the face area and a preset head-body proportionality coefficient.
Step 1032, if there is no face area, the driver image is set as the upper body detection area.
Optionally, after face detection and hand detection are performed on the driver image by using the DBface network model, if the face area and the hand area are obtained, or only the face area is obtained, determining an upper body detection area according to the face area and a preset head-body proportionality coefficient; if only the hand region is obtained, the driver image is set as the upper body detection region.
It should be noted that the driver image acquired in step 101 may be in various directions, and accordingly, the head orientation in the face area in step 1031 may also be in various directions, so as to reduce the influence of the head orientation in the face area on the determination of the subsequent upper body detection area, and the head orientation in the face area needs to be detected.
In one possible implementation, when the DBFace network model performs face detection on the driver image, the face orientation detection is also performed on the driver image, so as to determine the face orientation.
Optionally, when the DBFace network model performs head orientation detection on the driver image, the step of determining the head orientation may specifically include: and adding a head orientation detection branch to detect the head orientation after the feature extraction layer of the DBface network model based on a Multi-head Attention (Multi-head Attention) mechanism, and determining the head orientation.
For example, in order to share the feature extraction layer of the network model to increase the detection speed, based on the Multi-head Attention mechanism, a head orientation detection branch is added after the feature extraction layer of the DBFace network model, so as to detect and classify the head orientation, so that the DBFace network model can detect the head orientation of the driver image when detecting the face of the driver image. Wherein the classification loss function employs a cross entropy loss (Cross Entropy Loss) function.
A schematic diagram of the head orientation is shown in fig. 5. Referring to fig. 5, the head orientation includes forward, reverse, left and right. Wherein the human head orientation is forward, in the driver image, the human head is upward and the angle of deviation of the human head orientation from the vertical direction is smaller than the preset angle (see (a) in fig. 5); the head orientation being reversed is that in the driver image, the head is downward and the head orientation is offset from the vertical by an angle smaller than a preset angle (see (b) in fig. 5); the human head facing left is in the driver image, the human head facing left and the human head being offset from the horizontal by an angle smaller than a preset angle (see (c) in fig. 5); the direction of the human head to the right is that the human head is directed to the right in the driver image and the deviation angle of the human head from the horizontal direction is smaller than the preset angle (see (d) in fig. 5), wherein the preset angle takes 45 degrees. The human head offset angle is smaller than the preset angle, and the human head clockwise offset angle and the counterclockwise offset angle are both smaller than the preset angle.
In one possible implementation, referring to fig. 6, in step 1031, specifically may include:
and S1, rotating the face area according to the head orientation so as to enable the head orientation in the face area to be positive.
And S2, determining an upper body detection area according to the rotated face area and a preset head-body proportional coefficient.
Optionally, in practical application, if it is determined that the orientation of the head in the face area is forward, the face area does not need to be rotated; if the head orientation is determined to be reverse, the head orientation needs to be rotated 180 degrees, namely the face area is rotated 180 degrees; if the head orientation is determined to be left, the head orientation needs to be rotated by 90 degrees in the clockwise direction, namely the face area is rotated by 90 degrees in the clockwise direction; if the head orientation is determined to be right, the head orientation needs to be rotated by 90 degrees in the anticlockwise direction, that is, the face region needs to be rotated by 90 degrees in the anticlockwise direction, so that the head orientation in the face region is positive, and the rotated face region is obtained.
Illustratively, the width and height of the upper body detection area are determined according to the width and height of the rotated face area and a preset head-body proportionality coefficient. According to human body measurement, the human body ratio of the Viterbi Lu Wei is known that the ratio of the head height to the height is that the head height is more than or equal to 1:8, and the driver sits on the driving position in the automobile cab, so that the ratio of the head height to the upper half height is that the head height is more than or equal to 1:4. Considering the situations that a driver can stretch out and become lazy, the ratio of the head width to the upper body width can be more than or equal to 1:8. In the embodiment of the application, the preset height proportionality coefficient in the preset head-body proportionality coefficient is taken to be 4, and the preset width proportionality coefficient is taken to be 8.
The width and height formula of the upper body detection area is:
Width half-body ＝8×Width face +α
Height half-body ＝4×Height face in +β, width half-body Height, the width of the upper body detection area half-body To the height of the upper body detection area, width face Height is the width of the rotated face region face For the height of the rotated face region, α and β are compensation values in a preset head-body proportionality coefficient, so that the Width and height of the upper body detection region are more accurate, and tests show that α=0.1×width is taken in the embodiment of the present application face ，β＝0.1×Height face The width and height of the upper body detection area are made more accurate.
In another possible implementation, step 1031 may include: and determining an upper body detection area according to the face area and a preset head-body proportionality coefficient by taking the head orientation as a reference.
For example, the face region need not be rotated according to the head orientation, but after the head orientation is determined, the upper body detection region is determined based on the head orientation, for example, the head orientation is determined to be reverse, and then the upper body detection region is determined based on the face region and the preset head-body proportionality coefficient.
Step 104, determining the upper body posture of the driver based on the upper body of the driver detected by the upper body detection area.
In one possible implementation, referring to fig. 7, in step 104, specifically may include:
and 1041, performing upper body detection on the upper body detection area by adopting a Yolox-S network model.
Optionally, in the embodiment of the present application, the yellow-S network model based on Anchor-free is used to detect the upper body detection area, so as to determine the upper body of the driver. Before the upper body detection area is detected by using the Yolox-S network model, model training is required for the Yolox-S network model.
Step 1042, when the upper body of the driver is detected, estimating the posture of the upper body of the driver by using an openPose network model, and determining an upper body skeleton point, wherein the upper body skeleton point comprises a target position center point.
Optionally, the target site center point includes any one or more of a face center point and a hand center point. In the embodiment of the application, an openelse network model is adopted to estimate the gesture of the upper body of the driver so as to determine an upper body skeleton point including a central point of the target part (as shown in fig. 8). Before the openPose network model is adopted to estimate the gesture of the upper body of the driver, model training is required to be carried out on the openPose network model.
For example, if the upper body of the driver is not detected, the driver image is re-acquired and detected, i.e., step 101 and subsequent steps are re-performed.
And 105, determining an optimal target area according to the upper body posture and the target area, and detecting the target position of the driver based on the optimal target area.
Optionally, the target site includes any one or more of a face and a hand.
In one possible implementation, referring to fig. 9, in step 105, specifically may include:
step 1051, determining a face region center point according to the rotated face region, and determining a hand region center point according to the hand region.
Optionally, the center point of the rotated face area is taken as the center point of the face area, and the center point of the hand area is taken as the center point of the hand area.
Step 1052, determining an optimal face center point according to the face center point and the face region center point, determining an optimal face region according to the optimal face center point and the face region, and/or determining an optimal hand center point according to the hand center point and the hand region center point, and determining an optimal hand region according to the optimal hand center point and the hand region.
In one possible implementation manner, average value calculation may be performed on the coordinates corresponding to the face center point and the coordinates corresponding to the face region center point, so as to determine the coordinates corresponding to the optimal face center point and the optimal face center point, and further determine the optimal face region according to the coordinates corresponding to the optimal face center point and the size of the face detection frame corresponding to the face region.
And/or performing average value calculation on the coordinates corresponding to the hand center point and the coordinates corresponding to the hand region center point, determining the coordinates corresponding to the optimal hand center point and the optimal hand center point, and further determining the optimal hand region according to the coordinates corresponding to the optimal hand center point and the size of the hand detection frame corresponding to the hand region.
In another possible implementation manner, a weighted average calculation may be performed on the coordinates corresponding to the face center point and the coordinates corresponding to the face region center point, to determine the coordinates corresponding to the optimal face center point and the optimal face center point, where the weights may be set according to the actual situation, and are not specifically limited herein. And further determining the optimal face area according to the coordinates corresponding to the optimal face center point and the size of the face detection frame corresponding to the face area.
And/or, calculating a weighted average of coordinates corresponding to the hand center point and coordinates corresponding to the hand region center point, and determining coordinates corresponding to the optimal hand center point and the optimal hand center point, wherein weights can be set according to actual conditions and are not specifically limited herein. And further determining the optimal hand area according to the coordinates corresponding to the optimal hand center point and the size of the hand detection frame corresponding to the hand area.
Step 1053, detecting a face of the driver based on the optimal face region and/or detecting a hand of the driver based on the optimal hand region.
According to the method and the device for detecting the target part of the driver, the target part of the driver image is detected, the target area is determined, wherein the target part detection comprises face detection and hand detection, the target area comprises any one or more of the face area and the hand area, the upper body detection area is determined according to the target area, the upper body posture of the driver is determined based on the upper body of the driver detected through the upper body detection area, the optimal target area is determined according to the upper body posture and the target area, the target part of the driver is detected based on the optimal target area, the target part comprises any one or more of the face and the hand, the problem that the face or the hand is erroneously detected or the detection is inaccurate can be solved, and the detection precision of multi-part detection is improved.
It should be understood that the sequence number of each step in the foregoing embodiment does not mean that the execution sequence of each process should be determined by the function and the internal logic of each process, and should not limit the implementation process of the embodiment of the present application in any way.
Fig. 10 is a flowchart of a method for detecting abnormal driving behavior of a driver according to an embodiment of the present application. As shown in fig. 10, the method in the embodiment of the present application may include:
step 201, determining multiple groups of optimal target areas according to a driver multi-position detection method, wherein the optimal target areas comprise any one or more of an optimal face area and an optimal hand area.
Optionally, the above method for detecting multiple parts of a driver may be the method for detecting multiple parts of a driver provided in any embodiment of the present application.
Step 202, a first deep learning model is adopted, whether the change trend of face contour points included in a plurality of groups of optimal face areas accords with a fitting sample or not is judged, and whether the duration reaches a preset time or not is judged, if yes, a fatigue driving signal and an alarm signal are output; and calculating the face attitude angle according to the change trend of the face contour points included in the multiple groups of optimal face areas, judging whether the face attitude angle exceeds a preset angle threshold, and if so, outputting a distraction driving signal and an alarm signal.
In one possible implementation manner, whether the face in the obtained optimal face area is the face of the driver is determined, if so, the next operation is performed on the optimal face area, otherwise, the image of the driver is re-obtained and detected, that is, step 101 and the subsequent steps are re-performed.
The method includes the steps of determining whether a variation trend of face contour points included in a plurality of obtained groups of optimal face areas accords with a fitting sample and whether duration time reaches a preset time based on a first deep learning model, specifically, determining whether mouth contour points are in a large trend and accord with a fitting yawning sample, simultaneously determining whether the duration time reaches a first preset time and stability among the plurality of groups of optimal face areas reaches a first preset threshold value, determining whether eye contour points are in a closing trend and accord with a fitting dozing sample, and simultaneously determining whether the duration time reaches a second preset time and stability among the plurality of groups of optimal face areas reaches a second preset threshold value. If the judgment is yes, outputting a fatigue driving signal and an alarm signal.
Optionally, the face gesture angle is calculated according to the face contour point variation trend included in the multiple groups of optimal face areas, specifically, the face gesture angle includes a face pitching angle and a face shifting angle, and a sight pitching angle and a sight shifting angle, whether the face gesture angle exceeds a corresponding preset angle threshold value is judged, the duration reaches a third preset time, and the stability among the multiple groups of optimal face areas reaches the third preset threshold value. If the judgment is yes, outputting a distraction driving signal and an alarm signal.
And 203, judging whether a plurality of groups of optimal hand areas have target objects by adopting a second deep learning model, judging the types of the target objects when the target objects exist, and outputting dangerous behavior types and alarm signals according to the types of the target objects.
In one possible implementation, it is determined whether the hand gesture in the obtained optimal hand area is compliant, if not, the next operation is performed on the optimal hand area, otherwise, the driver image is re-obtained and detected, i.e. step 101 and the subsequent steps are re-performed.
Optionally, based on the second deep learning model, judging whether the obtained multiple groups of optimal hand areas have target objects, and whether the existence time of the target objects is longer than a fourth preset time, if the judgment is yes, judging the category of the target objects, and outputting dangerous behavior categories and alarm signals according to the category of the target objects. The target object may be a mobile device, a cigarette, food or drink, etc.
It should be noted that, during the execution of step 202, if an exception or an exit request is detected, the execution of step 202 is stopped; if an exception or exit request is detected during execution of step 203, execution of step 203 is stopped.
Alternatively, the first deep learning model may be a mobiletv 2 network model and the second deep learning model may be a Yolov5 network model. The alarm signal may be at least one of a buzzer alarm signal, a voice alarm signal, a meter alarm signal, a seat belt tightening signal, etc., and is not particularly limited herein.
According to the method for detecting the multiple parts of the driver, multiple groups of optimal target areas are determined according to the method for detecting the multiple parts of the driver, the optimal target areas comprise any one or more of optimal face areas and optimal hand areas, further, whether the change trend of face contour points included in the multiple groups of optimal face areas accords with a fitting sample and whether the duration reaches preset time is judged based on a deep learning model, and fatigue driving signals and alarm signals are output when the change trend accords with the fitting sample and the duration reaches the preset time; and calculating the face attitude angle according to the change trend of the face contour points included in the multiple groups of optimal face areas, judging whether the face attitude angle exceeds a preset angle threshold, and outputting a distraction driving signal and an alarm signal when the face attitude angle exceeds the preset angle threshold. And/or based on the deep learning model, judging whether a plurality of groups of optimal hand areas have target objects, judging the types of the target objects when the target objects are present, and outputting dangerous behavior types and alarm signals according to the types of the target objects, so that the problem of false detection or inaccurate detection of faces or hands can be avoided, the detection precision of multi-part detection is improved, the detection precision of abnormal driving behavior detection of a driver is further improved, and traffic accidents caused by poor driving behaviors of the driver are avoided.
In a possible implementation manner, the upper body of the driver is detected in step 104, the pose of the upper body of the driver is estimated, and the upper body skeleton points (mainly including the face skeleton points, the upper limb skeleton points and the hand skeleton points) are determined, which can also be applied to the interaction between the driver and the vehicle-mounted voice image. At this time, the detection of the upper body of the driver may be based on the acquired driver video. For example, after the upper body of the driver is detected and the posture of the upper body of the driver is estimated, the driver is confirmed to present the clapping action based on the determined upper body skeleton point, and then the vehicle-mounted voice image can be awakened, so that subsequent interaction is performed.
In practical application, when a driver feels tired driving, the driver can wake up the vehicle-mounted voice image after stopping, after the upper body of the driver is detected and the gesture of the upper body of the driver is estimated, the driver is confirmed to select a tired eliminating option through a specific action based on the determined upper body skeleton point, the vehicle-mounted voice image is enabled to execute another set of specific action based on the tired eliminating option, the upper body of the driver is detected again and the gesture of the upper body of the driver is estimated, and if the action of the driver is confirmed to be consistent with the specific action of the vehicle-mounted voice image based on the determined upper body skeleton point, the driver can be confirmed to have tired driving eliminated, and the vehicle can be driven normally.
In another possible implementation manner, the method in step 104 detects the upper body of the driver, performs gesture estimation on the upper body of the driver, determines the upper body skeleton points (mainly including the face skeleton points, the upper limb skeleton points and the hand skeleton points), and may also be applied to the vehicle-carried induction game. At this time, the detection of the upper body of the driver may be based on the acquired driver video. For example, when a body feeling game is performed on the upper body of a driver in an automobile cabin, the body movement of the driver is confirmed based on the determined upper body skeleton point after the upper body of the driver is detected and the posture of the upper body of the driver is estimated, so that the subsequent game operation is completed.
The upper body skeleton point is determined by detecting the upper body of the driver and estimating the gesture of the upper body of the driver, and the method is applied to interaction between the driver and the vehicle-mounted voice image or in a vehicle-mounted somatosensory game, so that fatigue driving can be eliminated and user experience can be improved.
The following are device embodiments of the present application, for details not described in detail therein, reference may be made to the corresponding method embodiments described above.
Fig. 11 is a schematic structural diagram of a driver multi-position detection device according to an embodiment of the present application, and for convenience of explanation, only the portions related to the embodiment of the present application are shown, and the details are as follows:
As shown in fig. 11, the driver multi-location detection device provided in this embodiment may include: an acquisition module 301, a detection module 302, a first determination module 303 and an output module 304.
The acquiring module 301 is configured to acquire a driver image.
The detection module 302 is configured to perform target location detection on the driver image, and determine a target area, where the target location detection includes face detection and hand detection, and the target area includes any one or more of a face area and a hand area.
The first determining module 303 is configured to determine an upper body detection area according to the target area, and determine an upper body posture of the driver based on the upper body of the driver detected by the upper body detection area.
And the output module 304 is configured to determine an optimal target area according to the upper body gesture and the target area, and detect a target portion of the driver based on the optimal target area, where the target portion includes any one or more of a face and a hand.
Optionally, the first determining module 303 is specifically configured to: when a face area exists, determining an upper body detection area according to the face area and a preset head-body proportional coefficient; when the face region does not exist, the driver image is set as the upper body detection region.
Optionally, the detection module 302 is specifically configured to: and detecting a target part of the driver image by adopting the DBface network model, determining a target area, detecting the head orientation of the driver image, and determining the head orientation.
Optionally, the first determining module 303 is further specifically configured to: rotating the face area according to the orientation of the head so as to make the orientation of the head in the face area be positive; and determining an upper body detection area according to the rotated face area and a preset head-body proportionality coefficient.
Optionally, the detection module 302 is further specifically configured to: based on a multi-head attention mechanism, adding a head orientation detection branch after a feature extraction layer of the DBface network model, detecting the head orientation, and determining the head orientation; the head orientation includes forward, reverse, left and right.
Optionally, the first determining module 303 is further specifically configured to: determining the width and the height of the upper body detection area according to the width and the height of the rotated face area and a preset head-body proportionality coefficient;
the width and height of the upper body detection area are:
Width half-body ＝8×Width face +0.1×Width face
Height nalf-body ＝4×Height face +0.1×Height face
wherein Width half-body Height, the width of the upper body detection area half-body To the height of the upper body detection area, width face Height is the width of the rotated face region face Is the height of the rotated face region.
Optionally, the first determining module 303 is further specifically configured to: the method comprises the steps of detecting the upper body of a driver in an upper body detection area by adopting a Yolox-S network model; when the upper body of the driver is detected, estimating the posture of the upper body of the driver by adopting an OpenPose network model, and determining an upper body skeleton point; the upper body skeleton point comprises a target position center point, and the target position center point comprises any one or more of a face center point and a hand center point.
Optionally, the output module 304 is specifically configured to: determining a face region center point according to the rotated face region, and determining a hand region center point according to the hand region; determining an optimal face center point according to the face center point and the face region center point, and determining an optimal face region according to the optimal face center point and the face region; and/or determining an optimal hand center point according to the hand center point and the hand region center point, and determining an optimal hand region according to the optimal hand center point and the hand region; the face of the driver is detected based on the optimal face region and/or the hand of the driver is detected based on the optimal hand region.
Optionally, the detection module 302 is further specifically configured to: when the face region and the hand region are not detected or the upper body is not detected, the driver image is re-acquired and detected.
It should be noted that, because the content of information interaction and execution process between the above devices/units is based on the same concept as the method embodiment of the present application, specific functions and technical effects thereof may be referred to in the method embodiment section, and will not be described herein again.
Fig. 12 is a schematic structural diagram of a driver abnormal driving behavior detection device provided in an embodiment of the present application. As shown in fig. 12, the device for detecting abnormal driving behavior of a driver provided in this embodiment may include: a second determination module 401 and an execution module 402.
The second determining module 401 is configured to determine a plurality of sets of optimal target areas according to the driver multi-location detection method described in any one of the foregoing embodiments, where the optimal target areas include any one or more of an optimal face area and an optimal hand area.
The execution module 402 is configured to determine, by using the first deep learning model, whether a face contour point variation trend included in the plurality of groups of optimal face regions conforms to a fitting sample and the duration reaches a preset time, and output a fatigue driving signal and an alarm signal when the fitting sample conforms to the duration reaches the preset time; calculating a face attitude angle according to the change trend of face contour points included in the multiple groups of optimal face areas, judging whether the face attitude angle exceeds a preset angle threshold, and outputting a distraction driving signal and an alarm signal when the face attitude angle exceeds the preset angle threshold; and/or judging whether the target object exists in the plurality of groups of optimal hand areas by adopting a second deep learning model, and judging the category of the target object when the target object exists; and outputting dangerous behavior categories and alarm signals according to the categories of the target objects.
The present application also provides a computer program product having a program code which, when run in a corresponding processor, controller, computing device or terminal, performs the steps of any of the driver multi-site detection method embodiments described above, and/or performs the steps of the driver abnormal driving behavior detection method embodiments described above, such as steps 101 to 105 shown in fig. 3, and/or steps 201 to 203 shown in fig. 10.
Those skilled in the art will appreciate that the methods and apparatus presented in the embodiments of the present application may be implemented in various forms of hardware, software, firmware, special purpose processors, or a combination thereof. The special purpose processor may include an Application Specific Integrated Circuit (ASIC), a Reduced Instruction Set Computer (RISC), and/or a Field Programmable Gate Array (FPGA). The proposed method and device are preferably implemented as a combination of hardware and software. The software is preferably installed as an application program on a program storage device. Which is typically a machine based on a computer platform having hardware, such as one or more Central Processing Units (CPUs), random Access Memory (RAM), and one or more input/output (I/O) interfaces. An operating system is also typically installed on the computer platform. The various processes and functions described herein may either be part of the application program or part of the application program which is executed by the operating system.
Fig. 13 is a schematic structural diagram of an electronic control unit according to an embodiment of the present application. As shown in fig. 13, the electronic control unit 10 of this embodiment includes: a processor 510, a memory 520, and a computer program 521 executable on the processor 510 is stored in the memory 520. The steps of any of the various method embodiments described above, such as steps 101 through 105 shown in fig. 3, and/or steps 201 through 203 shown in fig. 10, are implemented when the processor 510 executes the computer program 521. Alternatively, the processor 510, when executing the computer program 521, performs the functions of the modules/units of the apparatus embodiments described above, e.g., the functions of the modules 301 to 304 shown in fig. 11, and/or the functions of the modules 401 to 402 shown in fig. 12.
By way of example, computer program 521 may be partitioned into one or more modules/units that are stored in memory 520 and executed by processor 510 to perform the schemes provided herein. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions for describing the execution of the computer program 521 in the electronic control unit 10.
It will be appreciated by those skilled in the art that fig. 13 is merely an example of an electronic control unit and is not limiting of the electronic control unit, and may include more or fewer components than shown, or may combine certain components, or different components, such as input-output devices, network access devices, buses, etc.
The processor 510 may be a central processing unit (Central Processing Unit, CPU), but may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 520 may be an internal storage unit of the terminal device, for example, a hard disk or a memory of the terminal device, or an external storage device of the terminal device, for example, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash Card (Flash Card) or the like, which are provided on the terminal device. The memory 520 may also include both internal storage units of the terminal device and external storage devices. The memory 520 is used to store computer programs and other programs and data required by the terminal device. The memory 520 may also be used to temporarily store data that has been output or is to be output.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions. The functional units and modules in the embodiment may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of a software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working process of the units and modules in the above system may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and in part, not described or illustrated in any particular embodiment, reference is made to the related descriptions of other embodiments.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus/terminal device and method may be implemented in other manners. For example, the apparatus/terminal device embodiments described above are merely illustrative, e.g., the division of the modules or units is merely a logical function division, and there may be additional divisions in actual implementation, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection via interfaces, devices or units, which may be in electrical, mechanical or other forms.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated modules/units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the present application may implement all or part of the flow of the method of the above embodiment, or may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, and when the computer program is executed by a processor, the computer program may implement the steps of each method embodiment described above. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth.
Fig. 14 is a schematic structural view of a vehicle according to an embodiment of the present application. As shown in fig. 14, the vehicle 600 of this embodiment includes: an electronic control unit 10. The electronic control unit 10 may be the electronic control unit 10 provided in the embodiments of the present application, and specific functions and technical effects thereof may be referred to in the method embodiment section, and are not described herein.
Furthermore, the features of the embodiments shown in the drawings or mentioned in the description of the present application are not necessarily to be construed as separate embodiments from each other. Rather, each feature described in one example of one embodiment may be combined with one or more other desired features from other embodiments, resulting in other embodiments not described in text or with reference to the drawings.
The above embodiments are only for illustrating the technical solution of the present application, and are not limiting; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application, and are intended to be included in the scope of the present application.

Claims (13)

1. A driver multi-site detection method, comprising:
acquiring a driver image;
performing target position detection on the driver image, and determining a target area, wherein the target position detection comprises face detection and hand detection, and the target area comprises any one or more of a face area and a hand area;
determining an upper body detection area according to the target area, and determining an upper body posture of the driver based on the upper body of the driver detected through the upper body detection area;
and determining an optimal target area according to the upper body gesture and the target area, and detecting a target part of a driver based on the optimal target area, wherein the target part comprises any one or more of a human face and a hand.
2. The driver multi-site detection method according to claim 1, characterized in that the determining an upper body detection region from the target region includes:
if the human face region exists, determining the upper body detection region according to the human face region and a preset head-body proportionality coefficient;
and if the human face area does not exist, taking the driver image as the upper body detection area.
3. The driver multi-site detection method according to claim 2, wherein the target site detection is performed on the driver image, a target area is determined, and the head orientation detection is performed on the driver image, and a head orientation is determined, using a DBFace network model;
the determining the upper body detection area according to the face area and the preset head-body proportionality coefficient comprises the following steps:
rotating the face area according to the head orientation so as to enable the head orientation in the face area to be positive;
and determining the upper body detection area according to the rotated face area and a preset head-body proportionality coefficient.
4. A driver multi-site detection method according to claim 3, wherein the DBFace network model performs head orientation detection on the driver image, and determining head orientation comprises:
based on a multi-head attention mechanism, adding a head orientation detection branch after a feature extraction layer of the DBface network model, detecting the head orientation, and determining the head orientation; the head orientation includes forward, reverse, left and right.
5. The driver multi-site detection method of claim 3, wherein the determining the upper body detection region according to the rotated face region and a preset head-body proportionality coefficient comprises:
Determining the width and the height of the upper body detection area according to the width and the height of the rotated face area and a preset head-body proportionality coefficient;
the width and height of the upper body detection area are:
Width half-body ＝8×Width face +0.1×Width face
Height half-body ＝4×Height face +0.1×Height face
wherein Width half-body Height, the width of the upper body detection area half-body To the height of the upper body detection area, width face Height is the width of the rotated face region face Is the height of the rotated face region.
6. The driver multi-site detection method according to claim 3, characterized in that the determining the upper body posture of the driver based on the upper body of the driver detected by the upper body detection region includes:
performing upper body detection on the upper body detection area by using a Yolox-S network model;
when the upper body of the driver is detected, carrying out posture estimation on the upper body of the driver by adopting an OpenPose network model, and determining an upper body skeleton point; the upper body skeleton point comprises a target position center point, and the target position center point comprises any one or more of a face center point and a hand center point.
7. The driver multi-site detection method according to claim 6, wherein the determining an optimal target area from the upper body posture and the target area and detecting a target site of the driver based on the optimal target area includes:
Determining a face region center point according to the rotated face region, and determining a hand region center point according to the hand region;
determining an optimal face center point according to the face center point and the face region center point, and determining an optimal face region according to the optimal face center point and the face region;
and/or determining an optimal hand center point according to the hand center point and the hand region center point, and determining an optimal hand region according to the optimal hand center point and the hand region;
and detecting the face of the driver based on the optimal face region and/or detecting the hand of the driver based on the optimal hand region.
8. The driver multi-site detection method according to any one of claims 1 to 7, characterized in that the method further includes:
if the face area and the hand area are not detected, or the upper body is not detected, the driver image is re-acquired and detected.
9. A method for detecting abnormal driving behavior of a driver, comprising:
the driver multi-site detection method according to any one of claims 1 to 8, determining a plurality of sets of optimal target areas including any one or more of an optimal face area and an optimal hand area;
Judging whether the change trend of the face contour points included in the plurality of groups of optimal face areas accords with the fitting sample or not and whether the duration reaches the preset time or not by adopting a first deep learning model, and if so, outputting a fatigue driving signal and an alarm signal; calculating a face attitude angle according to the face contour point change trend included in the plurality of groups of optimal face areas, judging whether the face attitude angle exceeds a preset angle threshold, and if so, outputting a distraction driving signal and an alarm signal;
and/or judging whether the target object exists in the plurality of groups of optimal hand areas by adopting a second deep learning model, and judging the category of the target object when the target object exists; and outputting dangerous behavior categories and alarm signals according to the categories of the target objects.
10. A driver multi-site detection apparatus, characterized by comprising:
the acquisition module is used for acquiring the driver image;
the detection module is used for detecting a target part of the driver image and determining a target area, wherein the target part detection comprises face detection and hand detection, and the target area comprises any one or more of a face area and a hand area;
A first determination module configured to determine an upper body detection area according to the target area, and determine an upper body posture of a driver based on the upper body of the driver detected by the upper body detection area;
and the output module is used for determining an optimal target area according to the upper body gesture and the target area, detecting a target part of a driver based on the optimal target area, wherein the target part comprises any one or more of a human face and a hand.
11. An electronic control unit comprising a memory and a processor, the memory having stored therein a computer program executable on the processor, characterized in that the processor, when executing the computer program, implements the steps of the driver multi-site detection method according to any one of claims 1 to 8 and/or the steps of the driver abnormal driving behavior detection method according to claim 9.
12. A vehicle comprising an electronic control unit according to claim 11.
13. A computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the driver multi-site detection method according to any one of claims 1 to 8, and/or the driver abnormal driving behavior detection method according to claim 9.
CN202210784656.1A 2022-06-29 2022-06-29 Driver multi-position detection method and device, electronic control unit and vehicle Pending CN117392642A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210784656.1A CN117392642A (en) 2022-06-29 2022-06-29 Driver multi-position detection method and device, electronic control unit and vehicle

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210784656.1A CN117392642A (en) 2022-06-29 2022-06-29 Driver multi-position detection method and device, electronic control unit and vehicle

Publications (1)

Publication Number Publication Date
CN117392642A true CN117392642A (en) 2024-01-12

Family

ID=89467195

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210784656.1A Pending CN117392642A (en) 2022-06-29 2022-06-29 Driver multi-position detection method and device, electronic control unit and vehicle

Country Status (1)

Country Link
CN (1) CN117392642A (en)

Similar Documents

Publication Publication Date Title
EP3033999B1 (en) Apparatus and method for determining the state of a driver
US9928404B2 (en) Determination device, determination method, and non-transitory storage medium
US9662977B2 (en) Driver state monitoring system
US20150279128A1 (en) Method of diagnosing operating characteristics
CN105286802B (en) Driver Fatigue Detection based on video information
EP2457499B1 (en) Line-of-sight estimation device
JP4529394B2 (en) Driver's vehicle driving characteristic estimation device
JP2016057839A (en) Facial direction detection device and warning system for vehicle
US11144756B2 (en) Method and system of distinguishing between a glance event and an eye closure event
US8594774B2 (en) Sleep determination device and sleep determination method
US7599546B2 (en) Image information processing system, image information processing method, image information processing program, and automobile
JP2016115117A (en) Determination device and determination method
US8503737B2 (en) Visual line estimating apparatus
CN115471824A (en) Eye state detection method and device, electronic equipment and storage medium
CN117392642A (en) Driver multi-position detection method and device, electronic control unit and vehicle
JP3036319B2 (en) Driver status monitoring device
US20220284718A1 (en) Driving analysis device and driving analysis method
CN115703471A (en) Driver attention and hand placement system and method
KR20150067679A (en) System and method for gesture recognition of vehicle
CN113753057A (en) State determination device and state determination method
JPH079880A (en) Abnormality warning device for driver
CN116152790B (en) Safety belt detection method and device
US20230394702A1 (en) Device, method, and computer program for estimating seat position
CN104364821B (en) People detects device
CN116798189A (en) State detection method, device and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination