JPWO2020188689A1 - Driver condition monitoring device and driver condition monitoring method - Google Patents

Driver condition monitoring device and driver condition monitoring method Download PDF

Info

Publication number
JPWO2020188689A1
JPWO2020188689A1 JP2021506843A JP2021506843A JPWO2020188689A1 JP WO2020188689 A1 JPWO2020188689 A1 JP WO2020188689A1 JP 2021506843 A JP2021506843 A JP 2021506843A JP 2021506843 A JP2021506843 A JP 2021506843A JP WO2020188689 A1 JPWO2020188689 A1 JP WO2020188689A1
Authority
JP
Japan
Prior art keywords
driver
light
image
determination unit
external light
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
JP2021506843A
Other languages
Japanese (ja)
Other versions
JP7158561B2 (en
Inventor
衣里 小寺
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Mitsubishi Electric Corp
Original Assignee
Mitsubishi Electric Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Mitsubishi Electric Corp filed Critical Mitsubishi Electric Corp
Publication of JPWO2020188689A1 publication Critical patent/JPWO2020188689A1/en
Application granted granted Critical
Publication of JP7158561B2 publication Critical patent/JP7158561B2/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Traffic Control Systems (AREA)
  • Image Analysis (AREA)

Abstract

運転者状態監視装置は、車両の運転者の顔に向けて近赤外線を照射する近赤外発光素子（１２）および運転者の顔を撮像する近赤外線カメラ（１１）を制御し、近赤外線発光素子（１２）をパルス発光させるとともに、近赤外線発光素子（１２）の発光期間および消光期間と近赤外線カメラ（１１）の撮像タイミングとを同期させる撮像制御部（２１）と、発光期間に近赤外線カメラ（１１）が撮像した発光画像および消光期間に近赤外線カメラ（１１）が撮像した消光画像を取得する画像取得部（２２）と、発光画像から運転者の顔情報を検出し、顔情報に基づいて運転者の状態を判定する運転者状態判定部（２３）と、消光画像から運転者の眼周辺の輝度を検出し、輝度が輝度閾値以上の場合、運転者の眼に外光が照射されている状態と判定する外光状態判定部（２４）と、を備える。 The driver condition monitoring device controls a near-infrared light emitting element (12) that irradiates near infrared rays toward the driver's face of the vehicle and a near infrared camera (11) that images the driver's face, and emits near infrared rays. An imaging control unit (21) that causes the element (12) to emit pulse light and synchronizes the emission period and extinguishing period of the near-infrared light emitting element (12) with the imaging timing of the near-infrared camera (11), and the near-infrared ray during the emission period. The image acquisition unit (22) that acquires the light-emitting image captured by the camera (11) and the light-emitting image captured by the near-infrared camera (11) during the light-extinguishing period, and the driver's face information is detected from the light-emitting image and used as face information. The driver state determination unit (23), which determines the driver's state based on the light, detects the brightness around the driver's eye from the extinguished image, and when the brightness is equal to or higher than the brightness threshold, the driver's eye is irradiated with external light. It is provided with an external light state determination unit (24) for determining the state of being in the state.

Description

この発明は撮像装置の撮像した画像に基づいて運転者の状態を監視する運転者状態監視装置及び運転者状態監視方法に関するものである。 The present invention relates to a driver condition monitoring device and a driver condition monitoring method for monitoring a driver's condition based on an image captured by the imaging device.

従来、投光器から近赤外光を運転者の顔に向けて照射し、運転者の顔で反射した近赤外光を近赤外線カメラで撮像することで、外光に大きく影響されることなく、運転者の顔を鮮明に撮像するドライバモニタリング装置が提案されている(下記特許文献1)。 Conventionally, near-infrared light is emitted from a floodlight toward the driver's face, and the near-infrared light reflected by the driver's face is imaged by a near-infrared camera without being significantly affected by external light. A driver monitoring device that clearly images the driver's face has been proposed (Patent Document 1 below).

特開２０１０−０９７３７９号公報 Japanese Unexamined Patent Publication No. 2010-097379

上記特許文献１のように、外光に大きく影響されることなく、運転者の顔を鮮明に撮像するには、外光よりも強い近赤外線を投光器から運転者の顔に投光する必要がある。しかし、そのような場合、運転者の顔に外光が当たっている等の外光情報が失われてしまう。外光情報が失われてしまう場合、例えば、運転者が眼を細めているときに、外光が眩しくて眼を細めているのか、眠くて眼を細めているかが区別できなくなってしまう。
一方、外光情報を取得するために、投光器から投光する近赤外光の強度を外光よりも弱くした場合、運転者の顔を鮮明に撮像することは困難になる。
また、近赤外線カメラに加え照度計を設けることで、外光の情報を取得することは可能となるが、照度計を追加する分コストが増加してしまう。As in Patent Document 1, in order to clearly image the driver's face without being significantly affected by the outside light, it is necessary to project near infrared rays stronger than the outside light from the floodlight onto the driver's face. be. However, in such a case, the external light information such as the external light shining on the driver's face is lost. When the outside light information is lost, for example, when the driver is squinting, it becomes impossible to distinguish whether the outside light is dazzling and squinting, or sleepy and squinting.
On the other hand, when the intensity of the near-infrared light projected from the floodlight is weaker than that of the external light in order to acquire the external light information, it becomes difficult to clearly image the driver's face.
Further, by providing an illuminometer in addition to the near-infrared camera, it is possible to acquire information on external light, but the cost increases due to the addition of the illuminometer.

この発明は上記した問題点を解決するためになされたものであり、近赤外線カメラで運転者の顔を鮮明に撮像しつつ、外光の情報を取得することができる運転者状態監視装置を提供することを目的とする。 The present invention has been made to solve the above-mentioned problems, and provides a driver condition monitoring device capable of acquiring information on external light while clearly capturing the driver's face with a near-infrared camera. The purpose is to do.

この発明に係る運転者状態監視装置は、車両の運転者の顔に向けて近赤外線を照射する近赤外発光素子および運転者の顔を撮像する近赤外線カメラを制御し、近赤外線発光素子をパルス発光させるとともに、近赤外線発光素子の発光期間および消光期間と近赤外線カメラの撮像タイミングとを同期させる撮像制御部と、発光期間に近赤外線カメラが撮像した発光画像および消光期間に近赤外線カメラが撮像した消光画像を取得する画像取得部と、発光画像から運転者の顔情報を検出し、顔情報に基づいて運転者の状態を判定する運転者状態判定部と、消光画像から運転者の眼周辺の輝度を検出し、輝度が輝度閾値以上の場合、運転者の眼に外光が照射されている状態と判定する外光状態判定部と、を備える。 The driver state monitoring device according to the present invention controls a near-infrared light emitting element that irradiates near infrared rays toward the driver's face of a vehicle and a near infrared camera that images the driver's face, and controls the near infrared light emitting element. An imaging control unit that synchronizes the emission period and extinguishing period of the near-infrared light emitting element with the imaging timing of the near-infrared camera, and the emission image captured by the near-infrared camera during the emission period and the near-infrared camera during the extinguishing period. An image acquisition unit that acquires the captured extinguished image, a driver state determination unit that detects the driver's face information from the emitted image and determines the driver's condition based on the face information, and a driver's eye from the extinguished image. It is provided with an external light state determination unit that detects the brightness of the surroundings and determines that the driver's eyes are irradiated with external light when the brightness is equal to or higher than the brightness threshold.

この発明の運転者状態監視装置は、近赤外線カメラで運転者の顔を鮮明に撮像しつつ、外光の情報を取得することができる。 The driver condition monitoring device of the present invention can acquire information on external light while clearly capturing the driver's face with a near-infrared camera.

本実施の形態1に係る運転者状態監視装置２の構成を示すブロック図である。It is a block diagram which shows the structure of the driver condition monitoring apparatus 2 which concerns on this Embodiment 1. 本実施の形態１に係る近赤外線発光素子１１の撮像タイミング及び近赤外線発光素子１２の発光タイミングを説明するタイミングチャートである。It is a timing chart for explaining the imaging timing of the near-infrared light emitting element 11 and the light emitting timing of the near infrared light emitting element 12 according to the first embodiment. 本実施の形態１に係る運転状態監視装置２の眼周辺の輝度を検出する動作を示すフローチャートである。It is a flowchart which shows the operation which detects the brightness around the eye of the operation state monitoring apparatus 2 which concerns on this Embodiment 1. 本実施の形態１に係る運転者の顔にできる影のパターンから外光の方向を判定する方法を説明するための図である。It is a figure for demonstrating the method of determining the direction of the outside light from the pattern of the shadow formed on the driver's face which concerns on Embodiment 1. FIG. 図５Ａ及び図５Ｂは、本実施の形態１に係る運転者状態監視装置２のハードウェア構成例を示す図である。5A and 5B are diagrams showing a hardware configuration example of the driver condition monitoring device 2 according to the first embodiment. 本実施の形態１に係る運転者状態監視装置２の動作を示すフローチャートである。It is a flowchart which shows the operation of the driver condition monitoring apparatus 2 which concerns on this Embodiment 1. 本実施の形態２に係る運転者状態監視装置２Ｂの構成を示すブロック図である。It is a block diagram which shows the structure of the driver condition monitoring apparatus 2B which concerns on embodiment 2. FIG. 本実施の形態２に係る運転者状態監視装置２Ｂにおける運転者状態判定部２３Ｂの動作を示すフローチャートである。It is a flowchart which shows the operation of the driver state determination part 23B in the driver state monitoring device 2B which concerns on this Embodiment 2.

以下、この発明を実施するための形態について、図を参照して説明する。なお、各図中の同一又は相当する部分には同一の符号を付しており、その重複説明は適宜に簡略化ないし省略する。 Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. The same or corresponding parts in each drawing are designated by the same reference numerals, and the duplicated description thereof will be appropriately simplified or omitted.

実施の形態1
図１は本実施の形態1に係る運転者状態監視装置２の構成を示すブロック図である。
車両には、撮像装置１、運転者状態監視装置２、制御装置３、警報装置４、遮光制御装置５、自動ブレーキ装置６、ヘッドアップディスプレイ装置７及びカーナビゲーション装置８が搭載されており、運転者状態監視装置２は撮像装置１及び制御装置３に電気的に接続される。また、制御装置３は警報装置４、遮光装置５、自動ブレーキ装置６、ヘッドアップディスプレイ装置７及びカーナビゲーション装置８に電気的に接続される。
なお、本実施の形態１においては、運転者状態監視装置２と制御装置３を別の装置としているが、運転者状態監視装置２の中に制御装置３が含まれるような構成としてもよい。Embodiment 1
FIG. 1 is a block diagram showing a configuration of a driver condition monitoring device 2 according to the first embodiment.
The vehicle is equipped with an image pickup device 1, a driver status monitoring device 2, a control device 3, an alarm device 4, a shading control device 5, an automatic braking device 6, a head-up display device 7, and a car navigation device 8. The person state monitoring device 2 is electrically connected to the image pickup device 1 and the control device 3. Further, the control device 3 is electrically connected to the alarm device 4, the light shielding device 5, the automatic braking device 6, the head-up display device 7, and the car navigation device 8.
In the first embodiment, the driver condition monitoring device 2 and the control device 3 are separate devices, but the driver condition monitoring device 2 may include the control device 3.

撮像装置１は、近赤外線の光を撮像する近赤外線カメラ１１及び近赤外線の光を発光する近赤外線発光素子１２を備えており、近赤外線カメラ１１の撮像範囲および近赤外線発光素子１２の照射範囲に車両の運転者の顔が存在するような位置および角度で車両内部に設置されている。なお、近赤外線カメラ１１及び近赤外線発光素子１２は一つのモジュールとなっていてもよいし、分離されていてもよい。
上記のように撮像装置１は、近赤外線発光素子１２によって照らされた被写体を撮像することで、太陽光等の外光の影響を受けることなく、コントラスト鮮やかな被写体の画像を取得することができる。The image pickup device 1 includes a near-infrared camera 11 that captures near-infrared light and a near-infrared light emitting element 12 that emits near-infrared light, and includes an imaging range of the near-infrared camera 11 and an irradiation range of the near-infrared light emitting element 12. It is installed inside the vehicle at a position and angle so that the driver's face of the vehicle is present in the vehicle. The near-infrared camera 11 and the near-infrared light emitting element 12 may be one module or may be separated.
As described above, the imaging device 1 can acquire an image of a subject with vivid contrast without being affected by external light such as sunlight by imaging the subject illuminated by the near-infrared light emitting element 12. ..

運転者状態監視装置２は、撮像制御部２１、画像取得部２２、運転者状態判定部２３及び外光状態判定部２４を備えている。
撮像制御部２１は、近赤外線発光素子１２のパルス発光タイミングを設定し、近赤外線発光素子１２をパルス発光させる。発光タイミングは固定値としてもよいし、任意に変更可能としてもよい。
更に、撮像制御部２１は、近赤外線カメラ１１の撮像タイミングを当該発光タイミングに同期させ、近赤外線素子１２が発光期間と消光期間に画像を撮像するように制御する。具体的には、図２を用いて説明する。The driver condition monitoring device 2 includes an image pickup control unit 21, an image acquisition unit 22, a driver condition determination unit 23, and an external light condition determination unit 24.
The image pickup control unit 21 sets the pulse emission timing of the near-infrared light emitting element 12, and causes the near-infrared light emitting element 12 to emit pulse light. The light emission timing may be a fixed value or may be arbitrarily changed.
Further, the image pickup control unit 21 synchronizes the image pickup timing of the near infrared camera 11 with the light emission timing, and controls the near infrared element 12 to take an image during the light emission period and the quenching period. Specifically, it will be described with reference to FIG.

図２は、本実施の形態１に近赤外線カメラ１１の撮像タイミング及び近赤外線発光素子１２の発光タイミングを説明するタイミングチャートである。
図２においてＴ１は近赤外線カメラ１１が光信号を取り込む期間（撮像期間）であり、Ｔ２は近赤外線カメラ１１が光信号を取り込まない期間（ブランキング期間）である。
図２に示すように近赤外線発光素子１２は発光と消光を繰り返しており、ブランキング期間に近赤外線発光素子１２の発光が立ち上がり、ブランキング期間に近赤外線発光素子１２の発光が立ち下がる。このように撮像制御部２１が近赤外線カメラ１１および近赤外線発光素子１２を制御することで、近赤外線発光素子１２が発光している時の発光画像と、近赤外線素子１２が消光している時の消光画像とを交互に撮像することができる。FIG. 2 is a timing chart illustrating the imaging timing of the near-infrared camera 11 and the emission timing of the near-infrared light emitting element 12 according to the first embodiment.
In FIG. 2, T1 is a period during which the near-infrared camera 11 captures an optical signal (imaging period), and T2 is a period during which the near-infrared camera 11 does not capture an optical signal (blanking period).
As shown in FIG. 2, the near-infrared light emitting element 12 repeats light emission and quenching, the light emission of the near infrared light emitting element 12 rises during the blanking period, and the light emission of the near infrared light emitting element 12 falls during the blanking period. By controlling the near-infrared camera 11 and the near-infrared light emitting element 12 in this way, the image pickup control unit 21 controls the light emitting image when the near infrared light emitting element 12 is emitting light and when the near infrared element 12 is extinguished. It is possible to alternately capture the extinguished image of.

近赤外線発光素子１２が発光期間に撮像する場合、近赤外線発光素子１２によって照らされた運転者の顔を撮像するため、外光の影響を受けずに、運転者の顔が鮮明に映った画像を撮像することができる。一方、消光画像は、外光によって照らされた運転者の顔を撮像するため、運転者の顔は発光画像よりも不鮮明になるものの、画素の輝度から外光の情報を取得することができる。つまり、上述したように、近赤外線発光素子１２が発光している時の発光画像と、近赤外線発光素子１２が消光している時の消光画像とを交互に撮像することによって、外光を検出するための照度計などを追加せずに、１台の近赤外線カメラによって、鮮明な運転者の顔の画像と外光の情報を含んだ画像を両方撮像することが可能となる。 When the near-infrared light emitting element 12 takes an image during the light emitting period, the driver's face illuminated by the near infrared light emitting element 12 is imaged, so that the driver's face is clearly reflected without being affected by external light. Can be imaged. On the other hand, since the quenching image captures the driver's face illuminated by the external light, the driver's face becomes less clear than the luminescent image, but the information on the external light can be obtained from the brightness of the pixels. That is, as described above, the external light is detected by alternately capturing the light emitting image when the near-infrared light emitting element 12 is emitting light and the extinguished image when the near infrared light emitting element 12 is extinguished. With one near-infrared camera, it is possible to capture both a clear image of the driver's face and an image including information on external light without adding a luminometer or the like.

また、撮像制御部２１は近赤外線カメラ１１の撮像タイミング及び近赤外線発光素子１２の発光タイミングを画像取得部２２に逐次送信する。
なお、上記では近赤外線発光素子１２の発光タイミングに対して、近赤外線カメラ１１の撮像タイミングを同期させるようにしたが、近赤外線カメラ１１の撮像タイミングに対して近赤外線発光素子１２の発光タイミングを同期させるようにしてもよい。Further, the image pickup control unit 21 sequentially transmits the image pickup timing of the near infrared camera 11 and the light emission timing of the near infrared light emitting element 12 to the image acquisition unit 22.
In the above, the imaging timing of the near-infrared camera 11 is synchronized with the emission timing of the near-infrared light emitting element 12, but the emission timing of the near-infrared emitting element 12 is synchronized with the imaging timing of the near-infrared camera 11. It may be synchronized.

画像取得部２２は、近赤外線カメラ１１が撮像した画像を取得し、撮像制御部２１から受信した撮像タイミング及び発光タイミングに基づいて、取得した画像が発光画像か消光画像かを判別する。そして、取得した画像が発光画像の場合は運転者状態判定部２３に画像を送信し、取得した画像が消光画像の場合は外光状態判定部２４に画像を送信する。 The image acquisition unit 22 acquires an image captured by the near-infrared camera 11, and determines whether the acquired image is a light emitting image or a quenching image based on the imaging timing and the light emitting timing received from the imaging control unit 21. Then, when the acquired image is a light emitting image, the image is transmitted to the driver state determination unit 23, and when the acquired image is a quenching image, the image is transmitted to the external light state determination unit 24.

運転者状態判定部２３は、画像取得部２２から発光画像を取得した場合、発光画像から運転者の顔を検出し、運転者の眼の開眼度や顔向きといった顔情報を検出する。そして、検出した顔情報に基づいて、運転者が車両の運転に適さない状態か否かを判定する。ここで、運転に適さない状態とは、例えば、居眠り状態や脇見状態、運転不能状態（デッドマン状態）等である。
運転者状態判定部２３は、運転者が運転に適さない状態と判定した場合、制御装置３に対し、運転者が運転に適さない状態である旨の通知を行う。When the driver state determination unit 23 acquires a light emitting image from the image acquisition unit 22, the driver's face is detected from the light emitting image, and face information such as the degree of opening of the driver's eyes and the face orientation is detected. Then, based on the detected face information, it is determined whether or not the driver is in a state unsuitable for driving the vehicle. Here, the state unsuitable for driving is, for example, a dozing state, an inattentive state, an inoperable state (deadman state), or the like.
When the driver determines that the driver is not suitable for driving, the driver state determination unit 23 notifies the control device 3 that the driver is not suitable for driving.

外光状態判定部２４は、画像取得部２２から消光画像を取得した場合、消光画像から運転者の眼周辺の輝度を検出する。例えば、外光状態判定部２４は、眼の位置を基準にした予め定められた範囲に含まれる画素の平均輝度を検出する。なお、消光画像から運転者の眼を検出可能な場合は、消光画像から検出した眼の位置に基づいて眼周辺の輝度を検出し、消光画像から運転者の眼を検出不可能な場合、外光状態判定部２４は、運転者状態判定部２３が発光画像から検出した眼の位置を取得し、発光画像から検出した眼の位置に基づいて眼周辺の輝度を検出する。具体的には、図３を用いて説明する。 When the external light state determination unit 24 acquires a quenching image from the image acquisition unit 22, the external light state determination unit 24 detects the brightness around the driver's eye from the quenching image. For example, the external light state determination unit 24 detects the average brightness of pixels included in a predetermined range based on the position of the eye. If the driver's eye can be detected from the extinguished image, the brightness around the eye is detected based on the position of the eye detected from the extinguished image, and if the driver's eye cannot be detected from the extinguished image, the outside. The light state determination unit 24 acquires the position of the eye detected by the driver state determination unit 23 from the luminescent image, and detects the brightness around the eye based on the position of the eye detected from the luminescent image. Specifically, it will be described with reference to FIG.

図３は、本実施の形態１に係る運転状態監視装置２の眼周辺の輝度を検出する動作を示すフローチャートである。
外光情報取得部２４は、消光画像を取得すると、消光画像から運転者の眼を検出可能か判定し（ステップＳ００１）、消光画像から運転者の眼を検出可能な場合（ステップＳ００１：ＹＥＳ）、消光画像から検出した運転者の眼の位置に基づいて眼周辺の輝度を検出し（ステップＳ００２）、処理を終了する。一方、消光画像から運転者の眼を検出できない場合（ステップＳ００１：ＮＯ）、外光状態判定部２４は、運転者状態判定部２３が、発光画像から検出した眼の位置を、運転者状態判定部２３から取得し（Ｓ００３）、取得した発光画像から検出した眼の位置に基づいて眼周辺の輝度を検出する（Ｓ００４）。そして、処理を終了する。FIG. 3 is a flowchart showing an operation of detecting the brightness around the eyes of the operating condition monitoring device 2 according to the first embodiment.
When the external light information acquisition unit 24 acquires the extinguished image, it determines whether the driver's eye can be detected from the extinguished image (step S001), and if the driver's eye can be detected from the extinguished image (step S001: YES). , The brightness around the eye is detected based on the position of the driver's eye detected from the extinguished image (step S002), and the process is terminated. On the other hand, when the driver's eye cannot be detected from the extinguished image (step S001: NO), the external light state determination unit 24 determines the position of the eye detected by the driver state determination unit 23 from the light emission image. Acquired from the unit 23 (S003), the brightness around the eye is detected based on the position of the eye detected from the acquired luminescent image (S004). Then, the process ends.

このように、消光画像から運転者の眼を検出できない場合に、外光状態判定部２４は、発光画像から検出した眼の位置に基づいて眼周辺の輝度を検出することで、例えば、強い太陽光が運転者の顔にあたっており、消光画像に映る運転者の顔が白とびしてしまっている状況においても、外光の影響を受けない発光画像から検出される運転者の眼の位置に基づいて運転者の眼周辺の輝度を検出することができる。
なお、上記では消光画像から運転者の眼を検出できない場合に、発光画像から検出した眼の位置に基づいて眼周辺の輝度を検出することとしたが、外光状態判定部２４は、常に運転者状態判定部２３が、発光画像から検出した眼の位置に基づいて運転者の眼周辺の輝度を検出するようにしてもよい。In this way, when the driver's eye cannot be detected from the extinguished image, the external light state determination unit 24 detects the brightness around the eye based on the position of the eye detected from the emitted image, for example, the strong sun. Based on the position of the driver's eyes detected from the luminescent image that is not affected by external light, even when the light hits the driver's face and the driver's face reflected in the extinguished image is overexposed. It is possible to detect the brightness around the driver's eyes.
In the above, when the driver's eye cannot be detected from the extinguished image, the brightness around the eye is detected based on the position of the eye detected from the emitted image, but the external light state determination unit 24 is always in operation. The person state determination unit 23 may detect the brightness around the driver's eye based on the position of the eye detected from the luminescent image.

そして、運転者の眼周辺の輝度が輝度閾値以上であるかを判定し、運転者の眼周辺の輝度が輝度閾値以上の場合、運転者の眼に外光が照射されている状態であると判定する。
外光状態判定部２４は、運転者の眼に外光が照射されている状態であると判定した場合、制御装置３に対し、運転者の眼に外光が照射されている状態である旨の通知を行う。
また、外光状態判定部２４は、運転者の眼に外光が照射されている状態と判定した場合に、どちらの方向から運転者の顔に外光が照射されているかを検知するようにしてもよい。Then, it is determined whether the brightness around the driver's eyes is equal to or higher than the brightness threshold value, and when the brightness around the driver's eyes is equal to or higher than the brightness threshold value, it means that the driver's eyes are irradiated with external light. judge.
When the external light state determination unit 24 determines that the driver's eyes are irradiated with external light, the control device 3 is in a state where the driver's eyes are irradiated with external light. Notify.
Further, the external light state determination unit 24 detects from which direction the driver's face is irradiated with the external light when it is determined that the driver's eyes are irradiated with the external light. You may.

運転者の顔にできる影のパターンから外光の照射方向を判定する方法について図４を用いて説明する。
図４は、本実施の形態１に係る運転者の顔にできる影のパターンから外光の方向を判定する方法を説明するための図である。
図４Ａは、車両前方左上に太陽が存在するときの運転者の顔にできるパターンを表しており、一方図４Ｂは、車両前方右上に太陽が存在するときの運転者の顔にできる影のパターンである。図４Ａ、４Ｂからわかるように、運転者の顔の出っ張っている部位、特に鼻付近には外光の照射方向とは反対の方向に影ができる。したがって、外光の照射方向に対する運転者の顔にできる影のパターンを予め記憶させておくことで、消光画像から得られる運転者の顔にできる影のパターンから外光の照射方向を判定することが可能となる。A method of determining the irradiation direction of external light from the shadow pattern formed on the driver's face will be described with reference to FIG.
FIG. 4 is a diagram for explaining a method of determining the direction of external light from the shadow pattern formed on the driver's face according to the first embodiment.
FIG. 4A shows a pattern formed on the driver's face when the sun is present in the upper left of the front of the vehicle, while FIG. 4B shows a pattern of shadows formed on the driver's face when the sun is present in the upper right of the front of the vehicle. Is. As can be seen from FIGS. 4A and 4B, a shadow is formed on the protruding part of the driver's face, especially near the nose, in the direction opposite to the irradiation direction of the external light. Therefore, by storing in advance the shadow pattern formed on the driver's face with respect to the irradiation direction of the external light, the irradiation direction of the external light can be determined from the shadow pattern formed on the driver's face obtained from the quenching image. Is possible.

なお、外光状態取得部２４が外光の照射方向を判定した場合、外光の照射方向を制御装置３に対して出力する。 When the external light state acquisition unit 24 determines the irradiation direction of the external light, the external light irradiation direction is output to the control device 3.

制御装置３は、警報制御部３１と遮光制御部３２と車両制御部３３と表示制御部３４とを備えている。
警報制御部３１は、運転者状態判定部２３から運転者が運転に適さない状態の旨が通知された場合、警報装置４を制御し、運転者に対して警報を鳴らす。
遮光制御部３２は、外光状態判定部２４から運転者の眼に外光が照射されている状態である旨が通知された場合、車両に設けられた遮光装置５を制御し、運転者の眼に照射されている外光を遮光する。なお、遮光装置５は例えば、サンバイザーや透過率可変のフロントガラスである。
また、遮光制御部３２は、外光状態取得部から外光の照射方向を取得した場合は、当該外光の照射方向に基づいて遮光装置５を制御することで、より効率的に外光を遮光することが可能となる。The control device 3 includes an alarm control unit 31, a light-shielding control unit 32, a vehicle control unit 33, and a display control unit 34.
The alarm control unit 31 controls the alarm device 4 and sounds an alarm to the driver when the driver status determination unit 23 notifies that the driver is in an unsuitable state for driving.
When the external light state determination unit 24 notifies the driver's eyes that the external light is being irradiated, the light-shielding control unit 32 controls the light-shielding device 5 provided in the vehicle to control the driver's eyes. It blocks the external light shining on the eyes. The light-shielding device 5 is, for example, a sun visor or a windshield having a variable transmittance.
Further, when the light-shielding control unit 32 acquires the irradiation direction of the external light from the external light state acquisition unit, the light-shielding control unit 32 controls the light-shielding device 5 based on the irradiation direction of the external light to more efficiently generate the external light. It is possible to block light.

車両制御部３３は、運転者状態判定部２３から運転者が運転者に適さない状態の旨が通知された場合または外光状態判定部２４から運転者の眼に外光が照射されている状態の旨が通知された場合、自動ブレーキ装置６を制御し、自動ブレーキが通常よりも早めに作動するようにする。 The vehicle control unit 33 is in a state where the driver status determination unit 23 notifies that the driver is not suitable for the driver, or the external light condition determination unit 24 irradiates the driver's eyes with external light. When notified to that effect, the automatic braking device 6 is controlled so that the automatic braking operates earlier than usual.

表示制御部３４は、外光状態判定部２４から運転者の眼に外光が照射されている状態の旨が通知された場合、ヘッドアップディスプレイ装置７やカーナビゲーション装置８の表示の輝度を明るくし、運転者が眩しいと感じている状態においてもヘッドアップディスプレイ装置７やカーナビゲーション装置８が視認しやすいようにする。なお、輝度を明るくする以外にも、表示の色を視認しやすい色に変更してもよい。 When the display control unit 34 is notified by the external light state determination unit 24 that the driver's eyes are being irradiated with external light, the display control unit 34 brightens the display brightness of the head-up display device 7 and the car navigation device 8. However, the head-up display device 7 and the car navigation device 8 are made easy to see even when the driver feels dazzling. In addition to increasing the brightness, the display color may be changed to a color that is easy to see.

次に、運転者状態監視装置２のハードウェア構成例を説明する。
図５Ａ及び図５Ｂは、運転者状態監視装置２のハードウェア構成例を示す図である。
運転者状態監視装置２における撮像制御部２１、画像取得部２２、運転者状態判定部２３及び外光状態判定部２４の各機能は、処理回路によって実現される。即ち、運転者状態監視装置２は上記各機能を実現するための処理回路を備える。当該処理回路は、図５Ａに示すように専用のハードウェアである処理回路１００ａであってもよいし、図５Ｂに示すようにメモリ２００ｃに格納されているプログラムを実行するプロセッサ１００ｂであってもよい。Next, a hardware configuration example of the driver condition monitoring device 2 will be described.
5A and 5B are diagrams showing a hardware configuration example of the driver condition monitoring device 2.
Each function of the image pickup control unit 21, the image acquisition unit 22, the driver condition determination unit 23, and the external light condition determination unit 24 in the driver condition monitoring device 2 is realized by a processing circuit. That is, the driver condition monitoring device 2 includes a processing circuit for realizing each of the above functions. The processing circuit may be a processing circuit 100a which is dedicated hardware as shown in FIG. 5A, or may be a processor 100b which executes a program stored in the memory 200c as shown in FIG. 5B. good.

図５Ａに示すように、撮像制御部２１、画像取得部２２、運転者状態判定部２３及び外光状態判定部２４が専用のハードウェアである場合、処理回路１００ａは、例えば、単一回路、複合回路、プログラム化したプロセッサ、並列プログラム化したプロセッサ、ＡＳＩＣ（Ａｐｐｌｉｃａｔｉｏｎ Ｓｐｅｃｉｆｉｃ Ｉｎｔｅｇｒａｔｅｄ Ｃｉｒｃｕｉｔ）、ＦＰＧＡ（Ｆｉｅｌｄ−ｐｒｏｇｒａｍｍａｂｌｅ Ｇａｔｅ Ａｒｒａｙ）、またはこれらを組み合わせたものが該当する。撮像制御部２１、画像取得部２２、運転者状態判定部２３及び外光状態判定部２４の各部の機能をそれぞれ処理回路で実現してもよいし、各部の機能をまとめて１つの処理回路で実現してもよい。 As shown in FIG. 5A, when the image pickup control unit 21, the image acquisition unit 22, the driver state determination unit 23, and the external light state determination unit 24 are dedicated hardware, the processing circuit 100a is, for example, a single circuit. It corresponds to a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-programmable Gate Array), or a combination thereof. The functions of the image pickup control unit 21, the image acquisition unit 22, the driver state determination unit 23, and the external light state determination unit 24 may be realized by processing circuits, or the functions of each unit may be combined into one processing circuit. It may be realized.

図５Ｂに示すように、撮像制御部２１、画像取得部２２、運転者状態判定部２３及び外光状態判定部２４がプロセッサ１００ｂである場合、各部の機能は、ソフトウェア、ファームウェア、またはソフトウェアとファームウェアとの組み合わせにより実現される。ソフトウェアまたはファームウェアはプログラムとして記述され、メモリ１００ｃに格納される。プロセッサ１００ｂは、メモリ１００ｃに記憶されたプログラムを読み出して実行することにより、撮像制御部２１、画像取得部２２、運転者状態判定部２３及び外光状態判定部２４の各機能を実現する。即ち、撮像制御部２１、画像取得部２２、運転者状態判定部２３及び外光状態判定部２４は、プロセッサ１００ｂにより実行されるときに、図３及び後述する図６、図８に示す各ステップが結果的に実行されることになる。また、これらのプログラムは、撮像制御部２１、画像取得部２２、運転者状態判定部２３及び外光状態判定部２４の手順または方法をコンピュータに実行させるものであるともいえる。 As shown in FIG. 5B, when the image pickup control unit 21, the image acquisition unit 22, the driver state determination unit 23, and the external light state determination unit 24 are processors 100b, the functions of each unit are software, firmware, or software and firmware. It is realized by the combination with. The software or firmware is described as a program and stored in the memory 100c. The processor 100b realizes each function of the image pickup control unit 21, the image acquisition unit 22, the driver state determination unit 23, and the external light state determination unit 24 by reading and executing the program stored in the memory 100c. That is, when the image pickup control unit 21, the image acquisition unit 22, the driver state determination unit 23, and the external light state determination unit 24 are executed by the processor 100b, each step shown in FIG. 3 and FIGS. Will eventually be executed. Further, it can be said that these programs cause the computer to execute the procedures or methods of the image pickup control unit 21, the image acquisition unit 22, the driver state determination unit 23, and the external light state determination unit 24.

ここで、プロセッサ１００ｂとは、例えば、ＣＰＵ（Ｃｅｎｔｒａｌ Ｐｒｏｃｅｓｓｉｎｇ Ｕｎｉｔ）、処理装置、演算装置、プロセッサ、マイクロプロセッサ、マイクロコンピュータ、またはＤＳＰ（Ｄｉｇｉｔａｌ Ｓｉｇｎａｌ Ｐｒｏｃｅｓｓｏｒ）などのことである。メモリ１００ｃは、例えば、ＲＡＭ（Ｒａｎｄｏｍ Ａｃｃｅｓｓ Ｍｅｍｏｒｙ）、ＲＯＭ（Ｒｅａｄ Ｏｎｌｙ Ｍｅｍｏｒｙ）、フラッシュメモリ、ＥＰＲＯＭ（Ｅｒａｓａｂｌｅ Ｐｒｏｇｒａｍｍａｂｌｅ ＲＯＭ）、ＥＥＰＲＯＭ（Ｅｌｅｃｔｒｉｃａｌｌｙ ＥＰＲＯＭ）等の不揮発性または揮発性の半導体メモリであってもよいし、ハードディスク、フレキシブルディスク等の磁気ディスクであってもよいし、ミニディスク、ＣＤ（Ｃｏｍｐａｃｔ Ｄｉｓｃ）、ＤＶＤ（Ｄｉｇｉｔａｌ Ｖｅｒｓａｔｉｌｅ Ｄｉｓｃ）等の光ディスクであってもよい。 Here, the processor 100b is, for example, a CPU (Central Processing Unit), a processing device, a computing device, a processor, a microprocessor, a microcomputer, a DSP (Digital Signal Processor), or the like. The memory 100c may be, for example, a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Program ROM), or an EPROM (Electrical EPROM). However, it may be a magnetic disk such as a hard disk or a flexible disk, or an optical disk such as a mini disk, a CD (Compact Disc), or a DVD (Digital Versaille Disc).

次に、運転者状態監視装置２の動作について、図６を用いて説明する。
図６は、実施の形態１に係る運転者状態監視装置２の動作を示すフローチャートである。
運転者状態監視装置２の電源がＯＮになった場合、撮像制御部２１は、近赤外線発光素子１２のパルス発光タイミングを決定し、近赤外線発光素子１２をパルス発光させる（ステップＳ１０１）。そして、撮像制御部２１は、近赤外線発光素子１２の発光期間および消光期間に近赤外線カメラ１１が画像を撮像するように、近赤外線発光素子１２の発光タイミングに近赤外線カメラ１１の撮像タイミングを同期させ（ステップＳ１０２）、撮像タイミング及び発光タイミングを画像取得部２２に出力する（ステップＳ１０３）。画像取得部２２は近赤外線カメラが撮像した画像を取得し（ステップＳ１０４）、撮像制御部２１から受信した撮像タイミング及び発光タイミングに基づいて取得した画像が発光画像か消光画像かを判定する（ステップＳ１０５）。画像取得部２２が取得した画像が発光画像であった場合（ステップＳ１０５：ＹＥＳ）、発光画像を運転者状態判定部２３に送信し、運転者状態判定部２３は送信された発光画像に基づいて運転者が運転に適さない状態か否かを判定する（ステップＳ１０６）。そして、ステップＳ１０６での判定結果を制御装置３に出力し（Ｓ１０７）、ステップＳ１０４の処理に戻る。一方、画像取得部２２が取得した画像が消光画像であった場合（ステップＳ１０５：ＮＯ）、消光画像を外光状態判定部２４に送信し、外光状態判定部２４は送信された消光画像に基づいて運転者の眼に外光が照射されている状態か否かを判定する（ステップＳ１０８）。そして、ステップＳ１０６での判定結果を制御装置３に出力し（Ｓ１０７）、ステップＳ１０４の処理に戻る。Next, the operation of the driver condition monitoring device 2 will be described with reference to FIG.
FIG. 6 is a flowchart showing the operation of the driver condition monitoring device 2 according to the first embodiment.
When the power of the driver condition monitoring device 2 is turned on, the image pickup control unit 21 determines the pulse emission timing of the near infrared light emitting element 12, and causes the near infrared light emitting element 12 to emit pulse light (step S101). Then, the image pickup control unit 21 synchronizes the imaging timing of the near-infrared camera 11 with the emission timing of the near-infrared light emitting element 12 so that the near-infrared camera 11 captures an image during the emission period and the quenching period of the near-infrared light emitting element 12. (Step S102), and the imaging timing and the light emission timing are output to the image acquisition unit 22 (step S103). The image acquisition unit 22 acquires an image captured by the near-infrared camera (step S104), and determines whether the acquired image is a light emitting image or a quenching image based on the imaging timing and the light emitting timing received from the imaging control unit 21 (step S104). S105). When the image acquired by the image acquisition unit 22 is a light emitting image (step S105: YES), the light emitting image is transmitted to the driver state determination unit 23, and the driver state determination unit 23 is based on the transmitted light emission image. It is determined whether or not the driver is in a state unsuitable for driving (step S106). Then, the determination result in step S106 is output to the control device 3 (S107), and the process returns to the process in step S104. On the other hand, when the image acquired by the image acquisition unit 22 is a quenching image (step S105: NO), the quenching image is transmitted to the external light state determination unit 24, and the external light state determination unit 24 is used for the transmitted quenching image. Based on this, it is determined whether or not the driver's eyes are irradiated with external light (step S108). Then, the determination result in step S106 is output to the control device 3 (S107), and the process returns to the process in step S104.

以上のように、実施の形態１によれば、車両の運転者の顔に向けて近赤外線を照射する近赤外発光素子および運転者の顔を撮像する近赤外線カメラを制御し、近赤外線発光素子をパルス発光させるとともに、近赤外線発光素子の発光期間および消光期間と近赤外線カメラの撮像タイミングとを同期させる撮像制御部と、発光期間に近赤外線カメラが撮像した発光画像および消光期間に近赤外線カメラが撮像した消光画像を取得する画像取得部と、発光画像から運転者の顔情報を検出し、顔情報に基づいて運転者の状態を判定する運転者状態判定部と、消光画像から運転者の眼周辺の輝度を検出し、輝度が輝度閾値以上の場合、運転者の眼に外光が照射されている状態と判定する外光状態判定部と、を備える構成としたので、近赤外線カメラで運転者の顔を鮮明に撮像しつつ、外光の情報を取得することができる。 As described above, according to the first embodiment, the near-infrared light emitting element that irradiates the driver's face of the vehicle with near-infrared rays and the near-infrared camera that captures the driver's face are controlled to emit near-infrared rays. An imaging control unit that causes the element to emit pulse light and synchronizes the emission period and extinguishing period of the near-infrared light emitting element with the imaging timing of the near-infrared camera, and the emission image captured by the near-infrared camera during the emission period and near-infrared ray during the extinguishing period. An image acquisition unit that acquires the extinguished image captured by the camera, a driver state determination unit that detects the driver's face information from the emitted image and determines the driver's condition based on the face information, and a driver from the extinguished image. A near-infrared camera is provided because it is configured to include an external light state determination unit that detects the brightness around the eyes and determines that the driver's eyes are irradiated with external light when the brightness is equal to or higher than the brightness threshold. It is possible to acquire information on external light while clearly capturing the driver's face.

実施の形態２
図７は、本実施の形態２に係る運転者状態監視装置２Ｂの構成を示すブロック図である。
本実施の形態２に係る運転者状態監視装置２Ｂは、運転者が通常よりも眼を細めているときに、運転者が眠たくて眼を細めているのか、外光が眩しくて眼を細めているのかを判別するようにした。Embodiment 2
FIG. 7 is a block diagram showing the configuration of the driver condition monitoring device 2B according to the second embodiment.
In the driver condition monitoring device 2B according to the second embodiment, when the driver is squinting more than usual, the driver may be sleepy and squinting, or the outside light is dazzling and the eyes are squinted. I tried to determine if it was there.

図８は、本実施の形態２に係る運転者状態監視装置２Ｂにおける運転者状態判定部２３Ｂの動作を示すフローチャートである。
運転者状態判定部２３Ｂは、画像取得部２２から発光画像を取得した場合、取得した発光画像から逐次運転者の眼の開眼度を検出する（ステップＳ２０１）。そして、開眼度が閾値以下の状態が予め定められた時間継続したか否かを判定し（ステップＳ２０２）、開眼度が開眼度閾値以下の状態が予め定められた時間継続したと判定した場合（ステップＳ２０２：ＹＥＳ）、外光状態判定部２４Ｂは、消光画像から運転者の眼周辺の輝度を検出する（ステップＳ２０３）。一方、開眼度が開眼度閾値以下の状態が予め定められた時間継続しなかった場合（ステップ２０２：ＮＯ）、ステップ２０１の処理に戻る。外光状態判定部２４Ｂは、ステップＳ２０３で検出した運転者の眼周辺の輝度が輝度閾値以下か否かを判定し（ステップＳ２０４）、運転者の眼周辺の輝度が輝度閾値以下の場合（ステップＳ２０４：ＹＥＳ）、運転者が居眠り状態であると判定する（ステップＳ２０５）。一方、運転者の眼周辺の輝度が輝度閾値より高い場合（ステップＳ２０４：ＮＯ）、運転者が外光を眩しいと感じている状態であると判定する（ステップＳ２０６）。そして、処理を終了する。FIG. 8 is a flowchart showing the operation of the driver condition determination unit 23B in the driver condition monitoring device 2B according to the second embodiment.
When the driver state determination unit 23B acquires a light emission image from the image acquisition unit 22, the driver state determination unit 23B sequentially detects the degree of eye opening of the driver's eyes from the acquired light emission image (step S201). Then, it is determined whether or not the state in which the eye opening degree is below the threshold value has continued for a predetermined time (step S202), and it is determined that the state in which the eye opening degree is below the threshold value has continued for a predetermined time (step S202). Step S202: YES), the external light state determination unit 24B detects the brightness around the driver's eye from the extinguished image (step S203). On the other hand, when the state in which the eye opening degree is equal to or less than the eye opening degree threshold value does not continue for a predetermined time (step 202: NO), the process returns to the process of step 201. The external light state determination unit 24B determines whether or not the brightness around the driver's eyes detected in step S203 is equal to or less than the luminance threshold value (step S204), and when the brightness around the driver's eyes is equal to or less than the luminance threshold value (step S204). S204: YES), it is determined that the driver is in a dozing state (step S205). On the other hand, when the brightness around the driver's eyes is higher than the brightness threshold value (step S204: NO), it is determined that the driver feels that the outside light is dazzling (step S206). Then, the process ends.

以上のように、実施の形態２の運転者状態判定部２３Ｂは、開眼度が閾値以下の状態が予め定められた時間継続した場合に、外光状態判定部２４Ｂが検出する運転者の眼周辺の輝度が輝度閾値以下か否かを判定し、運転者の眼周辺の輝度が輝度閾値以下の場合は運転者が居眠り状態と判定し、運転者の眼周辺の輝度が輝度閾値より高い場合は運転者が外光を眩しいと感じている状態であると判定する。これによって、運転者が通常よりも眼を細めているときに、運転者が眠たくて眼を細めているのか、外光が眩しくて眼を細めているのかを判別するようにした。 As described above, the driver state determination unit 23B of the second embodiment detects the area around the driver's eye that is detected by the external light state determination unit 24B when the state in which the eye opening degree is equal to or less than the threshold value continues for a predetermined time. If the brightness around the driver's eye is less than or equal to the brightness threshold, it is determined that the driver is in a dozing state, and if the brightness around the driver's eye is higher than the brightness threshold. It is determined that the driver feels that the outside light is dazzling. As a result, when the driver is squinting more than usual, it is possible to determine whether the driver is sleepy and squinting, or the outside light is dazzling and squinting.

１ 撮像装置、２、２Ａ 運転者状態監視装置 、３ 制御装置、４ 警報装置、５ 遮光装置、６ 自動ブレーキ装置、７ ヘッドアップディスプレイ装置、８ カーナビゲーション装置、１１ 近赤外線カメラ、１２ 近赤外線発光素子、２１ 撮像制御部、２２ 画像取得部、２３、２３Ｂ 運転者状態判定部、２４、２４Ｂ 外光状態判定部、３１ 警報制御部、３２ 遮光制御部、３３ 車両制御部、３４ 表示制御部、１００ａ 処理回路、１００ｂ プロセッサ、１００ｃ メモリ 1 Imaging device, 2, 2A Driver status monitoring device, 3 Control device, 4 Alarm device, 5 Shading device, 6 Automatic braking device, 7 Head-up display device, 8 Car navigation device, 11 Near-infrared camera, 12 Near-infrared light emission Element, 21 Imaging control unit, 22 Image acquisition unit, 23, 23B Driver status determination unit, 24, 24B External light condition determination unit, 31 Alarm control unit, 32 Shading control unit, 33 Vehicle control unit, 34 Display control unit, 100a processing circuit, 100b processor, 100c memory

Claims (7)

車両の運転者の顔に向けて近赤外線を照射する近赤外発光素子および前記運転者の顔を撮像する近赤外線カメラを制御し、前記近赤外線発光素子をパルス発光させるとともに、前記近赤外線発光素子の発光期間および消光期間と前記近赤外線カメラの撮像タイミングとを同期させる撮像制御部と、
前記発光期間に前記近赤外線カメラが撮像した発光画像および前記消光期間に前記近赤外線カメラが撮像した消光画像を取得する画像取得部と、
前記発光画像から前記運転者の顔情報を検出し、前記顔情報に基づいて前記運転者の状態を判定する運転者状態判定部と、
前記消光画像から前記運転者の眼周辺の輝度を検出し、前記輝度が輝度閾値以上の場合、前記運転者の眼に外光が照射されている状態と判定する外光状態判定部と、
を備える運転者状態監視装置。A near-infrared light emitting element that irradiates near infrared rays toward the driver's face of a vehicle and a near infrared camera that images the driver's face are controlled to cause the near infrared light emitting element to emit pulses and emit the near infrared rays. An imaging control unit that synchronizes the light emitting period and extinguishing period of the element with the imaging timing of the near-infrared camera.
An image acquisition unit that acquires a light emitting image captured by the near-infrared camera during the light emitting period and a quenching image captured by the near infrared camera during the quenching period.
A driver state determination unit that detects the driver's face information from the light emission image and determines the driver's state based on the face information.
An external light state determination unit that detects the brightness around the driver's eyes from the extinguished image and determines that the driver's eyes are irradiated with external light when the brightness is equal to or higher than the brightness threshold value.
A driver condition monitoring device equipped with. 前記運転者状態判定部は、前記発光画像から検出された前記運転者の眼の開眼度が開眼度閾値以下の状態が、予め定められた時間以上継続した場合、前記外光状態判定部が検出した前記運転者の眼周辺の輝度が輝度閾値以下であれば前記運転者が居眠り状態と判定する請求項１記載の運転者状態監視装置。 The driver state determination unit detects when the driver's eye opening degree detected from the light emission image continues to be equal to or less than the eye opening degree threshold value for a predetermined time or longer. The driver condition monitoring device according to claim 1, wherein if the brightness around the eyes of the driver is equal to or less than the brightness threshold value, the driver determines that the driver is in a dozing state. 前記運転者状態判定部は、前記発光画像から前記運転者の眼の位置を検出し、
前記外光状態判定部は、前記運転者状態判定部が検出した前記運転者の眼の位置に基づいて前記運転者の眼周辺の輝度を検出する請求項１記載の運転者状態監視装置。The driver state determination unit detects the position of the driver's eyes from the light emission image and determines the position of the driver's eyes.
The driver condition monitoring device according to claim 1, wherein the external light condition determination unit detects the brightness around the driver's eyes based on the position of the driver's eyes detected by the driver condition determination unit. 前記外光状態判定部は、前記消光画像から前記運転者の眼の位置を検出できる場合は、前記消光画像から検出された眼の位置に基づいて記運転者の眼周辺の輝度を検出し、前記消光画像から前記運転者の眼の位置を検出できない場合、前記運転者状態判定部が検出した前記運転者の眼の位置に基づいて前記運転者の眼周辺の輝度を検出する請求項３記載の運転者状態監視装置。 When the position of the driver's eye can be detected from the extinguished image, the external light state determining unit detects the brightness around the driver's eye based on the position of the eye detected from the extinguished image. The third aspect of claim 3, wherein when the position of the driver's eye cannot be detected from the extinguished image, the brightness around the driver's eye is detected based on the position of the driver's eye detected by the driver state determination unit. Driver condition monitoring device. 前記外光状態判定部は、前記消光画像から前記運転者の顔の輝度を検出し、前記運転者の顔の輝度に基づいて外光の照射方向を判定する請求項１記載の運転者状態監視装置。 The driver condition monitoring according to claim 1, wherein the external light state determination unit detects the brightness of the driver's face from the extinguished image and determines the irradiation direction of the external light based on the brightness of the driver's face. Device. 前記運転者の眼に外光が照射されている状態と判定した場合、外光を遮光するように前記車両に備えられた遮光装置を制御する遮光制御部を更に備え、前記遮光制御部は、前記外光状態判定部が判定した外光の照射方向に応じて遮光装置を制御する請求項５記載の運転者状態監視装置。 When it is determined that the driver's eyes are irradiated with external light, the light-shielding control unit further includes a light-shielding control unit that controls a light-shielding device provided in the vehicle so as to block the external light. The driver condition monitoring device according to claim 5, wherein the light-shielding device is controlled according to the irradiation direction of the external light determined by the external light state determination unit. 車両の運転者の顔に向けて近赤外線を照射する近赤外発光素子および前記運転者の顔を撮像する近赤外線カメラを制御し、前記近赤外線発光素子をパルス発光させるとともに、前記近赤外線発光素子の発光期間および消光期間に同期して前記近赤外線カメラを撮像させるステップと、
前記発光期間に前記近赤外線カメラが撮像した発光画像および前記消光期間に前記近赤外線カメラが撮像した消光画像を取得するステップと、
前記発光画像から前記運転者の顔情報を検出し、前記顔情報に基づいて前記運転者の状態を判定するステップと、
前記消光画像から前記運転者の眼周辺の輝度を検出し、前記輝度が輝度閾値以上の場合、前記運転者の眼に外光が照射されている状態と判定するステップと、
を備える運転者状態監視方法。A near-infrared light emitting element that irradiates near infrared rays toward the driver's face of a vehicle and a near infrared camera that images the driver's face are controlled to cause the near infrared light emitting element to emit pulses and emit the near infrared rays. The step of imaging the near-infrared camera in synchronization with the light emitting period and the extinguishing period of the element,
A step of acquiring a light emitting image captured by the near-infrared camera during the light emitting period and a quenching image captured by the near infrared camera during the quenching period.
A step of detecting the driver's face information from the luminescent image and determining the driver's state based on the face information.
A step of detecting the brightness around the driver's eye from the extinguished image, and determining that the driver's eye is irradiated with external light when the brightness is equal to or higher than the brightness threshold.
A driver condition monitoring method.
JP2021506843A 2019-03-18 2019-03-18 DRIVER'S CONDITION MONITORING DEVICE AND DRIVER'S CONDITION MONITORING METHOD Active JP7158561B2 (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2019/011119 WO2020188689A1 (en) 2019-03-18 2019-03-18 Driver state monitoring device and driver state monitoring method

Publications (2)

Publication Number Publication Date
JPWO2020188689A1 true JPWO2020188689A1 (en) 2021-10-14
JP7158561B2 JP7158561B2 (en) 2022-10-21

Family

ID=72520598

Family Applications (1)

Application Number Title Priority Date Filing Date
JP2021506843A Active JP7158561B2 (en) 2019-03-18 2019-03-18 DRIVER'S CONDITION MONITORING DEVICE AND DRIVER'S CONDITION MONITORING METHOD

Country Status (2)

Country Link
JP (1) JP7158561B2 (en)
WO (1) WO2020188689A1 (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20240179383A1 (en) * 2022-11-30 2024-05-30 Gentex Corporation Monitoring system

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2006251926A (en) * 2005-03-08 2006-09-21 Nissan Motor Co Ltd Direct light detecting device
JP2017143889A (en) * 2016-02-15 2017-08-24 ルネサスエレクトロニクス株式会社 Eye opening degree detection system, doze detection system, automatic shutter system, eye-opening degree detection method and eye-opening degree detection program
JP2017161965A (en) * 2016-03-07 2017-09-14 オムロンオートモーティブエレクトロニクス株式会社 Face image processing apparatus
JP2017219885A (en) * 2016-06-02 2017-12-14 株式会社Ｓｕｂａｒｕ Driver condition monitoring device

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2006251926A (en) * 2005-03-08 2006-09-21 Nissan Motor Co Ltd Direct light detecting device
JP2017143889A (en) * 2016-02-15 2017-08-24 ルネサスエレクトロニクス株式会社 Eye opening degree detection system, doze detection system, automatic shutter system, eye-opening degree detection method and eye-opening degree detection program
JP2017161965A (en) * 2016-03-07 2017-09-14 オムロンオートモーティブエレクトロニクス株式会社 Face image processing apparatus
JP2017219885A (en) * 2016-06-02 2017-12-14 株式会社Ｓｕｂａｒｕ Driver condition monitoring device

Also Published As

Publication number Publication date
WO2020188689A1 (en) 2020-09-24
JP7158561B2 (en) 2022-10-21

Similar Documents

Publication Publication Date Title
JP4853389B2 (en) Face image capturing device
CN105313762B (en) Illuminator
JP5212927B2 (en) Face shooting system
JP6175430B2 (en) Method and apparatus for recognizing an object in a surrounding area of a vehicle
US10511789B2 (en) Infrared imaging device, control method thereof, and vehicle
CN106458083B (en) Vehicle head lamp control device
JP6168870B2 (en) Imaging apparatus, camera system, and control method
JP2013175914A (en) Imaging controller and program
JP2015069369A (en) Operation input device, operation input method, and program
US11093768B2 (en) Occupant monitoring device and occupant monitoring method
CN109561817B (en) Electronic endoscope system
JP2008109412A (en) Imaging apparatus, imaging apparatus control method, and computer program
WO2020188689A1 (en) Driver state monitoring device and driver state monitoring method
CN111398975A (en) Active sensor, object recognition system, vehicle, and vehicle lamp
JP2006252363A (en) Detector for object around vehicle, and method of detecting object around vehicle
JP4451195B2 (en) Gaze detection device
CN113691730B (en) Task switching control method and device of camera, medium and electronic equipment
US10829042B2 (en) Imaging apparatus with image sensor for detecting light in an atmospheric peak and vehicle having same
JP5206190B2 (en) Driver monitoring device
JP6541510B2 (en) Image pickup apparatus, control method thereof and program
JP6123137B2 (en) Camera device
JP5670509B2 (en) Detection device for opening and closing part
JP2017124037A (en) Visual line detection device
EP4010099B1 (en) Systems and methods for detecting specular surfaces
JP5437614B2 (en) Illumination apparatus and face authentication system using the same

Legal Events

Date Code Title Description
A621 Written request for application examination

Free format text: JAPANESE INTERMEDIATE CODE: A621

Effective date: 20210212

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20220426

A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20220519

TRDD Decision of grant or rejection written
A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

Effective date: 20220913

A61 First payment of annual fees (during grant procedure)

Free format text: JAPANESE INTERMEDIATE CODE: A61

Effective date: 20221011

R150 Certificate of patent or registration of utility model

Ref document number: 7158561

Country of ref document: JP

Free format text: JAPANESE INTERMEDIATE CODE: R150