JP4313717B2 - Gaze detection device - Google Patents

Gaze detection device Download PDF

Info

Publication number
JP4313717B2
JP4313717B2 JP2004117723A JP2004117723A JP4313717B2 JP 4313717 B2 JP4313717 B2 JP 4313717B2 JP 2004117723 A JP2004117723 A JP 2004117723A JP 2004117723 A JP2004117723 A JP 2004117723A JP 4313717 B2 JP4313717 B2 JP 4313717B2
Authority
JP
Japan
Prior art keywords
infrared
driver
line
photographing
sight
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
JP2004117723A
Other languages
Japanese (ja)
Other versions
JP2005296382A (en
Inventor
昭夫 高橋
亮人 木俣
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Honda Motor Co Ltd
Original Assignee
Honda Motor Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Honda Motor Co Ltd filed Critical Honda Motor Co Ltd
Priority to JP2004117723A priority Critical patent/JP4313717B2/en
Publication of JP2005296382A publication Critical patent/JP2005296382A/en
Application granted granted Critical
Publication of JP4313717B2 publication Critical patent/JP4313717B2/en
Anticipated expiration legal-status Critical
Expired - Fee Related legal-status Critical Current

Links

Images

Landscapes

  • Eye Examination Apparatus (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Description

本発明は、視線検出装置に関する。   The present invention relates to a line-of-sight detection device.

従来、例えば可視光領域にて撮影可能なカメラ、あるいは、赤外線領域にて撮影可能なカメラを用いて視線を検出する方法が知られており（例えば、非特許文献１参照）、可視光領域にて撮影可能なカメラを用いる方法では、カメラにより撮影した眼球像から眼の特徴点（目尻、瞼など）および虹彩を検出し、特徴点に対する虹彩の相対位置に基づき視線を検出するようになっている。ただし、この方法では、視線方向の変化に対する虹彩の位置変化が相対的に小さいことから、検出精度を向上させることが困難であるという問題がある。
このような問題に対して、赤外線領域にて撮影可能なカメラを用いる方法では、赤外線の点光源から眼球に赤外線を照射し、撮影画像から検出された瞳孔中心および角膜表面における反射像（プルキニエ像）の相対位置に基づき視線を検出するようになっており、この方法では相対的に高精度の視線検出を行うことが可能である。
大野健彦著，「視線を用いたインタフェース」，情報処理，社団法人情報処理学会，２００３年７月，第４４巻，第７号，ｐ．７２６−７３２ Conventionally, for example, a method for detecting a line of sight using a camera that can shoot in the visible light region or a camera that can shoot in the infrared region is known (see, for example, Non-Patent Document 1). In the method using a camera that can be photographed, eye feature points (eye corners, eyelids, etc.) and iris are detected from an eyeball image photographed by the camera, and the line of sight is detected based on the relative position of the iris with respect to the feature point. Yes. However, this method has a problem that it is difficult to improve the detection accuracy because the change in the position of the iris with respect to the change in the line-of-sight direction is relatively small.
To solve this problem, the method using a camera capable of photographing in the infrared region irradiates the eyeball with infrared rays from an infrared point light source, and the reflected image (Purkinje image) at the pupil center and corneal surface detected from the photographed image. ) Is detected based on the relative position of (). With this method, it is possible to detect the line of sight with relatively high accuracy.
Takeo Ohno, “Interface Using Eyes,” Information Processing, Information Processing Society of Japan, July 2003, Vol. 44, No. 7, p. 726-732

ところで、上記従来技術において、赤外線領域にて撮影可能なカメラを用いる方法では、運転者が眼鏡を装着している際に、赤外線の点光源から眼球に照射される赤外線が運転者の顔の向きに応じて眼鏡で反射されてしまい、角膜表面の反射像を明確に撮影することができない場合があり、この場合には視線検出の検出精度が低下あるいは視線検出が不可となる虞がある。
本発明は上記事情に鑑みてなされたもので、眼鏡の有無に関わらず視線を検出することが可能な視線検出装置を提供することを目的とする。 By the way, in the above-described conventional technique, in the method using a camera capable of photographing in the infrared region, when the driver wears spectacles, the infrared ray irradiated to the eyeball from the infrared point light source is the direction of the driver's face. Accordingly, there is a case where the reflected image of the cornea surface cannot be clearly taken, and in this case, there is a possibility that the detection accuracy of the gaze detection is lowered or the gaze detection is not possible.
The present invention has been made in view of the above circumstances, and an object thereof is to provide a line-of-sight detection device capable of detecting a line of sight regardless of the presence or absence of glasses.

上記課題を解決して係る目的を達成するために、請求項１に記載の本発明の視線検出装置は、自車両の運転者の眼球あるいは顔を可視光領域にて撮影する可視光撮影手段（例えば、実施の形態での可視光カメラ２１）と、自車両の運転者の眼球に赤外線を照射する赤外線照射手段（例えば、実施の形態での赤外線発光ダイオード（赤外線ＬＥＤ）２３）と、自車両の運転者の眼球を赤外線領域にて撮影する赤外線撮影手段（例えば、実施の形態での赤外線カメラ２２）と、前記可視光撮影手段の撮影により得られる可視光画像に基づき運転者が眼鏡を装着しているか否かを判定する眼鏡装着判定手段（例えば、実施の形態での眼鏡装着判定部３１）と、前記眼鏡装着判定手段の判定結果に応じて、少なくとも前記可視光撮影手段の撮影により得られる可視光画像または前記赤外線撮影手段の撮影により得られる赤外線画像に基づき運転者の視線を検出する視線検出手段（例えば、実施の形態での可視光視線算出部３２および赤外線視線算出部３４）とを備えることを特徴としている。   In order to solve the above-described problems and achieve the object, the line-of-sight detection device of the present invention according to claim 1 is a visible light photographing means for photographing an eyeball or a face of a driver of the own vehicle in a visible light region. For example, the visible light camera 21) in the embodiment, the infrared irradiation means (for example, the infrared light emitting diode (infrared LED) 23 in the embodiment) that irradiates the driver's eyeball with infrared rays, and the own vehicle An infrared photographing means for photographing the driver's eyeball in the infrared region (for example, the infrared camera 22 in the embodiment), and the driver wears glasses based on a visible light image obtained by photographing by the visible light photographing means According to the spectacle wearing determination means (for example, the spectacle wearing determination unit 31 in the embodiment) for determining whether or not the eyewear is worn, and at least by the photographing of the visible light photographing means according to the determination result of the spectacle wearing determination means. Line-of-sight detection means (for example, visible-light line-of-sight calculation unit 32 and infrared line-of-sight calculation unit 34 in the embodiment) for detecting a driver's line of sight based on a visible light image or an infrared image obtained by photographing with the infrared photographing means; It is characterized by having.

上記構成の視線検出装置によれば、運転者が眼鏡を装着しているか否かに応じて少なくとも可視光画像または赤外線画像に基づき運転者の視線を検出することにより、視線検出が不可となる状態が発生することを抑制しつつ、視線検出の検出精度が低下することを防止することができる。   According to the gaze detection device having the above configuration, the gaze detection is disabled by detecting the gaze of the driver based on at least a visible light image or an infrared image depending on whether or not the driver wears glasses. It is possible to prevent the detection accuracy of the line-of-sight detection from being lowered while suppressing the occurrence of.

さらに、請求項２に記載の本発明の視線検出装置は、前記赤外線撮影手段の撮影により得られる赤外線画像に基づき、前記赤外線照射手段により運転者の眼球に照射された赤外線が運転者の眼鏡で反射したか否かを判定する赤外線反射判定手段（例えば、実施の形態での赤外線反射判定部３３）を備え、前記眼鏡装着判定手段により運転者が眼鏡を装着していると判定され、かつ、前記赤外線反射判定手段により運転者の眼球に照射された赤外線が運転者の眼鏡で反射したと判定された場合に、前記視線検出手段は前記可視光撮影手段の撮影により得られる可視光画像に基づき運転者の視線を検出することを特徴としている。   Furthermore, the line-of-sight detection device according to the second aspect of the present invention is based on an infrared image obtained by photographing with the infrared photographing means, and the infrared rays irradiated to the driver's eyeball by the infrared radiating means are the glasses of the driver. Infrared reflection determination means (for example, infrared reflection determination unit 33 in the embodiment) for determining whether or not the light is reflected, it is determined by the spectacle wearing determination means that the driver is wearing spectacles, and When it is determined by the infrared reflection determination means that the infrared ray applied to the driver's eyeball is reflected by the driver's glasses, the line-of-sight detection means is based on a visible light image obtained by photographing with the visible light photographing means. It is characterized by detecting the driver's line of sight.

上記構成の視線検出装置によれば、赤外線撮影手段の撮影により得られる赤外線画像において、赤外線照射手段により運転者の眼球に照射された赤外線の角膜表面における反射像（プルキニエ像）を検出することができない場合であっても、可視光画像に基づき運転者の視線を継続して検出することができる。   According to the line-of-sight detection device having the above-described configuration, it is possible to detect a reflection image (Purkinje image) of the infrared cornea surface irradiated on the driver's eyeball by the infrared irradiation unit in the infrared image obtained by the infrared imaging unit. Even if this is not possible, the driver's line of sight can be continuously detected based on the visible light image.

さらに、請求項３に記載の本発明の視線検出装置は、前記眼鏡装着判定手段により運転者が眼鏡を装着していないと判定された場合に、前記視線検出手段は前記赤外線撮影手段の撮影により得られる赤外線画像に基づき運転者の視線を検出することを特徴としている。   Further, in the eye gaze detection device according to the third aspect of the present invention, when the eyeglass wearing judgment means judges that the driver is not wearing eyeglasses, the eye gaze detecting means is obtained by photographing with the infrared imaging means. It is characterized by detecting the driver's line of sight based on the obtained infrared image.

上記構成の視線検出装置によれば、運転者が眼鏡を装着していない場合、つまり赤外線照射手段により運転者の眼球に照射された赤外線が運転者の眼鏡で反射される虞がない場合には、赤外線画像に基づき運転者の視線を検出することにより、視線検出の検出精度が低下することを防止することができる。   According to the line-of-sight detection device having the above configuration, when the driver does not wear spectacles, that is, when there is no possibility that the infrared rays irradiated to the driver's eyeball by the infrared irradiation means are reflected by the driver's glasses. By detecting the driver's line of sight based on the infrared image, it is possible to prevent the detection accuracy of the line of sight detection from being lowered.

本発明の視線検出装置によれば、視線検出が不可となる状態が発生することを抑制しつつ、視線検出の検出精度が低下することを防止することができる。
さらに、請求項２に記載の本発明の視線検出装置によれば、赤外線照射手段により運転者の眼球に照射された赤外線の角膜表面における反射像（プルキニエ像）を検出することができない場合であっても、可視光画像に基づき運転者の視線を継続して検出することができる。
さらに、請求項３に記載の本発明の視線検出装置によれば、視線検出の検出精度が低下することを防止することができる。 According to the gaze detection apparatus of the present invention, it is possible to prevent the detection accuracy of gaze detection from being lowered while suppressing the occurrence of a state in which gaze detection is not possible.
Further, according to the line-of-sight detection device of the present invention described in claim 2, it is a case where the reflected image (Purkinje image) of the infrared ray on the corneal surface irradiated to the driver's eyeball by the infrared irradiation means cannot be detected. However, it is possible to continuously detect the driver's line of sight based on the visible light image.
Furthermore, according to the gaze detection apparatus of the present invention described in claim 3, it is possible to prevent the detection accuracy of gaze detection from being lowered.

以下、本発明の一実施形態に係る視線検出装置について添付図面を参照しながら説明する。
本実施の形態による視線検出装置１０は、例えば車両の走行安全装置に具備され、図１に示すように、自車両の運転者の視線を検出するためにＣＰＵ等を含む電子回路により構成された検出制御装置１１と、視線センサ１２とを備えて構成されている。 Hereinafter, a gaze detection apparatus according to an embodiment of the present invention will be described with reference to the accompanying drawings.
The line-of-sight detection device 10 according to the present embodiment is provided in, for example, a vehicle travel safety device, and is configured by an electronic circuit including a CPU and the like to detect the line of sight of the driver of the host vehicle as shown in FIG. A detection control device 11 and a line-of-sight sensor 12 are provided.

視線センサ１２は、例えば図２に示すように、車室内のインストルメントパネルやダッシュボード上部等に設けられ、例えばＣＣＤカメラやＣ−ＭＯＳカメラ等の可視光領域にて撮影可能な可視光カメラ２１および赤外線領域にて撮影可能な赤外線カメラ２２と、運転者の眼球に向けて赤外線を照射する赤外線発光ダイオード（赤外線ＬＥＤ）２３と、可視光画像処理部２４および赤外線画像処理部２５とを備えて構成されている。
可視光カメラ２１は、運転者の顔や眼球から反射される可視光を撮影する。
赤外線カメラ２２は、検出制御装置１１の制御により赤外線ＬＥＤ２３から運転者の眼球に向けて照射された赤外線の反射を撮影する。
各画像処理部２４，２５は、各カメラ２１，２２により撮影して得た可視光画像および赤外線画像に対して、例えばフィルタリングや二値化処理等の所定の画像処理を行い、二次元配列の画素からなる画像データを生成して検出制御装置１１へ出力する。 For example, as shown in FIG. 2, the line-of-sight sensor 12 is provided on an instrument panel, an upper part of a dashboard, or the like in a vehicle interior, and a visible light camera 21 capable of photographing in a visible light region such as a CCD camera or a C-MOS camera. And an infrared camera 22 capable of photographing in the infrared region, an infrared light emitting diode (infrared LED) 23 that irradiates infrared rays toward the driver's eyeball, a visible light image processing unit 24, and an infrared image processing unit 25. It is configured.
The visible light camera 21 captures visible light reflected from the driver's face and eyeball.
The infrared camera 22 captures the reflection of infrared rays emitted from the infrared LED 23 toward the driver's eyeball under the control of the detection control device 11.
The image processing units 24 and 25 perform predetermined image processing such as filtering and binarization processing on the visible light image and the infrared image obtained by capturing with the cameras 21 and 22, respectively, Image data composed of pixels is generated and output to the detection control device 11.

検出制御装置１１は、視線センサ１２から可視光画像の画像データが入力される眼鏡装着判定部３１および可視光視線算出部３２と、視線センサ１２から赤外線画像の画像データが入力される赤外線反射判定部３３および赤外線視線算出部３４と、赤外線ＬＥＤ２３の発光状態を制御する発光制御部３５とを備えて構成されている。   The detection control device 11 includes a spectacle wearing determination unit 31 and a visible light line-of-sight calculation unit 32 to which image data of a visible light image is input from the line-of-sight sensor 12, and an infrared reflection determination to which image data of an infrared image is input from the line-of-sight sensor 12. Unit 33 and infrared line-of-sight calculation unit 34, and a light emission control unit 35 that controls the light emission state of the infrared LED 23.

眼鏡装着判定部３１は、視線センサ１２から入力される可視光画像の画像データに対して、例えば眼鏡を検知対象物とした特徴量算出および形状判別等の認識処理を行い、運転者が眼鏡を装着しているか否かを判定し、この判定結果を赤外線反射判定部３３および赤外線視線算出部３４へ出力する。
可視光視線算出部３２は、赤外線反射判定部３３にて赤外線ＬＥＤ２３から運転者の眼球に向けて照射される赤外線が運転者の眼鏡で反射されていると判定された際に、視線センサ１２から入力される可視光画像の画像データに対して、例えば運転者の顔や眼球を検知対象物とした特徴量算出および形状判別等の認識処理を行い、認識した検知対象物に基づき、運転者の視線ベクトル（視線方向）を算出し、さらに、この視線ベクトルに基づき、視線の対象位置を算出する。
例えば、運転者の眼球を検知対象物とした特徴量算出および形状判別では、目頭の位置と虹彩の中心位置との相対距離に基づき視線ベクトルを算出する。
なお、画像データの認識処理において、特徴量算出の処理では、例えば二値化処理後の画像データに対して、画素の連続性に基づく検知対象物の抽出およびラベリングを行い、抽出した検知対象物の重心および面積および外接四角形の縦横比等を算出する。また、形状判別の処理では、例えば予め記憶している所定パターン（例えば輪郭等）に基づき画像データ上の検索を行い、所定パターンとの類似性に応じて検知対象物を抽出する。 The spectacle wearing determination unit 31 performs recognition processing such as feature amount calculation and shape determination using the spectacles as the detection target for the image data of the visible light image input from the line-of-sight sensor 12, and the driver wears spectacles. It is determined whether or not it is attached, and the determination result is output to the infrared reflection determination unit 33 and the infrared line-of-sight calculation unit 34.
The visible light line-of-sight calculation unit 32 receives the infrared ray from the line-of-sight sensor 12 when it is determined by the infrared reflection determination unit 33 that the infrared rays emitted from the infrared LED 23 toward the driver's eyeball are reflected by the driver's glasses. The input image data of the visible light image is subjected to recognition processing such as feature amount calculation and shape determination using, for example, the driver's face and eyeball as a detection target, and based on the detected detection target, A line-of-sight vector (line-of-sight direction) is calculated, and a target position of the line of sight is calculated based on the line-of-sight vector.
For example, in the feature amount calculation and shape determination using the driver's eyeball as a detection target, the line-of-sight vector is calculated based on the relative distance between the position of the eye and the center position of the iris.
In the image data recognition processing, in the feature amount calculation processing, for example, extraction and labeling of detection objects based on pixel continuity is performed on the image data after binarization processing, and the extracted detection objects The center of gravity and area of the image and the aspect ratio of the circumscribed rectangle are calculated. In the shape determination process, for example, a search on image data is performed based on a predetermined pattern (for example, an outline) stored in advance, and a detection target is extracted according to the similarity to the predetermined pattern.

赤外線反射判定部３３は、眼鏡装着判定部３１にて運転者が眼鏡を装着していると判定された際に、視線センサ１２から入力される赤外線画像の画像データに対して、例えば所定値以上の輝度および大きさを有する領域を検知対象物とした特徴量算出および形状判別等の認識処理を行い、赤外線ＬＥＤ２３から運転者の眼球に向けて照射される赤外線が運転者の眼鏡で反射されているか否かを判定し、この判定結果を可視光視線算出部３２および赤外線視線算出部３４へ出力する。この赤外線反射判定部３３による認識処理では、例えば図３（ａ）に示すように、運転者の眼球を撮影した赤外線画像の画像データに対して、所定値以上の輝度および大きさを有する領域を抽出することで、例えば図３（ｂ）に示すような赤外線の各反射像Ｑ１，Ｑ２，Ｑ３を検出する。
赤外線視線算出部３４は、眼鏡装着判定部３１にて運転者が眼鏡を装着していないと判定された際、あるいは、赤外線反射判定部３３にて赤外線ＬＥＤ２３から運転者の眼球に向けて照射される赤外線が運転者の眼鏡で反射されていないと判定された際に、視線センサ１２から入力される赤外線画像の画像データに対して、例えば運転者の眼球を検知対象物とした特徴量算出および形状判別等の認識処理を行い、認識した検知対象物に基づき、運転者の視線ベクトル（視線方向）を算出し、さらに、この視線ベクトルに基づき、視線の対象位置を算出する。この赤外線視線算出部３４による認識処理では、例えば角膜表面における反射像（プルキニエ像）の中心位置と瞳孔の中心位置との相対距離に基づき視線ベクトルを算出する。 The infrared reflection determination unit 33 determines, for example, a predetermined value or more with respect to the image data of the infrared image input from the line-of-sight sensor 12 when the spectacle wearing determination unit 31 determines that the driver is wearing spectacles. Recognition processing such as feature amount calculation and shape discrimination using an area having a brightness and size of the object as a detection target, and infrared rays emitted from the infrared LED 23 toward the driver's eyeball are reflected by the driver's glasses And the determination result is output to the visible light line-of-sight calculation unit 32 and the infrared line-of-sight calculation unit 34. In the recognition processing by the infrared reflection determination unit 33, for example, as shown in FIG. 3A, an area having luminance and size greater than or equal to a predetermined value with respect to image data of an infrared image obtained by photographing a driver's eyeball. By extracting, for example, infrared reflected images Q1, Q2, and Q3 as shown in FIG. 3B are detected.
The infrared line-of-sight calculation unit 34 is irradiated from the infrared LED 23 toward the driver's eye when the spectacle wearing determination unit 31 determines that the driver is not wearing spectacles, or the infrared reflection determination unit 33. When it is determined that the infrared light is not reflected by the driver's glasses, for example, feature amount calculation using the driver's eyeball as a detection target is performed on the image data of the infrared image input from the line-of-sight sensor 12 and A recognition process such as shape discrimination is performed, a driver's line-of-sight vector (line-of-sight direction) is calculated based on the recognized detection object, and a line-of-sight target position is calculated based on the line-of-sight vector. In the recognition processing by the infrared line-of-sight calculation unit 34, for example, the line-of-sight vector is calculated based on the relative distance between the center position of the reflected image (Purkinje image) on the cornea surface and the center position of the pupil.

本実施の形態による視線検出装置１０は上記構成を備えており、次に、この視線検出装置１０の動作について説明する。   The line-of-sight detection apparatus 10 according to the present embodiment has the above-described configuration, and the operation of the line-of-sight detection apparatus 10 will be described next.

先ず、例えば図４に示すステップＳ０１においては、赤外線画像を取得する。
次に、ステップＳ０２においては、可視光画像を取得する。
次に、ステップＳ０３においては、取得した可視光画像の画像データに対して、例えば眼鏡を検知対象物とした特徴量算出および形状判別等の認識処理を行う。
そして、ステップＳ０４においては、運転者が眼鏡を装着しているか否かを判定する。
この判定結果が「ＮＯ」の場合には、ステップＳ０５に進み、視線センサ１２から入力される赤外線画像の画像データに対して、例えば運転者の眼球を検知対象物とした特徴量算出および形状判別等の認識処理を行い、認識した検知対象物に基づき、運転者の視線ベクトルを算出し、さらに、この視線ベクトルに基づき、視線の対象位置を算出し、一連の処理を終了する。
一方、この判定結果が「ＹＥＳ」の場合には、ステップＳ０６に進む。 First, for example, in step S01 shown in FIG. 4, an infrared image is acquired.
Next, in step S02, a visible light image is acquired.
Next, in step S03, recognition processing such as feature amount calculation and shape determination using eyeglasses as a detection target is performed on the acquired image data of the visible light image.
In step S04, it is determined whether the driver is wearing glasses.
If this determination is “NO”, the flow proceeds to step S 05, and feature amount calculation and shape determination are performed on the infrared image image data input from the line-of-sight sensor 12 using, for example, the driver's eyeball as a detection target. The driver's line-of-sight vector is calculated based on the recognized detection object, and the line-of-sight target position is calculated based on the line-of-sight vector, and the series of processes ends.
On the other hand, if this determination is “YES”, the flow proceeds to step S 06.

ステップＳ０６においては、取得した赤外線画像の画像データに対して、例えば所定値以上の輝度および大きさを有する領域を検知対象物とした特徴量算出および形状判別等の認識処理を行い、赤外線ＬＥＤ２３から運転者の眼球に向けて照射される赤外線が運転者の眼鏡で反射されているか否かを判定する。
この判定結果が「ＮＯ」の場合には、上述したステップＳ０５に進む。
一方、この判定結果が「ＹＥＳ」の場合には、ステップＳ０７に進む。
ステップＳ０７においては、取得した可視光画像の画像データに対して、例えば運転者の顔や眼球を検知対象物とした特徴量算出および形状判別等の認識処理を行い、認識した検知対象物に基づき、運転者の視線ベクトルを算出し、さらに、この視線ベクトルに基づき、視線の対象位置を算出し、一連の処理を終了する。 In step S06, recognition processing such as feature amount calculation and shape determination is performed on the acquired image data of the infrared image using, for example, a region having a luminance and size greater than or equal to a predetermined value as a detection target. It is determined whether or not the infrared ray irradiated toward the driver's eyeball is reflected by the driver's glasses.
If this determination is “NO”, the flow proceeds to step S 05 described above.
On the other hand, if this determination is “YES”, the flow proceeds to step S07.
In step S07, recognition processing such as feature amount calculation and shape determination using the driver's face and eyeball as a detection target is performed on the acquired image data of the visible light image, and the recognition data is detected based on the detected detection target. Then, the driver's line-of-sight vector is calculated, and further, the target position of the line-of-sight is calculated based on the line-of-sight vector, and the series of processing ends.

上述したように、本実施の形態による視線検出装置１０によれば、運転者が眼鏡を装着しているか否かに応じて少なくとも可視光画像または赤外線画像に基づき運転者の視線を検出することにより、視線検出が不可となる状態が発生することを抑制することができる。すなわち、赤外線カメラ２２の撮影により得られる赤外線画像において、赤外線ＬＥＤ２３により運転者の眼球に照射された赤外線の角膜表面における反射像（プルキニエ像）を検出することができない場合であっても、可視光画像に基づき運転者の視線を継続して検出することができる。
また、運転者が眼鏡を装着していない場合、つまり赤外線ＬＥＤ２３により運転者の眼球に照射された赤外線が運転者の眼鏡で反射される虞がない場合には、赤外線画像に基づき運転者の視線を検出することにより、視線検出の検出精度が低下することを防止することができる。 As described above, according to the line-of-sight detection device 10 according to the present embodiment, by detecting the driver's line of sight based on at least a visible light image or an infrared image depending on whether or not the driver is wearing glasses. It is possible to suppress the occurrence of a state in which the line-of-sight detection is impossible. That is, in the infrared image obtained by photographing with the infrared camera 22, even if it is not possible to detect the reflection image (Purkinje image) of the infrared ray irradiated on the driver's eyeball by the infrared LED 23 on the corneal surface. The driver's line of sight can be continuously detected based on the image.
Further, when the driver does not wear spectacles, that is, when there is no possibility that the infrared rays irradiated to the driver's eyeballs by the infrared LED 23 are reflected by the driver's glasses, the driver's line of sight is based on the infrared image. By detecting this, it is possible to prevent the detection accuracy of the line-of-sight detection from being lowered.

本発明の一実施形態に係る視線検出装置の構成図である。It is a lineblock diagram of a gaze detection device concerning one embodiment of the present invention. 図１に示す視線センサを搭載した車両を示す斜視図である。It is a perspective view which shows the vehicle carrying the gaze sensor shown in FIG. 運転者の眼鏡で反射される赤外線の反射像の一例を示す図である。It is a figure which shows an example of the infrared reflected image reflected with a driver | operator's spectacles. 図１に示す視線検出装置の動作を示すフローチャートである。It is a flowchart which shows operation | movement of the gaze detection apparatus shown in FIG.

符号の説明Explanation of symbols

１０ 視線検出装置
２１ 可視光カメラ（可視光撮影手段）
２２ 赤外線カメラ（赤外線撮影手段）
２３ 赤外線発光ダイオード（赤外線照射手段）
３１ 眼鏡装着判定部（眼鏡装着判定手段）
３２ 可視光視線算出部（視線検出手段）
３３ 赤外線反射判定部（赤外線反射判定手段）
３４ 赤外線視線算出部（視線検出手段）

10 Gaze Detection Device 21 Visible Light Camera (Visible Light Imaging Means)
22 Infrared camera (infrared imaging means)
23 Infrared light emitting diode (infrared irradiation means)
31 Glasses wearing determination unit (glasses wearing determining means)
32 Visible light gaze calculation unit (gaze detection means)
33 Infrared reflection determination unit (infrared reflection determination means)
34 Infrared line-of-sight calculation unit (line-of-sight detection means)

Claims (3)

自車両の運転者の眼球あるいは顔を可視光領域にて撮影する可視光撮影手段と、
自車両の運転者の眼球に赤外線を照射する赤外線照射手段と、
自車両の運転者の眼球を赤外線領域にて撮影する赤外線撮影手段と、
前記可視光撮影手段の撮影により得られる可視光画像に基づき運転者が眼鏡を装着しているか否かを判定する眼鏡装着判定手段と、
前記眼鏡装着判定手段の判定結果に応じて、少なくとも前記可視光撮影手段の撮影により得られる可視光画像または前記赤外線撮影手段の撮影により得られる赤外線画像に基づき運転者の視線を検出する視線検出手段と
を備えることを特徴とする視線検出装置。 Visible light photographing means for photographing the eyeball or face of the driver of the own vehicle in the visible light region;
An infrared irradiation means for irradiating the eyeball of the driver of the host vehicle with infrared rays;
An infrared photographing means for photographing an eyeball of a driver of the own vehicle in an infrared region;
Glasses wearing determination means for determining whether or not the driver is wearing glasses based on a visible light image obtained by photographing by the visible light photographing means;
Line-of-sight detection means for detecting a driver's line of sight based on at least a visible light image obtained by photographing with the visible light photographing means or an infrared image obtained by photographing with the infrared photographing means in accordance with the determination result of the glasses wearing determination means. A line-of-sight detection device comprising: 前記赤外線撮影手段の撮影により得られる赤外線画像に基づき、前記赤外線照射手段により運転者の眼球に照射された赤外線が運転者の眼鏡で反射したか否かを判定する赤外線反射判定手段を備え、
前記眼鏡装着判定手段により運転者が眼鏡を装着していると判定され、かつ、前記赤外線反射判定手段により運転者の眼球に照射された赤外線が運転者の眼鏡で反射したと判定された場合に、前記視線検出手段は前記可視光撮影手段の撮影により得られる可視光画像に基づき運転者の視線を検出することを特徴とする請求項１に記載の視線検出装置。 Based on an infrared image obtained by photographing by the infrared photographing means, comprising infrared reflection judging means for judging whether or not the infrared ray irradiated to the driver's eyeball by the infrared irradiating means is reflected by the driver's glasses,
When it is determined by the spectacle wearing determination means that the driver is wearing spectacles, and it is determined by the infrared reflection determination means that the infrared ray applied to the driver's eyeball is reflected by the driver's spectacles The line-of-sight detection device according to claim 1, wherein the line-of-sight detection unit detects a driver's line of sight based on a visible light image obtained by photographing with the visible light photographing unit. 前記眼鏡装着判定手段により運転者が眼鏡を装着していないと判定された場合に、前記視線検出手段は前記赤外線撮影手段の撮影により得られる赤外線画像に基づき運転者の視線を検出することを特徴とする請求項１に記載の視線検出装置。

When it is determined by the glasses wearing determination means that the driver does not wear glasses, the line-of-sight detection means detects the driver's line of sight based on an infrared image obtained by photographing by the infrared photographing means. The visual line detection device according to claim 1.

JP2004117723A 2004-04-13 2004-04-13 Gaze detection device Expired - Fee Related JP4313717B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP2004117723A JP4313717B2 (en) 2004-04-13 2004-04-13 Gaze detection device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
JP2004117723A JP4313717B2 (en) 2004-04-13 2004-04-13 Gaze detection device

Publications (2)

Publication Number Publication Date
JP2005296382A JP2005296382A (en) 2005-10-27
JP4313717B2 true JP4313717B2 (en) 2009-08-12

Family

ID=35328642

Family Applications (1)

Application Number Title Priority Date Filing Date
JP2004117723A Expired - Fee Related JP4313717B2 (en) 2004-04-13 2004-04-13 Gaze detection device

Country Status (1)

Country Link
JP (1) JP4313717B2 (en)

Families Citing this family (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4692496B2 (en) * 2007-02-23 2011-06-01 株式会社デンソー Vehicle display device
JP5949389B2 (en) * 2012-09-24 2016-07-06 富士通株式会社 Detection apparatus, detection program, and detection method
JP2014082585A (en) * 2012-10-15 2014-05-08 Denso Corp State monitor device and state monitor program
KR20140066848A (en) * 2012-11-22 2014-06-02 경북대학교 산학협력단 Face detecting device and method for detect of face
KR101619661B1 (en) 2014-12-08 2016-05-10 현대자동차주식회사 Detection method of face direction of driver
KR101867915B1 (en) 2015-01-16 2018-07-19 현대자동차주식회사 Method for executing vehicle function using wearable device and vehicle for carrying out the same
CN104739363B (en) * 2015-04-17 2017-02-01 京东方科技集团股份有限公司 Vision detecting method, device and system
JP6617662B2 (en) 2016-08-24 2019-12-11 株式会社Ｊｖｃケンウッド Gaze detection device, gaze detection method, and computer program
KR102209595B1 (en) * 2017-09-01 2021-01-28 매직 립, 인코포레이티드 Detailed eye shape model for robust biometric applications
KR102288753B1 (en) * 2017-10-23 2021-08-11 현대모비스 주식회사 Apparatus for monitoring driver and method thereof
KR102495359B1 (en) 2017-10-27 2023-02-02 삼성전자주식회사 Method and apparatus for tracking object
JP6885444B2 (en) * 2019-11-11 2021-06-16 株式会社Ｊｖｃケンウッド Line-of-sight detector, line-of-sight detection method, and computer program
JP7096509B2 (en) * 2020-10-23 2022-07-06 富士通クライアントコンピューティング株式会社 Information processing equipment and programs
WO2023053556A1 (en) * 2021-09-30 2023-04-06 富士フイルム株式会社 Information processing device, information processing method, and program

Also Published As

Publication number Publication date
JP2005296382A (en) 2005-10-27

Similar Documents

Publication Publication Date Title
JP2007004448A (en) Line-of-sight detecting apparatus
JP4313717B2 (en) Gaze detection device
US8102417B2 (en) Eye closure recognition system and method
JP6617662B2 (en) Gaze detection device, gaze detection method, and computer program
JP2009254525A (en) Pupil detecting method and apparatus
JP6651074B2 (en) Method for detecting eye movement, program thereof, storage medium for the program, and apparatus for detecting eye movement
WO2011158463A1 (en) External light glare assessment device, line of sight detection device and external light glare assessment method
JP4491604B2 (en) Pupil detection device
WO2006035890A1 (en) Method for detecting line of sight
CN107169403B (en) Face image processing apparatus
JP6855872B2 (en) Face recognition device
JP4451195B2 (en) Gaze detection device
CN111522431B (en) Classifying glints using an eye tracking system
JP4696571B2 (en) Eye position detection device
WO2017154356A1 (en) Sight line detection device and sight line detection method
JP5417272B2 (en) Eyeball imaging device
JP5206190B2 (en) Driver monitoring device
JP2018190213A (en) Face recognition device and sight line detection device
KR20210136623A (en) A device tracking gaze and method threrefor
JP4376685B2 (en) Gaze detection device
CN111200709A (en) Method for setting light source of camera system, camera system and vehicle
JP2020024750A (en) Line-of-sight detection device, line-of-sight detection method, and computer program
JPH06323832A (en) Interface for vehicle
US11156831B2 (en) Eye-tracking system and method for pupil detection, associated systems and computer programs
JP5024161B2 (en) Face position estimation device

Legal Events

Date Code Title Description
A621 Written request for application examination

Free format text: JAPANESE INTERMEDIATE CODE: A621

Effective date: 20061128

A977 Report on retrieval

Free format text: JAPANESE INTERMEDIATE CODE: A971007

Effective date: 20090423

TRDD Decision of grant or rejection written
A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

Effective date: 20090507

A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

A61 First payment of annual fees (during grant procedure)

Free format text: JAPANESE INTERMEDIATE CODE: A61

Effective date: 20090515

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20120522

Year of fee payment: 3

R150 Certificate of patent or registration of utility model

Free format text: JAPANESE INTERMEDIATE CODE: R150

LAPS Cancellation because of no payment of annual fees