WO2017154356A1 - Dispositif de détection de ligne de visée et procédé de détection de ligne de visée - Google Patents

Dispositif de détection de ligne de visée et procédé de détection de ligne de visée Download PDF

Info

Publication number
WO2017154356A1
WO2017154356A1 PCT/JP2017/001476 JP2017001476W WO2017154356A1 WO 2017154356 A1 WO2017154356 A1 WO 2017154356A1 JP 2017001476 W JP2017001476 W JP 2017001476W WO 2017154356 A1 WO2017154356 A1 WO 2017154356A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
pupil
center
coordinates
corneal reflection
Prior art date
Application number
PCT/JP2017/001476
Other languages
English (en)
Japanese (ja)
Inventor
河内　隆宏
Original Assignee
アルプス電気株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by アルプス電気株式会社 filed Critical アルプス電気株式会社
Publication of WO2017154356A1 publication Critical patent/WO2017154356A1/fr

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
    • A61B3/113—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for determining or recording eye movement
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/038—Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes

Definitions

  • the present invention relates to a line-of-sight detection device and a line-of-sight detection method capable of detecting the line-of-sight direction of a driver of a vehicle and other subjects.
  • the gaze detection device described in Patent Literature 1 acquires a bright pupil image, a dark pupil image, and a corneal reflection image, and uses an intermediate value between the maximum value and the minimum value of a difference image between the bright pupil image and the dark pupil image as a threshold value. , The difference image is binarized. Further, this line-of-sight detection device detects the outer periphery of the pupil image by performing an elliptic approximation process or the like on the pupil image vicinity region searched from the binarized image, and specifies the center of the detected ellipse as the center position of the pupil image.
  • the gaze detection device described in Patent Document 1 has the following problems. That is, the light source used to acquire each of the bright pupil image and the dark pupil image does not match the conditions such as wavelength, position, incidence / reflection angle with respect to the eye, and light emission timing, so that the corneal reflection in the bright pupil image and the dark pupil image There is a possibility that the positions of the images do not completely coincide. When such a difference between the bright pupil image and the dark pupil image is taken, the corneal reflection image is not removed because the coordinates, brightness, and size of the corneal reflection image are different, and a part of the corneal reflection image remains in the difference image. May end up.
  • the remaining image of the corneal reflection image remaining in this way is located in the vicinity of the pupil image, this remaining image is also included in the target of the ellipse approximation process for detecting the outer periphery of the pupil image. In addition, the center position cannot be accurately detected.
  • An object of the present invention is to provide a gaze detection apparatus and a gaze detection method that can be calculated.
  • the eye gaze detection device of the present invention acquires a bright pupil image and a dark pupil image, respectively, and creates a difference image between the bright pupil image and the dark pupil image, and a difference image.
  • An image processing unit that creates a binarized binary image and selects a pupil candidate from one or more bright spots in the binary image, and calculates the coordinates of the corneal reflection light center in the dark pupil image as the reflection light center coordinates
  • a line-of-sight detection comprising a corneal reflection light center detection unit and a determination unit for determining whether or not the corneal reflection light center is within the first range of the binary image including the pupil candidate based on the reflection light center coordinates.
  • the image processing unit When the determination unit determines that the corneal reflection light center is within the first range by the determination unit, the image processing unit reduces the luminance of the pixels within the second range including the corneal reflection light center. Create a pupil image that is
  • the output device includes a pupil center calculation unit that calculates the coordinates of the pupil center based on the pupil image, and a gaze direction calculation unit that calculates the gaze direction based on the reflected light center coordinates and the pupil center coordinates. It is said.
  • the gaze detection method of the present invention includes a step of acquiring a bright pupil image and a dark pupil image, a difference image between the bright pupil image and the dark pupil image, and binarizing the difference image to generate a binary image.
  • image producing step is characterized by comprising: a pupil center calculation step of calculating a pupil center of the coordinates based on the pupil image, and calculating the sight
  • the line-of-sight detection device and line-of-sight detection method of the present invention it is determined whether or not the corneal reflection light center is within the first range, and if it is within the first range, Since the brightness of the second range including the corneal reflection light center is reduced, even if a part of the corneal reflection image remains in the difference image, it is possible to suppress the influence, thereby accurately calculating the pupil center. Therefore, the accuracy of line-of-sight detection can be further increased.
  • the image processing unit calculates an estimated value of the coordinates of the corneal reflection light center in the binary image. It is preferable to create a pupil image in which the luminance of the pixels in the second range including the coordinates is reduced.
  • the pupil image creation step determines that the corneal reflection light center is within the first range in the determination step
  • the coordinates of the corneal reflection light center in the binary image are estimated. It is preferable to calculate a value and create a pupil image in which the luminance of the pixels in the second range including the coordinates corresponding to the coordinates is reduced.
  • the accuracy of calculating the pupil center can be further increased.
  • the image processing unit when the determination unit determines that the corneal reflection light center is within the first range, the image processing unit preferably sets the luminance of the pixels within the second range to zero. . Thereby, since the image of the corneal reflection light in the dark pupil image remaining in the binary image can be completely removed, the pupil center can be calculated with higher accuracy.
  • a first camera and a second camera which are arranged apart from each other and each acquire an image of an area including at least eyes, a first light source arranged close to the first camera, It is preferable that the second light source is disposed close to the second camera, and the pupil image extraction unit acquires the bright pupil image and the dark pupil image from the images acquired by the respective cameras.
  • the pupil image extraction unit acquires the bright pupil image and the dark pupil image from the images acquired by the respective cameras.
  • the first range is preferably an elliptical range calculated based on the pupil candidate selected in the pupil candidate selection step.
  • an appropriate range can be set according to the shape of the eyes of various subjects, which can contribute to accurate pupil center calculation.
  • the first range is a range of pixels of the pupil candidate selected in the pupil candidate selection step.
  • the first range can be accurately set based on the pupil candidates, which can contribute to accurate pupil center calculation.
  • the pupil center calculation step calculates an elliptical center as the pupil center.
  • the setting accuracy of the center of the pupil can be kept constant, which can contribute to highly accurate line-of-sight detection.
  • the influence of the remaining image can be suppressed, and thereby the center of the pupil can be accurately determined. Can be calculated.
  • FIG. 3 is an enlarged view of FIG. It is a flowchart which shows the flow of a process of the gaze detection method which concerns on 1st Embodiment of this invention. It is a flowchart which shows the flow of a process of the gaze detection method which concerns on 2nd Embodiment of this invention.
  • FIG. 1 is a block diagram illustrating a configuration of a visual line detection device 10 according to the first embodiment.
  • 2A is an example of a bright pupil image acquired by the line-of-sight detection device 10
  • FIG. 2B is an example of a dark pupil image
  • FIG. 2C is the bright pupil image of FIG. It is a figure which shows the difference image of the dark pupil image of (B).
  • FIG. 3 is an enlarged view of FIG. As shown in FIG.
  • the line-of-sight detection device 10 includes two light sources 11, 12, two cameras 21, 22, a control unit 30, two light source control units 31, 32, two image acquisition units 41, 42, and a pupil.
  • An image extraction unit 51, a pupil center calculation unit 54, a corneal reflection light center detection unit 55, a gaze direction calculation unit 56, an image processing unit 57, and a determination unit 58 are provided.
  • the line-of-sight detection device 10 is installed, for example, on an instrument panel or an upper part of a windshield in a vehicle cabin so as to face the driver's face as a subject.
  • the cameras 21 and 22 are arranged such that the optical axes are separated from each other by a predetermined distance.
  • the cameras 21 and 22 have, for example, CMOS (complementary metal oxide semiconductor) as an image sensor.
  • CMOS complementary metal oxide semiconductor
  • This image sensor acquires an image of a face including the driver's eyes, and light is detected by a plurality of pixels arranged in the horizontal direction and the vertical direction.
  • Each of the first light source 11 and the second light source 12 includes, for example, a plurality of LED (light emitting diode) light sources.
  • the first light source 11 is disposed so as to surround the lens outside the lens of the first camera 21, and the second light source 12 is disposed so as to surround the lens outside the lens of the second camera 22, for example.
  • band pass filters it is preferable to arrange band pass filters in accordance with the wavelengths of the detection lights emitted from the two light sources 11 and 12.
  • the pupil image extraction unit 51 can extract the pupil image and the gaze direction calculation unit 56 can calculate the gaze direction with high accuracy.
  • the LED light source of the first light source 11 and the LED light source of the second light source 12 both emit infrared light (near infrared light) of 800 nm or more and 1000 nm or less, and give this detection light to the driver's eyes. It is arranged to be able to.
  • 850 nm is a wavelength with a low light absorption rate in the eyeball of a human eye, and this light is easily reflected by the retina at the back of the eyeball.
  • the distance between the optical axes of the LED light sources of the first camera 21 and the first light source 11 is determined with respect to the distance between the optical axes of the first camera 21 and the second camera 22 in consideration of the distance between the visual line detection device 10 and the driver. And short enough. Therefore, the first light source 11 can be regarded as having a substantially coaxial optical axis with respect to the first camera 21. Similarly, since the distance between the optical axes of the LED light sources of the second camera 22 and the second light source 12 is sufficiently shorter than the distance between the optical axes of the first camera 21 and the second camera 22, the second light source 12, the optical axes of the second camera 22 can be regarded as being substantially coaxial.
  • the optical axes of the first camera 21 and the second camera 22 is sufficiently long, the optical axes of the first light source 11 and the first camera 21, and the second light source 12 and the second Each optical axis of the camera 22 is not coaxial.
  • the above arrangement may be expressed as two members being substantially coaxial and the like, and the two members being non-coaxial.
  • the determination unit 58 includes a CPU and a memory of a computer, and processing by each unit is performed by executing software installed in advance.
  • the light source control unit 31 and the light source control unit 32 respectively control lighting / non-lighting of the first light source 11 and the second light source 12 according to an instruction signal from the control unit 30.
  • the cameras 21 and 22 acquire images according to the instruction signal from the control unit 30, and images captured by the cameras 21 and 22 are acquired by the image acquisition units 41 and 42 for each frame.
  • the images acquired by the image acquisition units 41 and 42 are read into the pupil image extraction unit 51 for each frame.
  • the pupil image extraction unit 51 includes a bright pupil image detection unit 52 and a dark pupil image detection unit 53.
  • the bright pupil image detection unit 52 detects an eye image when the combination of the light source and the camera satisfies one of the following bright pupil photographing conditions (a).
  • the dark pupil image detection unit 53 detects the following dark pupils: An eye image when the combination of the light source and the camera satisfies any one of the imaging conditions (b) is detected.
  • (A) Bright pupil photographing condition (a-1) An image is acquired by the first camera 21 substantially coaxial with the first light source 11 during the lighting period.
  • the infrared light emitted from the first light source 11 has a low absorption rate in the eyeball reaching the retina of the driver's eye, and thus is easily reflected by the retina. Therefore, when the first light source 11 is turned on, in the image acquired by the first camera 21 substantially coaxial with the first light source 11, the infrared light reflected by the retina is detected through the pupil, and the pupil looks bright. This image is extracted as a bright pupil image by the bright pupil image detection unit 52. The same applies to an image acquired by the second camera 22 that is substantially coaxial with the second light source 12 when it is turned on.
  • the wavelength of the infrared light emitted from the first light source 11 and the second light source 12 is set to 800 nm or more and 900 nm or less (especially 850 nm), the light absorption rate in the eyeball of the human eye is low and clearer. A bright pupil image can be acquired.
  • the infrared light reflected by the retina is transmitted to the second camera 22.
  • the pupil appears dark because it is hardly incident. Therefore, this image is extracted by the dark pupil image detection unit 53 as a dark pupil image.
  • the wavelength of the infrared light emitted from the first light source 11 and the second light source 12 is set to be greater than 900 nm and less than or equal to 1000 nm (particularly, 950 nm), the light absorption rate in the eyeball of the human eye is high and clearer.
  • a dark pupil image can be acquired.
  • the dark pupil image detected by the dark pupil image detection unit 53 is subtracted from the bright pupil image detected by the bright pupil image detection unit 52 to create a difference image. That is, the difference image is an image in which an image region that is not common to the bright pupil image and the dark pupil image remains after subtraction of both images. Furthermore, a range of a predetermined size including both eyes is cut out as the pupil detection range.
  • the clipped image is given to the image processing unit 57, and binarized by image processing to create a binary image.
  • the created binary image is stored in a memory in the image processing unit 57.
  • labeling image processing is performed, and a label is attached to each of one or more bright spots in the binary image.
  • the image processing unit 57 selects a bright spot as a pupil candidate of the pupil image from one or more binary images based on the brightness, shape, etc. of the bright spot in the binary image.
  • the selection of the pupil candidate is performed, for example, so as to select the bright spot closest to the circle as the image of the pupil.
  • the binary image, the selected pupil candidate, and the label attached to the pupil candidate are output to the pupil center calculation unit 54 and the determination unit 58.
  • the pupil center calculation unit 54 calculates an area image corresponding to the shape and area of the pupil for the bright spot selected as the pupil candidate of the pupil image. Further, an ellipse including the area image is extracted, and the center of the ellipse shape, that is, the intersection of the major axis and the minor axis of the ellipse shape is calculated as the center position of the pupil. Various methods can be used for extracting the ellipse (ellipse fitting). For example, an ellipse inscribed by the area image is calculated, and this ellipse is selected.
  • the pupil center calculation unit 54 sends the calculated coordinate data of the center position of the pupil to the line-of-sight direction calculation unit 56 and outputs a signal indicating that the coordinate data has been calculated to the determination unit 58.
  • the dark pupil image signal detected by the dark pupil image detection unit 53 is given to the corneal reflection light center detection unit 55.
  • the dark pupil image signal includes a luminance signal based on reflected light from the reflection point of the cornea.
  • the reflected light from the reflection point of the cornea forms a Purkinje image, and is acquired as a spot image with a very small area by the imaging device of the camera.
  • the corneal reflection light center detection unit 55 performs image processing on the spot image, and obtains the coordinates of the center of the reflection light (corneal reflection light) from the reflection point of the cornea in the dark pupil image as the reflection light center coordinates.
  • the corneal reflection light center detection unit 55 also calculates the area of the corneal reflection image in the dark pupil image.
  • the center coordinates of the reflected light and the area of the corneal reflection image thus calculated are output to the determination unit 58.
  • the calculation of the reflected light center coordinate in the corneal reflected light center detection unit 55 is performed when the dark pupil image is input from the dark pupil image detection unit 53 or the detection of the corneal reflected light center is performed from the determination unit 58. Performed when a signal to instruct is received.
  • the determination unit 58 refers to the reflected light center coordinates calculated by the cornea reflected light center detection unit 55 on the binary image created by the image processing unit 57, and reflects the reflected light within the first range of the binary image. It is determined whether or not there is a center coordinate.
  • the first range is, for example, an elliptical range extracted by the pupil center calculation unit 54 for calculating the pupil center position, or a circle having a radius of a predetermined number of pixels centered on the pupil center position. Or the range of an ellipse.
  • the result of determination by the determination unit 58 is output to the image processing unit 57.
  • the determination unit 58 receives a signal indicating that the coordinates of the center position of the pupil are calculated from the pupil center calculation unit 54, the detection result of the corneal reflection light center is input from the corneal reflection light center detection unit 55. If not input, a signal instructing execution of detection of the corneal reflection light center is sent to the corneal reflection light center detection unit 55.
  • the image processing unit 57 determines the binary image based on the shape and luminance information of the image at the reflection light center coordinates of the binary image.
  • An estimated value (estimated coordinates) of the coordinates of the corneal reflection light center at is calculated. For example, when the residual image of the corneal reflection light in the binary image has a shape of a part of a circle, the coordinates when the shape is complemented with the circle shape are the coordinates of the corneal reflection light center. Estimated as (estimated coordinates).
  • the image processing unit 57 creates an image in which the luminance of the pixels within the second range including the estimated coordinates is reduced in the binary image, and updates the binary image with this image.
  • the luminance is reduced at a constant rate, and the rate may be zero or half, for example.
  • the second range is, for example, a range corresponding to a corneal reflection image in the dark pupil image.
  • a pupil image 72 is obtained by subtracting the pupil image 71 in the dark pupil image 62 from the pupil image 71 in the bright pupil image 61.
  • a crescent-shaped cornea reflection image 73 remains in the difference image 63 due to the positional shift of the cornea reflection images in the bright pupil image 61 and the dark pupil image 62. Therefore, on the difference image 63, a pupil image 72 and a corneal reflection image 73 appear as pupil candidates (bright spots) of the pupil image.
  • an ellipse 82 or an ellipse 83 is set as a first range having an elliptical shape centered on the pupil image 72.
  • the cornea reflection image 73 does not exist within this range, but when the ellipse 83 is the first range, the cornea reflection image 73 exists within this range.
  • the circle range 84 centered on the center 84c of the circle obtained by complementing the crescent shape of the cornea reflection image 73 is set as the second range, Set the pixel brightness to zero.
  • the pupil center calculation unit 54 can calculate the center of the pupil image more accurately by calculating the center position of the pupil based on the updated binary image.
  • the pupil center calculation value calculated by the pupil center calculation unit 54 and the corneal reflection light center calculation value calculated by the corneal reflection light center detection unit 55 are given to the gaze direction calculation unit 56.
  • the line-of-sight direction calculation unit 56 detects the direction of the line of sight from the pupil center calculated value and the corneal reflection light center calculated value.
  • FIG. 4 is a flowchart showing a flow of processing of the visual line detection method according to the first embodiment.
  • the operation of the first camera 21 will be described.
  • the first light source 11 is turned on for obtaining a pupil image, and a bright pupil image is obtained by obtaining an image in accordance with the turning on (step S1).
  • the second light source 12 is turned on, and a dark pupil image is obtained by acquiring an image in accordance with the lighting (step S2).
  • the operation of the second camera 22 will be described.
  • the first light source 11 is turned on for pupil image acquisition under the control of the control unit 30, and when an image is acquired in accordance with this lighting, the second camera 22 obtains a dark pupil image.
  • Step S1 When the second light source 12 is turned on and an image is acquired in accordance with this lighting, a bright pupil image is obtained by the second camera 22 (step S2).
  • step S1 the image captured by the first camera 21 is read from the image acquisition unit 41 into the bright pupil image detection unit 52 of the pupil image extraction unit 51, and the bright pupil image detection unit 52 detects the bright pupil image.
  • the image captured by the second camera 22 is read from the image acquisition unit 42 into the dark pupil image detection unit 53 of the pupil image extraction unit 51, and the dark pupil image detection unit 53 detects the dark pupil image.
  • step S2 the image captured by the first camera 21 is read from the image acquisition unit 41 into the dark pupil image detection unit 53 of the pupil image extraction unit 51, and the dark pupil image detection unit 53 detects the dark pupil image.
  • the image captured by the second camera 22 is read from the image acquisition unit 42 into the bright pupil image detection unit 52 of the pupil image extraction unit 51, and the bright pupil image detection unit 52 detects the bright pupil image.
  • the pupil image extraction unit 51 the dark pupil image detected by the dark pupil image detection unit 53 is subtracted from the bright pupil image detected by the bright pupil image detection unit 52 to create a difference image (step S3). Further, the pupil image extraction unit 51 cuts out an image having a predetermined size including both eyes of the subject as a detection range (step S4).
  • the image cut out in step S4 is sent to the image processing unit 57, and a binary image is created by image processing (step S5).
  • the created binary image is stored in a memory in the image processing unit 57.
  • the image processing unit 57 executes a labeling process on the binary image, and labels each of one or more bright spots included in the binary image (step S6). Further, the image processing unit 57 selects a pupil candidate (candidate label) of the pupil image based on the shape of the bright spot or the like (step S7).
  • the selected pupil candidate data is output to the pupil center calculation unit 54.
  • the pupil center calculation unit 54 calculates an area image corresponding to the shape and area of the pupil for the bright spot selected by the image processing unit 57. Further, the pupil center calculation unit 54 extracts an elliptical shape including the calculated area image (ellipse fitting), and calculates coordinates (XP, YP) of the elliptical center position (step S8). The calculated coordinate value is output to the determination unit 58.
  • the determination unit 58 determines whether or not the corneal reflection light center position has been detected (step S9).
  • the determination unit 58 instructs the pupil image extraction unit 51 to calculate the coordinates of the corneal reflection light center.
  • the pupil image extraction unit 51 uses, as an image of a range for detecting the coordinates of the corneal reflection light center, an image of a predetermined size including both eyes of the subject in the dark pupil image. Cut out (step S10).
  • the corneal reflection light center detection unit 55 calculates the coordinates (XG, YG) of the corneal reflection light center from the spot image in the dark pupil image based on the extracted detection range (step S11).
  • the calculation result of the coordinates of the corneal reflection light center in step S11 is output to the determination unit 58.
  • the determination unit 58 determines whether or not the coordinates (XG, YG) of the corneal reflection light center are within the elliptical range including the area image calculated in step S8 (step S8). S12).
  • the determination result is output to the image processing unit 57.
  • the image processing unit 57 estimates the coordinates (estimated coordinates) of the corneal reflection light center in the binary image. (Step S13).
  • the image processing unit 57 creates an image in which the luminance of the pixels in the second range including the estimated coordinates in the binary image, that is, the peripheral pixels of the pupil in the elliptical range is zero,
  • the binary image is updated in the memory (step S14).
  • the updated binary image is also output to the pupil image extraction unit 51, and the processing after step S8 is executed again based on this binary image.
  • step S9 if it is determined in step S9 that the corneal reflection light center has already been detected (Y in step S9), it is determined in step S12 that the coordinates of the corneal reflection light center are not within the elliptical range of the pupil. If so (N in step S12), the coordinates of the pupil and the coordinates of the corneal reflection light center are determined (step S15), and the line-of-sight direction calculation unit 56 calculates the line-of-sight direction of the subject based on these coordinates.
  • the corneal reflection light center is within the first range of the binary image including the pupil candidate of the pupil, and it is determined that the corneal reflection light center is within the first range
  • the line-of-sight direction is calculated based on the coordinates of the pupil center calculated based on the coordinates and the center coordinates of the reflected light. According to this, even if a part of the corneal reflection image remains in the difference image, it is possible to suppress the influence, and thereby it is possible to accurately calculate the pupil center, thereby further improving the accuracy of line-of-sight detection. Can be increased.
  • the image processing unit When it is determined that the corneal reflection light center is within the first range, an estimated value of the coordinates of the corneal reflection light center in the binary image is calculated, and a second value including coordinates corresponding to the coordinates is calculated. An image in which the luminance of the pixels in the range is reduced is created, and the binary image is updated with this image. For this reason, since the image of the corneal reflected light remaining in the binary image is accurately removed, it is possible to further increase the calculation accuracy of the pupil center.
  • the image processing unit when it is determined by the determination unit that the corneal reflection light center is within the first range, the image processing unit preferably sets the luminance of the pixels within the second range to zero. Thereby, since the image of the corneal reflection light remaining in the binary image can be completely removed, the pupil center can be calculated with higher accuracy.
  • the line-of-sight detection device 10 includes the first light source 11 and the first camera 21 arranged substantially coaxially, and the second light source 12 and the second camera 22 arranged substantially coaxially.
  • a bright pupil image is acquired with a camera that is substantially coaxial with the light source that is turned on, and a dark pupil image is acquired with a non-coaxial camera.
  • the line-of-sight detection device 10 remains in the difference image of these images. The influence of the residual image of the corneal reflection light can be suppressed.
  • the wavelengths of the light emitted from the two light sources 11 and 12 are both low in the light absorption rate in the eyeball, but the wavelength of the light emitted from one light source is the light absorption rate.
  • the wavelength may be high and difficult to reach the retina (for example, greater than 900 nm and 1000 nm or less, particularly 950 nm), and may be used as a light source for bright pupil image capturing and a light source for dark pupil image capturing. Further, the number of light sources and cameras is not limited to two.
  • the first range is preferably an elliptical range calculated based on the selected pupil candidate. Furthermore, it is preferable to calculate the center of the elliptical shape as the pupil center.
  • the first range is preferably a range of pixels of the pupil candidate selected in the pupil candidate selection step. In this case, the first range may be the same as the range of the candidate pupil pixels, but may be extended outward by a predetermined number of pixels from the range. As a result, an appropriate range can be set according to the shape of the eyes of various subjects, which can contribute to accurate pupil center calculation.
  • FIG. 5 is a flowchart showing a flow of processing of the visual line detection method according to the second embodiment.
  • Acquisition of a bright pupil image (step S21), acquisition of a dark pupil image (step S22), creation of a difference image (step S23), extraction of a pupil detection range (step S24), creation of a binary image in the second embodiment (Step S25), labeling (Step S26), and selection of a pupil candidate label (Step S27) are the same as Steps S1 to S7 of the first embodiment.
  • a notification signal is transmitted from the image processing unit 57 to the determination unit 58.
  • the determination unit 58 determines whether or not the corneal reflection light center position has been detected (step S28).
  • the determination unit 58 instructs the pupil image extraction unit 51 to calculate the coordinates of the corneal reflection light center.
  • the pupil image extraction unit 51 uses, as an image of a range for detecting the coordinates of the corneal reflection light center, an image of a predetermined size including both eyes of the subject in the dark pupil image. Cut out (step S29).
  • the corneal reflection light center detection unit 55 calculates the coordinates (XG, YG) of the corneal reflection light center from the spot image based on the extracted detection range (step S30).
  • the calculation result of the coordinates of the corneal reflection light center in step S30 is output to the determination unit 58.
  • the determination unit 58 determines whether or not the coordinates (XG, YG) of the corneal reflection light center are within the range of the pupil candidates (step S31). This determination is also performed when it is determined in step S28 that the corneal reflection light center has already been detected (Y in step S28).
  • the image processing unit 57 estimates the coordinates (estimated coordinates) of the corneal reflection light center in the binary image. (Step S32). Further, the image processing unit 57 creates an image in which the luminance of the pixels in the second range including the estimated coordinates in the binary image, that is, the peripheral pixels of the pupil in the range of the pupil candidates is zero, The binary image is updated in the memory (step S33).
  • step S31 If it is determined in step S31 that the coordinates of the corneal reflection light center are not within the range of pupil candidates for the pupil (N in step S12), and if the binary image is updated in step S33, pupil center calculation is performed.
  • the unit 54 an area image corresponding to the shape and area of the pupil is calculated for the bright spot selected by the image processing unit 57.
  • the pupil center calculation unit 54 extracts an elliptical shape including the calculated area image (ellipse fitting), and calculates the coordinates (XP, YP) of the elliptical center position (step S34).
  • step S35 the coordinates of the pupil and the coordinates of the corneal reflected light center are determined (step S35), and the line-of-sight direction calculation unit 56 calculates the line-of-sight direction of the subject based on these coordinates.
  • Other operations and effects are the same as those in the first embodiment.
  • the line-of-sight detection device and line-of-sight detection method according to the present invention provide the detection angle of the detection light to the subject's eyes, the shooting angle of the camera, and the camera in the acquisition of the bright pupil image and the dark pupil image. Even if the position of the corneal reflection light center in the bright pupil image and the dark pupil image is shifted due to a shift in the photographing timing, and the corneal reflection image may remain in the difference image between these images, This is useful in that the influence can be suppressed.
  • Eye-gaze detection apparatus 11 1st light source 12 2nd light source 21 1st camera 22 2nd camera 30 Control part 31, 32 Light source control part 41, 42 Image acquisition part 51 Pupil image extraction part 52 Bright pupil image detection part 53 Dark pupil image Detection unit 54 Pupil center calculation unit 55 Corneal reflection light center detection unit 56 Gaze direction calculation unit 57 Image processing unit 58 Discrimination unit 61 Bright pupil image 62 Dark pupil image 63 Difference image 71 Pupil image 72 Pupil image 73 Corneal reflection image 81 Detection Range 82, 83 Ellipse (first range) 84 circle range (second range)

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Molecular Biology (AREA)
  • Veterinary Medicine (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Surgery (AREA)
  • Animal Behavior & Ethology (AREA)
  • General Health & Medical Sciences (AREA)
  • Public Health (AREA)
  • Medical Informatics (AREA)
  • Geometry (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Biomedical Technology (AREA)
  • Ophthalmology & Optometry (AREA)
  • Biophysics (AREA)
  • Eye Examination Apparatus (AREA)

Abstract

[Problème] L'invention a pour objet de fournir un dispositif de détection de ligne de visée et un procédé de détection de ligne de visée qui, même lorsqu'une partie d'une image de réflexion cornéenne reste dans une image de différence pour une image de pupille claire et une image de pupille sombre, peut supprimer l'effet de l'image restante et calculer ainsi avec précision le centre d'une pupille. [Solution] L'invention propose un dispositif de détection de ligne de visée qui a une unité de traitement d'image qui, lorsqu'une unité de détermination a déterminé que le centre de la lumière de réflexion cornéenne se trouve dans une première zone, crée une image de pupille dans laquelle la luminance de pixels dans une seconde zone qui comprend le centre de la lumière de réflexion cornéenne a été réduite. Le dispositif de détection de ligne de visée comprend : une unité de calcul de centre de pupille qui calcule les coordonnées d'un centre de pupille sur la base de l'image de pupille; et une unité de calcul de direction de ligne de visée qui calcule une direction de ligne de visée sur la base des coordonnées de centre de lumière de réflexion et des coordonnées du centre de pupille.
PCT/JP2017/001476 2016-03-09 2017-01-18 Dispositif de détection de ligne de visée et procédé de détection de ligne de visée WO2017154356A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2016-045717 2016-03-09
JP2016045717 2016-03-09

Publications (1)

Publication Number Publication Date
WO2017154356A1 true WO2017154356A1 (fr) 2017-09-14

Family

ID=59789228

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2017/001476 WO2017154356A1 (fr) 2016-03-09 2017-01-18 Dispositif de détection de ligne de visée et procédé de détection de ligne de visée

Country Status (1)

Country Link
WO (1) WO2017154356A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111723773A (zh) * 2020-06-30 2020-09-29 创新奇智(合肥)科技有限公司 遗留物检测方法、装置、电子设备及可读存储介质
CN114067420A (zh) * 2022-01-07 2022-02-18 深圳佑驾创新科技有限公司 一种基于单目摄像头的视线测量方法及装置

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH08308801A (ja) * 1995-05-16 1996-11-26 Olympus Optical Co Ltd 視線検出装置
JP2011115460A (ja) * 2009-12-04 2011-06-16 Saga Univ 視線制御装置、視線制御方法、及びそのプログラム
JP2016024616A (ja) * 2014-07-18 2016-02-08 国立大学法人静岡大学 眼球計測システム、視線検出システム、眼球計測方法、眼球計測プログラム、視線検出方法、および視線検出プログラム

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH08308801A (ja) * 1995-05-16 1996-11-26 Olympus Optical Co Ltd 視線検出装置
JP2011115460A (ja) * 2009-12-04 2011-06-16 Saga Univ 視線制御装置、視線制御方法、及びそのプログラム
JP2016024616A (ja) * 2014-07-18 2016-02-08 国立大学法人静岡大学 眼球計測システム、視線検出システム、眼球計測方法、眼球計測プログラム、視線検出方法、および視線検出プログラム

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111723773A (zh) * 2020-06-30 2020-09-29 创新奇智(合肥)科技有限公司 遗留物检测方法、装置、电子设备及可读存储介质
CN111723773B (zh) * 2020-06-30 2024-03-29 创新奇智(合肥)科技有限公司 遗留物检测方法、装置、电子设备及可读存储介质
CN114067420A (zh) * 2022-01-07 2022-02-18 深圳佑驾创新科技有限公司 一种基于单目摄像头的视线测量方法及装置

Similar Documents

Publication Publication Date Title
CN107533362B (zh) 眼睛跟踪设备和用于操作眼睛跟踪设备的方法
US10722113B2 (en) Gaze detection apparatus and gaze detection method
US10698482B2 (en) Gaze tracking using non-circular lights
EP2881891B1 (fr) Procédé et dispositif de traitement d'images
JP6346525B2 (ja) 視線検出装置
JP6201956B2 (ja) 視線検出装置および視線検出方法
US11023039B2 (en) Visual line detection apparatus and visual line detection method
JP2009254525A (ja) 瞳孔検知方法及び装置
JP5626039B2 (ja) 視線検出装置及び視線検出方法
US20170116736A1 (en) Line of sight detection system and method
WO2017203769A1 (fr) Dispositif de détection de ligne de visée
JP5601179B2 (ja) 視線検出装置及び視線検出方法
US20210012105A1 (en) Method and system for 3d cornea position estimation
WO2018164104A1 (fr) Dispositif de traitement d'image oculaire
JP4313717B2 (ja) 視線検出装置
US10748301B2 (en) Corneal reflex position detection device, eye-gaze tracking device, and corneal reflex position detection method
WO2017154356A1 (fr) Dispositif de détection de ligne de visée et procédé de détection de ligne de visée
JP2016028669A (ja) 瞳孔検出装置、および瞳孔検出方法
JP6011173B2 (ja) 瞳孔検出装置および瞳孔検出方法
JP4451195B2 (ja) 視線検出装置
US11796802B2 (en) Device tracking gaze and method therefor
WO2017154370A1 (fr) Dispositif et procédé de détection de ligne de visée
JP2017162233A (ja) 視線検出装置および視線検出方法
KR20160055621A (ko) 시선추적장치 및 그의 반사점 검출 방법
JP6693149B2 (ja) 瞳孔検出装置、および瞳孔検出方法

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17762702

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 17762702

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: JP