WO2015175435A1

WO2015175435A1 - Driver health and fatigue monitoring system and method

Info

Publication number: WO2015175435A1
Application number: PCT/US2015/030219
Authority: WO
Inventors: David Breed
Original assignee: Automotive Technologiesinternational, Inc.
Priority date: 2014-05-12
Filing date: 2015-05-11
Publication date: 2015-11-19
Also published as: CN106687026A

Abstract

Method and system for monitoring a vehicular driver in which images of the driver are obtained using an image obtaining device (200) and brain waves from the driver may be obtained using one or more sensors (310) on the seatbelt (304) worn by the driver. A processor (210) analyzes the images to derive a measure of flow of blood in at least one blood vessel, capillary and vein in the face of the occupant and then analyzes the blood flow over time and the obtained brain waves to determine whether the driver has lost the ability to continue to control the vehicle. Loss of ability to continue to control the vehicle exemplifies the driver becoming drowsy, falling asleep or otherwise being incapable of controlling the vehicle after initially having been awake or otherwise capable of controlling the vehicle.

Description

DRIVER HEALTH AND FATIGUE MONITORING SYSTEM AND METHOD TECHNICAL FIELD

The present invention relates generally to systems and methods for optically and capacitively monitoring a driver of a vehicle to determine at least one characteristic, condition, property and/or state of the driver, for example, whether the driver or operator of the vehicle is falling asleep or otherwise unable to operate the vehicle.

BACKGROUND ART

There are several different vehicular situations that are lumped together and called the distracted or inattentive driver problem. These situations include cases where the driver of a vehicle is wide-awake but eating, putting on make-up, texting and talking on the phone; cases where the driver is physically fatigued and may or may not also be sleepy; medical condition cases where the driver suffers a medical condition such as a stroke or heart attack; cases where the driver is under the influence of alcohol or drugs; and cases where the driver is just sleepy. At least one embodiment of the present invention is primarily concerned with detecting a drowsy or sleepy driver but a number of the other causes of the distracted driver also affect the parameters that are the focus of embodiments of the invention and thus will also be addressed, and are encompassed within the scope of the invention in its entirety. A primary focus of at least one embodiment of the present invention is to measure the heartbeat rate and respiration rate of the driver using optics, and through analysis of these rates, predict whether the driver is losing his/her ability to safely operate the vehicle. Driver inattention is a larger problem than driving sleepiness and fatigue. When possible, an analysis of brain waves is also used to further improve the prediction based on heartbeats.

A large number of patents and technical papers address systems for detection of a drowsy driver but despite the potential for such systems, researchers have been largely unsuccessful in finding a feasible way to identify sleepiness or inattention. Many systems have been described in the patent and non-patent literature that attempt to use optical methods, such as video cameras, to detect blinking, head movement, yawning etc., as indications of the drowsy driver. The most successful study has led to the development of the PERCLOS method whereby the percentage of time that the eyes are closed is used to measure drowsiness. Unfortunately, none of these systems have demonstrated a forecasting success rate that permits action to be taken in time for the driver to get to a location where he/she can exit the road on which the vehicle is traveling in time. Studies have shown that only ECG and EEG measurements have this ability to predict that a driver is falling asleep with sufficient time before the driver is unable to operate the vehicle to enable the vehicle to exit the road. At least one embodiment of the present invention, when using optical techniques, provides a method of measuring the heartbeat and respiration rates optically that, with appropriate analysis of the variability of these rates, can achieve a highly accurate prediction of the drowsiness state of the driver. When augmented with brain wave analysis, this accuracy is further improved. A commonly available device, the pulse oximeter, uses light to determine the pulse rate of a patient as well as the amount of oxygen in the blood. This is a device that usually clips to a finger, ear lobe or other part of the body, i.e., physically interacts and contacts the patient's body. A pulse oximeter is generally considered to be a medical device that indirectly monitors the oxygen saturation of a patient's blood (as opposed to measuring oxygen saturation directly through a blood sample) and changes in blood volume in the skin, producing a photoplethysmogram. It will be disclosed below that this can also be accomplished in an unobtrusive manner for the driver of a vehicle. As reported in U.S. Pat. No. 7,277,741, pulse oximetry takes advantage of the fact that in live human tissue, hemoglobin is a strong absorber of light between the wavelengths of 500 and 1100 nm. Pulsation of arterial blood through tissue is readily measurable, using light absorption by hemoglobin in this wavelength range. A graph of the arterial pulsation waveform as a function of time is referred to as an optical plethysmograph. The amplitude of the plethysmographic waveform varies as a function of the wavelength of the light used to measure it, as determined by the absorption properties of the blood pulsing through the arteries. By combining plethysmographic measurements at a plurality of different wavelength regions where oxy- and deoxy-hemoglobin have different absorption coefficients, e.g., two such wavelengths, the oxygen saturation of arterial blood can be estimated. Typical wavelengths employed in commercial pulse oximeters are 660 nm and 890 nm.

Optical measurements of heartbeats, however, are not as accurate as those obtainable using the pulse oximeter. An ECG involving the attachment of multiple sensors onto the skin of a person provides a much better measurement; however, this is impractical for driver applications. However, an intermediate technique using a capacitance sensing device such as disclosed in U.S. Pat. No. 8,694,084 has been found to provide nearly as accurate measurement of the heartbeat as the normal ECG, but without the need for skin attachment. The accuracy achievable is proportional to the distance of the device from the skin and thus when this distance exceeds a few centimeters, an optical method is preferred. Finally it has been found, as reported in U.S. Pat. No. 8,391,967, that signals containing both heartbeat information and brain wave information, similar to both an ECG and an EEG, can be obtained by such a capacitance based sensor when positioned near to the skin of a person providing proper analytical techniques are used to separate the two signals. In U.S. Pat. No. 7,885,700, a technique using two sensors allows the removal of "noise" resulting in a cleaner heartbeat signal. The "noise" contains the brain wave data.

SUMMARY OF THE INVENTION

A method for optically monitoring a driver of a vehicle in accordance with the invention includes obtaining images of the driver using at least one image obtaining device, obtaining brain waves from the driver using at least one sensor, analyzing, using a processor, the images to derive a measure of flow of blood in at least one blood vessel, capillary and vein in the face of the occupant, and analyzing, using the same processor or an additional processor, the blood flow over time and the obtained brain waves to determine whether the driver has lost the ability to continue to control the vehicle. The loss of ability to continue to control the vehicle arises from, for example, the driver becoming drowsy, falling asleep or otherwise being incapable of controlling the vehicle after initially having been awake or otherwise capable of controlling the vehicle.

As a result of the determination of the driver losing the ability to continue to control the vehicle, action may be taken automatically to effect a change in the operation of the vehicle, e.g., brake control, throttle control, steering control or a combination of these. Additionally or alternatively, action by the driver may be required to indicate regaining of the ability to control movement of the vehicle. Additionally or alternatively, the driver may be notified of the determination that they have lost the ability to continue to control the vehicle, required to generate a response to the notification which response may be monitored and then vehicular operation controlled based on the monitoring of the driver for the response to the notification.

Analysis of the images may entail determining a heartbeat of the driver, in which case, analysis of properties of the face over time to determine whether the driver has lost the ability to continue to control the vehicle entails analyzing variability of the heartbeat of the driver.

Analysis of the images may additionally or alternatively entail determining the blood oxygenation, color or motion of the face of the occupant, in which case, analysis of the face over time to determine whether the driver has lost the ability to continue to control the vehicle entails analyzing variability of the color or motion the face of the driver.

The frequency of the electromagnetic radiation may be selected to ensure reflection from blood in the driver. For example, the frequency may be between 500 nm to about 600 nm or more specifically, between about 540 nm to about 580 nm. The illumination may also be modulated.

Brain waves from the driver may be obtained from at least one and possibly two or more capacitance sensors on a seatbelt worn by the driver.

A system for optically monitoring a driver of a vehicle in accordance with the invention includes at least one illumination device that illuminates a portion of the driver with electromagnetic radiation, at least one image obtaining device that obtains images of the illuminated driver, and a processor coupled to the image obtaining device(s) and that analyzes the images to derive a measure of the blood flow, oxygenation, color or motion of the face of the occupant and then analyzes the face over time to determine whether the driver has lost the ability to continue to control the vehicle.

At least one embodiment of the invention further contemplates the parallel use of a pair of capacitance sensors mounted onto the seatbelt to provide better heartbeat information when the seatbelt is worn and positioned near to the driver's chest. Since this is not always the case, when the driver is wearing a heavy winter coat for example, this technique is used to provide improved heartbeat measurements plus an approximate EEG to allow for more accurate analysis and thus an earlier prediction of driver impairment when possible. BRIEF DESCRIPTION OF DRAWINGS

The following drawings are illustrative of embodiments of the system developed or adapted using the teachings of at least one of the inventions disclosed herein and are not meant to limit the scope of the invention as encompassed by the claims.

FIG. 1 is a block diagram illustrating the relationship between the occupant sensing unit and the vehicle safety system in accordance with the invention.

FIG. 2 is a flow chart showing the operation of the driver health and fatigue monitoring routine in accordance with the invention.

FIG. 3 is a flow chart showing the operation of the warning and feedback system routine in accordance with the invention.

FIG. 4A is a view of an optical monitoring system in accordance with the invention that monitors a portion of the face of the driver from an imager mounted near the top of the windshield with an alternate mounting location on the rear view mirror.

FIG. 4B is a view of another embodiment of an optical monitoring system in accordance with the invention that monitors a portion of the face of the driver from an imager mounted in the instrument panel facing so that it reflects off a portion of the windshield.

FIG. 5 illustrates a pair of capacitance sensors mounted onto a seatbelt positioned near the chest of the driver. BEST MODES FOR CARRYING OUT INVENTION

Although many of the examples below relate to a particular vehicle, an automobile, the invention is not limited to any particular vehicle and is thus applicable to all relevant vehicles including all compartments of a vehicle including, for example, the passenger or other compartment of an automobile, truck, farm tractor, construction machine, train, airplane and boat.

"Or" and "and" as used in the specification and claims shall be read in the conjunctive and in the disjunctive wherever they appear as necessary to make the text inclusive rather than exclusive, and neither of these words shall be interpreted to limit the scope of the text.

There is a continuing need to recognize when a driver is distracted in order to regain his attention before an accident results from such distraction.

An inebriated driver can be detected by monitoring his or her heartbeat and/or respiration rate.

Driver behavior indicative of inebriation can be detected optically with pattern recognition software and/or by monitoring the driving behavior of the driver with respect to a map database and accurate vehicle location system or optically using a camera that monitors the vehicle position on the roadway. In both cases, pattern recognition software applied or executed by a computer or other processing unit can be trained to recognize abnormal behavior. Such pattern recognition software is typically embodied on non-transitory computer-readable media available to a processor that is coupled to any system that provides information about the driver or operator. The pattern recognition software may also be in the form of a computer program stored on non-transitory media and that may be installed on the vehicle prior to sale to the consumer. The software and computer program may be updated via a wired link or a wireless link, e.g., via the Internet or another communications network.

Detecting the drowsy or health impaired driver is a different issue generally requiring different hardware and software. Monitoring the eyes of the driver is one approach when the eyes are visible. However, it may be difficult to diagnose a driver having a heart attack by his eye motion, or lack thereof, in time to take remedial action. Also, a driver with sunglasses or even ordinary glasses or a hat with a shade or veil may present difficulties for eye tracking software and then the driver may not be looking at the camera at the time that an event occurs.

Various solutions to detecting a drowsy driver or one that is having a heart attack, stroke or other health related event are disclosed in U.S. patents and published applications assigned to the assignee of this invention or its affiliated company, including: 5829782, 5845000, 20070025597, 6116639, 7887089, 7769513, 6513833, 7660437, 7009502, 6950022, 7788008, 7676062, 7738678, 6793242, 7596242, assigned at issuance to Automotive Technologies International, Inc. and 20080195261, 7629899, 7983802, and 7295925 assigned at issuance to Intelligent Technologies International, Inc. Also, additional solutions are disclosed in U.S. Pat. Appl. Publ. No. 2014/0276090 assigned to Automotive Vehicular Services, Inc. In these patents and applications, the methods disclosed are optical, radar and driving behavior related, for example, to maps. A method that has not been entirely discussed in the published patents and applications and will be discussed here is measuring the variations in the appearance of a patch of skin on the face, or the entire face, of a driver due to heartbeat, respiration rate, blood pressure and temperature. An additional method is to also use data from a seatbelt-mounted capacitance sensor, or a pair of such sensors, to obtain, when possible, a more accurate measurement of the driver' s heartbeat and a measure of his/her brain waves (see the discussion of FIG. 5).

By monitoring driver heartbeat rates and/or respiration rates, a determination can be made as to whether the driver is drowsy and at risk of falling asleep and/or whether he or she is experiencing a health event that could interfere with his or her ability to operate a vehicle. This determination can be significantly improved with accurate heartbeat measurements and brain wave measurements.

The facial monitoring approach, which is being developed at the MIT Media Lab and which has been considered for patient monitoring, uses a facial image to determine the flow of blood in the face vessels and then can derive an accurate pulse reading. The respiration rate and blood pressure readings can also be determined from the facial image. While these techniques are in the experimental stage, their incorporation into a vehicle for the purpose of driver health monitoring has not been considered by others and is believed to be unique to inventions disclosed herein. With this system, there is only a camera that can view the driver's face with green and, in some cases, IR illumination. Neither green nor IR illumination has been disclosed in the MIT study, although green frequencies reflected from the face from natural or broad spectrum illumination is discussed. Thus, a single camera can determine eye tracking, closure state, blinking rate and pulse and respiration measurements, as well as blood pressure and blood oxygenation.

When possible, a more accurate measurement of heartrate and additionally a measure of brain waves can be obtained, as discussed above, through the use of one, or better two, capacitance sensors mounted onto a seatbelt near the chest of the driver.

All driver monitoring systems are imprecise and thus subject to false alarms and consequently, before any action is taken to take control of any vehicle systems, the diagnosis should ideally be verified through driver feedback. This feedback can take many forms such as employing a heads-up display on the windshield or elsewhere in the view of the driver that display a query asking the driver to respond, such as orally or by depressing the horn or other switch or button, or making a gesture, for example. Other feedback systems include an oral alarm or verbal statement requesting a response from the driver or any other visual or audible alarm or message requiring the driver's attention and subsequent action indicating that the sensed event was a false alarm. Failing to receive a proper response, the driver monitoring system can repeat the request a number of times, wherein the specific number may be dependent on or determined by the urgency of the situation (i.e., a more urgent situation might require more requests than a less urgent situation). In one example, the driver monitoring system can ask the driver "are you alright?" and if the driver responds with something like "yes", then the system can be reset. If the driver fails to respond to a simple verbal request for a simple verbal response, the request can be changed to a more complex request such as "if you are alright, depress the horn button or raise your hand." If the driver fails to respond appropriately to this more complex request, one or more of a variety of actions can be taken by the vehicle system depending on an assessment on the vehicle as to what is most appropriate.

The variations at particular frequency ranges corresponding to the human heartbeats (about 30 Hz to about 200 Hz) and respiration rates (about 0.05 to about 2 Hz) in particular are monitored as indicators of drowsiness. Additionally, if seatbelt-mounted capacitance sensors are present close enough to the driver's chest, the brain waves captured by the capacitance sensors can determine the presence and relative magnitudes of the delta (up to 3 Hz), theta (4 Hz to 8 Hz), alpha (8 Hz to 12 Hz), beta (12 Hz to about 30 Hz), and gamma (approximately 26 Hz to 300 Hz) brain waves. As described in U.S. Pat. No. 8,391,967, these waves can provide a good indication of the attentiveness state of the driver.

Gross occupant movement or even the movement of his or her extremities can also be monitored and, even without knowing the exact motion detected, this information can be used to collaborate, confirm and/or verify the analysis of the respiration, heartbeat and brain wave signals. The passenger can tell if the driver is having a heart attack or falling asleep. Therefore, an optical or other monitoring system can also do so.

Thus, the driver monitoring system in accordance with the invention includes an algorithm embodied on storage media that is accessed by the processor to select requests, i.e., a request selection algorithm, based in part on a history of requests that may be maintained in a memory unit also accessed by the processor. The actions that may be taken can include a gradual slowing of the vehicle and flashing the emergency lights, sounding the horn for example in a pulsed alarm mode, taking control of the steering to maintain the vehicle in its lane using lane monitoring cameras, if available, or GPS or equivalent with an accurate map database if available, and avoiding a collision, notifying a remote site, allowing a remote site to assess the situation and take partial control of the vehicle, etc. The action or actions are determined by an action selection algorithm embodied on storage media that is accessed by the processor that can factor in various factors, including, for example, the severity of the situation.

If the driver is instructed by the system to brake or slow the vehicle motion, and there is no indication of the driver doing so, then the system may use this information along with other information to conclude that the driver is incapacitated, and possibly assume partial or total control of the vehicle or take other action (of the type described herein) and notify the driver that the optical monitoring system detects that he or she may no longer be capable of operating the vehicle and ask for feedback. A message to the driver can be in any form, such as a display, sound, light or any other system that can get the attention of the driver. The health monitoring system ECU 210, as shown in FIG. 4B, can also receive information from an optical occupant information providing sensor which sensor may use illumination and acquire images to enable a determination of information about the occupant.

In a preferred implementation, a recorded or synthesized voice can speak a message using speaker-microphone 206 to the driver and ask him or her to perform some task such as respond by saying a word or by depressing the horn or button or performing a gesture such as raising his/her hand or pointing. The ECU 210 can include appropriate command circuitry (hardware/software) to enable the speaker-microphone 206 to generate the sound message, e.g., a message recorded in storage media at the speaker-microphone 206 or at the ECU 210.

A time delay between the message and the action by the driver can be a good indication of the driver's condition. This time delay is computed by a processor in, for example, the health monitoring system ECU 210 that is provided with the time at which the message is conveyed to the driver via the speaker-microphone 206 or other device, and the time at which the driver provides the requested response thereto.

In some installations, it can be desirable to test and score the driver's reaction when he/she starts the vehicle. The health monitoring system can ask one or more times for a response, as above, from the driver in order to determine the alert response time of the driver and to train the driver as to what to expect so that he/she is not surprised at a later time. In some cases, this initial test can determine that the driver is not fit to drive the vehicle and prevent him/her from doing so (via any vehicle operation suppression system known to those skilled in the art). If the driver is the vehicle owner, then he/she should be familiar with the operation of this system and if he/she responds differently than at some previous time, then again vehicle operation can be suppressed. A block diagram illustrating one possible relationship between an occupant sensing unit 120 and a vehicle safety system 130 is illustrated in FIG. 1. An optical and capacitance monitoring system 121 is connected to a diagnostics and data logging module 123 which verifies that the system 121 is operational and logs any error codes. Optical and capacitance monitoring system 121 can also log particular characteristics of a particular driver if a set of individual preferences are identified and associated with the normal drivers of the vehicle. Logging of error codes and particular characteristics of an identified driver may be performed in a manner known to those skilled in the art, with the logged data being stored in storage medium accessible by a processor.

In this manner, the optical and capacitance monitoring system 121 would know what the normal heartbeat rate and respiration rate and the normal alert brain wave patterns of driver 1 is and can then better identify anomalies as they occur, once the driver is identified by accessing the logged data previously entered into the storage medium. Such a driver, for example, may have a history of skipping beats and it must be known that this should not be a concern of the health monitoring system, unless the rate of skipped beats increases which can be attributed to a potential medical problem, alcohol or drug abuse. Similarly caused anomalies can be detected in the brain waves of the driver. Such a system may also be used as a biometric measure to identify the driver along with normal brain waves, heartbeat rate and respiration rate, for example.

Optical and capacitance monitoring system 121 can also provide data for the passenger occupant identification and location system 122, the conventional use of optical sensors, and the driver health and fatigue monitoring system 124 (see FIG. 1). Occupant identification and location system 122 connects with the vehicle airbag system 131 and controls the vehicle airbag system 131, such control being variable and including causing inflation or deployment of one or more airbags in the airbag system 131 to be suppressed for a rear-facing child seat, for example.

Driver health and fatigue monitoring system 124 similarly connects to a vehicle warning and verification system 132 which tests the driver to see whether he or she is capable of operating the vehicle and if not, control of the vehicle can be taken over by a vehicle control system 133.

An exemplifying, non-limiting flow chart showing the operation of the driver health and fatigue monitoring routine, for any of the embodiments described herein, is shown in FIG. 2. The routine is illustrated generally at 140 and when invoked, starts to acquire optical image and brain wave data 141 which acquires sufficient images to permit determination of the heart rate, respiration rate, color and motion of the driver and sufficient brain waves to determine the relevant frequency data. The color can be used both to aid in determining the heart rate and as a general indicator of driver health. Once sufficient images and brain wave data are acquired, control passes to the modules that acquire respiration 142, acquire heartbeat 143, acquire motion and color 144 and separate brain waves from heart waves 160 and establish or record normal respiration 145, normal heartbeat 146, normal motion frequency and color 147, and normal brain waves 161, what will be termed the established respiration rate, heartbeat rate, color and typical occupant motion and brain waves. The time periods for each of these functions will often differ significantly. Less than one minute is generally sufficient to establish representative brain waves, color and heartbeat rate, several minutes is generally sufficient to establish a representative respiration rate and a longer time may be needed to establish how the particular driver typically moves while operating the vehicle. Establishment of a normal motion is the least important of the four since it will mainly become important when an anomaly has been detected in the heartbeat, brain waves, color or respiration rates. If the driver has stopped his normal motion when there is a significant heartbeat event, then this may confirm that there may be a problem with the driver, rather than a failure of the optical and capacitance monitoring system 121 (alternatively considered a health monitoring system). Thereafter, the respiration, heartbeat, motion and brain waves are acquired at 148, 149, 150, and 162 respectively.

Steps 151, 152, 153 and 163 determine whether any of the measured heartbeat, color, respiration, motion and brain wave rates are normal and if not, i.e., if one is not normal, a warning and feedback system is invoked or activated at 154. If the driver fails a feedback test generated by the warning and feedback system, step 155, then a routine to control the vehicle is invoked or activated at 156. The vehicle control routine that is invoked at step 156 may involve, for example, signaling to a remote site or facility for attempted driver communication and then for exercising vehicle control provided there is sufficient equipment on the vehicle to allow the remote operator to observe the driving conditions, traffic, etc. and thus permit a remote operator to slow the vehicle and guide it off onto a shoulder. Since, in most cases, this may not be possible, sufficient apparatus must be on the vehicle to permit this routine to flash emergency lights, sound the horn and/or bring the vehicle to a safe stop. At this point, the routine can optionally contact the proper authorities so that someone is dispatched to the vehicle.

The vehicle control routine 156 may also be performed partly or entirely by equipment on the vehicle upon command by a software program that is executed when needed. The equipment necessary for the control unit to invoke control of the vehicle includes, but is not limited to, devices that are coupled to the steering wheel or steering system, the brakes or braking system, the throttle system, transmission and/or engine. These devices may be controlled by software to reduce the speed of the vehicle to a stop and direct the vehicle to a safe area, e.g., the side or shoulder of a road. The vehicle control routine 156 may therefore correspond to or encompass a processor and control equipment that is coupled to vehicular parts that affect operation of the vehicle, e.g., the brakes, steering wheel and throttle, and possibly servos or actuators therefor. The manner in which this structure operates to affect vehicular control would be easily understood by those skilled in the automotive art in view of the disclosure herein.

Although not illustrated in the flow chart, the presence and characteristics of the heartbeats and brain waves of vehicle occupants can be valuable information that can and should be communicated by the communications unit to EMS personnel as part of the automatic collision notification message, if it is available. Therefore, the EMS personnel will know whether the occupants are alive and some indication as to their health state. This may be effected using a communication system or a transmission system on the vehicle (not shown) that is coupled to the vehicle safety system 130, or a part thereof, and obtains and transmits the information about the occupant(s) of the vehicle. The transmission may be accompanied by a location of the vehicle, obtained using a location determining system on the vehicle or apart from the vehicle, and may be effected using any known communications protocol, including using the Internet.

An exemplifying, non-limiting flow chart showing the operation of an exemplifying, non- limiting warning and feedback system routine, for any of the embodiments described herein, is shown in FIG. 3. The routine is illustrated generally at 180. A first step in the routine 180, at 181, is to initiate a warning set counter. The second step 182 is to generate and send a message in some form to get the attention of the driver. This message can be in any of several forms, e.g., audible, visual, tactile, such as a warning buzzer or a flashing light which can suffice, but this may confuse a driver that is unfamiliar with the system. If the vehicle is equipped with a heads-up display or other display that will get the attention of an alert driver, then a message can be displayed on one or more of such displays. A preferred message is to have a synthesized or recorded voice announce a message to the driver. A combination of the above warnings can be better still. The message can instruct the driver to take some deliberate action which will not interfere with his ability to operate the vehicle but will nevertheless indicate that he has received the message and responded in a timely manner.

Another display that can be used when sufficient data is available is a light or other visual display that shows or otherwise depicts the calculated awake state of the driver. A driver can monitor this display and get an indication as to whether the system thinks that he/she is falling asleep or otherwise experiencing a decrease in his/her ability to operate the vehicle. A driver that sees that he/she is gradually becoming drowsy can, and will ideally, plan when to stop and rest or get a cup of coffee. Also if the driver is sure that the system is in error, he/she can provide feedback to the system that can be taken into account to improve the system accuracy. One candidate device is the Ambient Orb from Ambient Devices, Cambridge, MA which is programmed to change color based upon the output of a state detection system.

After the message is sent to the driver at step 182, the counter is incremented at 183 and the system waits for a response at 184. If the response is not the expected response or if it is tardy (tardiness being determined relative to a predetermined time threshold that may be a function of, for example, the message or the current situation), then the response is judged to not be acceptable at step 185. If the response is judged acceptable, then control is passed back to the health monitoring routine at step 188. The count of the counter is checked at step 186 after an unacceptable driver response and if the counter exceeds some predetermined limit, such as three, then the vehicle control routine is invoked at step 187. If the counter is below its limit, then control is passed to step 182 and the driver is given another chance to respond to the message. As shown in FIG. 3, the routine returns to step 182 to send a message to the driver. However, if the message is a continuous message, such as one on a display, then the routine could be programmed to return to the increment counter step 183 (see the dotted line in FIG. 3).

Since the situation can be life threatening, it is important to provide only a limited amount of time for the driver to react to the message (the receive driver response step at 184). In the case of a drowsy driver, the system can also provide some advice, especially after a number of detected abnormalities that perhaps the driver should take a break. If the driver does not take the advice and there is a likelihood that the driver will in fact fall asleep, then the authorities can be notified. Such notification may be additional or alternative to the invoke vehicle control routine 187, see step 189 in FIG. 3. This notification may be provided by a communications system or transmission system (not shown in FIG. 3) that is coupled to the optical and capacitance monitoring system 121 and generates and sends an appropriate message relating to the lack of the driver providing a response for the predetermined amount of time to the sent message.

Another feedback system that has been suggested is to provide some change in vehicle operation that requires a driver response and then measure the response time. One such idea is to perturb the steering wheel either with a pulse or with a gradual drift of the vehicle to one side of the road and then measure how quickly and even if the driver makes a response.

FIGS. 2 and 3 show algorithms that may be executed by one or more processors of the occupant sensing unit 120 and warning and vehicle safety system 130 in FIG. 1, or components thereof. The internal structure of the processor(s) is therefore one or more computer programs embodied in or on computer-readable medium that when executed by the processor(s), performs the actions set forth in the flow charts illustrated in FIGS. 2 and 3. The instructions for the processor(s) would be computer code compiled according to the algorithms shown in FIGS. 2 and 3.

FIG. 4A is a view of an optical monitoring system that monitors the face of the driver from a camera and illumination system 200 mounted on or near a ceiling 204. One preferred location, for example, is on or adjacent to the rear view mirror 202 (schematically illustrated) since that location usually has a good view of the driver's face and is not obstructed by the visor. The optical monitoring system comprises a camera (representing at least one image obtaining system) and a source of illumination (representing at least one illumination device), which are collectively represented by the camera and illumination system 200, which can be green or in the non-visible part of the electromagnetic spectrum and in particular, in the infrared (IR) portion of the electromagnetic spectrum. The optical monitoring system also comprises a processor or processing unit coupled to the camera, and optionally the illumination source (the processor may be resident or part of the ECU 210 described below with reference to FIG. 4B). This processor is configured to analyze the images to obtain information or data therefrom, and specifically, from images of blood vessels, capillaries and/or veins or generally of a representative skin patch in a face of the occupant (driver) being imaged. The processor is also configured to determine a normal color of the face of the driver, and then analyze the current color relative to the normal color (e.g., a comparison) to determine whether the driver has lost the ability to continue to control the vehicle.

Although in this described embodiment, the face of the driver is being illuminated and images thereof are obtained during illumination, this is partly as a result of the face being the part of the driver that is most likely to be exposed, i.e., uncovered by a material that would block the illumination. Nevertheless, other parts of the driver may be illuminated and images thereof obtained. These parts should preferably be exposed parts of the driver, where reflections from the skin and blood vessels, capillaries and/or veins and color of the skin and blood may be obtained.

Blood, in particular, reflects in the IR portion of the spectrum and thus permits a clearer image of the blood in the blood vessels, capillaries and veins in the driver's face to be obtained and visible in the images being obtained by the camera of camera and illumination system 200. Pulsations of the blood flow can then be used to measure the heartbeat rate of the driver. In particular, the heartbeat variability can be analyzed by the processor as this permits a determination of the drowsiness and/or other attributes of the driver, or more generally, the loss of ability of the driver to continue to control the vehicle (discussed below).

Another use of this technology is to use the pattern of blood vessels, capillaries and veins in the driver's face as a biometric identifier of the driver. The processor would perform this analysis to identify the driver and then, control starting of the vehicle, adjust seating conditions, adjust environmental conditions, etc., based on pre-set information about the biometrically-identified driver.

If the camera is sensitive to longer wave IR, or if a separate imager is used, the temperature of the driver's face or portions thereof can also be measured by a temperature measurement device or algorithm and compared to ambient temperature as measured by a temperature sensor to determine whether the driver's temperature is normal or indicative of a medical problem. The color and overall motion of the driver's face can also be tracked by the processor, e.g., using a tracking algorithm or under control or command of software available to the processor, to determine the respiration rate of the driver. The face location can be determined in the images using, for example, pattern recognition techniques operated by the processor. Finally, the degree to which the blood capillaries, vessels or veins change in size during a beat or their absolute size, as determined by the processor, can provide a measure of the driver' s blood pressure.

In addition to near and far IR illumination, the pulse rate has been found to be best determined from the green part of the optical spectrum. Since blood is particularly absorbent of frequencies in the 500 nm to 600 nm range, the illumination can contain frequencies in this range. Since green light can be distracting to the driver and vehicle occupants, care must be exercised as to how it is used. Once the driver's face location has been determined by appropriate analysis software embodied on computer - readable media that is executed by the processor, a patch on the forehead, cheek, neck or other convenient place can be chosen as the spot to be illuminated. Then, the amount and frequency of illumination can be adjusted, via control of the source of illumination by the processor or control unit more generally, so that it is barely above, typically 1% to 10% above, that present in the ambient light level. Finally, it can be carefully focused so that it doesn't spill over and illuminate the eyes of the driver. The location of the source of illumination should also be carefully chosen so that reflections do not strike the eyes of other vehicle passengers. Multiple sources of illumination may be also provided and used independently or in combination to ensure the selected spot can always be illuminated.

The green light can also be modulated, via a modulation device or technique, so that it can be separated from ambient light reflections. This may require that the receiving device for the green light be a photodiode or equivalent to permit the modulations to be sensed and separated from the ambient light.

It has been found by several investigators that since the green frequencies, particularly at 540 nm and 580 nm, are the most absorbed, the variations in reflected light are greatest and thus the variations in blood flow rate most easily identified and measured from light in this frequency range.

Once the camera, processor and processing software has chosen a spot to monitor on the driver's face, it is beneficial to track this spot over time. Techniques for accomplishing this tracking are discussed in Borgobello, Bridget "MIT developing webcam-based health monitoring mirror". Press release MIT News Office, October 5, 2010, and in other publications by MIT on this subject referenced in US 20140276090, all of which may be used in the invention.

Camera and illumination system 200 can also be used to track the eye and eyelid motions of the driver as is discussed in prior art eye tracking systems for determining driver alertness. As mentioned above, such systems can be used to augment the monitoring system discussed herein but, when relied on alone, suffer from problems associated with sunglasses and even ordinary glasses which frequently block IR.

FIG. 4B illustrates an alternate arrangement for monitoring the face of a driver. In this implementation, a camera 206 is mounted below the top surface of the instrument panel in such a position as to view the driver's face through a reflection off of a surface 208 of the windshield in a manner similar to heads-up displays. This implementation has the advantage that a larger camera can be used with greater resolution and can contain focusing hardware and even pointing hardware under control of an ECU 210 to obtain a better view of the face of the driver and to more exactly project the artificial illumination and particularly the patch of green or other color light discussed above. This patch projection will move as the driver's head moves, under control of a processor and appropriate control algorithm or program in the ECU 210. In both cases, an alternate approach is to illuminate a significant portion of the driver's face with the green or other color light but to do so at a level slightly above the ambient light level and use more care in separating it from the ambient reflections. Otherwise, the operation in FIG. 4B is similar to that of FIG. 4 A. The same ECU 210 can be used to analyze the driver for signs of fatigue and initiate a warning such as a light 214 which requires a feedback response from the driver, such as depressing button 212, speaking a word, pointing etc. as described above. Γη many cases, it is desirable to displace the illumination source from the receiver which can make variations in blood flow more observable.

There are a number or additional monitoring methods that can yield important information about the health and status of the vehicle driver. A few of these will now be mentioned but many others appear in the medical literature on patient monitoring.

Techniques which are particularly useful for monitoring heart rates and variations thereof are disclosed in references 27 and 28 of US 20140276090 wherein ordinary cameras are used in conjunction with appropriate algorithms to allow a clear measurement of the driver's heart rate. Additionally reference 29 of US 20140267090 discloses the determination of an inebriated driver using thermal imaging. All of the techniques disclosed in these references are encompassed by this invention for the use of driver monitoring.

It is expected that the analysis program will take other information into account when it is available such as the temperature, blood pressure and oxygen level of the blood. This will depend on the particular suite of sensors that are used and such information will not be available, for example, if optical sensors are the only ones being used. Other information that can also be incorporated includes lane departure, steering wheel motion, vehicle speed and acceleration, time of day, head and body motion, state of the eyes and blinking rates, gaze direction and duration and any other measured properties of the occupant, vehicle and how it is being driven and/or the environment.

Other factors that can be included in the analysis depending on the information available include:

1. Pulse recovery can be aided by removing signals with frequency lower than about 0.67 Hz (40 bmp) and higher than about 1.67 Hz (100 bmp).

2. The heart rate decreases during sleep by about 5-40 %.

3. Blood pressure decreases during sleep to about 77-80 % of its value when the driver is awake.

4. The total spectrum power and its individual components in absolute values in different sleep stages HF (ms2) 1148 + 1148 (Wakefulness) 2277 + 2878 (Stage 2 non-REM) 1777 + 2789 (Stage 4 non-REM) 2073 + 3158 (REM sleep)

5. A reduction of the LF/HF ratio in the course of falling asleep.

6. The "HRV" is much larger during sleep. That is, one beat interval may be about 0.9 seconds and the next one about 1.3 seconds in the sleeping person while in the awake person, about 0.7 to about 0.8 seconds. (The heartbeat rate varies over time, and is known by the acronym HRV for heart rate variability. HRV is a measure of the oscillation of the interval between consecutive heartbeats.)

7. Typically, the differences in the cardiac dynamics during sleep and wake phases are reflected in the average (higher in sleep) and standard deviation (lower in sleep) of the interbeat intervals. It is important to keep in mind that EEG results demonstrate that it is feasible to accurately estimate quantitatively driving performance. Thus, the best system will be to measure EEG which is not economically feasible at this time but may be in the future. In particular, the Plessey EPIC sensor suite shows promise of providing this capability.

Other factors mentioned above include the blood oxygenation, color or motion of the face of the occupant.

FIG. 5 illustrates the placement of the capacitance sensors 310 for obtaining the brain waves mounted on a seatbelt 304 as worn by occupant 302. The construction of these sensors 310 is described in patents mentioned above and incorporated by reference herein. For some embodiments of the invention, the sensors 310 are arranged on, mounted on or integrated into the seatbelt 304 so that the will be on a portion of the seatbelt 304 in front of the occupant 302 when the seatbelt 304 is worn, and ideally close to or in front of the occupant' s heart. This positioning may be determined in a preparatory stage by having different sized occupants sit in the seat, and then put the seatbelt 304 on, and then mark on the seatbelt 304 where their heart is located. The marked locations can then be considered for placement of the sensors 310, with a view toward covering as many different sized occupants as possible. Different seatbelts 304 may have different positions of sensors 310.

The occupant 302 may be the driver of the vehicle, in which case, data about the driver derived from data or signals from sensors 310 may be used according to the techniques described above. Instead of capacitance sensors 310, other sensors or sensor arrangements of one or more sensors that enable the same data or signals to be obtained may be used and arranged or mounted on, or integrated into, the seatbelt 304.

Other reactive components may be coupled to the processor and perform cognitive tests including requiring an oral response, visual response or gesture response, for example, to the cue. Thus, the reactive component may be one which requires feedback from the driver, i.e., a detection of speech. In other words, if the driver is already looking at the road, then he can orally tell the system that he is awake and attentive.

Another factor to be used by the processor is the amount of time when the driver is not looking at the road ahead. If this exceeds, e.g., 2 seconds, then the processor could use this factor in its determination of inattentiveness. Yet another factor is head orientation which may be determined by analysis of the images of the driver. Most of the time the driver is looking forward, so the position in which the driver's head is in most of the time may be considered the normal position and variations from this position for an extended period of time may be indicative of the driver's inattentiveness.

Although the primary application of technology disclosed above is for preventing accidents caused by driver drowsiness, other applications are to apply the technology to airplane pilot and train conductors (or more generally vehicular operators) and other passengers in automobiles and planes (or more generally vehicular occupants) to detect medical conditions that should receive immediate attention, such as heart attacks, seizures and strokes as will be discussed below. Thus, as used herein, a driver should be considered to include an operator or a vehicle, a conductor of a vehicle, a pilot of a vehicle, etc.

The invention can be implemented in numerous ways, including potentially as a process; an apparatus; a system; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor. In this specification, these implementations, or any other form that the invention may take, may be referred to as techniques. In general, the order of the steps of disclosed processes may be altered within the scope of the invention

Although several preferred embodiments are illustrated and described above, there are possible combinations using other geometries, sensors, materials and different dimensions for the components that perform the same functions. At least one of the inventions disclosed herein is not limited to the above embodiments and should be determined by the following claims. There are also numerous additional applications in addition to those described above. Many changes, modifications, variations and other uses and applications of the subject invention will, however, become apparent to those skilled in the art after considering this specification and the accompanying drawings which disclose the preferred embodiments thereof. All such changes, modifications, variations and other uses and applications which do not depart from the spirit and scope of the invention are deemed to be covered by the invention which is limited only by the following claims.

Claims

1. A method for monitoring a driver in a vehicle, comprising:

obtaining images of the driver using at least one image obtaining device;

obtaining brain waves from the driver using at least one sensor;

analyzing, using a processor, the images to derive a measure of flow of blood in at least one blood vessel, capillary and vein in the face of the occupant; and

analyzing, using the same processor or an additional processor, the blood flow over time and the obtained brain waves to determine whether the driver has lost the ability to continue to control the vehicle that arises from the driver becoming drowsy, falling asleep or otherwise being incapable of controlling the vehicle after initially having been awake or otherwise capable of controlling the vehicle.

2. The method of claim 1, further comprising taking action to effect a change in the operation of the vehicle upon a determination that the driver has lost the ability to continue to control the vehicle.

3. The method of claim 1, further comprising requiring action by the driver to indicate regaining of the ability to control movement of the vehicle after a determination that the driver has lost the ability to continue to control the vehicle has been made.

4. The method of claim 1, further comprising:

notifying the driver of the determination that they have lost the ability to continue to control the vehicle;

requiring the driver to generate a response to the notification;

monitoring the driver for the response to the notification; and then

controlling vehicular operation based on the monitoring of the driver for the response to the notification.

5. The method of claim 1, further comprising analyzing the obtained images using the same processor or an additional processor to derive a measure of color of a face of the driver, the step of analyzing the blood flow over time and the obtained brain waves comprising analyzing the color of the face of the driver over time in combination with analysis of the blood flow over time and the obtained brain waves to determine whether the driver has lost the ability to continue to control the vehicle comprises analyzing variability of the color of the face of the driver.

6. The method of claim 5, wherein the step of analyzing the images using a processor to derive a measure of color of a face of the driver comprises determining a normal color of the face of the driver, and the step of analyzing the color of the face of the driver over time to determine whether the driver has lost the ability to continue to control the vehicle comprises comparing the current color of the face of the driver to the normal color of the face of the driver.

7. The method of claim 1, further comprising using pattern recognition to determine a location of the face of the driver from images being obtained by the at least one imaging device.

8. The method of claim 1, wherein the step of obtaining images of the driver using at least one image obtaining device comprises obtaining at least one initial image,

further comprising analyzing the at least one initial image to determine location of the face of the driver and selecting a portion of the face of the driver for infrared illumination; and then

illuminating only the selected portion of the face of the driver with electromagnetic radiation using at least one illumination device.

9. The method of claim 1, further comprising analyzing the images to detect presence of alcohol or drugs in blood in at least one blood vessel, capillary and vein in the face of the occupant.

10. The method of claim 1, further comprising illuminating a portion of the driver with electromagnetic radiation using at least one illumination device prior to obtaining images of the illuminated driver using at least one image obtaining device such that the images are of the illuminated portion of the driver.

11. The method of claim 10, wherein the at least one illumination device and the at least one imaging device are situated in a common unit.

12. The method of claim 10, wherein the step of illuminating the driver with electromagnetic radiation using at least one illumination device comprising illuminating the driver with infrared illumination in a range from about 500 nm to about 600 nm.

13. The method of claim 10, wherein the at least one illumination device, the at least one imaging device and the processor are co-located, further comprising transmitting analysis of the blood flow over time and the obtained brain waves to a remote location separate and apart from the location at which the at least one illumination device, the at least one imaging device and the processor are co- located.

14. The method of claim 10, wherein the at least one illumination device, and the at least one imaging device are arranged on a common frame.

15. The method of claim 10, further comprising determining the portion of the driver to illuminate with electromagnetic radiation using the at least one illumination device by:

illuminating the person with electromagnetic radiation using the at least one illumination device;

obtaining an initial image of the illuminated person using the at least one image obtaining device;

analyzing the initial image using the processor to determine location of the face of the person; and then

selecting the portion of the person being illuminated based on the determined face location, which selected portion is smaller than a part of the person illuminated to obtain the initial image.

16. The method of claim 15, further comprising monitoring movement of the portion of the face of the driver selected for illumination.

17. The method of claim 1, wherein the step of obtaining brain waves from the driver using at least one sensor comprises obtaining brain wave data from at least one capacitance sensor on a seatbelt worn by the driver.

18. A system for monitoring a driver in a vehicle, comprising:

an optical sensing unit comprising

at least one image obtaining device that obtains images of the driver; at least one sensor that obtain brain wave data about the driver; and

a processor coupled to said at least one image obtaining device and to said at least one sensor and that analyzes the obtained images and the obtained brain wave data to derive a measure of flow of blood in at least one blood vessel, capillary and vein in the face of the occupant, and then analyzes the blood flow over time and the obtained brain waves to determine whether the driver has lost the ability to continue to control the vehicle that arises from the driver becoming drowsy, falling asleep or otherwise being incapable of controlling the vehicle after initially having been awake or otherwise capable of controlling the vehicle; and

a vehicle safety system coupled to said occupant sensing unit and comprising

a warning and verification system configured to tests the driver to see whether the driver is capable of operating the vehicle after having been determined to have lost the ability to continue to control the vehicle; and

a vehicle control system that controls the vehicle when said warning and verification system determines that the driver is not able to operate the vehicle.

19. The system of claim 18, further comprising a seatbelt configured to be worn by the driver, said at least one sensor being arranged on a portion of said seatbelt adapted to be in front of the driver when the driver is wearing said seatbelt.

20. The system of claim 18, wherein said processor is further configured to analyze the obtained images to derive a measure of color of a face of the driver, and then analyzes the color of the face of the driver over time in combination with analysis of the blood flow over time and the obtained brain waves to determine whether the driver has lost the ability to continue to control the vehicle that arises from the driver becoming drowsy, falling asleep or otherwise being incapable of controlling the vehicle after initially having been awake or otherwise capable of controlling the vehicle. (Fig. 4B)