WO2016061351A1

WO2016061351A1 - Wearable device for operator degraded performance notification with location-based remedial response

Info

Publication number: WO2016061351A1
Application number: PCT/US2015/055734
Authority: WO
Inventors: Dan Atlas
Original assignee: Dan Atlas
Priority date: 2014-10-15
Filing date: 2015-10-15
Publication date: 2016-04-21

Abstract

A wearable device for an operator to wear upon the wrist, for automatically detecting the onset of degraded performance in the operator; and methods for using such a device. The device utilizes technologies such as surface Electromyography to non-invasively determine the gripping strength of the operator on an operating handle, such as a steering wheel. False positives are eliminated by performing a psychomotor vigilance test (PVT) to confirm a detected onset of degraded performance or to filter out a spurious or anomalous detection of the onset of degraded performance. In other embodiments, a degraded performance notification is sent to a remote server along with the operator's current location to provide a remedial location-based response to relieve the onset of degraded performance. In still other embodiments, a dedicated GPS navigation device provides location-based information to relieve the onset of degraded performance.

Description

WEARABLE DEVICE FOR OPERATOR DEGRADED PERFORMANCE NOTIFICATION WITH LOCATION-BASED REMEDIAL RESPONSE

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

None.

CROSS REFERENCE TO RELATED APPLICATIONS

[001 ] This application claims priority to and the benefit of U.S. Provisional Patent Application Serial No. 62/063,960, filed October 15, 2014, entitled "Device for detecting and alerting physiological onset of operator drowsiness and inattention", which is hereby incorporated by reference in its entirety.

BACKGROUND

[002] There is considerable interest in systems for monitoring a human operator of equipment or machinery, such as vehicles, and issuing an alerting signal whenever signs of operator degraded performance are detected. In addition to the common hazard of fatigue and drowsiness, similar behavioral lapses include, but are not limited to: mental overload, side-effects of medication, texting, boredom, general inattentiveness, or any other factors which distract the operator or otherwise reduce alertness and performance fitness. A monitoring and alerting system for such conditions is denoted herein as a "degraded performance monitoring system" (DPMS), and the term "degraded performance" herein denotes any condition which results in a decrease of attentiveness, vigilance, and/or any other diminishing of the operator's ability to properly carry out operations,

[003] Typically a degraded performance monitoring system includes multiple sensors attached to the operator's body and/or to an operational handle of the equipment, such as a control of a vehicle. In the particular case of a vehicle, such sensors are typically attached to, or imbedded in, the steering wheel, or in a sleeve covering the steering wheel. In prior art fatigue monitoring systems, output signals from gripping pressure sensors are typically processed (such as by filtering and normalization), and the intensities are compared with respective threshold levels. An alerting signal is generated whenever certain signals or a function thereof crosses one or more threshold levels.

[004] Several disadvantages are inherent in prior art processing of the sensor signals. The primary disadvantage is due to the fact that gripping pressure, even by a fully alert operator, may vary with time, as a function of, for example, road conditions. Comparison of signal intensity with a fixed threshold may yield varying results, wherein the state of operator degraded performance may be misinterpreted, and thus give rise to false alerting signals (false positives), or lack of alerting signals (false negatives). Another disadvantage is the general variability of output signals among the sensors as well as in time. Prior art processing does not account for this variability, resulting in reduced sensitivity and accuracy. A further prior art disadvantage lies in the manner of selecting (if any selection is done) only those signals from sensors that are responsive to gripping pressure at a particular time - proper selection is necessary to minimize noise in the signal, without which system accuracy is diminished. Moreover, beyond providing an alerting signal, prior art systems offer the operator no options for remedial action to relieve or compensate for the onset of degraded performance. This disadvantage is especially acute in vehicular environments, where the operator may be alone and in an unfamiliar location.

[005] Thus, it would desirable to have a. degraded performance monitoring system that overcomes the above disadvantages. This goal is attained by embodiments of the present invention. SUMMARY

[006] A wearable device for an operator to wear upon the wrist, for automatically detecting the onset of degraded performance in the operator; and methods for using such a device. Embodiments of the present invention provide a wrist-mounted wearable device for detecting the onset of operator degraded performance, in a manner that obviates the need for sensors mounted on the equipment (such as steering wheel gripping sensors, although similar operational algorithms are applicable), and whose accuracy is unaffected by variability in sensor output. In various embodiments of the present invention, the wrist-mounted wearable device communicates with a remote server to provide a location-based remedial response to the operator, for relieving or compensating for the onset of degraded performance.

[007] In the descriptions and figures herein, certain embodiments of the present invention are illustrated for clarity and simplicity in terms of the non-limiting case of a motor vehicle operator. It is understood that embodiments of the present invention are not limited to this case, but may be used in other environments as well.

[008] Therefore, according to various embodiments of the present invention, there is provided a method for detecting the onset of degraded performance in an operator, the method including: (a) providing a wearable device for a wrist of the operator, the wearable device having a sensor for determining a grip strength of the operator on an operating handle; (b) initializing the wearable device when worn by the operator when the operator is not gripping the operating handle; (c) establishing a grip strength baseline and a reaction time baseline when the operator initially grips the operating handle; (d) while the operator is operating, comparing, by the wearable device, a current grip strength to the grip strength baseline; (e) if a decrease of the grip strength from the grip s trength baseline exceeds a grip strength decrease threshold, then performing, by the wearable device, a psychomotor vigilance test (PVT), which includes: (f) comparing an increase in a PVT reaction time over the reaction time baseline to a reaction time increase threshold; and (g) if the increase in the P VT reaction time exceeds the reaction time increase threshold, then generating a degraded performance notification.

[009] In addition, according to another embodiment of the present invention there is provided a wearable device for a wrist of an operator, for detecting the onset of degraded performance in the operator, the device including: (a) a sensor for determining a grip strength of the operator on an operating handle; and (b) a data processor; (c) wherein the device is arranged to: (d) establish a grip strength baseline and a reaction time baseline when the operator initially grips the operating handle; (e) compare a current grip strength to the grip strength baseline; (f) perform a psychomotor vigilance test (PVT), which includes: (g) comparing an increase in a PVT reaction time over the reaction time baseline to a reaction time increase threshold; and (h) if the increase in the PVT reaction time exceeds the reaction time increase threshold, then generating a degraded performance notification.

BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The subject matter disclosed may best be understood by reference to the following detailed description when read with the accompanying drawings in which:

[001 1] Fig. 1A conceptually illustrates a wearable wrist-mounted w^rearable device according to an embodiment of the present invention.

[0012] Fig. IB conceptually illustrates a. wearable wrist-mounted wearable device according to another embodiment of the present in vention. [0013] Fig. 2 conceptually illustrates the wearable device of Fig. 1A being worn by an operator of a motor vehicle, according to an embodiment of the present invention.

[0014] Fig. 3 is a flowchart of a method for detecting the onset of operator degraded performance, according to an embodiment of the present invention.

[0015] Fig. 4 conceptually illustrates a configuration for providing the operator with a remedial response for the onset of degraded performance, according to an embodiment of the present invention.

[0016] Fig, 5 is a flowchart of a method for receiving a degraded performance notification and providing a remedial response thereto to relieve or compensate for the onset of operator degraded performance according to an embodiment of the present invention.

[0017] For simplicity and clarity of illustration, elements shown in the figures are not necessarily drawn to scale, and the dimensions of some elements may be exaggerated relative to other elements. In addition, reference numerals may be repeated among the figures to indicate corresponding or analogous elements.

DETAILED DESCRIPTION

[0018] Fig. 1A conceptually illustrates a wearable wrist-mounted wearable device 100 according to an embodiment of the present invention. In various embodiments, device 100 does not have a display or direct user input capabilities. In certain other embodiments, however, a face screen 101 provides a display of information to the wearer, such as for use as a wristwatch. Tn another related embodiment, face screen 101 is a touchscreen for input by the wearer, as in a smart watch. In a related embodiment, a wrist strap top portion 103.4 can be removably affixed to a wrist strap bottom portion 103B to hold wearable device 100 on the wearer's wrist. In a related embodiment, strap top portion 1Θ3Α is connected to strap bottom portion 103B by a clasp. A wearable device back surface 105 provides an area for transducers and/or sensors to contact the dorsal aspect (back of hand- side) of the wearer's wist; an inner strap surface 107 provides an area for transducers, EMG electrodes, and/or other sensors to contact the volar aspect (palm-side) of the wearer's wrist,

[0019] According to various embodiments of the present invention, wearable device 100 is configured with sensors and/or transducers to facilitate detection of the onset of degraded performance in the wearer. In one embodiment, a sensor having a multiplicity of electrodes is deployed on inner strap surface 107 for use in surface Electromyography (sEMG), to detect the activity of the muscles in the wearer's forearm, and thus detect a grip strength of the wearer. According to certain embodiments of the invention, the area of inner strap surface 107 is optimal for surface Electromyography, because it is proximate to the wearer's lower wrist surface, which typically is devoid of hair, thereby providing a better surface contact with the skin to the electrodes. According to another embodiment, sensors are deployed for use in Tensionmyography (TMG), to detect the activity of the muscles in the wearer's forearm.

[0020] According to certain embodiments of the invention, wearable device back surface 105 contains transducers for providing tactile vibration to the wearer's wrist, for signaling to the user that a psychomotor vigilance test (PVT) has been initiated.

[0021] According to additional embodiments of the invention, wearable device 100 includes an internal data processor 109 for performing data processing functions associated with the functioning of wearable device 100: and a wireless data communication interface

[1] for establishing a wireless data link 113 to an external device. [0022] Fig. IB conceptually illustrates a compact wearable wrist-mounted wearable device 150 according to additional embodiments of the present invention. In these embodiments, device ISO does not have a display or direct user input capabilities. The compact nature of device 150 permits wearing adjacent to a regular wristwatch as well as in place thereof. In a related embodiment, a wrist strap top portion 153 A can be removably affixed to a wrist strap bottom portion 153B to hold wearable device 150 on the wearer's wrist. In a related embodiment, strap top portion 153 A is connected to strap bottom portion 153B by a clasp. A device back surface 155 provides an area, for transducers and/or sensors to contact the dorsal aspect (back of hand-side) of the wearer^'s wrist; an inner strap surface 157 provides an area for transducers, EMG electrodes, and/or other sensors to contact the volar aspect (palm-side) of the wearer's wrist.

[0023] According to further embodiments of the invention, wearable device 150 includes an internal data processor 159 for performing data processing functions associated with the functioning of wearable device 150; and a wireless data communication interface 161 for establishing a wireless data link 163 to an external device.

[0024] Fig. 2 conceptually illustrates wearable device 160 being worn by an operator 201 of a motor vehicle, according to an embodiment of the present invention. Operator 201 is shown gripping a steering wheel 203.

[0025] Fig. 3 is a flowchart of a method 300 for detecting the onset of operator degraded performance, according to an embodiment of the present invention. In a step 301 wearable device 100 is affixed to the wrist of operator 201 (i.e., the operator is the "wearer"). In a step 303 wearable device 100 is initialized when operator 201 is not gripping an operating handle 203. In a related embodiment, operating handle 203 is a steering wheel of a motor vehicle. In a step 305 a reaction time baseline 307 and a grip strength baseline 309 are established when operator 201 initially grips handle 203. Then, in a step 311, operator 2(11 begins operating. In the related embodiment, operating is driving the motor vehicle. In a step 313 a current grip strength is compared to grip strength baseline 309. At a decision point 315 a decrease of the current grip strength from grip strength baseline 3Θ9 is checked to see if it exceeds a grip strength decrease threshold of threshold data 317, which contains values of multiple thresholds associated with the respective parameters to which the thresholds apply. If the grip strength decrease threshold is not exceeded, then step 313 is repeated. Ideally, if operator 201 remains alert is and not performance-impaired, step 313 will simply be repeated continually throughout the operating. If, however, the decrease in grip strength exceeds the grip strength decrease threshold, then in a step 321 a psychomotor vigilance test (PVT) is performed by wearable device 100. Wearable device 100 perforins the PVT as previously noted, wherein a vibration of any surface of wearable device 100 notifies the wearer to strongly and rapidly grip operating handle 203 for the PVT to be performed.

[0026] The PVT either confirms that operator 201 is experiencing the onset of degraded performance, or determines that the detected grip strength decrease was incidental or anomalous, and did not accurately indicate the onset of degraded performance. In a step 323 the PVT grip strength and PVT reaction time are compared to grip strength baseline 309 and reaction time baseline 307, respectively. At a decision point 325 a detected decrease of PVT grip strength and/or a detected increase of PVT reaction time are checked to see if they exceed respective thresholds in threshold data 317. In a related embodiment of the invention, both thresholds must be exceeded in order to qualify as exceeding the thresholds at decision point 325. In another related embodiment, exceeding either threshold qualifies as exceeding the thresholds at decision point 325. In the case, where the PV T shows that operator 201 is experiencing the onset of degraded performance, exceeding the threshold leads to a step 331 in which a degraded performance notification is generated. In the case where the PVT shows that operator 201 is not experiencing the onset of degraded performance, not exceeding the threshold leads to a repeat of step 313. The PVT offers an additional benefit of helping sustain the operator for a period of time, by rousing the operator to a state of greater alertness, even if only temporarily.

[0027] In an embodiment of the present invention, one or more of the thresholds in threshold data 317 can be modified to account for changes in environmental conditions. In a related embodiment, if the speed of a v ehicle operated by operator 201 decreases below a speed threshold in threshold data 317, then the grip strength decrease threshold and the reaction time increase threshold are disabled. In a non-limiting example, if the vehicle is stopped at a traffic light, operator 201 may temporarily release steering wheel 203, without triggering a degraded performance notification.

[0028] Fig. 4 conceptually illustrates a configuration for providing the operator with a remedial response for the onset of degraded performance, according to an embodiment of the present invention. In this embodiment, operator 201 is driving a motor vehicle while wearing wearable device 100. In a related embodiment, wearable device 100 has a wireless data, link 407 (such as a Bluetooth link) to a vehicle wireless data access point 421, and from thence to a vehicle data gateway 405 which has a wireless link 415 to Internet 461, and from thence via a link 443 to a remote server 441 having access to map data 445. In another related embodiment, wearable device 100 has a wireless data link 403 to an external device 401, from thence via a wireless link 413 to Internet 461, and from thence via link 443 to remote server 441 as before. In a related embodiment, external device 401 is a smartphone. In another related embodiment, external device 401 is a tablet computer, in still another related embodiment, wireless data access point 421 functions as an external device with data processing and storage capabilities similar to those of external device 401. In various embodiments of the invention, wearable device 100 has a virtual connection 451 to remote server 441, to which it can send a degraded performance notification 431 and from which it can receive a remedial response 433. In addition, both wireless data access point 421 and external device 401 are equipped with GPS receivers to provide GPS location data 419. In yet another related embodiment, device 100 contains a GPS receiver and directly receives GPS location data 419. In a further related embodiment, a dedicated GPS navigation device 409 receives location data and provides navigational information to device 100 via a wireless link 411 without the need for an Internet connection, via internal map data 447,

[0029] In a related embodiment, remote server 441 provides navigational and/or mapping services, non-limiting examples of which include Google Maps and Waze. A fatigued driver is alerted to possible safe stops ahead, with priorities linked to the number of PVT's issued, and thus to the state of performance degradation.

[0030] Fig, 5 is a flowchart of a method 500 for receiving degraded performance notification 431 and providing remedial response 433 to relieve or compensate for the onset of operator degraded performance according to an embodiment of the present invention. In a step 501 degraded performance notification 431 is received. According to a. related embodiment, in a step 503 degraded performance notification 431 sent along with GPS location data 419 to a remote server 441. GPS location data 491. identifies the geographical location of the operator. In another related embodiment, GPS location and navigation functions are performed by device 100, and in still another related embodiment, GPS location and navigation functions are performed by dedicated GPS navigation device 409 which has access to map data 447 without using a remote server.

[0031 ] In a related embodiment of the present invention, degraded performance notification 431 is generated and sent by wearable device 100, and step 503 is performed by wearable device ίθθ. In another related embodiment, steps 501 and 503 are performed by an intermediate device, such as by data access point 421, or by external device 401.

[0032] In a step 51)5 map data 445 or map data 447 is accessed, and in a step 507, location-based information relevant to the remedy or relief of the degraded performance condition of operator 201 is retrieved from map data 445 or map data 447. In a non- limiting example, such information could include navigation instructions to the nearest available rest stop or similar area.,

[0033] In a step 509, the retrieved information is sent to operator 201 as part of remedial response message 433, which is according to the retrieved location-based information. In the non-limiting example above, remedial response message 433 could include a suggestion such as: "... your driving responses indicate drowsiness or fatigue - please take these directions to a nearby rest stop, and rest for a few minutes to refresh yourself before continuing..." followed by the retrieved navigational directions.

[0034] According to various embodiments, steps 505, 507, and 509 are performed by remove server 441; and remedial response 433 is displayed to the operator. In a related embodiment, the display is via wearable device 100. In another related embodiment, the remedial response display is via external device 401. In a further related embodiment, the display is visual; and in an additional embodiment, the display is aural, such as via. synthesized or recorded speech, sound effects, and so forth. In various other embodiments, visual and aural displays are combined for the remedial response display. [0035] According to other various embodiments, steps SOS, £507, and 509 are performed by dedicated GPS navigation device 409. In still further embodiments, steps 505, 587, and 509 are performed by device 100,

[0036] A method for detecting the onset of degraded performance in an operator, the method comprising: providing a wearable device for a wrist of the operator, the wearable device having a sensor for determining a grip strength of the operator on an operating handle; initializing the wearable device when worn by the operator when the operator is not gripping the operating handle; establishing a grip strength baseline and a reaction time baseline when the operator initially grips the operating handle; while the operator is operating, comparing, by the wearable device, a current grip strength to the grip strength baseline; if a decrease of the grip strength from the grip strength baseline exceeds a grip strength decrease threshold, then performing, by the wearable device, a psychomotor vigilance test (PVT), which includes: comparing an increase in a PVT reaction time over the reaction time baseline to a reaction time increase threshold; and if the increase in the PVT reaction time exceeds the reaction time increase threshold, then generating a degraded performance notification.

[0037] A wearable device for a wrist of an operator, for detecting the onset of degraded performance in the operator, the device comprising: a sensor for determining a grip strength of the operator on an operating handle; and a data processor; wherein the device is arranged to: establish a grip strength baseline and a, reaction time baseline when the operator initially grips the operating handle; compare a current grip strength to the grip strength baseline; perform a psychomotor vigilance test (P VT), which includes: comparing an increase in a PVT reaction time over the reaction time baseline to a reaction time increase threshold; and if the increase in the PVT reaction time exceeds the reaction time increase threshold, then generating a degraded performance notification,

[0038] AH patents, patent applications, and published references cited herein are hereby incorporated by reference in their entirety. It should be emphasized that the above- described embodiments of the present disclosure are merely possible examples of implementations, merely set forth for a clear understanding of the principles of the disclosure. Many variations and modifications may be made to the above-described embodiment(s) without departing substantially from the spirit and principles of the disclosure. It will be appreciated that several of the above-disclosed and other features and functions, or alternatives thereof, may be desirably combined into many other different systems or applications. All such modifications and variations are intended to be included herein within the scope of this disclosure, as fall within the scope of the appended claims.

Claims

CLAIMS What is claimed is:

1. A method for detecting the onset of degraded performance in an operator, the method comprising:

providing a wearable device for a wrist of the operator, the wearable

device having a sensor for determining a grip strength of the operator on an operating handle;

initializing the wearable device when worn by the operator when the

operator is not gripping the operating handle;

establishing a grip strength baseline and a reaction time baseline when the operator initially grips the operating handle; while the operator is operating, comparing, by the wearable device, a

current grip strength to the grip strength baseline; if a decrease of the grip strength from the grip strength baseline exceeds a grip strength decrease threshold, then performing, by the wearable device, a psychomotor vigilance test (PVT), which includes:

comparing an increase in a PVT reaction time over the

reaction time baseline to a reaction time increase threshold; and

if the increase in the PVT reaction time exceeds the reaction

time increase threshold, then generating a degraded performance notification.

2. The method of claim 1 , wherein the psychomotor vigilance test (PVT) further includes:

comparing the current grip strength to the grip strength baseline;

if a decrease of a P VT grip strength from the grip strength baseline exceeds the grip strength decrease threshold, then generating a degraded performance notification.

3. The method of claim 1 , wherein the psychomotor vigilance test (PVT) further includes:

comparing the current grip strength to the grip strength baseline;

if the increase in the PVT reaction time exceeds the reaction time

increase threshold,, and also

if the decrease of a PVT grip strength from the grip strength baseline exceeds the grip strength decrease threshold, then generating a degraded performance notification,

4. The method of claim 1 , wherein the degraded performance notification is generated by the wearable device.

5. The method of claim 1, wherein the wearable device has a data link to an external device, and wherein the degraded performance notification is generated by the external device.

6. The method of claim 1, wherein the operator is a driver of a motor vehicle and wherein the operating handle is a steering wheel of the motor vehicle.

7. The method of claim 6, wherein the wearable device has a data link to an external device.

8. The method of claim 7, further comprising:

sending the degraded performance notification and location data

identifying a. geographical location of the operator to a

remote server;

accessing, by the remote server, map data related to the location

data;

retrieving, by the remote server, location-based information relevant to a remedy of degraded performance; and

sending, by the remote server, a remedial response to the operator

in response to the degraded performance notification,

according to the location-based information.

9. The method of claim 7, wherein the external device is a dedicated GPS navigation device, the method further comprising: providing, by the dedicated GPS navigation device, location-based information relevant to a remedy of degraded performance.

10. A wearable device for a wrist of an operator, for detecting the onset of degraded performance in the operator, the device comprising:

a sensor for determining a grip strength of the operator on an

operating handle; and

a data processor;

wherein the device is arranged to:

establish a grip strength baseline and a reaction time

baseline when the operator initially grips the operating handle;

compare a current grip strength to the grip strength

baseline;

perform a psychomotor vigilance test (PVT), which

includes :

comparing an increase in a PVT reaction time over the reaction time baseline to a reaction time increase threshold; and

if the increase in the PVT reaction time exceeds the reaction time increase threshold, then generating a degraded performance

notification.

1 1. The device of claim 10, wherein the psychomotor vigilance test (PVT) further includes:

comparing the current grip strength to the grip strength baseline;

and

if a decrease of a PVT grip strength from the grip strength baseline exceeds the grip strength decrease threshold, then generating a degraded performance notification.

12. The device of claim 10, wherein the psychomotor vigilance test (PVT) further includes:

comparing the current grip strength to the grip strength baseline;

if the increase in the PVT reaction time exceeds the reaction time

increase threshold, and also

if the decrease of a PVT grip strength from the grip strength baseline exceeds the grip strength decrease threshold, then generating a degraded performance notification.

13. The device of claim 10, wherein the device further comprises a data communication interface arranged to establish a wireless data communication link to an external device.