METHOD AND SYSTEM FOR PRESENTING REAL TIME PHYSIOLOGICAL INFORMATION THROUGH A HEAD MOUNTED
DISPLAY
Reference to Related Applications This application claims priority from U.S. Provisional Application No.
60/331,314 of the same title filed November 14, 2001, the entire disclosure of which is incorporated by reference in its entirety.
Rights in the Invention
This invention was made, in part, with support from the United States
Government. Accordingly, the United States Government may have certain rights to this invention.
Background
1. Field of the Invention The present invention relates in general to information delivery and presentation using a head mounted display. Specifically, the present invention provides methods and systems for presenting and delivering physiological information through a head mounted display in a medical and/or clinical setting.
2. Description of the Background Viewing monitors is an important way for clinicians to collect information about patients at the sites of clinical procedures, such as operating rooms, emergency rooms, intensive care units, ambulances, and other patient care areas. A familiar scene of a hospital operating room, for example, is a roomful of medical instruments and display monitors surrounding teams of clinicians. To correctly perform most procedures, clinicians must look at their hands, the patient, and clinical monitors displaying how the patient is doing. Often in these situations,
relevant data of patient physiological states are presented on one display that falls outside of the field of view of the clinicians who are performing the procedure. As such, significant logistical and ergonomic challenges are posted for the doctors to timely acquire the critical visual information as they concentrate on carrying out the procedure on hand - a challenge that directly bears on the success of the procedure and the safety of the patient.
Similarly, but on a larger scale, medical practitioners in a hospital or any medical and/or clinical facility - particularly outside of the treatment areas - are without visual access to relevant physiological mformation of a patient in real time unless they go to the particular devices or displays that present such information - often at the patient's bed site or in a central monitoring station. Therefore, the lack of efficiency in relaying information and the less-than-timely discharge of critical information to the responsible practitioners hinder the overall efficiency of the delivery of health services, as, for example, valuable time is expended for each practitioner to acquire necessary information and to convey the same to his or her peers when evaluating a medical situation and preparing for taking part in a procedure.
There is thus a need to unify and bring relevant information directly to a doctor or a medical practitioner as he or she performs an operation or activity or is to perform or evaluate such an operation or activity. Particularly, such information, while visual, may be overlaid into the field of vision of the doctor or practitioner.
Head mounted displays have been used to present visual information in various research and application areas, including telecommunication, aviation, and defense. See, e.g., Fiorino F. Aviation Week & Space Technology, August 13, 2001 , p.52; Soldiers, Dec. 2000, cover page; IEEE Spectrum, Oct. 2000, cover page; Weiss P., Science News, 1999, 156:330; Wilson A., Vision Systems Design, March, 2000, p.9. Many specific features and advantages of different kinds of head mounted displays have been described to date. See, e.g., US Patent No. HI, 790; US
Patent No. 6,037,914; US Patent No. 5,949,388; US Patent No. 4,961,626; US Patent No. 6, 101 ,431. However, little work has been done to utilize head mounted displays in facilitating medical and clinical operations or activities, e.g., for real time presentation and delivery of physiological information.
Summary of the Invention
The present invention overcomes the problems and disadvantages associated with current strategies and designs and provides methods and systems for presenting and delivering physiological information using a head mounted display thereby facilitating medical operations and/or clinical activities. The delivery and presentation of information as such into the eyes of the users (doctors and the like) improve the logistics, ergonomics, and efficiency of the operations or activities, for example, by releasing the users of the burden of reaching out of their way to different devices and displays to acquire needed information while performing an operation or activity or preparing for performing the same.
One embodiment of the invention is directed to methods for presenting physiological information generated from one or more devices through a head mounted display, comprising: transferring the physiological mformation to a receiver which connects to the head mounted display; and presenting through the head mounted display the physiological information. The receiver is wearable together with the head mounted display in a particular embodiment. The transferring may be on wire (with any form of tethered data cable) or wireless in various embodiments of the invention.
Another embodiment of the invention is directed to a method for delivering physiological information generated from one or more devices to a multiplicity of loci, comprising: broadcasting the physiological information to a plurality of receivers at the multiplicity of loci, each of which connects to a head mounted display and is wearable together therewith; and presenting through the head mounted display the physiological information. In a particular embodiment, the broadcasting
is in multiple channels and each of the plurality of receivers tunes into one of the multiple channels.
Another embodiment of the invention is directed to a system for presenting physiological mformation generated from one or more devices through a head mounted display, comprising: a head mounted display capable of presenting the physiological information; and a receiver capable of receiving the physiological information, wherein the receiver connects to the head mounted display and is wearable together therewith. According to certain embodiments, the receiver receives the physiological information on wire; according to other embodiments, the receiver receives the physiological information by wireless means.
Another embodiment of the invention is directed to a system for delivering physiological information generated from one or more devices to a multiplicity of loci, comprising: a first plurality of head mounted displays, each capable of presenting the physiological information; and a second plurality of receivers at the multiplicity of loci, each capable of receiving the physiological information, wherein each of the second plurality connects to one of the first plurality and is wearable together therewith and wherein the physiological information is broadcast to the second plurality and presented through the first plurality. In a particular embodiment, the system further comprises a third plurality of channels each carrying a selection of the physiological information, wherein each of the second plurality tunes into one of the third plurality.
Preferably, physiological information is information selected from the group consisting of electrocardiogram, ST segment analysis, non-invasive blood pressure, arterial blood pressure, pulmonary artery blood pressure, pulmonary wedge pressure, central venous pressure, intracranial pressure, cerebral perfusion pressure, cardiac output, temperature, oxygen saturation of the arterial blood, oxygen saturation of the venous blood, oxygen saturation of the mixed venous blood, end tidal carbon dioxide, fraction inspired/fraction expired of inhaled anesthetic agents plus oxygen,
carbon dioxide, nitrous oxide, helium, carbon monoxide, and nitrogen, ventilatory parameters, electroencephalogram, somatoscensory evoked potentials, brain stem auditory evoked potentials, visual evoked potentials, and bispectral spectroscopy. Other types of physiological information may be used in other embodiments of this invention.
Preferably, devices may be selected from the group of imaging devices consisting of MRI, X-Ray radiography, cine radiography, camera, CAT scan, doppler flow, ultrasound, and flexible and rigid scopes. Presenting may be performed in a numerical, textual, or graphical format, or any combination thereof. According to still another embodiment, the presenting is multi-dimensional. According to a further embodiment, the presenting is monochrome. Preferably, presenting may be in full color.
Presenting may also be performed in a substantially simultaneous manner as the physiological information is generated. Preferably the head-mounted display is monocular or binocular, and the head mounted display may be transparent or opaque.
Other features and advantages of the invention will become apparent from the following description. It should be understood, however, that the description and the examples are given by way of illustration, and various changes and modifications within the spirit and scope of the invention will become apparent to those skilled in the art from this description.
Description of the Drawings
Fig. 1 is a visual image presented via a head mounted display that has patient vital signs and physiological information overlaid into a clinician's field of vision during a procedure of insertion and placement of a pulmonary artery catheter in the patient.
Description of the Invention
The following terms are to be understood consistently with their typical meanings established in the relevant art, i.e. the art of biology, physiology, medicine, anesthesiology, and any areas related thereto.
EKG (or ECG), as used herein, refers to electrocardiogram. It reflects the electrical activity generated by the heart muscle. The EKG information is typically represented as a two dimensional (time and magnitude) graphed line and several additional sealers or numerals.
ST segment analysis refers to ST segment elevation or depression analysis displayed as positive or negative numeric value representing changes in the ST segment of the EKG.
Non-Invasive Blood Pressure (NIBP) refers to the blood pressure of the systemic arterial circulation; it is measured with a cuff. This information is typically represented as a set of three sealers or numerals, e.g., 125/75 (95) mm Hg.
Arterial Blood Pressure (ABP) refers to blood pressure of the systemic arterial circulation; it is measured by puncturing the skin and placing a sensor in direct contact with the arterial blood. The arterial blood pressure information is typically represented as a two dimensional (time and magnitude) graphed line and a set of three sealers or numerals.
Pulmonary Artery Blood Pressure (PABP) refers to blood pressure in the pulmonary (lung) artery. It is measured by inserting and directing a catheter through the right side of the heart and then on into the pulmonary artery. The pulmonary artery blood pressure information is typically represented as a two dimensional (time and magnitude) graphed line and a set of three sealers or numerals.
Pulmonary Wedge Blood Pressure (PAWP or PCWP) refers to blood pressure in the pulmonary circulation when the flow has been stopped by occluding (wedging) the lumen of the artery by inflating a balloon at the tip of the catheter.
The pulmonary wedge pressure information is typically represented as a two dimensional (time and magnitude) graphed line and a sealer or numeral.
Central Venous Blood Pressure (CVP) refers to blood pressure in the central veins. The central venous pressure information is typically represented as a two dimensional (time and magnitude) graphed line and/or a sealer or numeral.
Intracranial Blood Pressure (ICP) refers to pressure in the skull. The intracranial pressure information is tjφically represented as a two dimensional (time and magnitude) graphed line and a sealer or numeral.
Cerebral Perfusion Pressure (CPP) is typically calculated from other measured values, and represented as a two dimensional (time and magnitude) graphed line.
Cardiac Output (CO) refers to the amount of blood flow pumped through the heart. This cardiac output information is typically represented first as a two dimensional (time and magnitude) graphed line, and then converted into a sealer or numeral.
Temperature (Tl , T2, Tx), as used herein, may be measured with various probes in a number of locations, including, for example, at skin surface, oral, esophageal, rectal, and on every cardiac output catheter. The temperature information is represented as a sealer or numeral.
SaO2 (or SpO2) refers to oxygen saturation of the arterial blood. The SaO2 information is represented as a two dimensional (time and magnitude) graphed line and as a sealer or numeral.
SvO2 refers to oxygen saturation of mixed venous blood; the measurement and information representation are similar to those of SaO2, respectively.
ETCO2 refers to end tidal carbon dioxide, that is, the amount of C02 expired. The ETCO2 mformation is typically represented as a two dimensional (time and magnitude) graphed line and as a sealer or numeral.
FI/FE gases refer to fraction inspired/fraction expired of inhaled anesthetic agents plus oxygen, carbon dioxide, helium, carbon monoxide, and nitrogen; it is measured continuously from gas flowing into and out of patient. This information is typically represented as a two dimensional (time and magnitude) graphed line and as a sealer or numeral.
Ventilatory parameters, as used herein, refer to control setting parameters for a mechanical ventilator (e.g., rate, tidal volume, airway pressures, inspiratory/expiratory duration ratios) and parameters monitored during spontaneous ventilation (e.g., rate, tidal volume, airway pressures). The ventilatory parameters are represented as sealers or numerals.
EEG refers to electroencephalogram, that is, brain waves. The EEG information is typically represented as two dimensional (time and magnitude) graphed lines.
SSEP refers to somatoscensory evoked potentials. The SSEP information is typically represented as a two dimensional (time and magnitude) graphed line and a sealer or numeral.
BAEP refers to brain stem auditory evoked potentials. It measures auditory nerve conduction. The BAEP information is typically represented as a two dimensional (time and magnitude) graphed line and a sealer or numeral.
VEP refers to visual evoked potentials. It measures visual nerve conduction. The VEP information appears as a two dimensional (time and magnitude) graphed line and a sealer or numeral.
BIS values refer to bispectral spectroscopy measurements; they reflect the level of consciousness. The information are represented as a two dimensional (time and magnitude) graphed line and a sealer or numeral.
The term "head mounted display" (HMD), as used herein, encompasses any device which provides a visual display or direct retinal projection and which may be worn on the head. A head mounted display can be made from various materials such as plastic, glasses, metal, and alloy, among other things. It may be opaque or transparent; and accordingly, a user may see through the HMD for the surroundings, if it is transparent or, if it is opaque, the user may only see a display presented from a designated mformation or data source that connects to the HMD. A HMD may connect to a designated information or data source through wired network - hence tethered HMD - or through wireless means - hence wireless HMD. Any form of tethered data cable may be used according to various embodiments, including, e.g., electrically conducting wires and optical fibers. And, any form of wireless or remote transfer means may be used in this context, including, e.g., infrared (IR), radio frequency (RF). Any HMD as described here, whether homemade or commercially available, may be used in the methods and systems of the present invention.
The term "physiological information," as used herein, refers to any information that relates to the physiological well-being of a patient or a person of interest, which may be measured or monitored by an instrument or device in a medical or clinical setting and visually displayed for use by clinicians. The physiological information may be in a textual, numerical, or graphical format, or any combination thereof. Typically, therefore, the values of physiological parameters of the patient is physiological information. Examples of physiological information according to the present invention include, among other things, electrocardiogram, ST segment analysis, non-invasive blood pressure, arterial blood pressure, pulmonary artery blood pressure, pulmonary wedge pressure, central venous pressure, intracranial pressure, cerebral perfusion pressure, cardiac output,
temperature, oxygen saturation of the arterial blood, oxygen saturation of the arterial/venous blood, end tidal carbon dioxide, fraction inspired/fraction expired of inhaled anesthetic agents plus oxygen, carbon dioxide, nitrous oxide, helium, carbon monoxide, and nitrogen, ventilatory parameters, electroencephalogram, somatoscensory evoked potentials, brain stem auditory evoked potentials, visual evoked potentials, and bispectral spectroscopy.
Head Mounted Display Used for Presentation and Delivery of the Physiological Information
The head mounted display of the present invention may include any kind of HMD that is sold commercially by various vendors (e.g., Kaiser Electro-Optics, Inc., Microvision Inc., etc.) or made in a noncommercial setting. The HMD may be monocular or binocular according to various embodiment of the invention. There is at least one eye piece, preferably capable of being displayed to either eye.
In certain embodiments, the HMD is transparent; the user may look through the HMD lenses or eye pieces, which have displayed thereon information of the user's choosing from designated data sources, and see normal vision of the surroundings. The displayed images are thus overlaid or superimposed into the normal field of vision of the user. Images may be of contrasting colors or hues to the immediate field of vision for obtaining desired resolution, or simply bright enough to be visible against illuminated surrounding areas. Alternatively, the displayed images may be segregated to a portion of the user's field of vision. For example, the user could focus on the particular area of interest in the field of vision, which may be placed at one or a pluraity of different focal points such that to focus on one point would place the other points out of focus. The transparent HMDs may cover the field directly in front of the eyes, as well as that just out and above the eyes to allow adequate view when looking up. In other embodiments, the HMD is opaque; the user may only view the information or images of his or her choosing,
which are displayed on the eye pieces from a predetermined information or data source. Transparent HMDs may be used as opaque HMDs if desired.
In a particular embodiment, the HMD is a direct retinal projector which have displayed thereon information of the user's choosing from designated data sources, and see normal vision of the surroundings; the images are preferably bright enough to be visible against illuminating surrounding areas. There is at least one eye piece, preferably capable of being displayed into either eye.
The information or data sources may be any device or instrument and combinations thereof used in medical and clinical settings that measure the physiological state of a patient and that is capable of transferring information visually. In certain embodiments, a HMD may connect to such a designated information or data source through wired network; it is therefore tethered. In other embodiments, a HMD may connect to the designated information or data source through wireless means. Tethered HMDs are less likely to interfere with other wireless communications in the surroundings compared to wireless HMDs; they may be used in emergency rooms, operating rooms, or at intensive care units, post anesthesia care units, procedure rooms, and any other areas where patients are monitored. Wireless HMDs are more preferably used by roving users, those move around and need real-time visual access to relevant physiological information. These wireless HMDs are especially applicable in a multiple loci, multiple users, and data broadcasting situation which is discussed in detail infra. Any form of wireless or remote transfer means may be used according to the present invention. For example, signals may be transferred via radio frequency (RF), e.g., Bluetooth, or infrared (IR).
The HMD of this invention is wearable by the user and is preferably light weighted. The HMD connects to a receiver for receiving data and information from the aforementioned sources. The receiver may be a video input, for example, which is capable of receiving video signals generated from computer video cards. A
receiver may be part of the HMD or it may be a small peripheral of the HMD and is wearable together with the HMD. The HMD of this invention may receive and display video signals from any regular image generator without regard to the format of the signals - whether interlaced NTSC composite, interlaced NTSC component (e.g., S-Video, or RGB, etc.), progressive, or digital video.
The physiological information presented and delivered by a HMD according to the present invention include textual or numerical data as well as live or static images. The images may be multidimensional; two-dimensional in certain embodiments and three-dimensional in other embodiments. For example, real-time images from doppler flow, ultrasound, flexible and rigid scopes, and cine radiography, and a video camera and static video signals showing images from X- Ray, MRI scan, Cat scan, and a camera may be delivered and displayed. In certain embodiments, an HMD of the invention is monochrome; and in other embodiments, the HMD is capable of displaying images and/or data in full color. Typically, the displayed color matches the color of the clinical monitors from which the HMDs receives the data and images.
In a preferred embodiment the physiological information is displayed real time in a manner that provides an instantaneous record of information for a given preceding time period. In practice this embodiment is best carried out by horizontal or vertical scrolling of plotted graphical data. In representative embodiments, a continuously updated data set from the previous minute, 10 minutes, 30 minutes or hour are presented. Other time periods can be set and presented as desired. By displaying a constant section of most recent history with respect to a physiological variable this embodiment allows a medical practitioner to watch for and quickly respond to trends in a patient's condition. In some situations this new style of presenting a patient's condition in the HMD may be critical to the patient's prognosis. Previous use of heads up displays in other fields such as avionics do not address sufficiently this presentation format, which is particularly preferred for practice of embodiments of the invention.
In another embodiment, an HMD user controls the presentation of scrolled physiological data by voice command. Preferably, voice recognition software is located in a computer that controls the transfer of physiological information to the HMD. Most preferably, voice commands to that computer can halt data presentation (stop scrolling) to allow more time to see details at a specific time point, and also to go back and display older data. Yet another command in this context can change the time scale of data that is presented in real time format, to allow viewing of physiological changes that occur over shorter or longer time intervals. This control of real time graphical physiological data presentation is critical in a medical situation and has not been appreciated in the prior art. Even more important and unappreciated is the ability of the HMD user to intervene by voice command and directly cause medical treatment changes that influence the very parameters that are presented in real time. Thus, the practitioner can perceive, not just fixed data but physiological trends and intervene in those trends, all by voice command.
In certain embodiments, the HMDs of this invention is monochrome; and in other embodiments, the HMDs are capable of displaying images and/or data in full color. Typically, the displayed color matches the color of the clinical monitors from which the HMDs receives the data and images.
Monitoring Patients' Vital Signs and Guiding Medical Procedures Head mounted displays may be used according to this invention to present physiological information to a user, e.g., a clinician or supervisory clinician. The HMDs are especially useful for monitoring patients' vital signs and physiological information. For example, it is known that patients suffer abrupt changes in their cardio-pulmonary functions during an intubation procedure. A clinician wears a HMD that receives and presents all the vital signs and physiological information of the patient will be able to closely observe the patient's vital signs without periodically detracting from performing the procedure in order to evaluate the physiology state of the patient.
Another example is insertion of a pulmonary catheter. This is a procedure whereby a doctor insert a catheter into and through the heart and on into the pulmonary artery. It requires continuous and real-time knowledge of the pressure sensed at the tip of the catheter. Wearing a HMD that receives and displays the real- time blood pressure mformation, the doctor will be able to see his or her hands manipulating the catheter while having the patient's blood pressure readings continuously in view. A detailed discussion is included infra in Example 1.
Therefore, the HMDs of the present invention are particularly useful in guiding medical procedures. Similar to pulmonary catheter insertion, successful insertion of a bronchoscope also requires continuous knowledge of the location and direction of the tip of the bronchoscope. Wearing a HMD that receives and displays the image from the tip of the bronchoscope will allow the doctor to conveniently keep his or her hands as well as the tip of the bronchoscope in view at all times during the procedure. To guide invasive medical procedures, not only real-time patient-specific monitoring data may be used as discussed above, other general or historical medical and anatomic data or images may also be used in a similar manner. For example, textbook anatomic images and physiological data and images in the patient's medical history may be transferred to the HMD and overlaid into its display field as needed, to assist the doctor as he or she navigates through the patient's body to perform a specific procedure.
Receiving and Presenting Multi-Channel Broadcast of the Physiological Information
Head mounted displays of the present invention are useful not only within treatment areas such as operating rooms and emergency rooms as discussed supra, but also outside of treatment areas, particularly in a larger clinical and medical setting where a multitude of patient physiological data are relayed among many clinicians.
For example, in a hospital, clinic, or any other medical treatment facility, every clinician may have available a HMD which he or she can wear when needs to access relevant visual information of a given patient whose physiological condition is being monitored. Multiple patients are monitored and operated upon at the same time in various treatment areas in the facility. The physiological information gathered as such may be managed by a centralized information system, which assigns a unique information channel - e.g., a radio frequency - to the physiological data from each patient. The system thus broadcasts in real time such data in multiple channels throughout the hospital facility. A clinician wearing a HMD may selectively tune into a desired channel and receive physiological information regarding a particular patient. Therefore, the use of HMDs in such a system enables practitioners to conveniently gain access to real-time visual information of patient physiological states while on the move. It improves the efficiency and timeliness of the information relay in the medical treatment facility and helps clinicians to evaluate patient conditions and prepare for participating in treatment activities at a distance. Also, users of the system may switch from channel to channel and observe the operations and activities throughout the facility. The system thus provides a means for effective overall monitoring and surveillance, useful especially for those who bear the responsibility to oversee a good number of patients or operations at the same time.
In another embodiment, HMDs of the present invention may provide feedback control to the user. Feedback control means that the user is able to act on information received through the HMD. For example, for embodiments whereby the display provides respiratory information of a patient receiving respiratory assistance during a surgical procedure, and the user believes that the patient should have increased respiration, through an interactive means, the user may signal that increased respiration assistance be immediately provided. The interactive means may be provided through body movement, hand or finger movement, head or eye movement, or another means known in the art, that permits feedback control to the desired mechanism. Preferably, the interactive means is voice control whereby the
user speaks into a microphone causing the apparatus to change or alter the particular physiological parameter of the patient (e.g. respiratory assistance, amount of anesthesia or any drug being administered, cardiac rhythm control). The apparatus may also provide for confirmation that the requested change is being implemented through a visible indication back through the HMD or an audible signal. Such an embodiment allows the user complete and, more importantly, immediate control of the procedure and over the patient.
The following examples are offered to illustrate embodiments of the present invention, but should not be viewed as limiting the scope of the invention.
Example 1: Insertion and Placement of a Pulmonary Artery Catheter
The insertion and placement of a pulmonary artery catheter was performed on a patient. The clinician who performed the procedure wore a transparent head mounted display during the procedure, through which patient vital signs and physiological information were presented. Fig. 1 depicts what the clinician saw from looking through the transparent HMD. The continuous PA pressure trace (102) was used to guide the insertion. The clinician may simultaneously see the guiding PA pressure trace and his hands while inserting the catheter (106). There was no tactile information in the catheter itself to be felt by the fingers that can guide insertion.
At the moment when the image of Fig. 1 was captured, the catheter has just reached its correct, final position. The clinician knew this because the PA trace just changed from the large pulsatile wave form (104) down to the much flatter wave form seen across the rest of the trace (108), and the PAP was reported as 7 mm Hg (110). While the clinician held the catheter from moving with his right hand, he begins to deflate the balloon at the end of the catheter with his left hand.
In summary, the HMD provided the clinician with the following information: the status of the alarms: ALARMS SUSPENDED (112); EKG lead II wave form and Heart Rate value: 70 bpm (114); EKG lead V wave form and changes in ST1 and ST2 (116); systemic arterial blood pressure (ABP) wave form and systolic, diastolic, and mean values: 130/75 (97) mm Hg (118); pulmonary artery pressure (PAP) wave form and current value: 7 mm Hg (102, 104, 108, 110); pulse Ox, or oxygen saturation (SpO2) wave forms and value: 97 % (120); central venous pressure (CVP) value: 3 mm Hg (122); end tidal carbon dioxide (ETCO2) wave form: 0 mm Hg, sensor not connected to patient (124); airway respiratory rate (AWRR) value: 0 bpm, determined from ETCO2 trace, thus zero (126); and NIBP vale: not measured, thus no value was displayed (128).
It is to be understood that the description, specific examples and data, while indicating exemplary embodiments, are given by way of illustration and are not intended to limit the present invention. All references cited herein for any reason, including all U.S. Patents, are specifically and entirely incorporated by reference. Various changes and modifications within the present invention will become apparent to the skilled artisan from the discussion, disclosure and data contained herein, and thus are considered part of the invention.