AU2002301266B2

AU2002301266B2 - Method for detecting, evaluating and analyzing look sequences

Info

Publication number: AU2002301266B2
Application number: AU2002301266A
Authority: AU
Inventors: Ernst Pfleger
Original assignee: Individual
Current assignee: Individual
Priority date: 2001-10-01
Filing date: 2002-10-01
Publication date: 2007-08-30
Anticipated expiration: 2022-10-01
Also published as: ATA15512001A; AT412443B

Description

P/00/011 28/5/91 Regulation 3.2(2)

AUSTRALIA

Patents Act 1990 COMPLETE SPECIFICATION STANDARD PATENT Application Number: Lodged: Invention Title: METHOD FOR DETECTING, EVALUATING AND ANALYZING LOOK

SEQUENCES

The following statement is a full description of this invention, including the best method of performing it known to us Y N I The present invention relates to a method for detecting, evaluating, and analyzing look sequences of a test person using a look detection system, the visual field of the test person being detected by a first camera, which is directed forward and rigidly connected tO the head of the test person, and being recorded in a visual field video, the movement Of the pupil of the test person being detected by a second camera, which is also rigidly connected to the head of the test person, and being recorded in an eye video, and the eye video and the visual field video being recorded on a video system and synchronized in time, the pupil coordinates being established for each frame of the eye Video, the correlation function between pupil coordinates on the eye video and the coordinates of the corresponding visual point, the point upon which the test person fixes, on the visual field video being established, and, after completed establishment of the correlation function for each frame f:rom the pupil coordinates on the eye video, the coordinates of the corresponding visual point on the visual field video being extrapolated.

Methods of this type are known in the related art.

However, in these methods, the pupil coordinates are detected either not at all or very imprecisely and in a very expensive way.

2 The present invention seeks to improve the known systems and provide a method of the type initially described in which reliable recognition of the pupil coordinates for each frame of the eye video may be performed with a low technical outlay.

This is achieved according to the present invention in that, to establish the pupil coordinates for each frame of the eye video automatically using an image recognition program, the contrast of the pupils to the surroundings is registered, all points of the frame which are darker than a set degree of darkness are searched for, using these points, a dark area corresponding to the pupils is completely detected and delimited, and the focus of the dark area, which corresponds to the pupil center having the pupil coordinates, is established.

Through this method, first the entire pupil is detected as a dark area. The registration of the contrast between the dark pupil and the nearly white surroundings of the pupils allows particularly simple automatic detection of the dark area. Establishing the focus of this dark area is performed in a particular simple way and requires only a low outlay for computers, the focus able to be indicated with great precision, however. By equating the pupil center to the focus of the dark area, the pupil center may be established with great precision in a particularly simple way. The present invention is based on the general principle known in the related art of combining an eye video and a visual field video, and, due to the newly developed optical and mathematical methods in the background and the new analysis methods resulting therefrom, represents a significant innovation.

In order to correct errors due to unforeseen interference or incidences of light, according to a further embodiment of the present invention, the pupil coordinates are corrected by hand after the automatic establishment of the pupil coordinates.

A further problem in methods for detecting, evaluating, and analyzing look sequences of the type initially described is the imprecision in establishing the correlation function.

Therefore, the present invention also seeks to improve the known systems and provide a method of the type initially described in which reliable establishment of the correlation function may be performed with a low technical outlay.

This is achieved according to the present invention in that, to establish the correlation function, first, one or more sample look sequences of the test person on one or more specific, predetermined reference points are recorded and the assignment of the pupil coordinates on the eye video to the coordinates of the reference points on the visual field video is established, in that the pupil coordinates in the eye video are established for each frame in the eye video, the coordinates of the reference points in the corresponding frame on the visual field video are established, the pupil coordinates in the frame of the eye video are assigned to the coordinates of the reference point in the corresponding frame of the visual field video, this data set is stored, and the pupil coordinates on the eye video and the coordinates of the corresponding visual point on the visual field video are correlated from all data sets, preferably through quadratic regression.

Through this method, a correlation function is obtained which is based on corresponding coordinates on each frarme of the eye video and of the visual field video. Therefore, an unusually high precision of the correlation function results, the establishment thereof able to be performed in a simple way.

In order to make largely automatic establishment of the correlation function possible, in a further implementation of the present invention, the coordinates of the reference point on each frame of the visual field video are automatically established through an image recognition method.

According to another variant of the present invention, the coordinates of the reference point on each frame of the visual field video may be established by hand using a mouse click method. In this way, these coordinates may be established reliably even in the event of difficult conditions, such as insufficient contrast or undesired reflected glare due to flashes, etc.

In a further embodiment of the present invention, the sample look sequences may be obtained by rotating the head of the test person while he fixes on a single reference point. This represents a particularly simple method of obtaining different positions of the reference point on the visual field video, the definition of a single physical reference point sufficing.

In a further embodiment of the present invention, between and 100 positions of the reference point(s) may be used.

The very high precision of the correlation function desired results in this way, with reasonable computer outlay at the same time.

In a further embodiment of the present invention, a result video is generated from the visual field video and the established coordinates of the Visual point on the visual field video, on which the visual point is displayed by a clearly visible mark, particularly a cross. This allows a particularly graphic representation of the results, which is therefore advantageous for more rapid analysis of the look sequences.

Furthermore, the present invention relates to a method for detecting, evaluating, and analyzing look sequences of a test person using a look detection system, the visual field of the test person being detected by a first camera, which is directed forward and rigidly connected to the head of the test person, and being recorded in a visual field video, for each frame of the visual field video, the coordinates of the corresponding visual point, i.e. the point upon which the test person fixes, being established on the visual field video.

In this method, which is known in the related art, there is often the problem of evaluating the measured values. In particular, the problem results, during recording of all frames on a video system, of further processing and analyzing these significant quantities of data in compressed form.

Therefore, the present invention also seeks to compress the very large quantities of data resulting during detection of the look sequences into a form optimum for analysis and therefore offer a possibility of analyzing the detected look sequences in a simple way. The present invention further seeks to preprocess the highly technical content of the visual field analysis particularly simply and represent it graphically even for lay people without professional qualifications.

This is achieved according to the present invention in that multiple categories are defined and the visual point for each frame is assigned to a category, in that, for each frame, it is checked which category the visual point lies in the region of.

From this, the possibility results of straightforwardly arranging the large quantity of data, which results through the evaluation of the look sequehces, according to categories. Since the pupil coordinates of each frame no longer have to be processed further, but rather only the coordinates of the different defined categories, a drastic reduction of the data volume results 1 In a further embodiment of the present invention, for compressed representation of the time sequence of the association Of the visual point to specific categories of a look sequence a diagram is produced which has two axes, one axis corresponding to a time axis and the other axis corresponding to the individual categories, it being established for each frame in which category the visual point is located and a visual mark being entered in the diagram at the height of the time axis corresponding to the instant of the frame and the height of the other axis corresponding to the category. A diagram results from this which indicates the time sequence of the looks toward in a straightforward way and therefore allows simple and rapid further analysis.

According to another variant of the present inventioh, for compressed representation of the dwell time of the visual point oh Specific categories of a look sequence on a static image for each category, it is established for which frames the visual point lies in a category, the dwell time of the visual point on this category is established by addition of the duration of these frames, and an image is produced which corresponds to the static image, in which a graphic element, which has at least one parameter which is proportional to the dwell time, is positioned in the region of each category.

In this way, a diagram results which straightforwardly indicates the total duration of the looks toward individual categories. The essential information from all frames of the visual field video and all pupil coordinates Of the frames of the eye video is therefore summarized in this diagram, only one single graphic element having to be used per category.

In order to obtain a diagram which straightforwardly indicates the sequence of looks toward individual categories, according to a further embodiment of the present invention, for compressed representation of the sequence of look sequences, a perspective or equalized representation of the scene Observed by the test person is produced and an arrow is assigned to each visual point therein.

Furthermore, the present invention relates to a method for detecting, evaluating, and analyzing the looking behavior of various test persons in a predetermined examination situation.

Due to the lack of exact evaluation of the look sequences of a test person according to the present invention, studies involving quantitative comparison of the looking behavior of multiple test persons in one and the same situation have been unknown until now.

Therefore, the present invention further seeks to allow the simple and exact analysis of the looking behaviour of various test persons in a predetermined and exactly reproducible examination situation.

This is achieved according to the present invention in that the examination situation is recorded and this recording is played back for each test person, their look sequences being detected and evaluated using a look detection system, the visual field of the test person being detected by a first camera, which is directed forward and rigidly connected to the head of the test person, and being recorded in a visual field video, for each frame of the visual field video, the coordinates of the corresponding visual point, i.e. the point on which the test person fixes, being established on the visual field video.

In this way, the looking behaviour of various test persons may be analyzed very simply for the same examination situation. This makes it possible to evaluate, for example, looking behaviour which is a function of age or the physical state of the test persons.

In one aspect, the present invention provides a method for detecting, evaluating, and analyzing look sequences of a test person using a look detection system, the visual field of the test person being detected by a first camera which is directed forward and rigidly connected to the head of the test person, and being recorded in a visual field video, the movement of the pupils of the test person being detected by a second camera which is also rigidly connected to the head of the test person, and being recorded in an eye video, and the eye video and the visual field video being recorded on a video system and synchronized in time, the pupil coordinates (xa,ya) being established for each frame of the eye video, the correlation function between pupil coordinates (xa,ya) on the eye video and the coordinates (xb,yb) of the corresponding visual point the point on which the test person fixes, being established on the visual field video, and, after completed establishment of the correlation function the coordinates (xb,yb) of the corresponding visual point on the visual field video being extrapolated for each frame from the pupil coordinates on the eye video (xa,ya), characterized in that to establish the correlation function, first, one or more sample look sequences of the test person on one or more specific, predetermined reference points are recorded and the assignment of the pupil coordinates (xa,ya) on the eye video to the coordinates (xb,yb) of the reference points on the visual field video is established, in that the pupil coordinates (xa,ya) in the eye video are established for each frame in the eye video, the coordinates (xb,yb) of the reference point in the corresponding frame on the visual field video are established, the pupil coordinates (xa,ya) in the frame of the eye video are assigned to the coordinates (xb,yb) of the reference point in the corresponding frame of the visual field video, this data set is stored, and the pupil coordinates (xa,ya) on the eye video and the coordinates (xb,yb) of the corresponding visual point on the visual field video are correlated :from all data sets, preferably through quadratic regression.

The present invention will be described in more detail with reference to the attached drawing, in which particularly preferred exemplary embodiments are illustrated.

Fig. 1 shows a schematic illustration of the part of the image detection system connected to the head of the test person; N

I

Fig. 2 shows an illustration of the part of the image detection system connected to the head of the test person; which includes glasses 3; Fig. 3 shows a schematic illustration of two frames, one from the eye video and one from the visual field video; Fig; 4 shows an illustration of a video system for recording the two video signals; Fig. 5 shows the illustration of a user interface for establishing the pupil coordinates (xa,ya) according to the present invention; Fig. 6 shows a schematic illustration of two frames, one from the eye Video and one from the visual field video, during the recording of a sample look sequence; Fig. 7 shows the illustration of a user interface for establishing correlation function K according to the present invention; Fig. 8 shows a frame from a result video; Fig. 9 shows a sketch of an image to be analyzed; Fig. 10 shows a diagram according to the present invention for compressed representation of the time sequence of the association of visual point B to specific categories ki of a look sequence; Fig. 11 shows a diagram according to the present invention for compressed representation of the dwell time of visual point B on specific categories ki of a look sequence on the static image sketched in Fig. 9.

The object of look detection systems is to illustrate, with the greatest possible precision, which point of the visual field a test person looks at, upon which exact point the interest and/or the attention of the test person is directed.

In this case, the visual field is detected by a first camera 1, which is directed forward and rigidly connected to the head of the test person. Furthermore, the movement of the pupil of the test person is detected by a second camera 2, which is also rigidly connected to the head.

Rigidly connected in this connection means that both cameras 1, 2 are connected to the head of the test person in such a way that they move with it and/or follow all movements, the freedom of movement of the head and eyes not being restricted in any way, From the evaluation of these two recordings it is possible to indicate with great precision which point the test person looks at at any moment. In this way, exact statements about looks toward, staring, and the absence of looks are possible.

Such look detection systems are preferably used in the safety field, particularly in the field of accident research, but also in the field of advertising or in other investigations of human physiology.

Overall, looking behavior research represents a significant building block for the research of physiological causes of accidents. For example, new information for explaining accidents and reconstructing accidents in regard to human performance limits may be found through comprehensive look investigations.

Typically, for example, particularly dangerous places in street traffic are investigated using look detection systems; A test person equipped with such a look detection system travels through the dangerous location in this case, their looking behavior being recorded. The sum of the looks analyzed in this case will subsequently be referred to as a look sequence; From the analysis of the looking behavior, it is possible to reconstruct Which signposts or signals are not sufficiently observed due to their unfavorable placement, or where particularly obscured and/or little noted locations lie in an intersection.

In the field of work Safety, on construction sites, it may be investigated which sources of danger are detected very late by the test person, and which safety precautions would be necessary in this case.

A further important field of application of look detection Systems is the analysis of advertising posters or advertising spots. Tn this case as well, it is possible to detect very precisely which messages, texts, logos etc.

are fixed on by the test person 1 for how long, and in which sequence.

Fig. 1 Shows a part Of a look detection system for performing the method according to the present invention.

The visual field of the test person is detected by a first camera 1, which is directed forward and rigidly connected to the head of the test person. This first camera 1 therefore provides an approximate image of the look direction of the test person, which is purely defined by the position of the head. First camera 1 may, for example, be implemented by a CCD color camera, which records a large part of the visual field of the test person.

Preferably, first camera 1 and/or also second camera 2 may additionally be controlled using software and thus be adjusted to greatly differing external use conditions.

This ensures that there is no distortion of the pupil image due to the direct recording of the pupil and/or a large image exists due to the direct proximity to the eye and the device may be kept smaller as a whole. Both the Size and the generally poor image quality represented significant causes for imprecision in previous methods. This led not only to difficulties in the weight of the look detection system, but also to general restrictions in the looking behavior of the test person, which may be avoided through the method according to the present invention. Therefore, the look detection system according to the present invention may also be used by test persons wearing different clothes and protective measures, such as a helmetj without restrictions. Different cameras having different lenses may also therefore be used very easily, depending on the requirements of the experiment.

The high-quality cameras which are used in the system according to the present invention are preferably equipped with a control unit which allows white balancing, color balancing, and lighting to be performed automatically.

These values are typically also ManUally adjustable.

Through this control Unit, it is possible to optimally tailor the image quality to the experiment conditions.

Thereforej a very high image quality is ensured for further analysis. Furthermore, there is preferably the option of electronically enlarging an image detail as a digital zoom.

As a rule, other adjustment possibilities only have a limited influence on the image generated.

The movement of the pupil of the test person is detected by a second camera 2, which is also rigidly connected to the head of the test person j and which is pointed toward one of the two eyes of the test person. Second camera 2 may, for example, be implemented by a black and white CCD camera and register the eye movements of the right eye. The detection of the pupil position by second camera 2 is performed directly in the look detection system illustrated in the figures, second camera 2 being pointed directly toward the eye of the test person. The detection of the pupil position may, however, also be performed via optical deflection systems such as mirrors or glass fiber cables 1 using which the image of the eye is deflected to second camera 2.

Both cameras i, 2 are preferably attached to a helmet or glasses 3 (cf. Fig. or a similar carrier device, preferably easy to put on and take off, which is rigidly connected to the head of the test person. Rigidly connected is to be understood to mean, as already explained above, that the carrier device and both cameras 1, 2 follow all movements of the head, the freedom of movement of the head and eyes not, however, being restricted in any way.

The attachment of the camera onto glasses 3, as a carrier device which is easy to put on and take off, allows particularly great mobility of the test person and allows a much greater multiplicity of experiments than in typical systems.

Of course, it is also possible to provide multiple second cameras 2, in order to record both pupils of the test person, for example. Multiple first cameras 1 may also be provided, in order to Gompletely detect the visual field of the test person, if the focal distance of one single first camera 1 is not sufficient for this purpose; In this way, individual look sequences may be detected and, as described in the following, evaluated and analyzed. The term look sequence refers in this case to the sum of the respective looks recorded and analyzed.

Through the two cameras 1, 2, one obtains two video signals, referred to in the following as eye video and visual field video and schematically illustrated in Fig. 3, which are recorded on a video system illustrated in Fig. 4.

The term video system refers in this case to all devices which are suitable for recording film data. The carrier material for recording the video signals is unimportant here. Analog film materials may be used, as well as videotapes or digital storage media such as DVDs or similar things. The storage of single images in the memory of a computer is also considered recording according to the present invention. The format is also unimportant.

Different analog or digital film formats, such as DV or MPEG2, may be used. If CCD cameras are used, both sets of image information are preferably recorded on a digital video system, for example on 2 miniNDV recorders.

In a preferred embodiment, the connection between cameras I, 2 and video system is also implemented for the first time via a radio link. This allows wireless transmission of the video signals to the video system. In this way, the unhindered movement of the test person as a pedestrian, as a bicyclist in open terrain, or even in specific work uses, such as on scaffolds or construction sites, is advantageously made possible.

It is essential that both video signals are synchronized, that for each frame of the eye video, the corresponding frame of the visual field video may be found, and vice versa. The synchronization may be performed using a periodic signal generator and a time code. Preferably, the recording method is synchronized using a pulsed tone, which is also recorded on the respective audio tracks.

This method makes it possible to also simultaneously synchronize other external devices, such as accident data recorders, GPS systems, etc., in order to be able to bring further technical and medical parameters such as the current precise geographical position or even heart rate, skin resistance, etc. of the test person directly into synchronization with the looking behavior. The synchronization is indispensable for the later processing and/or evaluation of the two video signals according to the present invention, In the method according to the present invention, the precise coordinates (xa,ya) of the pupil center in the eye video are established by an image recognition program. In this case, the pupil coordinates (xa,ya) are established for each frame of the eye video. The pupil coordinates (xa,ya) in one frame of the eye video are sketched in Fig.

3.

The pupil coordinates (xa,ya) are preferably established automatically using an image recognition program. For this purpose, the contrast of the pupil to the surroundings is registered for each frame of the eye video and all points of the frame which are darker than a set degree of darkness are searched for. Using these points, a dark area is completely detected and delimited, and subsequently the focus of this dark area is established automatically.

Since the dark area corresponds to the pupil of the test person, the focus of the dark area represents the pupil center.

Preferably, the image recognition program offers adjustment variants for the corresponding contrast and the degree of darkness, so that the highest possible stage of precision may be achieved for all frames. Therefore, for differing illumination ratios, the respective best contrast may be ensured for each frame in the form of a grayscale threshold, which overall makes perfectly reliable determination of the pupil coordinates (xa,ya) possible.

The grayscale threshold is the value which, for example, in digitized form lies between 1 and 256 and defines the percentage proportion of black and/or white on a pixel.

The highest achievable value corresponds to completely black, the lowest value to White. Since the pupil never reaches the entirely black value during recording, a value is to be defined which at least for this image corresponds tO the actually existing pupil gray. The threshold value excludes all pixels which are lighter than the defined grayscale, all darker regions are used for finding the focus. Three parameters allow the threshold definition to be optimized. Since the illumination ratios often change greatly during the experiments within a sequence, this threshold definition is preferably also possible individually for each frame, All adjustments corresponding to the high requirements for each frame of the sequence may therefore be stored in a file. Through the method according to the present invention, the particularly high precision in the assignment of the pupil coordinates (xa,ya) on the visual field is possible. The respective precision stage may be visualized, for example, in the form of a scale. In this case, the quality of the area established is illustrated in the evaluation program using the scale, it primarily being important that a positive value is achieved. Negative values have the consequence that the respective frame is rejected, The higher the positive value 1 the more exactly light regions (surroundings) may be differentiated from dark regions (pupil).

In addition 1 an infrared filter may be provided in front of the camera for more precise localization of the pupil -center. Through this, the contrasts in the eye video are amplified. The IR filter has two important functions: firstly, the eye is illuminated using infrared lightemitting diodes (LED), which ensure good contrasts for the eye camera and for the further processing even in absolute darkness. The filter has the task of transmitting light emitted from the LED to the -camera chip, all other spectral regions of the light are suppressed in accordance with the filter transmission curve. Secondly, the reflections on the pupil caused by sunlight, which have massive negative effects on finding the focus; are primarily in the blue spectral region. In this case as well 1 the filter has the task of reducing the reflections on the pupil which are caused by sunlight.

In a further advantageous embodiment of the method according to the present invention, after the automatic establishment of the pupil coordinates (xaiya), there is also a manual check. A processor may manually change the image processing parameters in the event of an erroneous automatic detection (which may occur, for example in the event of sudden reflected glare on the eye surface, etc.).

It is also possible to correct the pupil coordinates (xa,ya) directly. A user interface for the computer localization of the pupil center according to the present invention and the manual check is shown in Fig. Overall, the pupil coordinates (xa,ya) are obtained for each frame of the eye Video in the form of a Cartesian pair of values, for example. Of course, other coordinate systems; such as polar coordinates, etc., may also be used.

Since both Cameras 1, 2 are rigidly connected to the head of the test person, a specific position of the pupil and/or of the pupil center in the eye Video always corresponds to a precisely defined visual point B in the visual field video. Therefore, it may be calculated precisely which point the test person looks at from the eye video and the visual field video.

For the assignment of the pupil coordinates (xa,ya) to the coordinates (xb,yb) of corresponding visual point B, i.e., the point oh Which the test person fixes, correlation function K between these two coordinate pairs (xa,ya) and (xb~yb) on the visual field video must first be established, The correlation between pupil coordinates (xa,ya) and visual point B on the visual field video is performed via an experimental series; For this purpose, the test person fixes on predetermined reference points P in sequence. Correlation function K between pupil coordinates (xa,ya) and the coordinates (xbjyb) in the visual field Video is produced with reference to the data measured in this case.

In the method according to the present invention, correlation function K between pupil coordinates (xa,ya) on the eye video and the coordinates (xb,yb) of corresponding visual point B On the visual field video is preferably established automatically. For this purpose, first one or more sample look sequences of the test person on one or more specific predetermined reference points P are recorded. Sample look sequence is to be understood as a look sequence which is only recorded for calibration and during which the test person looks at predetermined reference points P.

For example., a specific reference point P may be marked on a wall. In order to obtain the best possible contrast, a black mark on an otherwise white surface may be selected as reference point P, for example. Reference point P is, as a rule, a cross or a point of light or something similar.

The test person is instructed to fix on this reference point P, the visual field and the eye of the test person being recorded by both cameras 1, 2. In this way, multiple reference points P may be defined. It is also possible to mark only one reference point P and instruct the test person to perform different movements with the head as he fixes on this reference point P. Two frames of eyes and visual field videos obtained in this way are sketched in Fig. 6.

Since visual point B on the visual field video of the sample look sequence recorded in this way is given by known reference point P, in a next step, correlation function K between the pupil coordinates (xa,ya) on the eye video and the coordinates (xb,yb) of corresponding visual point B on the visual field video may be established.

For this purpose, the pupil coordinates (xa,ya) in the eye video are established according to the method described above for each frame in the eye video.

Furthermore, the coordinates (xb,yb) of reference point P in the corresponding frame oh the visual field video are established. This i.s preferably performed using an image recognition method, which establishes the coordinates (xb,yb) of reference point P, uniquely identifiable due to its contrast, on the visual field video.

However, it is also possible to determine the coordinates (xb,yb) of reference point P in the visual field video for each frame by hand, for example through a mouse click method. This allows evaluation of the sample look sequence even for difficult conditions in the field, in which automatic determination of the coordinates (xb,yb) of reference point P is not possible, for example due to a background which is too non-uniform.

Therefore, the pupil coordinates (xa,ya) in the frame of the eye video may be assigned to the coordinates (xb,yb) of reference point P in the corresponding frame Of the visual field video.

The corresponding coordinates in the eye video and in the visual field video are established and stored for each frame of the sample look sequence.

The pupil coordinates (xa,ya) on the eye video and the coordinates (xb,yb) of corresponding visual point B on the visual field video are correlated from all of the data sets thus obtained, preferably through quadratic regression. Of course, other methods such as linear regression or stochastic models are also possible for the correlation. A user interface for the establishment of correlation function K according to the present invention is illustrated in Fig. 7.

A correlation function K: (xa,ya) (xb,yb) is obtained, which uniquely assigns a specific set of the pupil coordinates (xa,ya) on the eye Video to the corresponding coordinates (xb,yb) of visual point B in the visual field video; For the greatest possible precision of correlation function K, at least 25 different positions of reference point P are to be used. Above approximately 100 different reference point positions, the precision achieved hardly increases any further, so that further elevation of the number of reference point positions is no ionger advisable.

Therefore, between 25 and oo00 reference point positions are preferably used.

Using correlation function K established in this way, all further video sequences of the same experimental series, in which there are no changes in regard to the camera positions on the head of the test person, may be calculated.

Non-linear connections may be detected through the numeric correlation of the two coordinate pairs. This offers a significant advantage in relation to known systems, in which the two video signals are simply superimposed. The result in these known methods is a result video which shows the visual field of the test person; The pupil of the test person is superimposed over the visual field, so that it may be seen Which point is fixed on by the test person.

These known methods are, however, very imprecise.

Geometric displacements between the position of the pupil in the eye video and the visual point must be compensated using video technology, In the method according to the present invention, these types of geometric factors do not have to be considered. The correct assignment of the pupil coordinates (xa,ya) to visual point B is performed via correlation function K.

After the calibration of the look detection system, individual look sequences may now be detected and analyzed.

After correlation function K is completely established, for each frame, the coordinates (xb,yb) of corresponding visual point B on the visual field video are extrapolated from the pupil coordinates (xa,ya) on the eye video.

The combination of the eye video and the visual field video into a result video is performed using software in such a way that calculated visual points B are positioned as centers of the looks toward on the visual field video.

Through the computer determination of the coordinates (xb,yb) of visual points B according to the present invention, particularly precise representation of the center of the look toward is possible.

Visual point B may be precisely drawn on the result video.

Preferably, visual point B is indicated on the result video by a clearly visible mark, for example by a cross. Fig. 8 shows a frame of a result video generated in such a way.

In a next step, the result video may be stored in its own file, typical video compression methods able to be used.

The method according to the present invention may be performed completely automatically. In this case, checking the pupil coordinates (xa,ya) by hand is not provided.

These coordinates are established directly from the frames of the eye video via the image recognition program.

Furthermore, the coordinates (xb,yb) of reference points P are automatically established during the calibration step.

Therefore, the further storing of the entire eye video described above is no longer absolutely necessary.

In a preferred embodiment of the present invention, the processing software offers the further advantageous possibility Of organizing the visual field into different units, categories ki. The respective image contents may be assigned to freely selectable and definable categories ki.

This division into categories ki is advantageous not only for the method according to the present invention, but also for other methods for detecting, evaluating, and analyzing look sequences, in which the visual field of the test person is detected by a first camera 1 and the coordinates (xb,yb) of corresponding visual point B on the visual field video are established for each frame of the visual field video.

Categories ki kl...kn may be static or moving elements of the visual field. For example, in a flight simulator, one category ki may be represented by a foreign airplane, a further category by an altimeter, etc. In advertising posters, one category ki may be given by a specific logo, another by a specific text, etc. For example, the image and/or sign or advertising poster sketched in Fig. 9 contains a graphic image 10, a logo 11, a title 12, and an explanatory advertising text 13. During the analysis of the look sequences on this static imnage, each of these elements tnay be assigned its o~h category ki. In Fig. 11, graphic iwfage 1-0 is assigned to category k1, logo 11 to category k2, title 12 to category k3, and explanatory advertising text 13 tO category k4.

The assigntnent of the individual visual points to individual categories ki may be performed automatically.

it is essential that Muiltiple Categories ki ate defined and visual point B for each frame is assigned to a category ki, in that for each frame it is checked which category ki visual point B lies ih the region Of. If Visual point B does hot lie in ahy of defined categories kii it may be assigned to an additional null category.

For example, in the case of a static imnage, individual categories ki Miay be defined on the basis of the frequencies-*of the visual points. Therefore, the division into categories ki occurs comtpletely aultomatically. For moving images, as in the case of the flight simnulator cited, the moving objects may be recognized and localized by special image recognition software. Communication between the flight simnulator and the imnage recognition system' is also possible, in which the coordinates (xbiyb) of the ind-iv-idual airplanes, and/or categories ki, may be read in directly.

On the other hand, the assignment of the visual points by hand to individual categories ki for each frame is also possible.

By providing the data in digitized form, the method according to the present invention allows further preprocessing of the data, particularly the assignment of individual visual points B to different categories ki, which greatly simplifies their analysis.

In an advantageous embodiment of the method, for compressed representation of the time sequence of the association of visual point B to a specific Category ki of a look sequence, a diagram is produced which has two axes, one axis corresponding to a time axis and the other axis corresponding to individual categories ki. For this purpose, it is established for each frame which category ki visual point B is located in and a visual mark 20, for example a line, is entered in the diagram at the height of the time axis corresponding to the instant of the frame and the height of the other axis corresponding to category ki.

The linear look band generated in this way, which is shown in Fig. 10, is a form of illustration in which the time, in seconds or frames, is represented on one axis and freely selectable categories ki are listed on the other axis. By entering single visual marks 20 and/or single lines for each frame in accordance with the assignment of visual point B to one of listed categories ki, a bar illustration results, in which each bar 21 indicates the beginning and the ending of a look toward a specific category ki. The temporal sequence of maintaining the look of the test person on specific categories ki is therefore clearly visible over time as a diagram. Of course, other visual marks 20, such as dots, crosses, etc. may also be used.

Of course, it is also possible to enter additional information which goes beyond video evaluations in the linear look band. This includes information which describes the action sequence, such as changes in the street space, but also additional information from other data sources, such as accident data recorders (speed, direction), or, furthermore, other data of human physiology such as the pulse rate or blink rate, the body temperature, etc.

In the linear look band, in accordance with the respective time sections, categories ki may be centrally illustrated by bars 21, which symbolize categories ki in various colors. Summaries may then be made within these single categories ki, which then may be differentiated by color according to their content, such as location differences or differences in content such as left/right or front/back.

The linear look band is, in principle, not limited in its length or in its categories ki. In order to allow a straightforward form of illustration, it is, however, advisable to summarize the contents of sections.

In a further advantageous form of illustration of the procedures during a look sequence on a static image, i.e., on a non-moving image, such as an advertising poster, the dwell times of visual point B on different categories ki may be graphically illustrated. For this purpose, it is established, for each category ki, during which frames visual point B lies in the region of this category ki. The times for which visual point B lies in the region of this category ki are added up, the sum established in this way corresponding to the dwell time of visual point B on this category ki. Subsequently, a graphic and/or an image is produced from the different dwell times, which essentially corresponds to the static image, however, each dwell time on individual categories ki being illustrated by a graphic element 30, which is placed in the region of category ki and has at least one parameter diameter, color, etc. which is proportional to the dwell time. For example, different dwell times may be illustrated by graphic elements 30 having different sizes (for example the circumference of graphic element 30 is larger the longer the dwell time is) or different colors. Fig. 11 shows the diagram obtained from the advertising poster of Fig. 9, in which the dwell times on individual categories ki are indicated by graphic elements 30, which are implemented as clouds of points having different diameters. Furthermore, the delimitations of four defined categories ki to k4 are indicated by dashed lines.

This diagram according to the present invention for compressed representation of the dwell tirne of visual point B on a specific category ki of a look sequence on a static image is therefore distinguished in that a graphic element, which is proportional to the dwell time, is positioned in the region of each category ki. The frequency of maintaining visual point B on a specific category ki is indicated in the diagram according to the present invention by a graphic element 30 proportional to the dwell time, such as a differently sized point.

Through this diagram, the frequencies and the time overhead analysis may be visualized particular clearly. The diagram according to the present invention allows high-quality and straightforward analysis of the frequencies the looks toward.

In a further form of illustration, the visual field sketch, the look sequences are listed in the sequence in which they appear in the result video. In this case, perspective illustrations or equalized illustrations of the scenes observed by the test person may be used, such as profile or outline sketches. For profile sketches, the look sequences may be drawn up graphically in a screen shot in photos drawn up in parallel, or also in sketches. For outline sketches, layout plans, stylized plans, or overview illustrations may be used for concrete contents. In the visual field sketch, each visual point, and/or each fixation, is assigned an arrow, independent of the fixation length. The fixation lengths and/or thematic connections may be described in more detail by colors, or also through additional information. Numberings and/or explanatory text are possible as additional information. Temporal connections and different color scales may also be displayed as a supplement, times may be bundled and identified in a concrete color having corresponding legends.

The method according to the present invention is particularly suitable for the analysis of the looking behavior of various test persons in a personal, predetermined examination situation. For this purpose, the examination situation may be recorded and analyzed. The recording of the examination situation may be performed, for example, using a simple video camera, this recording able to be played back to the test persons, via a video projector.

Claims

1. A method for detecting, evaluating, and analyzing look sequences of a test person using a look detection system, the visual field of the test person being detected by a first camera which is directed forward and rigidly connected to the head of the test person, and being recorded in a visual field video, the movement of the pupils of the test person being detected by a second camera which is also rigidly connected to the head of the test person, and being recorded in an eye video, and the eye video and the visual field video being recorded on a video system and synchronized in time, the pupil coordinates (xa,ya) being established for each frame of the eye video, the correlation function between pupil coordinates (xa,ya) on the eye video and the coordinates (xb,yb) of the corresponding visual point the point on which the test person fixes, being established on the visual field video, and, after completed establishment of the correlation function the coordinates (xb,yb) of the corresponding visual point on the visual field video being extrapolated for each frame from the pupil coordinates on the eye video (xa,ya), characterized in that to establish the correlation function, first, one or more sample look sequences of the test person on one or more specific, predetermined reference points are recorded and the assignment of the pupil coordinates (xa,ya) on the eye video to the coordinates (xb,yb) of the reference points on the visual field video is established, in that the pupil coordinates (xa,ya) in the eye video are established for each frame in the eye video, the coordinates (xb,yb) of the reference point in the corresponding frame on the visual field video are established, the pupil coordinates (xa,ya) in the frame of the eye video are assigned to the coordinates (xb,yb) of the reference point in the corresponding frame of the visual field video, this data set is stored, and the pupil coordinates (xa,ya) on the eye video and the coordinates (xb,yb) of the corresponding visual point on the visual field video are correlated :from all data sets, preferably through quadratic regression.

2. The method according to Claim 1, characterized in that to establish the pupil coordinates (xa,ya) for each frame of the eye video automatically using an image recognition program, the contrast of the pupils to the surroundings is registered, all points of the frame which are darker than a set degree of darkness are searched for, using these points, a dark area corresponding to the pupils is completely detected and delimited, and the focus of the dark area, which corresponds to the pupil center ha.ving the pupil coordinates (xa,ya), js established.

3. The method according to Claim 2, characterized in that, after the automatic establishment of the pupil coordinates (xa,ya), the pupil coordinates (xa,ya) are corrected by hand.

4. The method according to Claim 1, characterized in that the coordinates (xb,yb) of the reference point on each frame of the visual field video are automatically established by an image recognition method. The method according to Claim 1, characterized in that the coordinates (xb,yb) of the reference point on each frame of the visual field video are established by hand using a mouse click method.

6. The method according to one of Claims 1 to 5, characterized in that the sample look sequences are obtained by rotating the head of the test person while he fixes on an individual reference point 34

7. The method according to one of Claims 1 to 6, characterized in that between 25 and 100 positions of the reference point(s) are used.

8. The method according to one of the preceding claims, characterized in that a result video is generated from the visual field video and the established coordinates (xb,yb) of the visual point on the visual field video, On which the visual point is indicated by a clearly visible mark, particularly by a cross.

9. The method according to one of Claims 1 to 8, characterized in that multiple categories (ki) are defined and the visual point for each single image is assigned to a category in that it is checked for each frame which category (ki) the visual point(B) lies in the region of The method according to Claim 9, characterized in that, for compressed representation of the temporal sequence of the association of the visual point (B) to specific categories (ki) of a look sequence, a diagram is produced which has two axes, one axis corresponding to the time axis and the other axis corresponding to the individual categories it being established for each frame in which category (ki) visual point is located and a visual mark for example a line, being entered in the diagram at the height of the time axis corresponding to the instant of the frame and the height of the other axis corresponding to the category (ki).

11. The method according to Claim 9 or 10, characterized in that, for compressed representation of the dwell time of the visual point on specific categories (ki) of a look sequence on a static image, it is established for each category for which frames the visual point (B) lies in a category (ki), the dwell time of the visual point on this category (ki) is established by addition of the duration of these frames, and an image is produced, which corresponds to the static image, in which a graphic element which has at least one parameter proportional to the dwell time, is positioned in the region of each category (ki).

12. The method according to Claim 9, 10, or 11, characterized in that, for compressed representation of the sequence of look sequences, a perspective or equalized representation of the scene observed by the test person is produced and each visual point is assigned an arrow therein.

13. The method according to one of Claims 1 to 12, characterized in that the examination situation is recorded, and this recording is played back for each test person, their look sequences being detected and evaluated using a look detection system, the visual field of the test person being detected by a first camera which is directed forward and rigidly connected to the head of the test person, and being recorded in a visual field video, for each frame of the visual field video, the coordinates (xb,yb) of the corresponding visual point i.e. the point on which the test person fixes, being established on the visual field video.

14. A method for detecting, evaluating and analysing look sequences of a test person as hereinabove described with reference to the accompanying drawings. UNIV. PROF. DIPL.-ING DR ERNST PFLEGER WATERMARK PATENT TRADE MARK ATTORNEYS P21946AU00