WO2014024159A1

WO2014024159A1 - Interactive eye-control multimedia system for active and passive tracking

Info

Publication number: WO2014024159A1
Application number: PCT/IB2013/056488
Authority: WO
Inventors: Paolo Invernizzi; Gianluca Dal Lago
Original assignee: Sr Labs S.R.L.
Priority date: 2012-08-08
Filing date: 2013-08-08
Publication date: 2014-02-13
Also published as: ITFI20120165A1

Abstract

The present invention relates to an interactive multimedia apparatus and related method for providing both able-bodied users and disabled users with access to a set of complex information in a convenient and immediate manner, by using at least one eye-tracker device and a suitable user interface. The present invention further relates to an object and information display rack of the interactive, multimedia type.

Description

INTERACTIVE EYE-CONTROL MULTIMEDIA SYSTEM FOR ACTIVE AND PASSIVE TRACKING

Field of the invention

The present invention relates to the technical field of interactive apparatuses provided with eye-tracker devices.

Background art

The progress of information technology has multiplied the modes by which a user may access an information content, whether it is of the advertising type or of public utility. Moreover, the success of technologies related to the control of electronic devices and apparatuses solely by means of gaze paved the way for a very broad range of possibilities aimed at proving a user with access to information in places and through modes which not so long ago were unforeseen.

Moreover, we know that the so-called weak categories, comprising disabled and elderly people, represent an increasingly larger group of citizens whose needs have to be contemplated within the overall strategy aimed at guaranteeing an access to information which may be as much widespread as possible.

Since the above-mentioned technologies and the products implementing them are normally oriented to that group of people which falls within the definition of "able- bodied", it is necessary to intervene in order to make their use possible also for those physically challenged and with disabilities, even in view of the fact that technology has to be aimed at improving the quality of life and not at creating new barriers and boundaries.

Therefore, there is a need for the implementation of new technologies and the design of new devices capable of eliminating the above-mentioned problems, also allowing disabled people to find a set of even complex information in a convenient and quick manner.

The goal is thus to increase the usability of user interfaces of the devices and apparatuses through which said information is accessed, where usability means the extent to which the functions of the interface are adapted to the physical and mental needs of the user which, through it, wants to find information. Usability improves when learning the various functions of the user interface and their later use is an extremely simple and intuitive process, so as to be adapted to the mental status and to the different needs of each user who wants or needs to use it.

Therefore, a first object of the present invention is an interactive multimedia apparatus adapted to interact with a user by means of the analysis of said user's gaze, manage electromechanical actuators and display multimedia contents in response to the processing of the analysis of said user's gaze.

A second object of the present invention is an object and information display rack of the interactive multimedia type adapted to provide both able-bodied users and disabled users with access to a set of complex information in a convenient and immediate manner, by means of at least one eye-tracker device and a suitable user interface.

Brief description of the drawings

Fig. 1 shows a functional block diagram of the apparatus according to the present invention.

Fig. 2 shows an example of an object and information display rack according to the present invention.

Summary of the invention

The present invention relates to an interactive multimedia apparatus and related method for providing both able-bodied users and disabled users with access to a set of complex information in a convenient and immediate manner, by means of at least one eye-tracker device and a suitable user interface. The present invention further relates to an object and information display rack of the interactive multimedia type.

Detailed description of the invention

Given the above-described technical problem, an object of the present invention is thus the development of an apparatus adapted to allow a user, both able-bodied or disabled, to access a set of complex information in a convenient, effective and quick manner and without requiring the use of hands, but only the use of gaze, by means of eye-tracker devices.

The present invention is such to particularly improve accessibility of information apparatuses to disabled people which, for various problems, cannot use their hands. Such an improvement is also achieved by means of the provision of a graphic interface properly studied to be used through eye control. The present invention further allows to overcome, due to its high, simplicity, and straightforwardness of use, the wariness of technology which is present in some groups of people and especially among elderly people, simplifying their access to multimedia information contents from which they are usually barred.

The present invention has another advantageous application in the field of the so- called public interfaces, with particular reference to all those interactive information and advertising systems which are placed in public environments and which may be used, by means of the integration of the apparatus according to the invention therein, without requiring physical contact with their user. Thereby, a greater protection of both the apparatus and the user may be ensured with the elimination of problems normally related to wear and cleaning of the public interface, as well as to the hygiene of users.

In brief, the main results obtained by the introduction of the apparatus according to the present invention are as follows:

- Simplified access to information retrieval both for able-bodied and disabled people.

- Reduction of the time needed to perform the desired action.

- Reduction of the cognitive resources needed for the overall use of the system, which means greater ease of use.

- Elimination of physical contact between the equipment and the user, which implies a better preservation of efficiency and functionality of the equipment and elimination of any hygienic problems for the user.

The uses of the apparatus according to the present invention may be, inter alia, as follows:

- Display of virtual catalogs comprising graphic, audio and video contents, which may be browsed through and looked up by a user through the sole use of his/her gaze, with said virtual catalogs being aimed at illustrating the features of products and services being adapted to guiding the novice user to the correct use of the apparatus. In particular, in a preferred embodiment of the present invention, we find that:

o The user, once the product of interest has been selected by means of his/her gaze, may explore it by simply and naturally changing the orientation and magnification thereof and display its inner parts, o The user, once the product of interest has been selected by means of his/her gaze, may receive information about products related to the selected one, so as to promote cross-selling and marketing activities for the advertiser.

Display, on a suitable screen, of information related to the object present on the display rack the user is looking at. To this end, the eye-tracker device of the apparatus according to the present invention may, for example, be integrated into a window equipped with a screen where information related to the object the user is looking at, at that time, are passed. Therefore, the apparatus according to the present invention may be advantageously integrated into a display rack, an example of which is shown in the accompanying Figure 2, to detect which of the items displayed a potential customer observes, at a certain moment. Said display rack, for example, may be organized so that the displayed items are geometrically arranged around a monitor of which only the screen is visible. When the user observes a product, the stand on which the observed product is placed is lit; at the same time, on the monitor a video content is shown, which illustrates its operation or explains further features thereof by mentioning technical specifications, usage instructions, prices, etc. Display of installations and constructions of the "reactive" type, which is capable of providing interaction modes between people and environments. For example, in the field of interactive art, an image displayed on a high-definition display "disguised" as a painting may suddenly become animated when we look at it at certain precise spots.

Determination of the features of an image or a graphic interface which mostly attract the user's attention and thus promote the use thereof. This allows developing simple and effective-to-use products or services with a relevant competitive advantage. In particular, the following functions may be implemented:

o Monitoring and analysis of every aspect of the user/graphic interface interaction.

o Monitoring and analysis of the visual impact of a product or a communicative message through the consumer_'.s„eyes_(packaging_T video commercial, newsletter, printed media, etc.).

o Monitoring and analysis of the perception and exploring of an environment (supermarket aisle shelves, windows, etc.). With reference to the accompanying Figures 1 and 2, the apparatus according to the present invention comprises: a control module 10 - in turn comprising at least one digital signal microprocessor or processor - associated with means for detecting the presence of a user, with at least one eye-tracker device 11 adapted to detect the activity of said user's gaze, with image displaying means, and with actuating means preferably comprising drivers 16 for electric motors 17 and drivers 18 for lighting sources such as LED, filament or fluorescent lamps, said actuating means being adapted to act in response to the activity of said user's gaze detected by said at least one eye-tracker device 11.

Said control module 10 may comprise a personal computer provided with suitable hardware and software interfaces for managing and controlling said at least one eye-tracker device 11 , said user presence detecting means, said image-displaying means and said actuating means.

Said user presence detecting means preferably comprise sensitive pads 12, webcams 13 and infrared or ultrasound presence sensors 14 used individually or in combination.

Sensitive pads of prior art which may be used in the present invention are for example electro-pressure-sensitive devices for detecting the presence of people. One or more sensors adapted to detect a pressure, associated with a control module having a verification function, may be used. The operation principle of this kind of pads essentially consists in producing the change of status of an electrical signal when a certain pressure is exerted on the surface of the pad sensor. The presence of one or more people or part of the weight of one of these causes a contact placed inside the sensor to close; the closing of the circuit depending on said contact is processed by the control device, which in turn provides for the emission of a certain command.

Said image displaying means preferably comprise a liquid-crystal display (LCD) 15 optionally of the touch screen and multimedia type. The apparatus according to the present invention is adapted, to. detect -the presence of a possible user in the vicinity and determine where said possible user is directing his/her gaze and therefore his/her attention. Once the presence of a user has been detected and information about the direction of his/her gaze and thus the object of his/her attention have been collected, the apparatus according to the present invention processes said information so as to determine a sequence of following actions which, for example, may be aimed at: displaying, on said image displaying means, information, messages, audio/video multimedia contents or interactive interfaces for the user; actuating said actuating means to move movable planes or walls and/or orientating and switching on lighting means such as LED, filament or fluorescent lamps or yet still may be aimed at printing information or payment receipts on paper.

Thereby, the apparatus according to the present invention may be advantageously integrated in structures such as, for example, information and interactive public interfaces, display racks, business windows, ATMs and similar structures where making interaction with users more effective and efficient is required and desirable.

With reference to the accompanying Figure 2, the apparatus according to the present invention may be advantageously integrated in an object and information display rack, comprising: a structure 30 adapted to contain and display objects and comprising, in turn, pedestals and movable walls adapted to accommodate said objects; said actuating means being associated with suitable lighting means adapted to light said objects and with moving means associated with said pedestals and movable walls.

The apparatus according to the present invention is adapted to operate according to the following sequence of steps.

First, the possible presence of a user in the vicinity of the apparatus is detected by processing, through said control module 10, signals sent by said user presence detecting means, preferably comprising a sensitive pad 12, at least one infrared or ultrasound presence sensor 14 and/or a webcam 3, and then switching on the eye tracker 11 associated with the position of the user.

Once the presence of a user has been detected, said user may be recognized by processing the data collected by said eye tracker 11 and/or by said webcam 13, which processing operation is adapted to determine and store age, sex, facial expressions, etc., for example, in a convenient database. This operation may be carried out by using one of the several object recognition techniques aimed at recognizing the features of people available in prior art.

Said control module 10, which processes the data from the eye tracker 11 , then detects and stores the user's gaze, by carrying out a calibration thereof, and optionally repeating the operation if the user loses, even momentarily, his/her correct positioning. The calibration performed is stored in the data about the user in order to complete his/her profile.

Upon the recognition of the user, a check of the parameters detected is also performed in order to determine if this is a new user, otherwise the corresponding profile, already stored, is loaded without carrying out a new calibration.

Thereafter, the data about the user's gaze continuously captured by the eye- tracker device are processed.

Two different operating modes may occur, managed by said control module 10 and corresponding to two different operating modes of the eye-tracker device 11 : the first one sees the eye-tracker device to operate actively; the second one sees the eye-tracker device to operate passively.

In the first case (active mode), the interaction between the user and the apparatus according to the present invention is, in fact, of the active type and the user, by simply using his/her gaze, may give commands to the above-mentioned apparatus such as, for example, the command to act on said actuating means (e.g. in order to light a displayed object or to print information material), the command to display information on the apparatus display, etc.

This occurs by displaying, on said displaying means, information of suitable "interactors", i.e. the interactive portions - icon images or others - which may be actuated by the user through his/her own gaze, thus originating other corresponding actions.

Said interactors will have dimensions to be compatible with the type of calibration of the user's gaze previously made or recalled for said user.

If the calibration does not take place, the eye-tracker device is still able to distinguish the position of the user's gaze in a grid, wi.tb.in._the. space of the- interactive monitor, having dimensions of 3 x 3 or 4 x 4 at utmost. With the calibration, the eye-tracker device improves its performance of user's gaze position assessment, which may therefore be distinguished on finer grids, and, consequently, said interactors may have smaller dimensions and thus a lesser number of them may be displayed in the user interface on the interactive monitor. Hence, the apparatus according to the present invention takes advantage of said data about the user's gaze, continuously captured by said eye-tracker device, to perform the described functions. This makes it possible to achieve much higher accuracy and precision levels and to manage and distinguish a greater number of actions as compared to similar systems simply operating in response to the detection of user's presence or head or face position.

In detail, once a minimum accuracy angle has been established, defined as the smallest angle detectable between the directions of two consecutive gazes of the user, the interface interactors need to be spaced apart by at least such an angle to have a high probability that these are correctly distinguished by the control module of the apparatus according to the present invention. The better (or finer) the calibration performed by the apparatus according to the present invention, the lower the above-mentioned minimum accuracy angle and therefore the user interface shown on the monitor may have closer interactive items.

Therefore, in the active mode, said control module 10 displays, on said image displaying means, a specific set of interactors depending on the type of calibration performed or loaded and adapted to allow the user to give commands to the apparatus according to the present invention by simply using his/her own gaze, such as, for example, the command to act on said actuating means (e.g. in order to light a displayed object or to print information material), the command to display information on the apparatus display, etc.

In the second case (passive mode), the interaction between the user and the apparatus according to the present invention is of the passive type since the user's gaze is detected by the eye-tracker device without providing any type of feedback and these data are stored for later processing in order to carry out specific statistical assessments. In further detail, in the above-mentioned first case (active mode),, said control module 10 displays a suitable user interface on said image displaying means and modifies it as a consequence of the interaction of the user's gaze therewith, produces multimedia contents, yet still by said image displaying means, or actuates said actuating means in order to move movable planes or walls and/or orientate and switch on lighting means.

Claims

1. An interactive multimedia apparatus adapted to interact with a user by means of the analysis of said user's gaze, and to manage electromechanical actuators and display multimedia contents in response to the processing of the analysis of said user's gaze, comprising a control module (10) - in turn comprising at least one digital signal microprocessor or processor - associated with means for detecting the presence of a user, with at least one eye-tracker device (11 ) adapted to detect the activity of said user's gaze, with image displaying means, and with actuating means adapted to act in response to the activity of said user's gaze detected by said at least one eye-tracker device (11).

2. An apparatus according to claim 1 , wherein said means for detecting the presence of a user are selected from the group comprising sensitive pads (12), webcams (13) and infrared or ultrasound presence sensors (14), and are used individually or in combination.

3. An apparatus according to claims 1 - 2, wherein said control module 10 comprises a personal computer provided with suitable hardware and software interfaces for managing and controlling said at least one eye- tracker device (11 ), said means for detecting the presence of a user, said image displaying means and said actuating means.

4. An apparatus according to claims 1 - 3, wherein said image displaying means comprise liquid crystal displays (LCD) (15) of the touch screen and multimedia type.

5. An apparatus according to claims 1 - 4, wherein said actuating means are selected from the group comprising drivers for electric motors, drivers for lighting sources such as LEDs, filament or fluorescent lights, drivers for printers.

6. A method for interacting with a user by means of the analysis of said user's gaze, for managing actuating means and displaying audio/video multimedia contents in response to the processing of the analysis of said user's gaze, comprising the following steps: determining the possible presence of a user through the processing, by a suitable control module (10), of the signals received from suitable means for detecting the presence of. a user; detecting and recording the user's gaze detected by an eye-tracker device (11); processing the data related to the user's gaze collected by the eye- tracker device ( 1 ), said processing operation being adapted to use data from the eye-tracking device to provide the user with feedbacks, actuate electromechanical actuators or display multimedia contents on suitable displaying means.

7. A method according to claim 6, wherein said feedback provided to the user comprises displaying, on said displaying means, a plurality of interactors adapted to be actuated by the user through his/her gaze, thus originating other corresponding actions.

8. A method according to claims 6 - 7, wherein said processing of the data related to the user's gaze is further adapted to collect data from said eye- tracker device (11), without providing the user with any feedback, in a format suitable for further processing operations in order to carry out statistical assessments.

9. A method according to claims 6 - 8, wherein said determination of the possible presence of a user takes place through the processing of signals from devices selected from the group comprising sensitive pads (12), webcams (13) and infrared or ultrasound presence sensors (14), used individually or in combination.

10. A method according to claims 6 - 9, wherein the detection and recording of the user's gaze takes place through the following steps: switching on the eye-tracker device; recognizing the user through the processing of the image captured by said webcam (13); calibrating the user's gaze and storing the calibration parameters or loading a previously acquired calibration if the user is already known.

11. A method according to claims 7 - 10, wherein said interactors have dimensions to be compatible with the type of calibration of the user's gaze which has previously been made or recalled for said user, said interactors being reciprocally spaced apart, on said displaying means, by an angle greater than or equal to the smallest angle detectable between the directions of two consecutive gazes of the user.

12. An object and information display rack comprising: a structure adapted to contain and display objects and comprising, in turn, lighting means adapted to light said objects; an apparatus according to claims 1 - 5, wherein said actuating means are associated with said lighting means.

13. A display rack according to claim 12, further comprising pedestals and movable walls adapted to accommodate said objects, said actuating means being further associated with said pedestals and movable walls.

14. A display rack according to claims 12 - 13, further comprising printing means adapted to print information, and wherein said actuating means further comprise drivers for printers associated with said printing means.