EP4264402A1

EP4264402A1 - Methods, techniques and hardware and software tools for monitoring the activity of users at a video display terminal (vdt)

Info

Publication number: EP4264402A1
Application number: EP21840132.1A
Authority: EP
Inventors: Vittorio Ferrari; Marco Ferrari; Daniele Marioli; Pietro APOSTOLI; Simone DALOLA
Original assignee: Universita degli Studi di Brescia
Current assignee: Universita degli Studi di Brescia
Priority date: 2020-12-16
Filing date: 2021-12-15
Publication date: 2023-10-25
Also published as: IT202000030989A1; WO2022130236A1

Abstract

A system (S) formonitoring the activity of users (U) at a video display terminal (VDT),to determine the actual and active usage time (T) of thevideo display terminal (VDT) by each specific user (U_i) and to record the work sessions of the user (Ui) with their durations (T_partial), wherein the actual and active usage time (T) of the video display terminal (VDT) by a user (U_i) is determined by the combined analysis of the time traces related to the activities of attentive observation (100) of the monitor (10) by the user (U_i) and the active interaction (102,104,106) of the user (U_i) with the video display terminal (VDT)by means of the input peripherals (20). The system (S) comprises at least one optical sensor (50) configured to identify the user (U_i) who is using the video display terminal (VDT) and to detect their attentive observation of the monitor (10), at least one interface unit (60,60a) configured to detect the usage actions of the input peripherals (20), without recording any information input by the user (U_i), at least one hardware processing unit (70,70a) configured to process the information and data related to the active usage (102,104,106) of the input peripherals (20) and the attentive observation (100) of the monitor (10), and configured to provide the partial usage time of attentive observation (100) of the monitor (10) and active interaction (102, 104, 106) of each individual peripheral (20) and to combine the time traces which represent the sequence of partial activity intervals, i.e. referred to the single interaction with the VDT (attentive observation of the monitor or usage of a peripheral), to calculate the partial and/or combined usage times (T), and a set of software tools (80,80a) configured to detect and identify the user (U_i) at the video display terminal (VDT) using image processing techniques, intercepting the attentive observation (100) of the monitor (10) by the user (U_i), determining the active interaction (102,104,106) of the input peripherals (20), identifying the usage time of the video display terminal (VDT) by each user (U_i), recognizing and classifying the user (U_i), and determining and storing the usage sessions of the video display terminal (VDT). The combined analysis of the attentive observation (100) of the monitor (10) by the user (U_i) and the active interaction (102,104,106) of the user (U_i) with the video display terminal (VDT) through the input peripherals (20) comprises various combinations of the time traces which represent the sequence of the partial activity intervals, i.e. referring to the single interaction with the VDT (attentive observation of the monitor or the usage of the single peripheral) in which the combinations comprise operations such as AND, OR or combinations thereof.

Description

METHODS, TECHNIQUES AND HARDWARE AND SOFTWARE TOOLS FOR MONITORING THE ACTIVITY OF USERS AT A VIDEO DISPLAY TERMINAL (VDT)

FIELD OF THE INVENTION

The present invention relates to tools and methods for monitoring the activity of a user at a video display terminal (VDT).

In particular, the invention relates to tools and methods for non-invasively monitoring the activity of a user of a video display terminal (VDT), capable of quantitatively and objectively determining the active usage time of the VDT by the user according to specific parameters and specially developed metrics.

TECHNICAL PRIOR ART

Traditionally, the usage time of the VDT is estimated through questionnaires administered to the user, or through third-party surveillance and observation activities. The results of these forms of time determination may be inaccurate, not very repeatable, and poorly objective, thus overall unsatisfactory, as illustrated in A. Cristofolini, W. Versini, M. Tommasini, S. Forti, L. Barozzi, R. Mutinelli, “Valutazione dei tempi di utilizzo del videoterminale”, Giornale Italiano di Medicina del Lavoro ed Ergonomia (2008), 30 (4), pp. 340-344 (hereafter cited as Cristofolini).

The Guidelines of the Italian Society of Occupational Medicine and Industrial Hygiene (SIMLII) call for "VDT usage time to be assessed as objectively as possible" using equipment with "a rational objective approach," as described in the Cristofolini document.

Several tools are known in the prior art which are designed to monitor the performance of activities at the VDT by users using different hardware, software, or hybrid apparatuses, described in various publications and online sites, which will be presented below. However, each of the systems currently available in the prior art considers only partial user interaction with the VDT (typically either usage of the input peripherals or observation of the monitor) to determine the usage time of the VDT or the degree of user fatigue. Moreover, such known systems do not automatically take a global and objective measurement of the active usage time of the VDT by the user. Alternatively, there are apparatuses that, considering the degree of the sedentary lifestyle of the person determined in various manners, indirectly correlate such a sedentary lifestyle to the VDT usage.

The systems known in the prior art to monitor VDT usage have disadvantages and functional drawbacks. Firstly, they can be invasive and interfere with or even prevent normal VDT work. Secondly, in general, some known systems are manually activated; said systems only allow user monitoring following a specific enabling by the user without allowing continuous and automatic monitoring of the operator at the VDT.

Among the known solutions, it is possible to detect user activity and track the usage of the various applications installed on the VDT using software tools residing on the VDT itself to be monitored, as described in https https://whatpulse.org/, https://getklok.com/, and https://www.spyarsenal.com/familykeyloqger/.

Other solutions involve the use of hardware monitoring devices (sniffers), to be interposed between the VDT and the peripherals capable of intercepting keyboard keystrokes and/or movements of the mouse as described, e.g., in

Such tools, which are useful in work productivity measurement contexts, may be detrimental to user privacy.

The main limitation of such software tools is that they only allow determining the start-up and opening condition of single applications (e.g. browser, word processor, etc.), but cannot detect the actual presence of the user with the respective attentive observation of the monitor and active involvement in a work session.

Monitoring systems residing on the VDT to be monitored itself, capable of detecting the presence of a user only by controlling input devices (mouse, keyboard, etc.) have been suggested in literature. In particular, the publication S. Morelli, M. Grigioni, M. Ferrarin, A. Boschetto, M. Brocco, G. Maccioni, D. Giansanti, “A monitoring tool of workers’ activity at Video Display Terminals for investigating VDT- related risk of musculoskeletal disorder”, Computer Methods and Programs in Biomedicine, (2012), 107 (2), pp. 294-307 (hereafter cited as Morelli) presents a comparative study of the evaluation of VDT usage time carried out by the measurement obtained through a specially made software apparatus compared with the results of self-administered questionnaires. The software application installed on the computer of the user whose exposure to the VDT is to be measured monitors the user and determines the usage time of the VDT considering only the usage of the input peripherals (mouse and keyboard). These measurement tools do not consider the user's visual interaction which occurs when observing the monitor.

A different approach to solving the problem object of the present invention is represented by systems capable of making quantitative determinations of the VDT usage by measuring physical or biomedical parameters (e.g., electromyographic (EMG) recordings, electro-goniometers, motion analysis systems) related to the activity performed by the user are described in the literature, e.g., in D. Giansanti, M. Ferrarin, G. Maccioni, M. Grigioni, F. Draicchio, P. Dionisio, M. Piantoni, S. Morelli, “Rischio neuromuscolare degli operatori a videoterminale: progetto e applicazione di uno strumento innovativo per il monitoraggio dell’attivita”, Rapporti ISTISAN 11/21 , Istituto Superiore di Sanita (hereafter cited as Giansanti) and in document Y. Luo, N. Zhao, Y. Shen, “A wearable multi-modal human performance monitoring system for video display terminal users: Concept, development and clinical data validation”, 2016 IEEE International Conference on Real-time Computing and Robotics, RCAR, (2016), 7784019, pp. 163-168 (hereafter cited as Luo). Such tools are generally unsatisfactory because they are invasive for the user and mostly either prevent or limit the full and free activity at the VDT, bringing an unacceptable disruption of the phenomenon to be quantified. Furthermore, this method of determining the time of activity at the VDT requires analysis and interpretation of the measured data to be postponed relative to when the monitoring information was acquired. Such tools are more suited to analyzing the physical manner in which the VDT is used, the ergonomic aspects, and any repercussions on the health of the operator, rather than meeting the need for a synthetic quantification of the time of attentive and active usage of the VDT.

In particular, a system capable of detecting the actions performed by the user at the VDT is presented in the Morelli and Giansanti papers. The system consists of a software apparatus installed on the VDT to be monitored intended to detect keyboard and mouse interaction activities and a unit based on wearable devices which integrate accelerometers to monitor the motor activity of the user's upper limbs, as in Giansanti, to detect the actions of interfacing with the VDT. The system detects and records all mouse and keyboard activities, comprising the entered characters, the movements, and the mouse clicks. Due to the presence of devices to monitor the user's motor activity, such a system may be invasive and obstruct the user's normal activities of VDT usage. As a further limitation, the described system focuses on the analysis of the user's movements and the usage of the input peripherals, without placing the right emphasis on the quantification of the time for which the user observes the monitor, which may thus cause visual fatigue. Such a system lends itself more to diagnostic analysis related to the usage of the input peripherals and the posture of the upper limbs than to quantifying the work time at the VDT. In many application areas, such as call centers, in which the activity at the VDT is particularly intense and persistent, continuous and automatic monitoring of the user's activities carried at the VDT may be particularly useful, as described in the Morelli paper.

The Lou paper illustrates a laboratory system for assessing in real time the degree of fatigue of users at the VDT by monitoring physiological parameters (EEG, ECG, EMG/EOG, BIA, and PPG) detected by means of wearable devices. The results show that the system can monitor the degree of user fatigue at the VDT and provide feedback to prevent fatigue-related syndromes. Such a system is useful for quantifying the degree of fatigue produced in the user but it does not measure the active usage time of the VDT by the user; therefore, it cannot be used to determine the time of work at the VDT and to classify users based on such a time.

In the paper R. R. Fletcher, D. Chamberlain, D. Richman, N. Oreskovic, E. Taveras, “Wearable Sensor and Algorithm for Automated Measurement of Screen Time”, 2016 IEEE Wireless, WH, (2016); 7764564, pp. 109-116 (hereafter cited as Fletcher) a method for automatic detection of monitor observation time is suggested which uses optical sensors worn by the user and analyzes the wavelength of light emitted by the monitors and detected by the sensor. Furthermore, the user's movements can be monitored and identified through an accelerometer worn by the user. Besides the fact that the user is required to wear specific devices, the main limitation of the method is that the principle based on measuring the wavelength of the light emitted by the monitors only makes it possible to determine whether the user is near the monitor itself but not to discriminate whether the user is actively observing the monitor with a sufficient degree of attention.

Document EP 0 992 970 A1 "Apparatus and method for monitoring VDT operation, for preventing physical problems of the operator" describes a software apparatus and a method for monitoring operations performed at the VDT. The user must activate/stop the monitoring operation of the work sessions carried out at the VDT by means of the software apparatus. The system determines the VDT operating time based on the activation/interruption actions performed by the user and checks if the VDT usage time is greater than a set threshold. In such a case, the user is asked to stop working to take a break. In this system, there is a mechanism for determining the work time at the VDT solely based on the use of software timer-type applications without considering information from the use of input interfaces, let alone user observation of the monitor. The system acts as a user-activated time counter which thus makes it possible to derive the total duration of work performed and breaks to be taken.

Document JP2003070768 "System for detecting posture of VDT worker" presents a system which can detect the correct posture of a VDT worker (operator eye-monitor distance) and the user's action of observing the monitor. The system consists of an optical sensor (camera) placed on the monitor and an image processing unit capable of detecting the direction of the user's gaze. If the user's correct posture is continuously detected, a time counter is incremented and a warning is triggered if this interval exceeds a predetermined threshold. This device is only intended to signal the correct posture of the user and is not intended to calculate the active usage time of the VDT.

Document US 2008/0136650 A1 "System and method for ergonomic tracking for individual physical exertion" suggests a system and method for identifying the posture maintained over time of a user using a VDT to prevent possible disabling injuries which occur due to ergonomically incorrect posture. The system and method include an environmental module, in a possible embodiment comprising an image sensor and image processing unit to identify the physical environment in which the user operates, a biomechanical module to determine the user's posture, and an output module to indicate the correctness of the assumed posture to the user.

Document JP2001290632A "Method, device and system for monitoring VDT work and readable recording medium recording VDT work monitor program" suggests a system for monitoring VDT work consisting of a centralized apparatus connected with a plurality of computers and portable devices. The system makes it possible to group the information related to the use of the various devices in graphs and summary images. The suggested method and system appear to be more oriented towards the detection of the mere VDT usage rather than the actual quantification of the duration of work sessions.

Document JP2005267491 A "VDT work environment control device" relates to an ergonomic workstation for improving the conditions of activity at the VDT for the user by monitoring the user's physical condition and the workload to which the user is subjected. A plurality of sensors are used for this purpose, including an electromyographic sensor attached to the mouse, a chair pressure sensor to detect the user's posture, and sensors to measure typing speed on the keyboard. In particular, by means of electromyographic sensors, it is possible to determine the user's fatigue and, by means of image sensors, detect the presence and posture of the user and make fatigue assessments based on the blink rate analysis. This apparatus is oriented toward analyzing the user's fatigue state rather than determining the usage time of the VDT.

Document JPH05101243A, "VDC using time controller" describes an apparatus for the management of VDT usage in time slots by identification of the specific user through a password. The user is either allowed or not allowed use of the VDT during given time slots according to the associated usage authorization. The invention is more oriented towards managing a multi-user workstation and authorization of usage rather than measuring the time of active and attentive VDT usage by the user. US 2007/0250622 A1 "Alerts for monitoring user status" suggests a method for monitoring a user of a VDT, connected to a communication network, by another user. Such a method requires the monitored user to provide an enable signal so that the monitoring user can perform monitoring operations. The system provides automatic monitoring operations and notification of the activities performed by the first user once the second user has enabled monitoring. The implementation involves an analysis of the activities performed by the monitored user through appropriate software. Possible extensions of the invention provide for hardware input devices, such as webcams and microphones which can act as user monitoring events, indicating the start of specific activities performed at the VDT. The present invention does not permit continuous, automatic monitoring of the user in the absence of user enabling. Furthermore, attentive observation of the monitor by the user is not considered and detected.

US 7,421 ,413 B2 "Delicate metering of computer usage" presents a software apparatus for managing computers and VDTs shared among multiple users. The apparatus makes it possible to measure the usage of a computer shared with other users by a single user and provides for the enabling of the usage of the computer by the user who has usage rights. In particular, the system takes care of updating the share of time/resources used by the user and, if the user has purchased a greater share of PC usage than the time of usage they have already used, allows the user to use it. The activities performed at the VDT can be detected by monitoring different computer peripherals, such as mice, keyboards, printers, to determine the usage of the terminal and the associated peripherals and compare the usage rate of such peripherals with the available usage permissions. Such an invention does not aim to determine the time of attentive and active usage of the VDT, but only the mere and not better-specified activity in a context of shared usage of computer equipment according to the usage permissions granted to the various users.

Instead, US 2009/0006613 A1 "System and a method for updating a user presence status on a terminal by aggregating multisource information" describes a software method for detecting the physical presence of a user using a terminal connected to a telecommunications network, based on presence status information provided by at least one peripheral of the terminal. The method involves monitoring the status of the peripheral devices, such as webcams, mice, and keyboards. The patent further includes the ability to detect the physical presence of a user using sensors customarily used in home automation, such as volumetric presence sensors, infrared sensors, webcams, or a surveillance camera. Such sensors only determine the physical presence of a user and not the attentive observation of the monitor. The invention focuses on the detection of the presence of the user at the terminal using the computer operating system and aims to obtain information regarding the user's activity level determined by events due to keyboard usage, mouse movement, or physical movements detected by an inertial unit embedded in the terminal. The patent only aims to detect the physical presence and possible movements of the user of the VDT without distinguishing between the mere physical presence and the attentive and active presence, i.e. the interaction of the user with peripherals, such as mouse and keyboard, in addition to the attentive observation of the monitor.

Document US 2012/031 1447 A1 "Collecting, aggregating and presenting activity data" suggests a system capable of collecting and cataloging information related to the duration and type of activity generated by the usage of one or more computing devices. The system allows data to be displayed on user interfaces in different forms and aggregated by type of activity and/or user. The innovation is aimed at monitoring the productivity of various users and cataloging the activities they perform. The system can recognize the beginning of the activity at the terminal by analyzing the state of the peripherals or through devices capable of recognizing the use of voice through microphones, or gestural movements made by the user and detected through cameras, distance sensors, infrared sensors, and other devices, without evaluating the active and attentive interaction of the user with the VDT.

Document US 2014/0085077 A1 "Sedentary activity management method and apparatus using data from a data-capable band for managing health and wellness" describes an apparatus consisting of a hand-held terminal and wearable devices, such as smart bracelets, capable of detecting an individual's sedentary or non- sedentary activities to promote proper lifestyles and movement. The innovation presented is in the field of portable devices to monitor physical activity, sports, and recreation and consequently provide stimuli to the individual to devote adequate time to non-sedentary activities.

US 2016/0210839 A1 "Multimodal monitoring systems for physical activity" describes a system comprising a VDT, a hand-held telephone device and a data aggregator capable of monitoring a user's physical activity. The disclosed method is aimed at calculating the time the user performs sedentary activities. The system involves the use of a webcam solely to determine the user's sedentary posture. The data aggregator provided by the system may reside on the VDT, on the portable telephone device, on a server, or the communication network. Such an aggregator determines the sequences of activities carried out by the user and their duration and correlates the data relative to the physical activity carried out by the user, providing useful information about the degree of well-being linked to physical activity to promote correct lifestyles, without determining the time of attentive and active usage of the VDT.

Finally, US 9,713,444 B2 "Human-digital media interaction tracking" describes a software system for tracking interaction between a human and digital devices, which operates by storing information about a user's activity history. The software apparatus is responsible for providing information related to user behavior and cognitive activities performed. In particular, the system tracks the interaction between the user and the digital device by detecting its speed and duration. The apparatus includes a database which stores information about a user's activity history and a computing device which receives information about the user's behavior which indicates the usage by the user of an interactive software program inherent in specific cognitive activities. No reference is made to the detection of attentive and active usage of a VDT.

SUMMARY OF THE INVENTION

The present invention relates to solutions for non-invasively monitoring the activity of a user of a video display terminal (VDT), capable of quantitatively and objectively determining the active usage time of the VDT by the user according to specific parameters and specially developed metrics.

The usage time of the VDT is determined by the combined analysis of the user's attentive observation of the monitor, detected by image sensors, and the user's active interaction with the VDT through the input peripherals (keyboard, mouse, touchpad, joystick, graphics tablet, etc.), i.e. considering the complex of interactions that the user makes with the VDT.

The solution suggested here relates to a system for monitoring the activity of users at a video display terminal to determine the actual and active usage time of the video display terminal by each specific user and to record the user's work sessions with their durations, wherein the actual and active usage time of the video display terminal by a user is determined by the combined analysis of attentive observation of the monitor by the user and the active interaction of the user with the video display terminal by means of the input peripherals. The system comprises at least one optical sensor configured to identify the user who is using the video display terminal and to detect their attentive observation of the monitor, at least one interface unit configured to detect the usage actions of the input peripherals, without recording any user input information, at least one hardware processing unit configured to process the information and data related to the active usage of the peripherals and attentive observation of the monitor, and configured to provide the independent attentive observation time of the monitor and usage of each single peripheral and to combine information to calculate the usage time of the video display terminal. The system further comprises a set of software tools configured to detect and identify the user at the video display terminal using image processing techniques, intercept the user's attentive observation of the monitor, determine the usage of input peripherals, identify the time of video display terminal usage by each user, recognize and classify the user, and determine and store the usage sessions of the video display terminal. In various embodiments, the optical sensor and software tools are configured to implement facial recognition to distinguish different users who may share the same video display terminal at different times, to determine the corresponding specific usage time of the video display terminal for each user, and the system comprises a database of multiple facial images of the various users acquired in the initial stage and updated over time, to correlate the user who is using the video display terminal at a particular instant with the images in the database and recognize the user's identity.

Preferably, the interface unit is interposed between the input peripherals and the display screen.

Each input device is provided with hardware or software interface units by which hardware or software events related to the usage of the device are generated, respectively, and the processing unit is configured to determine the usage time of the video display terminal based on the hardware and software events that are generated by the interface units when the user uses the input devices or attentively observes at the monitor. The processing unit is configured to determine the usage time of a single peripheral or the combined usage time of the video display terminal. The software interface units connected to the peripherals are configured to generate software events, and the peripherals are motion sensors, optical or ultrasonic distance sensors, 3D scanners, optical triangulators, cameras in the visible, and infrared cameras.

In some embodiments, the system is self-contained and independent from the video display terminal.

In alternative embodiments, the system is either partially or fully integrated with the video display terminal.

The invention further relates to a corresponding method.

BRIEF DESCRIPTION OF THE DRAWINGS

Further features and advantages of the invention will be apparent from the reading of the following description provided by way of non-limiting example, with the aid of the figures shown in the accompanying tables, wherein:

Figure 1 shows a user during the usage of a video display terminal VDT, Figure 2 shows the method for determining the usage time T of the VDT, Figures 3a and 3b show an example of a monitoring system of the VDT, Figure 4 shows an example of a face, eye, and pupil detection of a user at the VDT which attest to the attentive observation of the monitor,

Figure 5 shows an example of event recording of attentive observation of the monitor and usage of the input peripherals (mouse and keyboard),

Figure 6 shows an example of a diagram of a general method of determining the usage time of the VDT,

Figure 7 shows an example of a block diagram for determining the usage time of the VDT,

Figure 8 shows an example flow-chart of the algorithm for determining the user's attentive observation of the monitor (interface software unit),

Figure 9 shows an example of the occurrence of events over time and the parameters used in the algorithm for determining the partial usage time relative to a single interaction with the VDT (attentive observation of the monitor or usage of a peripheral),

Figure 10 shows an example of a flow-chart related to the algorithm for determining the partial time of VDT usage for each individual event type, Figure 11 shows an example of summary graph of time traces representing the sequence of partial activity intervals, i.e., referring to the single interaction with the VDT (attentive monitor viewing or peripheral usage), during VDT usage - trial 1 ■ with Tthreshold = 0.5 s,

Figure 12 shows an example of summary graph of time traces representing the sequence of partial activity intervals, i.e., referring to the single interaction with the VDT (attentive monitor viewing or peripheral usage), during VDT usage - trial 1 ■ with Tthreshold = 1 s,

Figure 13 shows an example of summary graph of time traces representing the sequence of partial activity intervals, i.e., referring to the single interaction with the VDT (attentive monitor viewing or peripheral usage), during VDT usage - trial 1 ■ with Tthreshold = 2 S,

Figure 14 shows an example of summary graph of time traces representing the sequence of partial activity intervals, i.e., referring to the single interaction with the VDT (attentive monitor viewing or peripheral usage), during VDT usage - trial 1 ■ with Tthreshold = 5 S,

Figure 15 shows an example of summary graph of time traces representing the sequence of partial activity intervals, i.e., referring to the single interaction with the VDT (attentive monitor viewing or peripheral usage), during VDT usage - trial 1 ■ with Tthreshold = 10 S,

Figure 16 shows an example of flow-chart related to the algorithm for determining the combined usage time of the VDT,

Figure 17 shows an example of a graph of the raw data with Tthreshold = 1 s, Figure 18 shows an example of a graph of time traces which represent the attentive observation of the monitor (cam) and the usage of individual peripherals (mouse and keyboard) with Tthreshold = 1 s,

Figure 19 shows an example of a graph of VDT usage time traces combined with each other in AND, OR, or compound mode with Tthreshold = 1 s,

Figure 20 shows an example of a graph of tusage times of VDT usage for different RegN records with Tthreshold = 1 s and considering individual peripherals, Figure 21 shows an example of a graph of the percentage of monitor observation, mouse and keyboard usage for the different recordings with Tthreshold = 1 s considering the individual peripherals, and

Figure 22 shows an example graph of VDT usage times for different recordings with Tthreshold = 1 s and considering individual peripherals.

The parts according to the present description are shown in the drawings, where suitable, employing conventional symbols, showing only the specific details which are pertinent to the understanding of the embodiments of the present invention, so as not to highlight details which will be immediately apparent to the person skilled in the art, with reference to the description provided below.

DETAILED DESCRIPTION OF THE INVENTION

The present invention relates to tools and methods for monitoring the activity of a user U (U is usually referred to the individual user of a group Ui... Un) at the video display terminal VDT, adapted to objectively and quantitatively determine the actual and active usage time of the VDT by each specific user Ui (Ui indicates a specific user of the group) and to record the work sessions of the user Ui with the corresponding durations. Specifically, the actual and active usage time of the VDT by a user U is determined by the combined analysis of attentive observation of the monitor 10 and the user's active interaction with the VDT by means of the input peripherals 20 (keyboard, mouse, touchpad, joystick, graphics tablet, etc.) by each individual user U, i.e., considering the set of interactions that the user U performs with the VDT, as shown in Figure 1 . Such a combined analysis comprises various combinations (AND, OR or composition thereof, etc) of time traces which represent the sequence of partial activity intervals, i.e. referring to the single interaction with the VDT (attentive display of the monitor or usage of a peripheral). Naturally, only some of the different possible combination operations are indicated by way of nonlimiting example in the description.

Regulations (such as Legislative Decree no. 81 of 9 April 2008 - Transposition of article 1 of Law no. 123, 3 August 2007, regarding the protection of health and safety in the workplace, Official Gazette - Ordinary Supplement no. 101 of 30 April 2008) are aimed at ensuring health and safety in workplaces equipped with video display terminals and regulate the VDT usage by the individual user U based on exposure, i.e., considering the total usage time of the VDT typically on a weekly basis. The VDT usage is typically characterized by the observation of a monitor 10 by user U and by a series of actions that the user performs on the control and pointing systems, substantially consisting of typing and pressing, applied manually on the keyboard, touchpad, trackpoint, trackball, and mouse, i.e., by the interfacing of the user U with the VDT through the input peripherals 20, as highlighted in SIMLII, "Guidelines for health surveillance of employees working with video terminals", 2013.

The main risks associated with prolonged VDT usage are visual disturbances and neuro-muscular-skeletal disorders.

The suggested invention is applied, by way of non-limiting example, to the fields of Occupational Medicine and Safety and Prevention in the workplace, where it meets a widespread and unresolved need because it makes available a tool which can provide a quantitative and objective determination of the work time actually performed at the VDT by a user U. This aspect is essential in order to be able to correctly and objectively classify individual workers as VDT operatives or not according to current regulations. The suggested invention also makes available a set of measured data relating to the usage time and method of individual peripherals, making it possible to identify different usage behaviors and styles of the VDT by different categories of users (e.g., draughtspeople, word processors operatives, document proofreaders, data entry clerks, etc.).

The information generated by the suggested detection and monitoring system can provide useful information about any incorrect use of peripherals (such as intense and prolonged observation of the monitor, abrupt movements of the mouse, inappropriate use of the input devices, etc.) which may lead to damage or adverse effects on the user's health, such as eye-visual and musculoskeletal disorders.

Furthermore, the suggested system makes it possible to identify prolonged sessions of active work at the VDT with objective accuracy revealing whether they are interspersed with appropriate breaks.

It is the object of the present invention to define and develop methods and tools for monitoring the activity of different users Ui, U2, .., Un at the VDT, adapted to objectively and quantitatively determine the actual time of attentive and active usage of a VDT by each individual user U, to analyze the activities performed, to identify the conduct, style, and type of VDT usage and the duration of the various fractions of time in which the user U operates at the VDT. Such a purpose is achieved by means of a monitoring system consisting of a set of hardware devices and software apparatuses which interact appropriately. The actual time T of attentive and active usage of the VDT by a user U is determined by the combined analysis of the presence of the user U, the attentive observation of the monitor 10, and the active interaction with the VDT by means of one or more input peripherals 20 (keyboard, mouse, touchpad, joystick, graphics tablet, etc.) by each individual user U, i.e., considering the set of sensory interactions that the user U performs with the VDT, as shown in Figure 2.

In particular, the usage time T is determined by appropriate combinations of the time traces which represent the sequence of partial activity intervals respectively associated with attentive observation activities 100 of the monitor 10, mouse usage activities 102, keyboard usage activities 104, and usage activities 106 of any other input peripheral.

In an embodiment, the monitoring system S can be partially or fully integrated with the VDT to be monitored and share hardware/software units with the VDT itself, as shown in Figure 3a.

In another embodiment, the monitoring system S is autonomous, in conjunction with and independent of the VDT, i.e., it resides in an external processing unit provided with appropriate software apparatus which is connected to the VDT on which user U operates, but separate from it and not bounded to it, as shown in Figure 3b.

In such a second embodiment, advantageously it is not necessary to interfere in any manner with the VDT, in particular, it is not necessary to install any software on it. Therefore, the VDT does not require any changes to either the installed software, the operating system, or the need for additional drivers.

The system S, in the stand-alone version in combination with the VDT, consists of the following elements:

- at least one optical sensor (video camera) 50 to identify the user U who is using the VDT and to detect the user's attentive observation of the monitor;

- at least one hardware interface unit 60 capable of detecting the input peripheral usage actions (keyboard, mouse, touchpad, joystick, graphics tablet, etc.), without recording any information entered by the user U, to preserve the privacy of the user U;

- at least one hardware processing unit 70 consisting of a microprocessor electronic board with adequate computing capabilities, or a PC or equivalent system, capable of processing through appropriate algorithms the information and data related to the attentive observation of the monitor 10 and the active usage of the peripherals 20 by each individual user U, provide the independent usage time of each individual peripheral and combine said information to provide synthetic parameters and overall metrics which quantify the VDT usage, reflexively allowing the identification and discrimination of different behaviors and styles of operating at the VDT of each individual user U;

- a set of software tools, procedures, algorithms, apparatuses, and tools 80 residing in the processing hardware unit 70 capable of detecting and identifying, recognizing, and classifying the user at the VDT using image processing techniques, intercepting the attentive observation of the monitor 10, and determining the usage of the input peripherals 20 by each individual user U; identifying VDT usage time for each individual user U; identifying, recognizing, and classifying the user U; and determining and storing usage sessions (timestamp and duration) of the VDT by each individual user U;

- connection cables 25 between the VDT input peripherals 20 and the monitoring system S to connect keyboard and mouse or other peripherals to the monitoring system;

- connection cables 35 between the monitoring system S and the VDT to connect the monitoring system S to the VDT;

- connection cables, power supply, and input/output peripherals of the monitoring system S to ensure proper operation and management.

By introducing a distinctive and qualifying function, the present invention, by means of an optical sensor 50 (e.g., a video camera such as a webcam) and appropriate face recognition (face detection) algorithms, makes it possible to identify the presence of a user U and determine the position of the face, eyes, and pupils of the user U and consequently derive the direction in which the gaze of the user U is directed. By detecting the direction in which the user U is looking, it is possible to determine whether the user is looking attentively at the monitor 10 or is instead directing their attention to other objects, and by analyzing eye movements, it is possible to determine the degree of attention of the user U at the VDT.

An example of face detection V of a user U is shown in Figure 4. Following the acquisition of a frame and the detection of the face V of the user U, the monitoring system derives the position of face V, eyes O, and pupils P by deriving the direction in which the gaze of the user U is directed. In case the gaze is directed to the monitor 10, the system S generates an event of attentive observation of the monitor 10 by the user U. Conversely, no event is generated. The image processing algorithm then proceeds by reiterating the previously described acquisition and processing cycle to generate an event whenever attentive observation of the monitor 10 by the user U is recognized.

The present invention further provides user recognition and classification algorithms U (face recognition) to distinguish different users Ui, U2, ..Un which may share the same VDT at different times, to determine the corresponding specific usage time T1, T2, ..., T_n of the VDT for each user U1, U2, .., Un. To distinguish the various users U1, U2, ..., Un, the invention includes a database DB of multiple face images Vi, V2, .., Vn of the various users U1, U2, .., Un, acquired at the initial stage and updated over time, to correlate the user Ui who is using the VDT at a given instant with the n images in the DB database and recognize the identity of user Ui.

To detect the usage of the input peripherals 20, in the stand-alone implementation in conjunction with the VDT, the invention provides suitable hardware interface units 60 to be interposed between the input peripherals 20 and the VDT, such units being connected to the monitoring system S and an integral part thereof.

The hardware interface units 60 and related software apparatuses detect the usage of the input peripherals 20 and communicate this to the monitoring system S by matching a timestamp MT to the usage event. The data streams from the input peripherals 20 remain directed solely to the VDT and are not intercepted by the interface hardware units 60 which operate in a transparent and non-invasive manner. In this manner, the monitoring system S only receives information related to the plain usage of the input devices 20, without receiving information related to the content of the input data stream to safeguard the privacy of the user U. The hardware interface units 60, separate from the VDT, do not require the installation of any drivers on the VDT itself.

In a particular embodiment, each input peripheral 20i requires at least one interface hardware unit 60i to detect the usage thereof; in an alternative embodiment, shown in Figure 3b, a single interface unit 60 can detect the usage of one or more input peripherals 20i.

The system S, in its integrated version with the VDT, consists of the following elements: at least one optical sensor 50 (camera) integrated into or external to the VDT; at least one interface software unit 60a capable of detecting usage actions of input peripherals 20; the processing hardware unit 70a which consists of the VDT itself; a set of tools procedures, algorithms, apparatus, and software tools 80a resident on the VDT itself.

The elements composing the system S operate similarly to the stand-alone version in conjunction with the VDT.

The monitoring system S can derive information regarding the identification 200 of the user Ui among the n cataloged in the database DB, the attentive observation 210 of the monitor 10, and the active usage 220 of the input peripherals 20 made by each individual user Ui, U2, .., Un, recording the events and the time T at which they occurred on computer means, as shown in Figure 5.

Figure 6 shows a diagram describing the general method of determining the usage time T of the VDT.

The interface unit 60 receives the detection data from the camera 50, the keyboard 20a, the mouse 20b, and other input devices 22.

The usage time T of the VDT per individual user U is calculated from the user 200 recognition data, the observation detection 210 of the monitor 10, and the usage detection 220 of the peripherals 20.

The present invention provides specific parameters and specially developed metrics to be able to determine the actual attentive and active usage time T of the VDT.

For determining usage of the VDT, each input peripheral 20, indicated by Pi, P2, ..., PN, and PN+I, ... PM is provided with hardware interface units HWPi, HWP2,..., HWPN, or software interface units SWPN+I,.. SWPM as indicated by the block diagram in Figure 7 by which hardware events EP1, EP2, ..., EPN, or software EPN+I, ..., EPM related to the usage of the peripheral itself are generated, respectively. The numbering [1 ... N] indicates the peripheral devices with interface hardware units and in particular Pi could be mouse 20b, P2 could be keyboard 20a, while the numbering [N+1 ... M] indicates the devices with software interface units, and in particular PN+I could be the webcam 50. In the diagram in Figure 7, the processing unit 70 is responsible for determining the usage time of the VDT based on the hardware events EP1, EP2, .. EPN, and software EPN+I, ..., EPM which are generated by the hardware interface units HWP1, HWP2, .., HWPN, and software interface units SWPN+I, .., SWPM when the user U uses the input devices Pi, P2, ..., PN, and/or PN+I, .. PM, and/or attentively observe the monitor 10, making it possible to calculate the usage time of a single peripheral or the time T of combined usage of the various peripherals.

In the case of an input peripheral which is part of the set [1 ... N], for which a deliberate action of the user U towards the VDT is expected, such as, e.g., typing and manual strokes on the keyboard 20a or mouse movements 20b, it is possible to determine the actual usage of the peripheral by directly analyzing the electrical signals related to the communication between the VDT and the concerned peripheral. By means of the hardware interface unit [1 ... N] it is thus possible to analyze such electrical signals and generate a hardware event [EP1 .. EPN] with which a specific timestamp is associated whenever the peripheral EP1, EP2, ... EPN has actually been used by the user U.

By way of example, in the case of peripherals with USB communication protocol, two operating conditions may occur; only periodic synchronization signals SS are present between peripheral EP1, EP2, ... EPN and VDT in the absence of usage of the input devices 20, while conversely, in case of active usage, signals related to the specific usage SU of the peripheral are present in addition to the synchronization signals. By appropriately analyzing the electrical signals, it is thus possible to distinguish and eliminate by means of appropriate filtering the synchronization signals SS and keep the specific usage SU signals and thus generate hardware events EPi, EP2, ... EPN in the case of active usage of the peripheral through monostable circuits, by way of non-limiting example. Therefore, by exploiting this mode, it is possible to analyze the active usage of the peripheral Pi, P2, ... PN without detecting any sensitive information entered by the user U through the peripheral, thus preserving their privacy.

If peripheral belongs to the set [N+1 ... M], e.g., such as video cameras 50, optical or ultrasonic motion or distance sensors, it is not sufficient to consider the electrical signals related to the communication between the VDT and the peripheral, but it is necessary to analyze the information provided by the peripheral PN+I ,..., PM to determine whether there have been interactions between the user U and the VDT. Therefore, software interface units [N+1 ... M] are needed to determine the type of activities regarding the specifically concerned peripheral PN+I ,... PM and to generate the software events [SWPN+I ... SWPM], as shown in Figure 7.

Various devices and processing techniques can be used to determine the presence of the user U in front of the monitor, such as, by way of non-limiting example, optical or ultrasonic motion and distance sensors, 3D scanners, optical triangulators, visible and infrared cameras.

A particular implementation of the monitoring system S is the use of a video camera 50 (webcam). In this case, to determine the attentive observation of the monitor 10 by the user U, it is necessary to firstly determine the presence of a user U in front of the monitor 10 by recognizing a shape resembling a face V within the image captured by the camera 50, and secondly determine whether the user U is attentively observing the VDT by analyzing a sequence of frames to determine eye movements O.

In particular, the flow chart of the algorithm for determining the attentive observation of the monitor 10 by the user U using a video camera 10 is shown in Figure 8.

The process involves a step of starting 1000. In a step 1002, the video camera 50, appropriately located so that it can frame the face V of the user U who is operating at the VDT, is responsible for acquiring a sequence of individual frames. In step 1004, the face is recognized and in decision step 1006 if no face V is detected the control goes back to step 1002. If a face V is detected, the successive step 1008 performs eye recognition O. In decision step 1010 if the O eyes are not detected the control returns to step 1002. If the eyes O are detected, the position (x,y,t) of the eyes is determined in a step 1012. The face V is recognized in the successive step 1014. In the successive decision step 1016, the method checks whether the coordinates (x,y) of the position of the eyes O are compatible with the position of the monitor 10. In the negative case, the control goes back to step 1002. In the positive case, the time variation At=ti+i-ti (or At=ti-ti-i) is calculated in step 1018, the variation Ax=x(ti+i)-x(ti) (Ax=x(ti)-x(ti-i )) in the coordinate x of the eye position O is calculated in step 1020, and the variation Ay=y(ti+i)-y(ti) (or Ay=y(ti)-y(ti-i)) in the coordinate y of the eye position O is calculated in step 1022. In decision step 1024, the method checks whether the displacement speed of the position of the eyes O in the x- coordinate is between two minimum and maximum thresholds, \.e.,sx_min < x/ t < sx_max. In the negative case, the control goes back to step 1002. In the positive case, in decision step 1026, the method checks whether the displacement speed of the position of the eyes O in the y-coordinate is between two minimum and maximum thresholds, i.e., sy_min < y/ t < sy_max. In the negative case, the control goes back to step 1002. In the positive case, the active presence of a user U is recognized in step 1028 and a webcam event ECAM is generated in step 1030.

Every single frame acquired by the video camera 50 is analyzed by face detection techniques to detect the presence of a human face V and face recognition to then identify the user U. To perform the recognition of the presence of the face V, the algorithm attempts to identify the possible presence of a shape correlated to the one from the face V and, in case of actual recognition of presence, it continues with the search of the eyes O and the pupils P through shapes correlated thereto. The algorithm then provides the location and size thereof. Once the presence of a human face V is recognized, the face recognition operation is performed, i.e., the image acquired by the video camera 50 is compared with a set of images related to previously acquired reference identities present in a DB database to attribute the identity of the user U at the VDT relative to the previously surveyed identities. Successively, using the pupil position coordinates, the interface software unit verifies whether the gaze of the user U is directed towards an area compatible with the size of the monitor 10 and the location of the monitor relative to the camera 50. The information regarding the size and location of the monitor 10 relative to the camera 50 is entered in advance during the step of configuring and initializing of the monitoring system S, either manually or using semi-automatic tools or procedures. Such tools or procedures ask the user U, by way of example, to direct their gaze at particular areas of the monitor 10 or to follow with their gaze given paths or markers shown appropriately on the monitor 10. The algorithm, by processing of a sequence of frames, also evaluates whether there is an attentive and active observation of the monitor 10, also to avoid false detections due to the presence of static images, such as in the case of framing a photograph. The algorithm then evaluates the variations of the positions of the eyes Ax and Ay occurred in time At through the analysis of sequences of frames, to assess the active presence of user U; if the displacement speed Ax/At and Ay/At of the eyes on the monitor in the two coordinates x and y is between appropriate thresholds settable during the step of configuring, it is considered that the user U has varied the observation zone of the monitor 10 and thus is actively interfacing with the VDT. If there are multiple people framed by the camera 50, the interface software unit attempts to recognize the various users U by face detection and face recognition techniques and generates software events related to the user U actually being monitored.

The activity at the VDT is characterized by a set of discontinuous actions over time which generate events detected by the monitoring system. These events are characterized by such a short duration that they can be considered negligible to calculate the actual usage time of the VDT. The duration of such events is negligible and thus the actual usage time of the VDT cannot be determined from the sum of the duration of the individual events. Furthermore, a work session at the VDT is characterized by a set of multiple interactions between the user U and the VDT distributed over time without strict regularity and consequently by a set of events without a predefined periodicity. It is thus necessary to define the methods of analysis for determining the actual usage time of the VDT based on the presence of events and their recurrence density.

Primarily, the time stamps MT of the events 100 of attentive observation of the monitor 10 and of usage 102,104,106 of each individual input peripheral 20 are determined from the indications provided by the hardware and software interface units 60 using the previously described dedicated algorithms. Among the possible alternatives, the activity at the VDT is determined by analyzing the presence of repeated events over time, assumed to be of zero duration.

For each type of detected event (attentive observation 100 of the monitor 10 or usage 102,104,106 of the single input peripheral 20) the time stamp MT associated with the single event is considered and the time interval ATi between two successive distinct events is determined, according to the formula: T, = ti+i - ti where ti is the time at which the i-th event occurs and ti+i the time at which the immediately successive event occurs. If the time ATi is lower than a certain settable threshold Threshold it is assumed that the considered activity (attentive observation of the monitor 10 or usage of the input peripheral 20) has been performed without interruption for a duration of time equal to ATi. If the time interval ATi is greater than the threshold Threshold, it is assumed that in such a time interval ATi the considered activity (attentive observation of monitor 10 or usage of input peripheral 20) has been interrupted.

For each type of interaction with the VDT (attentive display of the monitor or usage of a peripheral device) the temporal traces are then generated, i.e. as a function of time, representing the sequence and temporal distribution of partial activity intervals, i.e. referring to each specific interaction with the VDT.

The partial usage time Tpartiai of the monitor or each individual VDT peripheral involved is determined by incrementing the current value by adding to it the time interval ATi if it is less than the threshold Threshold. In the opposite case, i.e., if the time interval ATi is greater than the threshold Threshold, it is considered that in such a time interval ATi activity has been interrupted and consequently the partial usage time Tpartiai is not increased.

Therefore, the partial usage time Tpartiai for the monitor and each peripheral is determined by the formula:

Tpartiai = S AT; considering only the time intervals ATi < Tthreshold.

An example of how the algorithm for determining the partial usage time Tpartiai for the monitor or each peripheral operates is shown in the example in Figure 9.

Considering the events Ei, E2, E3, E4 and Es, occurred respectively in time ti, t2, ta, t4 and ts, the times between successive events AT1, AT2, AT3 and AT4 are calculated and compared with the threshold Tthreshoid. In the representation shown in Figure 9, the times AT1 , AT2 and AT4 are smaller than the Tthreshoid, whereas AT3 turns out to be greater, i.e., it is believed that between event E3 and the successive event E4 there was an interruption of activity at the VDT. Consequently, in the illustrated case, the partial usage time results: Tpartiai = AT1 + AT2 + AT4.

The aforesaid procedure for determining the Tpartiai time is formalized in the algorithm described by the flow chart in Figure 10.

The process includes a step 1032 of initiating, and a successive step 1034 of event reading.

The time between successive AT; = t_i+1 - q events is calculated in a step 1036, and in a decision step 1038 we check if this time is less than the threshold Tthreshoid and if not, the control returns to step 1034. In the positive case, in step 1040, the partial time is incremented by the value ATi. In a decision step 1042, the method checks if the session is terminated, and in the negative case the method returns to step 1034, and in the positive case in a step 1044 the value Tpartiai. is saved.

The graphs in Figures 1 1 -15 show the time traces which represent the sequence of activity intervals obtained from the experimental results of the process of determining the partial observation times of the monitor 100 and usage 102 and 104 of the individual peripherals obtained for different values of the threshold time Tthreshoid during a usage session of a VDT consisting of a PC having mouse 20b and keyboard 20a as input peripherals. The logical value "1 " is used for each time trace if the presence of input activity for the specific peripheral is detected, while the logical value "0" is reported if there was no activity. As is clearly visible from the graphs, the number of activity interruptions considered decreases as the duration of the threshold time Tthreshoid increases.

Regardless of the value of Tthreshoid used, a complete interruption from the usage of the VDT is observed from second 6805 to about second 8100, where neither activities of use of the mouse 102 and keyboard 104 nor the observation of the monitor 100 are found, corresponding to an interval of time of complete inactivity which is consistent with the actual documented interruption of the usage of the VDT. Table 1 shows the times of observation of the monitor 100 and of usages of the mouse 102 and keyboard 104 resulting for the same experimental trial with different settings of the threshold time values Threshold. It can be observed that as the set value of the threshold time Threshold increases, the overall estimated times of monitor observation and mouse and keyboard usage increase, because short interruptions which last less than the time Threshold are not considered as such.

Table 1 : Estimation of partial usage times of the VDT as the threshold time Threshold changes

The invention further comprises synthetic indices appropriately arranged to determine the time of attentive and active work at the VDT from the partial times of attentive observation of the monitor 100 and usage of the individual peripherals 102 and 104. A true interaction between user U and VDT requires performing combined operations of monitor observation 100 and input actions from mouse 20b and/or keyboard 20a. Therefore, it is significant to consider not only the partial times of work at the VDT determined individually by the observation of the monitor detected by video camera 50 and by the input actions through the usage of keyboard 20a and mouse 20b but also to examine an appropriate combination of the time traces which represent the sequence of partial activity intervals, i.e. referred to the single interaction with the VDT (attentive observation of the monitor or usage of a peripheral), to describe and quantify the situations of combined usage. Specifically, the following three different reference cases were considered (see Figure 19):

- Combined usage A: usage of at least one of the input devices 20 or monitor observation 10 (logical OR).

- Combined usage B: simultaneous usage of all input peripherals 20 and simultaneous observation of monitor 10 (logical AND).

- Combined usage C: usage of at least one input peripheral 20 and simultaneous monitor observation 10 (partial logical AND on a peripheral).

The algorithm for determining combined VDT usage time is described in the flowchart in Figure 16.

The procedure includes a step of starting 1046, and a successive step 1048 of calculating the partial usage time of peripheral 1 , a step 1050 of calculating the partial usage time of peripheral 2, and subsequent steps of calculating the partial times of the peripherals until step 1052 of calculating the partial usage time of the individual peripheral. In a step 1054, the combined usage time (AND, OR, etc.) is determined, and in the successive step 1056, the combined usage time is calculated and stored in a step 1058. The process ends in step 1060.

The processing unit 70 or 70a, represented by the block diagram in Figure 7, separately determines the partial usage times of the individual peripherals starting from the hardware events EPi, EP2, ..., EPN, and software events EPN+I, ..., EPM related to the usage of each individual peripheral. Successively, the processing unit 70 or 70a, considering for the individual peripherals the hardware events EP1, EP2, ..., EPN, and software events EPN+I, .., EPM, determines the combined usage time of the VDT taking into account the simultaneous presence of multiple user input actions U and/or attentive observation of the monitor 10.

By way of example, Figures 17-19 show in graphical form the experimental results of the monitoring process of a VDT consisting of a PC having mouse 20b and keyboard 20a as input peripherals, specifically using three specific graphs: Figure 17: "Unprocessed data": the graph shows with square markers the data provided by the VDT monitoring system broken down by type. The x-axis shows the number of seconds elapsed since the first event was recorded (Threshold = 1 s). Figure 18: "Single peripheral usage": three time traces are shown, indicating the activity intervals of mouse usage 102, keyboard usage 104, and attentive monitor observation interval 100 (Tthreshoid = 1 s), respectively. The logical value "1" is used for each time trace if the presence of input activity for the specific peripheral is detected, while the logical value "0" is reported if there was no activity. The x-axis shows the time and/or date of the recorded events.

Figure 19: "Combined peripheral use". Three time traces A, B, C are shown indicating the activity intervals for the combined usage conditions (mouse OR keyboard OR cam), (mouse AND keyboard AND cam) and ((mouse AND cam) OR (keyboard AND cam)), respectively. As in the previous cases, Tthreshoid = 1 s. The logical value "1 " is used for each time trace if the presence of input activity for the specific peripheral is detected, while the logical value "0" is reported if there was no activity. The x-axis shows the time and/or date of the recorded events.

It was possible to determine the times of VDT usage for the various cases by examining the time track of the combined usage of the peripherals. Table 2 shows the times of VDT usage considering separately the observation of the monitor, the usage of the mouse, the use of the keyboard, and considering also the combined use of the peripherals according to the reference cases A, B, and C previously described. All values of the usage times were determined considering the time interval Tthreshoid = 1 s.

Table 2: VDT usage times with threshold time Tthreshoid = 1 s Figure 20 shows in graphical form the partial times related to the interaction of user U with the VDT by means of single activities (observation of the monitor 100 or usage of the mouse 102 or usage of the keyboard 104) experimentally detected for different monitoring sessions called recordings RegN.

Figure 21 shows, for each individual recording RegN, the observation percentage of the monitor 100 and usage of mouse 102 and keyboard 104 relative to the sum of partial VDT usage times. As can be noted, except for the case of recording Reg04, there is a predominance of the observation times of the monitor 10 over the times in which user U performs input operations 20 using the mouse 20b and/or the keyboard 20a and a predominance of mouse usage over keyboard usage.

The times related to VDT usage, considering the combined modes of peripheral usage, are shown in graphical form in Figure 22.

The combined usage case A (mouse OR keyboard OR cam) makes it possible to consider the time in which the user U actively interacts at the VDT with at least one input peripheral 20 or looking intently at the monitor 10. With this analysis mode, even mere observation of the monitor 100 (e.g., watching a video or reading a document without input activity) is counted as work activity at the VDT.

If one intends to consider as work activity at the VDT not only the observation of the monitor 10 but also the simultaneous usage of at least one peripheral (mouse 20b and/or keyboard 20a), it seems useful to consider the case of combined usage C ((mouse AND cam) OR (keyboard AND cam)).

The combined usage case B (mouse AND keyboard AND cam) counts as actual working time only that in which the user U simultaneously uses all input peripherals 20 and manifests attentive observation of the monitor 10. Such a situation is unlikely to occur, especially for experienced users. For example, in writing a text typically the user U observes the monitor 10 and interacts with the VDT using the keyboard 20a and rarely uses the mouse 20b, while in the case of reading a text typically the user U observes the text and scrolls through that document either using the keyboard 20a or using the mouse 20b. Therefore, mode B may result in a tendency to underestimate VDT usage time.

The invention makes it possible to determine the usage time of the VDT considering different factors (e.g. usage of single peripherals, combined usage, etc.) with different weights which can be pre-selected by special configuration options, or freely set according to particular cases of users.

Starting from the measurement of the partial times related to the usage of the single peripheral devices, of the attentive observation of the monitor, and of the modality/sequence of actions with which the user interacts with the VDT, the present invention can provide objective metrics to determine the type of activity carried out by the user (e.g. making CAD drawings, data entry, word processing, reading documents, correcting documents, etc.) and to identify the different modalities/behaviors of user interaction with the VDT; indeed, each user tends to use the VDT with a particular and personal style which is correlated to the purpose and type of usage and degree of experience and familiarity with the IT tool.

Therefore, the present invention can be applied for an additional purpose represented by the quantitative determination and classification of VDT usage styles which can be associated, for example, with education and training in the usage of computer tools or the recognition of inadequate or ineffective behaviors.

A further function allowed by the invention concerns the possibility of distinguishing between different users, called to carry out at the VDT the same task taken as a reference, based on the imperceptible differences in conduct and style of action reflected in the partial usage times of the various peripherals and appropriate metrics derived therefrom. This provides the possibility of performing the equivalent of handwriting analysis in the domain of digital interaction, exploiting appropriate user-specific "signatures" which are not graphic, as is the case in traditional analysis, but are biometric/sensory.

A further functionality allowed by the invention consists in the automatic, objective, and real-time verification and monitoring of the degree of attention of a user engaged in an interactive activity with a VDT, also possibly remotely connected, e.g. during teleconferences, work sessions, or distance training.

The solution described here is based on tools and methods for monitoring the activity of a user at a video display terminal VDT. The solution employs an integrated computer and electronic system.

In its implementation, the monitoring system is autonomous, in combination with the VDT, and independent from the VDT, i.e. it resides in an external unit unrelated to the VDT itself and equipped with the appropriate software.

In another implementation, the monitoring system is integrated with the VDT being monitored.

The monitoring system is non-intrusive for the user, without disrupting normal activity at the VDT.

The solution suggested herein includes the presence of at least one camera to identify the user U who is using the VDT and to detect the user's attentive observation of the monitor and interaction with the VDT.

In various embodiments, the user detection and identification/recognition at the VDT are employed using image processing techniques (such as face detection and face recognition algorithms).

In the embodiments, the detection of the actual usage of input devices (keyboard, mouse, touchpad, joystick, graphic tablet, etc.), is done without recording any information entered by the user, to preserve privacy.

The solution described here allows for an objective and quantitative determination of the user's actual, attentive, and active VDT usage time, a determination of the partial usage time of the individual input peripherals, and a determination of metrics for evaluating the usage time of a VDT by combining the time traces of partial activity intervals, i.e., referring to the individual interaction with the VDT (attentive monitor viewing or peripheral usage).

With the solution described here, it is possible to identify the usage time of the VDT by each user and to determine and store the usage sessions (time stamp and duration) of the VDT.

The various embodiments of the solution allow a quantitative and objective determination, according to specific parameters and specially developed metrics, of the usage time of the VDT by a user through the combined analysis of the attentive observation of the monitor by the user and the active interaction of the user with the VDT by means of the input peripherals (keyboard, mouse, touchpad, joystick, graphics tablet, etc.), i.e., considering the whole of the interactions that the user makes with the VDT.

The various embodiments provide innovative ways of combining and representing the partial activities carried out by the user to determine in an objective and robust manner what can be defined as time spent actively and attentively using the VDT.

The solution described here allows the identification of metrics for quantifying the work done at the VDT and classifying the type of user according to the type of activity performed (e.g., CAD worker or data entry, word processing, etc.).

Claims

1 ) A system (S) for monitoring the activity of users (U) at a video display terminal (VDT), to determine the actual and active usage time (T) of the video display terminal (VDT) by each specific user (Ui) and to record the user's work sessions (Ui) with their durations (Tpartiai), wherein the actual and active usage time (T) of the video display terminal (VDT) by a user (Ui) is determined by the combined analysis of attentive observation (100) of the monitor (10) by the user (Ui) and the active interaction (102,104,106) of the user (Ui) with the video display terminal (VDT) by means of the input peripherals (20), wherein said system (S) comprises:

- at least one optical sensor (50) configured to identify the user (Ui) who is using the video display terminal (VDT) and to detect their attentive observation of the monitor (10);

- at least one interface unit (60,60a) configured to detect the usage actions of the input peripherals (20), without recording any user input information (Ui);

- at least one hardware processing unit (70,70a) configured to process the information and data related to the active usage (102,104,106) of the peripherals (20) and the attentive observation (100) of the monitor (10), and configured to provide the independent usage time (100, 102,104,106) of attentive observation of the monitor (10) of each single peripheral (20) and to combine said information to calculate the usage time (T); and

- a set of software tools (80,80a) configured to:

- detect and identify the user (Ui) at the video display terminal (VDT) using image processing techniques,

- intercept the attentive observation (100) of the monitor (10) by the user (Ui),

- determine the usage (102,104,106) of the input peripherals (20),

- identify the usage time of thevideo display terminal (VDT) by each user (Ui);

- recognize and classify the user (Ui); and

- determine and store the video display terminal (VDT) usage sessions, wherein the combined analysis of the attentive observation (100) of the

32 monitor (10) by the user (Ui) and the active interaction (102,104,106) of the user (Ui) with the video display terminal (VDT) through the input peripherals (20) comprises various combinations of the time traces representing the sequence of the partial activity intervals, i.e. referring to the single interaction with the VDT such as the attentive observation of the monitor (100) or the use of the single peripheral (102, 104,106).

2) The system (S) for monitoring the activity of users (U) at a video display device (VDT) according to claim 1 , wherein said various combinations comprise operations such as AND, OR or compositions thereof, of the time traces representing the sequence of partial activity intervals partial, i.e., referring to the single interaction with the VDT such as the attentive observation of the monitor (100) or the use of the single peripheral device (102, 104,106).

3) The system (S) for monitoring the activity of users (U) at a video display terminal (VDT) according to claim 1 or claim 2, wherein said optical sensor (50) and said software tools (80) are configured to implement facial recognition to distinguish different users (Ui, U2, ...Un) who may share the same video display terminal (VDT) at different times to determine for each user (Ui, U2, ...Un) the corresponding specific usage time (T1, T2, ...Tn) of the video display terminal (VDT), wherein said system (S) comprises a database (DB) of multiple face images (Vi, V2, ...Vn) of various users (Ui, U2, ...Un) acquired during the initial step and updated over time to correlate the user (Ui) who is using the video display terminal (VDT) at a given instant with the images in the database (DB) and recognize the identity of the user (Ui).

4) The system (S) for monitoring the activity of users (U) at a video display terminal (VDT) according to one or more of the preceding claims, wherein said interface unit (60,60a) is interposed between the input peripherals (20) and the video display terminal (VDT).

5) The system (S) for monitoring the activity of users (U) at a video display terminal (VDT) according to one or more of the preceding claims, wherein each input peripheral (20, Pi, P2, ..., PN) is provided with hardware interface units (60, HWP1, HWP2, ..., HWPN) by means of which hardware events are generated (EP1, EP2, ..., EPN) related to the usage of the peripheral (20, Pi, P2, ..., PN) and

33 said unit processing unit (70,70a) is configured to determine the usage time of the video display terminal (VDT) based on the hardware events (EPi, EP2, ... EPN) which are generated by the interface units (60, 60a, HWP1, HWP2, ..., HWPN) when the user (U) uses the input devices (Pi, P2, ..., PN) or attentively observes the monitor (10), and wherein said processing unit (70,70a) is configured to determine the usage time of a single peripheral or the combined usage time (T) of the various peripherals.

6) The system (S) for monitoring the activity of users (U) at a video display terminal (VDT) according to claim 5, wherein said software interface units (SWPN+I , ..., SWPM) connected to peripherals (PN+I , ... PM) are configured to generate software events (SWPN+I ... SWPM), and wherein said peripherals (PN+I , ... PM) are motion sensors, optical or ultrasonic distance sensors, 3D scanners, optical triangulation sensors, visible spectrum cameras and infrared cameras.

7) The system (S) for monitoring the activity of users (U) at a video display terminal (VDT) according to one or more of the preceding claims, wherein said system (s) is autonomous and independent of the video display terminal (VDT).

8) The system (S) for monitoring the activity of users (U) at a video display terminal (VDT) according to one or more of the preceding claims 1 to 6, wherein said system is either partially or fully integrated with the video display terminal (VDT).

9) A method for monitoring the activity of users (U) at a video display terminal (VDT) comprising the steps of:

- determining the actual and active usage time (T) of the video display terminal (VDT) by each specific user (Ui),

- recording the work sessions of the user (Ui) with their respective durations (Tpartiai), wherein the actual and active usage time (T) of the video display terminal (VDT) by a user (Ui) is determined by the combined analysis of the time traces of attentive observation (100) of the monitor (10) by the user (Ui) and of the time traces of the active interaction (102,104,106) of the user (Ui) with the video display terminal (VDT) by means of the input peripherals (20),

- providing at least one optical sensor (50) configured to identify the user (Ui) who is using the video display terminal (VDT) and to detect their attentive observation of the monitor (10);

- arranging at least one interface unit (60,60a) configured to detect the usage actions of the input peripherals (20), without recording any user input information (Ui);

- providing at least one processing hardware unit (70,70a) configured to process information and data related to active usage (102,104,106) of the peripherals (20) and attentive observation (100) of the monitor (10), and configured to provide the independent usage time (100, 102,104,106) of attentive observation of the monitor (10) and usage of each single peripheral (20) and to combine said information to calculate the usage time (T); and

- providing a set of software tools (80,80a) configured to:

- determine the usage (102,104,106) of the input peripherals (20),

- identify the usage time of thevideo display terminal (VDT) by each user (Ui);

- recognize and classify the user (Ui); and

- determine and store the video display terminal (VDT) usage sessions, wherein the combined analysis of the attentive observation (100) of the monitor (10) by the user (Ui) and the active interaction (102,104,106) of the user (Ui) with the video display terminal (VDT) through the input peripherals (20) comprises various combinations of the time traces representing the sequence of the partial activity intervals, i.e. referring to the single interaction with the VDT such as the attentive observation of the monitor (100) or the use of the single peripheral (102, 104, 106).

10) The system (S) for monitoring the activity of users (U) at a video display device (VDT) according to claim 9, wherein said various combinations comprise operations such as AND, OR or compositions thereof, of the time traces representing the sequence of partial activity intervals partial, i.e., referring to the single interaction with the VDT such as the attentive observation of the monitor (100) or the use of the single peripheral device (102, 104, 106).

1 1 ) The method for monitoring the activity of users (U) at a video display terminal (VDT) according to claim 9 or claim 10, wherein the step is provided of configuring said optical sensor (50) and said software tools (80) to implement facial recognition to distinguish different users (Ui, U2, ...Un) who may share the same video display terminal (VDT) at different times, to determine for each user (U1, U2, ...Un) the corresponding specific usage time (T1, T2, ...Tn) of the video display terminal (VDT), wherein said system (S) comprises a database (DB) of multiple face images (Vi, V2, ...Vn) of various users (U1, U2, ...Un) acquired during the initial step and updated over time to correlate the user (Ui) who is using the video display terminal (VDT) at a given instant with the images in the database (DB) and recognize the identity of the user (Ui).

12) The method (S) for monitoring the activity of users (U) at a video display terminal (VDT) according to one or more of the preceding claims 9-1 1 , wherein the step is provided of interposing said interface unit (60,60a) between the input peripherals (20) and the video display terminal (VDT).

13) The method for monitoring the activity of users (U) at a video display terminal (VDT) according to one or more of the preceding claims 9-12, wherein the steps are provided of:

- equipping each input device (20, Pi, P2, ..., PN) with hardware interface units (60, HWP1, HWP2,..., HWPN) by which hardware events are generated (EP1, EP2, ..., EPN) related to the usage of the peripheral (20, Pi, P2, ..., PN),

- configuring said processing unit (70,70a) to determine the usage time of the video display terminal (VDT) based on the hardware events (EP1, EP2, ... EPN) which are generated by the interface units (60, 60a, HWP1, HWP2, ..., HWPN) when the user (U) uses the input devices (Pi, P2, ..., PN) or attentively observes the monitor (10),

- configuring said processing unit (70,70a) to determine the usage time of a single peripheral or the combined usage time (T) of the various peripherals.

14) The method for monitoring the activity of users (U) at a video display terminal (VDT) according to claim 13, wherein the steps are provided of:

36 - configuring said software interface units (SWPN+I, SWPM) connected to peripherals (PN+I, ... PM) to generate software events (SWPN+I ... SWPM), and

- selecting said peripherals (PN+I, ... PM) from motion sensors, optical or ultrasonic distance sensors, 3D scanners, optical triangulation sensors, visible spectrum cameras, and infrared cameras.

15) The method (S) for monitoring the activity of users (U) at a video display terminal (VDT) according to one or more of the preceding claims 9 - 14, wherein said method is autonomous and independent of the video display terminal (VDT). 16) The method for monitoring the activity of users (U) at a video display terminal (VDT) according to one or more of the preceding claims 9-14, wherein said method is partially or fully integrated with the video display terminal (VDT).

37