WO2018218286A1 - Method and system for abnormality detection - Google Patents
Method and system for abnormality detection Download PDFInfo
- Publication number
- WO2018218286A1 WO2018218286A1 PCT/AU2018/050520 AU2018050520W WO2018218286A1 WO 2018218286 A1 WO2018218286 A1 WO 2018218286A1 AU 2018050520 W AU2018050520 W AU 2018050520W WO 2018218286 A1 WO2018218286 A1 WO 2018218286A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- data
- individual
- behavioural
- abnormality
- detection
- Prior art date
Links
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/117—Identification of persons
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0059—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
- A61B5/0062—Arrangements for scanning
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/01—Measuring temperature of body parts ; Diagnostic temperature sensing, e.g. for malignant or inflamed tissue
- A61B5/015—By temperature mapping of body part
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/172—Classification, e.g. identification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
- G16H40/60—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
- G16H40/63—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices for local operation
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
- G16H40/60—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
- G16H40/67—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices for remote operation
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/02—Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
- A61B5/024—Detecting, measuring or recording pulse rate or heart rate
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
- A61B5/1118—Determining activity level
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
- A61B5/1123—Discriminating type of movement, e.g. walking or running
Definitions
- the present disclosure generally relates to a method and system for abnormality
- detection e.g., for detecting physical or behavioural abnormalities of a person.
- surveillance cameras are commonly used for monitoring an indoor or
- Thermographic cameras such as infrared cameras have been installed in some public places, e.g., airports, ports and terminals, for monitoring body temperatures of passengers, in order to minimise public health risks such as the spread of infectious diseases.
- Some of these devices may compare extracted behavioural or physical features of a target person with predetermined ranges representing predetermined "normal” behavioural or physical features.
- An exemplary application is the monitoring of students within a learning environment, such as a school.
- an abnormality determination system for determining an abnormality of an individual within a learning environment, including: a monitor, including one or more imaging devices, said imaging devices configured to generate imaging signals; and at least one abnormality detection server device, including: a communications interface to receive data; at least one computer processor to execute program instructions; and a memory, coupled to the at least one computer processor, storing program instructions for execution by the at least one computer processor to automatically: receive imaging signals from the monitor; process the imaging signals to generate imaging data; process the imaging data to detect the individual, and to generate corresponding identity data representing the identity of the individual; and identify an abnormality of the individual based on: physical detection data representing the detection of a physical characteristic of the individual from the imaging data, and a corresponding physical profile of the individual; or behavioural detection data representing the detection of a behavioural characteristic of the individual from the imaging data, and a corresponding behavioural profile of the individual, and context data representing a context of the detected behavioural or physical characteristics within the learning environment.
- the present invention also provides an abnormality determination method for
- determining an abnormality of an individual within a learning environment including: receiving imaging signals from a monitor including one or more imaging devices, said imaging devices configured to generate the imaging signals; processing the imaging signals to generate imaging data; processing the imaging data to detect the individual, and to generate corresponding identity data representing the identity of the individual; and identifying an abnormality of the individual based on: physical detection data representing the detection of a physical characteristic of the individual from the imaging data, and a corresponding physical profile of the individual; or behavioural detection data representing the detection of a behavioural characteristic of the individual from the imaging data, and a corresponding behavioural profile of the individual, and context data representing a context of the detected behavioural or physical
- the present invention also provides a system, including:
- one or more data-capturing devices configured to generate observation data, the observation data representing a presence of an individual
- At least one abnormality detection device including:
- a communications interface to receive data
- At least one computer processor to execute program instructions; and at least one memory, coupled to the at least one computer processor, storing program instructions for execution by the at least one computer processor to automatically:
- process the observation data to generate identity data representing the identity of the individual, and to detect at least one physiological characteristic or behavioural characteristic of the individual;
- the present invention also provides a method, including:
- observation data from one or more data-capturing devices, the observation data representing a presence of an individual
- processing the observation data to generate identity data representing the identity of the individual, and to detect at least one physiological characteristic or behavioural characteristic of the individual;
- FIG. 1 is a schematic diagram of an example of a system for abnormality detection
- Fig. 2 is a flow chart of an example of a method for abnormality detection
- FIG. 3 is a flow chart of an exemplary working process of a server device in the
- FIG. 4 is a block diagram of an example of the server device
- FIG. 5 is a block diagram of exemplary software components of the server device
- FIG. 6 is a block diagram of a monitor in the system of Fig. 1;
- FIG. 7 is a schematic diagram of an example of a system for abnormality detection for use in a learning environment
- FIG. 8 is a flow chart of an example of a method for abnormality detection of a server device in the system of Fig. 7;
- FIG. 9 is a flow chart of an example of a working process of a server device in the system of Fig. 7;
- FIG. 10 is a flow chart of an example of a method for detecting a behavioural
- FIG. 11 is a schematic diagram of a behavioural event generation subsystem for the abnormality detection system of Fig. 7;
- Fig. 12 is a flow chart of an example of a method for generating behavioural events of the behavioural event generation subsystem of Fig. 11;
- Fig. 13 is a block diagram showing the operation of an object detector unit of the behavioural event generation subsystem of Fig. 11;
- Fig. 14 is a diagram of results for an intersection-over-union (IoU) metric for
- FIG. 15 is a block diagram showing the operation of an object tracker unit of the
- Fig. 16 is a block diagram showing a multi-stage Long Short-Term Memory (LSTM) Network for performing Action Recognition in accordance with the behavioural event generation subsystem of Fig. 11;
- LSTM Long Short-Term Memory
- FIG. 17 is a schematic diagram of an abnormality recognition subsystem for the
- Fig. 18 is a flow chart of an example of a method for high-level behavioural
- Fig. 19 is a schematic diagram showing an example of a behavioural fragment
- Fig. 20 is a flow chart of a method for performing behavioural fragment classification during the behavioural categorisation and abnormality determination process of Fig. 18;
- Fig. 21 is a flow chart of a method for determining the presence of a behavioural abnormality for an individual based on behavioural fragment classification, as performed by the abnormality recognition subsystem of Fig. 17;
- Fig. 22 is an example of performing behavioural abnormality determination with a particular implementation an anomaly detector of the abnormality recognition subsystem of Fig 17.
- Described herein is a system that records biometric data of a person and analyses it in respect of the person's own biometric history. Unlike existing systems which can only measure results against pre-set parameters, the system described herein collects data against a specific individual, builds up their own biometric profile, and uses it as the benchmark for analysis and abnormality detection. [35]
- the term "normal” used in this disclosure is intended to refer to a situation that is predefined as acceptable, for example, a value is within a pre-determined acceptable range.
- the system 100 includes at least one monitor 110, abnormality
- detection server 120 including at least one electronic processing device, and a biometric profile management system server 130.
- the monitor 110 includes one or more video cameras 112 for capturing video signals, and one or more thermal imaging devices 114 for capturing thermal imaging signals including a series of thermal images that are associated with the video signals captured by the video cameras 112.
- the video cameras 112 may be, for example, an RGB (red/green/blue) video camera.
- the thermal imaging devices 114 may be an infrared camera.
- the video cameras 112 and the thermal imaging devices 114 are in communication with the abnormality detection server 120 via one or more communication networks 140.
- the video signals and the thermal imaging signals acquired by the video cameras 112 and the thermal imaging devices 114 are sent, via the one or more communication networks 140, to the abnormality detection server 120 for processing.
- the abnormality detection server 120 uses the received video signals and the thermal imaging signals, detects at least one individual (e.g., a person) from the video signal, and detects at least one biometric characteristic of that individual.
- the abnormality detection server 120 generates biometric detection data representing the detected biometric characteristic.
- the biometric characteristic detected by the abnormality detection server 120 may include one or more physiological characteristics, for example any one or more of the following: a) body temperature; b) heart/pulse rate; c) breathing pattern; and d) pupil dilation.
- the biometric characteristic detected by the abnormality detection server 120 may alternatively include one or more behavioural characteristics, for example any one or more of the following: a) facial expression; b) body movement; and c) gesture.
- server 120 may include both physiological characteristics and behavioural characteristics.
- the biometric profile management system server 130 stores one or a plurality of biometric profiles. Each biometric profile includes information generated based on a specific person's biometric history, e.g., the biometric characteristic of that person detected previously.
- the biometric profile management system server 130 is in communication with the abnormality detection server 120, e.g., direct communication, or via one or more communication networks 150.
- the abnormality detection server 120 and the biometric profile management system server 130 may be integrated into a single device.
- the abnormality detection server 120 retrieves the biometric profile of the person from the biometric profile management system server 130, and identifies an abnormality of the individual based on the biometric detection data and the biometric profile of the person.
- the monitor 110 may include other data-capturing devices configured to generate data that represents a presence of the individual in an environment in which the system 100 is deployed.
- the data capturing devices may include biometric sensors that measure other biometric features of a person, for example, fingerprint, voice, face, predefined body motion, electrical capacitance tomography, and/or body weight.
- the monitor 110 further includes a face detector, which detects faces and sends the detection result to the abnormality detection server 120.
- the face detector may be integrated with the video cameras 112 and/or the thermal imaging devices 114 as a single device. Alternatively, the face detector may be a separated device.
- the detection server 120 may include a face detection module for detection of faces using video signals acquired by the video cameras 112.
- the face detection module may be a hardware module, or a software component within the video cameras 112.
- the face detection performed by the face detector or the abnormality detection server 120 may use known facial recognition techniques, wherein:
- the abnormality detection server 120 may further be in
- the third-party server 190 may be a school administration system, or an on-board purchasing system of a cruise ship.
- the method 200 includes: a) generating biometric detection data by detecting at least one biometric
- Step 210 identifying an abnormality of the individual based on the biometric detection data and a stored biometric profile of the individual (Step 220).
- the generation of the biometric detection data includes: a) receiving from the monitor 110: i) video signals; and ii) thermal imaging signals including a series of thermal images that are associated with the video signals; b) identifying the individual using the video signals; and c) generating the biometric detection data of the individual based on the thermal imaging signals.
- the monitor 110 including the video cameras 112 and the thermal imaging devices 114, may be installed in suitable places for monitoring individuals of interest, e.g., people entering a building, location or room, passengers boarding a ship, or students in a classroom.
- An exemplary process of identifying the individual and recording his/her thermal images includes:
- thermal images associated with a person are obtained, and can be used for detecting the biometric characteristic of that person.
- the biometric characteristics may include one or more physiological characteristic, such as body temperature, heart/pulse rate, breathing pattern, and/or pupil dilation.
- the biometric characteristic includes body temperature and heart rate, which can be detected based on the thermal images of a person.
- the abnormality detection server 120 generates biometric detection data based on the detection results.
- the biometric detection data may include values representing a person's body temperature, heart rate, and number of steps.
- the system may receive additional inputs from a wearable device 180 worn by the person being monitored.
- the wearable device 180 may provide the abnormality detection server 120 with additional information, e.g., values representing the person's heart rate and number of steps, associated with information representing the person's identity.
- the wearable device 180 is a wrist band that is wearable by individuals within the monitoring environment (e.g. students in a school, or workers in a factory, or residents in a nursing home), and that may provide functionality based on the implementation of the system in that environment (e.g. the ability to replace the seaPass card on a cruise ship).
- the wearable device 180 can be loaded with a software application which interfaces with the abnormality detection server 120.
- the abnormality detection server 120 then retrieves the biometric profile of the
- the biometric profile includes information representing a person's individuality in biometric characteristics that may be different from others.
- a person's biometric detection data is not only referred to a standard, representing what is normal for all target individuals, but also what is normal for that particular person, as indicated by their biometric profile. The system only identifies an abnormality if the detection result is outside an acceptable range for that particular individual.
- the person's own biometric profile is also referred to when determining whether there is an abnormality. This may reduce the probability of false alert; thereby increasing the accuracy of abnormal detection.
- each biometric profile includes information generated based on a specific person's biometric history, e.g., previously detected biometric characteristic of that person.
- the biometric profile of a person is built and updated over time by the system using the detected biometric characteristic of the person.
- the biometric profile may include biometric deviation data, representing a deviation of the biometric characteristic of the individual from the biometric characteristic of a group of individuals.
- the biometric reference data may be obtained by: a) detecting the biometric characteristic of a plurality of individuals; and b) averaging the detected biometric characteristic of the plurality of individuals.
- the group of individuals may include all students in a classroom.
- the biometric characteristic to be detected is body temperature
- the system may detect the body temperature of each student in the classroom, and calculate an average body temperature, which can subsequently be used as the biometric reference data. This may reduce the effect of ambient temperature variations due to seasonal changes or air- conditioning, and thereby increase the accuracy of the detection of abnormal body temperature.
- the abnormality detection server 120 may use the biometric reference data to determine whether the abnormality is abnormality.
- the relative biometric characteristic value may then be prepared with a predetermined threshold. If the relative biometric characteristic value exceeds the predetermined threshold, the abnormality detection server 120 may determine that an abnormality exists.
- the condition for an abnormality can be defined as:
- statical measures of the body temperature of the class may be used in addition to or instead of the average body temperature to indicate the central tendency.
- the statical measures may include, for example, medians or modes of the body temperatures of the class.
- the detected body temperatures may be checked by the system so that noise or poor readings may be filtered out. This may be performed, e.g., by comparing the detected body temperature value with a predetermined range of body temperature values.
- the biometric detection data may include a plurality of body temperature values, each representing the temperature of a predetermined body part.
- the body temperatures values may include, for example, a head temperature value and a hand temperature value.
- the difference between head temperature and other parts of the body may be referred to as the "relative body temperature”.
- the difference between the temperatures of the head and other parts of the body may be different when experiencing a fever compared to strenuous physical activity.
- condition for an abnormality When using the relative body temperature, the condition for an abnormality can be defined as:
- the detection of relative body temperature may be used by the system as the standard for abnormality detection.
- biometric characteristics e.g., heart rate of the person
- condition for an abnormality can be defined as:
- This may be used in conjunction with or separate from the detection based on a single body temperature according to Formula A, and/or the detection based on multiple temperatures according to Formula B.
- a person's biometric profile is built and updated over time, i.e., the abnormality detection server 120 updates the biometric profile stored in the biometric profile management system server 130 based on the biometric detection data.
- the server 120 sends the biometric data to the biometric profile management system server 130 to update the stored biometric profile based on the biometric detection data.
- EWMA exponentially weighted moving average
- NewAvg a*X + (1 - a)*01dAvg ("Formula E") where a is a weighting parameter between 0 and 1 that represents the degree of weighting decrease, and X is the new value.
- the parameter a may change depending on one or more predetermined factors, for example:
- the parameter a may be given a greater value when the biometric
- detection data is obtained with a higher level of confidence of facial recognition or higher level of confidence in skeletal tracking, and may be given a lesser value when a longer time has passed since last update, so that older values which are more likely to be out of date are given lower weights.
- biometric characteristics may have different values of parameter a.
- the abnormality detection server 120 may receive a user input indicating whether the result of the abnormality detection is correct. Accordingly, the abnormality detection server 120 may use this user input to update the biometric profile. [91] For example, if the abnormality detection server 120 detects a false abnormality, a user may make an input indicating that the result is a false abnormality. The abnormality detection server 120 may, for example, update the biometric detection data stored in the biometric profile with a decreased weight, for instance by giving the parameter a of Formula E a smaller value, e.g., 0.
- the abnormality detection server 120 may update the biometric detection data stored in the biometric profile with an increased weight, e.g., by giving the parameter a of Formula E a greater value.
- the system may be trained by receiving feedback data indicating the health condition of the individuals being monitored. For example, in a training period, each individual being monitored may be requested to provide feedback indicating their health condition, which may be input into the system for updating their biometric profile.
- the feedback may be automatically obtained by retrieving information from a third-party system or database, e.g., a medical system having health information of the monitored individuals.
- the feedbacks may be provided by having the individuals make a gesture (e.g., thumb up) or use other body language to inform the system whether they are healthy and feeling well.
- the gesture or body language may be captured by the monitor, and detected and recognized by the system.
- each biometric profile may include a person's:
- the exemplary working flow includes:
- Step 302 detecting body temperature of all people in a thermal image frame and calculating an average body temperature
- Step 304 for each person in the thermal image frame, calculating his/her biometric pattern
- Step 306 retrieving the stored biometric profile of that person
- Step 308 comparing the person's calculated biometric pattern with the biometric pattern in the biometric profile
- Step 310 checking whether the comparison result is within the tolerance range
- Step 312 if the comparison result is within the tolerance range, updating the biometric pattern in the biometric profile based on the calculated biometric pattern and the stored biometric pattern;
- Step 3114 if the comparison result is outside the tolerance range, flagging the anomaly;
- Step 316 checking whether a user input indicating a false positive has been received.
- Step 3128 if the user input indicating that the flagged anomaly is a false alarm, updating the stored biometric pattern with a decreased weight.
- Biometric characteristics of a person may vary over time in a day or in different
- the biometric profile may include a plurality of biometric patterns, each biometric pattern including: a) information representing a person's individuality in biometric characteristics that may be different from others; and b) time data indicating a time when the biometric characteristics are detected.
- the time data may include, for example, an absolute time such as a clock time.
- the time data may include a relative time, e.g., whether these biometric characteristics are detected: before a predetermined activity, during a predetermined activity, or after a predetermined activity.
- the abnormality detection server 120 when receiving the video signals and the thermal imaging signals from the monitor 110, the abnormality detection server 120 also obtains detection time information when these signals are acquired by the monitor 110, and generates detection time data accordingly.
- the detection time data can be included in the biometric detection data.
- the abnormality detection server 120 selects one biometric pattern from the plurality of biometric patterns that has a time data
- the abnormality detection server 120 may calculate a biometric pattern based on the plurality of biometric patterns that have time data, by giving different biometric patterns different weights, wherein the biometric pattern having a time data corresponding to the detection time data has a or the highest weight.
- biometric patterns may be used for modifying the biometric pattern used for abnormality detection.
- season, weather, temperature, the individual's activity prior to detection which may be obtained, e.g., by utilising a school timetable
- a table may be stored and maintained by the system, indicating how different additional factors (e.g., time, season, weather, temperature, and/or previous activity) affects each biometric characteristic, e.g., by indicating how much each biometric characteristic in a biometric pattern should be modified based on each factor.
- the biometric characteristics detected by the system may include: the body temperature, the temperatures of different body parts, and the heart rate; and each biometric pattern in the biometric profile may include:
- time data (e.g., shortly after physical activity, shortly after a meal, other time).
- each person's biometric profile may include three of such biometric patterns: a normal pattern, an after-meal pattern and an after-sport pattern.
- each person's biometric profile may include a plurality of biometric patterns, each being associated with a combination of multiple additional factors (e.g., time, season, weather, temperature, and/or previous activity), wherein the system compares the conditions at the time of detection to select a biometric pattern that best matches the conditions, or to calculate a biometric pattern based on the similarities between the conditions at the time of detection and the additional factors associated with each biometric pattern, e.g., a biometric pattern having factors more similar to the conditions at the time of detection is given a higher weight.
- additional factors e.g., time, season, weather, temperature, and/or previous activity
- biometric patterns included in the biometric profile may vary in different applications.
- any significant deviation from the pattern of any or all characteristics may indicate a change in health or other personal condition.
- the biometric profile of an individual may include one or more behavioural characteristics, for example any one or more of the following: a) predetermined body movements; b) predetermined gestures; and c) predetermined facial expressions.
- the predetermined body movements may include, for example, one or more of the following: a) walking; b) running; and c) falling down.
- the predetermined gestures may include, for example, one or more of the following: a) scratching; b) hitting/striking; c) nodding; d) hand raising; e) waving; f) thumbs up; and g) thumbs down.
- the predetermined facial expressions may include, for example, one or more of the following: a) smiling; b) frowning; c) eyes closed; and d) crying.
- the system may detect these behavioural characteristics using movement detection, gesture detection, and/or expression detection techniques.
- the system may allow a user to select which behavioural characteristics are to be
- detected may allow the user to define other gestures that are to be detected.
- a user may define that striking and hitting are abnormal behaviours in a classroom, and the detection of striking or hitting leads to an alert to be triggered by the system.
- the biometric profile of each person may include information representing a person's individuality in the selected behavioural characters that may be different from others.
- the biometric characteristics detected by the system include both physiological characteristics and behavioural characteristics. Accordingly, the system may identify in parallel abnormalities based on physiological characteristics and behavioural characteristics. Alternatively, the detected physiological characteristics and behavioural characteristics may be used in conjunction by the system to detect the abnormality.
- the system may detect that a person is doing a strenuous physical activity, such as running. The system may then determine that, if an increase in body temperature is detected after that activity, it should not be taken as an abnormality.
- the stored biometric profile of a person may further include facial features of the person, which can be used for facial recognition.
- the abnormality detection server 120 may perform a facial recognition process in detecting the biometric characteristics of a person, so as to find the corresponding biometric profile.
- An exemplary facial recognition process may include: a) for each stored frame, conducting facial recognition; and b) if the facial recognition results in a match to facial features in an existing
- biometric profile associating the detected biometric characteristics with that biometric profile.
- the matching result may be checked or reviewed by a human staff. For example, a plurality of identified photos may be displayed in one page on a screen to allow the human staff to skim through to efficiently confirm whether the recognition results are correct, and to identify any incorrect result.
- the corresponding biometric profile can then be used for the detection of whether an abnormality exists.
- the detected biometric characteristics may further be used for updating this biometric profile, e.g., using an exponentially weighted moving average.
- the system may create a new biometric profile based on the detected biometric characteristics, and store the facial features in this biometric profile.
- the faces that are not matched may be checked by a human to decide whether a new biometric profile needs to be created.
- the facial features may be stored in a separate document, with each set of facial features associated with a unique identifier, whereby each biometric profile is also associated with one of the unique identifiers, to allow a corresponding biometric profile to be retrieved based on the facial recognition result.
- the result of the facial recognition may further be used for
- an exemplary attendance recording process may include:
- the system can be used for embarkation/disembarkation control, and/or authority control.
- the system may trigger a notification, e.g., an alert or alarm, to notify the relevant staff.
- a notification e.g., an alert or alarm
- an audio alarm or a visual alert may be triggered if a student in a
- a notification message may be sent to a relevant teacher or a doctor of the school, to allow further examination of the student.
- the system may allow the user to configure the notification of abnormality, e.g., by setting different notifications for different levels of severity.
- the abnormality detection server 120 may further be in communication with at least one third-party server 190, via one or more communication networks 141. Communication can occur via a direct link between the third-party server 190 and the abnormality detection server 120, or via an
- the analysis or detection results generated by the abnormality detection server 120 can be sent to the third-party server 190 for further use.
- the third-party server 190 may be a school administration system, to which the detection results of potential health issues or abnormal behaviours of students may be sent for school administration purpose.
- the result of abnormal detection may be used for controlling authorization of a predetermined activity.
- the predetermined activity may include, for example, purchasing alcohol, driving a vehicle, or entering a specified area.
- the system may generate a notification or alert to notify the bar tender, or to reject a purchase transaction in a payment system that is associated with that customer.
- the system may not authorise the passenger to pass an automatic check-in gate.
- the communications networks 140, 150, and 141 providing communication between the monitor 110, the abnormality detection server 120, the biometric profile management system server 130, and the third-party server 190 may take any appropriate form of wired or wireless networks, including but not limited to mobile networks, private networks, such as an 802.11 network, the Internet, local area networks (LANs), WANs, as well as via direct or point-to-point connections, such as Bluetooth or near field communication (NFC) connections.
- the communications networks 140, 150 and 141 may include different networks. Alternatively, the
- communications networks 140, 150 and 141 may be a single network that provides onward connectivity between different components of system 100.
- FIG. 4 An exemplary structure of the abnormality detection server 120 is shown in Fig. 4.
- the abnormality detection server 120 includes at least one processor 401, a memory 402, and an external input/output interface 403, interconnected via a bus 404.
- the abnormality detection server 120 may additionally include an input/output device 405, such as a keyboard and/or a display.
- the I/O interface 403 can connect the abnormality detection server 120 to peripheral devices and/or networks, including other servers, devices, and local or remote databases.
- the processor 401 executes machine-readable instructions stored in the memory 402 to perform at least part of the process described hereinbefore for
- the machine-readable instructions may include one or more software components 410 (as described in further detail hereinafter), and may be executed in a suitable execution environment, such as an operating system environment.
- the abnormality detection server 120 may be formed from any suitable processing system, such as a suitably programmed computer system, PC, web server, or network server.
- the machine-readable instructions can be embodied in non-transitory computer-readable storage media, e.g., a hard drive.
- the software components 410 may include: a) a data acquisition component (DAC) 502; b) a data analysis and pattern recognition components (DAPRC) 504; c) a database interface components (DIC) 506; d) an abnormality notification component (ANC) 508; and e) a feedback collation component (FCC) 510.
- DAC data acquisition component
- DAPRC data analysis and pattern recognition components
- DIC database interface components
- ANC abnormality notification component
- FCC feedback collation component
- the DAC 502 controls the collection of data received from the monitor 110 via the I/O interface 403 of the abnormality detection server 120.
- the software components 410 may allow adding new data acquisition components to adapt to additional input signals, for example, adding inputs from additional sensors.
- the collected data is sent from the DAC 502 to the DAPRC 504 for analysis, and for recognition of biometric characteristics and patterns, for example: a) recognise a person in the video data; b) recognise a known or unknown person in the video data; c) detect biometric characteristic of a person; d) detect biometric characteristic of each person in a group; and e) calculate an average biometric measurement for the group;
- the DAPRC 504 accesses the database in the biometric profile management system server 130 through the DIC 506, to build, retrieve and maintain the biometric profiles. Based on the data collected from the DAC 502 and the biometric profile retrieved using the DIC 506, the DAPRC determines whether an abnormality exists.
- the DIC 506 may also provide access to other databases, for example, a database for maintaining records of the abnormal detection history, and/or a database for storing the received raw data for a predetermined period of time.
- the DAPRC 504 reports the abnormality to the ANC 508, which may trigger an alert or alarm.
- the ANC 508 communicates the abnormality or other detection results (which may include predefined "events") reported by DAPRC 504 to additional event subscribers, that will respond to those detection results or events.
- the FCC 510 monitors whether a feedback in relation to the detection result has been received. If a user input indicating a false detection is received, the FCC 510 may report this to the DAPRC 504, which may then update the biometric profile stored in the database accordingly, via the DIC 506.
- the DAPRC 504 receives data from DAC 502; b) analysis of received data and detecting biometric characteristics; c) retrieving biometric profiles via DIC 506; d) identifying abnormalities; e) reporting abnormalities to ANC 508; and f) receiving feedback from ANC 508 and updating biometric profiles via DIC 506 accordingly.
- the abnormality detection is performed in real time, and the result may be sent in real time through a network (e.g., the internet or any other suitable type of network).
- the abnormality detection is performed in a decoupled mode, whereby data are not received in real time.
- the DAPRC 504 may maintain an internal or external log of timespans and data segments that have been parsed and processed, wherein the data segments may be subtyped by the type of biometric characteristic. This may be used for post-processing of data.
- the internal or external log may be maintained via a log maintenance component (LMC) 512.
- LMC log maintenance component
- FIG. 6 An exemplary structure of the monitor 110 is shown in Fig. 6.
- the monitor 110 includes at least one processor 601, a memory 602 such as a flash memory, an I/O interface 603, an infrared light source 604, the video cameras 112, and the thermal imaging devices 114 in the form of infrared cameras.
- the monitor may additionally include an input/output device 605, such as a display, one or more LED lights, and/or one or more buttons.
- the input/output device 605 may further include one or more audio input/output devices.
- the I/O interface 603 can connect the monitor 110 to peripheral devices and/or networks, such as sending the video signals captured by the video cameras 112 and the thermal imaging signals captured by the infrared camera 114 to the abnormality detection server 120.
- a single I/O interface 603 is shown, this is for the purpose of example only, and in practice multiple interfaces using different communication protocols (e.g., Ethernet, serial, USB, HDMI, wireless) may be provided.
- the processor 601 executes machine-readable instructions stored in the
- the memory 602 to control the infrared light source 604 to emit infrared radiation, and control the video cameras 112 and the infrared camera 114 to capture video signals and thermal imaging signals respectively.
- the processor 601 may perform pre-processing to the captured signals before sending them to external devices through the I/O interface 603. Alternatively, the signals may be sent as raw data by the processor 601 to external devices through the I/O interface 603.
- the monitor 110 may include other sensors in addition to the video cameras 112 and the thermal imaging devices 114, e.g., additional biometric sensors that measure other biometric features of a person, such as: fingerprint, voice, face, predefined body motion, electrical capacitance tomography, and/or body weight.
- additional biometric sensors that measure other biometric features of a person, such as: fingerprint, voice, face, predefined body motion, electrical capacitance tomography, and/or body weight.
- At least one non-transitory computer-readable storage media having computer-executable instructions embodied thereon for abnormality detection, wherein when executed by at least one electronic processing device, the computer-executable instructions cause the electronic processing device to perform: a) generating biometric detection data using detection signals generated by at least one monitor, the biometric detection data representing at least one biometric characteristic of an individual; and b) identifying an abnormality of the individual based on the biometric detection data and a stored biometric profile of the individual.
- an elevated body temperature may be an indication that an individual has an illness. However, it may also be detected in a healthy individual who is, or has recently, engaged in vigorous physical activity (e.g., by participating in a gym).
- Accurate abnormality detection requires context- specific behavioural and physical modelling in order to distinguish between a deviation in physical and/or behavioural characteristics that are representative of an underlying abnormality of the individual, and deviations that are a consequence of the individual's interaction with, or existence within, the learning environment. This is particularly significant in schools since the individuals being monitored typically engage in behaviours that are not commonly exhibited outside of the learning environment. For example, individuals may exhibit gestures, such as hand raising, and/or particular facial expressions based on their participation in class activities. When deployed within the school environment, conventional abnormality detection methods and systems may interpret these behaviours erroneously.
- the embodiments described herein include an abnormality detection system (ADS) and process for detecting abnormalities affecting individuals within a school, or a similar type of learning environment.
- the abnormalities detected can be physical, such as the presence of an illness or physical condition, or psychological such as, for example, ADHD or anxiety.
- Embodiments of the abnormality detection system include a monitoring apparatus (also referred to as a "monitor"), with one or more imaging devices configured to generate imaging data representing video and thermal images of a school environment, such as a classroom or play-area, that is frequented by the individuals to be monitored (referred to herein as a "monitoring area").
- the imaging data is transferred over a communications network to a server device configured to execute an abnormality detection application (ADA) which receives and processes the imaging data to determine whether there is an abnormality associated with the individual.
- ADA abnormality detection application
- Abnormality detection is performed based on biometric information of the individual, including a physical profile and a behavioural profile of the individual, and on contextual information in relation to the individual and/or learning environment at the time of the abnormality assessment.
- the abnormality detection system (ADS) described herein includes an abnormality detection server device (i.e. a detection server) that is configured to interface with one or more devices, including a biometric profile management system (BPMS) device and individual computing workstations deployed throughout the school (referred to herein as "terminals").
- BPMS biometric profile management system
- the detection server is configured to generate abnormality alert data representing an alert message in respect of the abnormality, and to transmit the alert to the BPMS.
- the alert message informs the person of the existence of the abnormality.
- the monitoring apparatus includes three imaging
- particular physical characteristics such as a thermal signature or heart rate
- behavioural characteristics such as gestures or facial expressions
- the monitoring apparatus is in the form of a camera array which is configured to generate imaging signals representing images of individuals within the monitoring area.
- Other embodiments of the ADS can include multiple monitoring apparatuses, for example where each is deployed within a different classroom (i.e., monitoring area), in order to extend the abnormality detection capabilities of the system.
- the cameras of each monitoring apparatus can be physically separated and can operate independently from one another for the purpose of generating the imaging signals.
- the cameras are part of an integrated image detection hardware device, such as for example, a Kinect, Flir, Face++, Verilook or Oxehealth device.
- the image detection hardware device can include individual cameras which are specifically configured for gesture-based human-computer interaction, such as the Intel RealSense 3D cameras or their consumer variants (e.g. the Razer Stargazer and the Creative BlasterX Senz3D).
- the physical profile of each student includes a
- the temperature profile which represents the nominal temperature characteristics of the student.
- Thermal signature data is detected for a student, and is compared to their temperature profile data in order to determine the presence of a potential abnormality associated with a raised or lowered temperature.
- the comparison also involves the use of context data, as obtained from the BPMS or other data source.
- the context data can include an indication of the types of learning activities being undertaken by the individuals at the location, or a state representing the highest degree of physical activity which the individual has engaged in within a particular time period (e.g. within the past hour).
- This context data can be used in conjunction with biometric information of one or more individuals within the learning environment. For example, an increase in detected temperature values may not necessarily be considered to represent an abnormality if the students are, or have recently been, participating in a particular class where physical exertion is required (such as a gym class).
- the physical profile also includes a heart rate profile for the student representing their nominal heart rate levels.
- the heart rate and temperature profiles can include multiple models corresponding to different contextual states.
- the heart rate profile can include distinct models for: i) a "resting state" which represents the heart rate of the student when they have not undertaken vigorous physical activity (e.g., when sitting at a desk); and ii) a "high activity” state which represents the heart rate of the student when they are engaged in physical activity (e.g., during gym class).
- the behavioural profile of each student contains data representing the tendency of the student to engage in particular behaviours, including exhibiting particular gestures and facial expressions.
- the detection server is configured to detect gestures and facial expressions that are specific to the school environment (e.g., hand raising), and other gestures and expressions (e.g., frowning, eye movements, etc.).
- the detected gestures can include "user-defined" gestures that are customised and/or configured specifically for the system (as described below).
- the detection of a behavioural abnormality involves: i) the detection of a gesture and/or facial expression exhibited by the student; and ii) matching the occurrence of the gesture and/or facial expression to the behavioural profile of the student.
- the ADS is configured to utilise context data in determining the existence of an abnormality based on detected gestures and/or facial expressions and the behavioural profile.
- a deviation in the level of class participation of the student e.g. as measured by the frequency with which the student exhibits particular behaviours, such as hand-raising, within a time period
- the level indicated as normal in their behavioural profile may indicate an abnormality or extraneous issue, unless there is a contextual basis for the lack of participation (such as when the class is currently sitting a test or exam).
- context data includes environmental attributes
- Context data can be determined based on learning-environment specific data provided to the abnormality detection server from the BPMS, either automatically or on request. This ensures that the context for interpreting detected physical and behavioural characteristics remains accurate over time.
- context data may be determined by the abnormality detection server during an initial configuration or setup phase, such as in implementations where no significant changes to this information are expected over time.
- the biometric data and the context data are stored within a data storage device of the ADS, such as a profile database.
- the ADS is configured to dynamically update the biometric data, including the physical and behavioural profile data, of each student based on monitoring and abnormality detection activities performed over a period of time.
- the system can be configured to perform updates to the profile data selectively based on the result of the biometric data collection process and the comparison of the collected data with the student's existing profile data. That is, updates to the profile data of a student can be made in respect of collected physical and/or behavioural data that is considered to be normal for the student, or that is, at least, within a particular tolerance range relative to those nominal values.
- the system can be configured to marginalise the physical detection data representing this value during the update process (i.e., the detected heart rate values will not be used to update the student's physical profile in this case).
- the detection server is configured to generate update report data reporting the physical and/or behavioural characteristics of one or more students to the BPMS.
- the report information can be viewed at a terminal by the appropriate personnel.
- the detection server can be configured to transmit additional data to the BPMS, including data representing the attendance of individual students, and/or particular critical behaviours or physical characteristics detected for the one or more students (as described herein).
- the data is transmitted in real-time and is processed by the BPMS, such as, for example, to manage attendance records for particular classes.
- the attendance data transmitted to the BPMS include an identifier of a student detected to be entering or leaving the monitored location (e.g., a unique student ID value), and an indication of the time when the student enters or leaves the location.
- the ADS can be configured to maintain a set of critical physical characteristics and/or behaviours that, when detected with respect to a student, cause the detection server to transmit a critical event message to the BPMS.
- the ADS can be configured to provide critical event messages to one or more terminals of the school directly.
- the critical characteristics may be physical or behavioural traits that warrant
- the critical characteristics and behaviours can be defined globally (i.e., those characteristics and behaviours that are considered critical for each student monitored), and individually where the critical characteristics and behaviours may differ for each student.
- the presence of particular physical characteristics may trigger a critical event notification (or "alert") for a student with a medical condition, but not for other students.
- the ADS can also be configured to generate a critical event alert if a student exhibits a particular behavioural characteristic (e.g., striking or hitting another student).
- a platform for detecting abnormalities affecting school students which: a) monitors students within a pre-determined location using a monitoring apparatus consisting of imaging devices configured to generate video and thermal signals, and where the imaging data generated from the signals allows the identification of particular students and the capture of their physical and behavioural
- b) maintains physical and behavioural profile information for each student, where the information from each profile is used to determine the existence of an abnormality affecting a student when compared with detected physical and behavioural patterns; c) interfaces with existing biometric profile management systems to provide an
- FIG. 7 illustrates an exemplary implementation of the ADS 102, including a monitor 110 with camera devices 111, 113 ... UN, an abnormality detection server 120 (also referred to as a "detection server 120" to avoid confusion with the ADS 102), a biometric profile management system server 130 and a corresponding profile database 131, and workstation terminals 151 and 152.
- Cameras 111, 113 ... UN include at least one video camera for capturing video signals, and a thermal imaging device for capturing thermal imaging signals, including a series of thermal images that are associated with the video signals, as described hereinbefore.
- Monitor 110 is deployed such as to generate video signals representing a video feed that covers a predetermined monitoring area within the learning environment (e.g., the inside of a classroom, or a strategic point on the premises).
- Monitor 110 can be configured specifically for the learning environment, such as to maximise the field of view of the video cameras over the monitoring area.
- the monitor 110 is placed in a position at the front of a classroom (the monitoring area being the inside of the classroom) allowing the feed that is generated to cover every person in the room, including the teacher.
- monitor 110 is a single standalone device, where cameras 111, 113 ...
- UN include at least three cameras that are configured as follows: a) Camera 1 - generates video signals at a frame rate and resolution enabling the detection server 120 to detect individuals (e.g., students), and to employ facial recognition technology to identify students entering and leaving the monitoring area; and b) Cameras 2 and 3 - generate video signals at a frame rate and resolution enabling the detection server 120 to measure the biometric (i.e., physical and behavioural) characteristics of the students in the monitoring area. Specifically, camera 2 operates to obtain vital medical information on each student, including body temperatures (head and torso) and heart rate. Camera 3 captures movements on an ongoing basis, and is configured to generate video signals which allow the detection of behaviours, such as gestures, motion, or holding unknown objects (such as a weapon).
- biometric i.e., physical and behavioural
- Monitor 110 transmits imaging signals to the detection server 120 via a
- Detection server 120 is a server device of the kind described hereinbefore, and with the software components 410 illustrated in Fig. 5.
- a Data Acquisition Component (DAC) receives imaging signals from the monitor 110 and produces corresponding imaging data.
- the imaging data generated is in the form of a data stream indicating a position (x,y) and a value specific to the signal generated by the particular camera.
- the data generated for camera 1 can include the coordinates of each pixel (x,y) along with a colour value, while for camera 2 the data can include the co-ordinates (x,y) and a corresponding temperature value.
- a mapping operation is performed by the DAC to resolve the data values generated for each of the cameras 111, 113, ...
- the DAC would resolve each set of (x,y) co-ordinates obtained from camera 1 to the corresponding (x,y) values obtained from camera 2.
- the DAC also maintains physical and behavioural profile data for each student, and learning-environment specific data, as received from the BPMS server 130 (as described hereinafter).
- a Data Analysis and Pattern Recognition Component uses the raw data obtained by the DAC to compare detected data with corresponding profile data, and to determine the existence of an abnormality based on this comparison and on context data representing a context of the detected characteristics.
- the DAPRC maintains an internal log of timespans and data segments that have been parsed and processed, sub-typed by pattern type.
- monitoring area includes: i) Motion detected by monitor 110; ii) DAC receives a video stream signal from camera 111; iii) DAC processes the received video stream signal to produce video data; iv) DAC receives an infrared stream signal from cameras 112; v) DAC processes the infrared stream signal to produce infrared data; vi) DAC generates a data acquisition event and transmits the event to the
- DAPRC DAPRC
- DAPRC receives the data acquisition event
- DAPRC performs facial recognition using the video data
- DAPRC recognises a face and raises subject identified event to the biometric profile management system server 130
- DAPRC receives identification request for time segment
- DAPRC performs identification, and if recognition is over 94%, the DAPRC raises a subject-identified event to the biometric profile management system server 130, and additionally to any other systems that are configured to utilise the recognition information such as school administration systems or medical systems
- xii) DAPRC performs identification, and if recognition is under 46%, the DAPRC raises a manual-identification-required event to the biometric profile management system server 130
- xiii) DAPRC receives a Face Manually Identified event from another
- Case 2 processing the thermal signature of an individual: i) Motion detected by monitor 110; ii) DAC receives an infrared stream signal from cameras 112; iii) DAC processes the infrared stream signal to produce infrared data; iv) DAC generates a data acquisition event and transmits the event to the
- Biometric profile management system (BPMS) server 130 is a system that allows school management applications to interact with it through Schools Interoperability Framework (SIF) or web API, such as a school administration system (SAS) application.
- SIF Schools Interoperability Framework
- SAS school administration system
- the BPMS server 130 includes: a profile management component 132 configured to manage data associated with students (such as biometric data) and other individuals associated with the school (e.g., parents and staff); an attendance component 134 configured to manage the attendance of students with respect to the monitoring area; and a critical event component configured to store data relating to critical event detection and notification.
- a profile management component 132 configured to manage data associated with students (such as biometric data) and other individuals associated with the school (e.g., parents and staff)
- an attendance component 134 configured to manage the attendance of students with respect to the monitoring area
- a critical event component configured to store data relating to critical event detection and notification.
- the detection server 120 is configured to communicate with the biometric profile management system server 130 and terminals 151, 152 via communications network 141.
- the communications networks 140 and 141 can be local or wide area networks (as described hereinbefore), or a combination of a plurality of different local or wide area sub-networks.
- detection server 120 communicates with monitor 110, the biometric profile management system server 130 and terminals 151, 152 via a single communications network, or through direct connections between the respective devices, mitigating the need for data transmission over a communications network.
- Profile management component 132 is configured to communicate with a profile database 131.
- the profile database 131 stores data within one or more tables, or similar data structures, which are accessed via a database management system (DBMS) of the profile management component 132.
- DBMS database management system
- the DBMS uses SQL language to query the database 131, which is implemented as an object-oriented database in the described embodiments.
- Data stored within the profile database 131 includes: a) Identity data representing the identity of individuals associated with the school, including students, parents and staff, including data representing the name, address, phone number, student/staff ID number, and facial characteristics (i.e., parameters of respective facial feature models for each individual); b) Biometric data representing the physical and behavioural profiles of students, or other individuals, for whom abnormalities are to be detected by the detection server 120; c) Logging data representing a record of the arrival and/or departure of students relative to particular monitoring areas, expressed as a "logging event" (e.g., including an indication of the monitoring area, an identifier of the logged student, an indication of whether the logging event is an arrival or departure, and a date- time value representing the time of the event); and d) Learning environment specific data, representing properties of the school that may be interpreted to determine the context of particular detected physical or behavioural characteristics (e.g., timetable data providing an indication of the class or session being undertaken in the monitoring area at particular times throughout the day).
- logging event
- the detection server 120 is configured to: transmit, to the BPMS server 130, an
- the profile management component 132 is configured to retrieve data from the
- the profile management component 132 is configured to transmit particular data to the detection server 120, including identity, biometric and learning-environment specific data, during an initial system configuration phase.
- student management data is stored within the profile management component 132 of the BPMS server 130.
- student management data is not stored locally within the BPMS server 130, and the profile management component 132 operates as a pass-through to allow the ADS 120 to access and update the database 131.
- the profile management component 132 can also be configured to access the student management data from the database 131, and to transmit the data to the detection server 120 dynamically at predetermined times (such as during the configuration phase), or in response to an update request by the server 120.
- Reporting data received by the BPMS server 130 is processed to determine
- a group of students may attend an external venue for part of the school day or a student may have to be excused from the classroom to attend a medical appointment.
- Arrival or departure information within the reporting data is transmitted from the profile management component 132 to the attendance component 134.
- Attendance component 134 generates corresponding logging event data for storage in the profile database 131, and can transmit the logging data to one or more of the terminals 151, 152 in real-time if required such that the school staff are informed of exactly who is on the premises (e.g., in the case an emergency, etc.);
- Profile management component 132 can be configured to generate attendance report data based on the logging information received within the reporting data. For example, in the case where the monitored area is a classroom, at the start of each period the teacher can select the appropriate class (from a timetable module in the SAS application, as executed within the profile management component 132). The profile management component 132 will generate the attendance report data in respect of the correct class. The attendance report can be generated in respect of a predetermined period (e.g., one full teaching day), and/or selected classes which were undertaken within the monitoring area; and
- An alert is determined when a critical physical characteristic and/or behaviour is detected with respect to a student.
- Data representing the detection of a critical physical characteristic and/or behaviour for a student (referred to as a "critical event") is transmitted to the critical event component, which is configured to generate corresponding critical event messages of a predetermined form (e.g., an email or Short Message Service text message).
- the profile management component 132 can be configured to record the occurrence of a critical event by generating corresponding critical event record data, and storing this data in the profile database 131. [191] Fig.
- FIG. 8 illustrates a process 800 executed by the detection server 120 for determining an abnormality of an individual within a learning environment, including: receiving imaging signals from the monitor 110; processing the imaging signals to generate imaging data; processing the imaging data to detect the individual, and to generate corresponding identity data representing the identity of the individual; and identifying an abnormality of the individual based on at least one of: a) physical detection data representing the detection of a physical characteristic of the individual from the imaging data, and a corresponding physical profile of the individual; and b) behavioural detection data representing the detection of a behavioural
- the detection server 120 is configured to execute an application which includes: the initialisation of ADS 102 system hardware devices; a monitoring step loop; a recognition step loop (run at a lower priority, or when the system is idle); an attendance step loop (triggered by a timer); a detection process step (triggered by a timer); and a stop/shut down sequence.
- the monitoring, recognition, attendance and detection steps are performed iteratively in order to conduct the abnormality detection process 800 illustrated in Fig. 8.
- the DAPRC of the abnormality server 120 receives imaging data
- the imaging data is generated from imaging signals produced by the monitor 110.
- the imaging data is processed to detect the presence of the individual, and to determine an identity of the individual.
- Cameras 111, 113, ... 1 IN are configured to use skeletal tracking to detect when a person enters the field of view and to follow them until they leave.
- the Microsoft Kinect skeletal detection and tracking algorithm is implemented as the skeletal tracking routine, and is configured to perform the detection of up to six users in the field of view of a camera sensor, and to track skeletal motion for up to two of those users.
- the imaging data include one or more frames of the respective video signals produced by the cameras 111, 113 ... 1 IN.
- the DAPRC For each of the one or more frames where the individual is in the field of view, the DAPRC is configured to check the individual's orientation to determine the best frames (e.g., unobstructed view, facing the device). For each of the best frames, the DAPRC extracts one or more features representing the detected biometric (i.e., physical or behavioural) characteristics as described hereinafter. Once the individual is out of the field of view, the DAPRC stores the frames (or relevant portions of the frames).
- Identification of the individual involves the execution of a facial recognition process on each stored frame. Facial recognition is performed by extracting a set of facial features from the stored frame data, and comparing the extracted features to facial model data representing the unique facial characteristics of each individual enrolled within the BPMS. Facial model data is included within the identity data of each individual, as maintained by the profile database 131 and transmitted to the detection server 120 (as described hereinbefore). The facial recognition process is performed by the detection server 120 using facial model data stored within the detection server 120 (as received from the BPMS server 130 or otherwise), in conjunction with imaging data generated for detected individuals, as described hereinbefore.
- the ADS can be configured to use both facial model data and thermal imaging data during the recognition process in order to distinguish between two individuals with similar facial characteristics (such as identical twins). If the extracted facial features and imaging data match to those of an individual (to a sufficiently high level of confidence, as described hereinbefore), then the DAPRC generates an indication of the identity of the individual based on the
- the identity of the individual is represented by their unique student (or staff) ID number.
- the detection server 120 detects physical and/or behavioural characteristics of the identified student by generating biometric detection data (as described hereinbefore). For example, at step 806 the body temperature and/or heart rate of each student within the monitoring area may be detected. A physical anomaly is determined if the detection data values do not match to the corresponding biometric profile data for the individual (as described hereinbefore).
- the physical detection data values represent physical characteristics including:
- FIG. 3 illustrates an exemplary process 300 by which
- Fig. 9 illustrates an exemplary process 900 for determining: 1) a behavioural characteristic (e.g., a gesture or expression) being exhibited by the individual; and 2) whether the exhibited characteristic constitutes an abnormality. Determining the exhibited behavioural characteristic includes: processing the behavioural detection data to determine that the behavioural characteristic is one of a behaviour type including: a gesture; and a facial expression; comparing the behavioural detection data to one or more behavioural characteristic models of the determined behaviour type; and selecting a particular behavioural characteristic based on a result of the comparison.
- a behavioural characteristic e.g., a gesture or expression
- the behaviour characteristics are specific to the
- the aforementioned gestures (i)-(vi) are pre-defined by the abnormality detection application executing on the server 120.
- the ADS 102 can be configured to detect behavioural characteristics corresponding to other gestures that are customised according to the particular implementation of the system (referred to as "user- defined” gestures). This allows the end users of the ADS 102 to define and store their own gestures, using a gesture modelling software application. For example, where students are prone to using their mobile phones during class, the gestures peculiar to that activity could be defined and stored enabling the system 102 to detect mobile phone use by students within the monitoring area.
- a "spinning" gesture can be defined as the spinning of an individual's forefinger in a circular motion.
- This spinning gesture can be interpreted as an indication that the individual desires the teacher to speak up (i.e. raise his or her voice), and allows the individual to express this desire without needing to raise their hand and wait for the teacher to ask them what they want.
- the gesture modelling software application creates the user-defined gestures using one or more known gesture creation algorithms, such as the 'User-Defined Gestures for Surface Computing' algorithm by Wobbrock et al., as published in the Proceedings of the 2009 SIGCHI Conference on Human Factors in Computing Systems.
- the DAPRC of the abnormality server 120 receives imaging data generated by the DAC (as described hereinbefore).
- the imaging data is generated from imaging signals produced by the monitor 110.
- the DAPRC determines whether a behavioural characteristic is detected over a set of the best frames of the imaging data, where the best frames are determined as described hereinbefore.
- Fig. 10 illustrates the process of determining the particular characteristic exhibited by the individual.
- the DAPRC isolates the individual within the frames of the imaging data and processes the frames to extract particular features that are relevant to the determination of behavioural characteristics. For example, in the case where the individual raises their hand, the features can include the trajectories of particular regions of the individual's hand.
- the features are extracted from particular regions of the image data which are identified during a pre-processing stage (e.g. the areas corresponding to the hands and face of an individual).
- the extracted features are analysed to determine the presence of a facial expression or gesture (or both) (i.e., at steps 1006 and 1012 respectively).
- expression and gesture recognition and tracking is performed using an integrated algorithm, such as for example by the method proposed by Pateraki et al. in 'An integrated approach for visual tracking of hands, faces and facial features' as published in Proceedings of the 2011 IEEE Workshop on Measuring and Understanding Human Movements and Emotions.
- a comparison is performed between the extracted features and one or more facial expression models of the individual (i.e., at step 1008).
- the facial expression models are represented by facial characteristic data of the corresponding identity data of the individual.
- the DAPRC selects the facial expression of the best matching model as a detected behavioural characteristic.
- the detection server 120 is configured to utilise individual specific gesture models. For example, each student may have their own corresponding set of trained gesture models to represent the behaviour of the student when performing the gesture (as described below).
- a set of global gesture models can be defined such that the determination of the gesture (or corresponding "action class" as described herein below) involves a classification process that is independent of the identity of the individual for whom the characteristic is being determined.
- each gesture model may be a global model based on a large scale training data set (e.g., a universal background model (UBM) using training data from all students).
- UBM universal background model
- the detection server 120 is configured to allow the detection of gesture and facial expression behaviours that are simultaneously exhibited by an individual. For example, a student may (i) raise their hand while (ii) simultaneously frowning, and the detection server 120 is configured to detect the two behaviours as distinct actions (i.e., at steps 1010 and 1016, as described hereinbefore).
- the DAPRC uses context data representing a context of the detected behavioural characteristic within the learning environment.
- the DAPRC is configured to process learning environment specific data, as received by the detection server 120 from the BPMS server 130, to determine context data relevant to the identification of an abnormality.
- the context data can include an indication of the class or session being undertaken in the monitoring area when the behavioural characteristic was detected.
- behavioural characteristic detection involves the generation of behavioural event data representing particular actions (such as gestures) and/or emotions (as determined, for example, from facial expressions) of an identified individual.
- Abnormality server 120 includes a behavioural event generation subsystem which processes data gathered from detection devices (e.g. cameras) of the monitor 110 to generate action and emotion events representing instances of specific gestures and facial expressions exhibited by individuals that are monitored by the system.
- the action and emotion event data generated by the behavioural event generation (BEG) subsystem is stored in an action and emotion database, which contains records representing instances of the action and emotion events that are observed by the system. Each event is associated with an individual who is identified by the BEG during the behavioural event determination process.
- the stored action and emotion events are processed to produce: i) action and event models (such as gesture models, and/or facial expression models, as described above); and ii) behavioural profile data representing the expected (or nominal) behaviour of an individual with respect to the history of actions and emotions that have been recorded for the individual over time.
- the action and emotion event models are utilised to perform action/emotion recognition (i.e.
- an action or emotion such as a gesture or facial expression
- the behavioural profile is used to perform abnormality recognition (i.e. to determine whether an abnormality exists when an action or emotion is observed for the individual).
- the production and/or updating of the models and/or behavioural profiles can be performed during a dedicated training phase of the system.
- the training process can be performed offline during a training mode, in which the ADS 102 is configured to passively monitor individuals in the learning environment for the purpose of training action, event or behavioural models (i.e. rather than performing abnormality detection).
- the ADS 102 can update the action and event models, and the behavioural profile, of individuals dynamically during operation of the system based on the event data produced by the BEG subsystem.
- the BEG subsystem is invoked by the ADS 102 to process the video imaging data received from the monitor 110 (i.e. at step 802) by performing object and detection tracking procedures (as described below) to: identify the individual (i.e. steps 804); and generate one or more action or emotion events representing the observed behaviour of the individual (i.e. during the determination of the behavioural characteristic at step 904).
- an abnormality detector subsystem is invoked to determine the existence of an abnormality (i.e. at step 810) by processing the determined action and/or emotion events in conjunction with the context information and the individual's behavioural profile (as generated and retrieved at steps 906 and 908 respectively).
- Fig. 11 illustrates an implementation of a BEG subsystem 160 according to the
- the BEG subsystem 160 is implemented within the detection server 120, and includes: a data capture unit 1100 which receives input from the monitor 110; a video and metadata storage unit 1102; object detector 1106 and object tracker 1108 units; a frame detection storage unit 1104; an annotated tubelet storage unit 1110; an association and fusion unit 1112 which produces fused tubelet data 1114; a recogniser unit 1122 which produces fused, identified and classified tubelet data 1116; a conversion unit 1118; and an individualised action and emotion database 1120 for storing action and emotion event data generated by the behavioural event generation subsystem 160.
- Fig. 12 shows the process 1200 of operation of the described BEG subsystem 160, which involves: receiving imaging data in the form of a video signal from the monitor 110 (at step 1202); performing object detection and tracking on the received video signal (at step 1204); identifying the individual and the associated actions and/or emotions being performed or expressed by them (at steps 1206, 1208 and 1210); generating behavioural event data in the form of action and/or emotion events (at step 1212); and storing the generated behavioural event data in the action and emotion database 1120 (at step 1214).
- the data capture unit 1100 interacts with a number of detection devices of the monitor 110 to receive video imaging data.
- the detection devices are cameras 111 - 1 IN.
- the number of cameras N deployed within the monitor 110 can vary based on the size and layout of the monitoring environment, and the pixel resolution of individual cameras.
- additional types of optical sensors such as thermal and multispectral cameras may be added to improve the range and orthogonality of data available for analysis and decision making.
- all cameras share a common trigger signal which is used to enable synchronized image capture. This facilitates the fusion of data received from the cameras 111-1 IN when these cameras are deployed at separate locations (i.e. during the viewpoint fusion process performed by the conversion unit 1118).
- the data capture unit 1100 performs frame based processing of the received imaging data. For each captured image frame, the data capture unit 1100 generates video and metadata including: Image data, which is compressed using a format suitable for further analysis; Image meta-data, such as data representing exposure time to aid data normalisation and analysis; Timestamp data, representing the time of the capture of the data; and Camera ID data identifying the camera from which the raw imaging data was received. The generated data is stored in the video and metadata storage database 1102. [217] In some embodiments, the data capture unit 1100 also generates meta-data for each camera 111-1 IN, including the location of deployment of the camera, its orientation, its spectral range and one or more field of view parameters. The camera meta-data can be stored in the video and metadata storage database 1102 to facilitate the fusion of objects and events seen from multiple cameras at a later stage in the data processing pipeline.
- Image data which is compressed using a format suitable for further analysis
- Image meta-data such as data representing exposure time to aid data normalisation and analysis
- Timestamp data representing
- the data capture unit 1100 processes the image data according to configurable
- the frame rate This allows the data capture process to be customisable depending on the requirements of the application (e.g. accuracy in detection vs. minimising data storage). For example, in order to register brief actions such as blinking a relatively high frame rate is required (approximately 30 Hz).
- the cameras 111-1 IN can be therefore configured to capture data at a frame rate that is high enough to capture the briefest of events, such as blinking. While the data capture unit 1100 will receive all this data, it can be configured to only retain a fraction of the full set of frames (reducing the full-frame rate to well below 10 Hz), with the exception of particular regions of interest, as identified by the object detector 1106 based on the past few seconds of data. For these regions of interest, all camera frames are retained, and can later be used by the action recognition component to enable detection of brief events. In order to constrain computing requirements, the object detector 1106 and object tracker 1108 operate at the lower, sub 10 Hz frame rate.
- the BEG subsystem 160 can be configured to dynamically remove data from the video and metadata storage unit 1102 once the data has been used as required.
- the object detector 1106 and object tracker 1108 unit process the video and meta-data stored in storage unit 1102.
- the video and meta-data produced by the data capture unit 1100 may be fed directly to the detector 1106 and/or tracker 1108 units, in addition to the storage unit 1102, in order to improve processing efficiency.
- the object detector 1106 operates to process a single captured image frame of the video data and returns a set of detections. These detections estimate the location, size and class probability distribution of object instances within the image. Ideally a single detection is associated with every true object instance of interest.
- the object detector unit 1106 can be implemented as one of a variety of different pattern classifiers, such as for example Convolutional Neural Networks (CNNs) which are trained on large, manually annotated datasets.
- CNN object detector can be implemented as a two stage detector (such as DeNet and Mask RCNN), or as a single stage detector (such as RetinaNet).
- Two stage object detectors operate by first applying a CNN to estimate the likely regions in the image that may contain an object of interest (of any class). These regions are then classified into classes by a second CNN.
- a single stage object detector performs both localization and classification in a single step.
- Single stage detectors may offer advantages including ease of implementation and processing speed in comparison to two-stage detectors, but typically result in reduced localization
- the initial base CNN is pretrained on the very large image classification dataset ImageNet (which contains 1000 classes, and 1.2M Images).
- object detector 1106 is implemented as a cascade of 2 individual detectors, as illustrated in Fig. 13.
- Video frame data from the video and metadata storage unit 1102 is fed into a top level object detector 1304 which operates on a significantly downscaled camera frame (e.g., 1024x1024 pixels) and is responsible for detecting the bounding boxes of large and "high level" objects, e.g., people.
- These detected large objects are then cropped out from the original, high resolution camera frame and fed into a sub-component object detector 1306 which identifies the location of finer
- top-level object and associated sub-object data 1308 which is stored in the frame detection storage unit 1104 (e.g. within a database in the described embodiments).
- the use of cascaded detectors offers performed benefits since the second detector 1306 can be run at full resolution on only cropped image regions instead of the entire frame.
- the preliminary, per frame, object hierarchy implicitly constructed by the cascade based object detector 1106 will also produce hierarchical clue data that is used in the association and fusion unit 1112 (described below).
- the object detector 1106 is trained on a set of images which were manually annotated with bounding boxes indicating the location, size and class of specific objects of interest.
- these training images should be captured from environments in which the system is expected to perform (e.g. classrooms, etc) and must contain an annotation for every object instance in view.
- environments in which the system is expected to perform e.g. classrooms, etc
- 5-10K object instances must be annotated for each class of interest.
- the time to annotate a single bounding box is typically in the order of 30 seconds.
- MAP@IoU intersection-over-union
- IoU intersection over union
- the primary metric with MSCOCO is the MAP@IoU[0.5:0.95] which is the MAP@IoU (described above) averaged over a range of IoUs from 0.5 to 0.95.
- Table 1 MSCOCO results for DeNet, RetinaNet and Mask RCNN object detectors.
- the number of pixels occupied by an object in the image is another important design consideration, objects with fewer associated pixels are significantly more difficult to identify. For instance the MAP@IoU[0.5:0.95] for the 5 Hz DeNet Detector is 57.5% for large objects ⁇ area > 9216 pixels), 45.0% for medium objects (1024 pixels ⁇ area ⁇ 9216 pixels) and only 21.5% for small objects (area ⁇ 1024 pixels) where area is the number of pixels the object occupies in the input image.
- the object tracker unit 1108 performs multiple object
- Object tracking on the detected object data of the frame detection storage unit 1104 in order to track the locations of multiple objects through time.
- Object tracking is performed in terms of "tubelets", each of which describes a sequence of bounding boxes through time (i.e. appearing in a sequence of consecutive video frames). Ideally, a single tubelet is associated with each object of interest.
- MOT is achieved via a Detection Based Tracking (DBT) method.
- DBT Detection Based Tracking
- affinity model There are two major components to DBT, an affinity model and a data association method.
- a simple linear motion model is implemented for the affinity model in which it is assumed that objects travel in straight lines, with either a constant velocity or constant acceleration. This allows the estimation of the location of an object in a given frame based on its previous position and motion within prior frames.
- the affinity is simply the distance from the predicted position to the detection hypothesis position.
- the object tracker 1108 is configurable to utilise either deterministic
- data association may be performed using advanced probabilistic methods.
- Training of the object tracker 1108 can be performed with a relatively small set of videos (e.g. 10 videos, 2 min each) with per frame bounding box annotations and instance IDs for each object being tracked (as determined by the object detector 1106, as described above) to form a groundtruth.
- the groundtruth is used when manually optimizing the ttrraacckkeerr ppaarraammeetteerrss ffoorr bbeesstt ppeerrffoorrmmaannccee..
- tubelet data output by the object tracker unit 1108 is
- the object tracking unit 1108 is configured to generate tubelets over relatively short time sequences (e.g., a few minutes), in order to improve system performance.
- the association and fusion unit 1112 processes the annotated tubelet data representing the short sequences and fuses these into longer sequences.
- the association and fusion unit 1112 uses a similar method to the object tracking unit 1108, which includes an affinity model and data association methods, to fuse tubelets which are associated with the same instance. The main difference is that, instead of operating on detection hypotheses generated by the object detector 1106, the association and fusion unit 1112 uses tubelet hypotheses.
- a detection hypothesis is a single bounding box and classification from a single frame generated by the object detector 1106.
- a tubelet hypothesis is a sequence of multiple detection hypotheses determined to be of the same object instance spread over multiple frames. In the described embodiments, the tubelet hypotheses are limited in length, e.g., 10 seconds in order to improve system performance.
- the association and fusion unit 1112 observes multiple "tubelet hypotheses" and attempts to link those which are associated with the same instance to create new tubelets that are potentially much longer than 10 seconds. It also creates a hierarchy where all the tubelets for a single person are grouped together, e.g. it might associate the hands, face and body of a single person together.
- Fig. 15 illustrates a tubelet fusion and hierarchy construction process 1500 performed by the association and fusion unit 1112, which fuses the tubelets and builds a hierarchical model of the fused tubelets. Tracking and fusion is firstly performed (i.e. at step 1504) on the annotated short tubelet data 1502. For example, the object detector 1106 will attempt to detect potentially overlapping object classes such as person, face, arm, hand, bike, wheel etc. However, the object detector 1106 does not have an understanding of how its object classes are related and connected, and consequently produces short tubelets. The fusion component 1112 associates relates short tubelets to produce fused "long" tubelet data 1506.
- the "person” tubelets are passed to a human pose classifier to estimate a pose associated with the tubelets (i.e. at step 1508).
- the pose estimator framework used in the described embodiments is OpenPose (see https://gith»b.conx ' 'CMU-Percep iial-Compu mg- f , h oi-;c- ;p ⁇ ). however the association and fusion unit may be configured to use other frameworks.
- the human pose estimation step 1508 will try to match a skeletal model to the image of a person and return the results as set of vectors that define a body pose.
- the predicted location of the various body parts are matched to the location of nearby tubelets representing the corresponding classes (body parts) in order to generate a tubelet hierarchy (i.e. at step 1510).
- the partial, preliminary hierarchical information generated by the initial, two-level object detector 1106 is also taken into account at this stage to improve the robustness of object association and hierarchy construction. This process will, for example, allow for the association of eye tubelets with their corresponding face tubelets, and for the association of face tubelets with their parent body tubelets etc. This enables the construction of a fused hierarchical tubelet model 1512.
- the fused tubelet data 1114 includes the tubelet
- the object detector 1106 contains all the information necessary for action recognition (as described below).
- the features generated by the object detector 1106 are also stored in the frame detection storage unit 1104 and/or the annotated tubelet storage unit 1110.
- the type and form of the stored data may vary according to the implementation (e.g. to minimise computation time and/or storage requirements). However, all data required for action/emotion recognition is accessible using the tubelet as a reference irrespective of the storage implementation.
- the recogniser unit 1122 receives fused tubelet data 1114, and performs the identity recognition 1206, action recognition 1208 and emotion recognition 1210 steps of process 1200 depicted in Fig. 12. In some embodiments, processes 1206, 1208 and 1210 are performed sequentially, while in other embodiments the recogniser unit 1122 may be configured to execute the processes in parallel.
- Identity recognition is performed to identify a target person appearing within a fused tubelet as an individual monitored by the system (e.g. a student enrolled within the ADS 102).
- identity recognition is achieved via facial recognition. Factors influencing the accuracy of the facial recognition method
- the recogniser unit 1122 implemented by the recogniser unit 1122 include: i) the number of unique individuals that potentially need to be identified; ii) the nature of the monitoring environment; iii) the size of the monitoring area; and iv) the number of frames which the target individual appears within (on average).
- the number of unique individuals that would need to be identified by a particular installation can be assumed to be fairly small (i.e. typically less than 100);
- the monitoring environment will generally be relatively static, with cameras in controlled locations and individuals mainly situated in a limited range of locations/positions;
- the surveyed area is of limited size and camera resolution can be adjusted to ensure imagery of adequate resolution can be collected for all locations of interest; and
- each individual will be seen in a large number of consecutive frames, consecutive tubelets and potentially from multiple viewpoints by multiple cameras. Combined with object tracking, tubelet association and viewpoint fusion, this will enable the use of statistical methods on top of per-frame identification to boost accuracy.
- the recogniser unit 1122 can be configured with a facial recognition module to perform facial recognition using a conventional recognition technique (e.g. Eyeris EmoVu), combined with data fusion and statistical averaging algorithms.
- a conventional recognition technique e.g. Eyeris EmoVu
- the recogniser unit 1122 may perform facial model training where a priori models are constructed for all individuals that require identification. This could involve, for example, students having their photo taken and added to the database of known individuals and the execution of a training algorithm on this database.
- the input to the face recognition module consists of tubelets classified as containing human faces.
- a confidence measure for each individual image frame in the face tubelet is computed using a conventional face detector algorithm with support for confidence estimates. Alternatively, if using a face detector without support for confidence estimates, frames not detected as a face are assigned a confidence of zero.
- the implemented facial recognition method is then applied to each frame in the face tubelet with a non-zero confidence value.
- the output of the face detector is, for each frame in the tubelet, an indication of a single individual, from the set of preregistered individuals, to whom the face in the frame corresponds.
- the face recognition module assigns an identity to each particular tubelet based on the output of the face recognition algorithm. For each frame, the output is weighted by the confidence value assigned to that frame in the previous step, and the identity with the largest weighted sum is chosen as the identity of the particular tubelet. A normalised measure of identity confidence is saved for later use by the viewpoint fusion process (described below).
- Action recognition is performed by the recogniser unit 1122 to label to a snippet (i.e., a number of consecutive video frames featuring the target individual) with a snippet (i.e., a number of consecutive video frames featuring the target individual) with a snippet (i.e., a number of consecutive video frames featuring the target individual) with a snippet (i.e., a number of consecutive video frames featuring the target individual) with a snippet (i.e., a number of consecutive video frames featuring the target individual) with a snippet (i.e., a number of consecutive video frames featuring the target individual) with a snippet (i.e., a number of consecutive video frames featuring the target individual) with a snippet (i.e., a number of consecutive video frames featuring the target individual) with a snippet (i.e., a number of consecutive video frames featuring the target individual) with a snippet (i.e., a number of consecutive video frames featuring the target individual) with
- the recogniser unit 1122 implements an action recognition module which is
- Action classes including:
- Hand gestures e.g. raising, waving, clapping etc.
- Facial movements e.g. frowning, smiling, blinking, speaking etc.
- Body movements e.g. standing up, sitting down, turning back etc.
- Pre-classification of each fused tubelet is performed to determine whether the tubelet contains a person, and if so the tubelet is processed by the action recognition module.
- the position of body parts and gestures also provides
- the action recognition module can be configured to utilise input information from a variety of different forms (or “modalities") to perform action detection and classification. These sources can include: 1) raw input frames; 2) feature maps from the initial object detector 1106; 3) tubelet data for tubelets labeled as "person”; and 4) the human body pose estimation information extracted during the tubelet hierarchy construction (i.e. at step 1508). Motion information is extracted from each snippet by computing the dense optical flow of consecutive frames in the raw video data. [255] In the described embodiments, the action recognition module is configured to receive input data including: i) Features from video frames, which provide general appearance information about the whole action context. As an optimization specific to the proposed system, the lower resolution features used in the initial top level object detector 1304 are reused here. ii) Features from optical flow as computed from the full frame video. Optical flow provides general motion information from the whole action context. iii) Features from appearance feature maps inside the bounding box of the
- step 1508 for each frame in the action candidate snippet.
- the video frame and optical flow features are extracted from a deep CNN (e.g.,
- LSTM Long Short-Term Memory
- This model first processes the context-aware features, which encode global information about the entire frame (output of first Two-Stream network 1602). The output of this first stage is combined with person-level features (output of second Two-Stream network 1604) to provide a refined prediction.
- the output of second stage is combined with body part-level information (output of third Two-Stream network 1606) to produce a final prediction 1608 of the human gesture-specific action.
- body part-level information output of third Two-Stream network 1606
- extra information can be added to another stage of LSTM. Note that, for blinking, we can still rely on estimated body pose and joints since there are two key-points for eyes.
- the action recognition module is trained using a training data set of fixed-length snippets, where each snippet contains a single action.
- the first training stage involves training each of the Two-Stream networks for contextual representation, person-level representation, and body part-level representation respectively. In the described embodiments, this is achieved by fine-tuning a pre-trained ResNet classifier on frames of the action snippets.
- a conventional CNN training process is used, which is similar to the training processes employed to train a network for image classification using categorical cross-entropy loss function.
- the second training stage then involves training the multi-stage LSTM to utilise the trained two-steam networks for action classification.
- the size of the training data set and the nature of the training data used to train the action recognition module can vary based on the accuracy needed by higher level analysis layers and the variability in the monitoring environments (if any) in which the system is deployed over. In a typical classroom application, it is expected that the action recognition module will benefit most from a training data set that includes a large amount of relevant, annotated data in order to accurately determine the parameters of the stack of networks.
- the training data set consists of several hundred varied and representative video snippets of each single action of interest (i.e. that can be recognised by the module).
- a significantly larger training data set may be required (e.g. a set that includes well over a thousand snippets in each action class).
- the efficiency of the action recognition process may be
- a first stage rejection mechanism is implemented using background subtraction to reject tubelets where no activity occurs.
- a second rejection stage is then performed to compute the optical flow sequence for the tubelet (which can be reused by the action recognition module in subsequently processing stages), which is used to apply a thresholding operation.
- a third rejection stage involves the use of a CNN that processes the sequence of image and optical flow frames associated with the threshold tubelet and classifies the snippets as containing an action or not containing an action.
- the accuracy of the action recognition module can be influenced by a variety of factors including, for example: the number of cameras in the monitor 110 (which may affect the ability to compensate for occlusion); the resolution of each camera 111-1 IN; the available computing resources of the detection server 120 on which the behavioural event generation subsystem 160 is executed; and the quality and quantity of the training data.
- MAP Mean Average Precision
- recognition module can be expected to properly classify around 9 of 10 actual actions represented within active tubelets (given that the action represented by the tubelet is an action that is within the set of trained actions classes).
- Emotion recognition is performed by the recogniser unit 1122 on tubelets which are determined to contain a representation of a human face.
- an emotion recognition module is implemented to execute a conventional emotion recognition algorithm that produces an indication of an emotion exhibited on the face of a target individual within the tubelet.
- the emotion recognition module is trained using a publicly available dataset, such as the MMI Facial Expression Database (MMI Facial Expression Database,
- the determined identity and classification data (including action and/or emotion classifications) is associated with each fused tubelet.
- the resulting data stream of fused, identified and classified tubelets 1116 is received by conversion unit 1118 (as shown in Fig. 11).
- the conversion unit 1118 firstly invokes a viewpoint fusion module 1117 which maps the tubelets into a common 3D space and fuses together tubelets which are associated with the same real world entity. For example, if an individual is visible in multiple cameras 111-l lN of the monitor 110, then each camera 111-1 IN will generate its own tubelet and the viewpoint fusion module fuses these together.
- Identity confidence measures computed for each tubelet by the facial recognition module are used to create a final identity prediction.
- viewpoint conversion module 1119 to convert the identified, classified tubelets into a high level representation suitable for long term storage and higher level analysis. This involves segmenting the tubelet into shorter snippets according to the actions detected by the action recognition module (as described above). Each snippet is then matched with the output of emotion recognition module (if any) to assign an indication of an emotion that is associated with the action. Matching is performed according to one or more matching criteria, such as for example timestamp information.
- the conversion unit 1118 outputs the generated behavioural event data for each snippet.
- the generated behavioural event data is stored in the action and emotion database 1120 (i.e. at step 1214).
- the behavioural event data includes: i) an identifier (ID) value which identifies the individual recognised within the snippet; ii) an indication of an action class representing the action performed by the identified individual (if applicable); iii) an indication of an emotion (or expression) representing the emotion exhibited by the identified individual (if applicable); and iv) a time duration of the snippet.
- the behavioural event data is stored in the action and emotion database 1120 as a behavioural event record.
- Table 2 shows exemplary behavioural event records representing the stored behavioural event data.
- the database 1120 is configured to record timing information about the occurrence of the behavioural event, such as date and timestamp values, in addition to the event data features (i-iv) above.
- the records of the database 1120 are used to construct behavioural profile data for each individual monitored (i.e. during a behavioural profile training process, as described below).
- Table 2 Example behavioural event records stored in database 1120.
- the detection server 120 processes the detection data (i.e. the behavioural action and/or emotion events produced by the BEG subsystem), data representing the biometric profile of the student (for example, as retrieved by the DAPRC at step 908 for the detection of a behavioural characteristic), and the generated context data, to identify the presence of an abnormality of the individual (as described hereinbefore). Details of an exemplary action recognition subsystem which performs this function are included herein below. Identifying an Abnormality from Behavioural Events
- the generated behavioural event data is input into an abnormality recognition (AR) subsystem 170, as described below, which operates to recognise an abnormality by comparing long term trends of observed behavioural events, as represented by corresponding generated behavioural event data, to the appropriate behavioural profile of an individual.
- AR abnormality recognition
- the AR subsystem 170 includes an event classification and analysis unit 1702 which is in communication with a model and profile storage unit 1704 that maintains sets of high-level state models, and (individual specific) behavioural models (as described below).
- the AR subsystem 170 includes an anomaly detector unit 1710 which receives high-level state classification information generated by the event classification and analysis module 1702 (as described below) and context data from a context engine 1706 to determine the existence of an abnormality affecting an individual monitored by the system (as indicated by abnormality indication data 1712).
- the event classification and analysis unit 1702 operates to process behavioural event data and produce activity state classifications based on particular sequences of events.
- the activity states represent high-level behavioural categorisations of an individual, as inferred from the behavioural events that are observed from monitoring the individual over time.
- Fig. 18 illustrates the process 1800 by which the AR subsystem 170 performs high- level behavioural categorisation and subsequent abnormality determination for an individual monitored by the system.
- the event classification and analysis module 1702 receives behavioural event data generated by the BEG 160 representing instances of action and/or emotion events identified for a particular individual.
- the behavioural event data includes event data generated in real-time 1701 by the BEG 160 and event data stored in the action and emotion database 1120.
- a sequencing process is applied to organise the behavioural event data based on the time of occurrence of each event to produce a series of event vectors ⁇ ei, &2,..., £M ⁇ -
- Each event vector has elements representing the value of the time of occurrence of the event, the duration of the event, an indicator of the action and/or emotion class for the event.
- step 1804 the event classification and analysis unit 1702 generates event
- Each behavioural fragment represents a collection of sequentially occurring behavioural events.
- overlapping sequential time windowing is used to form fragments from corresponding sets of considered behavioural events.
- One or more behavioural fragments are generated for each considered set by applying a fragment window of a fixed duration (e.g. 60 seconds) and overlap (e.g. 30 seconds) to each considered event set such as to generate a fragment that captures all corresponding events that occur within the window.
- the considered set of behavioural events are the events which occur within a given observation window, which is of a predetermined size (e.g.
- observation window may be defined dynamically, such as to include event vectors generated from behavioural event data 1701 that is received in real time from the BEG 160.
- Each behavioural fragment represents the behaviour exhibited by an individual over a continuous period of time (i.e. the duration of the fragment window) within the observation window. Since there may be a time gap between adjacent sequential behavioural events that are grouped to form a fragment, the event classification and analysis unit 1702 is configured to define a NULL event indicating that no action or emotion was detected (for a particular individual) over a specified time (i.e. between the event occurrence time and an end time defined as the occurrence time + the duration of the event). Each behavioural fragment can include one or more NULL events, such that the fragment covers the length of the fragment window.
- Fig. 19 illustrates behavioural fragment generation for a set of action behavioural events ei to e 6 which occur over a 5 minute observation window.
- the fragment window size is set to 1 minute with a 30 second overlap, resulting in the generation of 11 fragments f lt ... , ef l .
- Each event is represented by data including an identifier (ID) of the individual, the action class, the emotion class, the event start time, and the event duration.
- ID an identifier
- the event classification and analysis unit 1702 generates NULL events em ...e to bridge the time gap between consecutively occurring action events ei to e 6 prior to grouping the action and null events that fall within each
- the behavioural event sequencing and fragment generation processes can be performed with events associated with any individual monitored by the system (i.e. when the subsystem operates in a training mode as described below), or specific to each individual. That is, by producing a series of behavioural fragments from the behaviour events logged by the BEG subsysteml60, the AR subsystem 170 is able to express the behavioural characteristics which are exhibited by each individual, or by the monitored population in general, over contiguous time intervals of arbitrary lengths.
- fragment generation is also specific to the particular type of behavioural event(s) that is represented by the corresponding event data. Separate sets of fragments are generated for action events and emotion events based on
- behavioural event data which represents the occurrence of, at least, behaviour of the specified type.
- fragment generation performed on the data represented by the records in Table 2 involves generating emotion behaviour fragments using records 11- 17, but not records 13 and 16 (where only an action is represented).
- fragment classification is performed in order to relate the generated behavioural fragments to one or more activity states, therefore providing a high-level categorisation of the behaviour of an individual based on their time sequential observed event behaviours.
- the event classification and analysis unit 1702 defines a set of activity states each corresponding to a high-level categorisation of particular action and emotion behaviour that is detected by the BEG subsystem 160.
- activity states are defined for action (i-iii) and emotion (iv- vi) type behaviours, including: i) physical; ii) personal interaction; iii) class participation; iv) positive emotion; v) neutral emotion; and vi) negative emotion.
- the event classification and analysis unit 1702 maintains a set of activity state models for each activity state.
- the activity state model parameters Prior to classification, the activity state model parameters are trained on behavioural event fragments (i.e. at step 1805) from a training data set.
- supervised model training is performed using the training data set which includes fragments representing sequences of behavioural events associated with one or more individuals. That is, the activity state models are universal models (as opposed to individual specific models) which represent a generic high-level abstraction of event class actions or emotions over the whole population of monitored individuals.
- Each fragment of the training set is labelled in relation to one or more of the activity state models via a transcription process.
- the labelling process varies based on the type of modelling performed and the corresponding classification technique employed.
- the transcription process can occur dynamically during the compilation of the training data.
- the AR subsystem 170 can be operated in a training mode in which the event classification and analysis unit 1702 presents generated behavioural fragment data to a user of the system (such as a teacher) allowing the user to assign classification labels to the fragment indicating the extent to which the fragment is representative of one or more activity states.
- the event classification and analysis unit 1702 can be any event classification and analysis unit 1702.
- Dynamic transcription can be configured by defining one or more rules for the association of behavioural event classes (as defined by the BEG subsystem 160) to corresponding activity states. For example, with respect to the example fragment efl above during automated training the unit 1702 may calculate that the subject of the fragment spent 3.33%, 53.33%, and 40% of the time performing each respective event action, and assign training labels (e.g. indicating the personal interaction activity state) to the fragment accordingly without user intervention.
- training labels e.g. indicating the personal interaction activity state
- event class to activity state association rules are
- class association matrix stored within a data module of the event classification and analysis unit 1702.
- the values of the class association matrix indicate a correlation between each event class and the activity states defined by the system.
- Other data structures may be used to store event class to activity state association data depending on the implementation.
- each activity state model is represented by a conventional multilayer perceptron neural network (MLP- NN).
- MLP- NN multilayer perceptron neural network
- Supervised network training is performed according to a "one vs all" approach in which the training data is dynamically partitioned into sample fragments which are indicative of the particular activity state (as determined by the labelling process), and those which are not.
- the number of internal nodes is chosen according to the
- the activity state model data consists of a single trained NN for each of the states (i-vi) outlined above.
- the event classification and analysis unit 1702 stores the activity model data (i.e. the NN parameter values according to the above described implementation) in the model and profile storage unit 1704.
- the model and profile storage unit 1704 is a database that is accessed by the event classification and analysis unit 1702 via a database management system. Model training is performed offline in a training mode, which is executed at least once prior to the monitoring of the individuals for the purpose of abnormality determination.
- each activity state model can be represented by a set of Gaussian Mixture Models (GMMs) or Hidden Markov Models (HMMs).
- the parameters of each submodel are trained using conventional techniques, such as, for example, a form of expectation maximisation for GMM submodels. Training is supervised using transcribed fragment data which is labelled according to the activity state represented by the data, and with an indication of the degree of that state.
- the action behavioural fragment ef 1 described above may be labelled as belonging to the personal interaction activity state, and may be assigned a 'high' degree. Fragment ef 1 would therefore be used as a training sample for the 'high' submodel of the personal interaction activity state model.
- fragment classification is performed (at step 1806) on new behavioural event fragments that are generated by the event classification and analysis unit 1702 in order to translate the fragment (i.e. the time sequential action/emotion events) into a high-level activity representation.
- Fig. 20 summarises the fragment classification procedure as conducted by the event classification and analysis unit 1702.
- the fragment event type is determined as either being an action event type or an emotion event type.
- the event classification and analysis unit 1702 retrieves the corresponding activity state model data from the model and profile storage unit 1704.
- behavioural and emotion activity states are modelled separately such that the fragments which are applied to these models (i.e. during training or recognition) contain events of either an 'action' or 'emotion' type.
- Separating the activity state models allows the AR subsystem 170 to model actions and events independently, and allows for the abnormality determination process (discussed below) to proceed based on deviations detected from an individual's nominal actions and/or emotions. This can be advantageous when the amount of training data is limited for one type of behavioural event, but not the other.
- the fragment is classified against each activity state model to produce a set of activity state scores or classification values. Specifically, the classification of a candidate fragment results in the generation of single output value for each activity state.
- the form of the activity state scores varies depending on the type of classification method implemented. For implementations using MLP-NNs as the activity state models, the output value is a real number.
- the output value is an indication of a discrete level corresponding to the activity state (e.g. indicating 'high', 'moderate' or 'low' as described above).
- the output values are concatenated to produce a fragment classification vector consisting of three values (one for each of the three action or emotion states, depending on the fragment event type).
- the fragment classification vector for a behavioural event fragment represents the high-level categorisation of the fragment in the activity state space. For example, inputting the action behavioural fragment ef 1 described above into each action activity state MLP-NN model produces a 3 dimensional fragment classification vector vef with higher relative scores for the physical and personal interaction states compared to the class participation state. From this classification vector, it can be inferred that the individual is exhibiting a reasonable level of physical activity, a high level of interactivity with fellow individuals, and a low level of class participation.
- fragment classification vector data produced from the classification of a fragment consisting of observed behavioural events exhibited by an individual is processed by the anomaly detector 1710 to determine the existence of a behavioural abnormality associated with the individual.
- the anomaly detector 1710 receives the observed activity state data from the event classification and analysis unit 1702, in conjunction with data representing the nominal behaviour of the individual with respect to the relevant activity states.
- the anomaly detector 1710 matches the observed activity state data to the nominal activity data in order to determine whether an abnormality exists, as described below.
- the nominal activity data is stored within a behavioural profile of the individual and includes nominal action and emotion models which model the normal, or expected, behaviour for the individual in terms of the activity states of each type (i.e. action and emotion).
- the nominal action and emotion models are individual specific. For example, consider the implementation described above involving action and emotion activity states (i-vi), and as deployed to monitor two students Johnny and Jane. Jonny has an extroverted personality and has attention deficit hyperactivity disorder. Accordingly, his behavioural profile includes a nominal action model which reflects that he typically exhibits a 'high' level for the physical activity and the personal interaction states. Jane is introverted but is otherwise energetic. Her behavioural profile includes a nominal action model which reflects that she typically exhibits a 'high' level for the physical activity state but a 'low' level for the personal interaction state.
- the nominal action and emotion models have an input feature space corresponding to the components of fragment classification vectors (i.e. a dimensionality of 3 in the described embodiments), and are defined by parameter data according to the specific classification technique implemented by the anomaly detector 1710.
- a number of conventional pattern classification methods are suitable, such as NN based classifiers or statistical classifiers (e.g. GMMs/HMMs) as discussed above for universal activity state modelling.
- Supervised training is performed to determine the nominal action and emotion model parameters for each individual.
- the training data includes a set of fragment classification vectors which collectively represents the normal behaviour of the individual across the activity states of the given type (e.g. the degree of physical activity, personal interaction, and class participation for the action type).
- Fig. 21 illustrates the process performed by the anomaly detector 1710 to determine the presence of a behavioural abnormality for an individual.
- the anomaly detector 1710 receives input data from the event classification and analysis unit 1702, including: i) observed activity state data in the form of at least one fragment classification vector representing a high level characterisation of a sequence of behavioural action or emotion events exhibited by an individual (i.e. as represented by the fragment); and ii) an identifier of the individual uniquely specifying that individual within the system.
- the anomaly detector 1710 also receives input from the context engine 1706.
- Context engine 1706 defines and tracks conditions that are relevant to whether an abnormality may be affecting individuals monitored by the system.
- the conditions can include factors influencing the learning environment, such as the type of class or session being undertaken, and/or environmental factors (e.g. the ambient temperature within the environment).
- the context engine 1706 is configured to define the relationship between conditions of relevance (e.g. the type of class, weather conditions, whether an exam/test is being sat by the class, etc.) and the activity state levels which characterise an individual's behaviour (e.g. physical activity, personal interaction, class participation, etc.).
- the context engine 1706 may define a strong positive relationship
- Data maintained by the context engine 1706 includes: i) condition specification data (e.g. 'Class type currently in progress'); ii) an indication of a present value for the condition (e.g. 'gym class'); and iii) condition modifier data indicating the affect that the present value of the condition has on each activity state.
- condition specification data e.g. 'Class type currently in progress'
- an indication of a present value for the condition e.g. 'gym class'
- condition modifier data indicating the affect that the present value of the condition has on each activity state.
- the context engine 1706 determines the present value for the condition dynamically from condition notification data which can be received as input from the teacher, or from another subsystem.
- the condition modifier data is generated from an association table which maps each possible value of the condition to a corresponding effect for each activity state.
- the modifier data may be a positive real number (0, ⁇ ) that is used by the anomaly detector 1710 to weight the individual scores produced by the activity state classification data on the individual's nominal model (as described below).
- the anomaly detector 1710 receives input from the context engine 1706 including, at least: condition specification data for conditions that are relevant to the monitoring environment; and condition modifier data indicating the affect that the present value of the event/condition has on each activity state.
- the anomaly detector 1710 retrieves, from the model and profile storage unit 1704, profile data of the identified individual.
- the nominal behavioural activity model data is extracted from the profile data.
- Classification is performed for each fragment classification vector of the observed activity state data using the nominal action or emotion model of the individual. For each fragment classification vector, a set of matching scores are produced from the
- the matching scores are represented as a matching score vector of 3 components in the described embodiments (i.e. each component being the classification score obtained from one of the nominal action (or emotion) models).
- the anomaly detector 1710 processes one or more of the generated
- the matching score values are augmented by the context modifier data received from the context engine 1706. For example, in a case where the condition 'Class type currently in progress' has a value 'gym class' the condition modifier data may be applied to de- emphasise the matching score of the physical activity state model component (since it is expected that the physical activity of individuals in the environment will be elevated as a consequence).
- Abnormality determination then proceeds by generating an overall matching score from the activity state component specific scores.
- the combination strategy used will depend on the classification technique implemented. For example, in the case of statistical classification an overall likelihood score L (c ⁇ A a ) can be produced from individual likelihood scores ⁇ (3 ⁇ 4
- the anomaly detector 1710 compares the calculated overall matching score to a
- the matching threshold value can be determined by the anomaly detector 1710 according to a desired confidence interval (e.g. such that scores fall below the threshold if they are outside of a 95% confidence interval of the corresponding model). Determination of the threshold value is configurable by a user of the AR subsystem 170 such as to control the level of sensitivity of the abnormality detection process. For example, it may be desirable to widen the interval of acceptable matching scores in implementations where the nominal models have been trained on sparse data, or where there are other factors which increase the difficulty of obtaining an accurate high-level behavioural classification for individuals (e.g. where there are many individuals to be monitored, and many action and/or emotion classes).
- an anomaly is deemed to exist with respect to the particular fragment classification vector.
- the presence of an anomaly for a single fragment classification vector is sufficient to determine that an abnormality is affecting the individual.
- the determination of an abnormality requires the detection of anomalies for multiple fragment classification vectors spanning a predetermined time interval.
- the anomaly detector 1710 When an abnormality is determined, the anomaly detector 1710 generates
- abnormality indication data 1712 which indicates the occurrence of the abnormality to other subsystems of the ADS 102.
- the abnormality indication data 1712 includes: i) an indication of the individual affected by the abnormality indication data 1712
- abnormality e.g. the individual's ID value
- ii an indication of the values of the behavioural activity states associated with the fragment(s); iii) a corresponding time period in which the abnormality occurred; and iv) condition specification data for any conditions that are relevant to the monitoring environment when the abnormality occurred.
- Fig. 22 illustrates an example of the above described classification process for
- the nominal action and emotion models are GMMs ⁇ ⁇ and X e depicted with a single mixture per component, where the components c x , c 2 , c 3 correspond to the activity state values (e.g. the classification scores for the 'physical', 'personal interaction' and 'class participation' states for the action model).
- Classification of the vector c on the nominal action model ⁇ ⁇ produces component level matching score vector
- the context modifier data is used to weigh each component of the matching score vector (i.e.
- the AR subsystem 170 is configured to receive physiological data 1703 representing physical characteristics of individuals monitored by the system.
- the physiological data can be in the form of physiological events representing changes in a physiological parameter of the individual (such as their head and/or body temperature). Windowing is applied to organise the physiological events into time contiguous sequences (i.e. physiological event fragments), as described above for the behavioural events.
- the event classification and analysis unit 1702 is configured to provide the anomaly detector 1710 with observed physiological state data, which is produced by classifying physiological event fragments using pre-trained models for a set of physiological states.
- the physiological states include head temperature, extremity temperature (such as hands), and body temperature.
- the anomaly detector 1710 matches the observed physiological state data to a nominal physiological model of the identified individual to produce matching scores similarly to the process described above for the behavioural abnormality determination.
- An overall matching score is then calculated based on a physiological matching score vector, allowing a comparison to be performed between the calculated overall matching score and a threshold value.
- Anomalies in the observed physiological state data can be detected based on the comparison result, and a (physiological) abnormality is determined when an anomaly is detected within at least a given fixed number of physiological fragment classification vectors, as described above for the processing of the behavioural data.
- Generation of the observed physiological state data from the physiological fragment data may involve different processing steps compared to the generation of the observed behavioural activity state data. For example, in detecting the head and body temperature characteristics, the detection server 120 may compare detected values of the student's head/body temperature with an average head/body temperature of the class, and offset the deviation based on the student' s biometric profile (according to the methods described herein hereinbefore). In some cases, a 2% difference in a child's temperature from the rest of the class after the offset may be taken by the system as insignificant, whereas a 5% difference after the offset may indicate a potential problem, and would therefore result in the determination of an abnormality.
- the detection server 120 is also configured to detect differences in body temperature between different parts of the body, for example, between the hand and the head. A consistently high temperature throughout the body may be caused by a recent activity of the student, e.g., the student has been running, whereas a relatively higher temperature in the head may indicate illness. Accordingly, in some embodiments the server 120 may also be configured to trigger an alert (i.e. detect an abnormality) when the difference between the head temperature and the hand temperature exceeds a predetermined threshold, while ignoring an insignificant increase in the overall body temperature (in accordance with the critical event reporting processes described herein). This can be achieved, for example, by modelling the temperature differential as a separate physiological state variable in the above described processes.
- an increased heart rate may be caused by the student's recent physical activity.
- the detection server 120 can compare the detected heart rate with the average heart rate of the whole cohort, and offset the deviation based on the student's biometric profile.
- the heart rate may be detected for a continuous 15-second period simultaneously for a whole group of students. If the system fails to obtain a continuous 15-second heart rate of a particular student, it may start the detection process again for that student only, until a valid result is obtained.
- the frequency of a child scratching his/her head may be detected as a behavioural characteristic (as described hereinbefore), and a high frequency of a student' s head scratching against their regular pattern and an average frequency of the whole cohort may suggest a lice infection, and may be detected as an abnormality by the system.
- the system 102 may trigger an alert or send a notification to the teacher, to enable the affected student to be removed from the class (in accordance with the processes described herein).
- a low frequency of hand-raising by a student as compared to a frequency that is indicated as normal by the student's profile, may also be detected as an abnormality, subject to the determined context of this behaviour (as described hereinafter).
- the ADS 102 can be configured to perform monitoring and detection activities for one or more selected students. For example, by only detecting behavioural characteristics of these students, or detecting more behavioural
- the system 102 can also be configured to detect a student's interaction with other people based on predetermined behaviour characteristics. If it is detected that a student is spending more time alone than usual, the system 102 may identify a behavioural abnormality, and may send a notification to the relevant teacher or welfare officer regarding the abnormal behaviour to allow further investigation of the situation. Context data representing the student's environment may be used to determine whether the student' s interaction (or lack thereof) with other people constitutes a behavioural abnormality.
- a lack of interaction representing the student spending more time alone than usual when the student is in a particular area where social activity is expected may result in the identification of an abnormality, while a lack of interaction detected in another area (e.g., a classroom) may not.
- the detection of the student's behavioural characteristics may be used in the detection of the student's physical characteristics.
- the detection of heart rate may be started after the student has been seated.
- the detection server 120 can be configured to transmit detection event data to the BPMS server 130 when particular behavioural characteristics are detected.
- the BPMS server 130 can be configured to process the detection event data to perform particular functions in relation to the facilities, such as for example locking and/or unlocking doors adjacent to, or within, the monitoring area for students whose amount of physical movements is significantly lower than other students.
- the DAPRC can be configured to interpret one or more attributes indicated by the context data as rules, or conditions. For example, without considering the context data, a decrease in the frequency with which a student raises their hand during class, compared to the frequency of hand raising which is expected of the student based on their behavioural profile, may result in the identification of an abnormality (e.g., indicating the possibility of an issue, such as domestic troubles or bullying). However, an indication that the class or session being undertaken in the monitoring area is an exam or test can be interpreted as a condition which prevents the determination of an abnormality associated with the detected lack of class participation.
- an abnormality e.g., indicating the possibility of an issue, such as domestic troubles or bullying.
- an indication that the class or session being undertaken in the monitoring area is an exam or test can be interpreted as a condition which prevents the determination of an abnormality associated with the detected lack of class participation.
- the context data can be used during the comparison between the detection data and the corresponding biometric profile data (such as, for example, as an input to a pattern classification process).
- This allows a weight to be assigned to attributes indicated by the context data (such as the type of instruction being undertaken in the monitoring area - e.g., a regular class, or an exam), as opposed to using these attributes as rules or conditions which prevent or allow the identification of particular abnormalities.
- the DAPRC can be configured to utilise the generated context data to determine one or more models of the student's biometric profile which are to be used to identify the abnormality (i.e., by comparison with the detected physical or behavioural
- the thermal model to be used for identifying an abnormality based on the detected thermal signature (i.e., "pattern") of a student can be determined from calendar or timetable data.
- Each student may have several thermal profile models depending on their previous classes and/or activities in the learning environment (e.g., a normal profile model, an after-lunch profile model, and an "after-sport" profile model). The number of required models may vary depending on the application.
- the detection server 120 reports the identified abnormalities to the
- the DAPRC invokes the ANC to generate update report data for reporting the physical and/or behavioural characteristics of one or more students to the BPMS server 130.
- the reporting process of step 812 can be configured to occur as part of: a scheduled abnormality reporting process in which abnormalities that have been identified during a particular period of time (e.g., over the course of a school day) are processed and transmitted to the BPMS as a batch; and/or a dynamic abnormality reporting process in which the update report data is transmitted from the ANC to the BPMS server 130 in realtime (i.e., as an abnormality is identified for a student).
- Scheduled reporting can be configured to occur over an arbitrary time period as
- Dynamic reporting is performed for critical events (i.e., when a critical physical characteristic and/or behaviour is detected), such as those described hereinbefore.
- the reporting process of the detection server 120 can also be configured to include attendance monitoring, which reports on the detection of the ingress and egress of each student in a class, for example based on facial recognition or other authentication techniques (as described hereinbefore) .
- the detected ingress and egress information may be used by the LMC to register the students' attendance.
- the LMC stores data
- Attendance event data is generated to represent the ingress and egress of each student in the monitoring area.
- the attendance data is transferred to the attendance module 134 of the BPMS server 130 during the reporting process, either as a scheduled and/or a dynamic (i.e., real-time) report.
- the attendance module 134 can be configured to process the attendance event data to generate logging event data (as described hereinbefore), or other corresponding data, to record the attendance of the student in the monitoring area.
- the detection server 120 can be configured to execute the reporting process of step 812 without the identification of any abnormality, such as for example to perform attendance detection for students within the monitoring area.
- the physical and/or behavioural detection data generated by the detection server 120 can be utilised to selectively update the corresponding physical and/or behavioural profiles for one or more individuals (i.e., at step 814).
- the detection data is not used to update the biometric profile data if the detection data indicates an abnormality for the individual (as determined in step 810 hereinbefore). For example, if a student's heart rate was sufficiently abnormal to trigger an alert, the reading would not be used to update the profile.
- the ADS 102 can be configured to perform an update of the biometric profile in cases where the detection data was found to indicate an abnormality.
- the relative influence of the detection data values on the biometric profile parameters may be adjusted when performing the update (such as, for example, by attributing a reduced weight to those values).
- the detection server 120 is configured to transmit biometric update data to the BPMS server 130 representing the updated physical and/or behavioural profiles of one or more students, as generated by the detection server 120.
- the biometric update data is processed by the profile management component 132 to generate corresponding biometric data representing the updated profiles, and to store the generated biometric data in the profile database 131.
- the system and method provided in this disclosure may be used to monitor health indicators of passengers and crews of ships, for example, by monitoring their biometric characteristics such as body temperature and/or heart rate.
- monitoring devices may be installed at embarkation points, to monitor the body temperature and heart rate of each boarding person.
- the system can compare the person's body temperature and heart rate with the average level of a group of passengers or crew, it may effectively reduce false alarms, especially in the circumstance that the passenger's body temperature has been affected by the terminal's condition, e.g., the departure hall is hot, or a group of passengers has just alighted from a stifling bus. If the person with the elevated reading is representative of the general cohort, then that person will not be signalled by the system as being at risk. If the body temperature of a passenger is significantly higher than the average of other passengers, the system may trigger an alert or send a notification to a doctor on the ship for investigation.
- the system may provide additional functions, including one or more of the following: a) identification upon embarkation and disembarkation at the port; b) providing access to their cabin and to restricted areas on the ship; and c) authorising the passenger's on-board purchases and/or managing their account for on-board purchases.
- the system identifies the passenger, e.g., by using facial recognition or biometric identification techniques, and may retrieve relevant information of that passenger.
- the passenger's name and photo may be retrieved and displayed on a screen at the check-in point, to allow the check-in staff to welcome the passenger by their name. If the passenger cannot be identified from the database, the facial photo acquired by the monitoring device may be displayed on the screen for the check-in staff to confirm and verify the identity of the passenger, and subsequently link their face to their personal information pre-recorded in the database if their identity is verified.
- Other indicators such as lights may be used. For example, during embarking or disembarking, a green light may indicate to the crew member that the passenger's identification has been confirmed and their embarkation/disembarkation has been recorded, while red light may indicate a problem.
- An alarm may be triggered if a person is not identified, enabling them to be apprehended.
- the embarkation/disembarkation process may be expedited, as the passengers do not need to locate and present their boarding cards or identity documents when carrying their luggage at the same time. Further, fewer crew are required at the embarkation points, as the device can automatically identify each passenger. The security of the ship may also be improved.
- the system monitors the biomedical characteristics of the passenger, such as body temperature and heart rate, and determines whether the passenger's health condition is suitable for the travel. Any abnormal readings from the initial scan may appear as an alert on the screen, prompting the check-in staff to have the passenger removed to a separate area for more detailed examination by a doctor on the ship.
- the facial recognition and the temperature checking may be instantaneous, while the heart rate check may last for a predetermined period of time, e.g., 15 seconds..
- the system may start building up a biometric profile for each passenger.
- Further monitoring devices may be installed in different areas on the ship, so as to provide additional monitoring of the passenger's health conditions, based on each passenger's biometric profile.
- the passenger's biometric profile may be stored in the database and may be reused in the passenger's future cruise travel.
- the system may also be used for monitoring and tracking the health condition of ship crews, whose biometric profiles may be recorded and used in different ship trips.
- the system may be used for providing a passenger access to their cabin and providing selected passengers or ship crew access to restricted areas on the ship, such as a VIP area, a business-class area, or a "ship within a ship" area on large ships for passengers in premium accommodations.
- This access control may use facial recognition or other biometric identification techniques.
- the system may further be configured to unlock or open a door when it is detected that an authorised passenger is approaching it. On the other hand, when a passenger or crewmember enters an unauthorised area, an alert may be triggered.
- the system may further be used for authorising the passenger's on-board purchases and/or managing their account for on-board purchases.
- a shipboard account may be set up for each passenger during the initial shoreside check-in process.
- the account may be associated with the passenger's facial features, which may be stored in the database of the system, or may have been acquired or updated during the embarkation process by the monitoring device.
- the facial features of a minor e.g., a child of a passenger
- the parent's account if the parent permits it.
- the steward When processing an on-board purchase, the steward is provided with a hand-held device that can take a photo of the passenger's face.
- the facial photo is sent from the device to the system to identify the passenger's shipboard account associated with that face.
- the shipboard account is charged accordingly to make the payment for the purchase.
- the passenger's further personal information may be sent to the hand-held device, such as the passenger's name, to allow the steward to address the passenger by their name.
- a notification may be sent by the system to the hand-held device that the purchase cannot proceed. For example, if a passenger is under the minimum legal drinking age, the system may reject the purchase request from that passenger.
- the order may be recorded by the hand-held device and sent automatically to a relevant supply staff, such as a bartender if the passenger is ordering a drink. This may reduce the turnaround time, as the supply staff may start the preparation of the good or service immediately.
- the actual charge to the passenger's account may be triggered when the good is supplied to the passenger, with a receipt printed for the passenger.
- the passenger's photo may be acquired by a device installed at a counter or a bar where the order can be placed.
- the system may detect biometric characteristics of the passenger, and
- the system may monitor the passenger's physical or behavioural
- the system may not authorise the purchase, or a notification may be sent to the steward or bartender to reject or terminate the order.
- a crew member may be notified to ensure the safety of the passenger.
- the behavioural characteristics of crew members may be monitored to ensure that they provide proper service to the passengers.
- the provided method and system may be used for monitoring staff in hospitals, medical centres and clinics.
- Front-line staff in these facilities are more prone than the general population to catch infections or infectious disease, and are at higher risk of passing on infections or infectious disease to patients.
- Monitoring devices may be installed at multiple locations in these premises to
- a server that receives data from the monitors may instantly check the data obtained against the profile of a person associated with it. Variations outside an acceptable range for that person may trigger an alert, allowing the person identified as ill to be pulled from duty immediately.
- the provided method and system may be used to monitor machine operators in
- the system provided in the present disclosure can reference the machine operator's detected behaviour to their usual pattern of behaviour at various times of the day, so that an alarm will sound only if the behaviour is abnormal for that person.
- a device focusing on each operator or a group of operators may be provided to
- the system server collects and analyses this data in real time, comparing it with the individual's own biometric profile or biometric pattern associated with that time. For example, it may be normal for an employee to become slightly tired towards the end of their shift, and the system may only trigger an alert if this tiredness becomes excessive.
- the system can build up a profile for each individual, it may self-tune to accept that certain behaviours are normal for that individual. For example, some workers may exhibit signs of stress or fatigue because of excessive fidgeting, while for some other workers these behaviours may be normal. Accordingly, the system may learn that some behaviours are normal for a particular person and avoid sending alert signals in these situations, but only triggers the alarm when a genuine abnormality is detected.
- the system may be configured to also monitor the physical characteristics of the workers, and to alert for any potential health issues that are detected.
- the system may be used to monitor machine operators or
- production line workers to check whether they are performing the work properly, e.g., to detect whether any worker is not working or not following a predetermined working process.
- the provided method and system may be used to monitor pilots for unusual physical or behavioural variations.
- the system may be configured to take into account predetermined factors related to physical or behavioural variations. For example, it would be normal for the heart rate of a pilot to increase while the plane is taking off or landing. However, if the heart rate rises mid-flight, the system may recognise it as an abnormality and therefore trigger an alert. The system may also be configured to determine whether a pilot is suitable to fly before the plane leaves the terminal, based on the pilot's physical condition or behaviour.
- the provided method and system may be used to monitor a driver's behaviour, and to further detect whether a driver is fatigued based on his or her behaviour. This may apply to car, truck, bus, train or tram drivers.
- provided in the present disclosure are configured to take into account an individual's specific behaviour patterns. For example, some drivers tend to make more movement during driving than other drivers. Using the system provided herein, frequent movement of this driver can be recognised as his or her normal behaviour, thereby effectively reducing false alarms. At the same time, out-of-character behaviour of the driver can be accurately detected to alert for situations such as fatigue driving or drunk driving. The system may further inhibit the vehicle to be started if the driver is detected to be fatigued or drunk.
- the system may further include the function of facial recognition, so that a separate profile can be created for each driver, to cater for the situation that a same vehicle may be driven by different drivers at different times.
- the system may further provide driver authorisation based on the result of facial recognition, so that the vehicle will not start if the driver is an unauthorised person, with the exception of allowing parking attendants, service personnel and the like to drive the vehicle if so authorised.
- the provided method and system may be used in childcare centres to monitor health conditions and behaviours of children, for example, their body temperatures. This may facilitate preventing or mitigating the spread of infectious diseases in these facilities.
- the system may be used to monitor the children entering a shared space, such as the playroom and may alert the staff of the childcare centre that a child is sick before he or she enters the playroom.
- the system may also monitor all the children in the childcare centre during the day, and detect signs of medical issues before they become apparent or for behavioural anomalies that the staff may not have noticed.
- system may also be used to monitor the health and/or behaviour of staff, e.g., whether they have a health issue, or whether they are treating the children appropriately.
- the provided method and system may be used for nursing homes or other aged care facilities. Similar functions to those for childcare centres may be provided.
- the provided method and system may be used to provide continuous and remote monitoring of one or more of the following biometric characteristics of a baby: a) body temperature; b) sleep patterns; c) movements/gestures; d) heart rate; and e) breathing patterns.
- an alarm may be triggered to notify the parents or care giver.
- the provided method may be used for detecting an abnormal event such as a terrorist attack.
- the system may be configured to trigger an alert if it detects someone is conducting an abnormal behaviour, such as holding a gun or a knife.
- a group of people (students in a classroom) moving erratically may also signify a terrorist situation.
- the provided method can be used in an Intelligent Security Systems (ISS).
- ISS Intelligent Security Systems
- the system may automatically detect abnormal behaviour. It may also be used for tracing a particular individual. For example, if stock has been stolen by an employee in a business or store, the stored identity data or biometric data of a particular person may be used to automatically find in the recorded video all footage that includes the person, rather than manually looking through the whole video recording. [390]
- the system may also check a person's identity and monitor whether that person is authorised to conduct a pre-determined behaviour or enters a restricted area. If the person conducts an unauthorised behaviour, the system may trigger an alert. For example, if it is detected that a cleaner has opened a cash drawer in a shop, or a customer has entered a staff-only area in a restaurant, an alarm may be triggered.
Landscapes
- Health & Medical Sciences (AREA)
- Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Medical Informatics (AREA)
- Physics & Mathematics (AREA)
- Pathology (AREA)
- Biophysics (AREA)
- Veterinary Medicine (AREA)
- Animal Behavior & Ethology (AREA)
- Surgery (AREA)
- Molecular Biology (AREA)
- Heart & Thoracic Surgery (AREA)
- General Business, Economics & Management (AREA)
- Business, Economics & Management (AREA)
- Primary Health Care (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Epidemiology (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Psychiatry (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Social Psychology (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Measuring And Recording Apparatus For Diagnosis (AREA)
Abstract
Description
Claims
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/250,077 US20210307621A1 (en) | 2017-05-29 | 2018-05-29 | Method And System For Abnormality Detection |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
AU2017902032 | 2017-05-29 | ||
AU2017902032A AU2017902032A0 (en) | 2017-05-29 | Method and system for abnormality detection | |
AU2017279806A AU2017279806B2 (en) | 2017-05-29 | 2017-12-22 | Method and system for abnormality detection |
AU2017279806 | 2017-12-22 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2018218286A1 true WO2018218286A1 (en) | 2018-12-06 |
Family
ID=64454180
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/AU2018/050520 WO2018218286A1 (en) | 2017-05-29 | 2018-05-29 | Method and system for abnormality detection |
Country Status (1)
Country | Link |
---|---|
WO (1) | WO2018218286A1 (en) |
Cited By (33)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109413213A (en) * | 2018-12-20 | 2019-03-01 | 泉州装备制造研究所 | Cluster monitoring system based on system on chip |
CN110084259A (en) * | 2019-01-10 | 2019-08-02 | 谢飞 | A kind of facial paralysis hierarchical synthesis assessment system of combination face texture and Optical-flow Feature |
CN110458013A (en) * | 2019-07-05 | 2019-11-15 | 中国地质大学(武汉) | A kind of traffic abnormal incident detection method of Case-based Reasoning grade attention mechanism |
CN110472728A (en) * | 2019-07-30 | 2019-11-19 | 腾讯科技(深圳)有限公司 | Target information determines method, target information determining device, medium and electronic equipment |
CN110580446A (en) * | 2019-07-16 | 2019-12-17 | 上海交通大学 | Behavior semantic subdivision understanding method, system, computer device and medium |
CN111222493A (en) * | 2020-01-20 | 2020-06-02 | 北京捷通华声科技股份有限公司 | Video processing method and device |
CN111242004A (en) * | 2020-01-10 | 2020-06-05 | 厦门瑞为信息技术有限公司 | Automatic alarm method and system based on elevator monitoring data processing |
CN111368770A (en) * | 2020-03-11 | 2020-07-03 | 桂林理工大学 | Gesture recognition method based on skeleton point detection and tracking |
CN111462233A (en) * | 2020-03-20 | 2020-07-28 | 武汉理工大学 | Recovery data processing method and system for ship cab and storage medium |
CN111523445A (en) * | 2020-04-21 | 2020-08-11 | 南通大学 | Examination behavior detection method based on improved Openpos model and facial micro-expression |
CN111523444A (en) * | 2020-04-21 | 2020-08-11 | 南通大学 | Classroom behavior detection method based on improved Openpos model and facial micro-expressions |
CN111738177A (en) * | 2020-06-28 | 2020-10-02 | 四川大学 | Student classroom behavior identification method based on attitude information extraction |
CN111798356A (en) * | 2020-07-09 | 2020-10-20 | 北京交通大学 | Rail transit passenger flow abnormal pattern recognition method based on big data |
WO2020225389A1 (en) * | 2019-05-08 | 2020-11-12 | Jaguar Land Rover Limited | Activity identification method and apparatus |
CN111950515A (en) * | 2020-08-26 | 2020-11-17 | 重庆邮电大学 | Semantic feature pyramid network-based small face detection method |
CN112233800A (en) * | 2020-11-19 | 2021-01-15 | 吾征智能技术(北京)有限公司 | Disease prediction system based on abnormal behaviors of children |
CN112669966A (en) * | 2020-12-14 | 2021-04-16 | 北京易华录信息技术股份有限公司 | Behavior analysis system based on monitoring data |
WO2021072479A1 (en) * | 2019-10-14 | 2021-04-22 | TBIAS Pty Ltd | An automated behavioural monitoring unit |
CN112866808A (en) * | 2020-12-31 | 2021-05-28 | 北京市商汤科技开发有限公司 | Video processing method and device, electronic equipment and storage medium |
CN113076772A (en) * | 2019-12-18 | 2021-07-06 | 广东毓秀科技有限公司 | Abnormal behavior identification method based on full modality |
WO2021184071A1 (en) * | 2020-03-17 | 2021-09-23 | Elenium Automation Pty Ltd | "self-service station having thermal imaging camera" |
CN113627330A (en) * | 2021-08-10 | 2021-11-09 | 北京百度网讯科技有限公司 | Method and device for identifying target type dynamic image and electronic equipment |
EP3925524A1 (en) * | 2020-06-18 | 2021-12-22 | Rockwell Collins, Inc. | Physiological state screening system |
EP3925521A1 (en) * | 2020-06-18 | 2021-12-22 | Rockwell Collins, Inc. | Contact-less passenger screening and identification system |
CN113836969A (en) * | 2020-06-23 | 2021-12-24 | 山西农业大学 | Abnormal event detection method based on double flows |
CN114005237A (en) * | 2021-10-30 | 2022-02-01 | 南京林业大学 | Forest fire identification method and equipment based on thermal imaging analysis technology and computer storage medium |
US11288498B2 (en) * | 2019-09-19 | 2022-03-29 | Hrl Laboratories, Llc | Learning actions with few labels in the embedded space |
CN115082571A (en) * | 2022-07-20 | 2022-09-20 | 深圳云游四海信息科技有限公司 | Anomaly detection method and system for in-road parking camera |
WO2023286313A1 (en) * | 2021-07-15 | 2023-01-19 | ソニーグループ株式会社 | Signal processing device and method |
US11666246B2 (en) | 2015-12-31 | 2023-06-06 | Cerner Innovation, Inc. | Methods and systems for assigning locations to devices |
CN116563797A (en) * | 2023-07-10 | 2023-08-08 | 安徽网谷智能技术有限公司 | Monitoring management system for intelligent campus |
US11721190B2 (en) | 2017-12-28 | 2023-08-08 | Cerner Innovation, Inc. | Utilizing artificial intelligence to detect objects or patient safety events in a patient room |
US12008880B2 (en) | 2023-05-09 | 2024-06-11 | Cerner Innovation, Inc. | Utilizing artificial intelligence to detect objects or patient safety events in a patient room |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6309342B1 (en) * | 1998-02-26 | 2001-10-30 | Eastman Kodak Company | Management of physiological and psychological state of an individual using images biometric analyzer |
US7027621B1 (en) * | 2001-03-15 | 2006-04-11 | Mikos, Ltd. | Method and apparatus for operator condition monitoring and assessment |
WO2008064431A1 (en) * | 2006-12-01 | 2008-06-05 | Latrobe University | Method and system for monitoring emotional state changes |
US20140006326A1 (en) * | 2012-06-28 | 2014-01-02 | Nokia Corporation | Method and apparatus for providing rapport management |
WO2015114554A1 (en) * | 2014-01-31 | 2015-08-06 | Visa International Service Association | Method and system for authorizing a transaction |
WO2016135069A1 (en) * | 2015-02-26 | 2016-09-01 | Koninklijke Philips N.V. | Context detection for medical monitoring |
-
2018
- 2018-05-29 WO PCT/AU2018/050520 patent/WO2018218286A1/en active Application Filing
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6309342B1 (en) * | 1998-02-26 | 2001-10-30 | Eastman Kodak Company | Management of physiological and psychological state of an individual using images biometric analyzer |
US7027621B1 (en) * | 2001-03-15 | 2006-04-11 | Mikos, Ltd. | Method and apparatus for operator condition monitoring and assessment |
WO2008064431A1 (en) * | 2006-12-01 | 2008-06-05 | Latrobe University | Method and system for monitoring emotional state changes |
US20140006326A1 (en) * | 2012-06-28 | 2014-01-02 | Nokia Corporation | Method and apparatus for providing rapport management |
WO2015114554A1 (en) * | 2014-01-31 | 2015-08-06 | Visa International Service Association | Method and system for authorizing a transaction |
WO2016135069A1 (en) * | 2015-02-26 | 2016-09-01 | Koninklijke Philips N.V. | Context detection for medical monitoring |
Cited By (42)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11937915B2 (en) | 2015-12-31 | 2024-03-26 | Cerner Innovation, Inc. | Methods and systems for detecting stroke symptoms |
US11666246B2 (en) | 2015-12-31 | 2023-06-06 | Cerner Innovation, Inc. | Methods and systems for assigning locations to devices |
US11721190B2 (en) | 2017-12-28 | 2023-08-08 | Cerner Innovation, Inc. | Utilizing artificial intelligence to detect objects or patient safety events in a patient room |
CN109413213A (en) * | 2018-12-20 | 2019-03-01 | 泉州装备制造研究所 | Cluster monitoring system based on system on chip |
CN110084259B (en) * | 2019-01-10 | 2022-09-20 | 谢飞 | Facial paralysis grading comprehensive evaluation system combining facial texture and optical flow characteristics |
CN110084259A (en) * | 2019-01-10 | 2019-08-02 | 谢飞 | A kind of facial paralysis hierarchical synthesis assessment system of combination face texture and Optical-flow Feature |
WO2020225389A1 (en) * | 2019-05-08 | 2020-11-12 | Jaguar Land Rover Limited | Activity identification method and apparatus |
CN110458013B (en) * | 2019-07-05 | 2022-05-06 | 中国地质大学(武汉) | Traffic abnormal event detection method based on instance-level attention mechanism |
CN110458013A (en) * | 2019-07-05 | 2019-11-15 | 中国地质大学(武汉) | A kind of traffic abnormal incident detection method of Case-based Reasoning grade attention mechanism |
CN110580446A (en) * | 2019-07-16 | 2019-12-17 | 上海交通大学 | Behavior semantic subdivision understanding method, system, computer device and medium |
CN110472728A (en) * | 2019-07-30 | 2019-11-19 | 腾讯科技(深圳)有限公司 | Target information determines method, target information determining device, medium and electronic equipment |
US11288498B2 (en) * | 2019-09-19 | 2022-03-29 | Hrl Laboratories, Llc | Learning actions with few labels in the embedded space |
WO2021072479A1 (en) * | 2019-10-14 | 2021-04-22 | TBIAS Pty Ltd | An automated behavioural monitoring unit |
CN113076772A (en) * | 2019-12-18 | 2021-07-06 | 广东毓秀科技有限公司 | Abnormal behavior identification method based on full modality |
CN111242004A (en) * | 2020-01-10 | 2020-06-05 | 厦门瑞为信息技术有限公司 | Automatic alarm method and system based on elevator monitoring data processing |
CN111222493A (en) * | 2020-01-20 | 2020-06-02 | 北京捷通华声科技股份有限公司 | Video processing method and device |
CN111368770A (en) * | 2020-03-11 | 2020-07-03 | 桂林理工大学 | Gesture recognition method based on skeleton point detection and tracking |
WO2021184071A1 (en) * | 2020-03-17 | 2021-09-23 | Elenium Automation Pty Ltd | "self-service station having thermal imaging camera" |
CN111462233A (en) * | 2020-03-20 | 2020-07-28 | 武汉理工大学 | Recovery data processing method and system for ship cab and storage medium |
CN111462233B (en) * | 2020-03-20 | 2024-02-13 | 武汉理工大学 | Method, system and storage medium for processing restored data of ship cab |
CN111523444A (en) * | 2020-04-21 | 2020-08-11 | 南通大学 | Classroom behavior detection method based on improved Openpos model and facial micro-expressions |
CN111523445A (en) * | 2020-04-21 | 2020-08-11 | 南通大学 | Examination behavior detection method based on improved Openpos model and facial micro-expression |
EP3925524A1 (en) * | 2020-06-18 | 2021-12-22 | Rockwell Collins, Inc. | Physiological state screening system |
EP3925521A1 (en) * | 2020-06-18 | 2021-12-22 | Rockwell Collins, Inc. | Contact-less passenger screening and identification system |
CN113836969A (en) * | 2020-06-23 | 2021-12-24 | 山西农业大学 | Abnormal event detection method based on double flows |
CN111738177A (en) * | 2020-06-28 | 2020-10-02 | 四川大学 | Student classroom behavior identification method based on attitude information extraction |
CN111738177B (en) * | 2020-06-28 | 2022-08-02 | 四川大学 | Student classroom behavior identification method based on attitude information extraction |
CN111798356B (en) * | 2020-07-09 | 2023-11-10 | 北京交通大学 | Rail transit passenger flow abnormal pattern recognition method based on big data |
CN111798356A (en) * | 2020-07-09 | 2020-10-20 | 北京交通大学 | Rail transit passenger flow abnormal pattern recognition method based on big data |
CN111950515A (en) * | 2020-08-26 | 2020-11-17 | 重庆邮电大学 | Semantic feature pyramid network-based small face detection method |
CN112233800A (en) * | 2020-11-19 | 2021-01-15 | 吾征智能技术(北京)有限公司 | Disease prediction system based on abnormal behaviors of children |
CN112669966A (en) * | 2020-12-14 | 2021-04-16 | 北京易华录信息技术股份有限公司 | Behavior analysis system based on monitoring data |
CN112866808A (en) * | 2020-12-31 | 2021-05-28 | 北京市商汤科技开发有限公司 | Video processing method and device, electronic equipment and storage medium |
CN112866808B (en) * | 2020-12-31 | 2022-09-06 | 北京市商汤科技开发有限公司 | Video processing method and device, electronic equipment and storage medium |
WO2023286313A1 (en) * | 2021-07-15 | 2023-01-19 | ソニーグループ株式会社 | Signal processing device and method |
CN113627330A (en) * | 2021-08-10 | 2021-11-09 | 北京百度网讯科技有限公司 | Method and device for identifying target type dynamic image and electronic equipment |
CN113627330B (en) * | 2021-08-10 | 2024-05-14 | 北京百度网讯科技有限公司 | Method and device for identifying target type dynamic image and electronic equipment |
CN114005237A (en) * | 2021-10-30 | 2022-02-01 | 南京林业大学 | Forest fire identification method and equipment based on thermal imaging analysis technology and computer storage medium |
CN115082571A (en) * | 2022-07-20 | 2022-09-20 | 深圳云游四海信息科技有限公司 | Anomaly detection method and system for in-road parking camera |
US12008880B2 (en) | 2023-05-09 | 2024-06-11 | Cerner Innovation, Inc. | Utilizing artificial intelligence to detect objects or patient safety events in a patient room |
CN116563797A (en) * | 2023-07-10 | 2023-08-08 | 安徽网谷智能技术有限公司 | Monitoring management system for intelligent campus |
CN116563797B (en) * | 2023-07-10 | 2023-10-27 | 安徽网谷智能技术有限公司 | Monitoring management system for intelligent campus |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20210307621A1 (en) | Method And System For Abnormality Detection | |
WO2018218286A1 (en) | Method and system for abnormality detection | |
US11276291B2 (en) | Utilizing artificial intelligence to detect objects or patient safety events in a patient room | |
Ramzan et al. | A survey on state-of-the-art drowsiness detection techniques | |
US11151383B2 (en) | Generating visual event detectors | |
Weng et al. | Driver drowsiness detection via a hierarchical temporal deep belief network | |
US8582832B2 (en) | Detecting behavioral deviations by measuring eye movements | |
US8954433B2 (en) | Generating a recommendation to add a member to a receptivity cohort | |
CN111095264B (en) | Object recognition system and method | |
Saini et al. | Kinect sensor-based interaction monitoring system using the BLSTM neural network in healthcare | |
US20100153390A1 (en) | Scoring Deportment and Comportment Cohorts | |
CN111344715A (en) | Object recognition system and method | |
JP2020502642A (en) | Patient identification system and method | |
Alvarez et al. | Multimodal monitoring of Parkinson's and Alzheimer's patients using the ICT4LIFE platform | |
Balasundaram et al. | Mask detection in crowded environment using machine learning | |
Sim et al. | Improving the accuracy of erroneous-plan recognition system for Activities of Daily Living | |
Al-Rammahi | Face mask recognition system using MobileNetV2 with optimization function | |
Singh et al. | Facial emotion recognition using convolutional neural network | |
Lamba et al. | Information retrieval from facial expression using voting to assert exigency | |
Khodabandehloo et al. | FreeSia: A Cyber-physical System for Cognitive Assessment through Frequency-domain Indoor Locomotion Analysis | |
Moyo et al. | COVID-19 Face Mask Detection Alert System | |
Sowmya et al. | Machine learning model for emotion detection and recognition using an enhanced Convolutional Neural Network | |
Dharanaesh et al. | Video based Facial Emotion Recognition System using Deep Learning | |
Pal et al. | Multimodal autism detection in children through facial and speech emotion | |
US12008880B2 (en) | Utilizing artificial intelligence to detect objects or patient safety events in a patient room |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 18809764 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
WPC | Withdrawal of priority claims after completion of the technical preparations for international publication |
Ref document number: 2017902032 Country of ref document: AU Date of ref document: 20191126 Free format text: WITHDRAWN AFTER TECHNICAL PREPARATION FINISHED Ref document number: 2017279806 Country of ref document: AU Date of ref document: 20191126 Free format text: WITHDRAWN AFTER TECHNICAL PREPARATION FINISHED |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 18809764 Country of ref document: EP Kind code of ref document: A1 |