EP2790140A1 - Queue analysis - Google Patents
Queue analysis Download PDFInfo
- Publication number
- EP2790140A1 EP2790140A1 EP20140163767 EP14163767A EP2790140A1 EP 2790140 A1 EP2790140 A1 EP 2790140A1 EP 20140163767 EP20140163767 EP 20140163767 EP 14163767 A EP14163767 A EP 14163767A EP 2790140 A1 EP2790140 A1 EP 2790140A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- queue
- image
- time
- trajectory
- images
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0631—Resource planning, allocation, distributing or scheduling for enterprises or organisations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
Definitions
- the present invention relates to a system and method for analysing queues and in particular for determining queue properties from image and video data.
- Queue members (in particular but not limited to people) move on a trajectory through a queue usually by new queue members joining the queue and the longest queuing members leaving the queue at the front when the purpose of the queue is fulfilled for them.
- Images are obtained or acquired from two or more positions within the queue. These may be still or moving images. As individual queue members move on their trajectory through the queue then they will be imaged at different times and appear in different images (different positions). The images are analysed and one or more individual queue members are identified in the images. When the same individual is identified in different images at different times then an inference may be made as to their movement and trajectory through the queue.
- each queue member will have substantially the same trajectory (although the queue may move faster or slower at different times). Therefore, determining the trajectory of one particular queue member provides an approximation of the progress or progression of the queue for a particular time period. Therefore, information about the queue, such as the total queuing time or the time taken to pass from one point in the queue to another, may be determined. Following different queue members through the queue may provide additional information and queuing statistics, such as the queue time at a particular time of day, for example.
- Improved queue time estimation or inference may be achieved by randomly, pseudo randomly or otherwise sampling two or more queue members (preferably visually stable queue members) to detect and track in order to yield a combined queue time estimation. Therefore, not all queue members need be detected and tracked (they may not all be visually distinct or stable due to occlusion and/or none distinctive clothing).
- a method of analysing a queue comprising the steps of:
- the method may further comprise the steps of obtaining further images of further positions within the queue and detecting the queue member within these further images. Additional images and queue member detections may improve the characterisation of the queue and its progress.
- the images overlap.
- the view of each image overlaps with at least one other image so that each portion of the queue may be monitored and analysed.
- positions within the queue are selected from the front of the queue, back of the queue, and the middle of the queue. There may be several positions within or at different stages along the queue. The actual or physical positions or locations in the queue may be measured and recorded.
- pairs or further pairs of images are obtained, used to detect the queue member, and determine their trajectory, wherein each pair of images includes at least one common point in the queue.
- the pairs of images may overlap or be separated.
- determining the trajectory may further include the step of forming the trajectory from two or more other trajectories. These other trajectories may be tracklets or partial trajectories within the queue.
- determining that the queue member detected within the second image is the same as the queue member detected within the first image may be based on similarities of visual appearances of the queue member within each image. Other criteria may be used.
- determining a trajectory of the queue member within the queue may further comprise the step of applying the Munkres (also known as the Hungarian) assignment algorithm to the two or more other trajectories and the similarities of the visual appearances of the queue member within each image.
- Munkres also known as the Hungarian
- Other algorithms may be used.
- detecting the queue member within an image further comprises selecting a group of pixels containing the queue member.
- a procedure may be executed to box or bound the queue member and isolate their image from the rest of the image in order to restrict processing or comparison to that particular portion.
- the images may be video images. Frames within the video images may be analysed or the analysis may be carried out on groups or moving images. Therefore, movement of individuals may be used as a further identifier of that individual or used in the comparison steps.
- determining a trajectory of the queue member within the queue may be further based on the queuing distance between the first position and the second position.
- the actual distance moved may also be considered.
- Distance may be in absolute terms (metres) but also in number of queue members (e.g. 10 people).
- the method may further comprise the step of determining from the trajectory a total queuing time for the queue member or queue. Other statistics may be determined or collected.
- the method according may further comprise iterating steps (a) to (f) to determine a plurality of trajectories. This improves accuracy as it facilitates averaging and other statistical techniques.
- Each of the plurality of trajectories may relate to a different queue member, for example.
- the method may further comprise determining an average trajectory from the plurality of trajectories.
- the method may further comprise the step of determining an average queue time from the average trajectory.
- the trajectory may include data describing how the queue member moves through the queue over time.
- a system for analysing a queue comprising a processor configured to:
- the system may further comprise a first and a second camera arranged to capture the first and second images, respectively.
- the images may be acquired from different sources including live images, recorded images, multiple cameras and the same camera.
- the first and second cameras may be arranged to have partially overlapping views of the queue.
- the system may further comprise a user interface configured to receive a signal indicating the first and the second positions of the queue on the first and second images. Therefore, the system may receive user input defining points or positions within the queue of interest or to define the extent of the queue.
- the first or second positions of the queue may be selected from: the start of the queue, one or more intermediate positions in the queue, and the end of the queue.
- the methods described above may be implemented as a computer program comprising program instructions to operate a computer.
- the computer program may be stored on a computer-readable medium or transmitted as a signal.
- system and method may be an automated visual analysis of queuing statistics within user-defined queuing regions spanning multiple and preferably contiguous camera views.
- the system and method allows analysis of people queuing and queue management. More specifically, the system facilitates interactive and automated analysis of video streams depicting people in queues for the derivation of queuing statistics (e.g. average queuing times), along with real-time graphical visualisation for human users.
- queuing statistics e.g. average queuing times
- the method and system may be interactive together with an automated queue analysis system configured to derive more accurate and useful statistics for queue management in a non-invasive manner.
- the method and system may operate by automatically detecting and tracking individuals within video streams from video cameras encompassing the length or substantially the length of a queue.
- the queue may be automatically or manually segmented into a fixed number of visual regions. Person tracks involve multiple associated individual detections (video frames) of the individual concerned, with transition times computed between the individual detections within a track and associated with the defined queue regions.
- the observed transition times for individual queue regions are then aggregated over time and overall queue estimates derived accordingly by integrating over individual queue region estimates.
- Corresponding plots may be generated for denoting observed overall queuing times within different time slots for human visualisation.
- the present method and system only requires two or more detections per individual in the form of partial "tracklets" encompassing a portion of the whole queue (i.e. less than the whole of the queue). This may be sufficient to derive useful statistics over time, and provides an advantage in enabling usability of the system in larger and more crowded environments where queuing individuals may at times undergo temporary occlusion.
- An interactive component may enable a human user to define the position and extent of a queue spanning multiple camera views, as well as to specify time slots for real-time visualisation of generated queuing statistics.
- the method may be arranged into several component steps. This is illustrated schematically as a process flow between different system components or modules in figure 1 .
- An interactive graphical user interface (GUI) based component 1 enables users to graphically delineate the position and extent of a queue spanning multiple partially-overlapping camera views.
- the interface 1 presents successive pairs of camera views on screen, which the user may mark with the mouse or other pointing device to indicate queue position landmarks corresponding to, for example: (a) the back of the queue (for the first camera view); (b) one or more connecting regions denoting the same physical position in intermediate pairs of camera views to characterise the "joins" between those views; and (c) the front of the queue (for the last camera view).
- Other landmarks may be defined.
- the resulting set of queue position landmarks indicates a region of interest for further analysis by describing a preferably seamless position and extent of the queue across the multiple camera views.
- FIG. 2 This is illustrated further in Figures 2 , where camera view 1 is of a first section of a queue and Figure 3 shows a camera view 2 that is of the last section of the queue.
- the front of the queue is marked as a first landmark 10.
- a middle region is marked as a second landmark 20.
- the end of the queue is marked as a third landmark 30.
- the middle region landmark 20 is visible in both camera views 1 and 2 (i.e. the views overlap).
- a processing component may consist of four modules for performing a four-stage person detection and analysis procedure.
- a module comprising a person detector 2 may denote the positions of individual people in each camera view. These denotations may take the form of a "bounding box" indicating a rectangular (or other shaped) region of pixels that make up a detected individual.
- Reference [1] illustrates a person detector 2 based on a mixtures of multiscale deformable part models, although other techniques may be used.
- the single component person model described with reference to figure 1 of reference [1] illustrates one type of person detector that may be used with the present system and method and the remainder of reference [1] provides technical details of an example person detector and associated mathematical functions. Reference [1] therefore provides the skilled person with the material and examples necessary to build a software-based person detector.
- Figure 1 shows a module for extracting a visual appearance descriptor of detected individuals 3, which may comprise, for example, 29 different types of texture and colour features. Examples of these features are shown in references [2] and [3]. These feature descriptors may be generated from image patches denoting individuals within the bounding boxes generated by the person detector 2 described above and in reference [1].
- Figure 1 also shows a module for associating individual detections on the basis of temporal, appearance and spatial consistency to derive "tracklets” characterising their partial trajectories along the user-defined queue path (module 4).
- the Munkres Assignment algorithm also known as the Hungarian algorithm (see references [4] and [5] for a detailed description of this algorithm) may be used by this module 4 to group the automatically detected individuals in the visual appearance descriptor 3 across different frames into small trajectory segments called “tracklets” (or sub-trajectories through the queue), based on space-time consistency (individuals' locations and speed of movement between frames) and similarity of appearance based on appearance descriptors generated in module 2.
- the result may be a description of an individual's position in multiple frames, capturing their movement along the queue over time.
- a module for merging "tracklets" on the basis of similarity in appearance and space-time consistency may also be provided.
- This module 5 may be used to combine multiple tracklets (or small portions of the queue) that correspond to the same individual to form a lager track or trajectory. More specifically, two or more tracklets may be merged by combining several measures of similarity, for example: (a) similarity of appearance; (b) consistency of position; and (c) consistency of speed [6]. If the similarity exceeds a fixed or predetermined threshold, then the tracklets may be merged into one. Such merged tracklets may be further combined if necessary. During system operation, each newly generated tracklet may be compared with a "pool" of all existing tracklets ( Figure 1 , tracklet pool 6) and either merged if suitably similar (as per the method above) or left to stand alone as part of the pool.
- An interactive processing system 7 is shown Figure 1 and is configured to derive statistical plots characterising queuing transition time trends, e.g. average queuing times, over user-defined time slots. More precisely, users interested in viewing queuing statistics may select, for example: (a) the time period within which to perform the analysis, such as between certain hours of the day or between certain months of the year; and (b) the time resolution of the plot, e.g. on a daily basis or an hourly basis. This component may then update the queuing statistics by analysing the tracklets generated over the selected time period, and generates an on-screen visualisation in the form of a bar graph or other display denoting the average queuing times observed over those time periods with a selectable time resolution.
- This component may then update the queuing statistics by analysing the tracklets generated over the selected time period, and generates an on-screen visualisation in the form of a bar graph or other display denoting the average queuing times observed over those time periods with a selectable time resolution.
- Figure 4 illustrates a system 100 for operating the method described above.
- video feeds are provided by cameras 110.
- a computer 120 executes the functionality of modules 1-7 described with reference to Figure 1 .
- Display 120 is used to define the queue landmarks and to provide queuing statistics and other data to be visualised.
- Data store 140 is used to record the video feeds, where necessary, and to record calculated results and statistics.
- Pointing device (mouse 150) is used by a human user to interact with the system 100 (e.g. to define queue landmarks).
- the computer may operate a suitable operating system such as Linux, UNIX, MS Windows, or Apple OSX, for example.
- the computer system and cameras 110 may be linked over a network, such as the internet or an intranet, for example.
- detection, extraction and matching of queue members may use a parts-based person detector [1] to detect individuals (queue members) and produce bounding boxes indicating the position and size of individuals in image frames. Image patches within these bounding boxes may then each be split into a number of equal horizontal segments (for example six). Within each segment, a comprehensive set of types of visual features are extracted (for example 29 types), encompassing the colour and texture appearance of individuals for matching [2]. More specifically, these colour features incorporate different colour spaces including RGB, Hue-Saturation and YCrCb, with texture features derived from Gabor wavelet responses at eight different scales and orientations, as well as thirteen differently parameterised Schmid Filters [9], for example.
- Normalised histograms are generated capturing the statistics of these features for each horizontal segment, and then concatenated into a single feature vector. Given 16 bins for the histogram corresponding to each of the 29 feature types for each of the six horizontal strips, we thus have a 2784-dimensional feature vector per bounding box, which may be used as an appearance descriptor. Rather than consider each feature type equally in terms of relevance, the system may dynamically learn the importance of each of these feature types to more strongly weight those features most relevant for matching across different cameras [2, 3, 10]. The resulting model may take the form of a support vector machine (SVM) known as RankSVM [8, 7]. This model may be used to compute matching scores between the appearance descriptors of detected individuals.
- SVM support vector machine
- the Munkres Assignment algorithm also known as the Hungarian algorithm [5, 4], may be employed as part of a multi-target tracking scheme to locally group detections in different frames as likely belonging to the same person or queue member. This process may yield tracklets encompassing individual detections over multiple frames.
- this cost is computed as a weighted combination of appearance descriptor dissimilarity and physical pixel distance.
- Predicted pixel locations ⁇ n,f for frame f are estimated by assuming constant linear velocity from the last known location and velocity.
- the Munkres Assignment algorithm maps rows to columns in C so as to minimise the cost, with each detection added accordingly to their mapped incomplete tracklets. Surplus detections are used to initiate new tracklets. In practice, an upper bound is placed on cost, with assignments exceeding the upper bound retracted and the detection concerned treated as surplus. Additionally, tracklets which have not been updated for a length of time may be treated as complete.
Landscapes
- Business, Economics & Management (AREA)
- Human Resources & Organizations (AREA)
- Engineering & Computer Science (AREA)
- Strategic Management (AREA)
- Entrepreneurship & Innovation (AREA)
- Economics (AREA)
- Operations Research (AREA)
- Game Theory and Decision Science (AREA)
- Development Economics (AREA)
- Marketing (AREA)
- Educational Administration (AREA)
- Quality & Reliability (AREA)
- Tourism & Hospitality (AREA)
- Physics & Mathematics (AREA)
- General Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
Abstract
Description
- The present invention relates to a system and method for analysing queues and in particular for determining queue properties from image and video data.
- Many organisations dealing with the public are required to manage the length of time people are required to queue for services, for example customers in a bank or passengers going through security checks at an airport. In order to conduct such queue management more effectively, they require statistical estimates of queuing times experienced by people within different time slots throughout the day, so as to optimise their operations and minimise queuing times accordingly.
- Currently, techniques employed for monitoring queue lengths and estimating queuing times in airports involve a combination of laser-based passenger counting systems (which cannot track specific individuals) and Bluetooth signal-tracking (which requires passengers to have an active Bluetooth device about their person). Such systems cannot be installed in all areas and are highly inaccurate. Most visual sensor based systems require the ability to accurately track all individuals in a queue in order to derive useful statistics, an unreasonable and unworkable assumption in even modest real-world scenarios where imaging conditions and crowd behaviour is relatively unrestricted.
- Therefore, there is required a system and method that overcomes these problems.
- Queue members (in particular but not limited to people) move on a trajectory through a queue usually by new queue members joining the queue and the longest queuing members leaving the queue at the front when the purpose of the queue is fulfilled for them.
- Images are obtained or acquired from two or more positions within the queue. These may be still or moving images. As individual queue members move on their trajectory through the queue then they will be imaged at different times and appear in different images (different positions). The images are analysed and one or more individual queue members are identified in the images. When the same individual is identified in different images at different times then an inference may be made as to their movement and trajectory through the queue.
- In general, each queue member will have substantially the same trajectory (although the queue may move faster or slower at different times). Therefore, determining the trajectory of one particular queue member provides an approximation of the progress or progression of the queue for a particular time period. Therefore, information about the queue, such as the total queuing time or the time taken to pass from one point in the queue to another, may be determined. Following different queue members through the queue may provide additional information and queuing statistics, such as the queue time at a particular time of day, for example.
- Improved queue time estimation or inference may be achieved by randomly, pseudo randomly or otherwise sampling two or more queue members (preferably visually stable queue members) to detect and track in order to yield a combined queue time estimation. Therefore, not all queue members need be detected and tracked (they may not all be visually distinct or stable due to occlusion and/or none distinctive clothing).
- In accordance with a first aspect there is provided a method of analysing a queue comprising the steps of:
- (a) obtaining a first image acquired at a first time of a first position within a queue;
- (b) obtaining a second image acquired at a second time of a second position within the queue;
- (c) detecting a queue member within the first image;
- (d) detecting a queue member with the second image;
- (e) determining that the queue member detected within the second image is the same as the queue member detected within the first image; and
- (f) determining a trajectory of the queue member within the queue based on a difference between the first time and the second time.
- Optionally, the method may further comprise the steps of obtaining further images of further positions within the queue and detecting the queue member within these further images. Additional images and queue member detections may improve the characterisation of the queue and its progress.
- Preferably, the images overlap. In other words, the view of each image overlaps with at least one other image so that each portion of the queue may be monitored and analysed.
- Preferably, positions within the queue are selected from the front of the queue, back of the queue, and the middle of the queue. There may be several positions within or at different stages along the queue. The actual or physical positions or locations in the queue may be measured and recorded.
- Optionally, pairs or further pairs of images are obtained, used to detect the queue member, and determine their trajectory, wherein each pair of images includes at least one common point in the queue. The pairs of images may overlap or be separated.
- Optionally, determining the trajectory may further include the step of forming the trajectory from two or more other trajectories. These other trajectories may be tracklets or partial trajectories within the queue.
- Optionally, determining that the queue member detected within the second image is the same as the queue member detected within the first image may be based on similarities of visual appearances of the queue member within each image. Other criteria may be used.
- Advantageously, determining a trajectory of the queue member within the queue may further comprise the step of applying the Munkres (also known as the Hungarian) assignment algorithm to the two or more other trajectories and the similarities of the visual appearances of the queue member within each image. Other algorithms may be used.
- Preferably, detecting the queue member within an image further comprises selecting a group of pixels containing the queue member. In other words, a procedure may be executed to box or bound the queue member and isolate their image from the rest of the image in order to restrict processing or comparison to that particular portion.
- Preferably, the images may be video images. Frames within the video images may be analysed or the analysis may be carried out on groups or moving images. Therefore, movement of individuals may be used as a further identifier of that individual or used in the comparison steps.
- Preferably, determining a trajectory of the queue member within the queue may be further based on the queuing distance between the first position and the second position. In other words, as well as the time being taken into consideration in determining movement within the queue of queue members, the actual distance moved (and perhaps the total length of the queue) may also be considered. Distance may be in absolute terms (metres) but also in number of queue members (e.g. 10 people).
- Optionally, the method may further comprise the step of determining from the trajectory a total queuing time for the queue member or queue. Other statistics may be determined or collected.
- Optionally, the method according may further comprise iterating steps (a) to (f) to determine a plurality of trajectories. This improves accuracy as it facilitates averaging and other statistical techniques. Each of the plurality of trajectories may relate to a different queue member, for example.
- Optionally, the method may further comprise determining an average trajectory from the plurality of trajectories.
- Optionally, the method may further comprise the step of determining an average queue time from the average trajectory.
- Preferably, the trajectory may include data describing how the queue member moves through the queue over time.
- According to a second aspect, there is provided a system for analysing a queue comprising a processor configured to:
- (a) receive a first image acquired at a first time of a first position within a queue;
- (b) receive a second image acquired at a second time of a second position within the queue;
- (c) detect a queue member within the first image;
- (d) detect a queue member with the second image;
- (e) determine that the queue member detected within the second image is the same as the queue member detected within the first image; and
- (f) determine a trajectory of the queue member within the queue based on a difference between the first time and the second time.
- Preferably, the system may further comprise a first and a second camera arranged to capture the first and second images, respectively. The images may be acquired from different sources including live images, recorded images, multiple cameras and the same camera.
- Optionally, the first and second cameras may be arranged to have partially overlapping views of the queue.
- Preferably, the system may further comprise a user interface configured to receive a signal indicating the first and the second positions of the queue on the first and second images. Therefore, the system may receive user input defining points or positions within the queue of interest or to define the extent of the queue.
- Optionally, the first or second positions of the queue may be selected from: the start of the queue, one or more intermediate positions in the queue, and the end of the queue.
- The methods described above may be implemented as a computer program comprising program instructions to operate a computer. The computer program may be stored on a computer-readable medium or transmitted as a signal.
- It should be noted that any feature described above may be used with any particular aspect or embodiment of the invention.
- The following numbered clauses provide illustrative examples:
- 1. A method of analysing a queue comprising the steps of:
- (a) obtaining a first image acquired at a first time of a first position within a queue;
- (b) obtaining a second image acquired at a second time of a second position within the queue;
- (c) detecting a queue member within the first image;
- (d) detecting a queue member with the second image;
- (e) determining that the queue member detected within the second image is the same as the queue member detected within the first image; and
- (f) determining a trajectory of the queue member within the queue based on a difference between the first time and the second time.
- 2. The method of
clause 1 further comprises the steps of obtaining further images of further positions within the queue and detecting the queue member within these further images. - 3. The method of
clause 1 orclause 2, wherein the images overlap. - 4. The method according to any previous clause, wherein the positions within the queue are selected from the front of the queue, back of the queue, and the middle of the queue.
- 5. The method according to any previous clause, wherein further pairs of images are obtained, used to detect the queue member, and determine their trajectory, wherein each pair of images includes at least one common point in the queue.
- 6. The method according to any previous clause, wherein determining the trajectory further includes the step of forming the trajectory from two or more other trajectories.
- 7. The method according to clause 6, wherein determining that the queue member detected within the second image is the same as the queue member detected within the first image is based on similarities of visual appearances of the queue member within each image.
- 8. The method of
clause 7, wherein determining a trajectory of the queue member within the queue further comprises the step of applying the Munkres assignment algorithm to the two or more other trajectories and the similarities of the visual appearances of the queue member within each image. - 9. The method according to any previous clause, wherein detecting the queue member within an image further comprises selecting a group of pixels containing the queue member.
- 10. The method according to any previous clause, wherein the images are video images.
- 11. The method according to any previous clause, wherein determining a trajectory of the queue member within the queue is further based on the queuing distance between the first position and the second position.
- 12. The method according to any previous clause further comprising the step of determining from the trajectory a total queuing time for the queue member.
- 13. The method according to any previous clause further comprising iterating steps (a) to (f) to determine a plurality of trajectories.
- 14. The method of clause 13 further comprising determining an average trajectory from the plurality of trajectories.
- 15. The method of clause 14, further comprising the step of determining an average queue time from the average trajectory.
- 16. The method according to any previous clause, wherein the trajectory includes data describing how the queue member moves through the queue over time.
- 17. A system for analysing a queue comprising a processor configured to:
- (a) receive a first image acquired at a first time of a first position within a queue;
- (b) receive a second image acquired at a second time of a second position within the queue;
- (c) detect a queue member within the first image;
- (d) detect a queue member with the second image;
- (e) determine that the queue member detected within the second image is the same as the queue member detected within the first image; and
- (f) determine a trajectory of the queue member within the queue based on a difference between the first time and the second time.
- 18. The system of clause 17 further comprising a first and a second camera arranged to capture the first and second images, respectively.
- 19. The system of clause 18, wherein the first and second cameras are arranged to have partially overlapping views of the queue.
- 20. The system according to any of clauses 17 to 19 further comprising a user interface configured to receive a signal indicating the first and the second positions of the queue on the first and second images.
- 21. The system of
clause 20, wherein the first or second positions of the queue are selected from: the start of the queue, an intermediate position in the queue, and the end of the queue. - 22. A method substantially as described with reference to any of the accompanying drawings.
- 23. An apparatus substantially as described and shown in any of the accompanying drawings.
- 24. A computer program comprising program instructions that, when executed on a computer cause the computer to perform the method of any of
clause 1 to 16. - 25. A computer-readable medium carrying a computer program according to clause 24.
- 26. A computer programmed to perform the method of any of
clauses 1 to 16. - The present invention may be put into practice in a number of ways and embodiments will now be described by way of example only and with reference to the accompanying drawings, in which:
-
Fig. 1 shows a flow diagram of a method for analysing a queue, given by way of example only; -
Fig. 2 shows a schematic diagram of a camera view of a first part of a queue; -
Fig. 3 shows a schematic diagram of a camera view of a second part of the queue offigure 2 ; and -
Fig. 4 shows a schematic diagram of a system for analysing the queue. - It should be noted that the figures are illustrated for simplicity and are not necessarily drawn to scale.
- In one example implementation, the system and method may be an automated visual analysis of queuing statistics within user-defined queuing regions spanning multiple and preferably contiguous camera views.
- The system and method allows analysis of people queuing and queue management. More specifically, the system facilitates interactive and automated analysis of video streams depicting people in queues for the derivation of queuing statistics (e.g. average queuing times), along with real-time graphical visualisation for human users.
- The method and system may be interactive together with an automated queue analysis system configured to derive more accurate and useful statistics for queue management in a non-invasive manner. The method and system may operate by automatically detecting and tracking individuals within video streams from video cameras encompassing the length or substantially the length of a queue. The queue may be automatically or manually segmented into a fixed number of visual regions. Person tracks involve multiple associated individual detections (video frames) of the individual concerned, with transition times computed between the individual detections within a track and associated with the defined queue regions. The observed transition times for individual queue regions are then aggregated over time and overall queue estimates derived accordingly by integrating over individual queue region estimates. Corresponding plots may be generated for denoting observed overall queuing times within different time slots for human visualisation.
- Deriving such estimates may be obtained in this way by relaxing requirements imposed by other prior art visual sensor based systems to accurately track every individual in the queue for the duration of their presence; i.e. prior art techniques require the detection of a queue member in every video frame and then generate a complete track characterising their movement along the queue in its entirety.
- In contrast, the present method and system only requires two or more detections per individual in the form of partial "tracklets" encompassing a portion of the whole queue (i.e. less than the whole of the queue). This may be sufficient to derive useful statistics over time, and provides an advantage in enabling usability of the system in larger and more crowded environments where queuing individuals may at times undergo temporary occlusion.
- An interactive component may enable a human user to define the position and extent of a queue spanning multiple camera views, as well as to specify time slots for real-time visualisation of generated queuing statistics.
- The method may be arranged into several component steps. This is illustrated schematically as a process flow between different system components or modules in
figure 1 . - An interactive graphical user interface (GUI) based
component 1 enables users to graphically delineate the position and extent of a queue spanning multiple partially-overlapping camera views. Theinterface 1 presents successive pairs of camera views on screen, which the user may mark with the mouse or other pointing device to indicate queue position landmarks corresponding to, for example: (a) the back of the queue (for the first camera view); (b) one or more connecting regions denoting the same physical position in intermediate pairs of camera views to characterise the "joins" between those views; and (c) the front of the queue (for the last camera view). Other landmarks may be defined. The resulting set of queue position landmarks indicates a region of interest for further analysis by describing a preferably seamless position and extent of the queue across the multiple camera views. This is illustrated further inFigures 2 , wherecamera view 1 is of a first section of a queue andFigure 3 shows acamera view 2 that is of the last section of the queue. The front of the queue is marked as afirst landmark 10. A middle region is marked as asecond landmark 20. The end of the queue is marked as athird landmark 30. Themiddle region landmark 20 is visible in bothcamera views 1 and 2 (i.e. the views overlap). - A processing component may consist of four modules for performing a four-stage person detection and analysis procedure.
- A module comprising a
person detector 2 may denote the positions of individual people in each camera view. These denotations may take the form of a "bounding box" indicating a rectangular (or other shaped) region of pixels that make up a detected individual. Reference [1] illustrates aperson detector 2 based on a mixtures of multiscale deformable part models, although other techniques may be used. The single component person model described with reference tofigure 1 of reference [1] illustrates one type of person detector that may be used with the present system and method and the remainder of reference [1] provides technical details of an example person detector and associated mathematical functions. Reference [1] therefore provides the skilled person with the material and examples necessary to build a software-based person detector. -
Figure 1 shows a module for extracting a visual appearance descriptor of detectedindividuals 3, which may comprise, for example, 29 different types of texture and colour features. Examples of these features are shown in references [2] and [3]. These feature descriptors may be generated from image patches denoting individuals within the bounding boxes generated by theperson detector 2 described above and in reference [1]. -
Figure 1 also shows a module for associating individual detections on the basis of temporal, appearance and spatial consistency to derive "tracklets" characterising their partial trajectories along the user-defined queue path (module 4). For example, the Munkres Assignment algorithm, also known as the Hungarian algorithm (see references [4] and [5] for a detailed description of this algorithm) may be used by this module 4 to group the automatically detected individuals in thevisual appearance descriptor 3 across different frames into small trajectory segments called "tracklets" (or sub-trajectories through the queue), based on space-time consistency (individuals' locations and speed of movement between frames) and similarity of appearance based on appearance descriptors generated inmodule 2. The result may be a description of an individual's position in multiple frames, capturing their movement along the queue over time. - A module for merging "tracklets" on the basis of similarity in appearance and space-time consistency (module 5) may also be provided. This module 5 may be used to combine multiple tracklets (or small portions of the queue) that correspond to the same individual to form a lager track or trajectory. More specifically, two or more tracklets may be merged by combining several measures of similarity, for example: (a) similarity of appearance; (b) consistency of position; and (c) consistency of speed [6]. If the similarity exceeds a fixed or predetermined threshold, then the tracklets may be merged into one. Such merged tracklets may be further combined if necessary. During system operation, each newly generated tracklet may be compared with a "pool" of all existing tracklets (
Figure 1 , tracklet pool 6) and either merged if suitably similar (as per the method above) or left to stand alone as part of the pool. - An
interactive processing system 7 is shownFigure 1 and is configured to derive statistical plots characterising queuing transition time trends, e.g. average queuing times, over user-defined time slots. More precisely, users interested in viewing queuing statistics may select, for example: (a) the time period within which to perform the analysis, such as between certain hours of the day or between certain months of the year; and (b) the time resolution of the plot, e.g. on a daily basis or an hourly basis. This component may then update the queuing statistics by analysing the tracklets generated over the selected time period, and generates an on-screen visualisation in the form of a bar graph or other display denoting the average queuing times observed over those time periods with a selectable time resolution. - These components enable more robust interactive real-time queue statistical analysis and graphical visualisation for human users in large crowded areas. It operates by automatically detecting and tracking individuals within video streams from video cameras encompassing the length of a queue, which may be automatically or manually segmented into a fixed number of visual regions. Person or queue member tracks may involve multiple associated individual detections (video frames) of the individual concerned, with transition times computed between the individual detections within a track and associated with the defined queue regions. The observed transition times for individual queue regions may then be aggregated over time and overall queue estimates derived accordingly by integrating over individual queue region estimates. Corresponding plots are generated for denoting observed overall queuing times within different time slots for human visualisation. This process continues indefinitely with real-time refinement of statistics and visualisation.
-
Figure 4 illustrates asystem 100 for operating the method described above. In particular, video feeds are provided bycameras 110. Acomputer 120 executes the functionality of modules 1-7 described with reference toFigure 1 .Display 120 is used to define the queue landmarks and to provide queuing statistics and other data to be visualised.Data store 140 is used to record the video feeds, where necessary, and to record calculated results and statistics. Pointing device (mouse 150) is used by a human user to interact with the system 100 (e.g. to define queue landmarks). The computer may operate a suitable operating system such as Linux, UNIX, MS Windows, or Apple OSX, for example. The computer system andcameras 110 may be linked over a network, such as the internet or an intranet, for example. - In one example implementation (others may be used), detection, extraction and matching of queue members may use a parts-based person detector [1] to detect individuals (queue members) and produce bounding boxes indicating the position and size of individuals in image frames. Image patches within these bounding boxes may then each be split into a number of equal horizontal segments (for example six). Within each segment, a comprehensive set of types of visual features are extracted (for example 29 types), encompassing the colour and texture appearance of individuals for matching [2]. More specifically, these colour features incorporate different colour spaces including RGB, Hue-Saturation and YCrCb, with texture features derived from Gabor wavelet responses at eight different scales and orientations, as well as thirteen differently parameterised Schmid Filters [9], for example. Normalised histograms are generated capturing the statistics of these features for each horizontal segment, and then concatenated into a single feature vector. Given 16 bins for the histogram corresponding to each of the 29 feature types for each of the six horizontal strips, we thus have a 2784-dimensional feature vector per bounding box, which may be used as an appearance descriptor. Rather than consider each feature type equally in terms of relevance, the system may dynamically learn the importance of each of these feature types to more strongly weight those features most relevant for matching across different cameras [2, 3, 10]. The resulting model may take the form of a support vector machine (SVM) known as RankSVM [8, 7]. This model may be used to compute matching scores between the appearance descriptors of detected individuals.
- The Munkres Assignment algorithm, also known as the Hungarian algorithm [5, 4], may be employed as part of a multi-target tracking scheme to locally group detections in different frames as likely belonging to the same person or queue member. This process may yield tracklets encompassing individual detections over multiple frames. An individual queue member D is accordingly represented as a trackelt TD = {αD,1, ..., αD,J } comprising J individual detections with appearance descriptors αD,j .
- More precisely, tracklets may be built up incrementally over time, with an incomplete set updated after each frame by assigning individual detections from that frame to a tracklet according to their appearance similarity and spatial proximity. That is, given: (1) a set S = {α1,f ,...,α M,f } of M appearance descriptors for detections in frame f with corresponding pixel locations {β 1,f , ..., βM,f }; (2) a current set of N incomplete tracklets R = {T̂ 1,..., T̂N } with their most recently added appearance descriptors {α̂n,f
n }; and (3) corresponding predicted pixel locations {β̂n,f }, an M x N cost matrix C is generated where each entry C m,n is computed as: - In essence, this cost is computed as a weighted combination of appearance descriptor dissimilarity and physical pixel distance. Predicted pixel locations β̂n,f for frame f are estimated by assuming constant linear velocity from the last known location and velocity. The Munkres Assignment algorithm maps rows to columns in C so as to minimise the cost, with each detection added accordingly to their mapped incomplete tracklets. Surplus detections are used to initiate new tracklets. In practice, an upper bound is placed on cost, with assignments exceeding the upper bound retracted and the detection concerned treated as surplus. Additionally, tracklets which have not been updated for a length of time may be treated as complete.
-
- [1] P. Felzenszwalb, R. Girshick, D. McAllester, and D. Ramanan. Object Detection with Discriminatively Trained Part Based Models. In IEEE Transaction on Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010.
- [2] B. Prosser, W. Zheng, S. Gong, and T. Xiang. Person Re-Identification by Support Vector Ranking. In British Machine Vision Conference, Sep. 2010.
- [3] W. Zheng, S. Gong, and T. Xiang. Person re-identification by Probabilistic Relative Distance Comparison. In IEEE Conference on Computer Vision and Pattern Recognition, June 2011.
- [4] J. Munkres. Algorithms for the Assignment and Transportation Problems. In Journal of the Society for Industrial and Applied Mathematics, 5(1):32-28, 1957.
- [5] H.W. Kuhn. The Hungarian Method for the Assignment Problem. In Naval Research Logistics Quarterly, 2:83-97, 1955.
- [6] J. Prokaj, M. Duchaineau, and G. Medioni. Inferring Tracklets for Multi-Object Tracking. In IEEE Conference on Computer Vision and Pattern Recognition, 2011.
- [7] O. Chapelle and S. Keerthi. Efficient algorithms for ranking with SVMs. In Information Retrieval, volume 32, 2010.
- [8] T. Joachims. Optimizing search engines using clickthrough data. In Knowledge Discovery and Data Mining, pages 133-142, 2010.
- [9] C. Schmid. Constructing models for content-based image retrieval. In IEEE Conference on Computer Vision and Pattern Recognition, pages 30{45, 2001.
- [10] W. Zheng, S. Gong, and T. Xiang. Re-identification by relative distance comparison. In IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(3):653-668, March 2013.
- As will be appreciated by the skilled person, details of the above embodiment may be varied without departing from the scope of the present invention, as defined by the appended claims.
- Many combinations, modifications, or alterations to the features of the above embodiments will be readily apparent to the skilled person and are intended to form part of the invention. Any of the features described specifically relating to one embodiment or example may be used in any other embodiment by making the appropriate changes.
Claims (15)
- A method of analysing a queue comprising the steps of:(a) obtaining a first image acquired at a first time of a first position within a queue;(b) obtaining a second image acquired at a second time of a second position within the queue;(c) detecting a queue member within the first image;(d) detecting a queue member with the second image;(e) determining that the queue member detected within the second image is the same as the queue member detected within the first image; and(f) determining a trajectory of the queue member within the queue based on a difference between the first time and the second time.
- The method of claim 1 further comprises the steps of obtaining further images of further positions within the queue and detecting the queue member within these further images.
- The method of claim 1 or claim 2, wherein the images overlap.
- The method according to any previous claim, wherein further pairs of images are obtained, used to detect the queue member, and determine their trajectory, wherein each pair of images includes at least one common point in the queue.
- The method according to any previous claim, wherein determining the trajectory further includes the step of forming the trajectory from two or more other trajectories.
- The method according to claim 5, wherein determining that the queue member detected within the second image is the same as the queue member detected within the first image is based on similarities of visual appearances of the queue member within each image.
- The method of claim 6, wherein determining a trajectory of the queue member within the queue further comprises the step of applying the Munkres assignment algorithm to the two or more other trajectories and the similarities of the visual appearances of the queue member within each image.
- The method according to any previous claim, wherein detecting the queue member within an image further comprises selecting a group of pixels containing the queue member.
- The method according to any previous claim, wherein determining a trajectory of the queue member within the queue is further based on the queuing distance between the first position and the second position.
- The method according to any previous claim further comprising iterating steps (a) to (f) to determine a plurality of trajectories.
- A system for analysing a queue comprising a processor configured to:(a) receive a first image acquired at a first time of a first position within a queue;(b) receive a second image acquired at a second time of a second position within the queue;(c) detect a queue member within the first image;(d) detect a queue member with the second image;(e) determine that the queue member detected within the second image is the same as the queue member detected within the first image; and(f) determine a trajectory of the queue member within the queue based on a difference between the first time and the second time.
- The system of claim 11 further comprising a first and a second camera arranged to capture the first and second images, respectively.
- The system of claim 12, wherein the first and second cameras are arranged to have partially overlapping views of the queue.
- The system according to any of claims 11 to 13 further comprising a user interface configured to receive a signal indicating the first and the second positions of the queue on the first and second images.
- A computer-readable medium carrying a computer program comprising program instructions that, when executed on a computer cause the computer to perform the method of any of claims 1 to 10.
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
GBGB1306313.6A GB201306313D0 (en) | 2013-04-08 | 2013-04-08 | Queue analysis |
Publications (1)
Publication Number | Publication Date |
---|---|
EP2790140A1 true EP2790140A1 (en) | 2014-10-15 |
Family
ID=48483523
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP20140163767 Ceased EP2790140A1 (en) | 2013-04-08 | 2014-04-07 | Queue analysis |
Country Status (4)
Country | Link |
---|---|
US (1) | US9245247B2 (en) |
EP (1) | EP2790140A1 (en) |
GB (1) | GB201306313D0 (en) |
HK (1) | HK1200230A1 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107066917A (en) * | 2015-12-03 | 2017-08-18 | 国际联合航空集团 | Queue monitor based on marking profile |
CN109840982A (en) * | 2019-01-02 | 2019-06-04 | 京东方科技集团股份有限公司 | It is lined up recommended method and device, computer readable storage medium |
Families Citing this family (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9965684B2 (en) * | 2014-12-18 | 2018-05-08 | Sensormatic Electronics, LLC | Method and system for queue length analysis |
JP6852675B2 (en) * | 2015-09-14 | 2021-03-31 | 日本電気株式会社 | Matrix detection system, method and program |
JP7158828B2 (en) * | 2017-05-31 | 2022-10-24 | キヤノン株式会社 | Information processing device, information processing method and program |
US11301692B2 (en) * | 2017-06-30 | 2022-04-12 | Nec Corporation | Information processing apparatus, control method, and program |
US10867393B2 (en) * | 2018-03-22 | 2020-12-15 | Texas Instruments Incorporated | Video object detection |
JP7237467B2 (en) * | 2018-05-30 | 2023-03-13 | キヤノン株式会社 | Information processing device, information processing method, and program |
CN109959378A (en) * | 2019-03-08 | 2019-07-02 | 联创汽车电子有限公司 | Multiple target tracking accuracy computation system and its calculation method |
US11120280B2 (en) | 2019-11-15 | 2021-09-14 | Argo AI, LLC | Geometry-aware instance segmentation in stereo image capture processes |
KR20230054182A (en) * | 2021-10-15 | 2023-04-24 | 주식회사 알체라 | Person re-identification method using artificial neural network and computing apparatus for performing the same |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070253595A1 (en) * | 2006-04-18 | 2007-11-01 | Sorensen Associates Inc | Still Image Queue Analysis System and Method |
US8224028B1 (en) * | 2008-05-02 | 2012-07-17 | Verint Systems Ltd. | System and method for queue analysis using video analytics |
US20130070974A1 (en) * | 2011-09-16 | 2013-03-21 | Arinc Incorporated | Method and apparatus for facial recognition based queue time tracking |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5581625A (en) | 1994-01-31 | 1996-12-03 | International Business Machines Corporation | Stereo vision system for counting items in a queue |
US5953055A (en) | 1996-08-08 | 1999-09-14 | Ncr Corporation | System and method for detecting and analyzing a queue |
US8010402B1 (en) * | 2002-08-12 | 2011-08-30 | Videomining Corporation | Method for augmenting transaction data with visually extracted demographics of people using computer vision |
US8107676B2 (en) * | 2007-07-30 | 2012-01-31 | International Business Machines Corporation | Line length estimation |
DE102009028604A1 (en) * | 2009-08-18 | 2011-02-24 | Robert Bosch Gmbh | Device snake recognition device, method and computer program |
US8438175B2 (en) * | 2010-03-17 | 2013-05-07 | Lighthaus Logic Inc. | Systems, methods and articles for video analysis reporting |
US9443148B2 (en) * | 2013-03-15 | 2016-09-13 | International Business Machines Corporation | Visual monitoring of queues using auxiliary devices |
-
2013
- 2013-04-08 GB GBGB1306313.6A patent/GB201306313D0/en not_active Ceased
-
2014
- 2014-04-07 EP EP20140163767 patent/EP2790140A1/en not_active Ceased
- 2014-04-08 US US14/247,412 patent/US9245247B2/en active Active
-
2015
- 2015-01-15 HK HK15100482.8A patent/HK1200230A1/en unknown
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070253595A1 (en) * | 2006-04-18 | 2007-11-01 | Sorensen Associates Inc | Still Image Queue Analysis System and Method |
US8224028B1 (en) * | 2008-05-02 | 2012-07-17 | Verint Systems Ltd. | System and method for queue analysis using video analytics |
US20130070974A1 (en) * | 2011-09-16 | 2013-03-21 | Arinc Incorporated | Method and apparatus for facial recognition based queue time tracking |
Non-Patent Citations (1)
Title |
---|
AUBERT D: "Passengers queue length measurement", IMAGE ANALYSIS AND PROCESSING, 1999. PROCEEDINGS. INTERNATIONAL CONFER ENCE ON VENICE, ITALY 27-29 SEPT. 1999, LOS ALAMITOS, CA, USA,IEEE COMPUT. SOC, US, 27 September 1999 (1999-09-27), pages 1132 - 1135, XP010354339, ISBN: 978-0-7695-0040-9, DOI: 10.1109/ICIAP.1999.797754 * |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107066917A (en) * | 2015-12-03 | 2017-08-18 | 国际联合航空集团 | Queue monitor based on marking profile |
CN109840982A (en) * | 2019-01-02 | 2019-06-04 | 京东方科技集团股份有限公司 | It is lined up recommended method and device, computer readable storage medium |
US11488321B2 (en) | 2019-01-02 | 2022-11-01 | Beijing Boe Technology Development Co., Ltd. | Queuing recommendation method and device, terminal and computer readable storage medium |
Also Published As
Publication number | Publication date |
---|---|
US9245247B2 (en) | 2016-01-26 |
HK1200230A1 (en) | 2015-07-31 |
US20140301602A1 (en) | 2014-10-09 |
GB201306313D0 (en) | 2013-05-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9245247B2 (en) | Queue analysis | |
US9117147B2 (en) | Marginal space learning for multi-person tracking over mega pixel imagery | |
US9953215B2 (en) | Method and system of temporal segmentation for movement analysis | |
CN105378752B (en) | The on-line study system for detecting and counting for people | |
CN110378259A (en) | A kind of multiple target Activity recognition method and system towards monitor video | |
Parham et al. | Animal population censusing at scale with citizen science and photographic identification | |
US20130155229A1 (en) | Assisted video surveillance of persons-of-interest | |
CN110781839A (en) | Sliding window-based small and medium target identification method in large-size image | |
Drusch et al. | Analysing eye-tracking data: From scanpaths and heatmaps to the dynamic visualisation of areas of interest | |
JP6120404B2 (en) | Mobile body behavior analysis / prediction device | |
CN109829382B (en) | Abnormal target early warning tracking system and method based on intelligent behavior characteristic analysis | |
JP2012243313A (en) | Image processing method and image processing device | |
KR101645959B1 (en) | The Apparatus and Method for Tracking Objects Based on Multiple Overhead Cameras and a Site Map | |
CN105389562A (en) | Secondary optimization method for monitoring video pedestrian re-identification result based on space-time constraint | |
WO2020079877A1 (en) | System and method for video anomaly detection and storage medium | |
WO2022078134A1 (en) | People traffic analysis method and system, electronic device, and readable storage medium | |
Lin et al. | Cross camera people counting with perspective estimation and occlusion handling | |
US20170053172A1 (en) | Image processing apparatus, and image processing method | |
CN115862113A (en) | Stranger abnormity identification method, device, equipment and storage medium | |
JP6516702B2 (en) | People count system, number count method, and view method of number count result | |
CN109977796A (en) | Trail current detection method and device | |
Zhang et al. | A Multiple Instance Learning and Relevance Feedback Framework for Retrieving Abnormal Incidents in Surveillance Videos. | |
CN106446837B (en) | A kind of detection method of waving based on motion history image | |
KR102460899B1 (en) | Method and System for People Count based on Deep Learning | |
Rollakanti et al. | Moving Object Detection using Background Subtraction |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
17P | Request for examination filed |
Effective date: 20140407 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
R17P | Request for examination filed (corrected) |
Effective date: 20150415 |
|
RBV | Designated contracting states (corrected) |
Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 1200230 Country of ref document: HK |
|
17Q | First examination report despatched |
Effective date: 20150819 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
APBK | Appeal reference recorded |
Free format text: ORIGINAL CODE: EPIDOSNREFNE |
|
APBN | Date of receipt of notice of appeal recorded |
Free format text: ORIGINAL CODE: EPIDOSNNOA2E |
|
APBR | Date of receipt of statement of grounds of appeal recorded |
Free format text: ORIGINAL CODE: EPIDOSNNOA3E |
|
APAF | Appeal reference modified |
Free format text: ORIGINAL CODE: EPIDOSCREFNE |
|
APBX | Invitation to file observations in appeal sent |
Free format text: ORIGINAL CODE: EPIDOSNOBA2E |
|
APBT | Appeal procedure closed |
Free format text: ORIGINAL CODE: EPIDOSNNOA9E |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE APPLICATION HAS BEEN REFUSED |
|
18R | Application refused |
Effective date: 20220527 |
|
REG | Reference to a national code |
Ref country code: HK Ref legal event code: WD Ref document number: 1200230 Country of ref document: HK |