US20230038289A1 - Cashier interface for linking customers to virtual data - Google Patents
Cashier interface for linking customers to virtual data Download PDFInfo
- Publication number
- US20230038289A1 US20230038289A1 US17/969,714 US202217969714A US2023038289A1 US 20230038289 A1 US20230038289 A1 US 20230038289A1 US 202217969714 A US202217969714 A US 202217969714A US 2023038289 A1 US2023038289 A1 US 2023038289A1
- Authority
- US
- United States
- Prior art keywords
- subject
- subjects
- checkout
- detecting
- item
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000003993 interaction Effects 0.000 claims abstract description 156
- 238000000034 method Methods 0.000 claims abstract description 153
- 238000012544 monitoring process Methods 0.000 claims abstract description 92
- 230000008520 organization Effects 0.000 claims abstract description 73
- 230000004044 response Effects 0.000 claims abstract description 27
- 230000003190 augmentative effect Effects 0.000 claims abstract description 19
- 238000012545 processing Methods 0.000 claims description 68
- 238000003384 imaging method Methods 0.000 claims description 43
- 230000008569 process Effects 0.000 claims description 33
- 238000004891 communication Methods 0.000 claims description 14
- 230000002452 interceptive effect Effects 0.000 claims description 10
- 238000009877 rendering Methods 0.000 claims description 5
- 230000006870 function Effects 0.000 description 30
- 238000007726 management method Methods 0.000 description 27
- 238000001514 detection method Methods 0.000 description 21
- 230000009471 action Effects 0.000 description 15
- 230000000694 effects Effects 0.000 description 12
- 230000008901 benefit Effects 0.000 description 10
- 238000013459 approach Methods 0.000 description 9
- 230000000007 visual effect Effects 0.000 description 9
- 238000004458 analytical method Methods 0.000 description 7
- 238000010801 machine learning Methods 0.000 description 6
- LFQSCWFLJHTTHZ-UHFFFAOYSA-N Ethanol Chemical compound CCO LFQSCWFLJHTTHZ-UHFFFAOYSA-N 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 239000008267 milk Substances 0.000 description 3
- 235000013336 milk Nutrition 0.000 description 3
- 210000004080 milk Anatomy 0.000 description 3
- 238000005303 weighing Methods 0.000 description 3
- 241000251468 Actinopterygii Species 0.000 description 2
- 235000013339 cereals Nutrition 0.000 description 2
- 238000012790 confirmation Methods 0.000 description 2
- 238000013499 data model Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- 238000002372 labelling Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 239000003550 marker Substances 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012552 review Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 230000002730 additional effect Effects 0.000 description 1
- 235000013334 alcoholic beverage Nutrition 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 235000015278 beef Nutrition 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 235000015220 hamburgers Nutrition 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 238000003331 infrared imaging Methods 0.000 description 1
- 235000008960 ketchup Nutrition 0.000 description 1
- 230000000670 limiting effect Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000036961 partial effect Effects 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000012913 prioritisation Methods 0.000 description 1
- 230000002829 reductive effect Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000004984 smart glass Substances 0.000 description 1
- 230000003997 social interaction Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 238000013179 statistical model Methods 0.000 description 1
- 230000002459 sustained effect Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q20/00—Payment architectures, schemes or protocols
- G06Q20/08—Payment architectures
- G06Q20/20—Point-of-sale [POS] network systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q20/00—Payment architectures, schemes or protocols
- G06Q20/08—Payment architectures
- G06Q20/20—Point-of-sale [POS] network systems
- G06Q20/208—Input by product or record sensing, e.g. weighing or scanner processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q20/00—Payment architectures, schemes or protocols
- G06Q20/08—Payment architectures
- G06Q20/20—Point-of-sale [POS] network systems
- G06Q20/203—Inventory monitoring
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- G—PHYSICS
- G07—CHECKING-DEVICES
- G07G—REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS
- G07G1/00—Cash registers
- G07G1/01—Details for indicating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/103—Static body considered as a whole, e.g. static pedestrian or occupant recognition
Definitions
- This invention relates generally to the field of customer service technology, and more specifically to a new and useful system and method for applications of computer vision for linking users with virtual data.
- FIG. 1 is a schematic representation of a system of a preferred embodiment
- FIG. 2 is a schematic representation of an exemplary scenario of the system and method applied across multiple operator stations
- FIG. 3 is a schematic representation of an exemplary user flow for automated checkout
- FIG. 4 is a schematic representation of an exemplary user flow for merging subjects during automated checkout
- FIG. 5 is a schematic representation of an exemplary scenario for adding items at a service counter
- FIG. 6 is a schematic representation of an exemplary scenario of an operator servicing customers with no pre-defined line, which illustrates changes in a subject management user interface based on orientation of an operator relative to customers;
- FIG. 7 is an exemplary screenshot of a subject management user interface using an image-based representation
- FIGS. 8 A- 8 C are exemplary variations of types of image-based representations
- FIG. 9 is a flowchart representation of a method
- FIG. 10 is a flowchart representation of an alternative variation of the method.
- FIG. 11 is a flowchart representation of a method applied to automated checkout.
- FIG. 12 is an exemplary system architecture that may be used in implementing the system and/or method.
- a system and method for applying computer vision (CV) for linking users with virtual data of a preferred embodiment functions to leverage CV monitoring in facilitating digital interactions with customers from physical computing device user interface.
- the system and method preferably enables a convenient user interface through which digital information and/or records associated with a person can be easily accessed by another user through a computing device.
- the system and method may have utility in enabling a cashier user interface on a computing device to link customers and other types of users with virtual data.
- the system and method enables a subject management user interface to intelligently present subject indicators (e.g., user icons) that are arranged based on CV-based analysis on the position, orientation, attention, and organization of subjects in the environment relative to the computing device of the subject management user interface.
- subject indicators e.g., user icons
- Alternative types of user interfaces could similarly be presented such as rendering a live or recent image of a line and making the people in the image selectable items of a user interface, where selecting the people accesses some person associated state information.
- an operator may more easily access the digital information linked (through CV monitoring and/or other forms sensor-based monitoring) to the users in proximity to the operator.
- the operator may access the digital information either by viewing digital information of the user associated with the subject indicator and/or updating, adding, or deleting information of a data record associated with the user.
- the system and method enables a novel and new capability for user-operated computing devices to simplify how digital information of a modeled environment can be accessed.
- the system and method can coordinates subject data tracking, subject relationship modeling from an operator perspective, and subject data synchronization with a device of the operator.
- a sensor-based monitoring system such as a CV monitoring system, generates or otherwise establishes an association between a subject and at least one piece of subject data (e.g., a user profile, subject associated information or labeling, etc.) and then tracks that subject within an environment.
- a sensor-based monitoring system such as a CV monitoring system
- Subject relationship modeling can use the same or different sensor-based monitoring system to interpret the contextual relationship of a particular device (or user) relative to a set of subjects in the environment. This modeling may be used to present an interface through which some operator of a device can more easily interact with the subject data. For example, a menu of selectable subject indicators can be presented in a way that prioritizes presentation of the subject indicators based on a modeled point of view of the device that is presenting the menu user interface.
- Subject data synchronization preferably allows actions taken at an operator device to be applied to subject data.
- Subject data synchronization can enable an operator to access and interact with the subject data detected with the monitoring system. This may involve reading and using subject data and/or adding, updating, deleting, or otherwise editing subject data.
- subject data tracking, subject relationship modeling, and subject data synchronization may be performed and managed across multiple subjects and operators in an environment.
- the system and method can be implemented across multiple customers and workers operating POS systems that using the capabilities of the system and method in parallel as shown in FIG. 2 .
- This coordinated sensing, modeling, and data synchronization and communication can be particularly useful in situations where ambient computing is sensing and modeling interaction state of one or more subjects in an environment.
- the system and method may be applied and implemented for a variety of applications.
- automated checkout can include fully automated and partially automated variations and is primarily characterized by a system or method that generates or maintains a checkout list (i.e., a virtual cart) during the shopping process of a user or users, with the objective of knowing the possessed or selected items for billing.
- the checkout list can be used in executing a checkout process to charge a customer for the selected items.
- the checkout process can occur when a customer is in the process of leaving a store.
- the system and method can be used so that a checkout list can be processed at a POS system, which may be the standard checkout process in the environment, a checkout option for customers without a connected checkout-free application or account, customers wanting to pay with cash or other payment options, and for any suitable reason.
- a POS system which may be the standard checkout process in the environment, a checkout option for customers without a connected checkout-free application or account, customers wanting to pay with cash or other payment options, and for any suitable reason.
- checkout may be processed in part through interactions with a cashier-operated device such as a POS system.
- the checkout process could alternatively occur when a customer is in the process of leaving a store or when any suitable condition for completing a checkout process is satisfied such as when a customer selects a checkout option within an application.
- a checkout list may be maintained and tracked during a shopping experience through use of one or more monitoring systems.
- a checkout list (alternatively referred to as a checkout list) is a digitally stored data record or records modeling contents of a virtual cart (or what items are to be purchased). Any suitable number of databases and records may be used in representing item contents, item information (price, count, discounts, etc.) for items of a checkout list, and/or other information of the checkout list.
- the system and method can charge a customer for the total of a shopping cart and/or alternatively automatically present the total transaction for customer completion. This may include entering the contents of a checkout list into a POS system for payment and/or supplying of payment information for checkout processing.
- a checkout list is preferably associated with a subject. May alternatively be associated with multiple subjects if multiple subjects are detected or otherwise specified as a group collaboratively building a shared checkout list
- Automated checkout is used herein to describe a variety of types of sensor-assisted forms of checkout.
- automated checkout is facilitated through image analysis using video cameras and more specifically using a CV monitoring system.
- Other variations of automated checkout can use additional or alternative forms of sensor-based monitoring such as using sensor-enabled shelves, smart carts, RF ID tracking or smart infrastructure.
- Automated checkout in some variations may also include user-assisted forms of checkout such as where a customer uses an application operable on a personal computing device (e.g., a smart phone) to perform some form of pre-scanning of items.
- a personal computing device e.g., a smart phone
- a checkout list may be generated in part or whole for a customer.
- the checkout list can be synchronized with (or otherwise transmitted to) a point of sale (POS) system manned by a worker so that at least a subset of items can be automatically entered into the POS system thereby alleviating manual entry of the items.
- POS point of sale
- the system and method may be used in exposing cashier control over when and how a checkout list of a customer is processed by a POS system.
- the system and method may additionally or alternatively be used for providing customer care solutions, adding service counter items, and/or performing other operations.
- the system and method may be used within a POS system in a retail environment with an automated checkout system.
- a CV-based monitoring system and/or other sensor-based monitoring system may be used to contextually present user icons for various customers within the POS system as shown in FIG. 3 .
- This is preferably used so that a worker operating the POS system can have a menu of nearby customers intelligently presented within an application of the POS system.
- the customers can be presented with visual representation of the customers and ordered in a sensible manner (e.g., an approximation of line order) so that it is easy for a worker to select a customer for checkout. For example, the customers next in line would generally be presented first in a list of other customers in line.
- the system and method in this use case may additionally enable an intuitive interface for “merging” checkout lists for grouping customers for a single checkout process as shown in FIG. 4 .
- An operator of the POS system could select one of the customers for checkout processing from a user interface customized to their particular perspective and the checkout list associated with that customer (or customers shopping together) can be processed for checkout.
- the system and method may be used to more easily update and edit the subject data associated with a subject.
- a CV-based monitoring system and/or other sensor-based monitoring system may be used to contextually present user icons for various customers within the POS system as shown in FIG. 3 .
- a worker could then use this contextual user interface to select a customer and then supply additional information.
- this may be used at a deli, fish, butcher, or bakery counter, where service counter items (which may be packaged and priced at the counter) are sold.
- the worker could prepare one or more service counter items for a customer, select the appropriate customer from a contextually generated user interface, and then enter the service counter items.
- the service counter items may then be added to a checkout list of the customer stored by an automated checkout system. This serves as an example of information being added to the subject data.
- the system and method may be used for a customer care system.
- a worker could similarly be presented with an array of graphical representations of customers (e.g., icons in the form of customer profile photos or in-store image captures) where the graphical representations are organized in a contextual manner based on visual interpretation of the relative position of the worker and customers in the store.
- a worker could quickly and seamlessly access a user profile for providing various forms of customer service. For example, a user may enter their customer care request through an app or some other computing device.
- a monitoring system could detect that the user is in the store and establish an association between the customer care request and the CV detected user.
- the system and method may even help customize the contextual presentation of subject indicators based on the orientation and/or attention of an operator device and/or the operator as shown in FIG. 6 . For example, the customer that the worker is facing and closest too may be presented first in the list of possible customers.
- the system and method in one implementation may be able to facilitate a customer service tool to help with sales, customer appointments, customer questions, and the like.
- the system and method could be used so that a worker could easily access the checkout list information of a customer to assist a customer anywhere in the environment. This may be done in addition to or as an alternative to facilitating the checkout process within an application of the POS system.
- the system and method can be particularly applicable for facilitating individualized/personalized digital experiences in group-based computing environments.
- Group-based computing environments as used herein are used to characterize a new form of computing where a monitoring system tracks individual state of multiple users in an environment.
- the system and method are primarily described herein as it could be used along with systems for automated checkout, but the system and method are not limited such use cases.
- the system and method have particular applicability in the area of automated checkout as it provides a new mechanism through which such an automated checkout system can be integrated with a POS system used by a cashier. This could additionally be used for self-service checkout kiosks where a user with a tracked checkout list could complete a transaction at a self-service kiosk.
- users that have tracked digital interaction information are described primarily described herein as subjects.
- customers may be used.
- Users that interact with the user interface are described herein as operators.
- Operators in a retail environment may be workers, and, in examples of retail settings, the term worker is sometimes used herein.
- an operator could also be a subject such as when the system and method is used with a self-help kiosk.
- the system and method may provide a number of potential benefits.
- the system and method are not limited to always providing such benefits, and are presented only as exemplary representations for how the system and method may be put to use.
- the list of benefits is not intended to be exhaustive and other benefits may additionally or alternatively exist.
- the system and method can increase efficiency by which workers serve customers.
- the system and method provide a convenient user interface for accessing customer information. Through this interface, users can better interact with the information generated and managed by a monitoring system.
- the system and method can avoid the clumsy process of gathering information from a user so that a digital record by searched for and found.
- the system and method may make access to user data more secure and private by limiting access to operator devices where it contextually makes sense. For example, a worker could be prevented from viewing any customer's checkout list except for those that queue in the worker's specific lane.
- the system and method provide an interface that can conveniently leverage the intuition of a human.
- a worker can use their human intuition of who is next in line and who is checking out as a group to select customers for checkout.
- the customers will generally form a line in front of the worker.
- the line will often deviate from an ideal line that can be easily interpreted from a CV monitoring system.
- a child may wander up and down the line away from a parent, but a worker observing social interactions would be able to interpret the situation to understand the child (and the products held by them) should be merged with the products of the parent.
- the system and method can easily handle this situation.
- groups of adults e.g., couples, friends, etc.
- groups of adults could be easily segmented by a human worker for separating or grouping purchases by selectively asking simple questions (e.g., “Are you two paying together?”).
- the system and method can provide a convenient and efficient technology-based solution for managing digitally stored user data like checkout lists.
- the system and method facilitates a user interface that maintains this valuable form of human input.
- the system and method can aid in minimizing errors and correcting issues.
- the system and method composes and transforms the sensed information into an intuitive user interface.
- the primary action of selecting an individual subject is directly made easier through recommendations and contextual organization of information.
- a secondary action such as merging digital information (e.g., checkout lists) of two or more subjects is additionally made easier.
- merging digital information e.g., checkout lists
- an operator could easily select the appropriate icons of users that the operator would like to merge into a single checkout transaction and then initiate the checkout process.
- the related action of splitting a checkout transaction is also maintained with similar user interface actions. For example, the operator if wanting to undo a merge or remove one or more subjects from a merged checkout transaction could similarly select the subjects to remove (or deselect subjects from the active subjects in the checkout transaction).
- a system for applying CV to a customer service application of a preferred embodiment can include a sensing system 100 , which includes at least a CV monitoring system no, and a subject management user interface (UI) 200 .
- the sensing system 100 is configured to determine individual state of a plurality of subjects in an environment.
- the individual state of a subject relates to a prediction of items selected by the subject that are intended for purchase within a retail environment.
- the CV monitoring is additionally configured to determine contextual organization of the subjects, which is used in presenting the UI of the subject management UI 200 .
- the sensing system 100 of a preferred embodiment is preferably configured to collect or determine at least some form of state information relating to a subject.
- the sensing system 100 can include one or more types of sensing systems.
- a sensing system 100 may collect data from the environment and then make a determination.
- One such sensing system 100 can include a CV monitoring system no.
- Alternative and/or additional sensing systems could include smart cart/baskets, smart shelving, radar-based motion detection, wearable computer sensing systems (e.g., collecting data from inertial measurement units (IMUs) on phones and other activity sensing devices), and/or other suitable forms of sensing systems.
- IMUs inertial measurement units
- a sensing system 100 may additionally or alternatively receive digital communication from a device such that the sensing system 100 facilitates tracking of various forms of information of devices in the environment.
- Examples of such sensing systems could include near field communication system used to detect presence of a phone or other type of device, a radio frequency identifier tracking (RFID) system for tracking RF tags other RF identifiable objects, user applications that log information, and/or systems used to track and relay information from a computing device.
- RFID radio frequency identifier tracking
- the sensing system 100 may operate in combination with one or more outside systems to facilitate rich forms of information modeling for subjects of the environment.
- One exemplary sensing system 100 is one used to facilitate some form of automated checkout.
- the sensing system 100 may therefore be configured to generate and track a checkout list for subjects in the store.
- the sensing system 100 may additionally or alternatively be applied towards other forms of customer service.
- subject profile information may be tracked such as product browsing history, which may be used by a salesperson when helping the subject.
- subject profile information can be used to enable a customer service representative to quickly access customer-submitted issues or questions when engaging with the customer within the environment.
- the sensing system 100 preferably includes one or more computer processes and specially configured machine-readable instructions stored on a computer-readable medium.
- the machine-readable instructions are preferably specially configured so that execution and processing by the one or more processors, cause the sensing system 110 to detect subjects in the environment and establish an association between the detected subjects and at least one piece of data. More specifically for an automated checkout variation, the machine-readable instructions are preferably specially configured so that execution and processing by the one or more processors, cause the sensing system 110 to track a set of subjects through the environment and, for each subject, detect item interaction events, and update items in a checkout list based on the item interaction event.
- the item interaction events can include item section events and optionally item deselection events.
- the machine-readable instructions are preferably specially configured so that execution and processing by the one or more processors cause the sensing system 110 to detect a contextual organization of subjects relative to an operator station. This can include detecting subjects in proximity to an operator device (and/or an operator) and modeling line order, subject priority, subject and/or operator attention, and/or performing other operations to detect a contextual organization of subjects.
- the same sensing system 100 is used for subject data tracking and subject relationship modeling.
- different sensing systems 100 may be used for subject data tracking and subject relationship modeling.
- the sensing system 100 is preferably deployed in at least one location of the environment. In some variations, the sensing system 100 is deployed across a region of the environment and in some cases across a significant region of the environment. A significant region of the environment could be a region with substantial sensing coverage, which may include coverage percentages such as 10% coverage, 50% coverage, and/or 90% or more coverage.
- a CV monitoring system 110 of a preferred embodiment functions to transform image data collected within the environment into observations relating in some way to subjects in the environment.
- the CV monitoring system no is used for detecting items (e.g., products), monitoring users, tracking user-item interactions, and/or making other conclusions based on image and/or sensor data.
- the CV monitoring system 110 will preferably include various computing elements used in processing image data collected by an imaging system.
- the CV monitoring system 110 will preferably include an imaging system and a set of modeling processes and/or other processes to facilitate analysis of user actions, item state, and/or other properties of the environment.
- the CV monitoring system 110 in one exemplary implementation is used to track a checkout list of subjects for offering automated checkout.
- the CV monitoring system 110 preferably provides specific functionality that may be varied and customized for a variety of applications.
- the CV monitoring system 110 may additionally facilitate operations related to person identification, checkout list generation, item interaction tracking, store mapping, and/or other CV-based observations.
- the CV monitoring system 110 can at least partially provide: person detection; person identification; person tracking; object detection; object classification; object tracking; gesture, event, or interaction detection; detection of a set of customer-item interactions, and/or forms of information.
- the system can use a CV monitoring system 110 and processing system such as the one described in the published US Patent Application 2017/0323376 filed on May 9, 2017, which is hereby incorporated in its entirety by this reference.
- the CV monitoring system 110 will preferably include various computing elements used in processing image data collected by an imaging system.
- the imaging system functions to collect image data within the environment.
- the imaging system preferably includes a set of image capture devices.
- the imaging system might collect some combination of visual, infrared, depth-based, lidar, radar, sonar, and/or other types of image data.
- the imaging system is preferably positioned at a range of distinct vantage points.
- the imaging system may include only a single image capture device.
- a small environment may only require a single camera to monitor a shelf of purchasable items.
- the image data is preferably video but can alternatively be a set of periodic static images.
- the imaging system may collect image data from existing surveillance or video systems.
- the image capture devices may be permanently situated in fixed locations.
- a subset of imaging devices can be mobile cameras (e.g., wearable cameras or cameras of personal computing devices).
- the system could operate partially or entirely using personal imaging devices worn by users in the environment (e.g., workers or customers).
- a mobile camera mounted to a worker can be used for generating a rendering a live or recent image of a line and making the people in the image selectable items of a user interface of an operator device.
- the imaging system preferably includes a set of static image devices mounted with an aerial view from the ceiling or overhead.
- the aerial view imaging devices preferably provide image data that observes at least the users in locations where they would interact with items.
- the image data includes images of the items and users (e.g., customers or workers). While the system (and method) are described herein as they would be used to perform CV as it relates to a particular item and/or user, the system and method can preferably perform such functionality in parallel across multiple users and multiple locations in the environment as shown in FIG. 2 . Therefor, the image data may collect image data that captures multiple items with simultaneous overlapping events.
- the imaging system is preferably installed such that the image data covers the area of interest within the environment.
- ubiquitous monitoring characterizes pervasive sensor monitoring across regions of interest in an environment.
- Ubiquitous monitoring will generally have a large coverage area that is preferably substantially continuous across the monitored portion of the environment. However, discontinuities of a region may be supported. Additionally, monitoring may monitor with a substantially uniform data resolution or at least with a resolution above a set threshold.
- a CV monitoring system 110 may have an imaging system with only partial coverage within the environment.
- a CV-based processing engine and data pipeline preferably manages the collected image data and facilitates processing of the image data to establish various conclusions.
- the various CV-based processing modules are preferably used in generating user-item interaction events, a recorded history of user actions and behavior, and/or collecting other information within the environment.
- the data processing engine can reside local to the imaging system or capture devices and/or an environment. The data processing engine may alternatively operate remotely in part or whole in a cloud-based computing platform.
- User-item interaction processing modules function to detect or classify scenarios of users interacting with an item.
- User-item interaction processing modules may be configured to detect particular interactions through other processing modules. For example, tracking the relative position of a user and item can be used to trigger events when a user is in proximity to an item but then starts to move away.
- Specialized user-item interaction processing modules may classify particular interactions such as detecting item grabbing or detecting item placement in a cart.
- User-item interaction detection may be used as one potential trigger for an item detection module.
- a person detection and/or tracking module functions to detect people and track them through the environment.
- a person identification module can be a similar module that may be used to uniquely identify a person. This can use biometric identification. Alternatively, the person identification module may use Bluetooth beaconing, computing device signature detection, computing device location tracking, and/or other techniques to facilitate the identification of a person. Identifying a person preferably enable customer history, settings, and preferences to be associated with a person. A person identification module may additionally be used in detecting an associated user record or account. In the case where a user record or account is associated or otherwise linked with an application instance or a communication endpoint (e.g., a messaging username or a phone number), then the system could communicate with the user through a personal communication channel (e.g., within an app or through text messages).
- a personal communication channel e.g., within an app or through text messages.
- a gesture, event, or interaction detection modules function to detect various scenarios involving a customer.
- One preferred type of interaction detection could be a customer attention tracking module that functions to detect and interpret customer attention. This is preferably used to detect if, and optionally where, a customer directs attention. This can be used to detect if a customer glanced in the direction of an item or even if the item was specifically viewed.
- the item detection module of a preferred embodiment functions to detect and apply an identifier to an object.
- the item detection module preferably performs a combination of object detection, segmentation, classification, and/or identification. This is preferably used in identifying products or items displayed in a store.
- a product can be classified and associated with a product SKU identifier.
- a product may be classified as a general type of product. For example, a carton of milk may be labeled as milk without specifically identifying the SKU of that particular carton of milk.
- An object tracking module could similarly be used to track items through the store. Tracking of items may be used in determining item interactions of a user in the environment.
- the item detection module in some variations may be integrated into a real-time inventory system.
- the real-time inventory system functions to detect or establish the location of inventory/products in the environment.
- CV-based processing modules may additionally be used such as customer sentiment analysis, clothing analysis, customer grouping detection (e.g., detecting families, couples, friends or other groups of customers that are visiting the store as a group), and/or the like.
- the system may include a number of subsystems that provide higher-level analysis of the image data and/or provide other environmental information such as a real-time virtual cart system.
- the real-time virtual cart system functions to model the items currently selected for purchase by a customer.
- the real-time virtual cart system can manage the checkout lists of the various subjects (i.e., customers) in a retail environment.
- the virtual cart system may enable forms of automated checkout such as automatic self-checkout (e.g., functionality enabling a user to select items and walk out) or accelerated checkout (e.g., selected items can be automatically prepopulated in a POS system for faster checkout).
- Product transactions could even be reduced to per-item transactions (purchases or returns based on the selection or de-selection of an item for purchase).
- the virtual cart system may be integrated with the system to provide purchase or planned purchase information, which may be used as a condition for delivering content.
- the type of content delivered to customer may be based in part on their current cart contents. For example, a coupon may be selected and delivered to a customer for a particular brand of ketchup buns based in part on the customer having hamburger buns and ground beef in the cart.
- the CV monitoring system 110 may additionally or alternatively include human-in-the-loop (HL) monitoring which functions to use human interpretation and processing of at least a portion of collected sensor data.
- HL monitoring uses one or more workers to facilitate review and processing of collected image data.
- the image data could be partially processed and selectively presented to human processors for efficient processing and tracking/generation of a checkout list for users in the environment.
- the CV monitoring system 110 can be used within the system for one or more modes of subject monitoring.
- One form of subject monitoring is a subject relationship modeling mode that functions to coordinate and direct how subject information can be presented in the subject management UI 200 .
- the CV monitoring is preferably used in this mode to detect subjects and their contextual organization (e.g., from a position, orientation, and attention approach) relative to a worker and/or subject management user interface.
- the CV monitoring system 110 is preferably used to approximate the priority or order in which subject icons should be presented in a cashier application when the subjects are lined up for checkout.
- the CV monitoring system 110 can be used in synchronizing subject information from some sensing system 100 (possibly also the CV monitoring system 110 ) with CV-detected subjects. This functions to associate collected state information with observed subjects.
- An additional mode of subject monitoring could be one used for collecting subject state information as was described above such as the generation of a checkout list.
- the subject state information may alternatively be tracked and monitored through alternative types of sensing systems.
- this collected state information can include the checkout list. It could additionally include other information like user activity during a visit or during previous visits to a store, alerts associated with the user, and/or any suitable type of information.
- the subject management UI of a preferred embodiment functions to facilitate some form of customer service as directed by a human operator.
- the subject management UI 200 preferably uses contextual organization status as detected by the CV monitoring system 110 to augment the presentation and interactions of the subject management UI 200 .
- the subject management UI 200 is preferably operable on a computing device.
- the computing device will preferably have a graphical user interface output but may alternatively or additionally have UI output in an audio format or other alternative medium of UI output.
- the subject management UI 200 is preferably implemented as an application or as part of a larger application, code-base, operating system, or other suitable type of software implemented solution.
- the computing device can be a desktop or laptop computer, a tablet, smart phone, smart watch, smart glasses, smart headphones, augmented reality device, virtual reality device, and/or any suitable type of computing device, and/or any suitable type of computing device.
- the computing device could additionally be a POS system or some other purpose-built computing device.
- Multiple subject management UI 200 instances can be operating simultaneously within an environment.
- the implementation of the subject management UI 200 can vary depending on the type of computing device and the intended use case of the system.
- the location and presence of a subject management UI 200 may be registered or otherwise configured within the system. Alternatively the location and presence may be dynamically detected.
- the computing device of the subject management UI 200 may broadcast an identifying signal to the system.
- the sensing system 100 may alternatively detect and determine a location of the computing device.
- the orientation of the computing device may be determined.
- a visual marker can be placed on or near the location of the computing device such that that location can be used as a reference point for an associated computing device.
- the subject management UI 200 is preferably part of a POS system. More specifically, the subject management UI 200 can be or be part of a cashier application operable on a tablet computer that is directly or communicatively coupled with a POS system. In this exemplary implementation, an operator (e.g., cashier) will have a tablet with a touch user interface that can present subject icons that can be used to determine who will be selected for a checkout transaction.
- the computing device of the subject management UI 200 may alternatively interface with a POS system and/or other additional systems. In another variation, the computing device ma be a POS processing system wherein checkout processing (e.g., price calculation, payment receipt, receipt generation, etc.) are handled by the computing device.
- the subject management UI 200 will preferably include a number of views.
- the subject management UI 200 will preferably include a subject selection view, which is configured to present a set of subjects based on modeling of the context and relationship between the operator device and subjects in the environment.
- subjects are presented in an order intended to approximate the expected line order. This modeling of line order can factor in the detected physical proximity and order within a lineup region.
- the modeling may additionally consider the time of arrival to some region in the environment, the attention of the operator and/or subject, the orientation of an operator device, the subject data associated with the subjects, and/or other suitable factors.
- the subject management UI 200 will then preferably facilitate accessing subject-related data. Accessing may include reading and displaying information based on the subject data. For example, the contents of a checkout list as determined by a CV monitoring system may be displayed in response to operator selection of a corresponding subject indicator. In another variation, data input received at the operator device may be added to subject data in response to selection of the subject. For example, an issue can be resolved for a checkout list of a real-time virtual cart system. In another example, an item may be added to a checkout list of a subject after selection of the appropriate subject in a subject management UI 200 .
- a method for applying CV to a customer service application of a preferred embodiment can include detecting interaction state data of a plurality of subjects in an environment S 110 ; detecting a contextual organization of subjects relative to an operator station S 120 ; and at a computing device of the operator station, augmenting a user interface based on the contextual organization of subjects S 130 .
- the subject-associated data of a subject is preferably accessible in the user interface.
- Augmenting the user interface S 130 in one preferred variation can include presenting subject indicators in the user interface with the subject indicators arranged in response to contextual organization S 132 ; and in response to user interaction with a subject indicator, accessing the digital interaction state data of a subject associated with the subject indicator S 134 .
- the method may be applied in a similar manner to any suitable type of user-associated data.
- the method can include detecting user-associated data of a plurality of subjects in an environment S 210 ; detecting a contextual organization of subjects relative to an operator station S 220 ; and at a computing device of the operator station, augmenting a user interface based on the contextual organization of subjects S 230 , which includes presenting subject indicators in the user interface with the subject indicators arranged in response to contextual organization S 232 , and, in response to user interaction with a subject indicator, accessing the user-associated data of a subject associated with the subject indicator S 234 .
- User-associated can be any suitable type of data that can be linked to a subject.
- the method may be used in presenting a menu of customer-associated icons (i.e., subject indicators) as selectable UI elements.
- the customer-associated icons can be presented in an order or manner based on the customers' organization around an operator station (or an operator). In a variation using a CV monitoring system, the customer-associated icons are presented based on interpretation of images capturing the organization around an operator station. For example, the order in which customers are waiting in line can be used to at least partially determine the order of the customer-associated icons.
- the customer-associated icons are used to access the customer information, which in this case could be information such as a checkout list information that could be automatically added to a POS system.
- Method is primarily described as it is applied to one operator station and computing device.
- the method will generally be used in a computer environment and/or for a use case where multiple subject management user interfaces are used.
- the method may alternatively be described as detecting digital interaction state data of a plurality of subjects in an environment S 110 ; across a set of operator stations, detecting a contextual organization of subjects relative to each particular operator station S 120 ; and at a computing device instance of each operator station, augmenting a user interface of the computing device instance based on the contextual organization of subjects detected for the associated operator station S 130 .
- Augmenting the user interface can similarly include presenting subject indicators in the user interface with the subject indicators arranged in response to contextual organization S 132 , and, in response to user interaction with a subject indicator, accessing the digital interaction state data of a subject associated with the subject indicator S 134 .
- each POS checkout station can apply the method in assisting a worker at that particular station.
- Each POS terminal of each checkout lane of a grocery store may be augmented through the method described here.
- Each individual computing device of a POS terminal may have an interface that intelligently presents subject indicators for the subjects waiting in that particular checkout lane as shown in FIG. 2 .
- the method could include tracking checkout list status of a plurality of shoppers S 310 ; across a plurality of POS checkout stations, detecting a contextual organization of at least a subset of shoppers relative to each POS checkout station S 320 ; and, at the POS checkout station, presenting shopper-associated UI elements in a UI of the POS checkout station, where the UI elements are organized by the contextual organization S 332 ; and upon detecting selection of a shopper associated UI element, selecting a checkout list associated with the shopper for checkout processing S 334 .
- Multiple selections can be used to merge checkout lists of two or more shoppers.
- deselection of a UI element may remove a checkout list from checkout processing.
- the method is preferably implemented through a system substantially similar to the one described above but may alternatively be implemented with any suitable system.
- the method is described primarily as it may be applied to a CV monitoring system for tracking checkout list information and then facilitating automated checkout through a POS system in a retail setting, but the method may additionally or alternatively be used for a variety of digital interactions offered in an environment shared by multiple subjects.
- Block S 110 which includes detecting digital interaction state of a plurality of subjects in an environment, functions to collect subject-associated information from a plurality of subjects in an environment.
- Detecting digital interaction state of a plurality of subjects is preferably implemented through remote sensing using at least one sensor-based monitoring system such as a CV monitoring system, sensor-enabled smart shelf system, smart carts, RF ID tracking system, other types of smart infrastructure, and/or additional sensor modalities.
- sensor-based monitoring system such as a CV monitoring system, sensor-enabled smart shelf system, smart carts, RF ID tracking system, other types of smart infrastructure, and/or additional sensor modalities.
- the method is preferably applied to digital interactions that are individualized for a plurality of subjects.
- this may include tracking a checkout list of items selected for checkout while in the store, tracking browsing history of individual customers, receiving customer service request information form multiple customers in the store, and/or other suitable applications.
- digital interaction state information can be dynamic and based on ongoing interactions within the environment.
- digital interaction state information could be state information detected through discrete, isolated events.
- digital interaction state information may be generated when a subject enters the environment (or for any other type of event).
- a user may perform some action like picking up a restricted item (e.g., alcohol) and this may be stored as interaction state data and used to indicate to a cashier which customers have picked up alcohol by labeling subject indicator icons with a graphical indicator, which may signal to the cashier that they should request ID.
- a restricted item e.g., alcohol
- sensing digital interaction state preferably includes sensing the digital interaction state (and more generally activity) through one or more sensor-based monitoring systems.
- Sensing activity can include sensing activity across an environment, detecting subjects within the environment, and individually associating data with a set of subjects. The approach for sensing activity may vary depending on the type of digital interaction state monitored for the subjects.
- item/product interactions may be monitored by sensing shelf activity through shelf-based sensors, detecting user-item interactions through CV analysis of image data (e.g., detecting item pickups and setdowns), and/or other techniques.
- detecting user presence may include detecting user presence through one or more of computer vision, proximity detection, user and/or device location tracking (GPS, RF-based location tracking, wireless signal triangulation, etc.).
- GPS GPS, RF-based location tracking, wireless signal triangulation, etc.
- the digital interaction state can include checkout list data and detecting digital interaction state can include tracking a set of subjects through the environment; and for each subject, detecting item interaction events including at least item selection events and updating items in a checkout list based on the item interaction event. Detecting item interaction events including at least item selection events and updating items in a checkout list based on the item interaction event function to track the accumulation of items by individual subjects.
- the checkout lists are preferably associated with individual subjects, but may alternatively be associated with multiple subjects. This process for maintaining a checkout list in some preferred implementations using CV image processing wherein the method includes collecting image data and processing the image data through a CV image processing pipeline.
- the digital interaction state may occur outside of a sensing system 100 and could simply be information associated with a user record that can be synchronized with a subject in the record.
- block S 110 may include setting digital interaction state of a user record within a data management system.
- the digital interaction state can be a complex information model like in the case of a checkout list.
- a checkout list may have an associated data model that specifies a list of items, prices or product codes associated with those items, exceptions or flags for the shopping experience or individual products (e.g., alcoholic drinks may be flagged for ID check, produce may be flagged for weighing).
- the approach for detecting item interaction events used to update a checkout list can depend on the type of sensor monitoring systems employed. In a CV variation, a CV monitoring system may be used in detecting items and detecting user-item interactions.
- a multi-sensor monitoring system may involve detecting item activity from a sensor-enabled shelf, detecting presence of a user near the location of the item activity through another sensor monitoring system (e.g., CV-based people tracking), and associating item activity with the user.
- detecting user-item interactions can include using computer vision processing of image data and smart shelf event data. Other alternative approaches may similarly be applied to tracking of a checkout list.
- the digital interaction state could be as simple as binary state variable signifying if a user is requesting customer care assistance. In this way, a worker could view which customers in close proximity to them desire assistance.
- any sort of data may be stored and associated with a user. Such information could be stored as customer profile data.
- the method could facilitate workers more quickly accessing information of a customer. Examples of types of customer profile data that could be useful for a worker to quickly access could include: a customer's service request, online orders being picked up in store, account status (e.g., type of account or membership status at a club, gym, bank, airport, etc.), activity history, and/or other types of customer information.
- the digital interaction state could be any suitable type of information, associated media files, and/or other suitable forms of digital information.
- a CV monitoring system is one preferred type of sensing system, and a CV monitoring system is preferably used at least in part for executing block S 120 .
- the method and more specifically can additionally include collecting image data, applying CV processing of the image data and generating digital interaction state resulting in part from the CV processing.
- generating digital interaction state can include generating digital interaction state indicating the checkout list resulting in part from the computer vision processing.
- Collecting image data in an environment functions to collect video, pictures, or other imagery of a region containing objects of interest (e.g., inventory items).
- Image data is preferably collected from across the environment from a set of multiple imaging devices.
- collecting image data occurs from a variety of capture points.
- the set of capture points include overlapping and/or non-overlapping views of monitored regions in an environment.
- the method may utilize a single imaging device, where the imaging device has sufficient view of the exercise station(s).
- the imaging data preferably substantially covers a continuous region.
- the method can accommodate for holes, gaps, or uninspected regions.
- the method may be robust for handling areas with an absence of image-based surveillance such as bathrooms, hallways, and the like.
- the imaging data may be directly collected, and may be communicated to an appropriate processing system.
- the imaging data may be of a single format, but the imaging data may alternatively include a set of different imaging data formats.
- the imaging data can include high resolution video, low resolution video, photographs from distinct points in time, imaging data from a fixed point of view, imaging data from an actuating camera, visual spectrum imaging data, infrared imaging data, 3D depth sensing imaging data, parallax, lidar, radar, sonar, passive illumination, active illumination, and/or any suitable type of imaging data.
- the method may be used with a variety of imaging systems, collecting imaging data may additionally include collecting imaging data from a set of imaging devices set in at least one of a set of configurations.
- the imaging device configurations can include: aerial capture configuration, shelf-directed capture configuration, movable configuration, and/or other types of imaging device configurations. Imaging devices mounted over-head are preferably in an aerial capture configuration and are preferably used as a main image data source. In some variations, imaging devices may include worn imaging devices such as a smart eyewear imaging device. This alternative movable configuration can be similarly used to extract information of the individual wearing the imaging device or other observed in the collected image data.
- an application of CV processing in detecting interaction state can include tracking a set of subjects through the environment; for each subject, detecting item interaction events, updating items in a checkout list based on the item interaction event (e.g., adding or removing items).
- the checkout list can be a predictive model of the items selected by a customer, and, in addition to the identity of the items, the checkout list may include a confidence level for the checkout list and/or individual items.
- the checkout list is preferably a data model of predicted or sensed interactions.
- the CV monitoring system may use algorithmic approaches applying traditional computer vision techniques, deep learning models, machine learning, heuristic modeling, and/or other suitable techniques in processing the image data.
- the CV monitoring system may additionally use HL in evaluating image data in part or whole.
- Block S 120 which includes detecting a contextual organization of subjects relative to an operator station, functions to analyze the setting in proximity to the operator station.
- Detecting the contextual organization of subjects preferably includes applying computer vision to interpreting relative position, orientation, attention and other properties of a subject in a region near the operator station.
- the contextual organization of subjects may predict or characterize the order of customers in a line (i.e., line order) as it would be relevant to a worker and operator of the operator station.
- the contextual organization of subjects preferably models from the point of view of the operator an appropriate and intuitive way of presenting subjects.
- An output of detecting contextual organization of subjects can be a prioritized list of subjects ordered in some manner.
- all subjects in the environment may be analyzed.
- a subset of subjects may be analyzed.
- subjects located in a predefined region may be analyzed.
- subjects within some threshold distance in proximity to an operator station are evaluated for contextual organization.
- up to a maximum number of subjects may be evaluated for contextual organization. These subjects will generally be those closest and in the correct direction or designated region. For example, the subjects close to an operator station but in the line for an adjacent operator station are preferably not considered and only the subjects in the line region for the operator station are evaluated.
- a CV monitoring system is preferably used in evaluating the contextual organization of subjects relative to the operator station.
- Image data is preferably collected for each operator station and analyzed by detecting location of subjects and locations of operator stations. Additionally, CV processing of the image data may include detecting designated line regions, detecting direction of attention or body pose orientation of subjects and workers, direction/orientation of a computing device of the operator station, and/or other parameters. These various factors may be modeled to evaluate the contextual organization of the subjects.
- the operator station could be defined by the physical device such as the POS system, the position of the operator (e.g., the worker), and/or a defined region in the environment.
- the method may include detecting presence of an operator station. Detecting presence of an operator station may include visually detecting the operator station through CV processing of the image data. A visual marker may be applied communicating an identifier of the operator station. In another variation, the location of the operator station can be configured within the CV monitoring system.
- the contextual organization may be continuously updated, periodically updated, in response to an event like a change in image data, or as a result of any suitable trigger.
- detecting contextual organization of subjects comprises detecting proximity of a subject to an operator station.
- Detecting proximity may include measuring position of the checkout station; for a set of subjects, detecting position; and measuring checkout station proximity.
- the proximity could be an absolute magnitude measurement.
- the proximity could be based on displacement in one or more dimensions.
- Various heuristics, filters, or machine learning processes may additionally be used to appropriate limit subjects of interest. For example, a subject filter could be used to ignore subjects lined up for an adjacent checkout station.
- detecting contextual organization of subjects may include detecting line order which functions to analyze orientation and position patterns. Detecting line order may be specifically built around assessing order of subjects along a line. Detecting line order can include analyzing orientation and position of subjects relative to the operator station. Various heuristics, statistical models, machine learning models, and/or other techniques may be used to make such assessment useful.
- detecting contextual organization of subjects can include detecting attention, which functions to interpret the attention of an operator and/or subjects.
- detecting body pose orientation of users through computer vision and then assigning attention metrics to subjects based on relative attention. For example, the operator detected to be facing a subject that facing the operator will have a high attention score. This may be used so that a list of customers can be presented based on whom a worker is facing.
- the attention and orientation of an operator or an operator device may be used to determine aspects of a line.
- detecting line order can include detecting a direction of attention of one of the operator station or an operator. This may be used to determine the most likely first person in a line.
- some subjects accompanying another subject may line up past the cashier and it will actually be the subject in position to pay that should be considered as the highest priority in the line.
- some situations may have a line form around a customer service representative with no defined lane for lining up as shown in FIG. 6 .
- an organic line may form based on the dominant orientation of the customer service representative. Detecting this dominant orientation may be used in classifying and assigning a line order to subjects in near proximity.
- detecting contextual organization of subjects may include detecting mutual attention of an operator and at least one subject, which functions to assign priority based in part on shared attention between an operator and one or more subjects. For example, a customer and a cashier looking at each other (or at least directing attention towards each other) can prioritize that subject relative to other subjects.
- subjects may be classified visually as being in a waiting state (i.e., a queuing state).
- a waiting state i.e., a queuing state
- a subject that is near an operator station but is walking away or participating in shopping activity may be determined to not be in a waiting state and may be excluded from a prioritized list of subjects.
- detecting contextual organization of subjects may include detecting social grouping of subjects, which functions to determine if and which subjects may be part of a group. This may additionally include associating multiple subjects with a single checkout list. Associating multiple subjects with a single checkout list can include merging checkout lists from multiple subjects of a group. In another variation, associating multiple subjects with a single checkout list can include associating subjects to one indicated checkout list. This can be used to group families and groups of friends. Detecting social grouping can include detecting a social association of two or more subjects from historical interactions in the environment. Accordingly, how two or more subjects interact in the environment can lead the monitoring system to determine they are some form of pairing.
- the child and adult may be paired for checkout.
- the contextual organization may reflect their pairing but not default to associating the subjects directly.
- the contextual organization may additionally depend on analysis and information that originates from image data when the subject is not near the operator station and is instead determined based on previous interactions in the environment. Detecting a social grouping could be achieved by detecting shared paths, sustained proximity, detecting interactions, detecting sharing of a cart, detecting one or more instances of communication (e.g., directed visual attention, directed speech, physical contact, etc.), and/or making other suitable observations.
- social grouping may alternatively be based on information from a digital platform such as a social network or any suitable source. For example, a subject may preregister or link their account with the accounts of their family members.
- the contextual organization may factor in digital interaction state information. For example, contextual organization may consider information of a checkout list like indication of exceptions, no items, needs weight check, and the like.
- machine learning model may be trained using a training set mapping a number of relevant contextual features can be detected and resulting user interface interactions. This training may be used to adapt to different environmental conditions. Different stores have different checkout line organizations, machine learning applied to improving prioritization of subject position can be used adapt to the various environments.
- the method can include establishing an association of a CV-detected person (a subject) with digital interaction state information.
- association is established between CV-detected person and a user record, and the digital interaction state information is data linked to the user record.
- Establishing an association can be facilitated through the CV monitoring system or other techniques such as those described in U.S. patent application Ser. No. 16/015,180, filed on 21 Jun. 2018, which is hereby incorporated in its entirety by this reference.
- Block S 130 which includes augmenting a user interface at a computing device of the operator station based on the contextual organization of subjects, functions to customize presentation or interactions of an application.
- the user interface is preferably augmented to present subject related information in a manner that is relevant to the perspective of the operator.
- the computing device can be a POS system, and augmenting a user interface can be used to provide a contextually aware menu from which a cashier can select one or more subjects for checkout. This can enable a cashier to better interact and facilitate automated forms of checkout.
- block S 130 can be used to enable an operator to be presented with a menu of possible subjects, where the subjects are organized based on the contextual organization (e.g., proximity, line order, who the operator is facing) and the like.
- block S 130 can include presenting subject indicators in the user interface with the subject indicators arranged in response to contextual organization S 132 ; and in response to user interaction with a subject indicator, accessing the digital interaction state data of a subject associated with the subject indicator S 134 . Selecting a subject indicator by an operator may be used to access a user interface view with more detailed information or to select that subject to perform some additional action. Accessing the digital interaction state data may include reading the digital interaction state data and/or writing to the digital interaction state data. In some variations, the digital interaction state data may in part be presented or used in determining the user interface prior to any user interaction. For example, checkout list contents may be displayed in part next to the subject indicators prior to any selection of a subject.
- Block S 132 which includes presenting subject indicators in the user interface with the subject indicators arranged in response to contextual organization, functions to display subject indicators in a manner that corresponds to the operator's perspective.
- the contextual organization preferably considers physical conditions of the environment (e.g., relative positions of subjects, the operator, and other items in the environment) as well as the objectives of the operator (e.g., to checkout shoppers in a line).
- a subject indicator is preferably a graphical representation of a subject.
- that may be a graphical icon that could show an image of the subject.
- the method in some implementations may include capturing an image of subjects while in the environment and using the image in the graphical representation.
- a dedicated image capture device may be used to capture images of the subjects' faces.
- a user profile image of a subject could be used.
- Other variations may include using other symbolic representations such as clothing indication (e.g., showing a red shirt), indicating items from cart, or providing other descriptors that could be used to identify a subject.
- the subject management UI can additionally present other information such as digital interaction state information.
- the subject indicators may be marked with indication of errors, cart exceptions, number of items in cart, tasks for the worker such as weighing produce, and/or other suitable information.
- the user interface preferably presents a set of subject options (i.e., subject indicators). This could be presented as a list view, a gallery, explorable cards, or other suitable interfaces. In general, the graphical representations of subjects are selectable. In some variations, multiple subjects are selectable. Once selected, one or more actions may be executed on behalf of the subjects. An executed action will preferably use the digital interaction state of the subject. In the case of automated checkout, the checkout list(s) of the selected subject(s) are entered into the POS system for checkout processing. The order of the subject options is preferably based on the contextual organization information from S 120 .
- presenting a set of subject indicators in the user interface with the subject indicators arranged in response to contextual organization can include ordering the subject indicators in an order corresponding to the line order.
- subject indicators are preferably presented in an order that reflects a real world line.
- Order may a sequential order, but in a user interface, order may determine priority within the user interface.
- User interface arrangements based on order can determine placement, sequential order, size of the UI element, visibility (which subject indicators to show and when), page order, graphical presentation (e.g., how the subject indicator is presented in terms of the information included, etc.), and/or other variables of a user interface.
- presenting subject indicators may include representing a group of subjects as a single subject indicator or a grouping of subject indicators, which functions to present subjects by social grouping. Interactions can be applied across the group. This social grouping in some cases override the real world line such as for grouping a child with an adult subject in the user interface. Attention of the subjects and/or operator may similarly alter organization.
- the digital interaction state data of each subject member of the group may be combined into one digital interaction state for the group. For example, the individual checkout lists of multiple customers can be combined into a single group checkout list.
- access to the digital interaction state in block S 134 may involve performing some action (e.g., updating data) to each of the individual digital interaction state data of each group member.
- presenting a subject indicator within the user interface includes enabling user selection of the subject indicator from the user interface. For example, a user may be able to select a subject indicator to indicate they want to perform a checkout operation for them. Additionally or alternatively a set of interactive items may be performed in connection to the subject indicator. For example, sub-menu could be presented in connection to a subject indicator allowing an operator to perform various actions like select the option to add an item. Various user interface approaches may be used.
- groups of subjects may be manually created by an operator, which can include receiving user input indicating an associated group of subjects, and in response to indication of an associated group of subjects, grouping the subject indicators. This may additionally include merging the digital interaction data of the subjects.
- Selecting a subject in the user interface will generally depend on device input such as touch input on a tablet-like device or mouse/keyboard input on a computer. Selecting a subject could additionally use contextual organization. For example, the directed attention of an operator may be used to automatically select a subject based on whom the operator is facing, looking at, or indicating (e.g., pointing at).
- the user interface may be updated in real-time, but, in some implementations, the order and presentation of the subjects may lock to avoid UI elements moving/changing as an operator attempts to interact with them.
- a refresh button may be provided in the UI to allow an operator to explicitly indicate when the presentation of the subjects should be refreshed.
- a UI indicator could be presented to signify when the system believes the UI is “out of date” and should be refreshed.
- a user interface presents an image-based representation of the scenario, which functions to use an image map as a user interface as shown in FIG. 7 .
- presenting subject indicators in the user interface can include generating an interactive image-based representation and presenting the image-based representation with selectable subject indicators at the location of the subjects represented in the image-based representation.
- the image-based representation is preferably an image-based representation of the position of subjects relative to the operator station.
- the image-based representation preferably allows visual selection of a subject.
- a cashier can be presented with an image showing the customers waiting in line. The cashier can select a customer for checkout by selecting the customer shown within the image.
- Generating an interactive image-based representation may include determining position of CV detected subjects in collected image data, presenting the interactive image-based representation with interactive a subject indicator UI elements corresponding to a CV detected subject at the position of each relevant CV detected subject.
- the image-based representation could be a synthesized representation.
- the synthesized representation could be an overhead graphical map showing the operator station and visual indication of the subjects in corresponding location as shown in FIG. 8 A .
- This variation functions to create a graphical map with representative indicators of subject position relative to the operator.
- the image-based representation could be a still or live video stream from an image capture device as shown in FIG. 8 B .
- the image capture device is preferably oriented to have a similar perspective as the operator (e.g., from behind the POS system).
- the method may include collecting live video and detecting user selection at a location of CV detected person within the presented image.
- the operator can select a subject.
- the contextual organization of subjects in one variation can be used in generating an interactive image-based representation such as by generating the synthesized representation or adding interactive elements to an image or video.
- the image-based representation variation may be implemented without block S 120 . Locating and marking of subjects in the appropriate image data may defer interpretation of the line order of the subjects to the operator, and instead the image-based representation of nearby subjects serves as a spatial map of nearby subjects.
- the image-based representation could be extended to augmented reality or virtual reality user interfaces.
- presenting the image-based representation can include rendering the image-based representation in an augmented reality display as shown in FIG. 8 C . In this way subject indicators may be overlaid or displayed in positions associated with viewed subjects.
- Block S 134 which includes accessing the digital interaction state data of a subject associated with the subject indicator in response to user interaction with a subject indicator, functions to perform an action in connection with the digital interaction state data of the subject.
- Accessing the digital interaction state data may include reading and updating the user interface with content based on the digital interaction state data. For example, after selecting a subject indicator, the checkout list of the subject associated with the subject indicator may be displayed. In this variation accessing the digital interaction state data may include communicating the digital interaction state data to the computing device of the operator station.
- the data will generally be stored in a remote data storage system and managed by the monitoring system.
- accessing the checkout list of the subject further can include presenting a representation of the checkout list within the user interface.
- accessing the digital interaction state data may include communicating the digital interaction state data to the computing device of the operator station.
- the data will generally be stored in a remote data storage system and managed by the monitoring system
- Reading of interactive state data may not be used solely for presenting information but also performing some actions such as completing a checkout transaction.
- accessing the checkout list of the subject can include executing a checkout process for items of the checkout list, which functions to enter the items of the checkout list for POS processing, receiving payment, and finalizing the checkout process.
- accessing the checkout list of the subject further can include, if a cart issue is associated with the checkout list, presenting a guided resolution interaction flow within the user interface to resolve the cart issue prior to executing the checkout process.
- Cart issues may include conditions with the checkout list related to an unidentified item, an unknown number of items, produce or by-weight items requiring weighing, restricted items needing ID or other forms of verification, and/or other issues.
- the resolution interaction flows preferably indicate the cart issue and supplies a user interface for resolving—typically by receiving operator input.
- an item exception may be flagged for the produce item in the checkout list of a subject.
- the operator can be guided through a process of pairing produce identifiers, such as Price Look Up (PLU) codes, with measured item weight.
- PLU Price Look Up
- the CV monitoring system may automatically supply the PLU code when detecting the item being weighed during the resolution workflow.
- the operator may enter the PLU code.
- the quantity of an item may not be accurately detected by the monitoring system, and as such a cart issue will be flagged in association with that item of the checkout list.
- a resolution interaction flow can trigger: at the user interface displaying an item count resolution view, receiving item count and updating the count of the associated item in the checkout list.
- the item count resolution view may display a request like “How many of this item did the customer select?” and then the operator can visually count and enter the correct number.
- the item in the checkout list can be appropriately updated with the verified item count.
- a more detailed description of the checkout process performed at the computing device can include: presenting the subject indicators in an arrangement based on line order, receiving operator selection of a subject indicator, accessing the checkout list of a subject associated with the subject indicator, optionally displaying the contents of the content list, if a cart issue is associated with the checkout list, presenting a guided resolution interaction flow within the user interface to resolve the cart issue, and receiving confirmation to proceed with checkout once a checkout list is valid, entering the items into a POS processing system, determining a checkout total, receiving payment, and finalizing the checkout process.
- the method may only involve supplying the item information of the checkout list to an external POS processing system that finalizes the checkout process.
- the method may include performing each step of checkout processing.
- Accessing the digital interaction state data may additionally or alternatively include updating the subject indicator state data, which functions to write or edit the subject indicator state data.
- issues and problems with a checkout list may be manually resolved by an operator and resolution input data can be used in updating the checkout list.
- accessing the checkout list of the subject can include adding or updating at least one item to the checkout list of the select subject based on received user interaction at the operator station.
- the method may be used to add items to a checkout list of a customer.
- This can have particular utility at service counters of a grocery store like at the deli, fish, butcher, and/or bakery counter where service counter items are individually prepared and priced.
- the method can include, presenting the subject indicators in an arrangement based on most likely subjects being served by the operator, receiving operator selection of a subject indicator, entering the item information (e.g., price, item description, etc.), receiving operator confirmation to add to the checkout list, communicating the information to a computer-implemented system managing the checkout list, and adding the item information to the checkout list of the selected customer as shown in FIG. 5 .
- This variation of the method serves to make the process of an operator updating digital information of a customer easier since they can more easily access their information.
- This checkout list may later be processed in any suitable manner.
- one preferred implementation of the method includes: detecting digital interaction state of a plurality of subjects in an environment using at least one sensor-based monitoring system; detecting a contextual organization of subjects relative to an operator station; at the operator station, augmenting the user interface based on the contextual organization of subjects which comprises of at least: presenting a set of subject indicators in the user interface with the subject indicators arranged in response to contextual organization, and in response to received user interaction with at least one selected subject indicator, accessing the digital interaction state of the subject associated the at least one subject indicator.
- detecting digital interaction state may include tracking a set of subjects through the environment and, for each subject, detecting item interaction events including at least item selection events and updating items in a checkout list based on the item interaction event; and wherein accessing the digital interaction state of the subject comprises accessing the checkout list of the subject an associated with the at least one subject indicator.
- accessing the checkout list of the subject can include executing a checkout process for items of the checkout list. This may include transmitting the checkout list to a checkout list processor (e.g., a POS processing system). This may alternatively include performing the checkout process and executing a transaction on behalf of the checkout list.
- a checkout list processor e.g., a POS processing system
- accessing the checkout list of the subject may include presenting a representation of the checkout list within the user interface. This can include identifying and communicating the checkout list to the computer device of the operator device. Presentation of the checkout list may happen after selection of the associated subject in the user interface.
- accessing the checkout list of the subject further comprises, if a cart issue is associated with the checkout list, presenting a guided resolution interaction flow within the user interface to resolve the cart issue prior to executing the checkout process.
- the cart issue could be detection of a by-weight item (e.g., a produce item), and the guided resolution interaction flow can be used in collecting item weight and pairing with product identifier (e.g., a PLU code) and updating the checkout list.
- product identifier e.g., a PLU code
- the cart issue could include detection of an unknown item count, and the guided resolution interaction flow can be used in collecting an item count for one or more item and updating the checkout list.
- the cart issue could alternatively include detection of an unknown item type, and the guided resolution interaction flow can be used in specifying a description of the item and requesting product identification.
- the operator may announce an unknown cereal box (optionally with additionally identifying characteristics) and the operator can resolve the issue by scanning the cereal box or asking the customer for the product identity.
- the cart issue could include a restricted item alert, and the guided resolution interaction flow could instruct the operator to verify ID for a list of restricted items, and once ID is confirmed, checkout can proceed.
- Other cart issue types could similarly be used.
- detecting contextual organization of subjects may include detecting social grouping of subjects and associating multiple subjects with a single checkout list. Grouping may be performed using various numbers of factors from the sensor monitoring system or from other data sources. Some variations may enable options for manual control of merging within the user interface.
- accessing the checkout list of the subject can include adding at least one item to the checkout list of the subject based on received user interaction at the operator station. This can be used to add service counter items, but could additionally be used in other situations.
- accessing the digital interaction data can include modifying the digital interaction data based on received user interaction at the operator station.
- detecting item interaction events can involve the use of a CV monitoring system and can include collecting image data, applying computer vision processing of the image data and detecting item interaction events in part from the computer vision processing.
- the CV monitoring system can additionally be used in collecting image data of the region in proximity to the operator station and detecting a contextual organization of subjects relative to an operator station can include analyzing the image data of the region in proximity to the operator station using CV processing and at least partially determining contextual organization. In one implementation this may include supplying the image data in a convolutional neural network or other suitable ML, neural network, deep learning, statistical, heuristical, or other type of model that outputs indication of appropriate subjects and preferably prioritizes the identified subjects by predicted line order. This may similarly be applied in the method more generally.
- detecting item interaction events can include detecting user-item interactions using computer vision processing of image data and a smart shelf event data.
- This is one example of combining multiple types of sensor-based monitoring systems. Any suitable combination of CV monitoring smart carts, RFID tracking, smart shelves, smart infrastructure and/or other monitoring systems may be used.
- detecting contextual organization comprises detecting line order by analyzing orientation and position of subjects relative to the operator station.
- detecting line order further may include detecting a direction of attention of one of the operator station or an operator.
- presenting a set of subject indicators in the user interface with the subject indicators arranged in response to contextual organization can include ordering the subject indicators in an order corresponding to the line order.
- presenting a set of subject indicators in the user interface can include generating an interactive image-based representation of the position of subjects relative to the operator station and presenting the image-based representation with selectable subject indicators at the location of the subjects represented in the image-based representation.
- presenting the image-based representation comprises rendering the image-based representation in an augmented reality display.
- the image-based representation may alternatively be a synthesized graphical representation or an annotated image captured of the operator station.
- the systems and methods of the embodiments can be embodied and/or implemented at least in part as a machine configured to receive a computer-readable medium storing computer-readable instructions.
- the instructions can be executed by computer-executable components integrated with the application, applet, host, server, network, website, communication service, communication interface, hardware/firmware/software elements of a user computer or mobile device, wristband, smartphone, or any suitable combination thereof.
- Other systems and methods of the embodiment can be embodied and/or implemented at least in part as a machine configured to receive a computer-readable medium storing computer-readable instructions.
- the instructions can be executed by computer-executable components integrated with apparatuses and networks of the type described above.
- the computer-readable medium can be stored on any suitable computer readable media such as RAMs, ROMs, flash memory, EEPROMs, optical devices (CD or DVD), hard drives, floppy drives, or any suitable device.
- the computer-executable component can be a processor but any suitable dedicated hardware device can (alternatively or additionally) execute the instructions.
- a system comprising of one or more computer-readable mediums storing instructions that, when executed by the one or more computer processors, cause a computing platform to perform operations comprising those of the system or method described herein such as: detecting user-associated data of a plurality of subjects in an environment; detecting a contextual organization of subjects relative to an operator station; and at a computing device of the operator station, augmenting a user interface based on the contextual organization of subjects, which includes presenting subject indicators in the user interface with the subject indicators arranged in response to contextual organization, and, in response to user interaction with a subject indicator, accessing the user-associated data of a subject associated with the subject indicator.
- FIG. 12 is an exemplary computer architecture diagram of one implementation of the system.
- the system is implemented in a plurality of devices in communication over a communication channel and/or network.
- the elements of the system are implemented in separate computing devices.
- two or more of the system elements are implemented in same devices.
- the system and portions of the system may be integrated into a computing device or system that can serve as or within the system.
- the communication channel 1001 interfaces with the processors 1002 A- 1002 N, the memory (e.g., a random access memory (RAM)) 1003 , a read only memory (ROM) 1004 , a processor-readable storage medium 1005 , a display device 1006 , a user input device 1007 , and a network device 1008 .
- the computer infrastructure may be used in connecting sensing system 1101 , CV monitoring system 1102 , imaging devices 1103 , User input device with a subject management application 1104 , POS processing systems 1105 , and/or other suitable computing devices.
- the processors 1002 A- 1002 N may take many forms, such CPUs (Central Processing Units), GPUs (Graphical Processing Units), microprocessors, ML/DL (Machine Learning/Deep Learning) processing units such as a Tensor Processing Unit, FPGA (Field Programmable Gate Arrays, custom processors, and/or any suitable type of processor.
- CPUs Central Processing Units
- GPUs Graphics Processing Units
- microprocessors ML/DL (Machine Learning/Deep Learning) processing units
- ML/DL Machine Learning/Deep Learning
- FPGA Field Programmable Gate Arrays
- custom processors and/or any suitable type of processor.
- the processors 1002 A- 1002 N and the main memory 1003 can form a processing unit 1010 .
- the processing unit includes one or more processors communicatively coupled to one or more of a RAM, ROM, and machine-readable storage medium; the one or more processors of the processing unit receive instructions stored by the one or more of a RAM, ROM, and machine-readable storage medium via a bus; and the one or more processors execute the received instructions.
- the processing unit is an ASIC (Application-Specific Integrated Circuit).
- the processing unit is a SoC (System-on-Chip).
- the processing unit includes one or more of the elements of the system.
- a network device 1008 may provide one or more wired or wireless interfaces for exchanging data and commands between the system and/or other devices, such as devices of external systems.
- wired and wireless interfaces include, for example, a universal serial bus (USB) interface, Bluetooth interface, Wi-Fi interface, Ethernet interface, near field communication (NFC) interface, and the like.
- Computer and/or Machine-readable executable instructions comprising of configuration for software programs (such as an operating system, application programs, and device drivers) can be stored in the memory 1003 from the processor-readable storage medium 1005 , the ROM 1004 or any other data storage system.
- software programs such as an operating system, application programs, and device drivers
- the respective machine-executable instructions When executed by one or more computer processors, the respective machine-executable instructions may be accessed by at least one of processors 1002 A- 1002 N (of a processing unit 1010 ) via the communication channel 1001 , and then executed by at least one of processors 1001 A- 1001 N.
- processors 1002 A- 1002 N of a processing unit 1010
- Data, databases, data records or other stored forms data created or used by the software programs can also be stored in the memory 1003 , and such data is accessed by at least one of processors 1002 A- 1002 N during execution of the machine-executable instructions of the software programs.
- the processor-readable storage medium 1005 is one of (or a combination of two or more of) a hard drive, a flash drive, a DVD, a CD, an optical disk, a floppy disk, a flash storage, a solid state drive, a ROM, an EEPROM, an electronic circuit, a semiconductor memory device, and the like.
- the processor-readable storage medium 1005 can include an operating system, software programs, device drivers, and/or other suitable sub-systems or software.
- first, second, third, etc. are used to characterize and distinguish various elements, components, regions, layers and/or sections. These elements, components, regions, layers and/or sections should not be limited by these terms. Use of numerical terms may be used to distinguish one element, component, region, layer and/or section from another element, component, region, layer and/or section. Use of such numerical terms does not imply a sequence or order unless clearly indicated by the context. Such numerical references may be used interchangeable without departing from the teaching of the embodiments and variations herein.
Landscapes
- Business, Economics & Management (AREA)
- Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Accounting & Taxation (AREA)
- Theoretical Computer Science (AREA)
- Finance (AREA)
- General Business, Economics & Management (AREA)
- Strategic Management (AREA)
- Multimedia (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Cash Registers Or Receiving Machines (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
A system and method for applications of computer vision in linking users with virtual data that can include detecting digital interaction state of a plurality of subjects in an environment using at least one sensor-based monitoring system; detecting a contextual organization of subjects relative to an operator station; at the operator station, augmenting the user interface based on the contextual organization of subjects which comprises of at least: presenting a set of subject indicators in the user interface with the subject indicators arranged in response to contextual organization, and in response to received user interaction with at least one selected subject indicator, accessing the digital interaction state of the subject associated the at least one subject indicator.
Description
- This application is a Continuation Application of U.S. patent application Ser. No. 16/806,946, filed on 2 Mar. 2020, which claims the benefit of U.S. Provisional Application No. 62/812,265, filed on 1 Mar. 2019, both of which are incorporated in their entirety by this reference.
- This invention relates generally to the field of customer service technology, and more specifically to a new and useful system and method for applications of computer vision for linking users with virtual data.
- There is a growing interest in building digital interaction experiences in physical retail spaces that leverage new forms of sensor-based monitoring. At times these sensor-based monitoring systems are used to augment existing forms of interactions. In other cases, the sensor-based monitoring systems are used to facilitate completely new forms of experiences and interactions for users. In particular, computer vision is beginning to be used, sometimes along with other forms of digital sensing, to provide automated checkout. Many such systems are limited however. In particular many such systems are not compatible with existing checkout systems. As such, the existing infrastructure of established retailers is largely not compatible with these systems. Furthermore, there are no existing solutions on how an administrator or worker can interact with such systems to perform interactions like review the status of a user with sensed information. Thus, there is a need in the customer service technology field to create a new and useful system and method for applications of computer vision for linking users with virtual data. This invention provides such a new and useful system and method.
-
FIG. 1 is a schematic representation of a system of a preferred embodiment; -
FIG. 2 is a schematic representation of an exemplary scenario of the system and method applied across multiple operator stations; -
FIG. 3 is a schematic representation of an exemplary user flow for automated checkout; -
FIG. 4 is a schematic representation of an exemplary user flow for merging subjects during automated checkout; -
FIG. 5 is a schematic representation of an exemplary scenario for adding items at a service counter; -
FIG. 6 is a schematic representation of an exemplary scenario of an operator servicing customers with no pre-defined line, which illustrates changes in a subject management user interface based on orientation of an operator relative to customers; -
FIG. 7 is an exemplary screenshot of a subject management user interface using an image-based representation; -
FIGS. 8A-8C are exemplary variations of types of image-based representations; -
FIG. 9 is a flowchart representation of a method; -
FIG. 10 is a flowchart representation of an alternative variation of the method; -
FIG. 11 is a flowchart representation of a method applied to automated checkout; and -
FIG. 12 is an exemplary system architecture that may be used in implementing the system and/or method. - The following description of the embodiments of the invention is not intended to limit the invention to these embodiments but rather to enable a person skilled in the art to make and use this invention.
- A system and method for applying computer vision (CV) for linking users with virtual data of a preferred embodiment functions to leverage CV monitoring in facilitating digital interactions with customers from physical computing device user interface. The system and method preferably enables a convenient user interface through which digital information and/or records associated with a person can be easily accessed by another user through a computing device. In particular, the system and method may have utility in enabling a cashier user interface on a computing device to link customers and other types of users with virtual data.
- In one exemplary implementation, the system and method enables a subject management user interface to intelligently present subject indicators (e.g., user icons) that are arranged based on CV-based analysis on the position, orientation, attention, and organization of subjects in the environment relative to the computing device of the subject management user interface. Alternative types of user interfaces could similarly be presented such as rendering a live or recent image of a line and making the people in the image selectable items of a user interface, where selecting the people accesses some person associated state information. As the user interface is intelligently customized to the context and perspective of the operator of the computing device, an operator may more easily access the digital information linked (through CV monitoring and/or other forms sensor-based monitoring) to the users in proximity to the operator. On selecting one of the subject indicators, the operator may access the digital information either by viewing digital information of the user associated with the subject indicator and/or updating, adding, or deleting information of a data record associated with the user.
- The system and method enables a novel and new capability for user-operated computing devices to simplify how digital information of a modeled environment can be accessed. In a sense, the system and method can coordinates subject data tracking, subject relationship modeling from an operator perspective, and subject data synchronization with a device of the operator.
- For subject data tracking, a sensor-based monitoring system, such as a CV monitoring system, generates or otherwise establishes an association between a subject and at least one piece of subject data (e.g., a user profile, subject associated information or labeling, etc.) and then tracks that subject within an environment.
- Subject relationship modeling can use the same or different sensor-based monitoring system to interpret the contextual relationship of a particular device (or user) relative to a set of subjects in the environment. This modeling may be used to present an interface through which some operator of a device can more easily interact with the subject data. For example, a menu of selectable subject indicators can be presented in a way that prioritizes presentation of the subject indicators based on a modeled point of view of the device that is presenting the menu user interface.
- Subject data synchronization, preferably allows actions taken at an operator device to be applied to subject data. Subject data synchronization can enable an operator to access and interact with the subject data detected with the monitoring system. This may involve reading and using subject data and/or adding, updating, deleting, or otherwise editing subject data.
- Additionally, the subject data tracking, subject relationship modeling, and subject data synchronization may be performed and managed across multiple subjects and operators in an environment. In a retail setting, the system and method can be implemented across multiple customers and workers operating POS systems that using the capabilities of the system and method in parallel as shown in
FIG. 2 . - This coordinated sensing, modeling, and data synchronization and communication can be particularly useful in situations where ambient computing is sensing and modeling interaction state of one or more subjects in an environment. The system and method may be applied and implemented for a variety of applications.
- In one preferred implementation, the system and method are used in combination with a monitoring system used for automated checkout such as checkout-free shopping and/or assisted checkout. Herein, automated checkout can include fully automated and partially automated variations and is primarily characterized by a system or method that generates or maintains a checkout list (i.e., a virtual cart) during the shopping process of a user or users, with the objective of knowing the possessed or selected items for billing. The checkout list can be used in executing a checkout process to charge a customer for the selected items. The checkout process can occur when a customer is in the process of leaving a store. The system and method can be used so that a checkout list can be processed at a POS system, which may be the standard checkout process in the environment, a checkout option for customers without a connected checkout-free application or account, customers wanting to pay with cash or other payment options, and for any suitable reason.
- In the context of the system and method, checkout may be processed in part through interactions with a cashier-operated device such as a POS system. The checkout process could alternatively occur when a customer is in the process of leaving a store or when any suitable condition for completing a checkout process is satisfied such as when a customer selects a checkout option within an application.
- A checkout list may be maintained and tracked during a shopping experience through use of one or more monitoring systems. A checkout list (alternatively referred to as a checkout list) is a digitally stored data record or records modeling contents of a virtual cart (or what items are to be purchased). Any suitable number of databases and records may be used in representing item contents, item information (price, count, discounts, etc.) for items of a checkout list, and/or other information of the checkout list. In performing an automated checkout process, the system and method can charge a customer for the total of a shopping cart and/or alternatively automatically present the total transaction for customer completion. This may include entering the contents of a checkout list into a POS system for payment and/or supplying of payment information for checkout processing.
- A checkout list is preferably associated with a subject. May alternatively be associated with multiple subjects if multiple subjects are detected or otherwise specified as a group collaboratively building a shared checkout list
- Automated checkout is used herein to describe a variety of types of sensor-assisted forms of checkout. In one variation, automated checkout is facilitated through image analysis using video cameras and more specifically using a CV monitoring system. Other variations of automated checkout can use additional or alternative forms of sensor-based monitoring such as using sensor-enabled shelves, smart carts, RF ID tracking or smart infrastructure. Automated checkout in some variations may also include user-assisted forms of checkout such as where a customer uses an application operable on a personal computing device (e.g., a smart phone) to perform some form of pre-scanning of items.
- A checkout list may be generated in part or whole for a customer. In some variations, the checkout list can be synchronized with (or otherwise transmitted to) a point of sale (POS) system manned by a worker so that at least a subset of items can be automatically entered into the POS system thereby alleviating manual entry of the items. The system and method may be used in exposing cashier control over when and how a checkout list of a customer is processed by a POS system. The system and method may additionally or alternatively be used for providing customer care solutions, adding service counter items, and/or performing other operations.
- As an exemplary implementation, the system and method may be used within a POS system in a retail environment with an automated checkout system. A CV-based monitoring system and/or other sensor-based monitoring system may be used to contextually present user icons for various customers within the POS system as shown in
FIG. 3 . This is preferably used so that a worker operating the POS system can have a menu of nearby customers intelligently presented within an application of the POS system. The customers can be presented with visual representation of the customers and ordered in a sensible manner (e.g., an approximation of line order) so that it is easy for a worker to select a customer for checkout. For example, the customers next in line would generally be presented first in a list of other customers in line. The system and method in this use case may additionally enable an intuitive interface for “merging” checkout lists for grouping customers for a single checkout process as shown inFIG. 4 . An operator of the POS system could select one of the customers for checkout processing from a user interface customized to their particular perspective and the checkout list associated with that customer (or customers shopping together) can be processed for checkout. - As another alternative implementation, the system and method may be used to more easily update and edit the subject data associated with a subject. Similar to above, a CV-based monitoring system and/or other sensor-based monitoring system may be used to contextually present user icons for various customers within the POS system as shown in
FIG. 3 . A worker could then use this contextual user interface to select a customer and then supply additional information. In a grocery store setting, this may be used at a deli, fish, butcher, or bakery counter, where service counter items (which may be packaged and priced at the counter) are sold. The worker could prepare one or more service counter items for a customer, select the appropriate customer from a contextually generated user interface, and then enter the service counter items. The service counter items may then be added to a checkout list of the customer stored by an automated checkout system. This serves as an example of information being added to the subject data. - As another alternative implementation, the system and method may be used for a customer care system. A worker could similarly be presented with an array of graphical representations of customers (e.g., icons in the form of customer profile photos or in-store image captures) where the graphical representations are organized in a contextual manner based on visual interpretation of the relative position of the worker and customers in the store. In this way, a worker could quickly and seamlessly access a user profile for providing various forms of customer service. For example, a user may enter their customer care request through an app or some other computing device. A monitoring system could detect that the user is in the store and establish an association between the customer care request and the CV detected user. When a worker approaches the customer, the worker could see through a customer care app of the system and method that the customer had submitted the customer care request and the associated information (e.g., information about their problem or help request). In a crowded environment, the system and method may even help customize the contextual presentation of subject indicators based on the orientation and/or attention of an operator device and/or the operator as shown in
FIG. 6 . For example, the customer that the worker is facing and closest too may be presented first in the list of possible customers. - The system and method in one implementation may be able to facilitate a customer service tool to help with sales, customer appointments, customer questions, and the like. In the case of an environment with automated checkout, the system and method could be used so that a worker could easily access the checkout list information of a customer to assist a customer anywhere in the environment. This may be done in addition to or as an alternative to facilitating the checkout process within an application of the POS system.
- The system and method can be particularly applicable for facilitating individualized/personalized digital experiences in group-based computing environments. Group-based computing environments as used herein are used to characterize a new form of computing where a monitoring system tracks individual state of multiple users in an environment.
- The system and method are primarily described herein as it could be used along with systems for automated checkout, but the system and method are not limited such use cases. The system and method have particular applicability in the area of automated checkout as it provides a new mechanism through which such an automated checkout system can be integrated with a POS system used by a cashier. This could additionally be used for self-service checkout kiosks where a user with a tracked checkout list could complete a transaction at a self-service kiosk.
- Herein, users that have tracked digital interaction information, such as customers in a store, are described primarily described herein as subjects. In examples of retail settings, the term customers may be used. Users that interact with the user interface are described herein as operators. Operators in a retail environment may be workers, and, in examples of retail settings, the term worker is sometimes used herein. In some cases, an operator could also be a subject such as when the system and method is used with a self-help kiosk.
- The system and method may provide a number of potential benefits. The system and method are not limited to always providing such benefits, and are presented only as exemplary representations for how the system and method may be put to use. The list of benefits is not intended to be exhaustive and other benefits may additionally or alternatively exist.
- As one potential benefit, the system and method can increase efficiency by which workers serve customers. The system and method provide a convenient user interface for accessing customer information. Through this interface, users can better interact with the information generated and managed by a monitoring system. The system and method can avoid the clumsy process of gathering information from a user so that a digital record by searched for and found.
- As another benefit, the system and method may make access to user data more secure and private by limiting access to operator devices where it contextually makes sense. For example, a worker could be prevented from viewing any customer's checkout list except for those that queue in the worker's specific lane.
- As another potential benefit, the system and method provide an interface that can conveniently leverage the intuition of a human. In the case of a POS system of an automated checkout system, a worker can use their human intuition of who is next in line and who is checking out as a group to select customers for checkout. For example, the customers will generally form a line in front of the worker. However, the line will often deviate from an ideal line that can be easily interpreted from a CV monitoring system. As one exemplary scenario, a child may wander up and down the line away from a parent, but a worker observing social interactions would be able to interpret the situation to understand the child (and the products held by them) should be merged with the products of the parent. The system and method can easily handle this situation. As another exemplary scenario, groups of adults (e.g., couples, friends, etc.) could be easily segmented by a human worker for separating or grouping purchases by selectively asking simple questions (e.g., “Are you two paying together?”). The system and method can provide a convenient and efficient technology-based solution for managing digitally stored user data like checkout lists. The system and method facilitates a user interface that maintains this valuable form of human input.
- As a related potential benefit, the system and method can aid in minimizing errors and correcting issues. The system and method composes and transforms the sensed information into an intuitive user interface. The primary action of selecting an individual subject is directly made easier through recommendations and contextual organization of information. A secondary action such as merging digital information (e.g., checkout lists) of two or more subjects is additionally made easier. For example, in a cashier application on a POS system, an operator could easily select the appropriate icons of users that the operator would like to merge into a single checkout transaction and then initiate the checkout process. The related action of splitting a checkout transaction is also maintained with similar user interface actions. For example, the operator if wanting to undo a merge or remove one or more subjects from a merged checkout transaction could similarly select the subjects to remove (or deselect subjects from the active subjects in the checkout transaction).
- As shown in
FIG. 1 , a system for applying CV to a customer service application of a preferred embodiment can include asensing system 100, which includes at least a CV monitoring system no, and a subject management user interface (UI) 200. Thesensing system 100 is configured to determine individual state of a plurality of subjects in an environment. In a variation used with a system for automated checkout, the individual state of a subject relates to a prediction of items selected by the subject that are intended for purchase within a retail environment. The CV monitoring is additionally configured to determine contextual organization of the subjects, which is used in presenting the UI of thesubject management UI 200. - The
sensing system 100 of a preferred embodiment is preferably configured to collect or determine at least some form of state information relating to a subject. Thesensing system 100 can include one or more types of sensing systems. Asensing system 100 may collect data from the environment and then make a determination. Onesuch sensing system 100 can include a CV monitoring system no. Alternative and/or additional sensing systems could include smart cart/baskets, smart shelving, radar-based motion detection, wearable computer sensing systems (e.g., collecting data from inertial measurement units (IMUs) on phones and other activity sensing devices), and/or other suitable forms of sensing systems. - A
sensing system 100 may additionally or alternatively receive digital communication from a device such that thesensing system 100 facilitates tracking of various forms of information of devices in the environment. Examples of such sensing systems could include near field communication system used to detect presence of a phone or other type of device, a radio frequency identifier tracking (RFID) system for tracking RF tags other RF identifiable objects, user applications that log information, and/or systems used to track and relay information from a computing device. - The
sensing system 100 may operate in combination with one or more outside systems to facilitate rich forms of information modeling for subjects of the environment. Oneexemplary sensing system 100 is one used to facilitate some form of automated checkout. Thesensing system 100 may therefore be configured to generate and track a checkout list for subjects in the store. As mentioned above, thesensing system 100 may additionally or alternatively be applied towards other forms of customer service. For example, subject profile information may be tracked such as product browsing history, which may be used by a salesperson when helping the subject. In another example, subject profile information can be used to enable a customer service representative to quickly access customer-submitted issues or questions when engaging with the customer within the environment. - The
sensing system 100 preferably includes one or more computer processes and specially configured machine-readable instructions stored on a computer-readable medium. The machine-readable instructions are preferably specially configured so that execution and processing by the one or more processors, cause thesensing system 110 to detect subjects in the environment and establish an association between the detected subjects and at least one piece of data. More specifically for an automated checkout variation, the machine-readable instructions are preferably specially configured so that execution and processing by the one or more processors, cause thesensing system 110 to track a set of subjects through the environment and, for each subject, detect item interaction events, and update items in a checkout list based on the item interaction event. The item interaction events can include item section events and optionally item deselection events. - As for subject relationship modeling, the machine-readable instructions are preferably specially configured so that execution and processing by the one or more processors cause the
sensing system 110 to detect a contextual organization of subjects relative to an operator station. This can include detecting subjects in proximity to an operator device (and/or an operator) and modeling line order, subject priority, subject and/or operator attention, and/or performing other operations to detect a contextual organization of subjects. - Preferably, the
same sensing system 100 is used for subject data tracking and subject relationship modeling. However, in some variations,different sensing systems 100 may be used for subject data tracking and subject relationship modeling. - The
sensing system 100 is preferably deployed in at least one location of the environment. In some variations, thesensing system 100 is deployed across a region of the environment and in some cases across a significant region of the environment. A significant region of the environment could be a region with substantial sensing coverage, which may include coverage percentages such as 10% coverage, 50% coverage, and/or 90% or more coverage. - A
CV monitoring system 110 of a preferred embodiment functions to transform image data collected within the environment into observations relating in some way to subjects in the environment. Preferably, the CV monitoring system no is used for detecting items (e.g., products), monitoring users, tracking user-item interactions, and/or making other conclusions based on image and/or sensor data. TheCV monitoring system 110 will preferably include various computing elements used in processing image data collected by an imaging system. In particular, theCV monitoring system 110 will preferably include an imaging system and a set of modeling processes and/or other processes to facilitate analysis of user actions, item state, and/or other properties of the environment. As mentioned theCV monitoring system 110 in one exemplary implementation is used to track a checkout list of subjects for offering automated checkout. - The
CV monitoring system 110 preferably provides specific functionality that may be varied and customized for a variety of applications. In addition to item identification, theCV monitoring system 110 may additionally facilitate operations related to person identification, checkout list generation, item interaction tracking, store mapping, and/or other CV-based observations. Preferably, theCV monitoring system 110 can at least partially provide: person detection; person identification; person tracking; object detection; object classification; object tracking; gesture, event, or interaction detection; detection of a set of customer-item interactions, and/or forms of information. - In one preferred embodiment, the system can use a
CV monitoring system 110 and processing system such as the one described in the published US Patent Application 2017/0323376 filed on May 9, 2017, which is hereby incorporated in its entirety by this reference. TheCV monitoring system 110 will preferably include various computing elements used in processing image data collected by an imaging system. - The imaging system functions to collect image data within the environment. The imaging system preferably includes a set of image capture devices. The imaging system might collect some combination of visual, infrared, depth-based, lidar, radar, sonar, and/or other types of image data. The imaging system is preferably positioned at a range of distinct vantage points. However, in one variation, the imaging system may include only a single image capture device. In one example, a small environment may only require a single camera to monitor a shelf of purchasable items. The image data is preferably video but can alternatively be a set of periodic static images. In one implementation, the imaging system may collect image data from existing surveillance or video systems. The image capture devices may be permanently situated in fixed locations. Alternatively, some or all may be moved, panned, zoomed, or carried throughout the facility in order to acquire more varied perspective views. In one variation, a subset of imaging devices can be mobile cameras (e.g., wearable cameras or cameras of personal computing devices). For example, in one implementation, the system could operate partially or entirely using personal imaging devices worn by users in the environment (e.g., workers or customers). In some variations, a mobile camera mounted to a worker can be used for generating a rendering a live or recent image of a line and making the people in the image selectable items of a user interface of an operator device.
- The imaging system preferably includes a set of static image devices mounted with an aerial view from the ceiling or overhead. The aerial view imaging devices preferably provide image data that observes at least the users in locations where they would interact with items. Preferably, the image data includes images of the items and users (e.g., customers or workers). While the system (and method) are described herein as they would be used to perform CV as it relates to a particular item and/or user, the system and method can preferably perform such functionality in parallel across multiple users and multiple locations in the environment as shown in
FIG. 2 . Therefor, the image data may collect image data that captures multiple items with simultaneous overlapping events. The imaging system is preferably installed such that the image data covers the area of interest within the environment. - Herein, ubiquitous monitoring (or more specifically ubiquitous video monitoring) characterizes pervasive sensor monitoring across regions of interest in an environment. Ubiquitous monitoring will generally have a large coverage area that is preferably substantially continuous across the monitored portion of the environment. However, discontinuities of a region may be supported. Additionally, monitoring may monitor with a substantially uniform data resolution or at least with a resolution above a set threshold. In some variations, a
CV monitoring system 110 may have an imaging system with only partial coverage within the environment. - A CV-based processing engine and data pipeline preferably manages the collected image data and facilitates processing of the image data to establish various conclusions. The various CV-based processing modules are preferably used in generating user-item interaction events, a recorded history of user actions and behavior, and/or collecting other information within the environment. The data processing engine can reside local to the imaging system or capture devices and/or an environment. The data processing engine may alternatively operate remotely in part or whole in a cloud-based computing platform.
- User-item interaction processing modules function to detect or classify scenarios of users interacting with an item. User-item interaction processing modules may be configured to detect particular interactions through other processing modules. For example, tracking the relative position of a user and item can be used to trigger events when a user is in proximity to an item but then starts to move away. Specialized user-item interaction processing modules may classify particular interactions such as detecting item grabbing or detecting item placement in a cart. User-item interaction detection may be used as one potential trigger for an item detection module.
- A person detection and/or tracking module functions to detect people and track them through the environment.
- A person identification module can be a similar module that may be used to uniquely identify a person. This can use biometric identification. Alternatively, the person identification module may use Bluetooth beaconing, computing device signature detection, computing device location tracking, and/or other techniques to facilitate the identification of a person. Identifying a person preferably enable customer history, settings, and preferences to be associated with a person. A person identification module may additionally be used in detecting an associated user record or account. In the case where a user record or account is associated or otherwise linked with an application instance or a communication endpoint (e.g., a messaging username or a phone number), then the system could communicate with the user through a personal communication channel (e.g., within an app or through text messages).
- A gesture, event, or interaction detection modules function to detect various scenarios involving a customer. One preferred type of interaction detection could be a customer attention tracking module that functions to detect and interpret customer attention. This is preferably used to detect if, and optionally where, a customer directs attention. This can be used to detect if a customer glanced in the direction of an item or even if the item was specifically viewed.
- The item detection module of a preferred embodiment, functions to detect and apply an identifier to an object. The item detection module preferably performs a combination of object detection, segmentation, classification, and/or identification. This is preferably used in identifying products or items displayed in a store. Preferably, a product can be classified and associated with a product SKU identifier. In some cases, a product may be classified as a general type of product. For example, a carton of milk may be labeled as milk without specifically identifying the SKU of that particular carton of milk. An object tracking module could similarly be used to track items through the store. Tracking of items may be used in determining item interactions of a user in the environment.
- The item detection module in some variations may be integrated into a real-time inventory system. The real-time inventory system functions to detect or establish the location of inventory/products in the environment.
- Alternative forms of CV-based processing modules may additionally be used such as customer sentiment analysis, clothing analysis, customer grouping detection (e.g., detecting families, couples, friends or other groups of customers that are visiting the store as a group), and/or the like. The system may include a number of subsystems that provide higher-level analysis of the image data and/or provide other environmental information such as a real-time virtual cart system.
- The real-time virtual cart system functions to model the items currently selected for purchase by a customer. The real-time virtual cart system can manage the checkout lists of the various subjects (i.e., customers) in a retail environment. The virtual cart system may enable forms of automated checkout such as automatic self-checkout (e.g., functionality enabling a user to select items and walk out) or accelerated checkout (e.g., selected items can be automatically prepopulated in a POS system for faster checkout). Product transactions could even be reduced to per-item transactions (purchases or returns based on the selection or de-selection of an item for purchase). The virtual cart system may be integrated with the system to provide purchase or planned purchase information, which may be used as a condition for delivering content. The type of content delivered to customer may be based in part on their current cart contents. For example, a coupon may be selected and delivered to a customer for a particular brand of ketchup buns based in part on the customer having hamburger buns and ground beef in the cart.
- The
CV monitoring system 110 may additionally or alternatively include human-in-the-loop (HL) monitoring which functions to use human interpretation and processing of at least a portion of collected sensor data. Preferably, HL monitoring uses one or more workers to facilitate review and processing of collected image data. The image data could be partially processed and selectively presented to human processors for efficient processing and tracking/generation of a checkout list for users in the environment. - The
CV monitoring system 110 can be used within the system for one or more modes of subject monitoring. One form of subject monitoring is a subject relationship modeling mode that functions to coordinate and direct how subject information can be presented in thesubject management UI 200. The CV monitoring is preferably used in this mode to detect subjects and their contextual organization (e.g., from a position, orientation, and attention approach) relative to a worker and/or subject management user interface. For example, theCV monitoring system 110 is preferably used to approximate the priority or order in which subject icons should be presented in a cashier application when the subjects are lined up for checkout. - In a related mode of subject monitoring, the
CV monitoring system 110 can be used in synchronizing subject information from some sensing system 100 (possibly also the CV monitoring system 110) with CV-detected subjects. This functions to associate collected state information with observed subjects. - An additional mode of subject monitoring could be one used for collecting subject state information as was described above such as the generation of a checkout list. However, the subject state information may alternatively be tracked and monitored through alternative types of sensing systems. In the automated checkout variation, this collected state information can include the checkout list. It could additionally include other information like user activity during a visit or during previous visits to a store, alerts associated with the user, and/or any suitable type of information.
- The subject management UI of a preferred embodiment functions to facilitate some form of customer service as directed by a human operator. In facilitating, actions related to customer service, the
subject management UI 200 preferably uses contextual organization status as detected by theCV monitoring system 110 to augment the presentation and interactions of thesubject management UI 200. - The
subject management UI 200 is preferably operable on a computing device. The computing device will preferably have a graphical user interface output but may alternatively or additionally have UI output in an audio format or other alternative medium of UI output. Thesubject management UI 200 is preferably implemented as an application or as part of a larger application, code-base, operating system, or other suitable type of software implemented solution. - The computing device can be a desktop or laptop computer, a tablet, smart phone, smart watch, smart glasses, smart headphones, augmented reality device, virtual reality device, and/or any suitable type of computing device, and/or any suitable type of computing device. The computing device could additionally be a POS system or some other purpose-built computing device. Multiple
subject management UI 200 instances can be operating simultaneously within an environment. The implementation of thesubject management UI 200 can vary depending on the type of computing device and the intended use case of the system. The location and presence of asubject management UI 200 may be registered or otherwise configured within the system. Alternatively the location and presence may be dynamically detected. The computing device of thesubject management UI 200 may broadcast an identifying signal to the system. Thesensing system 100 may alternatively detect and determine a location of the computing device. In some variations, the orientation of the computing device may be determined. In one variation, a visual marker can be placed on or near the location of the computing device such that that location can be used as a reference point for an associated computing device. - When used to assist in the checkout process, the
subject management UI 200 is preferably part of a POS system. More specifically, thesubject management UI 200 can be or be part of a cashier application operable on a tablet computer that is directly or communicatively coupled with a POS system. In this exemplary implementation, an operator (e.g., cashier) will have a tablet with a touch user interface that can present subject icons that can be used to determine who will be selected for a checkout transaction. The computing device of thesubject management UI 200 may alternatively interface with a POS system and/or other additional systems. In another variation, the computing device ma be a POS processing system wherein checkout processing (e.g., price calculation, payment receipt, receipt generation, etc.) are handled by the computing device. - The
subject management UI 200 will preferably include a number of views. Thesubject management UI 200 will preferably include a subject selection view, which is configured to present a set of subjects based on modeling of the context and relationship between the operator device and subjects in the environment. In one exemplary implementation, subjects are presented in an order intended to approximate the expected line order. This modeling of line order can factor in the detected physical proximity and order within a lineup region. The modeling may additionally consider the time of arrival to some region in the environment, the attention of the operator and/or subject, the orientation of an operator device, the subject data associated with the subjects, and/or other suitable factors. - The
subject management UI 200 will then preferably facilitate accessing subject-related data. Accessing may include reading and displaying information based on the subject data. For example, the contents of a checkout list as determined by a CV monitoring system may be displayed in response to operator selection of a corresponding subject indicator. In another variation, data input received at the operator device may be added to subject data in response to selection of the subject. For example, an issue can be resolved for a checkout list of a real-time virtual cart system. In another example, an item may be added to a checkout list of a subject after selection of the appropriate subject in asubject management UI 200. - As shown in
FIG. 9 , a method for applying CV to a customer service application of a preferred embodiment can include detecting interaction state data of a plurality of subjects in an environment S110; detecting a contextual organization of subjects relative to an operator station S120; and at a computing device of the operator station, augmenting a user interface based on the contextual organization of subjects S130. The subject-associated data of a subject is preferably accessible in the user interface. Augmenting the user interface S130 in one preferred variation can include presenting subject indicators in the user interface with the subject indicators arranged in response to contextual organization S132; and in response to user interaction with a subject indicator, accessing the digital interaction state data of a subject associated with the subject indicator S134. - More generally, the method may be applied in a similar manner to any suitable type of user-associated data. As shown in
FIG. 10 , the method can include detecting user-associated data of a plurality of subjects in an environment S210; detecting a contextual organization of subjects relative to an operator station S220; and at a computing device of the operator station, augmenting a user interface based on the contextual organization of subjects S230, which includes presenting subject indicators in the user interface with the subject indicators arranged in response to contextual organization S232, and, in response to user interaction with a subject indicator, accessing the user-associated data of a subject associated with the subject indicator S234. User-associated can be any suitable type of data that can be linked to a subject. - Specifically, the method may be used in presenting a menu of customer-associated icons (i.e., subject indicators) as selectable UI elements. The customer-associated icons can be presented in an order or manner based on the customers' organization around an operator station (or an operator). In a variation using a CV monitoring system, the customer-associated icons are presented based on interpretation of images capturing the organization around an operator station. For example, the order in which customers are waiting in line can be used to at least partially determine the order of the customer-associated icons. The customer-associated icons are used to access the customer information, which in this case could be information such as a checkout list information that could be automatically added to a POS system.
- Method is primarily described as it is applied to one operator station and computing device. The method will generally be used in a computer environment and/or for a use case where multiple subject management user interfaces are used. When applied across an environment the method may alternatively be described as detecting digital interaction state data of a plurality of subjects in an environment S110; across a set of operator stations, detecting a contextual organization of subjects relative to each particular operator station S120; and at a computing device instance of each operator station, augmenting a user interface of the computing device instance based on the contextual organization of subjects detected for the associated operator station S130. Augmenting the user interface can similarly include presenting subject indicators in the user interface with the subject indicators arranged in response to contextual organization S132, and, in response to user interaction with a subject indicator, accessing the digital interaction state data of a subject associated with the subject indicator S134.
- For example, the method is preferably usable across multiple customers in a retail environment, wherein each POS checkout station can apply the method in assisting a worker at that particular station. Each POS terminal of each checkout lane of a grocery store may be augmented through the method described here. Each individual computing device of a POS terminal may have an interface that intelligently presents subject indicators for the subjects waiting in that particular checkout lane as shown in
FIG. 2 . - In the case where its applied to automated checkout as shown in
FIG. 11 , the method could include tracking checkout list status of a plurality of shoppers S310; across a plurality of POS checkout stations, detecting a contextual organization of at least a subset of shoppers relative to each POS checkout station S320; and, at the POS checkout station, presenting shopper-associated UI elements in a UI of the POS checkout station, where the UI elements are organized by the contextual organization S332; and upon detecting selection of a shopper associated UI element, selecting a checkout list associated with the shopper for checkout processing S334. Multiple selections can be used to merge checkout lists of two or more shoppers. Alternatively, deselection of a UI element may remove a checkout list from checkout processing. - The method is preferably implemented through a system substantially similar to the one described above but may alternatively be implemented with any suitable system. The method is described primarily as it may be applied to a CV monitoring system for tracking checkout list information and then facilitating automated checkout through a POS system in a retail setting, but the method may additionally or alternatively be used for a variety of digital interactions offered in an environment shared by multiple subjects.
- Block S110, which includes detecting digital interaction state of a plurality of subjects in an environment, functions to collect subject-associated information from a plurality of subjects in an environment. Detecting digital interaction state of a plurality of subjects is preferably implemented through remote sensing using at least one sensor-based monitoring system such as a CV monitoring system, sensor-enabled smart shelf system, smart carts, RF ID tracking system, other types of smart infrastructure, and/or additional sensor modalities.
- The method is preferably applied to digital interactions that are individualized for a plurality of subjects. In the retail space, this may include tracking a checkout list of items selected for checkout while in the store, tracking browsing history of individual customers, receiving customer service request information form multiple customers in the store, and/or other suitable applications.
- Through execution of block S110, at least some subjects in the environment will be associated with digital interaction state information. In many usage scenarios, multiple subjects will be detected and associated with digital interaction state information. The digital interaction state information can be dynamic and based on ongoing interactions within the environment. Alternatively, digital interaction state information could be state information detected through discrete, isolated events. For example, digital interaction state information may be generated when a subject enters the environment (or for any other type of event). In one exemplary variation, a user may perform some action like picking up a restricted item (e.g., alcohol) and this may be stored as interaction state data and used to indicate to a cashier which customers have picked up alcohol by labeling subject indicator icons with a graphical indicator, which may signal to the cashier that they should request ID.
- As in the system described above, a variety of types of sensing systems may be used individually or in combination to collect the digital interaction state of a subject. Detecting digital interaction state preferably includes sensing the digital interaction state (and more generally activity) through one or more sensor-based monitoring systems. Sensing activity can include sensing activity across an environment, detecting subjects within the environment, and individually associating data with a set of subjects. The approach for sensing activity may vary depending on the type of digital interaction state monitored for the subjects. In a retail environment, item/product interactions may be monitored by sensing shelf activity through shelf-based sensors, detecting user-item interactions through CV analysis of image data (e.g., detecting item pickups and setdowns), and/or other techniques. Similarly, detecting user presence (which may be compared to item interactions for determining which user picked up an item for example) may include detecting user presence through one or more of computer vision, proximity detection, user and/or device location tracking (GPS, RF-based location tracking, wireless signal triangulation, etc.).
- In variations of the method used with automated checkout, the digital interaction state can include checkout list data and detecting digital interaction state can include tracking a set of subjects through the environment; and for each subject, detecting item interaction events including at least item selection events and updating items in a checkout list based on the item interaction event. Detecting item interaction events including at least item selection events and updating items in a checkout list based on the item interaction event function to track the accumulation of items by individual subjects. The checkout lists are preferably associated with individual subjects, but may alternatively be associated with multiple subjects. This process for maintaining a checkout list in some preferred implementations using CV image processing wherein the method includes collecting image data and processing the image data through a CV image processing pipeline.
- In some variations, the digital interaction state may occur outside of a
sensing system 100 and could simply be information associated with a user record that can be synchronized with a subject in the record. In this variation, block S110 may include setting digital interaction state of a user record within a data management system. - The digital interaction state can be a complex information model like in the case of a checkout list. A checkout list may have an associated data model that specifies a list of items, prices or product codes associated with those items, exceptions or flags for the shopping experience or individual products (e.g., alcoholic drinks may be flagged for ID check, produce may be flagged for weighing). The approach for detecting item interaction events used to update a checkout list can depend on the type of sensor monitoring systems employed. In a CV variation, a CV monitoring system may be used in detecting items and detecting user-item interactions. In another example, a multi-sensor monitoring system may involve detecting item activity from a sensor-enabled shelf, detecting presence of a user near the location of the item activity through another sensor monitoring system (e.g., CV-based people tracking), and associating item activity with the user. In one variation, detecting user-item interactions can include using computer vision processing of image data and smart shelf event data. Other alternative approaches may similarly be applied to tracking of a checkout list.
- As an alternative example, the digital interaction state could be as simple as binary state variable signifying if a user is requesting customer care assistance. In this way, a worker could view which customers in close proximity to them desire assistance. In a similar manner, any sort of data may be stored and associated with a user. Such information could be stored as customer profile data. The method could facilitate workers more quickly accessing information of a customer. Examples of types of customer profile data that could be useful for a worker to quickly access could include: a customer's service request, online orders being picked up in store, account status (e.g., type of account or membership status at a club, gym, bank, airport, etc.), activity history, and/or other types of customer information. The digital interaction state could be any suitable type of information, associated media files, and/or other suitable forms of digital information.
- As discussed, a CV monitoring system is one preferred type of sensing system, and a CV monitoring system is preferably used at least in part for executing block S120. Accordingly, the method and more specifically can additionally include collecting image data, applying CV processing of the image data and generating digital interaction state resulting in part from the CV processing. In the variation of using the method with automated checkout, generating digital interaction state can include generating digital interaction state indicating the checkout list resulting in part from the computer vision processing.
- Collecting image data in an environment, functions to collect video, pictures, or other imagery of a region containing objects of interest (e.g., inventory items). Image data is preferably collected from across the environment from a set of multiple imaging devices. Preferably, collecting image data occurs from a variety of capture points. The set of capture points include overlapping and/or non-overlapping views of monitored regions in an environment. Alternatively, the method may utilize a single imaging device, where the imaging device has sufficient view of the exercise station(s). The imaging data preferably substantially covers a continuous region. However, the method can accommodate for holes, gaps, or uninspected regions. In particular, the method may be robust for handling areas with an absence of image-based surveillance such as bathrooms, hallways, and the like.
- The imaging data may be directly collected, and may be communicated to an appropriate processing system. The imaging data may be of a single format, but the imaging data may alternatively include a set of different imaging data formats. The imaging data can include high resolution video, low resolution video, photographs from distinct points in time, imaging data from a fixed point of view, imaging data from an actuating camera, visual spectrum imaging data, infrared imaging data, 3D depth sensing imaging data, parallax, lidar, radar, sonar, passive illumination, active illumination, and/or any suitable type of imaging data.
- The method may be used with a variety of imaging systems, collecting imaging data may additionally include collecting imaging data from a set of imaging devices set in at least one of a set of configurations. The imaging device configurations can include: aerial capture configuration, shelf-directed capture configuration, movable configuration, and/or other types of imaging device configurations. Imaging devices mounted over-head are preferably in an aerial capture configuration and are preferably used as a main image data source. In some variations, imaging devices may include worn imaging devices such as a smart eyewear imaging device. This alternative movable configuration can be similarly used to extract information of the individual wearing the imaging device or other observed in the collected image data.
- In applying CV processing and generating digital interaction state a variety of techniques may be used. In some instances, applying CV processing is used in detecting a user and establishing association with stored digital interaction state data, which may originate from user profile data. In a variation of the method used with automated checkout, an application of CV processing in detecting interaction state can include tracking a set of subjects through the environment; for each subject, detecting item interaction events, updating items in a checkout list based on the item interaction event (e.g., adding or removing items). The checkout list can be a predictive model of the items selected by a customer, and, in addition to the identity of the items, the checkout list may include a confidence level for the checkout list and/or individual items. The checkout list is preferably a data model of predicted or sensed interactions. Other variations of the method may have the checkout list be tracking of the number of items possessed by a customer or detection of only particular item types (e.g., controlled goods like alcohol, or automatic-checkout eligible goods). The CV monitoring system may use algorithmic approaches applying traditional computer vision techniques, deep learning models, machine learning, heuristic modeling, and/or other suitable techniques in processing the image data. The CV monitoring system may additionally use HL in evaluating image data in part or whole.
- Alternative approaches may apply various forms of sensor fusion with CV or image-based monitoring systems.
- Block S120, which includes detecting a contextual organization of subjects relative to an operator station, functions to analyze the setting in proximity to the operator station. Detecting the contextual organization of subjects preferably includes applying computer vision to interpreting relative position, orientation, attention and other properties of a subject in a region near the operator station. The contextual organization of subjects may predict or characterize the order of customers in a line (i.e., line order) as it would be relevant to a worker and operator of the operator station. The contextual organization of subjects preferably models from the point of view of the operator an appropriate and intuitive way of presenting subjects.
- An output of detecting contextual organization of subjects can be a prioritized list of subjects ordered in some manner.
- In one variation, all subjects in the environment may be analyzed. Alternatively, a subset of subjects may be analyzed. For example, subjects located in a predefined region may be analyzed. In one implementation, subjects within some threshold distance in proximity to an operator station are evaluated for contextual organization. Additionally or alternatively, up to a maximum number of subjects may be evaluated for contextual organization. These subjects will generally be those closest and in the correct direction or designated region. For example, the subjects close to an operator station but in the line for an adjacent operator station are preferably not considered and only the subjects in the line region for the operator station are evaluated.
- A CV monitoring system is preferably used in evaluating the contextual organization of subjects relative to the operator station. Image data is preferably collected for each operator station and analyzed by detecting location of subjects and locations of operator stations. Additionally, CV processing of the image data may include detecting designated line regions, detecting direction of attention or body pose orientation of subjects and workers, direction/orientation of a computing device of the operator station, and/or other parameters. These various factors may be modeled to evaluate the contextual organization of the subjects.
- The operator station could be defined by the physical device such as the POS system, the position of the operator (e.g., the worker), and/or a defined region in the environment. The method may include detecting presence of an operator station. Detecting presence of an operator station may include visually detecting the operator station through CV processing of the image data. A visual marker may be applied communicating an identifier of the operator station. In another variation, the location of the operator station can be configured within the CV monitoring system.
- The contextual organization may be continuously updated, periodically updated, in response to an event like a change in image data, or as a result of any suitable trigger.
- In one preferred approach, detecting contextual organization of subjects comprises detecting proximity of a subject to an operator station. Detecting proximity may include measuring position of the checkout station; for a set of subjects, detecting position; and measuring checkout station proximity. In some variations, the proximity could be an absolute magnitude measurement. In other variations, the proximity could be based on displacement in one or more dimensions. Various heuristics, filters, or machine learning processes may additionally be used to appropriate limit subjects of interest. For example, a subject filter could be used to ignore subjects lined up for an adjacent checkout station.
- In a variation to detecting proximity, detecting contextual organization of subjects may include detecting line order which functions to analyze orientation and position patterns. Detecting line order may be specifically built around assessing order of subjects along a line. Detecting line order can include analyzing orientation and position of subjects relative to the operator station. Various heuristics, statistical models, machine learning models, and/or other techniques may be used to make such assessment useful.
- Additionally or alternatively, detecting contextual organization of subjects can include detecting attention, which functions to interpret the attention of an operator and/or subjects. In one implementation, detecting body pose orientation of users through computer vision and then assigning attention metrics to subjects based on relative attention. For example, the operator detected to be facing a subject that facing the operator will have a high attention score. This may be used so that a list of customers can be presented based on whom a worker is facing. In some variations, the attention and orientation of an operator or an operator device may be used to determine aspects of a line. Accordingly, detecting line order can include detecting a direction of attention of one of the operator station or an operator. This may be used to determine the most likely first person in a line. In a grocery store, some subjects accompanying another subject may line up past the cashier and it will actually be the subject in position to pay that should be considered as the highest priority in the line. In another application, some situations may have a line form around a customer service representative with no defined lane for lining up as shown in
FIG. 6 . In this way, an organic line may form based on the dominant orientation of the customer service representative. Detecting this dominant orientation may be used in classifying and assigning a line order to subjects in near proximity. - In a related variation, detecting contextual organization of subjects may include detecting mutual attention of an operator and at least one subject, which functions to assign priority based in part on shared attention between an operator and one or more subjects. For example, a customer and a cashier looking at each other (or at least directing attention towards each other) can prioritize that subject relative to other subjects.
- In another variation, subjects may be classified visually as being in a waiting state (i.e., a queuing state). A subject that is near an operator station but is walking away or participating in shopping activity may be determined to not be in a waiting state and may be excluded from a prioritized list of subjects.
- In another variation, detecting contextual organization of subjects may include detecting social grouping of subjects, which functions to determine if and which subjects may be part of a group. This may additionally include associating multiple subjects with a single checkout list. Associating multiple subjects with a single checkout list can include merging checkout lists from multiple subjects of a group. In another variation, associating multiple subjects with a single checkout list can include associating subjects to one indicated checkout list. This can be used to group families and groups of friends. Detecting social grouping can include detecting a social association of two or more subjects from historical interactions in the environment. Accordingly, how two or more subjects interact in the environment can lead the monitoring system to determine they are some form of pairing. In the case where one subject is detected as a child and the other as an adult, then the child and adult may be paired for checkout. In the case of two adults, where it can be difficult to know if the adults will want separate or shared checkouts, the contextual organization may reflect their pairing but not default to associating the subjects directly.
- The contextual organization may additionally depend on analysis and information that originates from image data when the subject is not near the operator station and is instead determined based on previous interactions in the environment. Detecting a social grouping could be achieved by detecting shared paths, sustained proximity, detecting interactions, detecting sharing of a cart, detecting one or more instances of communication (e.g., directed visual attention, directed speech, physical contact, etc.), and/or making other suitable observations. In one implementation, social grouping may alternatively be based on information from a digital platform such as a social network or any suitable source. For example, a subject may preregister or link their account with the accounts of their family members.
- As another variation, the contextual organization may factor in digital interaction state information. For example, contextual organization may consider information of a checkout list like indication of exceptions, no items, needs weight check, and the like.
- In one implementation, machine learning model may be trained using a training set mapping a number of relevant contextual features can be detected and resulting user interface interactions. This training may be used to adapt to different environmental conditions. Different stores have different checkout line organizations, machine learning applied to improving prioritization of subject position can be used adapt to the various environments.
- As part of S120 or as an additional step the method can include establishing an association of a CV-detected person (a subject) with digital interaction state information. In some implementations, association is established between CV-detected person and a user record, and the digital interaction state information is data linked to the user record. Establishing an association can be facilitated through the CV monitoring system or other techniques such as those described in U.S. patent application Ser. No. 16/015,180, filed on 21 Jun. 2018, which is hereby incorporated in its entirety by this reference.
- Block S130, which includes augmenting a user interface at a computing device of the operator station based on the contextual organization of subjects, functions to customize presentation or interactions of an application. The user interface is preferably augmented to present subject related information in a manner that is relevant to the perspective of the operator. In the automated checkout use case, the computing device can be a POS system, and augmenting a user interface can be used to provide a contextually aware menu from which a cashier can select one or more subjects for checkout. This can enable a cashier to better interact and facilitate automated forms of checkout. More generally, block S130 can be used to enable an operator to be presented with a menu of possible subjects, where the subjects are organized based on the contextual organization (e.g., proximity, line order, who the operator is facing) and the like.
- In one preferred variation, block S130 can include presenting subject indicators in the user interface with the subject indicators arranged in response to contextual organization S132; and in response to user interaction with a subject indicator, accessing the digital interaction state data of a subject associated with the subject indicator S134. Selecting a subject indicator by an operator may be used to access a user interface view with more detailed information or to select that subject to perform some additional action. Accessing the digital interaction state data may include reading the digital interaction state data and/or writing to the digital interaction state data. In some variations, the digital interaction state data may in part be presented or used in determining the user interface prior to any user interaction. For example, checkout list contents may be displayed in part next to the subject indicators prior to any selection of a subject.
- Block S132, which includes presenting subject indicators in the user interface with the subject indicators arranged in response to contextual organization, functions to display subject indicators in a manner that corresponds to the operator's perspective. The contextual organization preferably considers physical conditions of the environment (e.g., relative positions of subjects, the operator, and other items in the environment) as well as the objectives of the operator (e.g., to checkout shoppers in a line).
- A subject indicator is preferably a graphical representation of a subject. In one variation that may be a graphical icon that could show an image of the subject. The method in some implementations may include capturing an image of subjects while in the environment and using the image in the graphical representation. A dedicated image capture device may be used to capture images of the subjects' faces. In another variation, a user profile image of a subject could be used. Other variations may include using other symbolic representations such as clothing indication (e.g., showing a red shirt), indicating items from cart, or providing other descriptors that could be used to identify a subject.
- Along with the subject indicators the subject management UI can additionally present other information such as digital interaction state information. For automated checkout, the subject indicators may be marked with indication of errors, cart exceptions, number of items in cart, tasks for the worker such as weighing produce, and/or other suitable information.
- The user interface preferably presents a set of subject options (i.e., subject indicators). This could be presented as a list view, a gallery, explorable cards, or other suitable interfaces. In general, the graphical representations of subjects are selectable. In some variations, multiple subjects are selectable. Once selected, one or more actions may be executed on behalf of the subjects. An executed action will preferably use the digital interaction state of the subject. In the case of automated checkout, the checkout list(s) of the selected subject(s) are entered into the POS system for checkout processing. The order of the subject options is preferably based on the contextual organization information from S120. Accordingly, presenting a set of subject indicators in the user interface with the subject indicators arranged in response to contextual organization can include ordering the subject indicators in an order corresponding to the line order. For example, subject indicators are preferably presented in an order that reflects a real world line. Order may a sequential order, but in a user interface, order may determine priority within the user interface. User interface arrangements based on order can determine placement, sequential order, size of the UI element, visibility (which subject indicators to show and when), page order, graphical presentation (e.g., how the subject indicator is presented in terms of the information included, etc.), and/or other variables of a user interface.
- In the variation, where a group of subjects are grouped, presenting subject indicators may include representing a group of subjects as a single subject indicator or a grouping of subject indicators, which functions to present subjects by social grouping. Interactions can be applied across the group. This social grouping in some cases override the real world line such as for grouping a child with an adult subject in the user interface. Attention of the subjects and/or operator may similarly alter organization. In some variations, the digital interaction state data of each subject member of the group may be combined into one digital interaction state for the group. For example, the individual checkout lists of multiple customers can be combined into a single group checkout list. Alternatively, access to the digital interaction state in block S134 may involve performing some action (e.g., updating data) to each of the individual digital interaction state data of each group member.
- In general presenting a subject indicator within the user interface includes enabling user selection of the subject indicator from the user interface. For example, a user may be able to select a subject indicator to indicate they want to perform a checkout operation for them. Additionally or alternatively a set of interactive items may be performed in connection to the subject indicator. For example, sub-menu could be presented in connection to a subject indicator allowing an operator to perform various actions like select the option to add an item. Various user interface approaches may be used.
- In some variations, groups of subjects may be manually created by an operator, which can include receiving user input indicating an associated group of subjects, and in response to indication of an associated group of subjects, grouping the subject indicators. This may additionally include merging the digital interaction data of the subjects.
- Selecting a subject in the user interface will generally depend on device input such as touch input on a tablet-like device or mouse/keyboard input on a computer. Selecting a subject could additionally use contextual organization. For example, the directed attention of an operator may be used to automatically select a subject based on whom the operator is facing, looking at, or indicating (e.g., pointing at).
- The user interface may be updated in real-time, but, in some implementations, the order and presentation of the subjects may lock to avoid UI elements moving/changing as an operator attempts to interact with them. For example, a refresh button may be provided in the UI to allow an operator to explicitly indicate when the presentation of the subjects should be refreshed. In some variations, a UI indicator could be presented to signify when the system believes the UI is “out of date” and should be refreshed.
- In one alternative variation, a user interface presents an image-based representation of the scenario, which functions to use an image map as a user interface as shown in
FIG. 7 . In this variation, presenting subject indicators in the user interface can include generating an interactive image-based representation and presenting the image-based representation with selectable subject indicators at the location of the subjects represented in the image-based representation. The image-based representation is preferably an image-based representation of the position of subjects relative to the operator station. The image-based representation preferably allows visual selection of a subject. For example, a cashier can be presented with an image showing the customers waiting in line. The cashier can select a customer for checkout by selecting the customer shown within the image. Generating an interactive image-based representation may include determining position of CV detected subjects in collected image data, presenting the interactive image-based representation with interactive a subject indicator UI elements corresponding to a CV detected subject at the position of each relevant CV detected subject. - In one variation, the image-based representation could be a synthesized representation. For example, the synthesized representation could be an overhead graphical map showing the operator station and visual indication of the subjects in corresponding location as shown in
FIG. 8A . This variation functions to create a graphical map with representative indicators of subject position relative to the operator. In another variation, the image-based representation could be a still or live video stream from an image capture device as shown inFIG. 8B . The image capture device is preferably oriented to have a similar perspective as the operator (e.g., from behind the POS system). In this variation, the method may include collecting live video and detecting user selection at a location of CV detected person within the presented image. - From that image-based representation, the operator can select a subject. The contextual organization of subjects in one variation can be used in generating an interactive image-based representation such as by generating the synthesized representation or adding interactive elements to an image or video. In some alternative variations, the image-based representation variation may be implemented without block S120. Locating and marking of subjects in the appropriate image data may defer interpretation of the line order of the subjects to the operator, and instead the image-based representation of nearby subjects serves as a spatial map of nearby subjects. The image-based representation could be extended to augmented reality or virtual reality user interfaces. In an augmented reality variation, wherein presenting the image-based representation can include rendering the image-based representation in an augmented reality display as shown in
FIG. 8C . In this way subject indicators may be overlaid or displayed in positions associated with viewed subjects. - Block S134, which includes accessing the digital interaction state data of a subject associated with the subject indicator in response to user interaction with a subject indicator, functions to perform an action in connection with the digital interaction state data of the subject.
- Accessing the digital interaction state data may include reading and updating the user interface with content based on the digital interaction state data. For example, after selecting a subject indicator, the checkout list of the subject associated with the subject indicator may be displayed. In this variation accessing the digital interaction state data may include communicating the digital interaction state data to the computing device of the operator station. The data will generally be stored in a remote data storage system and managed by the monitoring system.
- In the method variation used with an automated checkout, accessing the checkout list of the subject further can include presenting a representation of the checkout list within the user interface. In this variation accessing the digital interaction state data may include communicating the digital interaction state data to the computing device of the operator station. The data will generally be stored in a remote data storage system and managed by the monitoring system
- Reading of interactive state data may not be used solely for presenting information but also performing some actions such as completing a checkout transaction. In some variations, accessing the checkout list of the subject can include executing a checkout process for items of the checkout list, which functions to enter the items of the checkout list for POS processing, receiving payment, and finalizing the checkout process.
- In a related variation, accessing the checkout list of the subject further can include, if a cart issue is associated with the checkout list, presenting a guided resolution interaction flow within the user interface to resolve the cart issue prior to executing the checkout process. Cart issues may include conditions with the checkout list related to an unidentified item, an unknown number of items, produce or by-weight items requiring weighing, restricted items needing ID or other forms of verification, and/or other issues. The resolution interaction flows preferably indicate the cart issue and supplies a user interface for resolving—typically by receiving operator input.
- For example, produce items will commonly be sold by weight and so for each produce item, an item exception may be flagged for the produce item in the checkout list of a subject. The operator can be guided through a process of pairing produce identifiers, such as Price Look Up (PLU) codes, with measured item weight. In some implementations and instances, the CV monitoring system may automatically supply the PLU code when detecting the item being weighed during the resolution workflow. In other instances, the operator may enter the PLU code. By collecting the weight at the operator station and the PLU (through automated entry by the monitoring system or manual entry) the item exception can be resolved and the item correctly updated in the checkout list.
- In another example, the quantity of an item may not be accurately detected by the monitoring system, and as such a cart issue will be flagged in association with that item of the checkout list. During checkout processing of the checkout list, a resolution interaction flow can trigger: at the user interface displaying an item count resolution view, receiving item count and updating the count of the associated item in the checkout list. The item count resolution view may display a request like “How many of this item did the customer select?” and then the operator can visually count and enter the correct number. The item in the checkout list can be appropriately updated with the verified item count.
- A more detailed description of the checkout process performed at the computing device can include: presenting the subject indicators in an arrangement based on line order, receiving operator selection of a subject indicator, accessing the checkout list of a subject associated with the subject indicator, optionally displaying the contents of the content list, if a cart issue is associated with the checkout list, presenting a guided resolution interaction flow within the user interface to resolve the cart issue, and receiving confirmation to proceed with checkout once a checkout list is valid, entering the items into a POS processing system, determining a checkout total, receiving payment, and finalizing the checkout process. In some variations, the method may only involve supplying the item information of the checkout list to an external POS processing system that finalizes the checkout process. Alternatively, the method may include performing each step of checkout processing.
- Accessing the digital interaction state data may additionally or alternatively include updating the subject indicator state data, which functions to write or edit the subject indicator state data. For example, issues and problems with a checkout list may be manually resolved by an operator and resolution input data can be used in updating the checkout list. In this variation, accessing the checkout list of the subject can include adding or updating at least one item to the checkout list of the select subject based on received user interaction at the operator station.
- In one particular scenario, the method may be used to add items to a checkout list of a customer. This can have particular utility at service counters of a grocery store like at the deli, fish, butcher, and/or bakery counter where service counter items are individually prepared and priced. In this scenario, when an operator prepares an item and wants to add it to the checkout list of the customer, the method can include, presenting the subject indicators in an arrangement based on most likely subjects being served by the operator, receiving operator selection of a subject indicator, entering the item information (e.g., price, item description, etc.), receiving operator confirmation to add to the checkout list, communicating the information to a computer-implemented system managing the checkout list, and adding the item information to the checkout list of the selected customer as shown in
FIG. 5 . This variation of the method serves to make the process of an operator updating digital information of a customer easier since they can more easily access their information. This checkout list may later be processed in any suitable manner. - The method and the variations described in any suitable combination and applied to the applications described herein as well as other applications that could benefit from a dynamic user interface used to facilitate linking presence of real users with associated virtual data.
- As described one preferred implementation of the method includes: detecting digital interaction state of a plurality of subjects in an environment using at least one sensor-based monitoring system; detecting a contextual organization of subjects relative to an operator station; at the operator station, augmenting the user interface based on the contextual organization of subjects which comprises of at least: presenting a set of subject indicators in the user interface with the subject indicators arranged in response to contextual organization, and in response to received user interaction with at least one selected subject indicator, accessing the digital interaction state of the subject associated the at least one subject indicator. Below are some detailed descriptions of some preferred variations. However, such variations may be combined in any suitable manner and used in combination with any suitable variation described herein.
- In a virtual cart related variation of the above implementation, detecting digital interaction state may include tracking a set of subjects through the environment and, for each subject, detecting item interaction events including at least item selection events and updating items in a checkout list based on the item interaction event; and wherein accessing the digital interaction state of the subject comprises accessing the checkout list of the subject an associated with the at least one subject indicator.
- In a checkout variation of the virtual cart related variation, accessing the checkout list of the subject can include executing a checkout process for items of the checkout list. This may include transmitting the checkout list to a checkout list processor (e.g., a POS processing system). This may alternatively include performing the checkout process and executing a transaction on behalf of the checkout list.
- In the checkout variation above, accessing the checkout list of the subject may include presenting a representation of the checkout list within the user interface. This can include identifying and communicating the checkout list to the computer device of the operator device. Presentation of the checkout list may happen after selection of the associated subject in the user interface.
- In the checkout variation above, accessing the checkout list of the subject further comprises, if a cart issue is associated with the checkout list, presenting a guided resolution interaction flow within the user interface to resolve the cart issue prior to executing the checkout process. The cart issue could be detection of a by-weight item (e.g., a produce item), and the guided resolution interaction flow can be used in collecting item weight and pairing with product identifier (e.g., a PLU code) and updating the checkout list. The cart issue could include detection of an unknown item count, and the guided resolution interaction flow can be used in collecting an item count for one or more item and updating the checkout list. The cart issue could alternatively include detection of an unknown item type, and the guided resolution interaction flow can be used in specifying a description of the item and requesting product identification. For example, it may announce an unknown cereal box (optionally with additionally identifying characteristics) and the operator can resolve the issue by scanning the cereal box or asking the customer for the product identity. The cart issue could include a restricted item alert, and the guided resolution interaction flow could instruct the operator to verify ID for a list of restricted items, and once ID is confirmed, checkout can proceed. Other cart issue types could similarly be used.
- In the checkout variation above, detecting contextual organization of subjects may include detecting social grouping of subjects and associating multiple subjects with a single checkout list. Grouping may be performed using various numbers of factors from the sensor monitoring system or from other data sources. Some variations may enable options for manual control of merging within the user interface.
- In the virtual cart variation above, accessing the checkout list of the subject can include adding at least one item to the checkout list of the subject based on received user interaction at the operator station. This can be used to add service counter items, but could additionally be used in other situations. As more general variation, accessing the digital interaction data can include modifying the digital interaction data based on received user interaction at the operator station.
- In the virtual cart variation above, detecting item interaction events can involve the use of a CV monitoring system and can include collecting image data, applying computer vision processing of the image data and detecting item interaction events in part from the computer vision processing. The CV monitoring system can additionally be used in collecting image data of the region in proximity to the operator station and detecting a contextual organization of subjects relative to an operator station can include analyzing the image data of the region in proximity to the operator station using CV processing and at least partially determining contextual organization. In one implementation this may include supplying the image data in a convolutional neural network or other suitable ML, neural network, deep learning, statistical, heuristical, or other type of model that outputs indication of appropriate subjects and preferably prioritizes the identified subjects by predicted line order. This may similarly be applied in the method more generally.
- In the virtual cart variation above, detecting item interaction events can include detecting user-item interactions using computer vision processing of image data and a smart shelf event data. This is one example of combining multiple types of sensor-based monitoring systems. Any suitable combination of CV monitoring smart carts, RFID tracking, smart shelves, smart infrastructure and/or other monitoring systems may be used.
- In variation of the method above, wherein detecting contextual organization comprises detecting line order by analyzing orientation and position of subjects relative to the operator station.
- In the line order variation above, detecting line order further may include detecting a direction of attention of one of the operator station or an operator.
- In the line order variation above, presenting a set of subject indicators in the user interface with the subject indicators arranged in response to contextual organization can include ordering the subject indicators in an order corresponding to the line order.
- In a variation of the method above, presenting a set of subject indicators in the user interface can include generating an interactive image-based representation of the position of subjects relative to the operator station and presenting the image-based representation with selectable subject indicators at the location of the subjects represented in the image-based representation. In a related variation, presenting the image-based representation comprises rendering the image-based representation in an augmented reality display. The image-based representation may alternatively be a synthesized graphical representation or an annotated image captured of the operator station.
- Other suitable variations and variation combinations may additionally be implemented, and the method is not limited to only this variation and implementations.
- The systems and methods of the embodiments can be embodied and/or implemented at least in part as a machine configured to receive a computer-readable medium storing computer-readable instructions. The instructions can be executed by computer-executable components integrated with the application, applet, host, server, network, website, communication service, communication interface, hardware/firmware/software elements of a user computer or mobile device, wristband, smartphone, or any suitable combination thereof. Other systems and methods of the embodiment can be embodied and/or implemented at least in part as a machine configured to receive a computer-readable medium storing computer-readable instructions. The instructions can be executed by computer-executable components integrated with apparatuses and networks of the type described above. The computer-readable medium can be stored on any suitable computer readable media such as RAMs, ROMs, flash memory, EEPROMs, optical devices (CD or DVD), hard drives, floppy drives, or any suitable device. The computer-executable component can be a processor but any suitable dedicated hardware device can (alternatively or additionally) execute the instructions.
- In one variation, a system comprising of one or more computer-readable mediums storing instructions that, when executed by the one or more computer processors, cause a computing platform to perform operations comprising those of the system or method described herein such as: detecting user-associated data of a plurality of subjects in an environment; detecting a contextual organization of subjects relative to an operator station; and at a computing device of the operator station, augmenting a user interface based on the contextual organization of subjects, which includes presenting subject indicators in the user interface with the subject indicators arranged in response to contextual organization, and, in response to user interaction with a subject indicator, accessing the user-associated data of a subject associated with the subject indicator.
-
FIG. 12 is an exemplary computer architecture diagram of one implementation of the system. In some implementations, the system is implemented in a plurality of devices in communication over a communication channel and/or network. In some implementations, the elements of the system are implemented in separate computing devices. In some implementations, two or more of the system elements are implemented in same devices. The system and portions of the system may be integrated into a computing device or system that can serve as or within the system. - The
communication channel 1001 interfaces with theprocessors 1002A-1002N, the memory (e.g., a random access memory (RAM)) 1003, a read only memory (ROM) 1004, a processor-readable storage medium 1005, a display device 1006, a user input device 1007, and a network device 1008. As shown, the computer infrastructure may be used in connecting sensing system 1101,CV monitoring system 1102,imaging devices 1103, User input device with a subject management application 1104,POS processing systems 1105, and/or other suitable computing devices. - The
processors 1002A-1002N may take many forms, such CPUs (Central Processing Units), GPUs (Graphical Processing Units), microprocessors, ML/DL (Machine Learning/Deep Learning) processing units such as a Tensor Processing Unit, FPGA (Field Programmable Gate Arrays, custom processors, and/or any suitable type of processor. - The
processors 1002A-1002N and the main memory 1003 (or some sub-combination) can form aprocessing unit 1010. In some embodiments, the processing unit includes one or more processors communicatively coupled to one or more of a RAM, ROM, and machine-readable storage medium; the one or more processors of the processing unit receive instructions stored by the one or more of a RAM, ROM, and machine-readable storage medium via a bus; and the one or more processors execute the received instructions. In some embodiments, the processing unit is an ASIC (Application-Specific Integrated Circuit). In some embodiments, the processing unit is a SoC (System-on-Chip). In some embodiments, the processing unit includes one or more of the elements of the system. - A network device 1008 may provide one or more wired or wireless interfaces for exchanging data and commands between the system and/or other devices, such as devices of external systems. Such wired and wireless interfaces include, for example, a universal serial bus (USB) interface, Bluetooth interface, Wi-Fi interface, Ethernet interface, near field communication (NFC) interface, and the like.
- Computer and/or Machine-readable executable instructions comprising of configuration for software programs (such as an operating system, application programs, and device drivers) can be stored in the
memory 1003 from the processor-readable storage medium 1005, theROM 1004 or any other data storage system. - When executed by one or more computer processors, the respective machine-executable instructions may be accessed by at least one of
processors 1002A-1002N (of a processing unit 1010) via thecommunication channel 1001, and then executed by at least one of processors 1001A-1001N. Data, databases, data records or other stored forms data created or used by the software programs can also be stored in thememory 1003, and such data is accessed by at least one ofprocessors 1002A-1002N during execution of the machine-executable instructions of the software programs. - The processor-
readable storage medium 1005 is one of (or a combination of two or more of) a hard drive, a flash drive, a DVD, a CD, an optical disk, a floppy disk, a flash storage, a solid state drive, a ROM, an EEPROM, an electronic circuit, a semiconductor memory device, and the like. The processor-readable storage medium 1005 can include an operating system, software programs, device drivers, and/or other suitable sub-systems or software. - As used herein, first, second, third, etc. are used to characterize and distinguish various elements, components, regions, layers and/or sections. These elements, components, regions, layers and/or sections should not be limited by these terms. Use of numerical terms may be used to distinguish one element, component, region, layer and/or section from another element, component, region, layer and/or section. Use of such numerical terms does not imply a sequence or order unless clearly indicated by the context. Such numerical references may be used interchangeable without departing from the teaching of the embodiments and variations herein.
- As a person skilled in the art will recognize from the previous detailed description and from the figures and claims, modifications and changes can be made to the embodiments of the invention without departing from the scope of this invention as defined in the following claims.
Claims (20)
1. A method comprising:
detecting digital interaction state of a plurality of subjects in an environment using at least one sensor-based monitoring system;
detecting a contextual organization of subjects relative to an operator station;
at the operator station, augmenting the user interface based on the contextual organization of subjects which comprises of at least:
presenting a set of subject indicators in the user interface with the subject indicators arranged in response to contextual organization, and
in response to received user interaction with at least one selected subject indicator, accessing the digital interaction state of the subject associated the at least one subject indicator.
2. The method of claim 1 , wherein detecting digital interaction state comprises of: tracking a set of subjects through the environment and, for each subject, detecting item interaction events including at least item selection events and updating items in a checkout list based on the item interaction event; and wherein accessing the digital interaction state of the subject comprises accessing the checkout list of the subject an associated with the at least one subject indicator.
3. The method of claim 2 , wherein accessing the checkout list of the subject comprises executing a checkout process for items of the checkout list.
4. The method of claim 3 , wherein accessing the checkout list of the subject further comprises presenting a representation of the checkout list within the user interface.
5. The method of claim 3 , wherein accessing the checkout list of the subject further comprises, if a cart issue is associated with the checkout list, presenting a guided resolution interaction flow within the user interface to resolve the cart issue prior to executing the checkout process.
6. The method of claim 3 , wherein detecting contextual organization of subjects may include detecting social grouping of subjects and associating multiple subjects with a single checkout list.
7. The method of claim 2 , wherein accessing the checkout list of the subject comprises adding at least one item to the checkout list of the subject based on received user interaction at the operator station.
8. The method of claim 2 , wherein detecting item interaction events comprises of collecting image data, applying computer vision processing of the image data and detecting item interaction events in part from the computer vision processing.
9. The method of claim 2 , wherein detecting item interaction events comprises of detecting user-item interactions using computer vision processing of image data and a smart shelf event data.
10. The method of claim 2 , wherein detecting contextual organization comprises detecting line order by analyzing orientation and position of subjects relative to the operator station.
11. The method of claim 10 , wherein detecting line order further comprises detecting a direction of attention of one of the operator station or an operator.
12. The method of claim 10 , wherein presenting a set of subject indicators in the user interface with the subject indicators arranged in response to contextual organization comprises ordering the subject indicators in an order corresponding to the line order.
13. The method of claim 1 , wherein presenting a set of subject indicators in the user interface comprises generating an interactive image-based representation of the position of subjects relative to the operator station and presenting the image-based representation with selectable subject indicators at the location of the subjects represented in the image-based representation.
14. The method of claim 13 , wherein presenting the image-based representation comprises rendering the image-based representation in an augmented reality display.
15. A system comprising:
a sensor-based monitoring system comprising at least a computer vision monitoring system with a set of imaging devices;
a computing device at an operator station, the computing device operating a subject management user interface; and
one or more computer-readable mediums in communication with the sensor-based monitoring system and computing device, the one or more computer readable mediums storing instructions that, when executed by the one or more computer processors, cause the computer processors to:
detect user-associated data of a plurality of subjects in an environment,
detect a contextual organization of subjects relative to the operator station, and
at the computing device, augment the subject management user interface based on the contextual organization of subjects, which includes presenting subject indicators in the user interface with the subject indicators arranged in response to contextual organization, and, in response to user interaction with a subject indicator, accessing the user-associated data of a subject associated with the subject indicator.
16. The system of claim 1 , wherein instructions to detect user-associated data comprise of: tracking a set of subjects through the environment and, for each subject, detecting item interaction events including at least item selection events and updating items in a checkout list based on the item interaction event; and wherein accessing the user-associated data of the subject comprises accessing the checkout list of the subject an associated with the at least one subject indicator.
17. The system of claim 16 , wherein detecting item interaction events comprises of collecting image data, applying computer vision processing of the image data and detecting item interaction events in part from the computer vision processing.
18. The system of claim 16 , wherein detecting item interaction events comprises of detecting user-item interactions using computer vision processing of image data and a smart shelf event data.
19. The system of claim 16 , wherein instructions to detect contextual organization comprise detecting line order by analyzing orientation and position of subjects relative to the operator station.
20. The system of claim 16 , wherein accessing the checkout list of the subject comprises executing a checkout process for items of the checkout list.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/969,714 US20230038289A1 (en) | 2019-03-01 | 2022-10-20 | Cashier interface for linking customers to virtual data |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201962812265P | 2019-03-01 | 2019-03-01 | |
US16/806,946 US11507933B2 (en) | 2019-03-01 | 2020-03-02 | Cashier interface for linking customers to virtual data |
US17/969,714 US20230038289A1 (en) | 2019-03-01 | 2022-10-20 | Cashier interface for linking customers to virtual data |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/806,946 Continuation US11507933B2 (en) | 2019-03-01 | 2020-03-02 | Cashier interface for linking customers to virtual data |
Publications (1)
Publication Number | Publication Date |
---|---|
US20230038289A1 true US20230038289A1 (en) | 2023-02-09 |
Family
ID=72236712
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/806,946 Active 2040-07-09 US11507933B2 (en) | 2019-03-01 | 2020-03-02 | Cashier interface for linking customers to virtual data |
US17/969,714 Pending US20230038289A1 (en) | 2019-03-01 | 2022-10-20 | Cashier interface for linking customers to virtual data |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/806,946 Active 2040-07-09 US11507933B2 (en) | 2019-03-01 | 2020-03-02 | Cashier interface for linking customers to virtual data |
Country Status (4)
Country | Link |
---|---|
US (2) | US11507933B2 (en) |
AU (1) | AU2020231365A1 (en) |
CA (1) | CA3131604A1 (en) |
WO (1) | WO2020180815A1 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11727479B2 (en) | 2016-05-09 | 2023-08-15 | Grabango Co. | Computer vision system and method for automatic checkout |
US20230351318A1 (en) * | 2020-02-20 | 2023-11-02 | Boe Technology Group Co., Ltd. | Commodity change information generation method and apparatus, commodity sorting method and apapratus, and electronic device |
Families Citing this family (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110462669B (en) | 2017-02-10 | 2023-08-11 | 格拉班谷公司 | Dynamic customer checkout experience within an automated shopping environment |
WO2018209156A1 (en) | 2017-05-10 | 2018-11-15 | Grabango Co. | Series-configured camera array for efficient deployment |
US20190005479A1 (en) * | 2017-06-21 | 2019-01-03 | William Glaser | Interfacing with a point of sale system from a computer vision system |
US20190079591A1 (en) | 2017-09-14 | 2019-03-14 | Grabango Co. | System and method for human gesture processing from video input |
US10963704B2 (en) | 2017-10-16 | 2021-03-30 | Grabango Co. | Multiple-factor verification for vision-based systems |
US11288648B2 (en) | 2018-10-29 | 2022-03-29 | Grabango Co. | Commerce automation for a fueling station |
JP7391513B2 (en) * | 2019-01-17 | 2023-12-05 | 東芝テック株式会社 | Product registration device and information processing program |
US10997414B2 (en) * | 2019-03-29 | 2021-05-04 | Toshiba Global Commerce Solutions Holdings Corporation | Methods and systems providing actions related to recognized objects in video data to administrators of a retail information processing system and related articles of manufacture |
US11580785B1 (en) | 2019-06-10 | 2023-02-14 | Amazon Technologies, Inc. | Detecting interactions with non-discretized items and associating interactions with actors using digital images |
US11315074B2 (en) * | 2019-10-06 | 2022-04-26 | AiFi Corp | Smart shelf system |
FR3103955A1 (en) * | 2019-11-29 | 2021-06-04 | Orange | Device and method for environmental analysis, and device and voice assistance method implementing them |
JP7456764B2 (en) * | 2019-12-10 | 2024-03-27 | 東芝テック株式会社 | store system |
US11379676B2 (en) * | 2020-04-01 | 2022-07-05 | Nec Corporation | RFID-based self-checkout systems using cycle counting |
EP3937071A1 (en) * | 2020-07-06 | 2022-01-12 | Bull SAS | Method for assisting the real-time tracking of at least one person on image sequences |
CN114338577B (en) * | 2020-10-12 | 2023-05-23 | 腾讯科技(深圳)有限公司 | Information processing method and device, electronic equipment and storage medium |
US20220344019A1 (en) * | 2021-04-21 | 2022-10-27 | Britta Kay Carlson | Media and medical device ecommerce system and methods |
US20230133569A1 (en) * | 2021-10-28 | 2023-05-04 | Ncr Corporation | Self-checkout offer processing |
US20230177777A1 (en) * | 2021-12-08 | 2023-06-08 | Bank Of America Corporation | Systems and methods for enhanced augmented reality emulation based on historical data |
US11855831B1 (en) | 2022-06-10 | 2023-12-26 | T-Mobile Usa, Inc. | Enabling an operator to resolve an issue associated with a 5G wireless telecommunication network using AR glasses |
Family Cites Families (223)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US3009863A (en) | 1957-04-24 | 1961-11-21 | Aluminum Co Of America | Methods for thermally processing carbon articles |
US5254853A (en) | 1990-02-14 | 1993-10-19 | Stefan Reich | Optical sensing device |
US5418567A (en) | 1993-01-29 | 1995-05-23 | Bayport Controls, Inc. | Surveillance camera system |
US5502564A (en) | 1994-09-13 | 1996-03-26 | Hughes Aircraft Company | Substrate thickness measurement using oblique incidence multispectral interferometry |
US6837436B2 (en) | 1996-09-05 | 2005-01-04 | Symbol Technologies, Inc. | Consumer interactive shopping system |
US6697103B1 (en) | 1998-03-19 | 2004-02-24 | Dennis Sunga Fernandez | Integrated network for monitoring remote objects |
US6970183B1 (en) | 2000-06-14 | 2005-11-29 | E-Watch, Inc. | Multimedia surveillance and monitoring system including network configuration |
US8005314B2 (en) | 2005-12-09 | 2011-08-23 | Amnis Corporation | Extended depth of field imaging for high speed object analysis |
US7107110B2 (en) | 2001-03-05 | 2006-09-12 | Microsoft Corporation | Audio buffers with audio effects |
US20040201754A1 (en) | 2001-10-12 | 2004-10-14 | Mcalister Micheal J. | Dual camera mounting arrangement for a wide screen imaging system |
US8321302B2 (en) | 2002-01-23 | 2012-11-27 | Sensormatic Electronics, LLC | Inventory management system |
US7274803B1 (en) | 2002-04-02 | 2007-09-25 | Videomining Corporation | Method and system for detecting conscious hand movement patterns and computer-generated visual feedback for facilitating human-computer interaction |
US7921036B1 (en) | 2002-04-30 | 2011-04-05 | Videomining Corporation | Method and system for dynamically targeting content based on automatic demographics and behavior analysis |
US20030210340A1 (en) | 2002-05-07 | 2003-11-13 | John Frederick Romanowich | Camera with a mechanical and electrical interface for couplable engagement with a standard lampholder |
US7227976B1 (en) | 2002-07-08 | 2007-06-05 | Videomining Corporation | Method and system for real-time facial image enhancement |
US8351647B2 (en) | 2002-07-29 | 2013-01-08 | Videomining Corporation | Automatic detection and aggregation of demographics and behavior of people |
US7053915B1 (en) | 2002-07-30 | 2006-05-30 | Advanced Interfaces, Inc | Method and system for enhancing virtual stage experience |
US8010402B1 (en) | 2002-08-12 | 2011-08-30 | Videomining Corporation | Method for augmenting transaction data with visually extracted demographics of people using computer vision |
US7225414B1 (en) | 2002-09-10 | 2007-05-29 | Videomining Corporation | Method and system for virtual touch entertainment |
US9711153B2 (en) | 2002-09-27 | 2017-07-18 | The Nielsen Company (Us), Llc | Activating functions in processing devices using encoded audio and detecting audio signatures |
US6996460B1 (en) | 2002-10-03 | 2006-02-07 | Advanced Interfaces, Inc. | Method and apparatus for providing virtual touch interaction in the drive-thru |
US7912246B1 (en) | 2002-10-28 | 2011-03-22 | Videomining Corporation | Method and system for determining the age category of people based on facial images |
US7283650B1 (en) | 2002-11-27 | 2007-10-16 | Video Mining Corporation | Method and system for printing of automatically captured facial images augmented with promotional content |
US7734070B1 (en) | 2002-12-31 | 2010-06-08 | Rajeev Sharma | Method and system for immersing face images into a video sequence |
US20040260513A1 (en) | 2003-02-26 | 2004-12-23 | Fitzpatrick Kerien W. | Real-time prediction and management of food product demand |
US7711155B1 (en) | 2003-04-14 | 2010-05-04 | Videomining Corporation | Method and system for enhancing three dimensional face modeling using demographic classification |
US7590261B1 (en) | 2003-07-31 | 2009-09-15 | Videomining Corporation | Method and system for event detection by analysis of linear feature occlusion |
US7505621B1 (en) | 2003-10-24 | 2009-03-17 | Videomining Corporation | Demographic classification using image components |
US20050096997A1 (en) | 2003-10-31 | 2005-05-05 | Vivek Jain | Targeting shoppers in an online shopping environment |
US7319779B1 (en) | 2003-12-08 | 2008-01-15 | Videomining Corporation | Classification of humans into multiple age categories from digital images |
US7100824B2 (en) | 2004-02-27 | 2006-09-05 | Evolution Robotics, Inc. | System and methods for merchandise checkout |
US7168618B2 (en) | 2004-08-12 | 2007-01-30 | International Business Machines Corporation | Retail store method and system |
US20060036485A1 (en) | 2004-08-13 | 2006-02-16 | International Business Machines Corporation | Methods and apparatus for presenting personalized information to consumers in a retail environment |
US7646887B2 (en) | 2005-01-04 | 2010-01-12 | Evolution Robotics Retail, Inc. | Optical flow for object recognition |
US9609287B2 (en) | 2005-03-02 | 2017-03-28 | En-Gauge, Inc. | Remote monitoring |
US10645347B2 (en) | 2013-08-09 | 2020-05-05 | Icn Acquisition, Llc | System, method and apparatus for remote monitoring |
US9036028B2 (en) | 2005-09-02 | 2015-05-19 | Sensormatic Electronics, LLC | Object tracking and alerts |
US20070091177A1 (en) | 2005-10-24 | 2007-04-26 | The Regents Of The University Of California | Remote unattended camera and computer integrated security system |
US20070126871A1 (en) | 2005-12-06 | 2007-06-07 | Henninger Paul E Iii | Modular surveillance camera system with self-identification capability |
US7911482B1 (en) | 2006-01-06 | 2011-03-22 | Videomining Corporation | Method and system for efficient annotation of object trajectories in image sequences |
JP4125328B2 (en) | 2006-04-17 | 2008-07-30 | キヤノン株式会社 | Electronic device, control method of peripheral device by electronic device, program, and storage medium |
US7681790B2 (en) | 2006-06-09 | 2010-03-23 | The Container Store | Computer supported retail shopping systems and methods |
US7930204B1 (en) | 2006-07-25 | 2011-04-19 | Videomining Corporation | Method and system for narrowcasting based on automatic analysis of customer behavior in a retail store |
US7974869B1 (en) | 2006-09-20 | 2011-07-05 | Videomining Corporation | Method and system for automatically measuring and forecasting the behavioral characterization of customers to help customize programming contents in a media network |
US7987111B1 (en) | 2006-10-30 | 2011-07-26 | Videomining Corporation | Method and system for characterizing physical retail spaces by determining the demographic composition of people in the physical retail spaces utilizing video image analysis |
US8380558B1 (en) | 2006-12-21 | 2013-02-19 | Videomining Corporation | Method and system for analyzing shopping behavior in a store by associating RFID data with video-based behavior and segmentation data |
US8189926B2 (en) | 2006-12-30 | 2012-05-29 | Videomining Corporation | Method and system for automatically analyzing categories in a physical space based on the visual characterization of people |
US8146811B2 (en) | 2007-03-12 | 2012-04-03 | Stoplift, Inc. | Cart inspection for suspicious items |
US8295597B1 (en) | 2007-03-14 | 2012-10-23 | Videomining Corporation | Method and system for segmenting people in a physical space based on automatic behavior analysis |
US7957565B1 (en) | 2007-04-05 | 2011-06-07 | Videomining Corporation | Method and system for recognizing employees in a physical space based on automatic behavior analysis |
EP2153641B2 (en) | 2007-04-13 | 2021-10-27 | Ari M. Presler | Digital cinema camera system for recording, editing and visualizing images |
US7848548B1 (en) | 2007-06-11 | 2010-12-07 | Videomining Corporation | Method and system for robust demographic classification using pose independent model from sequence of face images |
US7909248B1 (en) | 2007-08-17 | 2011-03-22 | Evolution Robotics Retail, Inc. | Self checkout with visual recognition |
US8520906B1 (en) | 2007-09-24 | 2013-08-27 | Videomining Corporation | Method and system for age estimation based on relative ages of pairwise facial images of people |
US10296936B1 (en) | 2007-09-26 | 2019-05-21 | Videomining Corporation | Method and system for measuring effectiveness of a marketing campaign on digital signage |
US20090158309A1 (en) | 2007-12-12 | 2009-06-18 | Hankyu Moon | Method and system for media audience measurement and spatial extrapolation based on site, display, crowd, and viewership characterization |
JP5781307B2 (en) | 2007-12-26 | 2015-09-16 | ジョンソン コントロールズ テクノロジー カンパニーJohnson Controls Technology Company | System and method for performing commercial transactions in a vehicle |
US8098888B1 (en) | 2008-01-28 | 2012-01-17 | Videomining Corporation | Method and system for automatic analysis of the trip of people in a retail space using multiple cameras |
EP2283309A1 (en) | 2008-02-04 | 2011-02-16 | FPS Food Processing Systems B.V. | Vision system with software control for detecting dirt and other imperfections on egg surfaces |
US20130215116A1 (en) | 2008-03-21 | 2013-08-22 | Dressbot, Inc. | System and Method for Collaborative Shopping, Business and Entertainment |
US8027521B1 (en) | 2008-03-25 | 2011-09-27 | Videomining Corporation | Method and system for robust human gender recognition using facial feature localization |
US8433612B1 (en) | 2008-03-27 | 2013-04-30 | Videomining Corporation | Method and system for measuring packaging effectiveness using video-based analysis of in-store shopper response |
US8219438B1 (en) | 2008-06-30 | 2012-07-10 | Videomining Corporation | Method and system for measuring shopper response to products based on behavior and facial expression |
DE202008009834U1 (en) | 2008-07-22 | 2009-12-10 | Liebherr-Elektronik Gmbh | Camera for use in a digital network |
US8448859B2 (en) | 2008-09-05 | 2013-05-28 | Datalogic ADC, Inc. | System and method for preventing cashier and customer fraud at retail checkout |
US8379937B1 (en) | 2008-09-29 | 2013-02-19 | Videomining Corporation | Method and system for robust human ethnicity recognition using image feature-based probabilistic graphical models |
US8577705B1 (en) | 2008-12-30 | 2013-11-05 | Videomining Corporation | Method and system for rating the role of a product category in the performance of a store area |
US8175388B1 (en) | 2009-01-30 | 2012-05-08 | Adobe Systems Incorporated | Recognizing text at multiple orientations |
EP2401865B1 (en) | 2009-02-27 | 2020-07-15 | Foundation Productions, Llc | Headset-based telecommunications platform |
US8892469B2 (en) | 2009-04-01 | 2014-11-18 | Igt | Gaming device security mechanism |
US9747497B1 (en) | 2009-04-21 | 2017-08-29 | Videomining Corporation | Method and system for rating in-store media elements |
US8254633B1 (en) | 2009-04-21 | 2012-08-28 | Videomining Corporation | Method and system for finding correspondence between face camera views and behavior camera views |
US8638211B2 (en) | 2009-04-30 | 2014-01-28 | Icontrol Networks, Inc. | Configurable controller and interface for home SMA, phone and multimedia |
WO2010135575A2 (en) | 2009-05-20 | 2010-11-25 | Express Imaging Systems, Llc | Long-range motion detection for illumination control |
US9740977B1 (en) | 2009-05-29 | 2017-08-22 | Videomining Corporation | Method and system for recognizing the intentions of shoppers in retail aisles based on their trajectories |
US8812344B1 (en) | 2009-06-29 | 2014-08-19 | Videomining Corporation | Method and system for determining the impact of crowding on retail performance |
US8325982B1 (en) | 2009-07-23 | 2012-12-04 | Videomining Corporation | Method and system for detecting and tracking shopping carts from videos |
US8412656B1 (en) | 2009-08-13 | 2013-04-02 | Videomining Corporation | Method and system for building a consumer decision tree in a hierarchical decision tree structure based on in-store behavior analysis |
JP5257335B2 (en) | 2009-11-24 | 2013-08-07 | オムロン株式会社 | Method for displaying measurement effective area in three-dimensional visual sensor and three-dimensional visual sensor |
CA2784321C (en) | 2009-12-13 | 2018-01-16 | AisleBuyer LLC | Systems and methods for purchasing products from a retail establishment using a mobile device |
US8803888B2 (en) | 2010-06-02 | 2014-08-12 | Microsoft Corporation | Recognition system for sharing information |
JP5548042B2 (en) | 2010-06-23 | 2014-07-16 | ソフトバンクモバイル株式会社 | User terminal device and shopping system |
WO2012003538A1 (en) | 2010-07-06 | 2012-01-12 | James Shaun O'leary | A system for electronic transactions |
JP2012028015A (en) | 2010-07-20 | 2012-02-09 | Toshiba Corp | Illumination control system and illumination control method |
US8488881B2 (en) | 2010-07-27 | 2013-07-16 | International Business Machines Corporation | Object segmentation at a self-checkout |
US8615254B2 (en) | 2010-08-18 | 2013-12-24 | Nearbuy Systems, Inc. | Target localization utilizing wireless and camera sensor fusion |
US8652033B2 (en) | 2010-09-23 | 2014-02-18 | Karl Storz Endovision, Inc. | Video stylet with directable tip |
US20120173351A1 (en) | 2010-12-29 | 2012-07-05 | Qthru, Llc | Mobile Electronic Shopping |
US10474858B2 (en) | 2011-08-30 | 2019-11-12 | Digimarc Corporation | Methods of identifying barcoded items by evaluating multiple identification hypotheses, based on data from sensors including inventory sensors and ceiling-mounted cameras |
US9367770B2 (en) | 2011-08-30 | 2016-06-14 | Digimarc Corporation | Methods and arrangements for identifying objects |
US9129277B2 (en) | 2011-08-30 | 2015-09-08 | Digimarc Corporation | Methods and arrangements for identifying objects |
US20130103537A1 (en) | 2011-10-14 | 2013-04-25 | Spencer Thornton Hewett | Interactive consumer self-service shopping system |
US20130284806A1 (en) | 2011-10-19 | 2013-10-31 | Ran Margalit | Automated purchasing system |
US10326921B2 (en) | 2011-11-14 | 2019-06-18 | Tseng-Lu Chien | Light device has built-in camera and related digital data device's functions |
AU2013207407A1 (en) * | 2012-01-05 | 2013-10-24 | Visa International Service Association | Transaction visual capturing apparatuses, methods and systems |
US8606645B1 (en) * | 2012-02-02 | 2013-12-10 | SeeMore Interactive, Inc. | Method, medium, and system for an augmented reality retail application |
WO2013134865A1 (en) | 2012-03-16 | 2013-09-19 | In Situ Media Corporation | System and method for scanning, tracking and collating customer shopping selections |
US9661047B2 (en) | 2012-04-30 | 2017-05-23 | Mobilatv Ltd. | Method and system for central utilization of remotely generated large media data streams despite network bandwidth limitations |
US9892438B1 (en) | 2012-05-03 | 2018-02-13 | Stoplift, Inc. | Notification system and methods for use in retail environments |
US9378747B2 (en) | 2012-05-07 | 2016-06-28 | Dolby International Ab | Method and apparatus for layout and format independent 3D audio reproduction |
US9147114B2 (en) | 2012-06-19 | 2015-09-29 | Honeywell International Inc. | Vision based target tracking for constrained environments |
US9774459B2 (en) | 2012-06-22 | 2017-09-26 | Sensormatic Electronics, LLC | Security system and method using wireless adapters and PoE cameras |
US9288299B2 (en) | 2012-09-14 | 2016-03-15 | Dewmobile, Inc. | Method and apparatus for file sharing in a network |
US20140082610A1 (en) | 2012-09-14 | 2014-03-20 | DewMobile Inc. | Mesh network and mesh network node application |
US9647906B2 (en) | 2012-11-02 | 2017-05-09 | Rockwell Automation Technologies, Inc. | Cloud based drive monitoring solution |
US20140155022A1 (en) | 2012-12-05 | 2014-06-05 | Anil Kandregula | Methods and apparatus to monitor usage of mobile devices |
US20140188601A1 (en) | 2013-01-03 | 2014-07-03 | International Business Machines Corporation | Integration between brick-and-mortar establishments and web based establishments |
US10062066B2 (en) | 2013-01-30 | 2018-08-28 | Walmart Apollo, Llc | Systems and methods for retrieving items for a customer at checkout |
US20140214564A1 (en) | 2013-01-31 | 2014-07-31 | Wal-Mart Stores, Inc. | Method to allow two or more mobile devices to contribute items to the same virtual shopping cart in mobile self checkout software |
US20140245160A1 (en) | 2013-02-22 | 2014-08-28 | Ubiquiti Networks, Inc. | Mobile application for monitoring and controlling devices |
US10198080B1 (en) | 2013-03-11 | 2019-02-05 | Amazon Technologies, Inc. | Virtual user interface |
US9392675B2 (en) | 2013-03-14 | 2016-07-12 | Lutron Electronics Co., Inc. | Digital load control system providing power and communication via existing power wiring |
US9330413B2 (en) | 2013-03-14 | 2016-05-03 | Sears Brands, L.L.C. | Checkout and/or ordering systems and methods |
US20140272855A1 (en) | 2013-03-15 | 2014-09-18 | FitStar, Inc. | Selecting available exercises for customizing an exercise session |
US8915429B2 (en) | 2013-03-15 | 2014-12-23 | Ncr Corporation | Methods of mobile self-checkout shopping for produce items in a retail grocery store |
US9098825B2 (en) | 2013-03-26 | 2015-08-04 | Leonard Bashkin | Storage container with inventory control |
US9213803B2 (en) | 2013-05-02 | 2015-12-15 | David Rolley | System and method for collecting, analyzing and reporting fitness activity data |
US9412099B1 (en) | 2013-05-09 | 2016-08-09 | Ca, Inc. | Automated item recognition for retail checkout systems |
US20140365334A1 (en) | 2013-06-07 | 2014-12-11 | Bby Solutions, Inc. | Retail customer service interaction system and method |
US20140363059A1 (en) | 2013-06-07 | 2014-12-11 | Bby Solutions, Inc. | Retail customer service interaction system and method |
US9329682B2 (en) | 2013-06-18 | 2016-05-03 | Microsoft Technology Licensing, Llc | Multi-step virtual object selection |
US10176456B2 (en) | 2013-06-26 | 2019-01-08 | Amazon Technologies, Inc. | Transitioning items from a materials handling facility |
US10268983B2 (en) | 2013-06-26 | 2019-04-23 | Amazon Technologies, Inc. | Detecting item interaction and movement |
US10217133B2 (en) * | 2013-07-19 | 2019-02-26 | Paypal, Inc. | Reverse showrooming and merchant-customer engagement system |
US10290031B2 (en) | 2013-07-24 | 2019-05-14 | Gregorio Reid | Method and system for automated retail checkout using context recognition |
US9473747B2 (en) | 2013-07-25 | 2016-10-18 | Ncr Corporation | Whole store scanner |
US20150046213A1 (en) | 2013-08-07 | 2015-02-12 | Motorola Solutions, Inc. | Method for identifying a shopping group in a store |
JP5920299B2 (en) | 2013-09-17 | 2016-05-18 | コニカミノルタ株式会社 | Image forming apparatus, remote control method, and remote control program |
JP6361103B2 (en) | 2013-09-17 | 2018-07-25 | 株式会社リコー | Management device, management system, object management method, and program |
US9141931B2 (en) | 2013-09-26 | 2015-09-22 | Amazon Technologies, Inc. | Inventory distribution method and system |
US9984352B2 (en) | 2013-09-26 | 2018-05-29 | Amazon Technologies, Inc. | Kiosk network in a kiosk management system |
US9474934B1 (en) | 2013-10-11 | 2016-10-25 | Fit Intuition, LLC | Biometric assessment in fitness improvement |
US10366445B2 (en) | 2013-10-17 | 2019-07-30 | Mashgin Inc. | Automated object recognition kiosk for retail checkouts |
US9183620B2 (en) | 2013-11-21 | 2015-11-10 | International Business Machines Corporation | Automated tilt and shift optimization |
US9772612B2 (en) | 2013-12-11 | 2017-09-26 | Echostar Technologies International Corporation | Home monitoring and control |
CN106062842B (en) | 2014-01-30 | 2019-11-05 | 飞利浦灯具控股公司 | Lighting system and controller and mobile subscriber terminal for controlling it |
US20150227890A1 (en) | 2014-02-07 | 2015-08-13 | Kristin Kaye Bednarek | Communications system and smart device apps supporting segmented order distributed distribution system |
US20150310601A1 (en) | 2014-03-07 | 2015-10-29 | Digimarc Corporation | Methods and arrangements for identifying objects |
US9948902B1 (en) | 2014-03-07 | 2018-04-17 | Alarm.Com Incorporated | Video camera and sensor integration |
US9120621B1 (en) | 2014-03-25 | 2015-09-01 | Amazon Technologies, Inc. | Verifying bin content in an automated materials handling facility |
US9317785B1 (en) | 2014-04-21 | 2016-04-19 | Video Mining Corporation | Method and system for determining ethnicity category of facial images based on multi-level primary and auxiliary classifiers |
US9918213B2 (en) | 2014-05-07 | 2018-03-13 | DewMobile Inc. | Discovery and networking of proximate wireless devices by acoustic messaging |
US20150373509A1 (en) | 2014-06-19 | 2015-12-24 | Dewmobile, Inc. | Group gaming platform for mobile devices in proximity |
JP5866522B2 (en) | 2014-07-08 | 2016-02-17 | パナソニックIpマネジメント株式会社 | Facility management support apparatus, facility management support system, and facility management support method |
US10129507B2 (en) | 2014-07-15 | 2018-11-13 | Toshiba Global Commerce Solutions Holdings Corporation | System and method for self-checkout using product images |
US9547964B2 (en) | 2014-07-24 | 2017-01-17 | Heathco, Llc | Integrated home lighting and notification system |
EP3009997B1 (en) | 2014-10-15 | 2016-11-23 | Axis AB | Arrangement for a monitoring camera device |
US20160110791A1 (en) | 2014-10-15 | 2016-04-21 | Toshiba Global Commerce Solutions Holdings Corporation | Method, computer program product, and system for providing a sensor-based environment |
US10157379B2 (en) | 2014-11-12 | 2018-12-18 | Toshiba Global Commerce Solutions Holdings Corporation | Self checkout with security checks based on categorized items |
US9262681B1 (en) | 2014-11-19 | 2016-02-16 | Amazon Technologies, Inc. | Probabilistic registration of interactions, actions or activities from multiple views |
US10594983B2 (en) | 2014-12-10 | 2020-03-17 | Robert Bosch Gmbh | Integrated camera awareness and wireless sensor system |
US10134004B1 (en) | 2014-12-23 | 2018-11-20 | Amazon Technologies, Inc. | Processing image data from a camera cluster |
ITUB20159725A1 (en) | 2014-12-30 | 2017-06-30 | Egoview S R L | METHOD AND BIOMECHANICAL ANALYSIS SYSTEM OF POSTURE OF A CYCLIST AND CUSTOMIZED AUTOMATIC REALIZATION OF THE COMPONENTS OF A BICYCLE |
US9990587B2 (en) | 2015-01-22 | 2018-06-05 | Preferred Networks, Inc. | Machine learning heterogeneous edge device, method, and system |
US9412034B1 (en) | 2015-01-29 | 2016-08-09 | Qualcomm Incorporated | Occlusion handling for computer vision |
US9706619B2 (en) | 2015-02-16 | 2017-07-11 | Cree, Inc. | Lighting fixture with image sensor |
US9853730B2 (en) | 2015-02-27 | 2017-12-26 | Xicato, Inc. | Lighting based authentication of a mobile electronic device |
US9270634B1 (en) | 2015-03-02 | 2016-02-23 | Dewmobile, Inc. | Building a proximate social networking database based on relative distance profiling of two or more operably coupled computers |
CA3183763A1 (en) | 2015-03-13 | 2016-09-22 | Lutron Electronics Co., Inc. | Control device having an illuminated portion controlled in response to an external sensor |
US9250712B1 (en) | 2015-03-20 | 2016-02-02 | Hand Held Products, Inc. | Method and application for scanning a barcode with a smart device while continuously running and displaying an application on the smart device display |
US10089520B2 (en) | 2015-03-26 | 2018-10-02 | Krishna V Motukuri | System for displaying the contents of a refrigerator |
US20160289964A1 (en) | 2015-04-02 | 2016-10-06 | Shawn Engberg | Suspended ceiling |
BR112017021258A2 (en) | 2015-04-03 | 2018-06-26 | Lucis Technologies Holdings Limited | Environmental Control System |
US10217120B1 (en) | 2015-04-21 | 2019-02-26 | Videomining Corporation | Method and system for in-store shopper behavior analysis with multi-modal sensor fusion |
US10565455B2 (en) | 2015-04-30 | 2020-02-18 | Ants Technology (Hk) Limited | Methods and systems for audiovisual communication |
US10037592B2 (en) | 2015-06-05 | 2018-07-31 | Mindaptiv LLC | Digital quaternion logarithm signal processing system and method for images and other data types |
CN107924522B (en) | 2015-06-24 | 2022-06-03 | 奇跃公司 | Augmented reality device, system and method for purchasing |
US9911290B1 (en) | 2015-07-25 | 2018-03-06 | Gary M. Zalewski | Wireless coded communication (WCC) devices for tracking retail interactions with goods and association to user accounts |
US10318976B2 (en) | 2015-07-28 | 2019-06-11 | Walmart Apollo, Llc | Methods for determining measurement data of an item |
US20190054347A1 (en) | 2015-08-18 | 2019-02-21 | Michael Saigh | Wearable sports guidance communication system and developers tool kit |
US10180469B2 (en) | 2015-10-28 | 2019-01-15 | Siemens Healthcare Gmbh | Gesture-controlled MR imaging system and method |
CN105335509B (en) | 2015-10-29 | 2017-06-30 | 广州神马移动信息科技有限公司 | A kind of method for recommending action message, device and server |
CA3006550A1 (en) | 2015-12-03 | 2017-06-08 | Walmart Apollo, Llc | Smart cart for self-checkout of retail merchandise |
US10915910B2 (en) | 2015-12-09 | 2021-02-09 | International Business Machines Corporation | Passive analysis of shopping behavior in a physical shopping area using shopping carts and shopping trays |
EP3391339A2 (en) | 2015-12-18 | 2018-10-24 | Iris Automation, Inc. | Real-time visual situational awareness system |
US10262331B1 (en) | 2016-01-29 | 2019-04-16 | Videomining Corporation | Cross-channel in-store shopper behavior analysis |
WO2017136151A1 (en) | 2016-02-02 | 2017-08-10 | Gaming Grids Wearables, Llc | Esports fitness and training system |
US10832311B2 (en) | 2016-02-26 | 2020-11-10 | Imagr Limited | Method and medium for shopping in a physical store |
WO2017149875A1 (en) | 2016-02-29 | 2017-09-08 | ソニー株式会社 | Image capture control device, image capture device, and image capture control method |
KR20170110824A (en) | 2016-03-24 | 2017-10-12 | 삼성전자주식회사 | Display apparatus and the control method thereof |
US10198625B1 (en) | 2016-03-26 | 2019-02-05 | Videomining Corporation | Association of unique person to a mobile device using repeat face image matching |
US10387896B1 (en) | 2016-04-27 | 2019-08-20 | Videomining Corporation | At-shelf brand strength tracking and decision analytics |
US10319198B2 (en) | 2016-05-02 | 2019-06-11 | Focal Systems, Inc. | Expedited checkout system through portable checkout units |
CN114040153B (en) | 2016-05-09 | 2024-04-12 | 格拉班谷公司 | System for computer vision driven applications within an environment |
IL294633B1 (en) | 2016-05-31 | 2024-05-01 | Theia Group Incorporated | System for transmission and digitization of machine telemetry |
US10354262B1 (en) | 2016-06-02 | 2019-07-16 | Videomining Corporation | Brand-switching analysis using longitudinal tracking of at-shelf shopper behavior |
US10380814B1 (en) | 2016-06-27 | 2019-08-13 | Amazon Technologies, Inc. | System for determining entry of user to an automated facility |
US10659247B2 (en) | 2016-07-09 | 2020-05-19 | Grabango Co. | Computer vision for ambient data acquisition |
US10083358B1 (en) | 2016-07-26 | 2018-09-25 | Videomining Corporation | Association of unique person to point-of-sale transaction data |
BR102016019151A2 (en) | 2016-08-18 | 2018-03-06 | De Souza Lopes Roberto | METHOD AND COMPUTER VEHICLE DISPLAY AND IDENTIFICATION SYSTEM THROUGH CAMERAS APPLIED IN FUEL STATIONS AND CONVENIENCE STORES |
US10409548B2 (en) | 2016-09-27 | 2019-09-10 | Grabango Co. | System and method for differentially locating and modifying audio sources |
US10002344B2 (en) | 2016-10-17 | 2018-06-19 | Conduent Business Services, Llc | System and method for retail store promotional price tag detection |
WO2018125712A1 (en) | 2016-12-30 | 2018-07-05 | Datalogic Usa, Inc. | Self-checkout with three dimensional scanning |
US10665051B2 (en) | 2017-02-02 | 2020-05-26 | Smartbins Inc. | Bulk food integrated scale system |
CN110462669B (en) | 2017-02-10 | 2023-08-11 | 格拉班谷公司 | Dynamic customer checkout experience within an automated shopping environment |
US20180240180A1 (en) | 2017-02-20 | 2018-08-23 | Grabango Co. | Contextually aware customer item entry for autonomous shopping applications |
US10458595B2 (en) | 2017-02-28 | 2019-10-29 | Cisco Technology, Inc. | Smart Power over Ethernet lift |
US10997421B2 (en) | 2017-03-30 | 2021-05-04 | Hrl Laboratories, Llc | Neuromorphic system for real-time visual activity recognition |
WO2018209156A1 (en) | 2017-05-10 | 2018-11-15 | Grabango Co. | Series-configured camera array for efficient deployment |
US20190005479A1 (en) * | 2017-06-21 | 2019-01-03 | William Glaser | Interfacing with a point of sale system from a computer vision system |
US10474988B2 (en) | 2017-08-07 | 2019-11-12 | Standard Cognition, Corp. | Predicting inventory events using foreground/background processing |
US10055853B1 (en) | 2017-08-07 | 2018-08-21 | Standard Cognition, Corp | Subject identification and tracking using image recognition |
US10133933B1 (en) | 2017-08-07 | 2018-11-20 | Standard Cognition, Corp | Item put and take detection using image recognition |
US10445694B2 (en) | 2017-08-07 | 2019-10-15 | Standard Cognition, Corp. | Realtime inventory tracking using deep learning |
US10650545B2 (en) | 2017-08-07 | 2020-05-12 | Standard Cognition, Corp. | Systems and methods to check-in shoppers in a cashier-less store |
WO2019032304A1 (en) | 2017-08-07 | 2019-02-14 | Standard Cognition Corp. | Subject identification and tracking using image recognition |
US10127438B1 (en) | 2017-08-07 | 2018-11-13 | Standard Cognition, Corp | Predicting inventory events using semantic diffing |
US11023850B2 (en) | 2017-08-07 | 2021-06-01 | Standard Cognition, Corp. | Realtime inventory location management using deep learning |
US10474991B2 (en) | 2017-08-07 | 2019-11-12 | Standard Cognition, Corp. | Deep learning-based store realograms |
US20190079591A1 (en) | 2017-09-14 | 2019-03-14 | Grabango Co. | System and method for human gesture processing from video input |
US10469768B2 (en) | 2017-10-13 | 2019-11-05 | Fyusion, Inc. | Skeleton-based effects and background replacement |
US10963704B2 (en) | 2017-10-16 | 2021-03-30 | Grabango Co. | Multiple-factor verification for vision-based systems |
US11481805B2 (en) | 2018-01-03 | 2022-10-25 | Grabango Co. | Marketing and couponing in a retail environment using computer vision |
US20190244161A1 (en) | 2018-02-02 | 2019-08-08 | Microsoft Technology Licensing, Llc | Inventory control |
US20190333039A1 (en) | 2018-04-27 | 2019-10-31 | Grabango Co. | Produce and bulk good management within an automated shopping environment |
US10282852B1 (en) * | 2018-07-16 | 2019-05-07 | Accel Robotics Corporation | Autonomous store tracking system |
US10282720B1 (en) | 2018-07-16 | 2019-05-07 | Accel Robotics Corporation | Camera-based authorization extension system |
US11087539B2 (en) * | 2018-08-21 | 2021-08-10 | Mastercard International Incorporated | Systems and methods for generating augmented reality-based profiles |
EP3847076A4 (en) | 2018-09-07 | 2022-06-08 | Gatekeeper Systems, Inc. | Shopping basket monitoring using computer vision and machine learning |
US11288648B2 (en) | 2018-10-29 | 2022-03-29 | Grabango Co. | Commerce automation for a fueling station |
US10347009B1 (en) | 2018-11-21 | 2019-07-09 | Juan Ramon Terven | Self callbrating camera system |
US10943204B2 (en) * | 2019-01-16 | 2021-03-09 | International Business Machines Corporation | Realtime video monitoring applied to reduce customer wait times |
US20200265494A1 (en) | 2019-02-17 | 2020-08-20 | Grabango Co. | Remote sku on-boarding of products for subsequent video identification and sale |
-
2020
- 2020-03-02 CA CA3131604A patent/CA3131604A1/en active Pending
- 2020-03-02 AU AU2020231365A patent/AU2020231365A1/en active Pending
- 2020-03-02 WO PCT/US2020/020698 patent/WO2020180815A1/en active Application Filing
- 2020-03-02 US US16/806,946 patent/US11507933B2/en active Active
-
2022
- 2022-10-20 US US17/969,714 patent/US20230038289A1/en active Pending
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11727479B2 (en) | 2016-05-09 | 2023-08-15 | Grabango Co. | Computer vision system and method for automatic checkout |
US20230351318A1 (en) * | 2020-02-20 | 2023-11-02 | Boe Technology Group Co., Ltd. | Commodity change information generation method and apparatus, commodity sorting method and apapratus, and electronic device |
US12026662B2 (en) * | 2020-02-20 | 2024-07-02 | Boe Technology Group Co., Ltd. | Commodity change information generation method and apparatus, commodity sorting method and apapratus, and electronic device |
Also Published As
Publication number | Publication date |
---|---|
US20200279240A1 (en) | 2020-09-03 |
WO2020180815A1 (en) | 2020-09-10 |
AU2020231365A1 (en) | 2021-09-16 |
CA3131604A1 (en) | 2020-09-10 |
US11507933B2 (en) | 2022-11-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11507933B2 (en) | Cashier interface for linking customers to virtual data | |
US20240144340A1 (en) | Remote SKU On-Boarding of Products for Subsequent Video Identification and Sale | |
US11847689B2 (en) | Dynamic customer checkout experience within an automated shopping environment | |
US20230017398A1 (en) | Contextually aware customer item entry for autonomous shopping applications | |
US11443598B2 (en) | Selective treatment of shopping receptacles in checkout | |
US20220374965A1 (en) | Associating digital activities with positions in physical retail stores | |
US11810067B2 (en) | Digitally managed shelf space marketplace | |
JP6498065B2 (en) | Information processing apparatus, processing method, and program | |
WO2022081518A2 (en) | Methods and systems for retail environments | |
US20240070608A1 (en) | Resolving misplaced items in physical retail stores | |
US20220044262A1 (en) | System and Method for Collecting Real-World Data Using a Location-Based Computing Device in Fulfillment of Observation Campaign Opportunities |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
AS | Assignment |
Owner name: GLICKBERG, DANIEL, NEW YORK Free format text: SECURITY INTEREST;ASSIGNOR:GRABANGO CO.;REEL/FRAME:067611/0150 Effective date: 20240528 |
|
AS | Assignment |
Owner name: FIRST-CITIZEN BANK & TRUST COMPANY, CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:GRABANGO CO.;REEL/FRAME:067637/0231 Effective date: 20240524 |