WO2020134100A1 - Control method, device and system for vending machine, and vending machine - Google Patents

Control method, device and system for vending machine, and vending machine Download PDF

Info

Publication number
WO2020134100A1
WO2020134100A1 PCT/CN2019/099758 CN2019099758W WO2020134100A1 WO 2020134100 A1 WO2020134100 A1 WO 2020134100A1 CN 2019099758 W CN2019099758 W CN 2019099758W WO 2020134100 A1 WO2020134100 A1 WO 2020134100A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
vending machine
risk
user
goods
Prior art date
Application number
PCT/CN2019/099758
Other languages
French (fr)
Chinese (zh)
Inventor
刘朋樟
刘巍
陈宇
Original Assignee
北京沃东天骏信息技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京沃东天骏信息技术有限公司 filed Critical 北京沃东天骏信息技术有限公司
Publication of WO2020134100A1 publication Critical patent/WO2020134100A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06K—GRAPHICAL DATA READING; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
    • G06K17/00—Methods or arrangements for effecting co-operative working between equipments covered by two or more of main groups G06K1/00 - G06K15/00, e.g. automatic card files incorporating conveying and reading operations
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07F—COIN-FREED OR LIKE APPARATUS
    • G07F11/00—Coin-freed apparatus for dispensing, or the like, discrete articles
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07F—COIN-FREED OR LIKE APPARATUS
    • G07F9/00—Details other than those peculiar to special kinds or types of apparatus
    • G07F9/02—Devices for alarm or indication, e.g. when empty; Advertising arrangements in coin-freed apparatus

Definitions

  • the present disclosure relates to the field of artificial intelligence technology, and particularly to a vending machine control method, device, system, and vending machine.
  • Unmanned vending machines are an effective expansion and extension of the online retail business to the offline market. Compared with unmanned shelves, unmanned vending machines use a closed space, which can reduce the risk of theft.
  • unmanned vending machines Similar to unmanned shelves, how to effectively prevent the theft of unmanned vending machine products and prevent harmful physical health products from being placed in unmanned vending machines is still a problem that needs to be solved. Since unmanned vending machines mainly sell food and daily necessities, if the above actions are not controlled, it will not only easily cause economic losses, but also may threaten the personal safety of the users of the goods.
  • a vending machine control method which includes: when the vending machine is open: collecting images within the effective range of the vending machine; performing a risk analysis on the collected images; if the collected images are determined If it is abnormal, it is determined that the collected image is a high-risk image; the high-risk image is reported to the vending machine control server in order to identify the user's risk behavior.
  • reporting the high-risk images to the vending machine control server includes: filtering the abnormal images to select some abnormal images; and reporting the selected abnormal images to the vending machine control server.
  • the image abnormality includes one or more of the following: according to image analysis, it is determined that one or more image acquisition devices are blocked; according to image analysis, it is determined that an abnormal object appears in the vending machine; the user performs abnormal operations on the goods, Abnormal operations include one or more of destroying the outer packaging of the goods, adding substances to the inside of the goods, taking substances from the goods or leaving the camera detection area with the goods and returning them; or, the size, color or transparency of the goods have changed.
  • reporting the high-risk image to the vending machine control server includes: judging whether there is a high-risk image if the time after the door of the sales office or keeping the door open state exceeds a predetermined first time; if there is a high-risk image , Then report high-risk images.
  • the vending machine control method further includes: determining the change of the goods based on the collected images; determining the change of the goods based on the gravity detection; and performing the determination and collection when the changes of the goods determined based on the images and the gravity detection are inconsistent
  • the image is a high-risk image operation.
  • the vending machine control method further includes: in the case where it is determined that the goods are taken from the vending machine based on the image or gravity detection, recording the unique identification of the taken out goods, wherein each item in the vending machine has a unique identification ; In the case of determining whether the goods are returned to the vending machine based on the image or gravity detection, determine whether the identification of the returned goods matches the identification of the removed goods; if they do not match, perform the operation of determining that the collected image is a high-risk image , If it matches, delete the recorded identification of the goods.
  • the vending machine control method further includes: receiving a door opening request from the user; if the user account is a restricted user, refusing to trigger the vending machine to open the door.
  • performing risk analysis on the collected images includes: performing risk analysis on the collected images based on local image features and image analysis models of the vending machine.
  • the vending machine control method further includes: when the vending machine control server receives the high-risk image, identify the high-risk image; if it is determined that the user performs the illegal operation by identifying the high-risk image, modify the user's illegal operation statistics Value; mark the user as a restricted user when the user’s violation operation statistical value reaches a predetermined threshold.
  • the vending machine control method further includes: if the vending machine control server determines that the user has not performed the illegal operation based on the high-risk image, updating the image analysis training data set using the high-risk image; analyzing the training data set based on the updated image Update the image features and image analysis models, and send them to the vending machine to update the image features and image analysis models of the vending machine.
  • a vending machine control device including: a high-risk image determination unit configured to perform a risk analysis on the collected images within the effective range of the vending machine when the vending machine is open If it is determined that the collected image is abnormal, the collected image is determined to be a high-risk image; the reporting unit is configured to report the high-risk image to the vending machine control server in order to identify user risk behavior.
  • the reporting unit is further configured to screen for abnormal images and select some abnormal images; report the selected abnormal images to the vending machine control server.
  • the image abnormality includes one or more of the following: according to image analysis, it is determined that one or more image acquisition devices are blocked; according to image analysis, it is determined that an abnormal object appears in the vending machine; the user performs abnormal operations on the goods, Abnormal operations include one or more of destroying the outer packaging of the goods, adding substances to the inside of the goods, taking substances from the goods or leaving the camera detection area with the goods and returning them; or, the size, color or transparency of the goods have changed.
  • the reporting unit is further configured to: determine whether there is a high-risk image if the time after the door of the sales office or keep the door open state exceeds a predetermined first time; if there is a high-risk image, report the high Risk image.
  • the vending machine control device further includes: an image processing unit configured to determine the change in goods based on the collected images; a gravity processing unit configured to determine the change in goods based on gravity detection; the high-risk image determination unit also It is configured to perform an operation of determining that the collected image is a high-risk image when the cargo change determined based on the image and the gravity detection is inconsistent.
  • the high-risk image determination unit is further configured to: in the case where it is determined based on the image or gravity detection that the goods are taken out of the vending machine, record the unique identification of the taken out goods, wherein each goods in the vending machine With a unique identification; in the case where the goods are returned to the vending machine based on the image or gravity detection, determine whether the identification of the returned goods matches the identification of the removed goods; if they do not match, the determination of the collected image is high risk If the operation of the image matches, the recorded cargo identification is deleted.
  • the vending machine control device further includes: a door opening control unit configured to: receive a door opening request from a user; if the user account is a restricted user, refuse to trigger the door opening of the vending machine.
  • a door opening control unit configured to: receive a door opening request from a user; if the user account is a restricted user, refuse to trigger the door opening of the vending machine.
  • the high-risk image determination unit is further configured to: perform risk analysis on the collected images based on the local image features and image analysis models of the vending machine; update the local area based on the image features and image analysis models from the vending machine control server Image features and image analysis models.
  • a vending machine control device including: a memory; and a processor coupled to the memory, the processor configured to execute any one of the above based on instructions stored in the memory Vending machine control method performed by a vending machine.
  • a computer-readable storage medium is provided on which computer program instructions are stored, which when executed by a processor implements the steps of any of the above vending machine control methods.
  • a vending machine control system including: any of the above vending machine control devices; and a vending machine control server configured to: in the case of receiving a high-risk image Next, identify high-risk images.
  • the vending machine is configured to perform risk analysis on the collected image based on the local image features and image analysis model of the vending machine; the vending machine control server is also configured to: if it is determined that the user has not performed the illegal operation based on the high-risk image , Use high-risk images to update the image analysis training data set; update the image features and image analysis model according to the updated image analysis training data set, and send them to the vending machine.
  • the vending machine control server is further configured to: if it is determined that the user performs a violation operation by identifying a high-risk image, modify the user violation operation statistical value; when the user violation operation statistical value reaches a predetermined threshold, the The user is marked as restricted user.
  • a vending machine including: a storage device configured to carry goods; an image acquisition device configured to collect images of the cargo area; and a gravity detection device configured In order to detect the change of the weight of the goods carried by the storage device; and, any one of the above vending machine control devices.
  • FIG. 1 is a flowchart of some embodiments of a vending machine control method of the present disclosure.
  • FIG. 2 is a flowchart of some other embodiments of the vending machine control method of the present disclosure.
  • FIG. 3 is a flowchart of still other embodiments of the vending machine control method of the present disclosure.
  • FIG. 4 is a flowchart of still other embodiments of the vending machine control method of the present disclosure.
  • FIG. 5 is a schematic diagram of some embodiments of a vending machine control device of the present disclosure.
  • FIG. 6 is a schematic diagram of other embodiments of the vending machine control device of the present disclosure.
  • FIG. 7 is a schematic diagram of some embodiments of a vending machine control system of the present disclosure.
  • FIG. 8 is a schematic diagram of some embodiments of the vending machine of the present disclosure.
  • FIG. 1 A flowchart of some embodiments of the vending machine control method of the present disclosure is shown in FIG. 1. Start the following operations when the vending machine is open:
  • step 101 the vending machine opens the door under the user's trigger.
  • the user can trigger the vending machine to open the door by scanning the QR code of the vending machine, or informing the identity of the vending machine by other means.
  • the vending machine collects images within the effective range of the vending machine through the image acquisition device.
  • the effective range may include one or more of the area directly in front of the vending machine, inside the vending machine, and the area between the door and the vending machine after the door is opened.
  • the vending machine includes more than two image acquisition devices, which can collect abnormal and non-abnormal images and cache them, and can also cache non-abnormal images at the time of abnormal images, so as to ensure the cache of high-risk images quality.
  • a five-channel camera may be used, which is located on the left, right, upper, and lower sides to detect the shelf, and to detect the front from the inside to the outside.
  • an image analysis algorithm is used to determine whether the collected image is abnormal.
  • the image abnormality can be determined using an image analysis algorithm.
  • image abnormality may refer to one or more image acquisition devices (such as cameras) being blocked. If it is determined that the collected image is abnormal, step 103 is performed.
  • the possibility of occurrence of an image abnormality may be analyzed based on the set image abnormality, and the image abnormality is determined when the probability exceeds a predetermined abnormality threshold.
  • step 103 it is determined that the currently acquired image is a high-risk image.
  • step 104 the high-risk image is reported to the vending machine control server, so that the server recognizes the user's risk behavior.
  • the image reporting operation may be performed immediately after it is determined that there is a high-risk image to improve the image reporting efficiency; in some embodiments, the door may be opened at the sales office, or the vending machine continues to open the door for more than a predetermined duration Perform the reporting process to reduce the frequency of image reporting and reduce the burden of communication and data processing.
  • FIG. 2 A flowchart of some other embodiments of the vending machine control method of the present disclosure is shown in FIG. 2.
  • step 201 the vending machine opens the door under the user's trigger.
  • step 202 image analysis is performed on the collected image, and it is determined whether the collected image is abnormal based on the image characteristics and the image analysis model stored locally at the vending machine.
  • the image abnormality may be that one or more cameras are blocked. If it is determined that the collected image is abnormal, step 203 is executed. If no abnormal image is found, step 204 is executed.
  • the image abnormality may include a variety of situations, such as: it is determined that an abnormal object has occurred in the vending machine according to the image analysis; the user performs abnormal operations on the goods, and the abnormal operations may include destroying the outer packaging of the goods and adding substances to the inside of the goods 1.
  • the abnormal operations may include destroying the outer packaging of the goods and adding substances to the inside of the goods 1.
  • One or more items taken from the goods or the user returns the goods after leaving the camera detection area with the goods; or, the size, color or transparency of the goods changes.
  • the image features and image analysis models locally stored in the vending machine may be used to perform feature matching analysis to determine whether the image is abnormal.
  • step 203 the images collected by each image collection device are cached as high-risk images, and then step 206 is executed.
  • the change in cargo is determined based on the image collected by the camera, and the change in cargo is determined based on gravity detection.
  • the only goods to be operated can be identified based on the image algorithm based on the collected image, and whether to put back or take out the operation can be identified based on the trajectory of the image.
  • gravity detection it is judged whether the goods are put back or taken out based on the weight increase or decrease.
  • the weight or the combination of goods that may be operated is determined based on the weight change in grams, and the weight reduction is determined, and which one or which goods are taken out is determined based on the weight of various pre-stored goods.
  • step 205 it is determined whether the cargo changes based on the image and the gravity detection are consistent.
  • the action determined according to the image trajectory is inconsistent with the action determined by the increase or decrease of gravity, or the operated product identified by the image does not coincide with the change in gram of gravity, it may indicate that the operation has occurred product occlusion or abnormality.
  • High-risk behaviors such as storage; when the image recognition and gravity recognition results are consistent, it is a normal shopping process. The shopping images are not cached, and the goods are added to or deleted from the shopping cart.
  • step 206 If the changes are consistent, go to step 206; if the changes are not consistent, go to step 203, and collect the image at the current moment as a high-risk image cache.
  • step 206 it is determined whether the vending machine closes the door, or whether the time for which the door remains open despite opening the door exceeds a predetermined first time. If the door of the sales office is triggered, or the opening time exceeds the predetermined first time, step 207 is executed.
  • step 207 may be performed only when the user closes the door to settle. In other embodiments, step 207 may be performed when the vending machine is idle.
  • step 207 it is determined whether high-risk images are cached in the current cache area. If high-risk images are cached, step 208 is executed.
  • image frames are filtered from the cached high-risk images.
  • the filtering operation may include selecting data frames with clear images that are not completely occluded, and retaining only one frame or a few frames for very similar data.
  • step 209 the filtered high-risk image frames are reported to the vending machine control server, and the high-risk images cached in the cache area are deleted.
  • the unmanned vending machine based on video images is more powerful than the unmanned vending machine based on pure gravity and RFID (Radio Frequency Identification), because of its wider selection space and better user experience. More and more attention.
  • the gravity-based unmanned vending machine has the pain point that it cannot monitor the user's shopping process.
  • the image-based unmanned vending machine needs to overcome the high-risk behaviors such as blocking the camera, placing dangerous goods, and covering the theft of goods with both hands. Will cause higher risks and higher cargo damage.
  • the related unmanned vending machine risk control mainly includes two conventional solutions: video image and gravity:
  • the risk control solution based on video images cannot provide effective real-time early warning. Only when the unmanned vending machine operation and maintenance personnel finds that the merchandise in the vending machine is abnormal (there are abnormal items) or the product is lost during verification, it can detect the risk. All videos need to be played back manually. In addition, the above solution first needs to save all the videos and occupy a lot of storage space of the unmanned vending machine. If the video is returned to the cloud through the network, it will occupy a lot of network bandwidth and the operation and maintenance cost will increase dramatically. At the same time, manually tracing video will increase labor costs and reduce work efficiency. The behavior recognition algorithm based on pure vision is not accurate enough at present.
  • the gravity sensor can monitor the weight of the goods taken or returned at any time, so as to realize the deduction.
  • the unmanned vending machine cannot promptly warn in time, resulting in high cargo damage and high risk.
  • there is no image monitoring function when it is found that there are abnormal goods in the unmanned vending machine, it cannot be traced back, and it cannot be located when and who has stolen the goods or placed dangerous goods.
  • abnormal behavior can be discovered based on the results of image analysis, and abnormal image reporting can be triggered in time when an abnormal situation is discovered.
  • the burden of data transmission is reduced, and manual interpretation and backtracking are reduced by orders of magnitude.
  • it can also improve the efficiency of abnormal situation detection, in addition, it can also reduce the probability of missed judgment and improve the safety of sales.
  • step 301 it is determined that the goods are taken out and sold according to the image or gravity detection.
  • each cargo may be configured with a unique identification, such as an RFID identification.
  • a unique identification such as an RFID identification.
  • step 303 is triggered to be executed.
  • step 303 the identification of the returned goods is collected, and it is determined whether it matches the identification of the removed goods. If the two match, go to step 305; if the two do not match, go to step 304.
  • step 304 the current time is buffered, or the collected image of the sales process is intercepted as a high-risk image.
  • step 305 it is determined whether the user has performed an abnormal operation on the goods.
  • the abnormal operation may include one or more of destroying the outer packaging of the goods, adding substances to the inside of the goods, taking substances from the goods, or returning from the camera detection area after the user holds the goods. If it is determined that the user makes an abnormal operation, step 304 is executed. If it is determined that the user has not performed an abnormal operation, step 306 is executed.
  • step 306 it is determined that the user returns the item normally, and the logo is deleted.
  • FIG. 4 A flowchart of still other embodiments of the vending machine control method of the present disclosure is shown in FIG. 4.
  • a door opening request from the user is received.
  • the user's door opening request may include the user's personal identity information, such as the user's account information for the vending machine, or the platform related to the vending machine, or ID number information.
  • step 402 the user identity information is uploaded to the platform to determine whether the user is a restricted user. If it is determined to be a restricted user, step 403 is performed; if it is determined to be a normal user, step 404 is performed.
  • step 403 refusal to trigger the vending machine to open the door.
  • step 404 the vending machine is triggered to open the door.
  • step 405 it is determined that a high-risk image was generated during the shopping process.
  • the method shown in the embodiments shown in FIGS. 2 and 3 may be used to determine whether to generate a high-risk image.
  • the high-risk image may be cached first for subsequent reporting. If it is determined that the high-risk image is generated (cached), step 406 is executed.
  • the high-risk image is reported to the server.
  • some frames may be extracted from the generated high-risk image first, and only the extracted image frames are reported, thereby reducing the data transmission burden.
  • step 407 the server issues an alarm so that the staff can manually identify it, or use a further image recognition method to interpret the high-risk image.
  • step 408 it is determined whether the user performs a violation operation based on the high-risk image. If it is determined that the user has performed the violation operation, step 409 is performed; if it is determined that the user has not performed the violation operation, it is considered that a misjudgment has occurred, and step 412 is performed.
  • step 409 it is determined whether the statistical value of the user's illegal operation has reached a predetermined threshold. If the predetermined threshold has been reached, step 411 is performed; if the predetermined threshold has not been reached, step 410 is performed.
  • the recorded user violation operation statistical value is increased.
  • the amount of increased statistical value may also be determined according to the type of the user's current violation operation, such as a common violation operation plus 1, a malicious violation manipulation plus 10, and so on.
  • step 411 the user right is modified to restrict the user.
  • step 412 the image analysis training data set is updated with high-risk images.
  • the image features and the image analysis model are updated according to the updated image analysis training data set, and sent to the vending machine to update the local image features and the image analysis model.
  • the image features and the image analysis model are updated according to the updated image analysis training data set, and sent to the vending machine to update the local image features and the image analysis model.
  • it can only be sent to the vending machine that provides the high-risk image, while avoiding excessive vending machine data, it can also target the vending machine's location.
  • the environment and the situation experienced are dynamically updated.
  • the user's authority can be dynamically changed according to the user's illegal operation, preventing the user from further illegal operation and improving security; it can also dynamically update the data foundation of image analysis, image extraction features and models, and improve the accuracy of vending machine image analysis To reduce the probability of wrong judgment.
  • the vending machine it is possible to mine difficult examples online through an effective online self-feedback mechanism, find images that are not easily recognized accurately and return them, so that the vending machine realizes self-iteration and optimization of the model, thus quickly adapting to deployment
  • the new environment and new product image recognition requirements save a lot of image acquisition costs of training images, and the improvement of image recognition accuracy will further improve user experience and reduce cargo losses.
  • FIG. 5 A schematic diagram of some embodiments of the vending machine control device of the present disclosure is shown in FIG. 5.
  • the high-risk image determining unit 501 can determine whether the collected image is abnormal when the door of the vending machine is opened, and determine that the collected image is a high-risk image if the abnormality is determined.
  • the vending machine includes more than two image acquisition devices, which can collect abnormal and non-abnormal images and cache them, and can also cache non-abnormal images at the time of abnormal images, so as to ensure the cache of high-risk images quality.
  • the image anomaly can be determined using an image analysis model and image features based on the local vending machine, and a second time using an image analysis algorithm.
  • image abnormality may refer to one or more image acquisition devices (such as cameras) being blocked.
  • the reporting unit 502 can report the high-risk image to the vending machine control server, so that the server recognizes the user's risk behavior.
  • the reporting process may be performed when the door of the vending machine or the vending machine continues to open for more than a predetermined period of time, reducing the frequency of image reporting and reducing the communication and data processing burden.
  • Such a vending machine control device can react in a timely manner when the image collection is abnormal, and upload the collected image to the server for the server to analyze, thereby realizing timely feedback of the image, improving the efficiency of abnormal discovery, and improving the safety of the vending machine.
  • the image abnormality may also include multiple situations: for example, according to image analysis, it is determined that an abnormal object has occurred in the vending machine; the user performs abnormal operations on the goods.
  • the abnormal operations include destroying the outer packaging of the goods, adding substances into the goods, One or more items taken from the goods or the user left the camera detection area with the goods and returned to the camera; or, the size, color or transparency of the goods were changed, thereby reducing the possibility of leaking violations and further improving safety Sex.
  • the reporting unit 502 may first select a portion of the high-risk image frames from the cached high-risk images, and then report the selected high-risk image frames to the vending machine control server, and delete the cached high-risk images. In some embodiments, some frames with low pixels can be deleted, or a few frames can be selected from very similar frames.
  • Such a vending machine control device can reduce the amount of data that needs to be reported to the server, on the one hand reduce the data transmission burden, on the other hand improve the data transmission efficiency, and facilitate the server side to quickly respond to illegal operations.
  • the vending machine control device may further include a gravity processing unit 503 and an image processing unit 504.
  • the gravity processing unit 503 can determine the change of the cargo based on the gravity detection. In some embodiments, the weight reduction can be determined according to the gravity detection, and which cargo or cargos are taken out can be determined based on the weight of various pre-stored cargoes.
  • the image processing unit 504 can determine the change of goods based on the image collected by the camera. In some embodiments, it can determine which goods or goods are taken out based on the collected image.
  • the high-risk image determination unit 501 can cache the high-risk image in the case where the change of the cargo determined based on the image and the gravity detection is inconsistent, so that the reporting unit can report.
  • Such a vending machine control device can determine whether the cached high-risk image is triggered and reported by the two ways of judging whether the image is abnormal and whether the result of the image and the change of gravity detection is consistent, which reduces the probability of missed event extraction and improves the vending machine The safety of selling goods.
  • the vending machine control device may further include a door opening control unit 505, which can determine whether the user is a restricted user when receiving the user's trigger of the door opening operation. If it is determined that the user is a restricted user, it refuses to trigger the vending machine to open the door; if the user is a normal user, the door opening operation is performed.
  • a door opening control unit 505 can determine whether the user is a restricted user when receiving the user's trigger of the door opening operation. If it is determined that the user is a restricted user, it refuses to trigger the vending machine to open the door; if the user is a normal user, the door opening operation is performed.
  • Such a vending machine control device can use the user credit system to determine whether to provide services to the user, which further improves security.
  • the vending machine control device includes a memory 601 and a processor 602.
  • the memory 601 may be a magnetic disk, flash memory, or any other non-volatile storage medium.
  • the memory is used to store the instructions in the corresponding embodiments of the vending machine control method above.
  • the processor 602 is coupled to the memory 601, and may be implemented as one or more integrated circuits, such as a microprocessor or a microcontroller.
  • the processor 602 is used to execute instructions stored in the memory, which can realize timely feedback of images, improve the efficiency of abnormal discovery, and improve the safety of the vending machine.
  • a computer-readable storage medium has stored thereon computer program instructions, which when executed by a processor, implements the steps of the method in the embodiment corresponding to the vending machine control method.
  • the embodiments of the present disclosure may be provided as methods, devices, or computer program products. Therefore, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer usable non-transitory storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code .
  • FIG. 7 A schematic diagram of some embodiments of the vending machine control system of the present disclosure is shown in FIG. 7.
  • the vending machine control device 71 may be any of the vending machine control devices mentioned above.
  • the vending machine control server 70 can interpret the high-risk image when receiving the high-risk image. In some embodiments, in order to improve the timeliness of risk assessment, an alarm may be issued.
  • the vending machine control device can react in a timely manner when the image acquisition is abnormal, upload the collected image to the server for the server to analyze, in order to determine the illegal operation in time, so as to achieve timely feedback through the image, Improve the efficiency of abnormal discovery and improve the security of vending machines; it can dynamically change the user's authority according to the user's illegal operation, prevent the user from further illegal operation, and improve safety.
  • the user authority is modified to restrict the user, so that when the user uses the vending machine again, the vending machine refuses to The user opens the door, so as to realize the statistics and effective application of user credit, and improve the safety of vending machine sales.
  • the vending machine control server 70 determines that the user has not performed the illegal operation after reading the high-risk image
  • the high-risk image may be used to update the image analysis training data set, and the image characteristics may be updated according to the updated image analysis training data set.
  • the image analysis model, and sent to the vending machine the vending machine control device updates the local image characteristics and image analysis model, and realizes the optimization of its own image analysis function.
  • Such a vending machine control system can dynamically update the data basis of image analysis, image extraction features and models, improve the accuracy of vending machine image analysis, and reduce the probability of erroneous judgment.
  • FIG. 8 A schematic diagram of some embodiments of the vending machine 80 of the present disclosure is shown in FIG. 8.
  • the storage device 81 can carry the goods for sale; the gravity detection device 82 can collect the weight change on the shelf; the image collection device 83 can collect images of the inside of the vending machine and the area where the user operates.
  • the gravity detection device and the image acquisition device may include multiple.
  • the vending machine control device 84 may be any of the vending machine control devices mentioned above.
  • Such a vending machine can react in a timely manner when the image acquisition is abnormal, and upload the collected image to the server for the server to analyze, thereby realizing timely feedback of the image, improving the efficiency of abnormal discovery, and improving the security of the vending machine.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Control Of Vending Devices And Auxiliary Devices For Vending Devices (AREA)

Abstract

A control method, device (71, 84) and system for a vending machine (80), and the vending machine (80). The control method for the vending machine (80) comprises: opening the door of the vending machine (80) (101); if a collected image is abnormal (102), determining that the current collected image is a high-risk image (103); and reporting the high-risk image to a vending machine control server (70) so as to identify the risky behavior of a user (104). By means of the method, a response can be made in a timely manner when image collection is abnormal, and the collected image is uploaded to the server (70) for analysis by the server (70), so that timely feedback of the image is realized, the efficiency of abnormality discovery is improved, and the security of the vending machine (80) is improved.

Description

售货机控制方法、装置、***和售货机Vending machine control method, device, system and vending machine
相关申请的交叉引用Cross-reference of related applications
本申请是以CN申请号为201811630378.4，申请日为2018年12月29日的申请为基础，并主张其优先权，该CN申请的公开内容在此作为整体引入本申请中。This application is based on the application with the CN application number 201811630378.4 and the application date is December 29, 2018, and claims its priority. The disclosure content of the CN application is hereby incorporated into this application as a whole.
技术领域Technical field
本公开涉及人工智能技术领域，特别是一种售货机控制方法、装置、***和售货机。The present disclosure relates to the field of artificial intelligence technology, and particularly to a vending machine control method, device, system, and vending machine.
背景技术Background technique
随着零售业高速发展，越来越多的互联网企业开始探索新的零售模式，无人售货机是线上零售业务向线下市场的有效扩展与延伸。无人售货机相对于无人货架采用了封闭的空间，能够降低偷盗的风险。With the rapid development of the retail industry, more and more Internet companies have begun to explore new retail models. Unmanned vending machines are an effective expansion and extension of the online retail business to the offline market. Compared with unmanned shelves, unmanned vending machines use a closed space, which can reduce the risk of theft.
类似于无人货架，如何有效防止无人售货机商品被偷盗，防止将有害身体健康的商品放入无人售货机仍然是需要解决的问题。由于无人售卖机主要销售食品与日常生活必需品，因此上述行为如果得不到控制，不仅仅容易引起经济损失，还可能会威胁到商品使用者的人身安全。Similar to unmanned shelves, how to effectively prevent the theft of unmanned vending machine products and prevent harmful physical health products from being placed in unmanned vending machines is still a problem that needs to be solved. Since unmanned vending machines mainly sell food and daily necessities, if the above actions are not controlled, it will not only easily cause economic losses, but also may threaten the personal safety of the users of the goods.
发明内容Summary of the invention
根据本公开的一些实施例的一个方面，提出一种售货机控制方法，包括：在售货机开门状态下：采集售货机有效范围内的图像；对采集的图像进行风险分析；若确定采集的图像异常，则确定采集的图像为高风险图像；将高风险图像上报售货机控制服务器，以便识别用户风险行为。According to an aspect of some embodiments of the present disclosure, a vending machine control method is proposed, which includes: when the vending machine is open: collecting images within the effective range of the vending machine; performing a risk analysis on the collected images; if the collected images are determined If it is abnormal, it is determined that the collected image is a high-risk image; the high-risk image is reported to the vending machine control server in order to identify the user's risk behavior.
在一些实施例中，将高风险图像上报售货机控制服务器包括：对异常图像进行筛选，挑选出部分异常图像；将挑选出的异常图像上报售货机控制服务器。In some embodiments, reporting the high-risk images to the vending machine control server includes: filtering the abnormal images to select some abnormal images; and reporting the selected abnormal images to the vending machine control server.
在一些实施例中，图像异常包括以下一项或多项：根据图像分析确定一个或多个图像采集设备被遮挡；根据图像分析确定售货机内出现了异常物体；用户对货物做出异常操作，异常操作包括破坏货物外包装、向货物内部加入物质、从货物中取物质或用户持货物离开摄像头探测区域后又返回中的一项或多项；或，货物的尺寸、颜色或 透明度发生改变。In some embodiments, the image abnormality includes one or more of the following: according to image analysis, it is determined that one or more image acquisition devices are blocked; according to image analysis, it is determined that an abnormal object appears in the vending machine; the user performs abnormal operations on the goods, Abnormal operations include one or more of destroying the outer packaging of the goods, adding substances to the inside of the goods, taking substances from the goods or leaving the camera detection area with the goods and returning them; or, the size, color or transparency of the goods have changed.
在一些实施例中，将高风险图像上报售货机控制服务器包括：在售货机关门后或保持开门状态的时长超过预定第一时长的情况下，判断是否存在高风险图像；若存在高风险图像，则上报高风险图像。In some embodiments, reporting the high-risk image to the vending machine control server includes: judging whether there is a high-risk image if the time after the door of the sales office or keeping the door open state exceeds a predetermined first time; if there is a high-risk image , Then report high-risk images.
在一些实施例中，售货机控制方法还包括：基于采集的图像确定货物变化情况；基于重力探测确定货物变化情况；在基于图像和基于重力探测确定的货物变化情况不一致的情况下，执行确定采集的图像为高风险图像的操作。In some embodiments, the vending machine control method further includes: determining the change of the goods based on the collected images; determining the change of the goods based on the gravity detection; and performing the determination and collection when the changes of the goods determined based on the images and the gravity detection are inconsistent The image is a high-risk image operation.
在一些实施例中，售货机控制方法还包括：在根据图像或重力探测确定货物被取出售货机的情况下，记录被取出的货物的唯一标识，其中，售货机中的每个货物具备唯一标识；在根据图像或重力探测确定货物被放回售货机的情况下，确定放回的货物的标识与取出的货物的标识是否匹配；若不匹配，则执行确定采集的图像为高风险图像的操作，若匹配，则删除记录的货物的标识。In some embodiments, the vending machine control method further includes: in the case where it is determined that the goods are taken from the vending machine based on the image or gravity detection, recording the unique identification of the taken out goods, wherein each item in the vending machine has a unique identification ; In the case of determining whether the goods are returned to the vending machine based on the image or gravity detection, determine whether the identification of the returned goods matches the identification of the removed goods; if they do not match, perform the operation of determining that the collected image is a high-risk image , If it matches, delete the recorded identification of the goods.
在一些实施例中，售货机控制方法还包括：接收来自用户的开门请求；若用户账户为限制用户，则拒绝触发售货机开门。In some embodiments, the vending machine control method further includes: receiving a door opening request from the user; if the user account is a restricted user, refusing to trigger the vending machine to open the door.
在一些实施例中，对采集的图像进行风险分析包括：基于售货机本地的图像特征和图像分析模型对采集的图像进行风险分析。In some embodiments, performing risk analysis on the collected images includes: performing risk analysis on the collected images based on local image features and image analysis models of the vending machine.
在一些实施例中，售货机控制方法还包括：售货机控制服务器在收到高风险图像的情况下，识别高风险图像；若通过识别高风险图像确定用户执行违规操作，则修改用户违规操作统计值；在用户的违规操作统计值达到预定门限的情况下，将用户标记为限制用户。In some embodiments, the vending machine control method further includes: when the vending machine control server receives the high-risk image, identify the high-risk image; if it is determined that the user performs the illegal operation by identifying the high-risk image, modify the user's illegal operation statistics Value; mark the user as a restricted user when the user’s violation operation statistical value reaches a predetermined threshold.
在一些实施例中，售货机控制方法还包括：若售货机控制服务器根据高风险图像确定用户未执行违规操作，则利用高风险图像更新图像分析训练数据集；根据更新后的图像分析训练数据集更新图像特征和图像分析模型，并发送给售货机以便更新售货机的图像特征和图像分析模型。In some embodiments, the vending machine control method further includes: if the vending machine control server determines that the user has not performed the illegal operation based on the high-risk image, updating the image analysis training data set using the high-risk image; analyzing the training data set based on the updated image Update the image features and image analysis models, and send them to the vending machine to update the image features and image analysis models of the vending machine.
根据本公开的另一些实施例的一个方面，提出一种售货机控制装置，包括：高风险图像确定单元，被配置为在售货机开门状态下对采集的售货机有效范围内的图像进行风险分析，若确定采集的图像异常，则确定采集的图像为高风险图像；上报单元，被配置为将高风险图像上报售货机控制服务器，以便识别用户风险行为。According to an aspect of other embodiments of the present disclosure, a vending machine control device is proposed, including: a high-risk image determination unit configured to perform a risk analysis on the collected images within the effective range of the vending machine when the vending machine is open If it is determined that the collected image is abnormal, the collected image is determined to be a high-risk image; the reporting unit is configured to report the high-risk image to the vending machine control server in order to identify user risk behavior.
在一些实施例中，上报单元还被配置为对异常图像进行筛选，挑选出部分异常图像；将挑选出的异常图像上报售货机控制服务器。In some embodiments, the reporting unit is further configured to screen for abnormal images and select some abnormal images; report the selected abnormal images to the vending machine control server.
在一些实施例中，图像异常包括以下一项或多项：根据图像分析确定一个或多个图像采集设备被遮挡；根据图像分析确定售货机内出现了异常物体；用户对货物做出异常操作，异常操作包括破坏货物外包装、向货物内部加入物质、从货物中取物质或用户持货物离开摄像头探测区域后又返回中的一项或多项；或，货物的尺寸、颜色或透明度发生改变。In some embodiments, the image abnormality includes one or more of the following: according to image analysis, it is determined that one or more image acquisition devices are blocked; according to image analysis, it is determined that an abnormal object appears in the vending machine; the user performs abnormal operations on the goods, Abnormal operations include one or more of destroying the outer packaging of the goods, adding substances to the inside of the goods, taking substances from the goods or leaving the camera detection area with the goods and returning them; or, the size, color or transparency of the goods have changed.
在一些实施例中，上报单元还被配置为：在售货机关门后或保持开门状态的时长超过预定第一时长的情况下，判断是否存在高风险图像；若存在高风险图像，则上报高风险图像。In some embodiments, the reporting unit is further configured to: determine whether there is a high-risk image if the time after the door of the sales office or keep the door open state exceeds a predetermined first time; if there is a high-risk image, report the high Risk image.
在一些实施例中，售货机控制装置还包括：图像处理单元，被配置为基于采集的图像确定货物变化情况；重力处理单元，被配置为基于重力探测确定货物变化情况；高风险图像确定单元还被配置为在基于图像和基于重力探测确定的货物变化情况不一致的情况下，执行确定采集的图像为高风险图像的操作。In some embodiments, the vending machine control device further includes: an image processing unit configured to determine the change in goods based on the collected images; a gravity processing unit configured to determine the change in goods based on gravity detection; the high-risk image determination unit also It is configured to perform an operation of determining that the collected image is a high-risk image when the cargo change determined based on the image and the gravity detection is inconsistent.
在一些实施例中，高风险图像确定单元还被配置为：在根据图像或重力探测确定货物被取出售货机的情况下，记录被取出的货物的唯一标识，其中，售货机中的每个货物具备唯一标识；在根据图像或重力探测确定货物被放回售货机的情况下，确定放回的货物的标识与取出的货物的标识是否匹配；若不匹配，则执行确定采集的图像为高风险图像的操作，若匹配，则删除记录的货物的标识。In some embodiments, the high-risk image determination unit is further configured to: in the case where it is determined based on the image or gravity detection that the goods are taken out of the vending machine, record the unique identification of the taken out goods, wherein each goods in the vending machine With a unique identification; in the case where the goods are returned to the vending machine based on the image or gravity detection, determine whether the identification of the returned goods matches the identification of the removed goods; if they do not match, the determination of the collected image is high risk If the operation of the image matches, the recorded cargo identification is deleted.
在一些实施例中，售货机控制装置还包括：开门控制单元，被配置为：接收来自用户的开门请求；若用户账户为限制用户，则拒绝触发售货机开门。In some embodiments, the vending machine control device further includes: a door opening control unit configured to: receive a door opening request from a user; if the user account is a restricted user, refuse to trigger the door opening of the vending machine.
在一些实施例中，高风险图像确定单元还被配置为：基于售货机本地的图像特征和图像分析模型对采集的图像进行风险分析；根据来自售货机控制服务器的图像特征和图像分析模型更新本地的图像特征和图像分析模型。In some embodiments, the high-risk image determination unit is further configured to: perform risk analysis on the collected images based on the local image features and image analysis models of the vending machine; update the local area based on the image features and image analysis models from the vending machine control server Image features and image analysis models.
根据本公开的又一些实施例的一个方面，提出一种售货机控制装置，包括：存储器；以及耦接至存储器的处理器，处理器被配置为基于存储在存储器的指令执行如上文中任意一种由售货机执行的售货机控制方法。According to an aspect of still other embodiments of the present disclosure, a vending machine control device is proposed, including: a memory; and a processor coupled to the memory, the processor configured to execute any one of the above based on instructions stored in the memory Vending machine control method performed by a vending machine.
根据本公开的再一些实施例的一个方面，提出一种计算机可读存储介质，其上存储有计算机程序指令，该指令被处理器执行时实现上文中任意一种售货机控制方法的步骤。According to an aspect of still other embodiments of the present disclosure, a computer-readable storage medium is provided on which computer program instructions are stored, which when executed by a processor implements the steps of any of the above vending machine control methods.
根据本公开的其中一些实施例的一个方面，提出一种售货机控制***，包括：上文中任意一种售货机控制装置；和售货机控制服务器，被配置为：在收到高风险图像 的情况下，识别高风险图像。According to an aspect of some of the embodiments of the present disclosure, a vending machine control system is proposed, including: any of the above vending machine control devices; and a vending machine control server configured to: in the case of receiving a high-risk image Next, identify high-risk images.
在一些实施例中，售货机被配置为基于售货机本地的图像特征和图像分析模型对采集的图像进行风险分析；售货机控制服务器还被配置为：若根据高风险图像确定用户未执行违规操作，则利用高风险图像更新图像分析训练数据集；根据更新后的图像分析训练数据集更新图像特征和图像分析模型，并发送给售货机。In some embodiments, the vending machine is configured to perform risk analysis on the collected image based on the local image features and image analysis model of the vending machine; the vending machine control server is also configured to: if it is determined that the user has not performed the illegal operation based on the high-risk image , Use high-risk images to update the image analysis training data set; update the image features and image analysis model according to the updated image analysis training data set, and send them to the vending machine.
在一些实施例中，售货机控制服务器还被配置为：若通过识别高风险图像确定用户执行违规操作，则修改用户违规操作统计值；在用户的违规操作统计值达到预定门限的情况下，将用户标记为限制用户。In some embodiments, the vending machine control server is further configured to: if it is determined that the user performs a violation operation by identifying a high-risk image, modify the user violation operation statistical value; when the user violation operation statistical value reaches a predetermined threshold, the The user is marked as restricted user.
另外，根据本公开的一些实施例的一个方面，还提出一种售货机，包括：储物装置，被配置为承载货物；图像采集装置，被配置为采集货物区域图像；重力探测装置，被配置为探测储物装置承载的货物重量变化情况；和，上文中任意一种售货机控制装置。In addition, according to an aspect of some embodiments of the present disclosure, a vending machine is also proposed, including: a storage device configured to carry goods; an image acquisition device configured to collect images of the cargo area; and a gravity detection device configured In order to detect the change of the weight of the goods carried by the storage device; and, any one of the above vending machine control devices.
附图说明BRIEF DESCRIPTION
此处所说明的附图用来提供对本公开的进一步理解，构成本公开的一部分，本公开的示意性实施例及其说明用于解释本公开，并不构成对本公开的不当限定。在附图中：The drawings described herein are used to provide a further understanding of the present disclosure and form a part of the present disclosure. The exemplary embodiments and descriptions of the present disclosure are used to explain the present disclosure and do not constitute an undue limitation on the present disclosure. In the drawings:
图1为本公开的售货机控制方法的一些实施例的流程图。FIG. 1 is a flowchart of some embodiments of a vending machine control method of the present disclosure.
图2为本公开的售货机控制方法的另一些实施例的流程图。FIG. 2 is a flowchart of some other embodiments of the vending machine control method of the present disclosure.
图3为本公开的售货机控制方法的又一些实施例的流程图。FIG. 3 is a flowchart of still other embodiments of the vending machine control method of the present disclosure.
图4为本公开的售货机控制方法的再一些实施例的流程图。FIG. 4 is a flowchart of still other embodiments of the vending machine control method of the present disclosure.
图5为本公开的售货机控制装置的一些实施例的示意图。5 is a schematic diagram of some embodiments of a vending machine control device of the present disclosure.
图6为本公开的售货机控制装置的另一些实施例的示意图。6 is a schematic diagram of other embodiments of the vending machine control device of the present disclosure.
图7为本公开的售货机控制***的一些实施例的示意图。7 is a schematic diagram of some embodiments of a vending machine control system of the present disclosure.
图8为本公开的售货机的一些实施例的示意图。8 is a schematic diagram of some embodiments of the vending machine of the present disclosure.
具体实施方式detailed description
下面通过附图和实施例，对本公开的技术方案做进一步的详细描述。The technical solutions of the present disclosure will be further described in detail below through the accompanying drawings and embodiments.
本公开的售货机控制方法的一些实施例的流程图如图1所示。在售货机开门的状态下开始执行如下操作：A flowchart of some embodiments of the vending machine control method of the present disclosure is shown in FIG. 1. Start the following operations when the vending machine is open:
在步骤101中，售货机在用户的触发下开门。在一些实施例中，用户可以通过扫描售货机二维码，或通过其他方式告知售货机自身的身份以触发售货机开门。In step 101, the vending machine opens the door under the user's trigger. In some embodiments, the user can trigger the vending machine to open the door by scanning the QR code of the vending machine, or informing the identity of the vending machine by other means.
当用户触发开门后，售货机通过图像采集设备采集售货机有效范围内的图像。在一些实施例中，有效范围可以包括售货机的正前方，售货机的里面，以及开门之后门与售货机之间的区域中的一种或多种。在一些实施例中，售货机包括2路以上图像采集设备，可以采集异常和非异常的图像且均缓存，还可以缓存存在异常图像的时刻的未异常的图像，从而保证缓存的高风险图像的质量。在一些实施例中，可以采用5路摄像头，分别位于左、右、上、下侧探测货架，以及由货架内向外探测前方。After the user triggers the door opening, the vending machine collects images within the effective range of the vending machine through the image acquisition device. In some embodiments, the effective range may include one or more of the area directly in front of the vending machine, inside the vending machine, and the area between the door and the vending machine after the door is opened. In some embodiments, the vending machine includes more than two image acquisition devices, which can collect abnormal and non-abnormal images and cache them, and can also cache non-abnormal images at the time of abnormal images, so as to ensure the cache of high-risk images quality. In some embodiments, a five-channel camera may be used, which is located on the left, right, upper, and lower sides to detect the shelf, and to detect the front from the inside to the outside.
在步骤102中，通过图像分析算法判断采集的图像是否异常。在一些实施例中，图像异常可以利用图像分析算法确定。在一些实施例中，图像异常可以指的是一个或多个图像采集设备(如摄像头)被遮挡。若确定采集的图像异常，则执行步骤103。In step 102, an image analysis algorithm is used to determine whether the collected image is abnormal. In some embodiments, the image abnormality can be determined using an image analysis algorithm. In some embodiments, image abnormality may refer to one or more image acquisition devices (such as cameras) being blocked. If it is determined that the collected image is abnormal, step 103 is performed.
在一些实施例中，可以基于设定的图像异常情况分析出现图像异常的可能性，当可能性超过预定异常阈值时确定图像异常。In some embodiments, the possibility of occurrence of an image abnormality may be analyzed based on the set image abnormality, and the image abnormality is determined when the probability exceeds a predetermined abnormality threshold.
在步骤103中，确定当前采集的图像为高风险图像。In step 103, it is determined that the currently acquired image is a high-risk image.
在步骤104中，将高风险图像上报售货机控制服务器，以便服务器识别用户风险行为。在一些实施例中，图像上报的操作可以在确定存在高风险图像后即时进行从而提高图像上报效率；在一些实施例中，可以在售货机关门，或售货机持续开门的时间超过预定时长时执行上报过程，降低图像上报的频率，降低通信和数据处理负担。In step 104, the high-risk image is reported to the vending machine control server, so that the server recognizes the user's risk behavior. In some embodiments, the image reporting operation may be performed immediately after it is determined that there is a high-risk image to improve the image reporting efficiency; in some embodiments, the door may be opened at the sales office, or the vending machine continues to open the door for more than a predetermined duration Perform the reporting process to reduce the frequency of image reporting and reduce the burden of communication and data processing.
通过这样的方法，能够在图像采集异常的情况下及时做出反应，将采集的图像上传给服务器供服务器解析，从而实现及时反馈图像，提高异常发现的效率，提高售货机的安全性。Through this method, it can react in time when the image collection is abnormal, and upload the collected image to the server for the server to analyze, so as to realize the timely feedback of the image, improve the efficiency of abnormal discovery, and improve the safety of the vending machine.
本公开的售货机控制方法的另一些实施例的流程图如图2所示。A flowchart of some other embodiments of the vending machine control method of the present disclosure is shown in FIG. 2.
在步骤201中，售货机在用户的触发下开门。In step 201, the vending machine opens the door under the user's trigger.
在步骤202中，对采集的图像进行图像分析，基于售货机本地存储的图像特征和图像分析模型判断采集的图像是否异常。在一些实施例中，图像异常可以为出现一个或多个摄像头被遮挡。若确定采集的图像异常，则执行步骤203。若未发现图像异常，则执行步骤204。In step 202, image analysis is performed on the collected image, and it is determined whether the collected image is abnormal based on the image characteristics and the image analysis model stored locally at the vending machine. In some embodiments, the image abnormality may be that one or more cameras are blocked. If it is determined that the collected image is abnormal, step 203 is executed. If no abnormal image is found, step 204 is executed.
在一些实施例中，图像异常可以包括多种情况，如：根据图像分析确定售货机内出现了异常物体；用户对货物做出异常操作，异常操作可以包括破坏货物外包装、向货物内部加入物质、从货物中取物质或用户持货物离开摄像头探测区域后又返回中的 一项或多项；或，货物的尺寸、颜色或透明度发生改变等。In some embodiments, the image abnormality may include a variety of situations, such as: it is determined that an abnormal object has occurred in the vending machine according to the image analysis; the user performs abnormal operations on the goods, and the abnormal operations may include destroying the outer packaging of the goods and adding substances to the inside of the goods 1. One or more items taken from the goods or the user returns the goods after leaving the camera detection area with the goods; or, the size, color or transparency of the goods changes.
在一些实施例中，可以利用售货机本地存储的图像特征和图像分析模型进行特征匹配分析，确定图像是否异常。In some embodiments, the image features and image analysis models locally stored in the vending machine may be used to perform feature matching analysis to determine whether the image is abnormal.
在步骤203中，缓存各个图像采集设备采集的图像作为高风险图像，进而执行步骤206。In step 203, the images collected by each image collection device are cached as high-risk images, and then step 206 is executed.
在步骤204中，基于摄像头采集的图像确定货物变化情况，基于重力探测确定货物变化情况。在一些实施例中，可以根据采集的图像，基于图像算法识别出***作的唯一货物，基于图像的轨迹识别出是放回还是拿出操作。根据重力探测，基于重量增加或减少判断是货物放回还是拿出。基于重量变化克数确定可能***作的货物或货物组合确定重量减少量，基于预存的各种货物的重量确定哪个或哪些货物被取出。In step 204, the change in cargo is determined based on the image collected by the camera, and the change in cargo is determined based on gravity detection. In some embodiments, the only goods to be operated can be identified based on the image algorithm based on the collected image, and whether to put back or take out the operation can be identified based on the trajectory of the image. According to gravity detection, it is judged whether the goods are put back or taken out based on the weight increase or decrease. The weight or the combination of goods that may be operated is determined based on the weight change in grams, and the weight reduction is determined, and which one or which goods are taken out is determined based on the weight of various pre-stored goods.
在步骤205中，基于图像和基于重力探测确定的货物变化情况是否一致。在一些实施例中，当根据图像轨迹判断出的动作与重力增减判断的动作不一致，或者图像识别的***作商品与重力变化克数不吻合时，说明该次操作可能出现了商品遮挡、异常置物等高风险行为；当图像识别与重力识别结果一致时，说明这是一次正常的购物流程，不缓存购物图像，将商品加入购物车或从购物车删减。In step 205, it is determined whether the cargo changes based on the image and the gravity detection are consistent. In some embodiments, when the action determined according to the image trajectory is inconsistent with the action determined by the increase or decrease of gravity, or the operated product identified by the image does not coincide with the change in gram of gravity, it may indicate that the operation has occurred product occlusion or abnormality. High-risk behaviors such as storage; when the image recognition and gravity recognition results are consistent, it is a normal shopping process. The shopping images are not cached, and the goods are added to or deleted from the shopping cart.
若变化情况一致，则执行步骤206；若变化情况不一致，则执行步骤203，采集当前时刻的图像作为高风险图像缓存。If the changes are consistent, go to step 206; if the changes are not consistent, go to step 203, and collect the image at the current moment as a high-risk image cache.
在步骤206中，判断售货机是否关门，或是否虽然开门但保持开门状态的时长超过预定第一时长。若触发售货机关门，或开门时长超过预定第一时长，则执行步骤207。In step 206, it is determined whether the vending machine closes the door, or whether the time for which the door remains open despite opening the door exceeds a predetermined first time. If the door of the sales office is triggered, or the opening time exceeds the predetermined first time, step 207 is executed.
在一些实施例中，可以只在用户关门结算时，执行步骤207。在另一些实施例中，可以在售货机通信闲时执行步骤207。In some embodiments, step 207 may be performed only when the user closes the door to settle. In other embodiments, step 207 may be performed when the vending machine is idle.
在步骤207中，判断当前缓存区是否缓存有高风险图像。若缓存有高风险图像，则执行步骤208。In step 207, it is determined whether high-risk images are cached in the current cache area. If high-risk images are cached, step 208 is executed.
在步骤208中，从缓存的高风险图像中筛选图像帧。在一些实施例中，筛选操作可以包括挑选图像清晰的，未完全被遮挡的数据帧，以及对于极为相似的数据只保留一帧或几帧等。In step 208, image frames are filtered from the cached high-risk images. In some embodiments, the filtering operation may include selecting data frames with clear images that are not completely occluded, and retaining only one frame or a few frames for very similar data.
在步骤209中，将筛选出的高风险图像帧上报售货机控制服务器，并删除缓存区缓存的高风险图像。In step 209, the filtered high-risk image frames are reported to the vending machine control server, and the high-risk images cached in the cache area are deleted.
通过这样的方法，能够通过图像是否异常，以及图像与重力探测的变化情况判读结果是否一致这两种途径判断是否触发缓存高风险图像并上报，降低了事件漏提取的 概率，提高了售货机贩卖货物的安全性；同时，也无需持续存储图像，降低了对售货机存储空间的要求。Through this method, whether the image is abnormal and whether the interpretation result of the image and the gravity detection are consistent can be judged whether the cache high-risk image is triggered and reported, which reduces the probability of missed event extraction and improves vending machine sales. The safety of the goods; at the same time, there is no need to continuously store images, reducing the storage space requirements of the vending machine.
相关技术中，基于视频图像的无人售货机相对于基于纯重力、RFID(Radio Frequency Identification，射频识别)的无人售货机，由于其具备更广阔的选品空间，更优质的用户体验，引起越来越多的关注。但是基于重力的无人售货机存在无法监控用户购物流程的痛点，基于图像的无人售货机需要克服真实场景中容易出现的遮挡摄像头、危险品放置、双手盖住商品盗窃等高风险行为，仍然会造成较高的风险及较高的货损。In the related art, the unmanned vending machine based on video images is more powerful than the unmanned vending machine based on pure gravity and RFID (Radio Frequency Identification), because of its wider selection space and better user experience. More and more attention. However, the gravity-based unmanned vending machine has the pain point that it cannot monitor the user's shopping process. The image-based unmanned vending machine needs to overcome the high-risk behaviors such as blocking the camera, placing dangerous goods, and covering the theft of goods with both hands. Will cause higher risks and higher cargo damage.
相关的无人售货机风险控制主要包含视频图像与重力两种常规解决方案：The related unmanned vending machine risk control mainly includes two conventional solutions: video image and gravity:
(1)通过布置监控摄像头持续监控顾客消费过程，当补货人员或无人售货机维护人员发现售货机内出现异常商品或商品丢失时，可以通过查看录像，还原所有用户的完整的购物过程。(1) Continuously monitor the customer's consumption process by arranging surveillance cameras. When the replenishment personnel or unmanned vending machine maintenance personnel find abnormal goods or lost goods in the vending machine, they can restore the complete shopping process of all users by viewing the video.
基于视频图像的风险控制解决方案无法提供有效的实时预警，只有当无人售货机运维人员发现售货机内的商品异常(有异常物品)或核对时发现商品丢失时，才能够发现风险，同时需要人工回放全部录像。此外，上述方案首先需要保存全部录像，占用无人售货机大量的存储空间，而如果通过网络回传视频到云端，则会占用大量网络带宽，运维成本急剧增加。同时，人工追溯视频会提高人力成本，且降低工作效率。基于纯视觉的行为识别算法目前尚不够准确，因此单独利用图像算法虽然能够节约存储空间，降低人力成本，但容易触发大量虚假报警，同时由于商品偷盗行为、异常置物行为与正常购物行为相似度高，很难利用算法区分，因此容易发生漏报，留下安全隐患。The risk control solution based on video images cannot provide effective real-time early warning. Only when the unmanned vending machine operation and maintenance personnel finds that the merchandise in the vending machine is abnormal (there are abnormal items) or the product is lost during verification, it can detect the risk. All videos need to be played back manually. In addition, the above solution first needs to save all the videos and occupy a lot of storage space of the unmanned vending machine. If the video is returned to the cloud through the network, it will occupy a lot of network bandwidth and the operation and maintenance cost will increase dramatically. At the same time, manually tracing video will increase labor costs and reduce work efficiency. The behavior recognition algorithm based on pure vision is not accurate enough at present. Therefore, although the image algorithm alone can save storage space and reduce labor costs, it is easy to trigger a large number of false alarms. At the same time, due to the theft of goods, the abnormal storage behavior and the normal shopping behavior are similar It is difficult to distinguish using algorithms, so false negatives are prone to occur, leaving security risks.
(2)当无人售货机发现重力异常时，可以提供后台报警，运维人员可根据警情，到现场查看售货机情况，排除隐患。(2) When an unmanned vending machine discovers gravity anomalies, it can provide a background alarm, and the operation and maintenance personnel can go to the scene to check the vending machine situation according to the police situation to eliminate hidden dangers.
基于重力的解决方案中，重力传感器能够时刻监控被取走或放回商品的重量，从而实现扣款。然而，当用户取走商品但是放回等重量的无用物品(比如沙子)或危险品时，无人售货机无法及时预警，从而导致高货损与高风险。同时由于不具备图像监控功能，因此当发现无人售货机出现异常商品时，无法进行追溯，无法定位何时、何人人偷盗了商品或置入了危险品。In the gravity-based solution, the gravity sensor can monitor the weight of the goods taken or returned at any time, so as to realize the deduction. However, when the user takes the goods but puts back the same weight of useless items (such as sand) or dangerous goods, the unmanned vending machine cannot promptly warn in time, resulting in high cargo damage and high risk. At the same time, because there is no image monitoring function, when it is found that there are abnormal goods in the unmanned vending machine, it cannot be traced back, and it cannot be located when and who has stolen the goods or placed dangerous goods.
通过本公开上述实施例中的方法，能够根据图像分析的结果发现异常行为，并在发现异常情况时及时触发上报异常图像，一方面降低了数据传输的负担，成数量级的 降低了人工判读、回溯的工作量；另一方面还能够提高异常情况发现的效率，另外，还能够降低漏判的概率，提高售货安全性。Through the method in the above embodiments of the present disclosure, abnormal behavior can be discovered based on the results of image analysis, and abnormal image reporting can be triggered in time when an abnormal situation is discovered. On the one hand, the burden of data transmission is reduced, and manual interpretation and backtracking are reduced by orders of magnitude. On the other hand, it can also improve the efficiency of abnormal situation detection, in addition, it can also reduce the probability of missed judgment and improve the safety of sales.
在一些实施例中，为了应对违规用户将从售货机中取出的物品中掺入异常物质(如注射有毒液体)，或置换为其他的外观相同、相似的物品重新放回售货机的情况，本公开的售货机控制方法的又一些实施例的流程图如图3所示。In some embodiments, in order to cope with the situation where the illegal user mixes an abnormal substance (such as an injection of toxic liquid) into the item taken out from the vending machine, or replaces it with another item of the same appearance and similarity and put it back in the vending machine, this book A flowchart of still other embodiments of the disclosed vending machine control method is shown in FIG. 3.
在步骤301中，根据图像或重力探测确定货物被取出售货机。In step 301, it is determined that the goods are taken out and sold according to the image or gravity detection.
在步骤302中，记录被取出的货物的唯一标识。在一些实施例中，每个货物可以配置有唯一标识，如RFID标识。当用户结账关闭售货机后，删除记录的货物的唯一标识。In step 302, the unique identification of the removed goods is recorded. In some embodiments, each cargo may be configured with a unique identification, such as an RFID identification. When the user closes and closes the vending machine, the unique identifier of the recorded goods is deleted.
当根据图像或重力探测确定货物被放回售货机时，触发执行步骤303。When it is determined that the goods are returned to the vending machine according to the image or gravity detection, step 303 is triggered to be executed.
在步骤303中，采集放回的货物的标识，并判断与取出的货物的标识是否匹配。若两者匹配，则执行步骤305；若两者不匹配，则执行步骤304。In step 303, the identification of the returned goods is collected, and it is determined whether it matches the identification of the removed goods. If the two match, go to step 305; if the two do not match, go to step 304.
在步骤304中，缓存当前时刻，或截取采集的本次售货过程的图像作为高风险图像。In step 304, the current time is buffered, or the collected image of the sales process is intercepted as a high-risk image.
在步骤305中，判断用户是否对货物做出异常操作。异常操作可以包括破坏货物外包装、向货物内部加入物质、从货物中取物质或用户持货物离开摄像头探测区域后又返回中的一项或多项。若确定用户做出异常操作，则执行步骤304。若确定用户未做出异常操作，则执行步骤306。In step 305, it is determined whether the user has performed an abnormal operation on the goods. The abnormal operation may include one or more of destroying the outer packaging of the goods, adding substances to the inside of the goods, taking substances from the goods, or returning from the camera detection area after the user holds the goods. If it is determined that the user makes an abnormal operation, step 304 is executed. If it is determined that the user has not performed an abnormal operation, step 306 is executed.
在步骤306中，确定用户正常放回物品，删除标识。In step 306, it is determined that the user returns the item normally, and the logo is deleted.
通过这样的方法，能够应对将从售货机中取出的物品中掺入异常物质(如注射有毒液体)，或置换为其他的外观相同、相似的物品重新放回售货机的情况，进一步提高了售货机贩卖物品的安全性。Through this method, it is possible to cope with the situation in which abnormal substances (such as injection of toxic liquids) are mixed into the items taken out of the vending machine, or replaced with other items with the same appearance and similarity and then put back into the vending machine, which further improves the sales. The safety of goods sold by cargo planes.
本公开的售货机控制方法的再一些实施例的流程图如图4所示。A flowchart of still other embodiments of the vending machine control method of the present disclosure is shown in FIG. 4.
在步骤401中，接收来自用户的开门请求。用户的开门请求中可以包括用户的个人身份信息，如用户对于售货机，或售货机相关的平台的账号信息，或身份证号码信息等。In step 401, a door opening request from the user is received. The user's door opening request may include the user's personal identity information, such as the user's account information for the vending machine, or the platform related to the vending machine, or ID number information.
在步骤402中，将用户身份信息上传平台判断用户是否为限制用户。若确定为限制用户，则执行步骤403；若确定为正常用户，则执行步骤404。In step 402, the user identity information is uploaded to the platform to determine whether the user is a restricted user. If it is determined to be a restricted user, step 403 is performed; if it is determined to be a normal user, step 404 is performed.
在步骤403中，拒绝触发售货机开门。In step 403, refusal to trigger the vending machine to open the door.
在步骤404中，触发售货机开门。In step 404, the vending machine is triggered to open the door.
在步骤405中，判断在购物过程中产生了高风险图像。在一些实施例中，可以采用如图2、3所示实施例中的方式确定是否产生高风险图像。在一些实施例中，当确定产生高风险图像时，可以先缓存高风险图像以便后续上报。若确定产生(缓存)了高风险图像，则执行步骤406。In step 405, it is determined that a high-risk image was generated during the shopping process. In some embodiments, the method shown in the embodiments shown in FIGS. 2 and 3 may be used to determine whether to generate a high-risk image. In some embodiments, when it is determined to generate a high-risk image, the high-risk image may be cached first for subsequent reporting. If it is determined that the high-risk image is generated (cached), step 406 is executed.
在步骤406中，向服务器上报高风险图像。在一些实施例中，可以先从产生的高风险图像中提取部分帧，只上报提取的图像帧，从而降低数据传输负担。In step 406, the high-risk image is reported to the server. In some embodiments, some frames may be extracted from the generated high-risk image first, and only the extracted image frames are reported, thereby reducing the data transmission burden.
在步骤407中，服务器发出告警，以便工作人员人工识别，或采用进一步的图像识别方法判读高风险图像。In step 407, the server issues an alarm so that the staff can manually identify it, or use a further image recognition method to interpret the high-risk image.
在步骤408中，根据高风险图像确定用户是否执行违规操作。若确定用户执行了违规操作，则执行步骤409；若确定用户未执行违规操作，则认为当前出现了误判，执行步骤412。In step 408, it is determined whether the user performs a violation operation based on the high-risk image. If it is determined that the user has performed the violation operation, step 409 is performed; if it is determined that the user has not performed the violation operation, it is considered that a misjudgment has occurred, and step 412 is performed.
在步骤409中，判断该用户违规操作统计值是否达到了预定门限。若已达到预定门限，则执行步骤411；若未达到预定门限，则执行步骤410。In step 409, it is determined whether the statistical value of the user's illegal operation has reached a predetermined threshold. If the predetermined threshold has been reached, step 411 is performed; if the predetermined threshold has not been reached, step 410 is performed.
在步骤410中，增加记录的用户违规操作统计值。在一些实施例中，还可以根据用户当前违规操作的种类确定增加的统计值的量，如普通违规操作加1，恶意违规操纵加10等。In step 410, the recorded user violation operation statistical value is increased. In some embodiments, the amount of increased statistical value may also be determined according to the type of the user's current violation operation, such as a common violation operation plus 1, a malicious violation manipulation plus 10, and so on.
在步骤411中，修改用户权限为限制用户。In step 411, the user right is modified to restrict the user.
在步骤412中，利用高风险图像更新图像分析训练数据集。In step 412, the image analysis training data set is updated with high-risk images.
在步骤413中，根据更新后的图像分析训练数据集更新图像特征和图像分析模型，并发送给提供给售货机更新售货机本地的图像特征和图像分析模型。在一些实施例中，考虑到售货机的存储空间和数据处理负担，可以只发送给提供了该高风险图像的售货机，在避免售货机数据量过多的同时，也能够针对售货机所处的环境、所经历的情况实现动态更新。In step 413, the image features and the image analysis model are updated according to the updated image analysis training data set, and sent to the vending machine to update the local image features and the image analysis model. In some embodiments, considering the storage space and data processing burden of the vending machine, it can only be sent to the vending machine that provides the high-risk image, while avoiding excessive vending machine data, it can also target the vending machine's location. The environment and the situation experienced are dynamically updated.
通过这样的方法，能够根据用户违规操作情况动态改变用户的权限，预防用户进一步违规操作，提高安全性；还能够动态更新图像分析的数据基础，图像提取特征和模型，提高售货机图像分析的准确性，降低错误判断的概率。由于在售货机使用过程中，能够通过有效的在线自反馈机制进行在线的难例挖掘，寻找不容易被准确识别的图像并回传，使售货机实现模型的自我迭代与优化，从而快速适应部署新环境、新商品的图像识别需求，节省大量训练图像的图像采集成本，同时图像识别准确度的提升也会进一步提高用户体验、减少货物损失。Through this method, the user's authority can be dynamically changed according to the user's illegal operation, preventing the user from further illegal operation and improving security; it can also dynamically update the data foundation of image analysis, image extraction features and models, and improve the accuracy of vending machine image analysis To reduce the probability of wrong judgment. During the process of using the vending machine, it is possible to mine difficult examples online through an effective online self-feedback mechanism, find images that are not easily recognized accurately and return them, so that the vending machine realizes self-iteration and optimization of the model, thus quickly adapting to deployment The new environment and new product image recognition requirements save a lot of image acquisition costs of training images, and the improvement of image recognition accuracy will further improve user experience and reduce cargo losses.
本公开的售货机控制装置的一些实施例的示意图如图5所示。A schematic diagram of some embodiments of the vending machine control device of the present disclosure is shown in FIG. 5.
高风险图像确定单元501能够在售货机开门的状态下，判断采集的图像是否异常，并在确定异常的情况下确定采集的图像为高风险图像。在一些实施例中，售货机包括2路以上图像采集设备，可以采集异常和非异常的图像且均缓存，还可以缓存存在异常图像的时刻的未异常的图像，从而保证缓存的高风险图像的质量。在一些实施例中，图像异常可以利用基于售货机本地的图像分析模型和图像特征，次用图像分析算法确定。在一些实施例中，图像异常可以指的是一个或多个图像采集设备(如摄像头)被遮挡。The high-risk image determining unit 501 can determine whether the collected image is abnormal when the door of the vending machine is opened, and determine that the collected image is a high-risk image if the abnormality is determined. In some embodiments, the vending machine includes more than two image acquisition devices, which can collect abnormal and non-abnormal images and cache them, and can also cache non-abnormal images at the time of abnormal images, so as to ensure the cache of high-risk images quality. In some embodiments, the image anomaly can be determined using an image analysis model and image features based on the local vending machine, and a second time using an image analysis algorithm. In some embodiments, image abnormality may refer to one or more image acquisition devices (such as cameras) being blocked.
上报单元502能够将高风险图像上报售货机控制服务器，以便服务器识别用户风险行为。在一些实施例中，可以在售货机关门，或售货机持续开门的时间超过预定时长时执行上报过程，降低图像上报的频率，降低通信和数据处理负担。The reporting unit 502 can report the high-risk image to the vending machine control server, so that the server recognizes the user's risk behavior. In some embodiments, the reporting process may be performed when the door of the vending machine or the vending machine continues to open for more than a predetermined period of time, reducing the frequency of image reporting and reducing the communication and data processing burden.
这样的售货机控制装置能够在图像采集异常的情况下及时做出反应，将采集的图像上传给服务器供服务器解析，从而实现图像的及时反馈，提高异常发现的效率，提高售货机的安全性。Such a vending machine control device can react in a timely manner when the image collection is abnormal, and upload the collected image to the server for the server to analyze, thereby realizing timely feedback of the image, improving the efficiency of abnormal discovery, and improving the safety of the vending machine.
在一些实施例中，图像异常还可以包括多种情况：如根据图像分析确定售货机内出现了异常物体；用户对货物做出异常操作，异常操作包括破坏货物外包装、向货物内部加入物质、从货物中取物质或用户持货物离开摄像头探测区域后又返回中的一项或多项；或，货物的尺寸、颜色或透明度发生改变，从而降低漏发现违规操作的可能性，进一步提高了安全性。In some embodiments, the image abnormality may also include multiple situations: for example, according to image analysis, it is determined that an abnormal object has occurred in the vending machine; the user performs abnormal operations on the goods. The abnormal operations include destroying the outer packaging of the goods, adding substances into the goods, One or more items taken from the goods or the user left the camera detection area with the goods and returned to the camera; or, the size, color or transparency of the goods were changed, thereby reducing the possibility of leaking violations and further improving safety Sex.
在一些实施例中，上报单元502可以先从缓存的高风险图像中筛选部分高风险图像帧，再将筛选出的高风险图像帧上报售货机控制服务器，并删除缓存区缓存的高风险图像。在一些实施例中，可以删除部分像素低的帧，或从极为相似的帧中选择几帧。这样的售货机控制装置能够降低需要上报给服务器的数据量，一方面降低数据传输负担，另一方面提高数据传输效率，便于服务器侧对于违规操作的迅速反应。In some embodiments, the reporting unit 502 may first select a portion of the high-risk image frames from the cached high-risk images, and then report the selected high-risk image frames to the vending machine control server, and delete the cached high-risk images. In some embodiments, some frames with low pixels can be deleted, or a few frames can be selected from very similar frames. Such a vending machine control device can reduce the amount of data that needs to be reported to the server, on the one hand reduce the data transmission burden, on the other hand improve the data transmission efficiency, and facilitate the server side to quickly respond to illegal operations.
在一些实施例中，如图5所示，售货机控制装置还可以包括重力处理单元503和图像处理单元504。重力处理单元503能够基于重力探测确定货物变化情况，在一些实施例中，可以根据重力探测确定重量减少量，基于预存的各种货物的重量确定哪个或哪些货物被取出。图像处理单元504能够基于摄像头采集的图像确定货物变化情况，在一些实施例中，可以根据采集的图像确定哪个或哪些货物被取出。In some embodiments, as shown in FIG. 5, the vending machine control device may further include a gravity processing unit 503 and an image processing unit 504. The gravity processing unit 503 can determine the change of the cargo based on the gravity detection. In some embodiments, the weight reduction can be determined according to the gravity detection, and which cargo or cargos are taken out can be determined based on the weight of various pre-stored cargoes. The image processing unit 504 can determine the change of goods based on the image collected by the camera. In some embodiments, it can determine which goods or goods are taken out based on the collected image.
高风险图像确定单元501能够在基于图像和基于重力探测确定的货物变化情况 不一致的情况下缓存高风险图像，以便上报单元上报。The high-risk image determination unit 501 can cache the high-risk image in the case where the change of the cargo determined based on the image and the gravity detection is inconsistent, so that the reporting unit can report.
这样的售货机控制装置能够通过图像是否异常，以及图像与重力探测的变化情况判读结果是否一致这两种途径判断是否触发缓存高风险图像并上报，降低了事件漏提取的概率，提高了售货机贩卖货物的安全性。Such a vending machine control device can determine whether the cached high-risk image is triggered and reported by the two ways of judging whether the image is abnormal and whether the result of the image and the change of gravity detection is consistent, which reduces the probability of missed event extraction and improves the vending machine The safety of selling goods.
在一些实施例中，售货机控制装置还可以包括开门控制单元505，能够在收到用户触发开门操作时，判断用户是否为限制用户。若确定用户为限制用户，则拒绝触发售货机开门；若用户为正常状态的用户，则执行开门操作。In some embodiments, the vending machine control device may further include a door opening control unit 505, which can determine whether the user is a restricted user when receiving the user's trigger of the door opening operation. If it is determined that the user is a restricted user, it refuses to trigger the vending machine to open the door; if the user is a normal user, the door opening operation is performed.
这样的售货机控制装置能够利用用户信用体系确定是否向用户提供服务，进一步提高了安全性。Such a vending machine control device can use the user credit system to determine whether to provide services to the user, which further improves security.
本公开售货机控制装置的一些实施例的结构示意图如图6所示。售货机控制装置包括存储器601和处理器602。其中：存储器601可以是磁盘、闪存或其它任何非易失性存储介质。存储器用于存储上文中售货机控制方法的对应实施例中的指令。处理器602耦接至存储器601，可以作为一个或多个集成电路来实施，例如微处理器或微控制器。该处理器602用于执行存储器中存储的指令，能够实现及时反馈图像，提高异常发现的效率，提高售货机的安全性。A structural schematic diagram of some embodiments of the vending machine control device of the present disclosure is shown in FIG. 6. The vending machine control device includes a memory 601 and a processor 602. The memory 601 may be a magnetic disk, flash memory, or any other non-volatile storage medium. The memory is used to store the instructions in the corresponding embodiments of the vending machine control method above. The processor 602 is coupled to the memory 601, and may be implemented as one or more integrated circuits, such as a microprocessor or a microcontroller. The processor 602 is used to execute instructions stored in the memory, which can realize timely feedback of images, improve the efficiency of abnormal discovery, and improve the safety of the vending machine.
在另一些实施例中，一种计算机可读存储介质，其上存储有计算机程序指令，该指令被处理器执行时实现售货机控制方法对应实施例中的方法的步骤。本领域内的技术人员应明白，本公开的实施例可提供为方法、装置、或计算机程序产品。因此，本公开可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本公开可采用在一个或多个其中包含有计算机可用程序代码的计算机可用非瞬时性存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。In other embodiments, a computer-readable storage medium has stored thereon computer program instructions, which when executed by a processor, implements the steps of the method in the embodiment corresponding to the vending machine control method. Those skilled in the art should understand that the embodiments of the present disclosure may be provided as methods, devices, or computer program products. Therefore, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer usable non-transitory storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code .
本公开的售货机控制***的一些实施例的示意图如图7所示。售货机控制装置71可以为上文中提到的任意一种售货机控制装置。售货机控制服务器70能够在收到高风险图像的情况下，以便判读高风险图像。在一些实施例中，为了提高风险评估的及时性，可以发出告警。A schematic diagram of some embodiments of the vending machine control system of the present disclosure is shown in FIG. 7. The vending machine control device 71 may be any of the vending machine control devices mentioned above. The vending machine control server 70 can interpret the high-risk image when receiving the high-risk image. In some embodiments, in order to improve the timeliness of risk assessment, an alarm may be issued.
这样的售货机控制***中，售货机控制装置能够在图像采集异常的情况下及时做出反应，将采集的图像上传给服务器供服务器解析，以便及时确定违规操作，从而实现通过图像的及时反馈，提高异常发现的效率，提高售货机的安全性；能够根据用户违规操作情况动态改变用户的权限，预防用户进一步违规操作，提高安全性。In such a vending machine control system, the vending machine control device can react in a timely manner when the image acquisition is abnormal, upload the collected image to the server for the server to analyze, in order to determine the illegal operation in time, so as to achieve timely feedback through the image, Improve the efficiency of abnormal discovery and improve the security of vending machines; it can dynamically change the user's authority according to the user's illegal operation, prevent the user from further illegal operation, and improve safety.
在一些实施例中，若根据高风险图像判读后确定用户执行违规操作，且违规操作统计值达到预定门限，则修改用户权限为限制用户，从而使该用户再次使用售货机时，售货机拒绝为用户开门，从而实现用户信用的统计和有效应用，提高售货机售货的安全性。In some embodiments, if it is determined after the interpretation of the high-risk image that the user performs the violation operation, and the violation operation statistical value reaches a predetermined threshold, the user authority is modified to restrict the user, so that when the user uses the vending machine again, the vending machine refuses to The user opens the door, so as to realize the statistics and effective application of user credit, and improve the safety of vending machine sales.
在一些实施例中，若售货机控制服务器70根据高风险图像判读后确定用户未执行违规操作，则可以利用高风险图像更新图像分析训练数据集，根据更新后的图像分析训练数据集更新图像特征和图像分析模型，并发送给售货机，售货机控制装置更新本地的图像特征和图像分析模型，实现自身图像分析功能的优化。In some embodiments, if the vending machine control server 70 determines that the user has not performed the illegal operation after reading the high-risk image, the high-risk image may be used to update the image analysis training data set, and the image characteristics may be updated according to the updated image analysis training data set. And the image analysis model, and sent to the vending machine, the vending machine control device updates the local image characteristics and image analysis model, and realizes the optimization of its own image analysis function.
这样的售货机控制***能够动态更新图像分析的数据基础，图像提取特征和模型，提高售货机图像分析的准确性，降低错误判断的概率。Such a vending machine control system can dynamically update the data basis of image analysis, image extraction features and models, improve the accuracy of vending machine image analysis, and reduce the probability of erroneous judgment.
本公开的售货机80的一些实施例的示意图如图8所示。储物装置81能够承载售卖的货物；重力探测装置82能够采集货架上的重量变化情况；图像采集装置83能够采集售货机内部，以及用户操作时所处区域的图像。在一些实施例中，重力探测装置和图像采集装置可以包括多个。售货机控制装置84可以为上文中提到的任意一种售货机控制装置。A schematic diagram of some embodiments of the vending machine 80 of the present disclosure is shown in FIG. 8. The storage device 81 can carry the goods for sale; the gravity detection device 82 can collect the weight change on the shelf; the image collection device 83 can collect images of the inside of the vending machine and the area where the user operates. In some embodiments, the gravity detection device and the image acquisition device may include multiple. The vending machine control device 84 may be any of the vending machine control devices mentioned above.
这样的售货机能够在图像采集异常的情况下及时做出反应，将采集的图像上传给服务器供服务器解析，从而实现图像的及时反馈，提高异常发现的效率，提高售货机的安全性。Such a vending machine can react in a timely manner when the image acquisition is abnormal, and upload the collected image to the server for the server to analyze, thereby realizing timely feedback of the image, improving the efficiency of abnormal discovery, and improving the security of the vending machine.
最后应当说明的是：以上实施例仅用以说明本公开的技术方案而非对其限制；尽管参照较佳实施例对本公开进行了详细的说明，所属领域的普通技术人员应当理解：依然可以对本公开的具体实施方式进行修改或者对部分技术特征进行等同替换；而不脱离本公开技术方案的精神，其均应涵盖在本公开请求保护的技术方案范围当中。Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure but not to limit them; although the present disclosure has been described in detail with reference to the preferred embodiments, persons of ordinary skill in the art should understand that: The specific embodiments disclosed are modified or equivalently replaced with some technical features; without departing from the spirit of the technical solutions of the present disclosure, they should all be included in the scope of the technical solutions claimed by the present disclosure.

Claims (17)

  1. 一种售货机控制方法，包括：A vending machine control method, including:
    在售货机开门状态下：With the door open:
    采集售货机有效范围内的图像；Collect images within the effective range of the vending machine;
    对采集的图像进行风险分析；Perform risk analysis on the collected images;
    若确定采集的图像异常，则确定采集的图像为高风险图像；If it is determined that the collected image is abnormal, it is determined that the collected image is a high-risk image;
    将高风险图像上报售货机控制服务器，以便识别用户风险行为。Report high-risk images to the vending machine control server to identify user risk behaviors.
  2. 根据权利要求1所述的方法，其中，所述将高风险图像上报售货机控制服务器包括：The method according to claim 1, wherein the reporting of the high-risk image to the vending machine control server includes:
    对所述异常图像进行筛选，挑选出部分异常图像；Screen the abnormal images and select some abnormal images;
    将挑选出的异常图像上报所述售货机控制服务器。The selected abnormal image is reported to the vending machine control server.
  3. 根据权利要求1或2所述的方法，其中，所述图像异常包括以下一项或多项：The method according to claim 1 or 2, wherein the image abnormality includes one or more of the following:
    根据图像分析确定一个或多个图像采集设备被遮挡；According to the image analysis, it is determined that one or more image acquisition devices are blocked;
    根据图像分析确定售货机内出现了异常物体；According to the image analysis, it is determined that abnormal objects appear in the vending machine;
    用户对货物做出异常操作，所述异常操作包括破坏货物外包装、向货物内部加入物质、从货物中取物质或用户持货物离开摄像头探测区域后又返回中的一项或多项；The user performs abnormal operations on the goods, the abnormal operations include one or more of destroying the outer packaging of the goods, adding substances into the goods, taking substances from the goods, or returning from the camera detection area when the user holds the goods;
    或，or,
    货物的尺寸、颜色或透明度发生改变。The size, color or transparency of the goods has changed.
  4. 根据权利要求1或2所述的方法，其中，所述将所述高风险图像上报售货机控制服务器包括：The method according to claim 1 or 2, wherein the reporting of the high-risk image to the vending machine control server includes:
    在售货机关门后或保持开门状态的时长超过预定第一时长的情况下，判断是否存在所述高风险图像；Judging whether the high-risk image is present after the door of the sales office or the duration of keeping the door open exceeds the predetermined first duration;
    若存在所述高风险图像，则上报所述高风险图像。If the high-risk image exists, the high-risk image is reported.
  5. 根据权利要求1所述的方法，还包括：The method of claim 1, further comprising:
    基于采集的图像确定货物变化情况；Determine the changes of goods based on the collected images;
    基于重力探测确定货物变化情况；Determine cargo changes based on gravity detection;
    在基于图像和基于重力探测确定的货物变化情况不一致的情况下，执行确定采集的图像为高风险图像的操作。In the case where the change of the cargo determined based on the image and based on the gravity detection is inconsistent, the operation of determining the collected image as a high-risk image is performed.
  6. 根据权利要求1所述的方法，还包括：The method of claim 1, further comprising:
    在根据图像或重力探测确定货物被取出售货机的情况下，记录被取出的货物的唯一标识，其中，所述售货机中的每个货物具备唯一标识；In the case where it is determined that the goods are taken from the vending machine according to the image or gravity detection, record the unique identification of the taken out goods, wherein each item in the vending machine has a unique identification;
    在根据图像或重力探测确定货物被放回售货机的情况下，确定放回的货物的标识与取出的货物的标识是否匹配；In the case where it is determined that the goods are returned to the vending machine based on images or gravity detection, determine whether the identification of the returned goods matches the identification of the removed goods;
    若不匹配，则执行确定采集的图像为高风险图像的操作；If it does not match, perform the operation of determining that the acquired image is a high-risk image;
    若匹配，则删除记录的所述货物的标识。If there is a match, the recorded identification of the goods is deleted.
  7. 根据权利要求1所述的方法，还包括：The method of claim 1, further comprising:
    接收来自用户的开门请求；Receive door opening requests from users;
    若用户账户为限制用户，则拒绝触发售货机开门。If the user account is a restricted user, it refuses to trigger the vending machine to open the door.
  8. 根据权利要求1所述的方法，其中，对采集的图像进行风险分析包括：The method according to claim 1, wherein performing risk analysis on the collected image includes:
    基于售货机本地的图像特征和图像分析模型对采集的图像进行风险分析。Based on the local image features and image analysis model of the vending machine, the risk analysis of the collected images is carried out.
  9. 根据权利要求1或7所述的方法，还包括：The method according to claim 1 or 7, further comprising:
    售货机控制服务器在收到所述高风险图像的情况下，识别所述高风险图像；The vending machine control server recognizes the high-risk image when receiving the high-risk image;
    若通过识别所述高风险图像确定用户执行违规操作，则修改用户违规操作统计值；If it is determined that the user performs the illegal operation by identifying the high-risk image, the statistical value of the user's illegal operation is modified;
    在所述用户的违规操作统计值达到预定门限的情况下，将用户标记为限制用户。When the statistical value of the illegal operation of the user reaches a predetermined threshold, the user is marked as a restricted user.
  10. 根据权利要求8所述的方法，还包括：The method of claim 8, further comprising:
    若所述售货机控制服务器根据所述高风险图像确定用户未执行违规操作，则利用所述高风险图像更新图像分析训练数据集；If the vending machine control server determines that the user has not performed a violation operation based on the high-risk image, the high-risk image is used to update the image analysis training data set;
    根据更新后的图像分析训练数据集更新图像特征和图像分析模型，并发送给售货机以便更新所述售货机的图像特征和图像分析模型。The image features and the image analysis model are updated according to the updated image analysis training data set, and sent to the vending machine to update the image features and the image analysis model of the vending machine.
  11. 一种售货机控制装置，包括：A vending machine control device, including:
    高风险图像确定单元，被配置为在售货机开门状态下对采集的售货机有效范围内的图像进行风险分析，若确定采集的图像异常，则确定采集的图像为高风险图像；The high-risk image determination unit is configured to perform a risk analysis on the collected images of the vending machine in the open state of the vending machine. If the collected image is determined to be abnormal, the collected image is determined to be a high-risk image;
    上报单元，被配置为将高风险图像上报售货机控制服务器，以便识别用户风险行为。The reporting unit is configured to report high-risk images to the vending machine control server in order to identify user risk behaviors.
  12. 一种售货机控制装置，包括：A vending machine control device, including:
    存储器；以及Storage; and
    耦接至所述存储器的处理器，所述处理器被配置为基于存储在所述存储器的指令执行如权利要求1至8任一项所述的方法。A processor coupled to the memory, the processor configured to perform the method according to any one of claims 1 to 8 based on instructions stored in the memory.
  13. 一种计算机可读存储介质，其上存储有计算机程序指令，该指令被处理器执行时实现权利要求1至8任意一项所述的方法的步骤。A computer-readable storage medium having computer program instructions stored thereon, which when executed by a processor implements the steps of the method of any one of claims 1 to 8.
  14. 一种售货机控制***，包括：A vending machine control system, including:
    权利要求11或12所述的售货机控制装置；和The vending machine control device according to claim 11 or 12; and
    售货机控制服务器，被配置为：在收到所述高风险图像的情况下，识别所述高风险图像。The vending machine control server is configured to recognize the high-risk image when receiving the high-risk image.
  15. 根据权利要求14所述的***，其中，The system of claim 14, wherein:
    所述售货机被配置为基于售货机本地的图像特征和图像分析模型对采集的图像进行风险分析；The vending machine is configured to perform risk analysis on the collected images based on the local image features and image analysis models of the vending machine;
    所述售货机控制服务器还被配置为：The vending machine control server is further configured to:
    若根据所述高风险图像确定用户未执行违规操作，则利用所述高风险图像更新图像分析训练数据集；If it is determined from the high-risk image that the user has not performed a violation operation, the image analysis training data set is updated using the high-risk image;
    根据更新后的图像分析训练数据集更新图像特征和图像分析模型，并发送给售货机。Update the image features and image analysis model according to the updated image analysis training data set, and send them to the vending machine.
  16. 根据权利要求14或15所述的***，其中，The system according to claim 14 or 15, wherein
    所述售货机控制服务器还被配置为：The vending machine control server is further configured to:
    若通过识别所述高风险图像确定用户执行违规操作，则修改用户违规操作统计值；If it is determined that the user performs the illegal operation by identifying the high-risk image, the statistical value of the user's illegal operation is modified;
    在所述用户的违规操作统计值达到预定门限的情况下，将用户标记为限制用户。When the statistical value of the illegal operation of the user reaches a predetermined threshold, the user is marked as a restricted user.
  17. 一种售货机，包括：A vending machine, including:
    储物装置，被配置为承载货物；Storage device, configured to carry cargo;
    图像采集装置，被配置为采集有效范围内图像；The image acquisition device is configured to acquire images within the effective range;
    重力探测装置，被配置为探测所述储物装置承载的货物重量变化情况；和，A gravity detection device configured to detect changes in the weight of the cargo carried by the storage device; and,
    权利要求11或12所述的售货机控制装置。The vending machine control device according to claim 11 or 12.
PCT/CN2019/099758 2018-12-29 2019-08-08 Control method, device and system for vending machine, and vending machine WO2020134100A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201811630378.4 2018-12-29
CN201811630378.4A CN109711359B (en) 2018-12-29 2018-12-29 Vending machine control method, vending machine control device and computer-readable storage medium

Publications (1)

Publication Number Publication Date
WO2020134100A1 true WO2020134100A1 (en) 2020-07-02

Family

ID=66259311

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/099758 WO2020134100A1 (en) 2018-12-29 2019-08-08 Control method, device and system for vending machine, and vending machine

Country Status (2)

Country Link
CN (1) CN109711359B (en)
WO (1) WO2020134100A1 (en)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109711359B (en) * 2018-12-29 2021-08-03 北京沃东天骏信息技术有限公司 Vending machine control method, vending machine control device and computer-readable storage medium
CN111080896A (en) * 2019-11-30 2020-04-28 嘉兴聚变信息科技有限公司 Anti-prying and anti-theft method for self-service vending machine
CN113837400B (en) * 2020-06-08 2024-04-16 菜鸟智能物流控股有限公司 Fault processing and device for logistic object delivery device
CN114419781A (en) * 2022-01-04 2022-04-29 合肥美的智能科技有限公司 Method for reminding that camera equipment is shielded, vending terminal, storage medium and device
CN116503994B (en) * 2023-06-29 2023-09-26 深圳市乐科智控科技有限公司 Commodity anti-theft detection method, device, equipment and storage medium

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101482063B1 (en) * 2014-05-09 2015-01-13 주식회사 로탬 Apparatus and method for managing vending information of vending machine
CN107680261A (en) * 2017-09-25 2018-02-09 福州米鱼信息科技有限公司 A kind of automatic vending system
CN108648334A (en) * 2018-04-11 2018-10-12 合肥美的智能科技有限公司 Self-service cabinet and its abnormal method for controlling reporting, self-service system
CN109711359A (en) * 2018-12-29 2019-05-03 北京沃东天骏信息技术有限公司 Vending machine control method, device and computer readable storage medium
CN109711360A (en) * 2018-12-29 2019-05-03 北京沃东天骏信息技术有限公司 Vending machine risk control method, device and control system

Family Cites Families (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106530529A (en) * 2016-11-11 2017-03-22 贤石金 Intelligent cash register
CN106781014B (en) * 2017-01-24 2018-05-18 广州市蚁道互联网有限公司 Automatic vending machine and its operation method
CN106920152A (en) * 2017-03-07 2017-07-04 深圳市楼通宝实业有限公司 Self-service vending method and system
CN106934890A (en) * 2017-03-09 2017-07-07 石道松 Automatic selling shops
CN107481414A (en) * 2017-08-21 2017-12-15 文彬 A kind of real-time shopping guide method and system of open unmanned sale apparatus
CN107622436A (en) * 2017-09-06 2018-01-23 上海金荣翔企业发展有限公司 A kind of vending system and good selling method in unmanned shop
CN107730782A (en) * 2017-10-19 2018-02-23 安徽小豆网络科技有限公司 Unmanned convenience store's operation system of the self-service cashier's machine of view-based access control model image recognition
CN107742365A (en) * 2017-12-01 2018-02-27 深圳市迪比科电子科技有限公司 Unmanned intelligent selling method and unmanned intelligent selling system
CN108171172A (en) * 2017-12-27 2018-06-15 惠州Tcl家电集团有限公司 Self-help shopping method, self-service sale device and computer readable storage medium
CN108154601A (en) * 2018-01-09 2018-06-12 合肥美的智能科技有限公司 Automatic vending machine and its control method
CN108335406A (en) * 2018-02-08 2018-07-27 合肥美的智能科技有限公司 Self-service equipment and its good selling method and self-service system
CN108492451A (en) * 2018-03-12 2018-09-04 远瞳（上海）智能技术有限公司 Automatic vending method
CN108416909B (en) * 2018-04-23 2024-05-21 福建自贸试验区厦门片区益客联信息科技有限公司 Intelligent sales counter and intelligent vending method
CN108734862A (en) * 2018-05-22 2018-11-02 孙海 Combined type vending system and its control method
CN108765702A (en) * 2018-05-23 2018-11-06 济南每日优鲜便利购网络科技有限公司 Automatic vending machine
CN108806074B (en) * 2018-06-05 2021-08-03 腾讯科技（深圳）有限公司 Shopping information generation method and device and storage medium
CN108921048A (en) * 2018-06-14 2018-11-30 深圳码隆科技有限公司 A kind of shopping settlement method, device and user terminal
CN108694779B (en) * 2018-07-09 2020-04-03 深圳码隆科技有限公司 Goods returning method and device for open type vending machine

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101482063B1 (en) * 2014-05-09 2015-01-13 주식회사 로탬 Apparatus and method for managing vending information of vending machine
CN107680261A (en) * 2017-09-25 2018-02-09 福州米鱼信息科技有限公司 A kind of automatic vending system
CN108648334A (en) * 2018-04-11 2018-10-12 合肥美的智能科技有限公司 Self-service cabinet and its abnormal method for controlling reporting, self-service system
CN109711359A (en) * 2018-12-29 2019-05-03 北京沃东天骏信息技术有限公司 Vending machine control method, device and computer readable storage medium
CN109711360A (en) * 2018-12-29 2019-05-03 北京沃东天骏信息技术有限公司 Vending machine risk control method, device and control system

Also Published As

Publication number Publication date
CN109711359A (en) 2019-05-03
CN109711359B (en) 2021-08-03

Similar Documents

Publication Publication Date Title
WO2020134100A1 (en) Control method, device and system for vending machine, and vending machine
US11756367B2 (en) Investigation generation in an observation and surveillance system
JP7245321B2 (en) Scan leakage identification method and apparatus, self-service cash register terminal and system
US9158975B2 (en) Video analytics for retail business process monitoring
US10460300B2 (en) Method of preventing fraud and theft during automated teller machine transactions and related system
CN111263224B (en) Video processing method and device and electronic equipment
CN106031165B (en) Method, system and the computer-readable medium of managing video surveillance system
US11710397B2 (en) Theft prediction and tracking system
JP6440327B2 (en) Crime prevention system, crime prevention method, and robot
US11501301B2 (en) Transaction terminal fraud processing
US20180157917A1 (en) Image auditing method and system
EP4075399A1 (en) Information processing system
CN111260685B (en) Video processing method and device and electronic equipment
CN115497169A (en) Warehouse operation behavior identification method, device and equipment
WO2018104999A1 (en) Information processing device, control method, and program
JP6536643B2 (en) INFORMATION PROCESSING APPARATUS, CONTROL METHOD, AND PROGRAM
CN108171286B (en) Unmanned selling method and system
CN109711360B (en) Vending machine risk control method, vending machine risk control device and vending machine risk control system
CN104052975B (en) Shop networking video alarm with passenger flow counting function
US11928942B2 (en) Systems and methods for theft prevention and detection
CN111583521A (en) Safety protection method and device
CN108279946B (en) Method and device for calling seller application
JP6531804B2 (en) INFORMATION PROCESSING APPARATUS, CONTROL METHOD, AND PROGRAM
CN115546703B (en) Risk identification method, device and equipment for self-service cash register and storage medium
CN115457697A (en) Anti-theft method and device for unmanned retail container, equipment and storage medium

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19904423

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19904423

Country of ref document: EP

Kind code of ref document: A1