CN117952507B - Intelligent shopping cart commodity returning identification method and system - Google Patents
Intelligent shopping cart commodity returning identification method and system Download PDFInfo
- Publication number
- CN117952507B CN117952507B CN202410348008.0A CN202410348008A CN117952507B CN 117952507 B CN117952507 B CN 117952507B CN 202410348008 A CN202410348008 A CN 202410348008A CN 117952507 B CN117952507 B CN 117952507B
- Authority
- CN
- China
- Prior art keywords
- commodity
- user
- information
- image
- shopping cart
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 89
- 238000005303 weighing Methods 0.000 claims abstract description 32
- 238000012544 monitoring process Methods 0.000 claims abstract description 11
- 230000008859 change Effects 0.000 claims abstract description 10
- 230000003993 interaction Effects 0.000 claims abstract description 10
- 230000008569 process Effects 0.000 claims description 35
- 230000006399 behavior Effects 0.000 claims description 34
- 238000012545 processing Methods 0.000 claims description 18
- 238000001514 detection method Methods 0.000 claims description 15
- 230000033001 locomotion Effects 0.000 claims description 12
- 238000004364 calculation method Methods 0.000 claims description 10
- 238000013527 convolutional neural network Methods 0.000 claims description 9
- 230000003287 optical effect Effects 0.000 claims description 6
- 238000003708 edge detection Methods 0.000 claims description 3
- 230000011218 segmentation Effects 0.000 claims 1
- 230000009471 action Effects 0.000 abstract description 4
- 238000013473 artificial intelligence Methods 0.000 abstract description 2
- 238000012015 optical character recognition Methods 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 6
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000012790 confirmation Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 230000002265 prevention Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 239000013585 weight reducing agent Substances 0.000 description 1
Landscapes
- Cash Registers Or Receiving Machines (AREA)
Abstract
The invention discloses a commodity returning identification method and system for an intelligent shopping cart, and belongs to the technical fields of Internet of things and artificial intelligence. Aiming at the problem that the intelligent shopping cart in the prior art cannot give consideration to accuracy, safety, convenience and rapidness in the goods returning operation, the invention recognizes the goods taking-out behavior of a user by collecting the image information in the monitoring area of the intelligent shopping cart, recognizes goods returning information by comprehensively comparing recognition marks, characters and image features, accurately judges the goods returning behavior of the user by combining the weight change information of the weighing system, confirms goods returning by the user through the interaction system, and changes the goods returning into goods returning by scanning codes of the user when the recognition fails. Due to the fact that the order of commodity information identification and the information quantity of graphic feature comparison are set, automation of goods returning operation can be achieved, goods returning actions of users and goods purchased in returning can be accurately and conveniently identified, algorithm efficiency is improved, user experience is improved, and risks of goods loss are reduced.
Description
Technical Field
The invention relates to the technical fields of Internet of things and artificial intelligence, in particular to a commodity returning identification method and system of an intelligent shopping cart.
Background
At present, the varieties of various shopping supermarkets are tens of thousands, and the specifications of many commodities are the same, but the prices are different by several yuan or even hundreds of yuan. In the self-help shopping process of using the intelligent shopping cart, if the whole process behavior is uncontrolled, a certain risk is brought to the supermarket, and the whole process monitoring and the identification of the user behavior and commodity information can also cause great increase of cost. Especially, after the commodity is added into the intelligent shopping cart, accurate identification of the return behavior of the user is most critical to supermarket damage prevention. Some intelligent shopping carts provide a scheme that a user can return goods by actively deleting the goods on the shopping list on the screen, which brings loopholes to loss prevention and possibly brings loss to supermarkets. If the goods returned are identified by simple code scanning, the condition that the code scanning is carried out on one type of goods and the goods returned are actually carried out on the other goods may occur.
Chinese patent application CN 106672042A, publication date 2017, month 5 and 17, describes an intelligent shopping cart and a method for determining shopping and returning processes of the shopping cart. In the invention, when a user uses the intelligent shopping cart to make shopping, if the user needs to return goods, the user firstly operates the computer to enter a goods returning program, then takes out goods from the shopping cart, then scans codes to confirm that the goods need to be purchased, and then judges that the weighing change of the intelligent shopping cart accords with the goods returning action, and the user can finish one-time goods returning operation. The judgment method provided by the invention improves the accuracy of identifying the return behavior of the user, but increases the time cost of the user and has poor experience.
Disclosure of Invention
1. Technical problem to be solved
Aiming at the problem that the intelligent shopping cart in the prior art cannot give consideration to accuracy, safety, convenience and quickness in the return operation, the invention provides a method capable of accurately identifying the user to return the purchased goods based on the intelligent shopping cart, which can realize optimization of the return operation, accurately and conveniently identify the user to return the purchased goods, promote user experience and reduce the risk of goods loss.
2. Technical proposal
The aim of the invention is achieved by the following technical scheme.
The intelligent shopping cart is at least provided with a shopping cart body provided with a shopping basket, and the cart body is provided with a processor, a weighing device, a commodity bar code scanner and image acquisition equipment. Wherein a virtual plane is arranged below a basket opening of the shopping basket and is an H1 plane; the weighing device is arranged at the bottom of the shopping basket, the scanning port of the commodity bar code scanner is overlapped with the H1 plane, and the image acquisition equipment is arranged above the H1 plane and used for monitoring the H1 plane and a moving object in a space above the H1 plane.
The first aspect of the invention provides an intelligent shopping cart return commodity identification method, which comprises the following steps:
And acquiring an intelligent shopping cart return commodity image. An identification mark such as a bar code or a two-dimensional code in the return commodity image is extracted. Preferably, the bar code detection algorithm used in extracting the identification mark is ZBar, EAST, OCR, or ORB technology. If at least one identification mark is extracted, setting the comprehensive similarity score Z to be 1, returning the commodity information and the Z value corresponding to the identification mark, and ending the returned commodity identification flow.
If the identification mark is not extracted, characters in the extracted image are compared with names of commercial products purchased by the user by using a cosine similarity calculation method, so that character comparison similarity T is obtained, and the value range of T is 0-1; and extracting deep features of the returned commodity image and the purchased commodity image by using a pre-trained convolutional neural network, and calculating the graph feature comparison similarity P, wherein the value range of P is 0-1.
Further, the method for extracting the characters in the image comprises the following steps: the text regions in the image are found by text detector EAST, and text content is extracted for each text region detected using TESSERACT as an OCR (optical character recognition) tool.
Further, the method for calculating the graphic feature comparison similarity P comprises the following steps: extracting the graphic features of returned goods by using a pretrained convolutional neural network, and extracting the deep features of goods images by using mainstream machine learning technologies such as VGG, resNet and the like optionally and not limited to; extracting features from the purchased commodity images by using the same convolutional neural network; and calculating the distance between the returned commodity image and the characteristic of each image of the purchased commodity by adopting the technical scheme of combination of the change detection, the convolution neural network and the Euclidean distance, and normalizing the distance value to obtain the distance L, wherein the image similarity P=1-L.
And obtaining a comprehensive similarity score Z according to the text comparison similarity T and the graphic feature comparison similarity P.
（1）
The character weight is V, and the graphic feature weight is U; the values of U and V are in the range of 0,1, and U+V=1; preferably, V takes a value of 0.4 and U takes a value of 0.6.
And returning commodity information corresponding to the highest Z value in the purchased commodities and the Z value, and ending the commodity returning identification process.
The second aspect of the invention provides a method for judging the commodity taking-out behavior of an intelligent shopping cart, which comprises the following steps:
The image acquisition equipment continuously monitors the H1 plane and acquires continuous image information; processing the image information, and judging whether the user has the action of continuously and stably moving the handheld object upwards from the position below the H1 plane; if yes, starting the weighing system to start weighing, wherein the weighing lasts for A seconds. If the weighing system continuously monitors A seconds to confirm weight reduction after the user holds the article to move upwards from the position below the H1 plane, the user is judged to take out the commodity from the shopping cart. By using the intelligent shopping cart return commodity identification method according to the first aspect of the invention, commodities taken out by a user are identified.
Preferably, MEDIAPIPE hand key point detection is combined with a Lucas-Kanade (L-K) optical flow algorithm to judge whether a user has a continuous and stable upward movement behavior of a handheld object from below an H1 plane, and the steps are as follows:
and processing the image by using the hand key point detection model to acquire the key points of the hand. Preferably, a MEDIAPIPE hand key point detection model is used to process the video frame and obtain the key points of the hand.
And determining a candidate area containing the handheld object according to the hand key points.
Items are detected within the candidate region using edge detection or color segmented image processing techniques.
If the object is detected, selecting edges or other obvious features on the object as object feature points, and simultaneously selecting key points of the hand as hand feature points; if no object is detected, only the key points of the hand are selected as the characteristic points.
For each pair of continuous images, calculating the optical flow of the object characteristic points and the hand characteristic points by using an L-K method; analyzing the component of the object feature point light flow vector and the ground vertical direction, and if more than 80% of object feature points move upwards along the ground vertical direction, indicating that objects move upwards in the continuous images; analyzing the initial position of the movement of the characteristic points of the article, and judging whether the movement of the article starts below the H1 plane; if the object starts to move from below the H1 plane and moves upwards along the vertical direction of the ground, the hand can be further confirmed to follow the object to move upwards in the pair of continuous images;
And analyzing the object and hand movement modes in all continuous images, and if more than 70% of judgment results are that the hand moves upwards along with the object, confirming that the handheld object continuously and stably moves upwards from the position below the H1 plane in the video.
The third aspect of the invention provides a method for identifying return behavior of an intelligent shopping cart, which specifically comprises the following steps:
pre-starting purchase withdrawal flow: the information acquisition module continuously acquires H1 plane image information, and when the fact that the handheld article of the user moves upwards from the position below the H1 plane is detected, the user is prejudged that the purchased commodity is possibly taken out, and the purchase return process is prestarted.
And (3) collecting information of returned goods: and the image acquisition unit, such as a camera, sends the acquired image to the commodity information classification module in real time, and stops photographing when the commodity is monitored to leave above the H1 plane.
Identifying returned goods: the commodity information classification module applies the intelligent shopping cart commodity return identification method provided by the first aspect of the invention to carry out commodity return identification on the shot image and report the matching result to the shopping decision service module; and in the commodity returning identification process, if the commodity identification mark in the shot image is extracted, informing the camera to stop shooting.
Monitoring the state of the electronic scale: when the weighing unit of the information acquisition module monitors that the electronic scale is reduced by one stable weight value, the user is judged to take out the commodity, the purchase return process is confirmed to be started, and the weight recognition result is given to the shopping decision module.
The shopping decision service module synthesizes the weight and the image recognition result to decide commodity information of purchase return: if the weight is consistent with the result of the image, directly giving an image identification result, and entering a purchase return process; otherwise, prompting the user to sweep the code to purchase and enter a purchase-withdrawal process.
The method for judging the consistency of the weight and the result of the image comprises the following steps: weight matching degree calculation is carried out on the image recognition result, if the weight matching degree X is larger than or equal to the weight matching degree boundary value M, commodity information recognized by the image is screened, and the purchasing process is carried out; otherwise, the identified commodity information is not given, the user is prompted to sweep the code to purchase and enter a purchase-withdrawal process.
The specific calculation formula of the weight matching degree X is as follows:
（2）
the weight matching degree boundary value M is in the range of 0-1. Preferably, M is set to 0.8.
In the present invention, a single operation process allows only one piece (a loose item, a package, or the like) to be removed.
Preferably, the steps of pre-starting the purchase return process, acquiring information of returned goods and identifying returned goods apply the intelligent shopping cart goods taking-out behavior judging method provided by the second aspect of the invention.
In a fourth aspect of the present invention, an intelligent shopping cart system is provided, and an intelligent shopping cart return behavior recognition method of the third aspect of the present invention is performed.
Shopping cart body. The intelligent shopping cart comprises a shopping cart body provided with a shopping basket, wherein the cart body is provided with a processor, a weighing device, a commodity bar code scanner and image acquisition equipment, and a virtual plane which is an H1 plane is arranged below a basket opening of the shopping basket. The weighing device is arranged at the bottom of the shopping basket, the scanning port of the commodity bar code scanner is overlapped with the H1 plane, and the image acquisition equipment is arranged above the H1 plane and used for monitoring the H1 plane and moving objects in the space above the H1 plane.
The information acquisition module comprises a weighing unit and an image acquisition unit; the weighing unit is used for collecting commodity weight information; the image acquisition unit is used for acquiring video information and commodity image information in the monitoring area;
The commodity information classification module: the image acquisition unit is responsible for carrying out identification and classification processing on the images and the image information acquired by the image acquisition unit, identifying commodity information in the images and sending an identification result to the shopping decision service module;
User behavior recognition module: the image acquisition unit is responsible for carrying out user behavior judgment on the image acquired by the image acquisition unit and sending the identification result to the shopping decision service module;
The shopping decision service module is responsible for comprehensively judging whether a return behavior occurs according to commodity weight information acquired by the weighing unit, user behaviors identified by the user behavior identification module and commodity information identified by the commodity information classification module, feeding back a specific return commodity identification result list to the user information interaction module, receiving input information of a user returned by the user information interaction module, and selecting a mode for ending a return purchase flow according to the information;
And a user information interaction module: the method is responsible for feeding back the change condition of order information to the user, and asking the user to confirm whether to return goods; and receiving input information of the user, and feeding the input information of the user back to the shopping decision service module.
3. Advantageous effects
Compared with the prior art, the invention has the advantages that:
(1) In the conventional goods returning scheme designed by the invention, the customer only needs to directly take out goods and make goods returning confirmation on the intelligent shopping cart, and the code scanning is not needed, so that one operation link is reduced, and the user experience is improved
(2) In the method for identifying the returned goods, the accuracy of comparison is improved through the multidimensional identification judgment of the identification mark, the characters and the graphics; meanwhile, the judgment sequence of the recognition mark priority, the character recognition and the comprehensive similarity scoring of the graphic features is set, so that the judgment efficiency is improved to the greatest extent. Meanwhile, in the text and graphic similarity comprehensive scoring link, the information used for comparison is limited to the purchased goods of the user, so that the comparison information database size is greatly reduced, the execution efficiency of the algorithm is improved to the greatest extent, and the execution time of the algorithm is shortened;
(3) The text and graphics recognition technology module and the like which are preferably combined and applied in the invention are all current mature lightweight programs, are suitable for the actual performance of the intelligent shopping cart vehicle-mounted processor, and have low use cost.
Drawings
FIG. 1 is a schematic diagram of an intelligent shopping cart;
FIG. 2 is a flow chart of a method for determining the behavior of an intelligent shopping cart to take out goods;
FIG. 3 is a flow chart diagram of a method for identifying return behavior of an intelligent shopping cart.
Detailed Description
The invention will now be described in detail with reference to the drawings and the accompanying specific examples. As shown in FIG. 1, the intelligent shopping cart is provided with at least one shopping cart body provided with a shopping basket, and the cart body is provided with a processor, a weighing device, a commodity bar code scanner and image acquisition equipment. Wherein a virtual plane is arranged below a basket opening of the shopping basket and is an H1 plane; the weighing device is arranged at the bottom of the shopping basket, the scanning port of the commodity bar code scanner is overlapped with the H1 plane, and the image acquisition equipment is arranged above the H1 plane and used for monitoring the H1 plane and a moving object in a space above the H1 plane.
The process of purchasing goods by a user using an intelligent shopping cart is generally as follows: the user firstly scans the commodity by the commodity bar code scanner. Once the merchandise is scanned, the camera and electronic scale are immediately activated. When the user places merchandise in the shopping basket, the camera captures this action and records the relevant video. Meanwhile, the weight change monitored by the electronic scale can be recorded. All of this information (video recordings and weight changes) is sent in real time to the shopping decision service for processing. The processed information is directly displayed on the computer of the shopping cart so as to be convenient for the user to confirm. The goods returning judgment is usually carried out by combining the goods scanning code and the weighing weight change information, and the operation is not intelligent and efficient enough.
Example 1
The first aspect of the invention provides an intelligent shopping cart return commodity identification method, which comprises the following steps:
And acquiring an intelligent shopping cart return commodity image. The method can collect the video information of the goods returned from the intelligent shopping cart, extract the image with the largest goods exposing area in each frame of image in the video as a key frame, and take the key frame as the information of the goods returned from the intelligent shopping cart. The serial images can be obtained by taking photos at specific intervals, the images meeting the conditions are selected as returned commodity images, the interval shooting time is preferably 50ms, and the resource requirements of transmission and processing are reduced while the identification requirements are met.
An identification mark such as a bar code or a two-dimensional code in the return commodity image is extracted. The removal of the item information can be determined one hundred percent by the identification mark. Preferably, the bar code detection algorithm used in extracting the identification mark is ZBar, EAST, OCR, or ORB technology. The ZBar technology can detect and decode bar codes in various formats, and the bar code comparison is a decoding result of directly comparing the bar codes. If at least one identification mark is extracted, setting the comprehensive similarity score Z to be 1, returning the commodity information and the Z value corresponding to the identification mark, and ending the returned commodity identification flow.
If the identification mark is not extracted, characters in the extracted image are compared with the names of the commodities purchased by the user, so that character comparison similarity T is obtained, and the value range of T is 0-1; and extracting deep features of the returned commodity image and the purchased commodity image by using a pre-trained convolutional neural network, and calculating the graph feature comparison similarity P, wherein the value range of P is 0-1.
Further, the method for extracting the characters in the image comprises the following steps: the text regions in the image are found by a text detector, and text content is extracted for each text region detected using an OCR (optical character recognition) tool. Preferably, the text detector is EAST and the OCR tool is TESSERACT. Preferably, the text comparison similarity calculation uses a cosine similarity calculation method.
Further, the method for calculating the graphic feature comparison similarity P comprises the following steps: extracting the graphic features of returned goods by using a pretrained convolutional neural network, and extracting the deep features of goods images by using mainstream machine learning technologies such as VGG, resNet and the like optionally and not limited to; the same convolutional neural network is used to extract features for the purchased commodity images. And calculating the distance between the returned commodity image and the characteristic of each image of the purchased commodity by adopting the technical scheme of combination of the change detection, the convolution neural network and the Euclidean distance, and normalizing the distance value to obtain the distance L, wherein the image similarity P=1-L. The smaller the distance, the higher the similarity of the images.
Obtaining a comprehensive similarity score Z according to the text comparison similarity T and the graphic feature comparison similarity P:
（1）
The character weight is V, and the graphic feature weight is U; the values of U and V are in the range of 0,1, and U+V=1; preferably, V takes a value of 0.4 and U takes a value of 0.6. The higher Z represents the higher the overall similarity of character recognition and graphic features.
And returning commodity information corresponding to the highest Z value in the purchased commodities and the Z value, and ending the commodity returning identification process. The commodity corresponding to the highest Z value is the returned commodity.
The number of supermarket commodities is very large, generally over ten thousand, and the user hopes that the feedback time of the system is in the millisecond level. In addition, in actual shopping in a supermarket, the gesture of taking out the commodity is not standard, and the shot taken-out commodity graph is incomplete, and the weight of the commodity is inaccurate due to basket touching. The commodity purchased by the user is used as a comparison identification object, so that the comparison magnitude is greatly reduced, and the calculated amount is reduced; meanwhile, multidimensional identification judgment is adopted, and the information of returned goods can be confirmed in the identification mark extraction ring, namely, the identification process is immediately finished, so that the identification efficiency is improved; if the identification mark is not extracted, the information of the text and the image is integrated, and the identification accuracy is improved.
Example 2
In a second aspect of the invention, a method for determining the behavior of an intelligent shopping cart to take out goods is provided. As shown in fig. 2, the method for judging the commodity taking-out behavior of the intelligent shopping cart specifically comprises the following steps:
the image acquisition device continuously monitors the H1 plane and acquires continuous image information. The image information collected here may be obtained by selecting key frames in the video stream to form a series of continuous images, or by taking pictures of the series at specific intervals.
Image information is processed, and MEDIAPIPE hand key point detection is combined with a Lucas-Kanade (L-K) optical flow algorithm to judge whether a user has the continuous, stable and upward movement of a handheld object from below an H1 plane. The algorithm is a two-frame differential optical flow estimation algorithm, and other algorithms capable of realizing similar functions can be applied to achieve the same technical effect. The method comprises the following steps:
and processing the continuous images by using the hand key point detection model to acquire key points of the hand. Preferably, the image is processed using a MEDIAPIPE hand keypoint detection model to obtain the keypoints of the hand.
Determining a candidate area containing the handheld object according to the hand key points; this area may contain a hand-held item, which may be located between the center of the palm and the fingertips in general.
Items are detected within the candidate region using edge detection or color segmented image processing techniques. If the object is detected, selecting edges or other obvious features on the object as object feature points, and simultaneously selecting key points of the hand as hand feature points; if no object is detected, only the key points of the hand are selected as the characteristic points. The condition that the commodity is not detected may be that the user does not take out the commodity, or that the commodity is blocked.
For each pair of successive images, the optical flow of the object feature points and hand feature points is calculated using the L-K method. Analyzing the component of the object feature point light flow vector and the ground vertical direction, and if more than 80% of object feature points move upwards along the ground vertical direction, indicating that objects move upwards in the continuous images; analyzing the initial position of the movement of the characteristic points of the article, and judging whether the movement of the article starts below the H1 plane; if the object starts to move from below the H1 plane and moves upward in the direction perpendicular to the ground, it can be further confirmed that the hand follows the object upward in the pair of continuous images.
And analyzing the object and hand movement modes in all the continuous images, and if more than 70% of judgment results are that the hand moves upwards along with the object, confirming that the handheld object continuously and stably moves upwards from the position below the H1 plane in the series of images.
The weighing system is started to start weighing, and the weighing lasts for A seconds. In the present embodiment, a is set to 3. If the weighing system continuously monitors the hand-held articles for 3 seconds after the hand-held articles move upwards from the position below the H1 plane to confirm that the weight is reduced, the user is judged to take out the articles from the shopping cart.
By using the intelligent shopping cart return commodity identification method according to the first aspect of the invention, commodities taken out by a user are identified.
Example 3
In a third aspect of the present invention, we provide a method for identifying return behavior of an intelligent shopping cart, as shown in fig. 3, specifically including the following steps:
Pre-starting purchase withdrawal flow: the information acquisition module continuously acquires the H1 plane image information, and by the user commodity taking-out behavior determination method provided in embodiment 2, when the hand key point detection and the motion detection detect that the user holds the commodity to move upwards from below the H1 plane, the user is prejudged that the user may take out the purchased commodity, and the purchase return process is prestarted.
And (3) collecting information of returned goods: and the image acquisition unit is used for shooting at specific intervals, sending the shot images to the image processing service in real time, and stopping shooting when the commodity is monitored to leave above the H1 plane. Preferably, the cameras are shot at equal intervals of 50 ms.
Identifying returned goods: the image processing module adopts the intelligent shopping cart goods returned identification method provided in the embodiment 1 to identify goods returned from the shot image, and reports the matching result to the shopping decision module. And in the commodity returning identification process, if the commodity identification mark in the shot image is extracted, informing the camera to stop shooting.
Monitoring the state of the electronic scale: when the electronic scale is monitored to reduce a stable weight value, the user is judged to take out the commodity, the purchase return process is confirmed to be started, and the weight identification result is given to the shopping decision module.
And the shopping decision module synthesizes the weight and the image result to decide commodity information of the purchase return. Specifically, if the weight is consistent with the result of the image, directly giving an image recognition result, and entering a purchase return process; otherwise, prompting the user to sweep the code to purchase and enter a purchase-withdrawal process.
The method for judging the consistency of the weight and the result of the image comprises the following steps: weight matching degree calculation is carried out on the image recognition result, if the weight matching degree X is larger than or equal to the weight matching degree boundary value M, commodity information recognized by the image is screened, and the purchasing process is carried out; otherwise, the identified commodity information is not given, the user is prompted to sweep the code to purchase and enter a purchase-withdrawal process.
The specific calculation formula of the weight matching degree X is as follows:
（2）
the weight matching degree boundary value M is in a value range of 0-1, and can be dynamically adjusted according to actual shopping scenes. In this embodiment, M is set to 0.8.
And (3) purchasing process: the shopping decision module feeds back the purchase return commodity information and waits for the user to confirm whether to purchase return.
Preferably, the shopping cart asks the user whether to purchase the commodity or not through the computer screen, and the user can also feed back and receive confirmation information of the user through the mobile phone APP or a short message, a micro-message public number, a micro-message applet, a sound and the like. Ending the purchase return flow: and acquiring explicit feedback of whether the user buys the shopping back or not and judging the user. And judging after clicking feedback of the button: if the user confirms the purchase return, the shopping decision service performs the purchase return determination operation and ends the purchase return flow. Otherwise, if the user feedback identification commodity result is wrong, starting a code scanning purchase return process, waiting for the commodity bar code scanner to report an identification mark, and determining the purchase return commodity; otherwise, if the user does not confirm the purchase return, waiting for the user to put back the commodity again, and not allowing the user to perform other operations in the period.
In this embodiment, a single operation allows only one piece (a loose item, a package, or the like) to be removed.
Example 4
In a fourth aspect of the present invention, an intelligent shopping cart system is provided, which implements the intelligent shopping cart commodity return behavior identification method of the third aspect of the present invention. Comprising the following steps:
The information acquisition module comprises a weighing unit and an image acquisition unit; the weighing unit is used for collecting commodity weight information; the image acquisition unit is responsible for acquiring image information in a monitoring area, and can acquire video information and take pictures.
The commodity information classification module: and the image acquisition unit is responsible for carrying out identification and classification processing on the images acquired by the image acquisition unit, identifying commodity information in the images and sending an identification result to the shopping decision service module.
User behavior recognition module: and the image acquisition unit is responsible for judging the user behavior of the image acquired by the image acquisition unit and sending the identification result to the shopping decision service module.
The shopping decision service module is responsible for comprehensively judging whether a return behavior occurs according to commodity weight information acquired by the weighing unit, user behaviors identified by the user behavior identification module and commodity information identified by the commodity information classification module, feeding back a specific return commodity identification result list to the user information interaction module, receiving input information of a user returned by the user information interaction module, and selecting a mode for ending a return purchase flow according to the information;
And a user information interaction module: the method is responsible for feeding back the change condition of order information to the user, and asking the user to confirm whether to return goods; and receiving input information of the user, and feeding the input information of the user back to the shopping decision service module.
Preferably, the shopping decision module and the image processing module run in a processor provided on the shopping cart body.
Preferably, in some embodiments of the invention, the processor on the shopping cart runs an android App.
Example 5
As a specific implementation of embodiment 4, the present invention provides an intelligent shopping cart system, including: the intelligent shopping cart, the edge computer and the service software are connected with the back-end server cluster and the service software through a wired or wireless network. Specific:
The intelligent shopping cart is provided with at least one shopping cart body provided with a shopping basket, and the cart body is provided with a processor, a weighing device, a commodity bar code scanner and an image acquisition device. Preferably, a processor arranged on the shopping cart body is responsible for executing a shopping application program, an AI image processing system and a shopping decision making program system. The operating program run by the shopping cart processor can be developed based on android, hong Mongolian, IOS and other operating systems.
And the edge computer and the service software are used for managing data acquisition, data processing, algorithm models and the like. Deployment is within the business segment.
The back-end server cluster and the service software are used for managing all other shopping services which are not supported by the edge computer and the service software and all shopping services which are not operated by the edge computer and the service software, and can be deployed in a business super or centralized in a group headquarter.
The intelligent shopping cart, the edge computer and the service software and the back-end server cluster and the service software can be in communication connection through various networking modes such as local area network, wide area network, wired mode, wireless mode and the like. Including but not limited to WIFI, ZIGBEE, bluetooth, etc.
The foregoing has been described schematically the invention and embodiments thereof, which are not limiting, but are capable of other specific forms of implementing the invention without departing from its spirit or essential characteristics. The drawings are also intended to depict only one embodiment of the invention, and therefore the actual construction is not intended to limit the claims, any reference number in the claims not being intended to limit the claims. Therefore, if one of ordinary skill in the art is informed by this disclosure, a structural manner and an embodiment similar to the technical scheme are not creatively designed without departing from the gist of the present invention, and all the structural manners and the embodiment are considered to be within the protection scope of the present patent. In addition, the word "comprising" does not exclude other elements or steps, and the word "a" or "an" preceding an element does not exclude the inclusion of a plurality of such elements. The various elements recited in the product claims may also be embodied in software or hardware. The terms first, second, etc. are used to denote a name, but not any particular order.
Claims (9)
1. The intelligent shopping cart return commodity identifying method includes the following steps:
the image acquisition equipment continuously monitors the H1 plane and acquires continuous image information;
Processing the image information by using a hand key point detection model to obtain key points of the hand;
determining a candidate area containing the handheld object according to the hand key points;
detecting an item within the candidate region using edge detection or color segmentation image processing techniques;
if the object is detected, selecting edges or other obvious features on the object as object feature points, and simultaneously selecting key points of the hand as hand feature points;
if no article is detected, only selecting the key points of the hand as the characteristic points;
for each pair of continuous images, calculating the optical flow of the object characteristic points and the hand characteristic points by using an L-K method; analyzing the component of the object feature point light flow vector and the ground vertical direction, and if more than 80% of object feature points move upwards along the ground vertical direction, indicating that objects move upwards in the continuous images; analyzing the initial position of the movement of the characteristic points of the article, and judging whether the movement of the article starts below the H1 plane; if the object starts to move from below the H1 plane and moves upwards along the vertical direction of the ground, the hand can be further confirmed to follow the object to move upwards in the pair of continuous images;
analyzing the motion modes of the articles and the hands in all continuous images, if more than 70% of judgment results are that the hands move upwards along with the articles, confirming that the hand-held articles in the video continuously and stably move upwards from the position below an H1 plane, starting a weighing system to start weighing, and weighing lasts for A seconds;
If the weight of the object held by the user is confirmed to be reduced after the object is upwards moved from the position below the H1 plane and the weighing system continuously monitors the time A seconds, the user is confirmed to take out the commodity from the shopping cart;
collecting an intelligent shopping cart return commodity image;
extracting identification marks in the returned commodity images; if at least one identification mark is extracted, setting the comprehensive similarity score Z as 1, returning the commodity information and the Z value corresponding to the identification mark, and ending the returned commodity identification flow;
If the identification mark is not extracted, characters in the extracted image are compared with the names of the commodities purchased by the user, and a cosine similarity calculation method is used for calculating the character comparison similarity T, wherein the value range of the T is 0-1; extracting deep features of returned commodity images and purchased commodity images by using a pre-trained convolutional neural network, and calculating a graph feature comparison similarity P, wherein the value range of P is 0-1;
Obtaining a comprehensive similarity score Z according to the text comparison similarity T and the graphic feature comparison similarity P:
（1）
the character weight is V, and the graphic feature weight is U; the values of U and V are in the range of 0,1, and U+V=1;
and returning commodity information corresponding to the highest Z value in the purchased commodities and the Z value, and ending the commodity returning identification process.
2. The method for identifying return goods of intelligent shopping cart according to claim 1, wherein the identification mark is a bar code or a two-dimensional code.
3. The method for identifying return goods of intelligent shopping cart as claimed in claim 1, wherein U is 0.6 and V is 0.4.
4. The method for identifying return goods of intelligent shopping cart according to claim 1, wherein the detection algorithm used in extracting the identification mark is ZBar, EAST, OCR or ORB.
5. The method for identifying return goods of intelligent shopping cart according to claim 1, wherein the method for calculating the comparison similarity of graphic features P is as follows: and calculating and normalizing the distance between the returned commodity image and the characteristics of each purchased commodity image to obtain a distance L, wherein the image similarity P=1-L.
6. The intelligent shopping cart return goods identification method according to claim 5, wherein the pretrained convolutional neural network is VGG or ResNet, and the algorithm for calculating the distance between the return goods image and the feature of each goods image purchased additionally is a change detection algorithm, a convolutional neural network algorithm or a euclidean distance algorithm.
7. The intelligent shopping cart return behavior recognition method comprises the following steps:
pre-starting purchase withdrawal flow: the information acquisition module continuously acquires H1 plane video information, and when detecting that a user holds an article upwards from below an H1 plane, prejudges that the user possibly takes out the purchased article and prestarts a purchase return process;
And (3) collecting information of returned goods: shooting at specific intervals by the camera, giving the shot images to an image processing service in real time, and stopping shooting when the commodity is monitored to leave above the H1 plane;
Identifying returned goods: applying the intelligent shopping cart return commodity identification method according to any one of claims 1-6, carrying out return commodity identification on the shot image, and reporting the matching result to a shopping decision service; in the commodity returning identification process, if the commodity bar code in the shot image is extracted, the camera is informed to stop shooting;
Monitoring the state of the electronic scale: when the electronic scale is monitored to reduce a stable weight value, judging that the user takes out the commodity, confirming starting a purchase return process, and giving a weight identification result to a shopping decision service;
the shopping decision service module synthesizes the weight and the image result to decide commodity information of purchase return: if the weight is consistent with the result of the image, directly giving an image identification result, and entering a purchase return process; otherwise, prompting the user to sweep the code to purchase and enter a purchase-withdrawal process.
8. The intelligent shopping cart return behavior recognition method of claim 7, wherein the method for judging that the weight is consistent with the result of the image is as follows: weight matching degree calculation is carried out on the image recognition result, if the weight matching degree X is larger than or equal to the weight matching degree boundary value M, commodity information recognized by the image is screened, and the purchasing process is carried out; otherwise, the identified commodity information is not given, the user is prompted to sweep the code to purchase and enter a purchase-withdrawal process;
The specific calculation formula of the weight matching degree X is as follows:
（2）
The weight matching degree boundary value M is in the range of 0-1.
9. An intelligent shopping cart system for performing the intelligent shopping cart return behavior identification method of any one of claims 7 or 8, comprising:
A shopping cart body;
the information acquisition module comprises a weighing unit and an image acquisition unit; the weighing unit is used for collecting commodity weight information; the image acquisition unit is used for acquiring video information and commodity image information in the monitoring area;
The commodity information classification module: the image acquisition unit is responsible for carrying out identification and classification processing on the images and the image information acquired by the image acquisition unit, identifying commodity information in the images and sending an identification result to the shopping decision service module;
User behavior recognition module: the image acquisition unit is responsible for carrying out user behavior judgment on the image acquired by the image acquisition unit and sending the identification result to the shopping decision service module;
The shopping decision service module is responsible for comprehensively judging whether a return behavior occurs according to commodity weight information acquired by the weighing unit, user behaviors identified by the user behavior identification module and commodity information identified by the commodity information classification module, feeding back a specific return commodity identification result list to the user information interaction module, receiving input information of a user returned by the user information interaction module, and selecting a mode for ending a return purchase flow according to the information;
And a user information interaction module: the method is responsible for feeding back the change condition of order information to the user, and asking the user to confirm whether to return goods; and receiving input information of the user, and feeding the input information of the user back to the shopping decision service module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410348008.0A CN117952507B (en) | 2024-03-26 | 2024-03-26 | Intelligent shopping cart commodity returning identification method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410348008.0A CN117952507B (en) | 2024-03-26 | 2024-03-26 | Intelligent shopping cart commodity returning identification method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117952507A CN117952507A (en) | 2024-04-30 |
| CN117952507B true CN117952507B (en) | 2024-06-18 |
Family
ID=90805243
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410348008.0A Active CN117952507B (en) | 2024-03-26 | 2024-03-26 | Intelligent shopping cart commodity returning identification method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117952507B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104331151A (en) * | 2014-10-11 | 2015-02-04 | 中国传媒大学 | Optical flow-based gesture motion direction recognition method |
| CN106672042A (en) * | 2016-11-16 | 2017-05-17 | 南京亿猫信息技术有限公司 | Intelligent shopping cart and shopping and sales return process judging method of shopping cart |
| CN110245594A (en) * | 2019-06-04 | 2019-09-17 | 天津大学 | A kind of commodity recognition method for cash register system |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104143105A (en) * | 2013-09-22 | 2014-11-12 | 腾讯科技（深圳）有限公司 | Graph recognition method, device and system and terminal device |
| CN104615721B (en) * | 2015-02-06 | 2018-02-16 | 北京京东尚科信息技术有限公司 | For the method and system based on return of goods related information Recommendations |
| CN109785543A (en) * | 2017-11-14 | 2019-05-21 | 秦皇岛中秦智能装备有限公司 | Intelligent barcode scanning identifies shopping cart |
| CN109166257B (en) * | 2018-08-23 | 2021-02-19 | 深圳码隆科技有限公司 | Shopping cart commodity verification method and device thereof |
| CN114331535A (en) * | 2021-12-29 | 2022-04-12 | 南京亿猫信息技术有限公司 | Information issuing system and method based on intelligent shopping cart |
| CN116109992A (en) * | 2023-01-16 | 2023-05-12 | 山东大学 | Goods shelf commodity identification method and system |
| CN117609583A (en) * | 2023-07-20 | 2024-02-27 | 浙江工业大学 | Customs import and export commodity classification method based on image text combination retrieval |
-
2024
- 2024-03-26 CN CN202410348008.0A patent/CN117952507B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104331151A (en) * | 2014-10-11 | 2015-02-04 | 中国传媒大学 | Optical flow-based gesture motion direction recognition method |
| CN106672042A (en) * | 2016-11-16 | 2017-05-17 | 南京亿猫信息技术有限公司 | Intelligent shopping cart and shopping and sales return process judging method of shopping cart |
| CN110245594A (en) * | 2019-06-04 | 2019-09-17 | 天津大学 | A kind of commodity recognition method for cash register system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117952507A (en) | 2024-04-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11861584B2 (en) | Self-service settlement method, apparatus and storage medium | |
| JP6869345B2 (en) | Order information determination method and equipment | |
| CN207742721U (en) | A kind of intelligent article retracting device | |
| CN110033293B (en) | Method, device and system for acquiring user information | |
| CN110866429A (en) | Missed scanning identification method and device, self-service cash register terminal and system | |
| EP3882877A1 (en) | Checkout flows for autonomous stores | |
| JP2019174959A (en) | Commodity shelf position registration program and information processing apparatus | |
| James et al. | Automated checkout for stores: A computer vision approach | |
| US20210168135A1 (en) | Linking a physical item to a virtual item | |
| CN117952507B (en) | Intelligent shopping cart commodity returning identification method and system | |
| US11568564B2 (en) | Mapping multiple views to an identity | |
| JP2011150425A (en) | Research device and research method | |
| CN114360057A (en) | Data processing method and related device | |
| EP4383171A1 (en) | Information processing program, information processing method, and information processing apparatus | |
| CN118097519B (en) | Intelligent shopping cart shopping behavior analysis method and system based on commodity track analysis | |
| US20240193995A1 (en) | Non-transitory computer-readable recording medium, information processing method, and information processing apparatus | |
| US20240211920A1 (en) | Storage medium, alert generation method, and information processing apparatus | |
| US20230093938A1 (en) | Non-transitory computer-readable recording medium, information processing method, and information processing apparatus | |
| US20240212355A1 (en) | Storage medium, alert generation method, and information processing device | |
| EP4383167A1 (en) | Information processing program, information processing method, and information processing apparatus | |
| US20240212321A1 (en) | Storage medium, data generation method, and information processing device | |
| CN112183691B (en) | Commodity display method, device and storage medium | |
| US20240212320A1 (en) | Storage medium, specifying method, and information processing device | |
| EP4390884A1 (en) | Alert generation program, alert generation method, and information processing apparatus | |
| WO2023148856A1 (en) | Purchase analysis device, purchase analysis method, and non-transitory computer-readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |