CN109558775A - A kind of refrigerator food management method - Google Patents
A kind of refrigerator food management method Download PDFInfo
- Publication number
- CN109558775A CN109558775A CN201710888847.1A CN201710888847A CN109558775A CN 109558775 A CN109558775 A CN 109558775A CN 201710888847 A CN201710888847 A CN 201710888847A CN 109558775 A CN109558775 A CN 109558775A
- Authority
- CN
- China
- Prior art keywords
- refrigerator
- image
- food materials
- empty
- handed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/68—Food, e.g. fruit or vegetables
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Biology (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- Human Computer Interaction (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the invention discloses a kind of refrigerator food management methods, this method comprises: acquisition refrigerator inside image；Detect the hand gestures and hand exercise situation on refrigerator inside image；The typing situation of food materials in refrigerator is judged according to hand gestures and hand exercise situation.Scheme through the embodiment of the present invention eliminates position error, improves discrimination.
Description
Technical field
The present embodiments relate to food materials to store equipment control technology, espespecially a kind of refrigerator food management method.
Background technique
The identification of refrigerator inside food materials is substantially all made of static knowledge otherwise at present, that is, acquires a picture, pass through knowledge
Other algorithm identifies the food materials in picture.This method unavoidably will appear food materials stacking and block, and food materials are largely hidden
Gear, will be unable to accurately identify.In an other sub-picture, food materials are positioned and are classified (i.e. target detection), not only there is positioning
Error, there is also error in classification, two are superimposed, and overall recognition accuracy is not high.
Summary of the invention
The embodiment of the invention provides a kind of refrigerator food management methods, can eliminate position error, improve discrimination.
In order to solve the above technical problems, the embodiment of the present invention adopts the following technical scheme that
A kind of refrigerator food management method, this method comprises:
Acquire refrigerator inside image；
Detect the hand gestures and hand exercise situation on refrigerator inside image；
The typing situation of food materials in refrigerator is judged according to hand gestures and hand exercise situation.
Optionally, acquisition refrigerator inside image includes:
The video image at refrigerator inside and/or chamber door entrance area is shot by camera preset in refrigerator.
Optionally, the hand gestures on detection refrigerator inside image include:
The target image for hand gestures detection is obtained from video image；
Target image is inputted into trained hand gestures detection model in advance；
The image in target image is identified by hand gestures detection model；
Hand gestures are exported according to recognition result.
Optionally, it is obtained from video image and includes: for the target image of hand gestures detection
A, the i-th frame image in the video image of shooting, and the background image compared as image are obtained；I is positive integer,
The initial value of i is 1；
B, the i-th frame image is compared with the i+1 frame image of shooting；
C, when the difference of the neighboring mean value of co-located region on the i-th frame image and i+1 frame image is less than preset the
When one threshold value, determine that the i-th frame image is identical as i+1 frame image, and return step A after i is added 1；When the i-th frame image and i-th+
When the difference of the neighboring mean value of co-located region is greater than or equal to preset first threshold on 1 frame image, the i-th frame image is determined
It is different from i+1 frame image, and enter step D；
D, using i+1 frame image as target image.
Optionally, hand gestures include: empty-handed, non-empty-handed and without hands；
Hand exercise situation includes: empty-handed immigration refrigerator, non-empty-handed removal refrigerator, non-empty-handed immigration refrigerator and empty-handed removal
Refrigerator.
Optionally, the hand exercise situation on detection refrigerator inside image includes:
Based on the hand gestures on frame image each in the video image determined, hand institute on each frame image is determined
In region；
According to the relative position variation of hand region in the shooting of video image sequence and adjacent two field pictures
Determine hand exercise situation.
Optionally, the typing situation for judging food materials in refrigerator according to hand gestures and hand exercise situation includes:
When hand gestures detection model output result is empty-handed immigration refrigerator and non-empty-handed removal refrigerator, determine from refrigerator
Interior taking-up food materials；
When hand gestures detection model output result is non-empty-handed immigration refrigerator and empty-handed removal refrigerator, determine to refrigerator
Inside it is put into food materials；
When hand gestures detection model output result is empty-handedly moves into refrigerator and empty-handed removal refrigerator, food in judgement refrigerator
Material information do not change or refrigerator in the positions of food materials be adjusted；
When hand gestures detection model output result is non-empty-handed immigration refrigerator and non-empty-handed removal refrigerator, refrigerator is determined
Interior food materials information does not change or is put into the first food materials and takes out the second food materials.
Optionally, this method further include:
Refrigerator and empty-handedly removal refrigerator are moved into be empty-handed when hand gestures detection model exports result, or is non-empty-handed shifting
When entering refrigerator and non-empty-handed removal refrigerator, whether detection history food materials image and food materials image in current refrigerator are identical；
When history food materials image is identical as food materials image in current refrigerator, determine that food materials information does not become in refrigerator
Change；
When history food materials image and food materials image in current refrigerator be not identical, and hand gestures detection model output result is
When empty-handed immigration refrigerator and empty-handed removal refrigerator, determine that the position of food materials in refrigerator is adjusted；
When history food materials image and food materials image in current refrigerator be not identical, and hand gestures detection model output result is
When non-empty-handed immigration refrigerator and non-empty-handed removal refrigerator, judgement is put into the first food materials and takes out the second food materials.
Optionally, this method further include:
When determining to take out food materials out of refrigerator, confirmation hand is in the non-empty-handed arbitrary a removed under refrigerator posture
In frame image, third food materials corresponding to the food materials image in hand images region, or confirm to carry out the non-empty-handed removal shape
The third food materials in the target image obtained when state judges；Third food materials are confirmed as to the food materials being removed, and to current
Food materials in refrigerator at the historical position of third food materials are identified；
When determining to be put into food materials into refrigerator, confirmation hand is in the non-empty-handed arbitrary b moved under refrigerator posture
In frame image, the 4th food materials corresponding to the food materials image in hand images region, or confirm to carry out the non-empty-handed immigration shape
The 4th food materials in the background image obtained when state judges；4th food materials are confirmed as the food materials being placed into, and to current refrigerator
The 4th interior food materials are identified；
When determining that food materials information does not change in refrigerator, food materials Information invariability in current refrigerator is kept；
When the position for determining food materials in refrigerator is adjusted or determines to be put into the first food materials and take out the second food materials, to user
Issue the prompting that food materials information update is carried out by user.
Optionally, this method further include: third food materials and the 4th food materials are identified by the following method:
When determine out of refrigerator take out food materials when, using preset image segmentation algorithm interception history food materials image in work as
The different image-region of food materials image in preceding refrigerator, is labeled as the first image-region, which includes described the
Three food materials；The first image region of interception is inputted in trained sorting algorithm model in advance, to pass through the classification
Algorithm model is identified and is classified to the food materials in the first image region；
When determining to be put into food materials into refrigerator, intercept in the current refrigerator in food materials image with the history food materials figure
As different image-region, it is labeled as the second image-region, which includes the 4th food materials；By interception
Second image-region inputs in trained sorting algorithm model in advance, with by the sorting algorithm model to described the
Food materials in two image-regions are identified and are classified.
The beneficial effect of the embodiment of the present invention may include:
1, the embodiment of the present invention acquires refrigerator inside image；Detect the hand gestures and hand exercise on refrigerator inside image
Situation；The typing situation of food materials in refrigerator is judged according to hand gestures and hand exercise situation.The example scheme will be in refrigerator
Image object test problems switch to target classification problem, eliminate position error, improve discrimination.
2, the embodiment of the present invention is shot at refrigerator inside and/or chamber door entrance area by camera preset in refrigerator
Video image.The example scheme is based on video and handles in real time, so that the picture information quantity obtained is bigger, avoids for single
The accuracy of food materials identification is influenced because picture shooting angle or effect are undesirable when picture is handled.
3, the hand gestures on detection of embodiment of the present invention refrigerator inside image include: to obtain from video image for hand
The target image of portion's attitude detection；Target image is inputted into trained hand gestures detection model in advance；Pass through hand gestures
Detection model identifies the image in target image；Hand gestures are exported according to recognition result.The example scheme uses
Preset model identifies hand gestures, reduces the operand that refrigerator system is calculated in real time.
4, the embodiment of the present invention is obtained from video image includes: for the target image of hand gestures detection
A, the i-th frame image in the video image of shooting, and the background image compared as image are obtained；I is positive integer,
The initial value of i is 1；B, the i-th frame image is compared with the i+1 frame image of shooting；C, when the i-th frame image and i+1 frame figure
When being less than preset first threshold as the difference of the neighboring mean value of upper co-located region, the i-th frame image and i+1 frame figure are determined
As identical, and return step A after i is added 1；When the neighboring mean value of co-located region on the i-th frame image and i+1 frame image
When difference is greater than or equal to preset first threshold, determine that the i-th frame image is different from i+1 frame image, and enter step D；D,
Using i+1 frame image as target image.The example scheme compares acquisition target classification by the method for neighboring mean value frame by frame
The image to be processed of problem improves discrimination so that the image to be processed obtained is more acurrate.
5, the embodiment of the present invention judges that the typing situations of food materials in refrigerator includes: according to hand gestures and hand exercise situation
When hand gestures detection model output result is empty-handed immigration refrigerator and non-empty-handed removal refrigerator, determine to take out food out of refrigerator
Material；When hand gestures detection model output result is non-empty-handed immigration refrigerator and empty-handed removal refrigerator, determine to put into refrigerator
Enter food materials；When hand gestures detection model output result is empty-handedly moves into refrigerator and empty-handed removal refrigerator, food in judgement refrigerator
Material information do not change or refrigerator in the positions of food materials be adjusted；When hand gestures detection model output result is non-empty-handed shifting
When entering refrigerator and non-empty-handed removal refrigerator, determine that food materials information does not change or is put into the first food materials and takes out second in refrigerator
Food materials.The example scheme, database picture need not be fixed under refrigerator scene and be trained, and such relative to target detection
Image labeling work, greatly reduce data mark/calculation amount and database construction cost.
Detailed description of the invention
Following further describes the present invention with reference to the drawings:
Fig. 1 is the refrigerator food management method flow chart of the embodiment of the present invention；
Fig. 2 is the hand gestures method flow diagram on the detection refrigerator inside image of the embodiment of the present invention；
Fig. 3 is the hand exercise situation method flow diagram on the detection refrigerator inside image of the embodiment of the present invention；
Fig. 4 be the embodiment of the present invention fisrt feature point for hand closer to refrigerator inside when empty-handedly inwardly move
Dynamic detection method schematic diagram；
Fig. 5 be the embodiment of the present invention fisrt feature point for hand closer to outside refrigerator when empty-handedly inwardly move
Dynamic detection method schematic diagram；
Fig. 6 is the operation chart of the taking-up third food materials of the embodiment of the present invention；
Fig. 7 is the operation chart for being put into the 4th food materials of the embodiment of the present invention.
Specific embodiment
Understand in order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with attached drawing pair
The embodiment of the present invention is described in detail.It should be noted that in the absence of conflict, embodiment and reality in the application
The feature applied in example can mutual any combination.
Step shown in the flowchart of the accompanying drawings can be in a computer system such as a set of computer executable instructions
It executes.Also, although logical order is shown in flow charts, and it in some cases, can be to be different from herein suitable
Sequence executes shown or described step.
Embodiment one
A kind of refrigerator food management method, as shown in Figure 1, this method may include S101-S103:
S101, acquisition refrigerator inside image.
Optionally, acquisition refrigerator inside image may include:
The video image at refrigerator inside and/or chamber door entrance area is shot by camera preset in refrigerator.
In embodiments of the present invention, one or more cameras can be set in refrigerator in advance, so as to refrigerator inside
And/or chamber door entrance area carries out video capture, obtains above-mentioned refrigerator inside image, thus true by the refrigerator inside image
Recognize the typing situation of the food materials in refrigerator.
In embodiments of the present invention, camera may be mounted at refrigerator side, horizontal field of view angle be greater than 90 degree (110 degree~
130 degree are advisable), image in refrigerator is acquired in real time.Frame per second may remain in 5 frames/second~15 frames/second, and frame per second is excessively high, need to handle
Data volume it is excessive, increase computation burden, frame per second is too low, action recognition inaccuracy.In addition, in order to ensure image definition suggestion
Use 720P image resolution ratio.
In embodiments of the present invention, it can be respectively provided with camera in one layer of the left and right sides of refrigerator, two sides are all made of identical
Algorithm process, even if one layer occlusion issue occurs, the other side can also correctly be identified.In addition, there are two knowledges when two sides are identified
When other result, it can be subject to recognition confidence the higher person.
S102, the hand gestures on detection refrigerator inside image and hand exercise situation.
Optionally, hand gestures may include: empty-handed, non-empty-handed and without hand；
Hand exercise situation may include: empty-handed to move into refrigerator, non-empty-handed removals refrigerator, non-empty-handed immigration refrigerator and empty-handedly
Remove refrigerator.
In embodiments of the present invention, it is summarized to the operating habit that refrigerator uses for user it is found that user is opening ice
After case, main includes empty-handedly moving into refrigerator, non-empty-handed removal refrigerator, non-empty-handed immigration refrigerator and empty-handedly removing four hands of refrigerator
Motion conditions mainly include empty-handed, non-empty-handed and without three kinds of postures of hand for hand gestures.
In embodiments of the present invention, in order to enable it is more acurrate to target classification, it can be by hand gestures and hand exercise feelings
Condition detection separately carries out.
Optionally, as shown in Fig. 2, the hand gestures on detection refrigerator inside image may include S201-S204:
S201, the target image detected for hand gestures is obtained from video image.
In embodiments of the present invention, in order to realize the accurate detection to hand gestures, and the real-time of refrigerator system is saved
Calculation amount can first obtain one or more target image for hand gestures detection from video image.Due to hand appearance
State from no hand to have hand, from have hand to no hand, from empty-handedly to it is non-it is empty-handed, empty-handedly change from non-to the various states such as empty-handed
When, the image of two frame of front and back can have greatly changed in pixel or brightness, therefore can be carried out by the two field pictures that are connected
Neighboring mean value compares to know whether two field pictures have occurred biggish variation, so as to tentatively judge that a later frame image is opposite
There is a possibility that generating hand gestures variation in previous frame image, and the frame image that will test out is inputted as target image
Preset hand gestures detection model is detected, to improve detection efficiency and detection accuracy.
In embodiments of the present invention, a background image can be obtained, and from video image in advance with the background image
For start frame, the gradually variation between more every two field pictures.And in a later frame image compared with background image, between the two
When the difference of neighboring mean value is less than a certain preset threshold value, it is believed that two field pictures variation less, and by a later frame image
As background image, continuation is compared with next frame.If a later frame image is compared with background image, neighborhood between the two
When the difference of mean value is greater than or equal to a certain preset threshold value, it is believed that two field pictures change greatly, it is likely that latter
There is hand gestures variation, therefore can be using a later frame image as target image in frame image.
In embodiments of the present invention, which can be the start frame of video image, be also possible to after filtering
A certain frame among the video image of acquisition.Below using the start frame of video image as illustrating the present invention for background image
The specific embodiment of the detection hand gestures of embodiment.
Optionally, it is obtained from video image and may include: for the target image of hand gestures detection
A, the i-th frame image in the video image of shooting, and the background image compared as image are obtained；I is positive integer,
The initial value of i is 1；
B, the i-th frame image is compared with the i+1 frame image of shooting；
C, when the difference of the neighboring mean value of co-located region on the i-th frame image and i+1 frame image is less than preset the
When one threshold value, determine that the i-th frame image is identical as i+1 frame image, and return step A after i is added 1；When the i-th frame image and i-th+
When the difference of the neighboring mean value of co-located region is greater than or equal to preset first threshold on 1 frame image, the i-th frame image is determined
It is different from i+1 frame image, and enter step D；
D, using i+1 frame image as target image.
In embodiments of the present invention, it is based on 5*5 neighborhood change mean, which can be set to 30, work as a later frame
When the difference of the neighboring mean value of image and the background image of setting is less than 30, can using a later frame image as background image,
It, can be by a later frame image when the difference of a later frame image and the neighboring mean value of the background image of setting is greater than or equal to 30
As target image.It should be noted that the first threshold can be adjusted according to conditions such as concrete application scene or illumination
It is whole, its specific value is limited, i.e., including but not limited to above-mentioned 30.
S202, target image is inputted into trained hand gestures detection model in advance.
In embodiments of the present invention, hand gestures detection model can be pre-established, which is deep
Spend neural network model, for the deep neural network model can acquire in advance a large amount of hand gestures image to the model into
Row training, to obtain the hand gestures detection model with hand gestures identification function.
S203, the image in target image is identified by hand gestures detection model.
In embodiments of the present invention, since hand gestures detection model is trained in advance, hand gestures inspection
Direct Recognition can be carried out to the image of input by surveying model.
S204, hand gestures are exported according to recognition result.
In embodiments of the present invention, after hand gestures detection model can identify the target image of input, directly
Hand gestures are exported according to recognition result.For example, can be exported if detecting the hand gestures of empty-handed state in target image
Empty-handedly.
In embodiments of the present invention, above scheme only detects one target of hand by preset model, and only needs to use
One mini Mod, hand differ greatly with food materials in refrigerator, and feature is obvious, has ensured recognition accuracy.
In embodiments of the present invention, based on above-mentioned hand gestures detection scheme, hand exercise feelings can further be detected
Condition.
Optionally, as shown in figure 3, the hand exercise situation on detection refrigerator inside image may include S301-S302:
S301, based on the hand gestures on frame image each in the video image determined, determine on each frame image
Hand region.
S302, according to the relative position of hand region in the shooting of video image sequence and adjacent two field pictures
Change and determines hand exercise situation.
In embodiments of the present invention, according to hand location in the shooting of video image sequence and adjacent two field pictures
The relative position in domain, which changes, determines that hand exercise situation can be realized especially by following proposal: based on the video image determined
In hand gestures on each frame image, determine hand region on each frame image, and determine inside refrigerator space
Any one fix fisrt feature point and hand region on any one fix second feature point；Judgement
The relative position of fisrt feature point and second feature point；The opposite position that will be judged in every two field pictures adjacent in video image
It sets and compares；Determine second feature o'clock relative to the first spy according to the variation tendency of relative position between every adjacent two field pictures
Levy the variation tendency of point；And the variation tendency according to second feature point relative to fisrt feature point determines hand exercise situation.
In embodiments of the present invention, if the fisrt feature point of setting for hand closer to refrigerator inside,
When the relative position of fisrt feature point and second feature point can determine hand in mobile, such as Fig. 4 to refrigerator inside when shortening
Shown in empty-handedly move inward shown in schematic diagram；When the relative position of fisrt feature point and second feature point can be true when extending
Hand is determined to moving outside refrigerator., whereas if the fisrt feature point of setting for hand closer to outside refrigerator,
For example, hand has been in refrigerator inside, and fisrt feature point is arranged near refrigerator doors, then fisrt feature point and second feature point
Relative position can be determined when shortening hand to moving outside refrigerator, the opposite position of fisrt feature point and second feature point
Hand can be determined when extending to refrigerator inside movement by setting, and as shown in Figure 5 empty-handedly moves inward shown in schematic diagram.
It in embodiments of the present invention, can in conjunction with the hand moving direction judgement of hand gestures above-mentioned judgement and the step
To accurately determine out the hands such as empty-handed immigration refrigerator, non-empty-handed removal refrigerator, non-empty-handed immigration refrigerator and empty-handed removal refrigerator
Motion conditions.In addition, when for detecting the posture for being in no hand in entire video data, it can be determined that user is only by refrigerator
Door is opened, and is not stretched out one's hand into refrigerator and is carried out the operation such as picking and placing.
S103, the typing situation that food materials in refrigerator are judged according to hand gestures and hand exercise situation.
Optionally, judge that the typing situation of food materials in refrigerator may include following according to hand gestures and hand exercise situation
Four kinds of situations:
Situation one, when hand gestures detection model output result be it is empty-handed move into refrigerator and non-empty-handed removal refrigerator when, sentence
It is fixed that food materials are taken out out of refrigerator；
Situation two, when hand gestures detection model output result is non-empty-handed immigrations refrigerator and empty-handed removal refrigerator, sentence
Food materials are put into orientation refrigerator；
Situation three, when hand gestures detection model output result be it is empty-handed move into refrigerator and it is empty-handed remove refrigerator when, determine
In refrigerator food materials information do not change or refrigerator in the positions of food materials be adjusted；
Situation four, when hand gestures detection model output result be non-empty-handed immigration refrigerator and non-empty-handed removal refrigerator when,
Determine that food materials information does not change or is put into the first food materials and takes out the second food materials in refrigerator.
In embodiments of the present invention, the judgement for situation one and situation two is more clear, and judgement is only needed to take out in next step
Food materials be what or the food materials that are put into what is.Also need further judge which belongs to for situation three and situation four
A kind of situation.
In the embodiment of the present invention, the history food materials image stored in refrigerator system can be recalled, with progress hand
Portion operates food materials image in later current refrigerator and compares, and determines whether two images change, so that it is determined that artificial situation
Three and situation four in food materials situation of change.It can specifically be realized by following scheme.
Optionally, this method can also include:
Refrigerator and empty-handedly removal refrigerator are moved into be empty-handed when hand gestures detection model exports result, or is non-empty-handed shifting
When entering refrigerator and non-empty-handed removal refrigerator, whether detection history food materials image and food materials image in current refrigerator are identical；
When history food materials image is identical as food materials image in current refrigerator, determine that food materials information does not become in refrigerator
Change；
When history food materials image and food materials image in current refrigerator be not identical, and hand gestures detection model output result is
When empty-handed immigration refrigerator and empty-handed removal refrigerator, determine that the position of food materials in refrigerator is adjusted；
When history food materials image and food materials image in current refrigerator be not identical, and hand gestures detection model output result is
When non-empty-handed immigration refrigerator and non-empty-handed removal refrigerator, judgement is put into the first food materials and takes out the second food materials.
In the embodiment of the present invention, which can be the last refrigerator doors stored in refrigerator and shuts
When the refrigerator storage chamber that shoots in food materials image, refrigerator system, which directly transfers storage information, can obtain the history food materials figure
Picture.In addition, refrigerator system can also be by wireless technology and cloud storage or user terminal (for example, mobile phone, IPAD, wearable
Equipment etc.) communication, it is obtained in the refrigerator storage chamber shot when last refrigerator doors are shut from the cloud storage or user terminal
Food materials image.Secondly, the history food materials image can also be most start in the video image of current shooting a few frame images (or
Background image in foregoing teachings) because hand not yet operates food materials, food materials and its position in this few frame image
It sets and has not occurred variation, can be used as history food materials image and referred to.
In embodiments of the present invention, by aforementioned schemes determine the corresponding food materials situation of change of hand exercise situation with
Afterwards, specific taken out food materials, be put into food materials and adjusted food materials can also be further determined that out according to following proposal.
Optionally, this method can also include:
When determining to take out food materials out of refrigerator, confirmation hand is in the non-empty-handed arbitrary a removed under refrigerator posture
In frame image, third food materials corresponding to the food materials image in hand images region, or confirm to carry out the non-empty-handed removal shape
The third food materials in the target image obtained when state judges；Third food materials are confirmed as to the food materials being removed, and to current
Food materials in refrigerator at the historical position of third food materials are identified；
When determining to be put into food materials into refrigerator, confirmation hand is in the non-empty-handed arbitrary b moved under refrigerator posture
In frame image, the 4th food materials corresponding to the food materials image in hand images region, or confirm to carry out the non-empty-handed immigration shape
The 4th food materials in the background image obtained when state judges；4th food materials are confirmed as the food materials being placed into, and to current refrigerator
The 4th interior food materials are identified；
When determining that food materials information does not change in refrigerator, food materials Information invariability in current refrigerator is kept；
When the position for determining food materials in refrigerator is adjusted or determines to be put into the first food materials and take out the second food materials, to user
Issue the prompting that food materials information update is carried out by user.
Embodiment two
The difference of the embodiment and embodiment one is, gives another kind and identifies to third food materials and the 4th food materials
Method.
Optionally, this method further include: third food materials and the 4th food materials are identified by the following method:
Using different with food materials image in current refrigerator in preset image segmentation algorithm interception history food materials image
Image-region is labeled as the first image-region；And the different image-region in current refrigerator in food materials image is intercepted, it marks
For the second image-region；
First image-region of interception and the second image-region are inputted in trained sorting algorithm model in advance, with logical
It crosses classification algorithm model the food materials in the first image-region and the second image-region are identified and classified.
In embodiments of the present invention, as shown in fig. 6, being directed to the operation for taking out third food materials, history can be obtained respectively
Image in food materials image (such as background image above-mentioned) and the currently later refrigerator of taking-up third food materials, and respectively to this two
Image is intercepted, and different image-region in two images, i.e., the first above-mentioned image-region and the second image district are obtained
Domain；At this moment due in current refrigerator food materials image be equivalent to and remove third food materials from history food materials image, first
Image-region and the second image-region are inevitable different, by preparatory trained sorting algorithm model to the first image district of interception
Domain and the second image-region carry out identification can go out third food materials with Direct Recognition.Alternatively, it is also possible to only by the first image of interception
Region (the third food materials region that do not take away also before third food materials for including in history food materials image) input is instructed in advance
In the sorting algorithm model perfected, because first image-region includes the image for the third food materials taken away, pass through sorting algorithm
Model is identified and is classified to the food materials of the first image-region, and Direct Recognition goes out third food materials.
In embodiments of the present invention, as shown in fig. 7, being directed to the operation for being put into the 4th food materials, history can be obtained respectively
Food materials image (such as background image above-mentioned) and it is currently put into image in the later refrigerator of the 4th food materials, and respectively to this two
Image is intercepted, and different image-region in two images, i.e., the first above-mentioned image-region and the second image district are obtained
Domain；At this moment due in current refrigerator food materials image be equivalent to and be put into the 4th food materials in history food materials image, first
Image-region and the second image-region are inevitable different, by preparatory trained sorting algorithm model to the first image district of interception
Domain and the second image-region carry out identification can go out the 4th food materials with Direct Recognition.Alternatively, it is also possible to only by the second image of interception
Region (the 4th food materials region having been placed in after the 4th food materials for including in food materials image in i.e. current refrigerator) input
In preparatory trained sorting algorithm model, because second image-region includes the image for the 4th food materials being put into, by dividing
Class algorithm model is identified and is classified to the food materials of the second image-region, and Direct Recognition goes out the 4th food materials.
In embodiments of the present invention, the sorting algorithm model can include but is not limited to: GoogleNet model,
VGG16 model or ResNet model.
In embodiments of the present invention, the picture for being trained to sorting algorithm model need not be defined under refrigerator scene
Picture, can be applied to refrigerator system before it is in advance that sorting algorithm model training is good, reduce refrigerator system and marked
Note, the data volume calculated, and the construction cost of database is greatly reduced.
Embodiment three
The difference of the embodiment and embodiment one and embodiment two is, gives another and eats to third food materials and the 4th
Material carries out knowledge method for distinguishing.
Optionally, this method can also include: and identify by the following method to third food materials and the 4th food materials:
When determine out of refrigerator take out food materials when, using preset image segmentation algorithm interception history food materials image in work as
The different image-region of food materials image in preceding refrigerator, is labeled as the first image-region, which includes described the
Three food materials；The first image region of interception is inputted in trained sorting algorithm model in advance, to pass through the classification
Algorithm model is identified and is classified to the food materials in the first image region；
When determining to be put into food materials into refrigerator, intercept in the current refrigerator in food materials image with the history food materials figure
As different image-region, it is labeled as the second image-region, which includes the 4th food materials；By interception
Second image-region inputs in trained sorting algorithm model in advance, with by the sorting algorithm model to described the
Food materials in two image-regions are identified and are classified.
In embodiments of the present invention, the analysis based on aforementioned schemes can be with, which only has for interception
The image of three food materials or the 4th food materials, which is put into preparatory trained sorting algorithm model, to carry out identification and simplifies identification process, mentions
High recognition efficiency.
Although disclosed herein embodiment it is as above, the content only for ease of understanding the present invention and use
Embodiment is not intended to limit the invention.Technical staff in any fields of the present invention is taken off not departing from the present invention
Under the premise of the spirit and scope of dew, any modification and variation, but the present invention can be carried out in the form and details of implementation
Scope of patent protection, still should be subject to the scope of the claims as defined in the appended claims.
Claims (10)
1. a kind of refrigerator food management method, which is characterized in that the described method includes:
Acquire refrigerator inside image；
Detect the hand gestures and hand exercise situation on the refrigerator inside image；
The typing situation of food materials in refrigerator is judged according to the hand gestures and hand exercise situation.
2. refrigerator food management method according to claim 1, which is characterized in that the acquisition refrigerator inside image packet
It includes:
The video image at the refrigerator inside and/or chamber door entrance area is shot by camera preset in refrigerator.
3. refrigerator food management method according to claim 2, which is characterized in that the detection refrigerator inside image
On hand gestures include:
The target image for hand gestures detection is obtained from the video image；
The target image is inputted into trained hand gestures detection model in advance；
The image in the target image is identified by the hand gestures detection model；
The hand gestures are exported according to recognition result.
4. refrigerator food management method according to claim 3, which is characterized in that described to be obtained from the video image
For the hand gestures detection target image include:
A, the i-th frame image in the video image of shooting, and the background image compared as image are obtained；I is positive integer,
The initial value of i is 1；
B, the i-th frame image is compared with the i+1 frame image of shooting；
C, it is preset when the difference of the neighboring mean value of co-located region on the i-th frame image and the i+1 frame image is less than
First threshold when, it is identical as the i+1 frame image to determine the i-th frame image, and return step A after i is added 1；Work as institute
The difference for stating the neighboring mean value of co-located region on the i-th frame image and the i+1 frame image is greater than or equal to preset the
When one threshold value, it is different from the i+1 frame image to determine the i-th frame image, and enter step D；
D, using the i+1 frame image as the target image.
5. refrigerator food management method according to any one of claims 1-4, which is characterized in that the hand gestures packet
It includes: empty-handed, non-empty-handedly and without hand；
The hand exercise situation includes: empty-handed immigration refrigerator, non-empty-handed removal refrigerator, non-empty-handed immigration refrigerator and empty-handed removal
Refrigerator.
6. refrigerator food management method according to claim 5, which is characterized in that the detection refrigerator inside image
On hand exercise situation include:
Based on the hand gestures on each frame image in the video image determined, hand on each frame image is determined
Portion region；
Changed according to the relative position of hand region in the shooting of video image sequence and adjacent two field pictures and is determined
The hand exercise situation.
7. refrigerator food management method according to claim 6, which is characterized in that described according to the hand gestures and hand
Portion's motion conditions judge that the typing situation of food materials in refrigerator includes:
When hand gestures detection model output result is empty-handed immigration refrigerator and non-empty-handed removal refrigerator, determine from refrigerator
Interior taking-up food materials；
When hand gestures detection model output result is non-empty-handed immigration refrigerator and empty-handed removal refrigerator, determine to refrigerator
Inside it is put into food materials；
When hand gestures detection model output result is empty-handedly moves into refrigerator and empty-handed removal refrigerator, food in judgement refrigerator
Material information do not change or refrigerator in the positions of food materials be adjusted；
When hand gestures detection model output result is non-empty-handed immigration refrigerator and non-empty-handed removal refrigerator, refrigerator is determined
Interior food materials information does not change or is put into the first food materials and takes out the second food materials.
8. refrigerator food management method according to claim 7, which is characterized in that the method also includes:
Refrigerator and empty-handedly removal refrigerator are moved into be empty-handed when the hand gestures detection model exports result, or is non-empty-handed shifting
When entering refrigerator and non-empty-handed removal refrigerator, whether detection history food materials image and food materials image in current refrigerator are identical；
When history food materials image is identical as food materials image in current refrigerator, determine that food materials information does not become in the refrigerator
Change；
When history food materials image and food materials image in current refrigerator be not identical, and hand gestures detection model output result is
When empty-handed immigration refrigerator and empty-handed removal refrigerator, determine that the position of food materials in the refrigerator is adjusted；
When history food materials image and food materials image in current refrigerator be not identical, and hand gestures detection model output result is
When non-empty-handed immigration refrigerator and non-empty-handed removal refrigerator, judgement is put into the first food materials and takes out the second food materials.
9. refrigerator food management method according to claim 7 or 8, which is characterized in that the method also includes:
When determining to take out food materials out of refrigerator, confirmation hand is in the non-empty-handed arbitrary a removed under refrigerator posture
In frame image, third food materials corresponding to the food materials image in hand images region, or confirm to carry out the non-empty-handed removal shape
The third food materials in the target image obtained when state judges；The third food materials are confirmed as the food materials being removed, and right
Food materials in current refrigerator at the historical position of the third food materials are identified；
When determining to be put into food materials into refrigerator, confirmation hand is in the non-empty-handed arbitrary b moved under refrigerator posture
In frame image, the 4th food materials corresponding to the food materials image in hand images region, or confirm to carry out the non-empty-handed immigration shape
The 4th food materials in the background image obtained when state judges；4th food materials are confirmed as the food materials being placed into, and right
The 4th food materials in current refrigerator are identified；
When determining that food materials information does not change in the refrigerator, food materials Information invariability in current refrigerator is kept；
When the position for determining food materials in the refrigerator is adjusted or determines to be put into the first food materials and take out the second food materials, to user
Issue the prompting that food materials information update is carried out by user.
10. refrigerator food management method according to claim 9, which is characterized in that the method also includes: by following
Method identifies the third food materials and the 4th food materials:
When determine out of refrigerator take out food materials when, using preset image segmentation algorithm interception history food materials image in current ice
The different image-region of food materials image in case, is labeled as the first image-region, which eats comprising the third
Material；The first image region of interception is inputted in trained sorting algorithm model in advance, to pass through the sorting algorithm
Model is identified and is classified to the food materials in the first image region；
When determining to be put into food materials into refrigerator, intercept in the current refrigerator in food materials image with the history food materials image not
Identical image-region, is labeled as the second image-region, which includes the 4th food materials；It will be described in interception
Second image-region inputs in advance in trained sorting algorithm model, with by the sorting algorithm model to second figure
As the food materials in region are identified and are classified.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710888847.1A CN109558775A (en) | 2017-09-27 | 2017-09-27 | A kind of refrigerator food management method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710888847.1A CN109558775A (en) | 2017-09-27 | 2017-09-27 | A kind of refrigerator food management method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109558775A true CN109558775A (en) | 2019-04-02 |
Family
ID=65863539
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710888847.1A Withdrawn CN109558775A (en) | 2017-09-27 | 2017-09-27 | A kind of refrigerator food management method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109558775A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110689560A (en) * | 2019-10-11 | 2020-01-14 | 海信集团有限公司 | Food material management method and equipment |
| CN110796051A (en) * | 2019-10-19 | 2020-02-14 | 北京工业大学 | Real-time access behavior detection method and system based on container scene |
| CN110837799A (en) * | 2019-11-05 | 2020-02-25 | 四川虹美智能科技有限公司 | Refrigerator food material input management method, device and system |
| CN111126133A (en) * | 2019-11-08 | 2020-05-08 | 博云视觉（北京）科技有限公司 | Intelligent refrigerator access action recognition method based on deep learning |
| WO2021057769A1 (en) * | 2019-09-25 | 2021-04-01 | 青岛海尔电冰箱有限公司 | Method for viewing and tracking stored items |
-
2017
- 2017-09-27 CN CN201710888847.1A patent/CN109558775A/en not_active Withdrawn
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021057769A1 (en) * | 2019-09-25 | 2021-04-01 | 青岛海尔电冰箱有限公司 | Method for viewing and tracking stored items |
| CN110689560A (en) * | 2019-10-11 | 2020-01-14 | 海信集团有限公司 | Food material management method and equipment |
| CN110689560B (en) * | 2019-10-11 | 2022-06-07 | 海信集团有限公司 | Food material management method and equipment |
| CN110796051A (en) * | 2019-10-19 | 2020-02-14 | 北京工业大学 | Real-time access behavior detection method and system based on container scene |
| CN110796051B (en) * | 2019-10-19 | 2024-04-26 | 北京工业大学 | Real-time access behavior detection method and system based on container scene |
| CN110837799A (en) * | 2019-11-05 | 2020-02-25 | 四川虹美智能科技有限公司 | Refrigerator food material input management method, device and system |
| CN111126133A (en) * | 2019-11-08 | 2020-05-08 | 博云视觉（北京）科技有限公司 | Intelligent refrigerator access action recognition method based on deep learning |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109558775A (en) | A kind of refrigerator food management method | |
| CN109919977B (en) | Video motion person tracking and identity recognition method based on time characteristics | |
| KR101964397B1 (en) | Information processing apparatus and information processing method | |
| CN102761706B (en) | Imaging device and imaging method | |
| Luber et al. | People tracking in rgb-d data with on-line boosted target models | |
| CN104601964B (en) | Pedestrian target tracking and system in non-overlapping across the video camera room of the ken | |
| CN103098076B (en) | Gesture recognition system for TV control | |
| JP2020061128A (en) | Binocular pedestrian detection system having dual-stream deep learning neural network and the methods of using the same | |
| CN103996046B (en) | The personal identification method merged based on many visual signatures | |
| CN110287907B (en) | Object detection method and device | |
| CN110232330A (en) | A kind of recognition methods again of the pedestrian based on video detection | |
| CN106097385B (en) | A kind of method and apparatus of target following | |
| CN110390685B (en) | Feature point tracking method based on event camera | |
| CN105741319B (en) | Improvement visual background extracting method based on blindly more new strategy and foreground model | |
| CN112581540B (en) | Camera calibration method based on human body posture estimation in large scene | |
| CN107507226A (en) | A kind of method and device of images match | |
| US11361534B2 (en) | Method for glass detection in real scenes | |
| US20140044342A1 (en) | Method for generating 3d coordinates and mobile terminal for generating 3d coordinates | |
| CN106934332A (en) | A kind of method of multiple target tracking | |
| WO2019068931A1 (en) | Methods and systems for processing image data | |
| Li et al. | Treat samples differently: Object tracking with semi-supervised online covboost | |
| CN107085729A (en) | A kind of personnel's testing result modification method based on Bayesian inference | |
| CN108053422A (en) | Mobile target monitoring method | |
| CN108765326A (en) | A kind of synchronous superposition method and device | |
| KR101853276B1 (en) | Method for detecting hand area from depth image and apparatus thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20190402 |
|
| WW01 | Invention patent application withdrawn after publication |