CN110674789B

CN110674789B - Food material management method and refrigerator

Info

Publication number: CN110674789B
Application number: CN201910966464.0A
Authority: CN
Inventors: 高桢; 曲磊; 赵启东; 李正义
Original assignee: Hisense Co Ltd
Current assignee: Hisense Co Ltd
Priority date: 2019-10-12
Filing date: 2019-10-12
Publication date: 2022-04-29
Anticipated expiration: 2039-10-12
Also published as: CN110674789A

Abstract

The application provides a food material management method and a refrigerator, wherein the method comprises the following steps: responding to the opening of a refrigerator door body, and acquiring a plurality of frames of first images of a food material storage area; determining a target ROI existing in each frame of the first image according to the similarity between the ROI of the region of interest of the multiple frames of the first image and the ROI of the corresponding position in the second image; the first image and the second image respectively comprise a plurality of ROIs (region of interest), the different ROIs correspond to different interlayers of the food material storage area, and the target ROI is an ROI corresponding to an interlayer on the uppermost layer in the interlayers with the similarity smaller than a preset similarity threshold; and if at least two continuous first images in each frame of first image have the target ROI, taking the interlayer of the uppermost layer corresponding to the target ROI in the at least two frames of first images as the food material access position. The method has high identification accuracy.

Description

Food material management method and refrigerator

Technical Field

The application relates to the technical field of household appliances, in particular to a food material management method and a refrigerator.

Background

With the development of artificial intelligence technology, artificial intelligence technology is used in more and more household appliances, such as refrigerators, televisions, air conditioners, washing machines, and the like. For the refrigerator, how to effectively manage food in the refrigerator becomes a core function of the intelligent refrigerator at present, and the problem is also very concerned by users. The precondition for realizing food management is to know the access position of food.

In the related art, the change conditions of the food materials in each interlayer before and after the food materials are stored and taken are detected through the cameras arranged in each layer in the refrigerator, so that the automatic identification of the storage and taking positions of the food materials is realized, but in the scheme, if the food materials in the refrigerator are shielded, the identification accuracy is low.

Disclosure of Invention

The application provides a food material management method and a refrigerator, so that accuracy of identification of a storage and taking position of the refrigerator is improved.

In a first aspect, the present application provides a food material management method, including:

responding to the opening of a refrigerator door body, and acquiring a plurality of frames of first images of a food material storage area;

determining a target ROI existing in each frame of the first image according to the similarity between the ROI of the region of interest of the first image and the ROI of the corresponding position in the second image of multiple frames; the second image is a collected background image of the food material storage area, the first image and the second image respectively comprise a plurality of ROIs, different ROIs correspond to different interlayers of the food material storage area, and the target ROI is an ROI corresponding to an interlayer on the uppermost layer in the interlayers with the similarity smaller than a preset similarity threshold;

and if at least two continuous first images in the first images of each frame have the target ROI, using the interlayer of the uppermost layer corresponding to the target ROI in the at least two continuous first images as a food material access position.

In a second aspect, an embodiment of the present application provides a refrigerator, including:

the refrigerator comprises a refrigerator main body, an image acquisition unit and a processor; the image acquisition unit and the processor are fixedly arranged in the refrigerator main body; the image acquisition unit is positioned at the top of the compartment of the refrigerator;

the image acquisition unit is used for responding to the opening of a refrigerator door body and acquiring a plurality of frames of first images of the food material storage area; the visual angle range of the image acquisition unit comprises the food material storage area;

wherein the processor is configured to:

According to the food material management method and the refrigerator, the opening of a door body of the refrigerator is responded, and a plurality of frames of first images of a food material storage area are obtained; determining a target ROI existing in each frame of the first image according to the similarity between the ROI of the region of interest of the first image and the ROI of the corresponding position in the second image of multiple frames; the second image is a collected background image of the food material storage area, the first image and the second image respectively comprise a plurality of ROIs, different ROIs correspond to different interlayers of the food material storage area, and a target ROI is an ROI corresponding to an interlayer on the uppermost layer in the interlayers with the similarity smaller than a preset similarity threshold; and if the target ROI exists in at least two continuous first images in each frame of the first images, using the interlayer of the uppermost layer corresponding to the target ROI in the at least two frames of the first images as the food material access position.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.

FIG. 1A is a schematic diagram of a refrigerator according to an embodiment of the present disclosure;

FIG. 1B is a schematic view of a refrigerator according to another embodiment of the present disclosure;

fig. 2 is a flowchart illustrating an embodiment of a food material management method provided in the present application;

FIG. 3A is a schematic view of an inner partition of a refrigerator according to an embodiment of the method provided herein;

FIG. 3B is a schematic view of an inner partition of a refrigerator according to another embodiment of the method provided herein;

FIG. 4 is a schematic ROI extraction of a captured image within the refrigerator shown in FIG. 3A;

FIG. 5 is a schematic diagram of a similarity determination process according to an embodiment of the method provided by the present application;

FIG. 6 is a schematic diagram of an access location sequence according to an embodiment of the method provided in the present application;

FIG. 7 is a sequence diagram of the sequence of access locations of FIG. 6 after processing;

FIG. 8 is a schematic structural diagram of an embodiment of a refrigerator provided herein;

fig. 9 is a schematic structural diagram of an embodiment of an electronic device provided in the present application.

With the foregoing drawings in mind, certain embodiments of the disclosure have been shown and described in more detail below. These drawings and written description are not intended to limit the scope of the disclosed concepts in any way, but rather to illustrate the concepts of the disclosure to those skilled in the art by reference to specific embodiments.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

The terms "comprising" and "having," and any variations thereof, in the description and claims of this application and the drawings described herein are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.

Firstly, the application scenario related to the present application is introduced:

the method provided by the embodiment of the application is applied to a scene of managing the refrigerator food materials, for example, the access position is identified, so that the accuracy of identifying the access position is improved.

In the related technology, as for the identification method of the food material access position, one method is mostly through manual input or user voice input, although the accuracy rate is high, the method depends on the participation of a user, and the user experience is not high; in another type of access position identification method based on image identification, a camera is generally installed on each layer of a refrigerator, image identification is performed on each layer after access is finished, and access change conditions of each layer of food materials are detected. Although the method realizes automatic identification of the access position, the method is easily influenced by food material shielding, and especially the method is easy to miss detection and error detection when the food material is accessed from deep layers.

According to the food material management method, the identification of the access position is realized when the food material is taken out of or put into the interlayer of the food material storage area, the method belongs to a dynamic identification method, and the influence of food material shielding in static identification can be greatly reduced. Meanwhile, according to the method, the storage and taking position can be recognized once in each round of storage and taking process, the centralized recognition of static recognition is avoided, the storage position of the food material can be updated in time, and the recognition rate of the storage and taking position is further improved.

The method provided by the application can be realized by a refrigerator such as a processor executing corresponding software codes, and can also be realized by the refrigerator through data interaction with a server while executing the corresponding software codes, for example, the server controls the refrigerator to realize the identification method. The refrigerator and the server can be connected through a network.

In some embodiments, the refrigerator may include an image collecting unit for collecting an image of a food material storage area during a process of a user accessing food materials; in an embodiment, the image capturing unit may be a color camera, a depth camera, or a combination of both. The color camera can be a common color camera or a wide-angle color camera; the depth camera may be a binocular camera, a structured light camera, or a camera based on time of flight (TOF).

In some embodiments, image capturing unit 10 may be mounted on the top of the interior of a refrigerator compartment (as shown in FIG. 1A), or image capturing unit 10 may be mounted on the top of the exterior of the refrigerator compartment (as shown in FIG. 1A)

Near the top of the refrigerator door 11, as shown in fig. 1B), the range of viewing angles of the image capture unit can cover the entire refrigerated area and/or the entire refrigerated area.

In some embodiments, the refrigerator may further include a processor and a memory, where the processor is configured to implement the food material management method of the present application, and in some embodiments, the processor may further perform preprocessing on an image acquired by the image acquisition unit. The processor may be a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), or a combination of a CPU and a GPU. The processor may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a Programmable Logic Device (PLD), or a combination thereof. The PLD may be a Complex Programmable Logic Device (CPLD), a field-programmable gate array (FPGA), a General Array Logic (GAL), or any combination thereof. The storage is connected with the processor through a bus or other manners, at least one instruction, at least one program, a code set or an instruction set is stored in the storage, and the at least one instruction, the at least one program, the code set or the instruction set is loaded by the processor and executes the food material management method. The memory may be a volatile memory (volatile memory), a non-volatile memory (non-volatile memory), or a combination thereof. The volatile memory may be a random-access memory (RAM), such as a Static Random Access Memory (SRAM) or a Dynamic Random Access Memory (DRAM). The nonvolatile memory may be a Read Only Memory (ROM), such as a Programmable Read Only Memory (PROM), an Erasable Programmable Read Only Memory (EPROM), and an electrically erasable programmable read-only memory (EEPROM). The nonvolatile memory may also be a flash memory (flash memory), a magnetic memory such as a magnetic tape (magnetic tape), a floppy disk (floppy disk), and a hard disk. The non-volatile memory may also be an optical disc.

The technical solution of the present application will be described in detail below with specific examples. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments.

Fig. 2 is a flowchart illustrating an embodiment of a food material management method provided in the present application. As shown in fig. 2, the method provided by this embodiment includes:

step 101, responding to the opening of a refrigerator door body, and acquiring a plurality of frames of first images of a food material storage area;

when identifying the food material storing and taking position of a user in the process of storing and taking food materials, responding to the opening of a refrigerator door body, and collecting at least two frames of first images by an image collecting unit arranged in the refrigerator. The at least two frames of first images are images of a food material storage area within the refrigerator. Wherein the image capturing unit may be disposed at a top of the compartment of the refrigerator.

102, determining a target ROI existing in each frame of first image according to the similarity between the ROI of the region of interest of the multiple frames of first images and the ROI of the corresponding position in the second image; the second image is a background image of the collected food material storage area, the first image and the second image respectively comprise a plurality of ROIs, different ROIs correspond to different interlayers of the food material storage area, and the target ROI is an ROI corresponding to an interlayer on the uppermost layer in the interlayers with the similarity smaller than a preset similarity threshold.

After at least two frames of first images acquired by the image acquisition unit are acquired, similarity analysis is performed on the acquired first images and a second image serving as a background image, for example, a Region of interest (ROI) in the first image is extracted, and each ROI in the first image is compared with an ROI in a corresponding position in the second image to perform similarity comparison. The first image and the second image respectively include a plurality of ROIs, and different ROIs correspond to different interlayers of the food material storage area, as shown in fig. 3A. For any ROI, the similarity of the ROI of the first image and the ROI of the corresponding position of the second image is calculated.

In one implementation, for the second image as the background image, the first captured frame image may be selected as the second image during the user's access to the food material, for example, the first captured frame image is triggered when it is detected that the refrigerator door is opened.

In other implementations, the second image may also be updated for multiple access processes, and specific implementations may be seen in the following embodiments.

For example, a user may open the refrigerator, place an apple in the refrigerator, and remove a bottle of beverage, which may be considered a two-pass access process.

Wherein the similarity may for example comprise at least one of: color similarity, structural similarity, and texture similarity.

In one implementation manner, for any ROI in any frame of the first image, if the similarity corresponding to the ROI is greater than the preset similarity threshold corresponding to the ROI, it is indicated that no access behavior occurs to the interlayer of the food material storage region corresponding to the ROI.

If the similarity corresponding to the ROI is smaller than the preset similarity threshold corresponding to the ROI, it is indicated that the access behavior may occur to the interlayer of the food material storage region corresponding to the ROI, for example, if the interlayer corresponding to the ROI is blocked by a limb of a user, the similarity between the interlayer and the interlayer in the background image is smaller, or if a certain food material is taken out from the interlayer, the similarity between the interlayer and the interlayer in the previous background image is smaller.

And if the number of the ROIs with the similarity smaller than the preset similarity threshold is one, directly determining the ROI as the target ROI.

If the number of ROIs with similarity smaller than the preset similarity threshold is greater than one, as shown in fig. 3B, for example, in the process that the user takes food from the interlayer on the uppermost layer of the food storage area, the two interlayers are simultaneously blocked by the arm, and at this time, the target ROI is actually the ROI corresponding to the interlayer on the uppermost layer.

The number of layers of the interlayer of the food material storage area is numbered from bottom to top in the following embodiments. In an embodiment, the predetermined similarity thresholds corresponding to the ROIs corresponding to the different partition layers may be the same or different.

And 103, if at least two continuous first images in each frame of first image have the target ROI, taking the interlayer of the uppermost layer corresponding to the target ROI in the at least two continuous first images as the food material access position.

Specifically, if the target ROIs exist in at least two frames of first images in a food material access process in a certain round, but the target ROIs of the first images in each frame are different, for example, the interlayer corresponding to the target ROI of the t-3 th frame of first image is the 1 st layer, the interlayer corresponding to the target ROI of the t-2 th frame of first image is the 2 nd layer, the interlayer corresponding to the target ROI of the t-1 th frame of first image is the 3 rd layer, the interlayer corresponding to the target ROI of the t +1 th frame of first image is the 3 rd layer, and the interlayer corresponding to the target ROI of the t +2 th frame of first image is the 2 nd layer, it is further necessary to determine which interlayer the food material access position is.

In one implementation, the uppermost interlayer of the interlayers may be used as the food material access position.

In an implementation manner, the uppermost interlayer in the interlayers having the occurrence times greater than the first preset time threshold in the at least two frames of first images having the target ROI may be used as the food material access position, for example, the 3 rd interlayer has the occurrence times greater than 2 times, and then the 3 rd interlayer is used as the food material access position, so that misjudgment in the food material access process can be avoided, for example, a user extends a hand to the 4 th layer, but finally takes a food material on the 3 rd layer. Or, the interlayer with the largest occurrence number in the at least two frames of the first images with the target ROI can be used as the food material access position.

In practical application, each frame of first image can be obtained in real time, a target ROI existing in each frame of first image is identified, and the food material access position is decided after multiple frames of first images are obtained, or whether the target ROI exists or not is sequentially identified and the food material access position is decided after multiple frames of first images are obtained.

The method comprises the steps of responding to the opening of a refrigerator door body, and obtaining a plurality of frames of first images of a food material storage area; determining a target ROI existing in each frame of the first image according to the similarity between the ROI of the region of interest of the first image and the ROI of the corresponding position in the second image of multiple frames; the second image is a collected background image of the food material storage area, the first image and the second image respectively comprise a plurality of ROIs, different ROIs correspond to different interlayers of the food material storage area, and the target ROI is an ROI corresponding to an interlayer on the uppermost layer in the interlayers with the similarity smaller than a preset similarity threshold; and if the target ROI exists in at least two continuous first images in each frame of the first images, using the interlayer of the uppermost layer corresponding to the target ROI in the at least two frames of the first images as the food material access position.

On the basis of the above embodiment, further, before step 102, the ROI of the first image may also be extracted, which may specifically be implemented as follows:

aiming at any frame of first image, determining initial position information of the edge area of each interlayer in the first image according to the position of at least one interlayer of the refrigerator;

fitting to obtain a boundary curve of the edge area of each interlayer according to the initial position information of the edge area of each interlayer in the first image;

and extracting the ROI of the first image according to the fitted boundary curve of the edge area of each interlayer.

Specifically, for a plurality of frames of first images acquired by an image acquisition unit in the food material storing and taking process, extracting a region of interest (ROI) of each frame of first image respectively. Based on the interlayer structure of the refrigerator, the position of at least one interlayer in the refrigerator can be determined, and further the initial position of each interlayer in the first image can be determined by combining the installation position of the image acquisition unit.

In some embodiments, further, the actual position of the boundary of the interlayer may be determined by fitting a boundary curve of the edge region, and finally the ROI is extracted according to the boundary of each interlayer.

In some implementations, the regions between the boundaries of different spacers are determined as ROIs for which different spacers correspond, such as the 3 ROIs in fig. 3B.

In some implementation manners, the edge regions of each interlayer in the refrigerator, which are close to the refrigerator door, are different from the main body of the interlayer in material, so that the edge regions of each interlayer can be well identified, and food materials can be generally placed on the interlayer through the edge regions in the food material storing and taking process by a user, for example, as shown in fig. 4, the ROI corresponding to each interlayer is only the edge region of the interlayer, so that whether the storing and taking action occurs in the interlayer can be determined based on the similarity of the edge regions of the interlayer, and then the target ROI corresponding to the first image of the frame can be determined.

Specifically, if the image acquisition unit adopts a wide-angle camera, the imaging of the wide-angle camera has certain distortion, and the closer the image acquisition unit is, the more serious the distortion degree occurs at the edge of the interlayer. Therefore, for the edge area of each interlayer with obvious distortion, a plurality of sections of straight line fitting can be adopted; straight line fitting may be used directly for the boundaries of the edge regions of the spacers that are not significantly distorted. If the image acquisition unit adopts a common camera, the image distortion degree is small, and straight line fitting can be adopted for the boundary of the edge area of each interlayer. For different interlayer structures in the refrigerator, different linear equations can be used for fitting. And further obtaining the ROI according to the fitted boundary curve of the edge area of each interlayer.

In one implementation, the boundary curve of the edge region of each interlayer obtained by fitting may be, for example, one boundary curve close to the refrigerator door, and the edge region may correspond to two boundary curves, as shown in fig. 4, and the other boundary curve may also be obtained by fitting in the above manner, or another boundary curve of the edge region may be obtained by moving the corresponding depth to the inside of the refrigerator in parallel, where the depth represents the distance between the two boundary curves, and may be known in advance based on the structure of the refrigerator.

In one implementation, the ROI of the first image of the t-th frame is denoted as A_t＝{A_t ¹,A_t ²,…,A_t ⁿWhere n is the total number of refrigerator compartments, A_t ⁱRepresenting the ith ROI of the first image of the t-th frame, e.g. A_t ¹The 1 st interlayer of the refrigerator in the first image of the t-th frame is shown, and the 1 st layer is the 1 st layer counted from bottom to top of the refrigerator in fig. 3B.

In one implementation, the ROI of the second image is extracted and characterized as B ═ B¹,B²,…,BⁿN is the total number of refrigerator compartments, BⁱRepresenting the ith second ROI in the second image, e.g. B¹Representing the layer 1 of the refrigerator in the second image.

On the basis of the foregoing embodiment, further, if the similarity includes at least one of the following: in one implementation, as shown in fig. 5, the similarity between the ith ROI in the first image of the t-th frame and the corresponding ith ROI in the second image is denoted as D_t ⁱ(A_t ⁱ,Bⁱ)。

For color similarity, color histogram vectors of the ith ROI in the first image and the corresponding ith ROI in the second image are extracted, and the distance (e.g. Euclidean distance) between the two color histogram vectors is calculatedFurther, the distance can be normalized to obtain the color similarity C_t ⁱ(A_t ⁱ,Bⁱ)。C_t ⁱ(A_t ⁱ,Bⁱ) Value range [0,1 ]]，C_t ⁱ(A_t ⁱ,Bⁱ) The larger the value of (a) indicates that the ROI is more similar to the ROI in the second image.

In other embodiments, for example, the cosine similarity of two color histogram vectors may also be calculated. For the structural similarity, the Structural Similarity (SSIM) between the ith ROI in the first image and the corresponding ith ROI in the second image is calculated, and the structural similarity is recorded as S_t ⁱ(A_t ⁱ,Bⁱ)。S_t ⁱ(A_t ⁱ,Bⁱ) Value range [0,1 ]]，S_t ⁱ(A_t ⁱ,Bⁱ) The larger the value of (a) indicates that the ROI is more similar to the ROI in the second image.

For the texture similarity, Local Binary Pattern (LBP) features of an ith ROI in the first image and a corresponding ith ROI in the second image are calculated respectively, then a distance between the two LBP features is calculated, and further the distance can be normalized to obtain the texture similarity T_t ⁱ(A_t ⁱ,Bⁱ)。T_t ⁱ(A_t ⁱ,Bⁱ) Value range [0,1 ]]，T_t ⁱ(A_t ⁱ,Bⁱ) The larger the value of (a) indicates that the ROI is more similar to the ROI in the second image.

Further, at least two items of the obtained color similarity, structure similarity and texture similarity can be weighted to obtain the similarity corresponding to the ROI.

In one implementation, the color similarity, the structure similarity and the texture similarity are weighted, and three corresponding weights w are given₁、w₂、w₃In one embodiment, three weights satisfy the constraint w₁+w₂+w ₃1. Ith R of t frameSimilarity D of OI and corresponding ith ROI in second image_t ⁱ(A_t ⁱ,Bⁱ) Can be calculated as: d_t ⁱ(A_t ⁱ,Bⁱ)＝w₁×C_t ⁱ(A_t ⁱ,Bⁱ)+w₂×S_t ⁱ(A_t ⁱ,Bⁱ)+w₃×T_t ⁱ(A_t ⁱ,Bⁱ). Based on the above calculation, if the t-th frame includes n ROIs, a similarity set { D ] of each ROI corresponding to the first image and the second image can be obtained_t ⁱ(A_t ⁱ,Bⁱ)}，(i＝1,2,…,n)。

On the basis of the above embodiment, further, in the process of accessing the food material by the user, the hand and the food material of the user may block the interlayer of the plurality of layers, particularly the interlayer below the user, so that for the t-th frame image, the similarity corresponding to each ROI may be analyzed from the interlayer on the uppermost layer.

In one implementation, for any frame of the first image, step a, if the similarity corresponding to the ith ROI in the first image is smaller than a similarity threshold, determining that the target ROI existing in the first image is the ith ROI;

b, if the similarity corresponding to the ith ROI in the first image is larger than or equal to a similarity threshold, subtracting i by one, and repeatedly executing the step a until the similarity corresponding to the ith ROI is smaller than the similarity threshold, or ending until i is equal to 1; and i represents the number of layers of the interlayer, the number of layers is arranged from the bottom of the refrigerator to the top, the initial value of i is N, and N is the total number of layers of the interlayer.

In one implementation, different similarity thresholds λ may be set for the interlayers of different layersⁱIf the similarity corresponding to the ith ROI is larger than the set similarity threshold lambdaⁱThen the layer is deemed to have no access activity. The ROI is judged from the uppermost layer one by one, if the Nth layer is determined to have access behavior, the target ROI corresponding to the first image of the frame is considered as the ROI corresponding to the N-th interlayer, and the ROI can be usedSetting a position mark corresponding to a first image of a frame as N; otherwise, the layer N-1 is continuously judged until the bottom layer. And if no access action occurs in each layer finally, setting the position mark corresponding to the first image of the frame as 0.

In summary, if there is a similarity corresponding to an ROI that is smaller than the similarity threshold corresponding to the ROI, it is determined that the target ROI existing in the first image of the frame is the ROI.

If the similarity corresponding to the multiple ROIs is smaller than the similarity threshold value corresponding to each ROI, the target ROI corresponding to the frame first image is determined to be the ROI corresponding to the uppermost interlayer in the multiple ROIs, for example, the similarity corresponding to the 2 nd interlayer and the 3 rd interlayer in the refrigerator is smaller than the similarity threshold value corresponding to each ROI, and the ROI corresponding to the 3 rd interlayer is used as the target ROI of the frame first image. In one implementation the levels may be arranged in order from the bottom of the refrigerator up.

In other embodiments of the present application, the similarity threshold of each ROI may be the same, which is not limited in the present application.

In one implementation, as shown in fig. 6, an access position sequence may be established to store the position mark of the target ROI corresponding to each frame, and the access position sequence may sequentially include: and marking the position of the target ROI corresponding to the first image of each frame. If the target ROI with the similarity smaller than the similarity threshold value does not exist in the first image, the position corresponding to the first image is marked as zero, otherwise, if the target ROI with the similarity smaller than the preset similarity threshold value exists in the first image, the position corresponding to the target ROI in the first access position sequence is marked as the layer number.

Further, the position markers included in the access position sequence may be subjected to a filtering process.

In one implementation, the filtering process may be implemented by:

dividing the first sequence of access positions into at least one subsequence;

for any one of the subsequences, if the number of the position markers which are not zero in the subsequences is smaller than a first number threshold, setting the position markers in the subsequences to be zero;

and if the number of the position marks which are not zero in the subsequence is larger than or equal to the first number threshold, setting the position marks which are zero in the subsequence as the mode of the position marks in the subsequence.

Specifically, in order to reduce the influence of noise in the process of identifying the food material access position, retrospective filtering is performed on the position mark of the target ROI of each frame. And setting the length of a sliding window to be delta n for the obtained access position sequence comprising the position mark of each frame, and performing uncovered sliding from the first frame in the access position sequence, namely dividing the access position sequence into at least one subsequence. If the number of the non-zero position markers in the sliding window (namely, the subsequence) is less than a set first quantity threshold value, setting the position markers corresponding to all frames in the sliding window to be 0; otherwise, the mark of the access position in the sliding window that is 0 is filled, for example, the mark of the access position is 0 and is filled as the mode of the non-zero position mark in the sliding window, as shown in fig. 7, the position mark corresponding to the 11 th frame is filled as 4, and the position mark corresponding to the 32 th frame is filled as 5. In other embodiments of the present application, the number of the spacers may be other values, such as the number of layers of the uppermost spacer, and the like, which is not limited in the present application. The sequence of access positions after the filtering process is shown in fig. 7.

Further, the collected first image may include multiple access processes of the user, for example, if the user opens a refrigerator, puts an apple in the refrigerator, and takes out a cola, the first image may be regarded as two access processes, in the following embodiment, one access process is referred to as one access cycle, and the access position sequence corresponding to each access cycle may be extracted from the access position sequence after the filtering processing. In one implementation, extracting the access position sequence corresponding to each access cycle may be implemented by:

taking a first non-zero position mark before a mark which is zero in the first access position sequence after the filtering processing as an end position mark of an access cycle;

searching a position mark which is in the shortest interval with the first non-zero position mark and is zero in the first access position sequence after the filtering processing, and taking the position mark which is adjacent to the position mark with the shortest interval and is zero as the initial position mark of the access period;

and acquiring a second access position sequence corresponding to the access period from the first access position sequence after the filtering processing according to the start position mark and the end position mark.

Specifically, based on the first access position sequence after the filtering process, each access period can be determined by detecting a non-zero portion between two subsequences with 0 duration. If the position mark of the current frame is 0, backtracking and searching the time which is not 0, and taking the time which is not 0 as the end time t of the access period₁I.e. to find a non-0 position marker, which is taken as the end position marker, from t₁Go on to trace back from time to t₁The time when the position mark with the shortest time interval is 0 is used as the starting time t of the access period₀That is, the position mark with the shortest interval of 0 before the position mark with non-0 is searched, the position mark with 0 is the next non-zero position mark as the initial position mark, and t is used₀To t₁The sequence of access locations between the instants serves as the second sequence of access locations of the access cycle, as shown in fig. 7. If the flag of the current frame is not 0, the target ROI of the first image after the identification is continued.

Further, after the target ROI corresponding to the first image of each frame in each access period is determined, the food material access position corresponding to each access period may be determined, which may specifically adopt the following manner:

and if the number of the at least one position mark in the second access position sequence is larger than the corresponding second number threshold, taking the interlayer of the uppermost layer corresponding to the at least one position mark as the food material access position corresponding to the access cycle.

Specifically, statistical analysis is performed on the position corresponding to the target ROI of each frame in the second access sequence, and finally, the food material access position corresponding to each access period is decided.

For any access cycle, as shown in FIG. 7, forObtained t₀To t₁The number { N of each position mark in the second access position sequence corresponding to the access cycle is counted_i}. Starting to judge the number N of the corresponding frames of each layer from the 5 th interlayer of the highest layer_iAnd a set second number threshold alpha_iThe magnitude relationship of (1). If the number N of the 5 th interlayer is corresponding to the number N of the interlayer₁Greater than a set threshold value alpha₁And if not, continuously checking the size relation between the number corresponding to the 4 th layer and the corresponding second number threshold value until the bottom layer.

In summary, if the number of one position mark is greater than the second number threshold corresponding to the position mark, the interlayer corresponding to the position mark is used as the food material access position corresponding to the access cycle.

And if the number of the position marks is larger than the corresponding second number threshold value, taking the interlayer on the uppermost layer in the interlayers corresponding to the position marks as the food material access position corresponding to the access cycle. For example, if the number of the partitions marked as 4 and 2 is greater than the corresponding second quantity threshold, the partition marked as 4 is used as the food material access position of the access cycle. The position marks are the number of layers corresponding to the inner partition layer of the refrigerator, and in one implementation mode, the number of layers can be arranged in an upward sequence from the bottom of the refrigerator.

In other embodiments of the present application, the second quantity threshold corresponding to each position mark may also be the same, and the present application does not limit this.

Further, the second image as the background image may be updated, specifically, the following method may be adopted:

and if the continuous occurrence frequency of the same target ROI is greater than a second preset frequency threshold value, and the similarity between the target ROIs of two adjacent frames of first images in the multiple frames of first images corresponding to the same target ROI is greater than a preset similarity threshold value, updating the second image into any one frame of first image in the multiple frames of first images corresponding to the same target ROI.

Specifically, if an image acquired when the refrigerator is opened is used as the background image, a relatively large difference may be always shown between the ROI in the first image and the ROI in the corresponding position in the second image in the subsequent similarity determination process, and the identified target ROI may continue to be the ROI corresponding to a certain layer. Therefore, it is necessary to determine whether to update the second image after each round of access process is finished.

If the ROI of the first image has a large difference from the ROI of the corresponding position of the second image, but the difference between the ROIs of the consecutive first images is very small, the second image needs to be updated. Thus, for the update of the second image, it is first determined whether a constraint of the update is fulfilled, e.g. the number of consecutive occurrences of the same target ROI is larger than a second preset number threshold. Further, it is determined whether the similarity between the target ROIs of two adjacent first images in the multiple frames of first images of the same target ROI is greater than a similarity threshold, if so, it is determined that the difference between the ROIs of the consecutive first images is very small, and it is determined that the second image needs to be updated to any one frame of first image in the multiple frames of first images of the same target ROI, for example, to the first frame of first image in the multiple frames of first images. Otherwise, the second image does not have to be updated. The plurality of frames of first images may be part or all of the first images of the same target ROI.

In an implementation manner, it may be further determined whether the constraint condition is satisfied through the position marker in the second access position sequence, that is, it is determined whether the number of consecutive occurrences of a certain position marker in the second access position sequence is greater than a second preset number threshold, and if the certain position marker exists, the similarity between the target ROIs in the multiple frames of the first image corresponding to the position marker is further determined.

For example, if a certain position mark is continuously output for more than a certain time Δ t, for example, the position mark of the target ROI starting from the t-th frame is a layer 2 interlayer, until the position mark of the target ROI of the t + Δ t frame is always a layer 2 interlayer, then it is determined whether the second image update is currently required. And calculating the similarity between the target ROIs of the first images of two continuous frames from the t frame to the t + delta t/4 frame, wherein the similarity calculation method refers to the previous embodiment. If the similarity between the target ROI of each two first images is larger than a set similarity threshold, the second image needs to be updated at the moment, namely, the new second image is updated to the first image of the t-th frame; otherwise, no update of the second image is necessary. However, Δ t/4 is only an example, and any continuous multi-frame first image between the tth frame and the t + Δ t frame may be selected in practical applications.

In summary, the method of the present application uses a machine vision method to realize dynamic identification of the food material access position. And identifying the access position by performing similarity calculation on the ROI corresponding to each interlayer of the image in the access process, and updating the second image serving as the background image when a certain condition is met. The access position of the food material is identified by using an image processing mode, so that inconvenience caused by manual input of the access position is avoided, and the user experience can be greatly improved; the dynamic identification of the position is realized by collecting the image of the food material access process in the access process, compared with a static identification method, the influence of food material shielding on an identification result is reduced, and the identification accuracy is higher; and the background image updating strategy provided in the identification process reduces the influence caused by the non-standard operation of the user in the access process, and improves the robustness of the algorithm.

Fig. 8 is a structural diagram of an embodiment of a refrigerator provided by the present application, and as shown in fig. 8, the refrigerator includes:

a refrigerator main body, an image acquisition unit 801, a processor 802; wherein, the image acquisition unit 801 and the processor 802 are fixedly arranged in the refrigerator main body; the image acquisition unit is positioned at the top of the compartment of the refrigerator;

the image acquisition unit 801 is used for responding to the opening of a refrigerator door body and acquiring a plurality of frames of first images of the food material storage area; the visual angle range of the image acquisition unit comprises the food material storage area;

the above components may communicate over one or more buses.

Optionally, a memory 803 may also be included that stores executable instructions for the processor 802.

Wherein the processor 802 is configured to:

In a possible implementation manner, the number of times that the target ROI corresponding to the food material access position appears in the at least two consecutive frames of the first image is greater than a first preset number threshold.

In one possible implementation, the processor 802 is configured to:

generating a first access position sequence according to a target ROI existing in the first image of each frame; the first access position sequence sequentially comprises position marks of target ROIs existing in the first images of all frames; if the target ROI with the similarity smaller than a preset similarity threshold does not exist in the first image, the position of the target ROI corresponding to the first image in the first access position sequence is marked as zero, and if the target ROI with the similarity smaller than the preset similarity threshold exists in the first image, the position of the target ROI corresponding to the first image in the first access position sequence is marked as the layer number.

In one possible implementation, the processor 802 is configured to:

filtering a position marker of the target ROI included in the first sequence of access positions.

In one possible implementation, the processor 802 is configured to:

dividing the first sequence of access positions into at least one subsequence;

and if the number of the non-zero position markers in the subsequence is greater than or equal to the first number threshold, setting the position markers which are zero in the subsequence as the mode of the position markers in the subsequence.

In one possible implementation, the processor 802 is configured to:

searching a position mark which is in the shortest interval with the first non-zero position mark and is zero in the first access position sequence after the filtering processing, and taking the position mark which is adjacent to the position mark which is in the shortest interval and is zero as the initial position mark of the access period;

In one possible implementation, the processor 802 is configured to:

and if the number of at least one position mark in the second access position sequence is larger than the corresponding second number threshold, taking the interlayer of the uppermost layer corresponding to the at least one position mark as the food material access position corresponding to the access cycle.

In one possible implementation, the processor 802 is configured to:

and if the continuous occurrence frequency of the same target ROI is greater than a second preset frequency threshold value, and the similarity between the target ROIs of two adjacent frames of first images in the multi-frame first images corresponding to the same target ROI is greater than a preset similarity threshold value, updating the second image into any one frame of first image in the multi-frame first images corresponding to the same target ROI.

In one possible implementation, the processor 802 is configured to:

for any frame of the first image, determining initial position information of edge areas of the interlayer of each layer in the first image according to the position of at least one interlayer of the refrigerator;

The refrigerator of this embodiment may be configured to execute the method corresponding to the foregoing method embodiment, and the specific implementation process of the refrigerator may refer to the foregoing method embodiment, which is not described herein again.

Fig. 9 is a block diagram of an embodiment of an electronic device provided in the present application, and as shown in fig. 9, the electronic device includes:

a processor 901, and a memory 902 for storing executable instructions for the processor 901.

Optionally, the method may further include: a communication interface 903 for enabling communication with other devices.

The above components may communicate over one or more buses.

The processor 901 is configured to execute the corresponding method in the foregoing method embodiment by executing the executable instruction, and the specific implementation process of the method may refer to the foregoing method embodiment, which is not described herein again.

The electronic device is, for example, a server, and may communicate with a refrigerator to acquire a captured image of an access process, thereby implementing the method in any of the above embodiments.

The embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the method in the foregoing method embodiment is implemented.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method of managing food materials, comprising:

2. The method of claim 1, wherein the target ROI corresponding to the food material access position appears in the first images of at least two consecutive frames more than a first preset number threshold.

3. The method according to claim 1 or 2, wherein after determining the target ROI present in the first image for each frame, further comprising:

4. The method of claim 3, wherein after generating the first sequence of access locations, further comprising:

5. The method according to claim 4, wherein said filtering the position marker of the target ROI comprised in the first sequence of access positions comprises:

dividing the first sequence of access positions into at least one subsequence;

6. The method according to claim 4, wherein after the filtering the position marker of the target ROI comprised in the first sequence of access positions, further comprising:

7. The method of claim 6, wherein the using, as the food material access position, an uppermost layer of the interlayer corresponding to the target ROI in the at least two frames of the first image comprises:

and if the number of at least one position mark in the second access position sequence is larger than the corresponding second number threshold, taking the interlayer of the uppermost layer corresponding to at least one position mark as the food material access position corresponding to the access cycle, wherein the position mark in the interlayer of the uppermost layer corresponding to at least one position mark is larger than the corresponding second number threshold.

8. The method of claim 1 or 2, further comprising:

9. The method according to claim 1 or 2, wherein before determining the target ROI existing in each frame of the first image according to the similarity between the ROI of the first image and the ROI at the corresponding position in the second image, the method further comprises:

10. A refrigerator, characterized by comprising:

the refrigerator comprises a refrigerator main body, an image acquisition unit and a processor; wherein the image acquisition unit and the processor are fixedly arranged in the refrigerator main body; the image acquisition unit is positioned at the top of the compartment of the refrigerator;

wherein the processor is configured to: