CN113124634A - Refrigerator with a door - Google Patents
Refrigerator with a door Download PDFInfo
- Publication number
- CN113124634A CN113124634A CN201911416098.8A CN201911416098A CN113124634A CN 113124634 A CN113124634 A CN 113124634A CN 201911416098 A CN201911416098 A CN 201911416098A CN 113124634 A CN113124634 A CN 113124634A
- Authority
- CN
- China
- Prior art keywords
- food material
- target
- image
- condition
- controller
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
- F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
- F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
- F25D29/00—Arrangement or mounting of control or safety devices
- F25D29/003—Arrangement or mounting of control or safety devices for movable devices
-
- F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
- F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
- F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
- F25D29/00—Arrangement or mounting of control or safety devices
- F25D29/005—Mounting of control devices
-
- F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
- F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
- F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
- F25D2500/00—Problems to be solved
- F25D2500/06—Stock management
Landscapes
- Engineering & Computer Science (AREA)
- Chemical & Material Sciences (AREA)
- Combustion & Propulsion (AREA)
- Physics & Mathematics (AREA)
- Mechanical Engineering (AREA)
- Thermal Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Cold Air Circulating Systems And Constructional Details In Refrigerators (AREA)
Abstract
The application discloses refrigerator belongs to electron technical field. The refrigerator includes: a cabinet including a storage compartment having an opening; the door is movably connected with the shell and used for shielding the opening; a controller to: determining food material information corresponding to the target food material condition at a first moment of storing a first food material meeting the target food material condition; wherein the food material information comprises at least one of shelf life and volume; at a second moment after the first moment, if the updating condition is met, updating the food material information corresponding to the target food material condition; wherein the update condition includes: storing or taking out food materials meeting the target food material conditions; the shelf life time T corresponding to the updated target food material condition is T0- (T2-T1), T0 represents the shelf life time corresponding to the target food material condition before updating, T1 represents the first time, and T2 represents the second time. The method and the device solve the problem that the information of the food materials determined by the controller is less. The application is used for storing food materials.
Description
Technical Field
The application relates to the technical field of electronics, in particular to a refrigerator.
Background
With the development of electronic technology, the requirements for the use intelligence and convenience of household appliances (such as refrigerators) are increasing, and the refrigerators are required to manage food materials stored therein.
In the related art, a refrigerator includes a storage chamber, a camera, and a controller. When a user accesses food materials in a storage chamber of the refrigerator, the camera can capture images of the food materials, and the controller can identify the images of the food materials to obtain types of the food materials. Further, the controller may determine what food material is stored in the refrigerator.
Since the controller of the current refrigerator can only determine the type of the food material stored in the refrigerator, and the information of the food material determined by the controller is less, the effect of managing the information of the food material in the refrigerator by the controller is poor.
Disclosure of Invention
The application provides a refrigerator, which can solve the problem that information of food materials determined by a controller is less. The refrigerator includes:
a cabinet including a storage compartment having an opening;
the door is movably connected with the shell and used for shielding the opening;
a controller to:
determining food material information corresponding to a target food material condition at a first moment of storing a first food material meeting the target food material condition; wherein the food material information comprises at least one of shelf life and volume;
at a second moment after the first moment, if the updating condition is met, updating the food material information corresponding to the target food material condition;
wherein the update condition includes: storing or taking out food materials meeting the target food material conditions;
the quality guarantee period T corresponding to the updated target food material condition is T0- (T2-T1), T0 represents the quality guarantee period corresponding to the target food material condition before updating, T1 represents the first moment, and T2 represents the second moment;
when the food materials meeting the target food material condition are stored at the second moment, updating the volume V of the food material information corresponding to the target food material condition as V0 plus delta V1; when the food materials meeting the target food material condition are taken out at the second moment, the updated volume V of the food material information corresponding to the target food material condition is V0-delta V2; v0 represents the volume of the food material information corresponding to the target food material condition before updating, Δ V1 represents the volume of the food material satisfying the target food material condition stored at the second time, and Δ V2 represents the volume of the food material satisfying the target food material condition extracted at the second time.
The beneficial effect that technical scheme that this application provided brought includes at least:
in the refrigerator provided by the application, the controller can determine food material information corresponding to the target food material condition when the first food material is stored, namely at least one of the shelf life and the volume of the food material is determined; and after the food materials meeting the target food material condition are accessed at a second moment after the first moment, the food material information corresponding to the target food material condition can be updated. According to the embodiment of the application, the controller can determine the quality guarantee period and the quality guarantee volume of the food materials stored in the storage chamber, so that more food material information of the food materials can be determined by the controller, and the management effect on the information of the food materials in the refrigerator can be further improved.
Drawings
Fig. 1 is a schematic structural diagram of a refrigerator provided in an embodiment of the present application;
fig. 2 is a flowchart of a method for updating food material information according to an embodiment of the present application;
FIG. 3 is a schematic structural diagram of another refrigerator provided in an embodiment of the present application;
fig. 4 is a schematic structural diagram of a sensing module according to an embodiment of the present disclosure;
fig. 5 is a flowchart of another food material information updating method according to an embodiment of the present application;
fig. 6 is a flowchart of a method for determining conditions of a target food material according to an embodiment of the present application.
Detailed Description
To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.
With the development of electronic technology, the requirements for various household appliances are higher and higher. At present, the intelligent management of food materials in the refrigerator is one of the trends of the refrigerator development, and the more food material information of the food materials that the refrigerator can determine, the better the intelligent management effect of the refrigerator on the food materials. At present, a refrigerator can acquire an image of food materials through a camera, and then recognize the image to determine the type of the food materials. However, only the type of food materials stored in the refrigerator is managed, which is far from the requirement of intelligent management, so that the effect of intelligent management of the food materials by the refrigerator is poor at present. The embodiment of the application provides a refrigerator, more food material information of food materials is obtained, and then the intelligent management effect of the food materials can be improved.
Fig. 1 is a schematic structural diagram of a refrigerator provided in an embodiment of the present application. As shown in fig. 1, the refrigerator 10 may include: a cabinet 101, a door 102, and a controller (not shown in fig. 1). The cabinet 101 includes a storage chamber having an opening K; the door 102 is movably connected with the casing 101 for shielding the opening K. Alternatively, the controller may be located anywhere inside the housing 101, enclosed by the housing 101; or the controller may be located anywhere outside the housing 101; or the controller may be located anywhere inside the door 102, enclosed by the door; or the controller may be located anywhere outside of the door 102; or the controller may also be independent of the casing 101 and the door 102, and the setting position of the controller is not limited in the embodiment of the present application. The controller may be used to perform the method of updating food material information as shown in fig. 2. As shown in fig. 2, the method may include:
In the following embodiments of the present application, a shelf life in the food material information corresponding to the target food material condition is referred to as a shelf life corresponding to the target food material condition, and a volume in the food material information corresponding to the target food material condition is referred to as a volume corresponding to the target food material condition.
Wherein the shelf life time T corresponding to the updated target food material condition is T0- (T2-T1), T0 represents the shelf life time corresponding to the target food material condition before updating, T1 represents the first time, and T2 represents the second time.
When the food materials meeting the target food material condition are stored at the second moment, the updated volume V in the food material information corresponding to the target food material condition is V0+ delta V1; when the food materials meeting the target food material condition are taken out at the second moment, the volume V in the food material information corresponding to the updated target food material condition is V0-delta V2; v0 represents the volume in the food material information corresponding to the target food material condition before updating, Δ V1 represents the volume of the food material satisfying the target food material condition stored at the second time, and Δ V2 represents the volume of the food material satisfying the target food material condition extracted at the second time.
It should be noted that, in the embodiment of the present application, the food materials meeting the target food material condition may be regarded as the two food materials being the same, and the two food materials are the same food material.
In the embodiment of the application, when a certain food material meeting the target food material condition is accessed in the storage chamber, the controller can update the food material information corresponding to the target food material condition. For example, when accessing the same food material as the first food material stored in the storage chamber, the controller may update the recorded shelf life and volume corresponding to the target food material condition. The shelf life corresponding to the updated target food material type is shorter than the shelf life before updating. When the food materials identical to the first food material are stored, the volume corresponding to the updated target food material condition is larger than the volume before updating; when the same food material as the first food material is taken out, it can be considered that at least part of the first food material is taken out, and the volume corresponding to the updated target food material type is smaller than the volume before updating.
In summary, in the refrigerator provided in the embodiment of the present application, the controller may determine the food material information corresponding to the target food material condition when the first food material is stored, that is, determine at least one of the shelf life and the volume of the food material; and after the food materials meeting the target food material condition are accessed at a second moment after the first moment, the food material information corresponding to the target food material condition can be updated. According to the embodiment of the application, the controller can determine the quality guarantee period and the quality guarantee volume of the food materials stored in the storage chamber, so that more food material information of the food materials can be determined by the controller, and the management effect on the information of the food materials in the refrigerator can be further improved.
Optionally, with continuing reference to fig. 1, in the embodiment of the present application, the refrigerator 10 may further include a camera 103 for capturing an image at the opening K. The controller may be in communication with the camera device 103 for acquiring images captured by the camera device 103. It should be noted that fig. 1 illustrates the image capturing apparatus 103 as being located at the top of the chassis 101, alternatively, the image capturing apparatus 103 may also be located at other positions, such as the top of the door 102 or the top of the storage room, which is not limited in this embodiment of the present application. It should be noted that the "top" of a structure described in the embodiments of the present application is the end of the structure away from the ground when the refrigerator is placed on the ground for normal use.
The refrigerator provided by the embodiment of the application can further comprise a memory, a door switch detector, a display screen, a loudspeaker and a microphone, wherein the memory, the door switch detector, the display screen, the loudspeaker and the microphone can be in communication connection with the controller.
Alternatively, the memory may be provided in the cabinet or the door, or may be independent of the cabinet and the door. The memory can be used for storing images collected by the camera equipment, identification results obtained by identifying the food material images by the controller and other information needing to be stored. Alternatively, a door opening and closing detector may be used to detect whether the door is in an open state or a closed state. For example, a door opening and closing detector may be provided to a position in the housing capable of contacting the door, and the door is determined to be in an open state when the door opening and closing detector does not contact the door, and the door is determined to be in a closed state when the door opening and closing detector contacts the door. Alternatively, the display screen may be provided on a surface of the door of the refrigerator remote from the cabinet. The controller may control the display screen to display the recognition result of the food material image or other information. The display screen can also be a touch display screen, and at this time, a user can interact with the refrigerator through the touch display screen, for example, the user can perform touch on the touch display screen to trigger the controller to generate a corresponding instruction, and execute the generated instruction.
Alternatively, the speaker and microphone may be provided on the cabinet or on the door. For example, the speaker and the microphone may be provided at the same position as the image pickup apparatus. Like speaker, microphone and camera equipment can constitute the perception module jointly, and this perception module sets up at the top of casing. The controller can control the loudspeaker to emit voice information to prompt the user to carry out corresponding operation. The controller can control the microphone to collect sound information in the environment where the refrigerator is located so as to generate corresponding instructions.
For example, fig. 3 is a schematic structural diagram of another refrigerator provided in an embodiment of the present application, and the refrigerator shown in fig. 3 may be a left side view of the refrigerator, and fig. 3 shows a state in which a door 102 in the refrigerator is closed. As shown in fig. 3, the refrigerator 10 further includes a driving part 104 and a sensing module 100 located at the top of the cabinet 101, and the sensing module 100 may be connected to the driving part 104. The drive assembly 104 is also communicatively coupled to the controller. Fig. 4 is a schematic structural diagram of a sensing module according to an embodiment of the present disclosure, and fig. 4 is a bottom view of the sensing module 100. As shown in fig. 4, the sensing module 100 includes: an image pickup apparatus 103, a speaker 105, and a microphone 106, the image pickup apparatus 103 including a depth camera 1031 and a color camera 1032. As shown in fig. 3, the field of view of the image pickup apparatus 103 is a conical region (region between two broken lines in fig. 3) having the image pickup apparatus 103 as a vertex. As shown in fig. 4, the microphone 106 may be a linear 4-microphone array, and the speaker 105 is located at the side of the sensing module, so that the distance between the microphone 106 and the speaker 105 can be increased.
Alternatively, the controller may control the driving part 104 to move the sensing module 100 according to the state (open state or closed state) of the door 102 or a voice command issued by a user. For example, when the controller determines that the door 102 is in the open state, the driving part 104 may be controlled to push the sensing module 100 out in a direction approaching the door 102, and when the sensing module is pushed out to a predetermined position, the driving part 104 may be controlled to stop the pushing-out action. And then triggers the image pickup device 103 in the sensing module 100 to work. When the controller determines that the door 102 is in the closed state, the image pickup apparatus 103 may be controlled to stop operating, and the driving part 104 may be controlled to retract the sensing module 100 in a direction away from the door 102, and when the sensing module is retracted to the home position, the driving part 104 may be controlled to stop the retracting action.
Optionally, the image capturing apparatus in the embodiment of the present application may include at least one of a depth camera and a color camera. The color camera can be a common color camera or a wide-angle color camera; the depth camera may be a binocular camera, a structured light camera, or a time of flight (TOF) based camera. The image captured by the depth camera may be a depth image (also referred to as range image), and the image captured by the color camera may be a color image. The pixel value of the pixel point in the depth image is the distance (also called depth) from the point corresponding to the pixel point to the depth camera in the scene covered by the visual field range of the depth camera, and the pixel value of the pixel point in the color image is a gray value. Alternatively, the frequency with which the depth camera and the color camera capture images may be the same, and the minimum time interval between the capture of the depth image and the capture of the color image may be less than the duration threshold. The multi-frame depth images collected by the depth camera and the multi-frame color images collected by the color camera can be in one-to-one correspondence, and the color image corresponding to each frame of depth image is the color image collected at the collecting time which is closest to the collecting time of the frame of depth image in the multi-frame color images. Furthermore, the scene difference between the corresponding depth image and the scene represented by the color image can be ensured to be small. For example, the depth camera and the color camera may each capture an image at the same time.
The pickup mode of the microphone can be selected according to the pickup distance, and can be near-field pickup or far-field pickup. A microphone can generally collect sound information within three meters in near-field sound pickup, and a microphone can generally collect sound information within a range of three meters to five meters in far-field sound pickup. The microphone can include native recording module when the near field picked up sound, also can include single microphone and disappear the module again. The microphone can include the module that includes when near field pickup when far field pickup, can also include many microphone module such as the six microphone modules of linear four microphone module one-level annular etc..
The controller may include a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), or a combination of a CPU and a GPU. The processor may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a Programmable Logic Device (PLD), or a combination thereof. The PLD may be a Complex Programmable Logic Device (CPLD), a field-programmable gate array (FPGA), a General Array Logic (GAL), or any combination thereof. It should be noted that the controller in the embodiment of the present application may be disposed in the refrigerator, or may be disposed in other devices, and it is only necessary to ensure that the controller can be in communication connection with each component in the refrigerator, and can control each component.
The memory is connected to the controller through a bus or other means, and at least one instruction, at least one program, a code set, or an instruction set is stored in the memory, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the controller to implement the method provided by the embodiment of the present application. The memory may be a volatile memory (or a nonvolatile memory), a non-volatile memory (or a combination thereof). The volatile memory may be a random-access memory (RAM), such as a Static Random Access Memory (SRAM) or a Dynamic Random Access Memory (DRAM). The nonvolatile memory may be a Read Only Memory (ROM), such as a Programmable Read Only Memory (PROM), an Erasable Programmable Read Only Memory (EPROM), and an electrically erasable programmable read-only memory (EEPROM). The non-volatile memory may also be a flash memory, a magnetic memory, such as a magnetic tape, a floppy disk, or a hard disk. The non-volatile memory may also be an optical disc.
Fig. 5 is a flowchart of another food material information updating method provided in the embodiment of the present application, and the method can be applied to a controller. As shown in fig. 5, the method may include:
Optionally, any food material condition may include: conditions of food material types, or conditions of food material types and conditions of food material states; the food material status may comprise a processed status or an unprocessed status. For example, apple, banana, and the like may be food material types of food materials. The certain food material condition comprises the following food material types: the food material type is apple, and the food material state conditions are as follows: if the food material state is an unprocessed state, the food material satisfying the food material condition is an unprocessed apple.
In the following embodiments, the target food material conditions are described as examples, including conditions of food material types and conditions of food material states. In the embodiment of the present application, the food materials satisfying the target food material condition are not stored in the storage chamber before the first time, and the controller does not manage the food material information corresponding to the target food material condition. The food material information corresponding to the target food material condition may be information of a food material satisfying the target food material condition.
The controller can control the camera device to acquire images at the opening of the storage room. The visual field range of the camera equipment at least needs to cover the opening of the storage chamber, and the food materials inevitably need to pass through the opening when the user accesses the food materials in the refrigerator, so that the camera equipment can be ensured to acquire images in the process of accessing the food materials by the user.
Alternatively, when the controller detects that the door is in an open state through the door switch detector, the controller may control the driving part to push out the sensing module, and then control the image pickup apparatus to start capturing an image. The controller may control the camera device to continuously acquire a certain number of images per second. When the controller determines that the door is changed from the open state to the closed state, the controller may control the image pickup apparatus to stop capturing images. Thus, in the process that the door is changed from the closed state to the open state and then to the closed state, the image pickup device can complete image pickup in one pickup cycle. It should be noted that, in the embodiment of the present application, the image capturing apparatus captures an image under the control of the controller as an example for explanation. Optionally, the image capturing apparatus may also perform image acquisition without being controlled by the controller, which is not limited in this embodiment of the application.
Optionally, in this embodiment of the present application, the image capturing apparatus includes a depth camera and a color camera, and the depth camera and the color camera may capture images simultaneously, so that the images captured by the image capturing apparatus at the same capture time include a depth image and a color image corresponding to the depth image. The image collected by the camera device at the opening of the storage room comprises a group of depth images and a group of color images which are in one-to-one correspondence with each frame of depth image in the group of depth images, and the group of depth images comprises a plurality of frames of depth images.
The multi-frame first image in the embodiment of the application may be an image acquired by the image pickup apparatus in one acquisition period. The controller may perform image recognition based on the multiple frames of the first images, and further determine the food material type and the food material state of the first food material in the multiple frames of images to obtain the condition of the food material type and the condition of the food material state satisfied by the first food material, and then determine the target food material condition satisfied by the first food material. The food material state comprises a processed state or an unprocessed state. And when the controller cannot determine the food material type or the food material state of the first food material only in an image recognition mode, the controller can play a prompt voice prompting the user to input through the controller loudspeaker so as to determine the food material type or the food material state of the first food material in a user input mode. It should be noted that, a manner of determining the target food material condition satisfied by the first food material is described in detail later, and details of the embodiment of the present application are not described herein.
For example, if the controller determines that the food material type of the first food material is apple and the food material state is an unprocessed state, the controller may determine that the food material type of the target food material conditions is apple and the food material state conditions are: the food material state is unprocessed state.
In an embodiment of the application, the controller may further determine a first moment at which the first food material is stored in the storage chamber. For example, the controller may determine the first time by:
in the first mode, the controller may determine any time between the start of the capturing of the first image and the stop of the capturing of the first image by the image capturing apparatus as a first time when the first food material is stored in the storage chamber. That is, the controller may determine any one time in the acquisition cycle in which the first image is acquired as the first time.
For example, the first time may be a time at which the controller determines that the door is changed from the open state to the closed state, or the first time may be a time at which the controller determines that the door is changed from the closed state to the open state. Since the time consumed in the storing process of the user food material is generally less, the error of determining any moment in the storing process as the storing moment of the first food material is less. In addition, the first time is used for determining the shelf life of the food material, and the time required by the user to access the food material can be ignored compared with the shelf life of the food material, so that any time in the acquisition cycle of the first image is determined as the first time for storing the first food material, and the determination of the shelf life cannot be influenced.
In the second aspect, the controller may determine a time when the user's hand is inserted into the storage chamber as a first time when the first food material is stored in the storage chamber.
For example, the controller may determine a movement trajectory of the hand of the user according to the first images of the plurality of frames, and then determine an image when the hand moves to a target position according to the movement trajectory, where the target position is a position farthest from the starting point in the movement trajectory. The controller may consider the target position to be located in the storage compartment, and may determine that the hand extends into the storage compartment when the hand is moved to the target position. Further, the controller may determine a timing at which the image capturing apparatus captures the image as the first timing.
In the second way, the controller may determine, from among the plurality of frames of first images acquired by the image capturing apparatus, a plurality of frames of images including a hand region, and the image including the hand region may be a depth image. Alternatively, the controller may acquire each frame of depth image acquired by the image capturing device in real time, that is, each time a frame of depth image is acquired by the image capturing device, the frame of depth image is acquired. The controller may then perform hand detection on the frame depth image to determine whether the frame depth image is an image that includes a hand region. Optionally, in the depth image acquired by the image pickup apparatus in one acquisition cycle, an image between the image of the first frame including the hand and the image of the last frame including the hand determined by the controller may be both an image including the hand.
After the controller determines the hand region in each frame of depth image, the controller may also determine a centroid position of the hand region in each frame of depth image, and further determine a movement trajectory of the hand by tracking the centroid position, for example, tracking the centroid position of the hand region by an adaptive high-order predictive tracking model. For example, the controller may cluster pixel points of the hand region in the depth image to determine a centroid position of the hand. The controller can also determine one track vector of the hand according to two adjacent images including the hand, and further obtain a plurality of track vectors according to the multi-frame images including the hand. The controller may determine a continuous movement trajectory of the hand based on the trajectory vector obtained by combining the plurality of trajectory vectors.
Optionally, the controller may train a hand recognition model through machine learning or a deep neural network, and then perform hand detection on the depth image by using the trained hand recognition model. The controller may, for example, express features of the hand region based on the depth information from a three-dimensional point cloud of the hand region in the depth image, such as determining a hand feature vector based on depth values of respective points in the image of the hand region. And then, training a hand recognition model by using the hand feature vectors extracted from the multi-frame depth images, and further recognizing the hand region in the image by using the hand recognition model. The three-dimensional point cloud of the hand region is a set of points in the hand region represented by three-dimensional coordinates.
It should be noted that the controller may determine an access status of the first food material according to the plurality of frames of the first image, where the access status may include a food material storing status and a food material taking status. The controller determines that the food material state of the first food material is a storing food material state, namely the controller determines that the user stores the first food material in the storage chamber. The controller determines that the food material state of the first food material is a food material taking-out state, namely the controller determines that the user takes out the first food material from the storage chamber. It should be noted that, in the embodiment of the present application, the controller determines the access state of the first food material as the stored food material state.
For example, the controller may determine, according to a multi-frame depth image including the hand, a multi-frame color image corresponding to the multi-frame depth image, further identify the multi-frame color image, and determine a change condition of the hand state. The hand state comprises a state of not taking food materials or a state of taking food materials. When the controller determines that the hand state is changed from the food material not taken state to the food material taken state according to the multi-frame color images, the access state of the food materials can be determined to be the food material taken state; when the controller determines that the hand state is changed from the food material taking state to the food material non-taking state according to the multi-frame color images, the access state of the food material can be determined to be the food material storing state.
After the controller determines the target food material condition satisfied by the first food material in step 501, the controller may search the food material information corresponding to the target food material condition in a food material information base, where the food material information base may include a corresponding relationship between various target food material conditions (i.e., a combination of a food material type and a food material state) and the food material information. For example, the food material information library may include the corresponding relationship shown in table 1 below. As shown in table 1 below, the food material information corresponding to the target food material condition may include, in addition to the shelf life and the volume: the food material storage method meeting the target food material condition and the reference image comprising the food material meeting the target food material condition are provided. Optionally, the food material information corresponding to the target food material condition may further include the weight of the food material or other information, which is not limited in the embodiment of the present application.
TABLE 1
It should be noted that the food material information corresponding to the target food material condition may include at least one of a shelf-life duration and a volume, and in the embodiment of the present application, the food material information includes both the shelf-life duration and the volume. Optionally, the food material information may also comprise only shelf life or volume.
Assuming that the first food material is unprocessed chives, that is, the food material type of the first food material is chives, and the food material state of the first food material is an unprocessed state, the target food material condition includes the food material type of the chives and the food material state is an unprocessed state. In step 502, the controller may determine that the shelf life corresponding to the target food material condition is 1 to 3 days and the volume is 2000 cubic centimeters according to the correspondence shown in table 1 above. It should be noted that, in the above table 1, the shelf-life time corresponding to the target food material condition is taken as an example of a time length range, optionally, the shelf-life time may also be a fixed time length, or the controller may determine any time length within the shelf-life time range corresponding to the target food material condition as the shelf-life time corresponding to the target food material condition.
It should be noted that, when the controller determines the food material information corresponding to the target food material condition, the food material information may be recorded, so as to manage the information related to the food materials in the refrigerator. For example, the controller may record the food material information in a certain record table stored in the memory. Optionally, the record table may record information such as a storage location of the food material in the storage chamber, an access state of the food material, and an access time, in addition to the volume and shelf life of the food material.
For example, the controller may obtain a record table as shown in table 2 below after recording the food material information corresponding to the target food material condition satisfied by the first food material in the record table.
TABLE 2
And 503, determining food material conditions met by the third food material taken out of the storage chamber according to a plurality of frames of third images acquired by the camera at a third time after the first time.
It should be noted that, each time the controller determines that there is a food material taken out of the storage chamber or a food material stored in the storage chamber, the controller can identify the food material to determine the type and state of the food material, and further determine the food material condition satisfied by the food material.
For example, it is assumed that after the first food material is stored in the storage chamber at the first time, the controller determines that the user takes out the third food material from the storage chamber, that is, the access state of the third food material is the food material taking-out state, and the controller may determine that the time when the user takes out the third food material is the third time. Further, the controller can also determine a food material type and a food material status of the third food material to determine the food material condition satisfied by the third food material.
It should be noted that the manner in which the controller determines the access state of the third food material is the same as the manner in which the access state of the first food material is determined, the manner in which the controller determines the access state of the third food material is the same as the manner in which the controller determines the access state of the first food material, the manner in which the controller determines the food material conditions met by the third food material is the same as the manner in which the controller determines the target food material conditions met by the first food material, and details of the embodiment of the present application are omitted.
In the embodiment of the present application, the third food material is also taken as the food material satisfying the target food material condition, that is, the food material condition satisfied by the third food material is taken as the target food material condition.
The controller can determine whether the food material condition satisfied by the third food material is the target food material condition to determine whether the third food material is the same as the first food material, and further determine whether the update condition is satisfied. The updated condition may comprise that the food material satisfying the target food material condition is stored in or taken out from the storage chamber, i.e. the same food material as the first food material is stored in or taken out from the storage chamber. When the food material condition satisfied by the third food material is the target food material condition, the controller may determine that the third food material is the same as the first food material, so that it may be determined that the food material satisfying the target food material condition stored in the storage chamber is changed, and then the food material information corresponding to the target food material condition may be updated.
Illustratively, the shelf life time T corresponding to the updated target food material condition is T0- (T3-T1), T0 represents the shelf life time corresponding to the target food material condition before updating, T1 represents the first time, and T3 represents the third time. When the food material satisfying the target food material condition is taken out of the storage chamber, the volume of the food material satisfying the target food material condition in the storage chamber is reduced, so that the volume V corresponding to the updated target food material condition is V0- Δ V2, V0 represents the volume corresponding to the target food material condition before updating, and Δ V2 represents the volume of the third food material (i.e., the taken food material satisfying the target food material condition). Therefore, the first food material information (i.e., the food material information corresponding to the updated target food material condition) can include T and V. Alternatively, since it is generally difficult to determine the specific volume of the food material, Δ V2 can be set as a fixed value, and the volume of the food material satisfying the target food material condition in the storage chamber can be estimated roughly, so as to achieve fuzzy management of the volume of the food material. Optionally, the specific volume of the third food material may also be measured, and then the accurate volume Δ V2 of the third food material is determined according to the measurement result.
Supposing that the third food material is taken out after the first food material is stored for one day, the first food material and the third food material are unprocessed Chinese chives, namely the first food material and the third food material both meet the target food material condition that the food material type is Chinese chives and the food material state is an unprocessed state; the volume Δ V2 of the third food material is a set fixed value, such as 1000 cubic centimeters. The controller may update the food material information corresponding to the target food material condition, for example, update the record table shown in table 2 above to obtain the record table shown in table 3 below. As shown in table 3 below, the volume of the food material information corresponding to the updated target food material condition may be 1000 cubic centimeters, and the shelf life may be 2 days.
TABLE 3
And 505, determining the food material conditions met by the second food material stored in the storage chamber according to a plurality of frames of second images acquired by the camera at a second moment after the third moment.
For example, it is assumed that after the third food material is taken out of the storage chamber at the third time, the controller determines that the user stores the second food material into the storage chamber, that is, the access state of the second food material is the food material storing state, and the controller may determine that the time when the user stores the second food material is the second time. Further, the controller can also determine the food material type and food material status of the second food material to determine the food material condition satisfied by the second food material.
It should be noted that the manner in which the controller determines the access state of the second food material is the same as the manner in which the access states of the first food material and the third food material are determined, the manner in which the controller determines the access state of the second food material is the same as the manner in which the controller determines the access states of the first food material and the third food material, and the manner in which the controller determines the food material conditions met by the second food material is the same as the manner in which the controller determines the food material conditions met by the first food material and the third food material, which is not described in detail in the embodiments of the.
The controller can determine whether the food material condition satisfied by the second food material is the same as the food material condition satisfied by the third food material to determine whether the second food material is the same as the third food material. Further, the controller may determine whether the food material stored in the storage chamber satisfying the target food material condition is changed again.
And 507, determining the food material information corresponding to the food material conditions met by the second food material.
The controller determines that the food material conditions met by the second food material are different from the food material conditions met by the third food material, and then the controller can determine that the second food material is different from the third food material, and further can determine the food material information corresponding to the food material conditions met by the second food material, and record the food material information corresponding to the food material conditions. The controller can determine that the food material (e.g., the second food material) satisfying the food material condition is stored in the storage chamber according to the record, and further manage the food material information of the food material satisfying the food material condition.
And step 508, determining whether the time difference between the second time and the third time is less than a duration threshold. When the time difference between the second time and the third time is smaller than the duration threshold, execute step 509; when the time difference between the second time and the third time is greater than or equal to the duration threshold, step 507 is executed.
When the food material condition satisfied by the second food material is the same as the food material condition satisfied by the third food material (i.e., both the target food material conditions), the controller may determine that the second food material is the same as the third food material, and thus may determine that the food material satisfying the target food material condition stored in the storage chamber has changed. At this time, the second food material is the food material which is stored at the second moment and meets the target food material condition. Further, the controller may further determine a specific change of the food material satisfying the target food material condition. For example, the controller may determine a time difference between the second time and the third time, and further determine a specific change condition of the food material satisfying the target food material condition according to a relationship between the time difference and the time length threshold.
When the time difference between the second time and the third time is greater than or equal to the duration threshold, the controller may determine that a second food material identical to the third food material is placed after the third food material is taken out of the storage chamber for a longer time, and may further consider the second food material as a food material to be purchased again. At this time, the food material information corresponding to the food material condition satisfied by the second food material may be redetermined, and the food material information corresponding to the food material condition satisfied by the second food material is recorded. Optionally, since the food material condition satisfied by the second food material is the target food material condition, the controller may update the food material information (i.e., the first food material information) corresponding to the recorded target food material condition at this time, and update the food material information to the food material information corresponding to the target food material condition in the food material information base.
When the time difference between the second time and the third time is less than the duration threshold, the controller may determine that the second food material identical to the third food material is put in after a short time after the third food material is taken out of the storage chamber. Since the user usually takes out the food material from the storage chamber, if the food material is not eaten, the user stores the remaining food material in the refrigerator again in a short time. Therefore, when the time difference between the second time and the third time is smaller than the duration threshold, the controller may determine that the second food material is likely to be the remaining food material in the extracted third food material.
For example, the controller may control the speaker to play a first prompt voice confirming whether the second food material is at least part of the third food material, so as to determine whether the second food material is the remaining food material in the taken-out third food material. For example, when the controller determines that the time difference between the second time and the third time is smaller than the duration threshold, the controller may send a voice playing instruction to the speaker, where the voice playing instruction may carry specific content of the first prompt voice, so that the speaker may play the first prompt voice for determining whether the second food material is at least part of the third food material according to the voice playing instruction. If the first prompt voice is "please confirm whether the food is the food taken out last time".
And step 510, acquiring a first response voice collected by the microphone within a first time length after the first prompt voice is played by the loudspeaker. Step 511 is performed.
Illustratively, the first duration may be 10 seconds, 20 seconds, or other duration. The first response voice can indicate whether the second food material is at least part of the third food material. After hearing the first prompt voice played by the loudspeaker, the user can speak the first response voice. For example, the first response voice is "yes", and the first response voice may indicate that the second food material is at least part of the third food material. Or the first response voice is "no", at which time the first response voice may indicate that the second food material is not at least part of the third food material, and the second food material is the same food material as the third food material that is newly purchased.
After the first response voice is acquired, the controller may analyze the first response voice, and may determine information indicated by the first response voice in combination with the first prompt voice.
And step 512, updating the food material information corresponding to the target food material condition to obtain second food material information.
When the controller determines that the first response voice indicates that the second food material is at least part of the third food material, the controller may determine that the food material meeting the target food material condition stored in the storage chamber is changed again, and may update the food material information corresponding to the target food material condition.
Illustratively, the shelf life time T corresponding to the updated target food material condition is T0- (T2-T3), T0 represents the shelf life time corresponding to the target food material condition before updating, T2 represents the second time, and T3 represents the third time. When the food materials satisfying the target food material condition are stored in the storage chamber, the volume of the food materials satisfying the target food material condition in the storage chamber increases, so that the volume V corresponding to the updated target food material condition is V0 +. DELTA.V 1, V0 represents the volume corresponding to the target food material condition before updating, and DELTA.V 1 represents the volume of the second food material (i.e., the stored food materials satisfying the target food material condition). Here, Δ V1 may be a fixed value or may be a specific volume of the second food material. Therefore, the second food material information (i.e., the food material information corresponding to the updated target food material condition) can include T and V. It should be noted that, at this time, the shelf life time T0 corresponding to the target food material condition before updating is the shelf life time in the first food material information, that is, the shelf life time T after updating in step 504; at this time, the volume V0 corresponding to the target food material condition before updating is the volume in the first food material information, that is, the volume V updated in step 504. Alternatively, in the embodiment of the present application, when Δ V1 may be a fixed value set together with Δ V2, Δ V1 may be equal to Δ V2, or Δ V1 may not be equal to Δ V2.
Optionally, in the embodiment of the present application, the controller may further determine in real time whether the quality guarantee period corresponding to the target food material condition is less than the time limit value, and control the speaker to play a third prompt voice indicating the quality guarantee period corresponding to the target food material condition when it is determined that the quality guarantee period is less than the time limit value. And then the user can determine that the food materials meeting the target food material condition are going to deteriorate when hearing the third prompt voice, and further can process the food materials in time.
Optionally, in this embodiment of the application, when the third food material is taken out, the controller may control the speaker to play a fourth prompt voice for determining whether the remaining food materials still exist in the storage chamber, where the food material conditions satisfied by the remaining food materials are the same as the food material conditions satisfied by the third food material. The fourth prompting voice can be used for prompting the user to confirm whether the food material identical to the third food material still exists in the storage chamber, namely, whether the third food material comprises the food material indicated by all the target food material information stored in the storage chamber. For example, the food material type of the third food material is apple, the food material status is processed status, and the fourth prompt voice may be "whether there are processed apples in the refrigerator".
Thereafter, the controller may obtain a fourth response voice collected by the microphone within a fourth time period after the fourth prompt voice is played by the speaker. The fourth response voice is used to indicate whether there is remaining food material in the storage room. For example, after hearing the fourth prompt voice, the user may speak a fourth response voice indicating whether there are remaining food materials in the storage room. If the fourth response voice is "present", the fourth response voice may indicate that the remaining food materials still exist in the storage room; if the fourth response voice is "absent", the fourth response voice may indicate that there is no remaining food material in the storage chamber, and the controller may determine that all the processed apples stored in the refrigerator have been taken out at this time.
Next, when the controller determines that there is no remaining food material in the storage chamber based on the fourth response voice, the controller may delete the food material information corresponding to the target food material condition. For example, when the food materials satisfying the target food material condition are not stored in the storage chamber, the controller does not need to manage the food material information of the food materials, so the controller can delete the food material information corresponding to the target food material condition.
In the embodiment of the present application, the food material conditions satisfied by the food materials are described as an example, where the food materials include both the type of the food material and the state of the food material. Optionally, the food material conditions met by the food material may also only include the type of the food material, for such a case, the controller only needs to omit the step of determining the food material state of the food material in the method shown in fig. 5, and other steps are the same as those in the method shown in fig. 5, which is not described herein again in the embodiments of the present application.
It should be noted that, in the embodiment of the present application, only the target food material condition includes the condition of the food material type and the condition of the food material state. The combined controller for one food material type and food material state can record only one food material information as an example. For example, only one correspondence between the processed apples and the food material information may exist in the stored record table. Optionally, the target food material condition may also include conditions of other information of the food material, such as conditions of a storage location. At this time, for each combination of the food material type, the food material state and the storage position, the controller can record the corresponding food material information. The record table as stored may include the correspondence of processed apples stored in the first layer of the refrigeration compartment to food material information and the correspondence of processed apples stored in the second layer of the refrigeration compartment to food material information. Wherein the storage location is a condition of the first floor of the cold storage compartment and the storage location is a storage location of the second floor of the cold storage compartment that is respectively one of the two target food material conditions. Optionally, the target food material condition may further include conditions of other information of the food material, and the correspondence between the target food material condition and the food material information recorded by the controller when the conditions of other information are included can be analogized according to the correspondence when the conditions of the storage location are included, which is not described in detail in the embodiment of the present application.
It should be noted that, in the embodiment of the present application, the first food material, the second food material, and the third food material are only used for distinguishing food materials in different access scenes, and names of the first food material, the second food material, and the third food material may be arbitrarily replaced. The first time, the second time and the third time in the embodiment of the application are only used for distinguishing different times, and the names of the first time, the second time and the third time can be replaced arbitrarily. In the above embodiment of the application, the second food material is stored at the second time, and optionally, the time when the food material meeting the target food material condition is taken out may also be referred to as the second time.
It should be noted that the content of each prompt voice and each response voice in the embodiment of the present application is only an exemplary description, and the content of the prompt voice and the content of the response voice in practical application may be arbitrarily changed, which is not limited in the embodiment of the present application.
In summary, in the refrigerator provided in the embodiment of the present application, the controller may determine the food material information corresponding to the target food material condition when the first food material is stored, that is, determine at least one of the shelf life and the volume of the food material; and after the food materials meeting the target food material condition are accessed at a second moment after the first moment, the food material information corresponding to the target food material condition can be updated. According to the embodiment of the application, the controller can determine the quality guarantee period and the quality guarantee volume of the food materials stored in the storage chamber, so that more food material information of the food materials can be determined by the controller, and the management effect on the information of the food materials in the refrigerator can be further improved.
The manner in which the controller determines the target food material condition that is met by the first food material in step 501 is described below. For example, the controller may determine the target food material condition by performing the method shown in fig. 6. It should be noted that the manner of determining the target food material condition satisfied by the first food material in fig. 6 can also be used to determine the food material condition satisfied by any food material, such as determining the food material conditions satisfied by the second food material and the third food material. As shown in fig. 6, the method may include:
The food material image may be a color image. The specific manner for determining the at least one frame of food material image will be described in detail later, and this embodiment of the present application is not described herein again.
The controller may directly input the food material image into the recognition model after each frame of food material image is determined, so as to obtain a recognition result of the frame of food material image output by the recognition model. Optionally, after determining all food material images in one acquisition period of the image pickup device, the controller may further input each frame of food material image into the identification model, so as to obtain an identification result of each frame of food material image output by the identification model.
The identification model in the embodiment of the application can be used for outputting the food material type and the food material state based on the input food material image. Optionally, the recognition model may also be used to: and outputting the type of the food material, the state of the food material and the confidence coefficient based on the input food material image, wherein the confidence coefficient can be the common confidence coefficient of the type of the output food material and the state of the food material. In the embodiment of the present application, the confidence corresponding to the combination of the food material type and the food material state is referred to as "confidence". For example, the recognition result of the recognition model for recognizing and outputting a certain frame of food material image may include: the food material type is apple, the food material state is a raw state, and the confidence coefficient is 60%.
It should be noted that, in the recognition result of at least one frame of food material image output by the recognition model, each recognition result may include a food material type, a food material state and a confidence level, or a partial recognition result may include only a food material type and a confidence level but not a food material state, or a partial recognition result may not include a food material type and a food material state. It should be noted that, when the identification result does not include the type of the food material, the identification result necessarily does not include the state of the food material. The partial recognition result only comprising the type of the food material and the confidence degree represents that the recognition model does not recognize the food material state of the first food material in the food material image. The partial recognition result excluding the food material type and the food material state indicates that the recognition model does not recognize the food material type of the first food material in the food material image, and at this time, the recognition result output by the recognition model may include information indicating that the food material type is not recognized.
It should be noted that, when the recognition model recognizes a frame of food material image, multiple combinations of food material types and food material states and the confidence corresponding to each combination may be output, and at this time, the recognition model may output the maximum confidence and the combination corresponding to the maximum confidence as a recognition result.
Optionally, in this embodiment of the application, the recognition model may be any one of network structures such as a deep neural network, a convolutional neural network, a deep belief network, a deep stack neural network, a deep fusion network, a deep recurrent neural network, a deep bayesian neural network, a deep generation network, and deep reinforcement learning, or a derivative model thereof.
The controller in the embodiment of the application can further calculate or judge according to the recognition result of at least one frame of food material image output by the recognition model, and further determine a target recognition result, namely a final recognition result obtained by recognizing the food material type and the food material state of the first food material stored by the user.
When the controller determines that the recognition result of the at least one frame of food material image does not have a recognition result including the type of food material, the controller may directly determine that the target recognition result is the type of food material without recognizing the food material image, and the target recognition result does not include any food material type and food material state.
When the controller determines that there is at least one recognition result including the food material type in the recognition results of the at least one frame of food material image in step 602, the controller may generate a target recognition result including the food material type. For example, the controller may determine the target recognition result in the following two determination manners.
In the first determination, the target recognition result may be a recognition result with the highest confidence degree included in the at least one recognition result.
For example, the controller determines five frames of food material images including the first food material, wherein the food material type in the recognition results of the three frames of food material images is a, the food material state is a processed state, and wherein the confidence in each recognition result is 0.5; the food material type in the recognition result of one frame of food material image is B, the food material state is a raw state, and the confidence coefficient in the recognition result is 0.8; and the recognition result of one frame of food material image only comprises the food material type without the food material state, the food material type is C, and the confidence coefficient of the food material type C in the recognition result is 0.7. The controller may determine that the highest confidence of the five recognition results is 0.8, and further, the controller may generate a target recognition result including the food material type B and the unprocessed state.
In the second determination method, since there may be combinations of food material types and food material states in part of the recognition results in the at least one recognition result that are the same, the controller may determine a comprehensive confidence of each combination according to the confidence of each food material type and food material state combination in each recognition result, and further generate a target recognition result according to the comprehensive confidence of each combination. In addition, in the embodiment of the present application, when a certain identification result only includes a food material type and does not include a food material state, the food material type is referred to as a combination in the identification result.
For example, suppose that the at least one recognition result comprises x recognition results, the x recognition results can comprise y combinations of food material types and food material states, and y is greater than or equal to 1 and less than or equal to x. For each of the y combinations, the controller may determine the most favorable combined confidence according to a target formula. The target formula isWherein s represents the number of the x recognition results including the recognition result of the combination, QaAnd representing the confidence coefficient in the a-th recognition result comprising the combination in the x recognition results, wherein a is more than or equal to 1 and less than or equal to s. Further, the controller may generate the target recognition result including a combination with the highest integrated confidence among the y combinations.
For example, the controller determines five frames of food material images including the first food material, wherein the food material type in the recognition results of the three frames of food material images is a, the food material state is a processed state, and wherein the confidence in each recognition result is 0.5; the food material type in the recognition results of two frames of food material images is B, the food material state is a raw state, and the confidence coefficient in each recognition result is 0.8. According to the target formula, the comprehensive confidence coefficient Q of the combination of the food material type A and the processed state can be calculatedAThe combined confidence of the combination of food material type B and raw state is Q, 0.5+0.5 × 3/5, 0.90B=(0.8+0.8)×2/5=0.64。QA>QBAccordingly, the controller may generate a target recognition result including the food material type a and the processed state.
When the target recognition result does not include the food material type, the controller may determine that the food material type of the first food material cannot be determined in an image recognition manner, and then the controller may control the speaker to play the second prompt voice so as to determine the food material type of the first food material in a user input manner.
For example, the second prompt voice may prompt the user to input the food material type of the target food material in the form of voice. For example, the second prompt voice may be "what material you are depositing". It should be noted that the content of the second prompting voice in the embodiment of the present application is only an exemplary description, and the content of the second prompting voice in practical application may be changed arbitrarily, which is not limited in the embodiment of the present application.
And 606, acquiring a second response voice collected by the microphone within a second time length after the second prompt voice is played by the loudspeaker. Step 607 is performed.
For example, the second time period may be 10 seconds, 20 seconds, or other time period. The response speech may indicate a food material type of the first food material. When the user hears the second prompt voice played by the speaker, the user may speak a second response voice, for example, the second response voice is "apple".
Step 607, determining the food material type of the first food material based on the second response voice. Step 6011 is performed.
The controller can extract information of the second response voice collected by the microphone to determine the food material type of the first food material indicated by the second response voice. It should be noted that, in the embodiment of the present application, the controller determines the food material type of the first food material in a form of collecting voice through the microphone. Optionally, the controller may also receive the food material type input by the user through the touch display screen. At this time, the second prompt voice can also prompt the user to manually input the food material type of the first food material through the touch display screen on the refrigerator.
And step 609, determining the conditions of the target food materials according to the target identification result.
When the controller determines that the target recognition result simultaneously comprises the food material type and the food material state, the controller can determine that the food material type and the food material state of the first food material are successfully recognized in an image recognition mode, and further can determine the condition of the food material type in the target food material condition according to the food material type in the target recognition result; and determining the conditions of the food material states in the target food material conditions according to the food material states in the target identification result. If the controller can determine that the conditions of the food material types in the target food material conditions are as follows: the food material type is a food material type of a first food material, and the conditions of the food material state are as follows: the food material state is a food material state of the first food material. That is, the target food material condition satisfied by the first food material comprises: the food material type is a food material type of the first food material, and the food material state is a food material state of the first food material.
When the controller determines that the target recognition result only includes the food material type, the controller may determine that only the food material type of the first food material is successfully recognized through image recognition, so that the food material type in the target recognition result may be determined as the food material type of the first food material.
The food material types of the first reference food material and the second reference food material are both the food material types of the first food material, the food material state of the first reference food material is a processed state, and the food material state of the second reference food material is an unprocessed state.
When the controller determines only the food material type of the first food material and does not determine the food material state of the first food material, the controller can continue with the determination of the food material state of the first food material. For example, the controller may determine which of the first food material and the first reference food material is more similar according to the similarity between the first food material and the first reference food material and the similarity between the first food material and the second reference food material, and further determine the food material state of the first food material.
The similarity between two food materials can be represented by the similarity between images containing the two food materials. The food material state of the first reference food material is a processed state, and the food material state of the second reference food material is an unprocessed state. If the first food material is more similar to the first reference food material, determining that the food material state of the first food material is a processed state; if the first food material is more similar to the second reference food material, the food material status of the first food material can be determined to be an unprocessed status.
For example, the controller may determine similarity between each frame of food material image in the at least one frame of food material image and each frame of first reference image in the at least one frame of first reference image, to obtain at least one first reference similarity, and then determine a maximum similarity among the at least one first reference similarity as the first similarity. The first reference image comprises a first reference food material. The controller can determine the similarity between each frame of food material image in the at least one frame of food material image and each frame of second reference image in the at least one frame of second reference image to obtain at least one second reference similarity, and further determine the maximum similarity in the at least one second reference similarity as the second similarity. Wherein the second reference image comprises a second reference food material. The food material corresponding to the greater similarity of the first similarity and the second similarity is the food material more similar to the first food material, and the controller can determine the food material state corresponding to the greater similarity as the food material state of the first food material.
Alternatively, the controller may determine the similarity between each frame of food material image and each frame of reference image by any one of a Structural Similarity Index (SSIM) method, a cosine similarity (cosin) determination method, and a histogram-based similarity determination method.
When the controller determines the food material state of the first food material, the controller can generate the target food material condition. The target food material conditions include: the food material type is a food material type of the first food material, and the food material state is a food material state of the first food material.
Referring to the manner of determining at least one frame of food material image in step 601, step 601 may include the following steps 601a to 601 d:
step 601a, acquiring n frames of target images including a hand region in a plurality of frames of first images acquired by the camera equipment.
Wherein n is more than or equal to 1. It should be noted that in the embodiment of the present application, an n-frame target image including a hand region may be determined in a set of depth images acquired by an image capturing apparatus, where the target image is a depth image. Alternatively, the controller may acquire each frame of depth image acquired by the image capturing device in real time, that is, each time a frame of depth image is acquired by the image capturing device, the frame of depth image is acquired. The controller may then perform hand detection on the frame depth image to determine whether the frame depth image is a target image including a hand region. Optionally, the controller may train a classifier through machine learning or a deep neural network, and then perform hand detection on the depth image by using the trained classifier.
In an example, the controller performs hand detection on all depth images acquired by the camera in one acquisition cycle, and then determines n frames of target images including a hand region. Optionally, in the depth image acquired by the image capturing apparatus in one acquisition cycle, the depth image between the first frame target image and the last frame target image determined by the controller may be the target image.
Optionally, in this embodiment of the application, after the at least one frame of target image is determined, a color image corresponding to the at least one frame of target image may also be directly used as the food material image.
Step 601b, at least one frame of auxiliary image is determined in the n frames of target images.
Optionally, the controller may perform filtering on the determined n frames of target images to obtain at least one frame of auxiliary image. Alternatively, the controller may determine the auxiliary image by:
step s11, determine the first frame of target image in the n frames of target images as a frame of auxiliary image.
For example, when the controller determines that a frame of depth image is a target image including a hand region for the first time during hand detection of a depth image acquired by the image capturing apparatus, the controller may directly determine the frame of target image as a frame of auxiliary image.
Step s12, when | hi-hi-1And when | ≧ epsilon, determining the ith frame target image in the n frame target images as a frame auxiliary image.
And the ith frame target image is any frame target image except the first frame target image in the n frame target images. h isiRepresenting the average depth value, h, of the hand region in the target image of the i-th framei-1And representing the average depth value of the hand area in the i-1 th frame target image in the n frames of target images, wherein epsilon represents a depth threshold value, and i is more than or equal to 1 and less than or equal to i and less than or equal to n.
When the controller determines each frame of the target image, the controller may obtain a pixel value of each pixel point in the hand region of the target image, and determine an average depth value of the hand region (that is, an average pixel value of each pixel point in the hand region) according to the pixel value of each pixel point in the hand region. In the present embodiment, the "average depth value of the hand region" is referred to as "hand depth". The controller can record the hand depth of each frame of target image, and further determine the change amplitude delta h ═ h of the hand depth in each frame of target imagei-hi-1And determining whether the change amplitude of the hand depth in each frame of the target image is greater than or equal to a depth threshold epsilon. When the change amplitude of the hand depth in a certain frame of target image is greater than or equal to the depth threshold epsilon, the controller determines the frame of target image as a frame of auxiliary image. The controller may be based on the determinationThe auxiliary image identifies the type of the food material in the image acquired by the camera device.
It should be noted that in the embodiment of the present application, the auxiliary image is determined by the variation range of the hand depth, that is, the target image with a large hand variation may be determined as the auxiliary image, so as to avoid the waste of computing resources caused by identifying all the multiple frame images with a high similarity.
Step s13, when | hi-hi-1And if the frame number of the image between the ith frame target image and the previous frame auxiliary image in the depth image acquired by the camera equipment is greater than or equal to the frame number threshold value, determining the ith frame target image as a frame auxiliary image.
When the change amplitude of the hand in a certain frame of target image is smaller than the depth threshold, the controller can also determine the number of frames of the depth image between the frame of target image and the determined previous frame of auxiliary image in the depth image acquired by the camera equipment, and determine whether the number of frames is larger than or equal to the frame number threshold Deltak. When the number of frames is greater than or equal to the frame number threshold, the frame target image is determined as a frame auxiliary image. That is, when the change width of the hand in the target image is smaller than the depth threshold, the controller determines one frame of auxiliary image every Δ k frame of image. Therefore, the situation that the auxiliary image cannot be determined due to the fact that the moving amplitude of the hand of the user is small all the time can be avoided, and the situation that the image for food material identification can be obtained is guaranteed.
Optionally, in this embodiment of the application, after the at least one frame of auxiliary image is determined, a color image corresponding to the at least one frame of auxiliary image may also be directly used as the food material image.
Step 601c, determining at least one frame of key image in the at least one frame of auxiliary image.
In an embodiment of the application, to reduce the number of images targeted for food material identification to reduce the calculation time, the controller may re-screen the key image from the determined at least one frame of auxiliary image for subsequent image processing and identification.
Alternatively, the controller may determine the key image by:
and step s21, determining a target hand area corresponding to the depth value range of the kth frame auxiliary image from a plurality of hand areas corresponding to the depth value ranges.
Wherein, the k frame auxiliary image is any one frame auxiliary image in the at least one frame auxiliary image. k is larger than or equal to 1, the pixel average value of the hand area of the k frame auxiliary image is positioned in the depth value range of the k frame auxiliary image, and in a plurality of hand areas corresponding to the depth value ranges, the depth value in the depth value range is in negative correlation with the corresponding hand area.
Optionally, the controller may determine m depth intervals according to the height of the refrigerator, where m is greater than 1, and the depth of each position in the scene where the refrigerator is located is the distance from the position to the image pickup device in the refrigerator. Illustratively, each depth zone includes [ Q0, Q1), [ Q2, Q3), …, [ Qm-1, Qm ], where Q0 is 0, and Qm may be greater than or equal to the height of the refrigerator. Alternatively, the lengths of the respective depth intervals may all be equal, for example, the lengths of the respective depth intervals may be 10 centimeters, 20 centimeters, or other values. Optionally, the multiple depth value ranges in the embodiment of the present application may include m depth value ranges in one-to-one correspondence with the m depth intervals, where each depth value range may be the same as the corresponding depth interval.
The depth value ranges in the embodiment of the present application may also correspond to the hand areas one by one, and the depth values in the depth value ranges are inversely related to the corresponding hand areas. The area of the hand corresponding to each depth value range is the area of the hand of the user in the image acquired by the image pickup device when the average depth of the hand of the user is within the depth value range. For example, the hand area in an image may be represented by the total number of pixel points included in the hand region in the image. When the distance between the hand of the user and the camera equipment is short, the hand area in the image acquired by the camera equipment is necessarily large, so the hand area is large; when the distance between the hand of the user and the camera shooting equipment is far, the hand area in the image collected by the camera shooting equipment is necessarily small, so that the hand area is small.
Optionally, when the hand of the user is located at different depths, the controller may cluster areas of hand regions in the multi-frame image collected by the camera device to obtain hand areas corresponding to different depth intervals, and then obtain hand areas corresponding to different depth value ranges.
In this embodiment, when the controller determines a frame of the auxiliary image, the controller may determine, according to the average pixel value of the hand region in the auxiliary image, a depth value range in which the average pixel value of the hand region is located, and further determine a target hand area corresponding to the depth value range.
And step s22, when the ratio of the area of the hand region in the k frame auxiliary image to the target hand area is larger than the ratio threshold, determining the k frame auxiliary image as a frame key image.
In this embodiment, when the controller determines a frame of the auxiliary image, the controller may determine an area of a hand region in the auxiliary image, that is, determine the number of pixels included in the hand region. The controller may then determine a scaling factor corresponding to the frame of auxiliary image, that is, a ratio of an area of a hand region in the frame of auxiliary image to a target hand area corresponding to the frame of auxiliary image. The controller may further detect whether the scale factor is greater than a ratio threshold, and determine the frame of auxiliary image as a frame of key image when the scale factor is greater than the ratio threshold.
Optionally, in the embodiment of the application, after the at least one frame of key image is determined in the auxiliary image, the at least one frame of key image may also be directly input as a food material image into the recognition model to obtain at least one recognition result. And determining a target identification result of the food material image according to the at least one identification result so as to obtain the type of the food material in the food material image.
Step 601d, determining at least one frame of food material image according to the determined at least one frame of key image.
In the embodiment of the application, in order to further improve the identification accuracy, a partial Region of the key image where the food material is taken by the hand may be used as a Region of Interest (ROI), and then the Region of Interest in the color image corresponding to the key image is captured according to the ROI to obtain the food material image, so as to perform subsequent food material identification on the food material image. Optionally, the region of interest in the color image corresponding to the key image is at the same position as the region of interest in the key image of the frame.
Alternatively, the controller may determine the food material image by:
and step s31, acquiring the expansion coefficient corresponding to the depth value range of each frame of key image from the expansion coefficients corresponding to the depth value ranges.
It should be noted that, in step s21, the m depth value ranges determined according to the m depth intervals may also correspond to the m expansion coefficients one by one, and the depth values in the respective depth value ranges may be negatively correlated to the corresponding expansion coefficients.
Step s32, determining the food material area in each frame of key image according to the expansion coefficient corresponding to the depth value range of each frame of key image and the hand area in each frame of key image, where the food material area in each frame of key image includes the hand area in each frame of key image.
Optionally, both the hand region and the food material region in the key image in the embodiment of the application may be rectangular, and the center, the length direction and the width direction of the hand region and the food material region are the same. Optionally, the length w' ═ e of the food material areapWidth h' ═ h × ep(ii) a Where w represents the length of the hand region, h represents the width of the hand region, and p represents the expansion coefficient corresponding to the depth value range of the key image.
And step s33, intercepting the food material area in the color image corresponding to each frame of key image as a frame of food material image corresponding to each frame of key image.
The relation that the length and the width of the food material image satisfy is the same as the relation that the food material area in the frame key image satisfies, and details are not repeated herein in the embodiments of the present application.
When the user accesses the food material, the area of the food material may be larger than the area of the hand, or the food material may stick out of the hand of the user. According to the embodiment of the application, the hand region in the key image is enlarged to determine the food material region, and then the food material region in the color image corresponding to the key image is intercepted as the food material image, so that more food material features can be ensured to be included in the food material image, and the problem of low identification accuracy caused by the fact that only the hand region is intercepted as the food material image and then food material identification is carried out is solved.
It should be noted that, in the embodiment of the present application, a determination method of the reference images (such as the first reference image and the second reference image) stored in the embodiment of the present application is the same as the determination method of the food material image, and details thereof are not repeated in the embodiment of the present application. In addition, only the food material images determined in the multiple frames of first images acquired by the camera equipment are input into the identification model for identification, so that the number of the food material images is small and the characteristics of the food materials are obvious, the time consumption in the identification calculation process can be reduced, and the real-time property of the identification model for identifying the food materials is improved. And when the controller is positioned in other equipment independent from the refrigerator, the data volume of the image transmitted to the controller can be reduced, and the real-time performance of the food material identification by the identification model is improved.
Optionally, after the controller identifies the food material through the identification model, if the target identification result obtained by identification is wrong, the controller may train the identification model again by using the target training data to update the identification model, so as to improve the accuracy of the identification model in identifying the food material image. For example, after the controller generates the target recognition result, the controller may determine whether the target recognition result is correct by acquiring a confirmation voice of the user. When the controller determines that the target recognition result is wrong, the recognition accuracy of the food material image by the recognition model is low, and then the retraining instruction can be received, and the recognition model is trained continuously to optimize the recognition model. The retraining instruction may carry the correct food material type and food material status.
It should be noted that, in the embodiment of the present application, different training methods may be adopted to continuously train the recognition model according to different recognition results output by the recognition model. For example, the recognition model may be determined to be retrained or incrementally trained according to the recognition result output by the recognition model. It should be noted that the retraining of the model is to train the model by using the newly added training data and the historical training data of the model to obtain a new model. And performing incremental training on the model, namely, adjusting parameters related to the newly added training data in the model according to the newly added training data to update the model. Optionally, retraining the model does not change the recognition result that the model can recognize, and only improves the accuracy of the model recognition, while performing incremental training on the model may increase the recognition result that the trained model can output.
Optionally, in the embodiment of the present application, the controller may further determine the food material type of the food material in other manners, and the controller may further determine other information of the food material in the refrigerator, such as a storage location of the food material.
In an optional embodiment of the present application, the refrigerator may further include a code scanning module. The controller can be in communication connection with the code scanning module and is used for acquiring the graphic code acquired by the code scanning module. And then the controller can also confirm the food material type of eating the material according to the figure code that the yard module of sweeping gathered. Illustratively, the graphic code may include a bar code or a two-dimensional code, or the like. The code scanning module can be positioned on the surface of the door close to the opening for illustration, and can also be positioned on the surface of the door far away from the opening. Alternatively, the code scanning module may also be located at any other position on the casing or the door, which is not limited in this embodiment of the application.
Optionally, the controller may further perform the following steps to determine the food material type of the first food material. The following manner of determining the food material type of the first food material may also be used to determine the food material types of the second food material and the third food material.
Step 901, determining an access state of the first food material and an identification state of the first food material according to the multiple frames of target images.
It should be noted that, after the controller determines multiple frame target images in step 601a, the controller may execute step 901. The access state comprises a food material storing state or a food material taking state, and the identification state comprises a graphic code carrying state or a graphic code not carrying state.
For example, the controller may identify the plurality of frames of target images to determine whether an image including a graphic code exists in the plurality of frames of target images. When the image comprising the graphic code exists in the multi-frame target image, the controller can determine that the identification state of the food material is the state carrying the graphic code; when the image including the graphic code does not exist in the multi-frame target image, the controller may determine that the identification state of the food material is a state without carrying the graphic code.
The controller can determine a multi-frame color image corresponding to the multi-frame depth image according to the multi-frame depth image including the hand, further recognize the multi-frame color image, and determine the change condition of the hand state. The hand state comprises a state of not taking food materials or a state of taking food materials. When the controller determines that the hand state is changed from the food material not taken state to the food material taken state according to the multi-frame color images, the access state of the food materials can be determined to be the food material taken state; when the controller determines that the hand state is changed from the food material taking state to the food material non-taking state according to the multi-frame color images, the access state of the food material can be determined to be the food material storing state.
And 902, detecting whether the code scanning module collects the graphic code within the target time length after the target identification result is determined. When the target time length internal code scanning module after the target identification result is determined acquires the graphic code, executing the step 903; and when the target time length internal code scanning module after the target identification result is determined does not acquire the graphic code, executing the step 906.
It should be noted that, when the controller determines in step 901 that the identification status of the first food material is the status carrying the graphic code, and determines in step 608 that the target recognition result does not include any food material type, the controller executes step 902.
Optionally, when the target recognition result does not include the food material type, that is, the controller does not determine the food material type of the first food material through image recognition, the controller may control the speaker to play an indication voice indicating that the user moves the target food material, so as to place the graphic code of the first food material within the visual field of the code scanning module. When hearing the prompt voice, the user can place the graphic code on the first food material in the visual field range of the code scanning module, so that the code scanning module can collect the graphic code on the first food material. Optionally, the code scanning module may start collecting the graphic code when the door is determined to be in the open state.
And step 903, analyzing the graphic code acquired by the code scanning module in the target time length to obtain an analysis result. Step 904 is performed.
It should be noted that any graphic code may correspond to a character string, and a graphic code on an object is generally used to indicate related information of the object. The user (such as a manufacturer of the object) can set the related information of the object indicated by the graphic code by himself, and then the related information indicated by the graphic code is input into the graphic code information base, so that other users can search the related information of the object indicated by the graphic code in the graphic code information base according to the graphic code. It should be noted that the graphic code information base may be stored in a memory in the refrigerator, or may be stored on the internet, or other devices connected to the controller.
The graphic code information base may include information corresponding to a plurality of character strings, and when the controller parses the graphic code, the controller may first determine a target character string corresponding to the graphic code, and then may search information corresponding to the target character string in the graphic code information base. If the information corresponding to the target character string exists in the graphic code information base, the information corresponding to the target character string may be determined as the analysis result of the graphic code, and the analysis result includes the related information of the object (i.e., the target object) indicated by the graphic code. If the information corresponding to the target character string does not exist in the graphic code information base, the analysis result can be determined to be the information of the object which is not indicated by the graphic code, and at this time, the analysis result does not include the related information of the object which is indicated by the graphic code. At this time, the graphic code may not be successfully analyzed, and the controller fails to analyze the graphic code.
Step 904, determining whether the parsing result includes the food material type. When the analysis result includes the food material type, execute step 905; when the parsing result does not include the food material type, step 906 is executed.
Illustratively, the information related to the food material may comprise one or more of a type of food material, a shelf life, a production date, a manufacturer, a volume, a weight, a price, and the like. The controller may analyze the analysis result of the graphic code to determine whether the analysis result includes the food material type, and then determine whether the food material type of the first food material can be determined according to the analysis result.
Step 905, determining the food material type of the first food material according to the analysis result.
When the analysis result of the graphic code includes the food material type, the controller may determine the food material type in the analysis result as the food material type of the first food material. Therefore, the controller can be regarded as analyzing the graphic code on the first food material to obtain the food material type of the first food material.
Step 906, determining whether the access state of the first food material is the food material taking-out state. When the access state of the first food material is the food material taking-out state, execute step 907; when the access status of the first food material is not the food material taking-out status, step 605 is executed.
Step 907, determining the similarity between each frame of food material image and each frame of reference image in at least one frame of reference image to obtain at least one reference similarity. Step 908 is performed.
The reference image comprises the food materials which are stored in the storage room and are marked to be in the state of carrying the graphic codes, namely the reference image is the image of the food materials carrying the graphic codes. The food material type of the food material in the reference image is known.
The controller determines the access status of the first food material as the food material taking status in step 906, that is, determines that an access condition is satisfied, where the access condition includes: the identification state of the first food material is a state carrying the graphic code, the target recognition result does not include the food material type, the graphic code is not collected by the code scanning module within the target time length after the controller determines the target recognition result, and the access state of the first food material is a state of taking out the food material. Therefore, the controller determines that the user takes out the first food material carrying the graphic code from the storage chamber, the food material type of the first food material is not identified in an image identification mode, and the user does not trigger the code scanning module to scan the graphic code of the first food material. Furthermore, the controller can determine at least one frame of reference image of the food material with known food material type stored in the storage chamber, determine the similarity between the food material image containing the first food material and each frame of reference image, obtain at least one reference similarity, and determine the similarity between the two frames of images, namely determine the similarity of the feature vectors of the two frames of images. The similarity between the food material image and the reference image can be represented as follows: the food material type of the first food material is the size of the probability referring to the food material type of the food material in the image.
For example, in step 601, the controller determines five frames of food material images including the first food material, three food materials carrying graphic codes are stored in the storage chamber, and the controller can acquire three frames of reference images of each of the three food materials, so the controller can acquire nine frames of reference images. And the controller determines the similarity between each food material image and each reference image, so that 5 × 9-45 reference similarities can be determined.
Alternatively, the controller may determine the similarity between each frame of food material image and each frame of reference image by any one of a Structural Similarity Index (SSIM) method, a cosine similarity (cosin) determination method, and a histogram-based similarity determination method.
For example, the memory in the refrigerator in the embodiment of the present application may store data related to food materials with graphic codes stored in the storage chamber. Optionally, the related data may include the food material type of the food material, the corresponding character string of the graphic code, the volume or weight, and the corresponding reference image. For example, the memory may store the data related to the food materials shown in table 4 below. The reference images in table 4 below are only used to illustrate that the food related data includes corresponding images, and specific image contents of the images are not illustrated.
TABLE 4
Step 908 determines whether the at least one reference similarity includes a similarity greater than a similarity threshold. When the at least one reference similarity includes a similarity greater than the similarity threshold, performing step 909; when the at least one reference similarity is less than or equal to the similarity threshold, step 605 is executed.
When the similarity between a frame of food material image and a frame of reference image is greater than a similarity threshold, it can be considered that the food material type of the first food material in the food material image is possibly the same as the food material type of the food material in the frame of reference image, the frame of reference image has a reference value for determining the food material type of the first food material, and then the food material type of the first food material can be further determined based on the frame of reference image. When the similarity between a frame of food material image and a frame of reference image is less than or equal to the similarity threshold, it can be considered that the food material type of the first food material in the food material image is unlikely to be the same as the food material type of the food material in the reference image, and the frame of reference image does not have a reference value for determining the food material type of the first food material.
And 909, determining the food material type of the food material in the reference image corresponding to the maximum similarity in the at least one reference similarity as the food material type of the first food material.
The controller may determine a reference image, which includes the food material with the food material type most likely identical to the food material type of the first food material, in all the reference images, and further determine the food material type of the food material in the reference image as the food material type of the first food material. The reference image is the image with the maximum reference similarity with any frame of food material image in all the reference images.
It should be noted that, because too many food materials with graphic codes are often not stored in the storage chamber of the refrigerator, the number of reference images can be small, so that the speed of determining the maximum similarity among at least one reference similarity can be increased, and the speed of determining the type of the food material of the first food material can be further ensured. In addition, in the embodiment of the present application, when the access state of the first food material is the food material taking-out state, the food material type of the first food material is determined by determining the similarity between the food material image and the reference image. Therefore, the code does not need to be scanned when the first food material is taken out (namely, the graphic code carried by the target food material is collected), the code is not required to be scanned when the food material is stored and taken out, and the user experience can be improved.
It should be noted that in the embodiment of the present application, the controller may execute step 605 when it is determined that the second condition is satisfied. Wherein the second condition comprises: the analysis result of the graphic code acquired by the code scanning module does not include any food material type; or the at least one reference similarity determined by the controller is smaller than the similarity threshold; or the identification state of the first food material is a state carrying the graphic code, the target identification result does not include any food material type, the graphic code is not collected by the code scanning module within the target duration, and the access state of the first food material is a state of storing the food material.
The controller determines that the food material type of the first food material cannot be determined by analyzing the graphic code acquired by the code scanning module, or the food material type of the first food material cannot be determined according to a mode of matching with the reference image when the food material is taken out, or the food material type of the first food material cannot be determined when the first food material is stored, and then the controller can control the loudspeaker to play the second prompt voice so as to determine the food material type of the first food material in a mode of user input.
Optionally, when the analysis result of the graphic code acquired by the code scanning module by the controller does not include any food material type, and the controller determines the food material type of the first food material through a user input mode, the controller may upload the corresponding relationship between the graphic code acquired by the code scanning module and the food material type of the first food material to the graphic code information base, so that the food material type of the first food material can be determined according to the graphic code when the graphic code is acquired subsequently. If the controller can store the corresponding relation between the character string corresponding to the graphic code and the food material type of the food material into the graphic code information base, the continuous updating of the graphic code information base can be guaranteed, the graphic code information included in the graphic code information base is enriched, and the limitation that the food material type is determined by adopting a graphic code analyzing mode is reduced.
Optionally, in the embodiment of the application, the user can also set the graphic code for the food material by himself, and input the information of the food material indicated by the graphic code into the graphic code information base, so that the food material which does not carry the graphic code originally can be identified by means of graphic code analysis, and the management effect of the refrigerator on the information of the stored food material is improved. Alternatively, the graphic code pasted by the user can be reused, such as arranging the graphic code on the food material through a clip or a vessel with the graphic code.
It should be noted that, in the related art, the food material type of the food material to be accessed is determined only in an image recognition manner, and the accuracy of the determination of the food material type is low, and the cases that the determination of the food material type is impossible are more, so that the limitation of determining the food material type is high. It should be noted that when the type of the food material is determined by analyzing only the graphic code carried by the food material, the graphic code analysis is prone to fail. In the embodiment of the application, the food material type of the food material can be determined by combining image recognition, graphic code analysis and voice input, so that the effective determination of the food material type of the food material can be ensured, the accuracy of determining the food material type is improved, the limitation of determining the food material type is reduced, and the intelligent management of a refrigerator on the food material information is facilitated. In addition, codes do not need to be scanned when food materials are stored and taken out, so that the process of determining the types of the food materials can be simplified, and the use experience of a user on the refrigerator is improved.
In another optional embodiment of the present application, the controller may further comprise a shelf and a weight sensor. The controller may determine the storage location of the food material in combination with the image captured by the camera and the weight detected by the weight sensor.
In an embodiment of the application, a shelf is located in the storage compartment for dividing the storage compartment into a plurality of storage layers; each weight sensor is used for detecting the weight of the corresponding shelf and the object carried on the corresponding shelf. It should be noted that fig. 1 illustrates an example in which three shelves (not shown in fig. 1) are provided in the storage chamber, and the three shelves can divide the storage chamber into 4 storage layers. Optionally, the number of the shelves may also be two or four or even more, which is not limited in the embodiments of the present application. Alternatively, the storage compartments in the embodiments of the present application may include a refrigerating compartment and a freezing compartment, and the door may include at least one door corresponding to the refrigerating compartment and at least one door corresponding to the freezing compartment, and fig. 1 illustrates only the structure in the refrigerating compartment, and reference may be made to the description of the refrigerating compartment for the freezing compartment.
Optionally, the weight sensor corresponding to each shelf in the embodiment of the present application may include a plurality of sub-weight sensors located on a bottom surface of the shelf, the bottom surface of the shelf is opposite to the bearing surface of the shelf, and the weight detected by the weight sensor corresponding to the shelf may be: an average value of the weights detected by the plurality of sub-weight sensors, or a sum of the weights detected by the plurality of sub-weight sensors. Alternatively, each shelf may also include only one sub-weight sensor, which is not limited in this embodiment.
Optionally, each shelf may also be located on a corresponding carrying structure, and a weight sensor corresponding to the shelf may be located between the shelf and the corresponding carrying structure to detect the weight of the shelf and the object carried by the shelf. Illustratively, the refrigerator may further include at least one set of bosses on a sidewall of the storage compartment, and each set of bosses may be one load bearing structure. The at least one group of bosses are in one-to-one correspondence with the at least one shelf, and for any group of bosses and the corresponding shelf, the group of bosses are used for bearing the shelf, and the weight sensors corresponding to the shelf are positioned between the shelf and the bosses. Alternatively, each set of bosses may include at least two bosses supporting opposite ends of the shelf. For example, the weight sensor may include a plurality of sub-weight sensors located at an edge region of the bottom surface of the shelf. For example, the weight sensor includes two sub-weight sensors, which may be respectively located at edge positions of opposite ends in the bottom surface of the shelf. For another example, the weight sensor includes four sub weight sensors, and the four sub weight sensors may be respectively located at four corners of the bottom surface of the shelf.
Alternatively, the load bearing structure may be a plate-like structure that may be secured in the storage compartment (e.g., by being supported by bosses on the side walls of the storage compartment), the rack may be located on a corresponding plate-like structure, and the rack's corresponding weight sensor may be located between the rack and the corresponding plate-like structure. At this time, the sub-weight sensor included in the weight sensor corresponding to the shelf may be located at a middle region of the bottom surface of the shelf.
The controller may further perform the following steps to determine the storage location of the food material:
and b11, determining the moving track of the hand according to the multi-frame target images.
It should be noted that, after the controller determines multiple frame target images in step 601a, the controller may execute step b 11. Thereafter, the controller may perform step b 12.
For example, after each frame of target image including the hand region is determined, the controller may further determine a centroid position of the hand region in each frame of target image, and then determine a movement trajectory of the hand by tracking the centroid position, such as tracking the centroid position of the hand region by an adaptive higher-order predictive tracking model. For example, the controller may cluster pixel points of the hand region in the image to determine a centroid position of the hand. The controller can also determine a track vector of the hand according to two adjacent frames of target images, and further obtain a plurality of track vectors according to the multiple frames of target images. The controller may determine a continuous movement trajectory of the hand based on the trajectory vector obtained by combining the plurality of trajectory vectors.
And b12, determining the target storage area passed by the hand according to the moving track of the hand. Step b13 is performed.
The storage compartment may include a plurality of storage regions, each of which may include at least two storage layers, and the storage layers of the plurality of storage regions are different. Illustratively, the storage compartment in the refrigerator shown in fig. 1 is divided into four storage layers by three shelves, and it is assumed that the four storage layers are sequentially referred to as a first storage layer, a second storage layer, a third storage layer, and a fourth storage layer from top to bottom. The four reservoir layers may belong to two reservoir regions, e.g., the two reservoir regions are referred to as a high confidence region and a low confidence region, respectively. Wherein the low confidence region may include a first reservoir layer and a second reservoir layer and the high confidence region may include a third reservoir layer and a fourth reservoir layer.
The controller may determine the m depth value ranges according to a height of the refrigerator. Please refer to step 601d for the introduction of the m depth value ranges, which is not described in detail in the embodiments of the present application. Alternatively, the plurality of depth value ranges may correspond one-to-one to a plurality of storage areas included in the storage room, and the depth value range corresponding to each storage area includes a distance from any one position in the storage area to the image pickup apparatus.
For example, the controller may determine an average depth value of a hand region in a target image according to a movement trajectory of the hand, where the target image is an image of the multi-frame image when the hand moves to a target position in the movement trajectory, and the target position is a position farthest from the starting point in the movement trajectory. Furthermore, the controller may determine the storage area corresponding to the depth value range in which the average depth value is located as the target storage area through which the hand passes; that is, the controller may determine that the hand passes through the target storage area when the average depth value is within the depth value range corresponding to the target storage area. In this way, the controller may determine that the user accesses food material from the target storage area, i.e., that the storage location of the accessed food material is located within the target storage area.
Step b13, determine whether the target storage area is a storage area near the image pickup apparatus among the plurality of storage areas. When the target storage area is not a storage area close to the image pickup apparatus, performing step b 14; when the target storage area is a storage area near the image pickup apparatus, step b15 is executed.
When the camera device collects images, a blind area often exists, at least part of areas in the storage area close to the camera device are located in the blind area, and when a user accesses food materials in the storage area close to the camera device, the accuracy of the position of the food materials to be accessed is determined to be lower through the images collected by the camera device. And camera equipment can gather the comparatively complete image apart from its distant storing district, and when the user accessed the food material in this storing district far away from camera equipment, the accuracy of the position of confirming access food material according to the image of camera equipment collection was higher. It is possible to determine the accuracy of determining the position of the access food material from the image captured by the image capturing apparatus by determining whether the target storage area is one of the plurality of storage areas that is close to the image capturing apparatus.
For example, the image pickup apparatus is located at the top of the cabinet or the top of the storage room, the storage area near the image pickup apparatus may be the low confidence area, and the storage area farther from the image pickup apparatus may be the high confidence area.
And b14, determining the target storage layer passed by the hand according to the moving track.
When the controller determines that the target storage area is not a storage area close to the camera equipment, the controller determines that the accuracy of the position for storing and taking food materials is higher according to the image acquired by the camera equipment; furthermore, the controller can directly determine the position of the food material to be stored and taken according to the image collected by the camera equipment.
For example, the depth value range corresponding to the storage area may include a plurality of sub-depth value ranges, the plurality of sub-depth value ranges may correspond to the storage layers in the storage area in a one-to-one manner, and the sub-depth value range corresponding to each storage layer includes a distance from any position in the storage layer to the image capturing apparatus. The controller can determine a target image according to the moving track of the hand, and then determine a sub-depth value range in which the average depth value of the hand area in the target image is located; furthermore, the controller may determine the storage layer corresponding to the sub-depth range as a target storage layer through which the hand passes, that is, determine the storage position of the stored food material as the target storage layer.
And b15, determining the target storage layer with the changed food material stored in the target storage area according to the weight detected by the weight sensor corresponding to at least one shelf positioned between the storage layers of the target storage area.
When the controller determines that the target storage area is a storage area close to the camera equipment, the controller determines that the accuracy of determining the position of storing and taking food materials according to the image acquired by the camera equipment is low; furthermore, the controller can determine the position of the food material to be accessed by combining the weight detected by the weight sensor so as to ensure the accuracy of determining the position of the food material to be accessed. Alternatively, in the embodiment of the present application, the weight sensors may be provided only on the bottom surfaces of the shelves in the storage area near the image pickup apparatus, that is, only the weight sensors may be provided for each of the shelves in the storage area near the image pickup apparatus. It should be noted that the shelf in the storage area near the image pickup apparatus in the embodiment of the present application refers to a shelf between the respective storage layers in the storage area. Optionally, a weight sensor may be disposed on a bottom surface of each shelf in the storage chamber, which is not limited in this application.
For example, when it is determined that the image captured by the image capturing apparatus includes a hand, that is, when the target image is captured, the controller may capture the weight detected by the weight sensor corresponding to each shelf in the storage room, and determine that the weight detected by a certain weight sensor captured within a set period of time is the same as the weight detected by the weight sensor, as an effective weight detected by the weight sensor. The controller can determine the storage layer with the changed stored food material according to the effective weight detected by each weight sensor, namely determine the storage position of the stored food material.
It should be noted that there are many ways to determine the target storage layer according to the weight detected by the weight sensor, and the embodiments of the present application are explained by taking the following two ways as examples.
In a first manner, the controller may determine the target reservoir layer based only on the weight detected by the weight sensor.
Alternatively, when Gi’-GiAnd when the storage layer is not equal to 0, the controller can determine the storage layer, which is close to the bearing surface of the ith shelf, in the target storage area as the target storage layer. Wherein i is more than or equal to 1, and the ith shelf is any one shelf in the target storage area; giIndicating the weight detected by the weight sensor corresponding to the ith shelf before the hand passes the target storage area, Gi' denotes the weight detected by the weight sensor corresponding to the ith shelf after the hand passes through the target storage area. GiAnd Gi' both are the effective weights detected by the weight sensors. Gi’-GiNot equal to 0 andthat is, after the hand passes through the target storage area, the weight detected by the weight sensor corresponding to the ith shelf changes.
For example, when the controller determines that the image captured by the image capturing device includes a hand, the controller may start to acquire the effective weight detected by the weight sensor corresponding to each shelf in the storage area close to the image capturing device, where the weight detected by the weight sensor corresponding to the ith shelf detected by the controller is Gi. Then, the controller can continuously obtain the effective weight detected by the weight sensor corresponding to each shelf of the controller, and when no hand is included in the image acquired by the camera equipment within the preset time, the controller can determine the weight detected by the weight sensor corresponding to the ith shelf as Gi’。
Because weight sensor is used for detecting the weight of the shelf that corresponds and the object that it bore, so when the effectual weight that certain weight sensor detected changes, the object that the shelf that this weight sensor corresponds bore can be thought to the controller changes, and then the controller can directly confirm to have carried out the access of eating the material in the storing layer of this shelf top. Thus, the controller can determine the storage tier in the target storage area that is proximate to the carrying surface of the shelf as the target storage tier. Therefore Gi’-GiAnd when the storage layer is not equal to 0, the controller can determine the storage layer, which is close to the bearing surface of the ith shelf, in the target storage area as the target storage layer.
In the second mode, the controller may further determine the access state of the food material in the target storage area according to the multi-frame image including the hand captured by the image capturing apparatus. The access state comprises a food material storing state or a food material taking state. Furthermore, the target storage layer is determined according to the weight detected by the weight sensor corresponding to at least one shelf between the storage layers of the target storage area and the access state of the food material in the target storage area. The controller determines that the access state of the food in the target storage area is a food storing state, namely that the food is stored in the target storage area; the controller determines that the access state of the food in the target storage area is the food taking-out state, namely that the food is taken out of the target storage area.
The controller can determine a multi-frame color image corresponding to the multi-frame depth image according to the multi-frame depth image including the hand, further recognize the multi-frame color image, and determine the change condition of the hand state. The hand state comprises a state of not taking food materials or a state of taking food materials. When the controller determines that the hand state is changed from the food material not taken state to the food material taken state according to the multi-frame color images, the access state of the food materials can be determined to be the food material taken state; when the controller determines that the hand state is changed from the food material taking state to the food material non-taking state according to the multi-frame color images, the access state of the food material can be determined to be the food material storing state.
In one case, when Gi’-GiWhen not equal to 0, the controller may determine whether the first condition and the second condition are satisfied, and then determine the target reservoir layer. Wherein the first condition comprises: the access state is the state of storing food material, and Gi’-Gi> 0, or the access state is the food material taking-out state, and Gi’-GiLess than 0; the second condition includes: the access state is the state of storing food material, and Gi’-GiLess than 0; or the access state is the food material taking-out state, and Gi’-Gi>0。Gi’-GiIf the weight is more than 0, namely after the food material is stored and taken, the weight detected by the weight sensor corresponding to the ith shelf is increased; gi’-GiThat is, after the food material is stored and taken out, the weight detected by the weight sensor corresponding to the ith shelf is reduced.
When the first condition is met, the controller can determine a storage layer in the target storage area adjacent to the ith shelf and proximate to the carrying surface of the ith shelf (which can also be referred to as a storage layer above the ith shelf) as the target storage layer. When the second condition is met, the controller can determine a storage layer in the target storage area that is adjacent to the ith shelf and away from the carrying surface of the ith shelf (which can also be referred to as a storage layer under the ith shelf) as the target storage layer.
In another case, there is only one shelf between the individual storage levels of the target storage area, i.e. the target storage area comprises only two storage levels. When the weight detected by the weight sensor corresponding to the shelf is not changed after the food material is stored and taken, the controller may determine the target storage layer for storing and taking the food material in the target storage area according to the storage and taking state of the food material.
For example, the one shelf in the target storage area is the ith shelf, and the weight detected by the weight sensor corresponding to the shelf is not changed after the food material is stored and taken, that is, Gi’-G i0. At this time, when the access state is the food material storing state, the controller may determine that the target storage layer is a storage layer of the bearing surface far from the ith shelf in the target storage area; when the access state is a food material taking-out state, the controller may determine that the target storage layer is a storage layer close to the carrying surface of the ith shelf in the target storage area.
It is assumed that the target storage area comprises two storage layers including the first storage layer and the second storage layer described above, and the shelf between the two storage layers is the first shelf. Table 5 below is a table of correspondence between the change in weight detected by the weight sensor, the access status, and the target storage layer, and table 5 below shows only six of the correspondence.
As shown in table 5 below, in the first corresponding relationship, the controller determines that there is food material stored in the target storage area, and the weight detected by the weight sensor corresponding to the first shelf is increased after the food material is stored in the target storage area, at this time, the controller may determine that the food material is stored in the first storage layer.
In the second corresponding relation, the controller determines that the food material is stored in the target storage area, and the weight detected by the weight sensor corresponding to the first shelf is reduced after the food material is stored and taken out, and at the moment, the controller can determine that the food material is stored in the second storage layer. And the controller can also determine that the scene is that more food materials are stored in the second storage layer, and the food materials are ejected to the first storage layer when the food materials are stored in the second storage layer.
In the third corresponding relation, the controller determines that food materials are stored in the target storage area, the weight detected by the weight sensor corresponding to the first shelf is not changed after the food materials are stored and taken, and at the moment, the controller can determine that the food materials in the first storage layer are not changed and the food materials are stored in the second storage layer.
In the fourth corresponding relationship, the controller determines that the food material is taken out of the target storage area, and the weight detected by the weight sensor corresponding to the first shelf is increased after the food material is stored and taken out, at this time, the controller may determine that the food material is taken out of the second storage layer. Since the weight sensor corresponding to the shelf below the first storage area inevitably detects a decrease in weight when the food material is taken out of the first storage area, it can be determined that the controller can determine that the food material is taken out of the second storage layer when the weight sensor detects an increase in weight when the food material is taken out of the target storage area. And the controller can also determine that the scene is that more food materials are stored in the second storage layer, the food materials are jacked to the first storage layer before the food materials are taken out from the second storage layer, and the weight detected by the weight sensor corresponding to the shelf is smaller than the actual weight of the shelf and the objects borne by the shelf. When the food material is taken out of the second storage layer, the weight detected by the weight sensor is increased due to the lack of the load of the food material of the second storage layer.
In a fifth corresponding relationship, the controller determines that the food material is taken out of the target storage area, and the weight detected by the weight sensor corresponding to the first shelf is reduced after the food material is stored and taken out, at this time, the controller may determine that the food material is taken out of the first storage layer.
In a sixth corresponding relationship, the controller determines that the food material is taken out of the target storage area, and the weight detected by the weight sensor corresponding to the first shelf is not changed after the food material is stored and taken out, at this time, the controller may determine that the food material is taken out of the first storage layer.
TABLE 5
Serial number | Access state of food material | Change in weight | Target reservoir layer |
1 | Food material storage state | Increase of | The first storage layer |
2 | Food material storage state | Reduce | Second storage layer |
3 | Food material storage state | Is not changed | Second storage layer |
4 | Food material taking-out state | Increase of | Second storage layer |
5 | Food material taking-out state | Reduce | The first storage layer |
6 | Food material taking-out state | Is not changed | Second storage layer |
It should be noted that, in the embodiment of the present application, only the weight sensor is disposed on the bottom surface of the shelf in the storage area close to the image pickup apparatus, and when it is determined that the hand passes through the storage area close to the image pickup apparatus, the target storage layer of the stored food material is determined according to the weight detected by the weight sensor. Optionally, when the weight sensor is further arranged on the bottom surface of the shelf in the storage area far away from the image pickup device, when the hand is determined to pass through the storage area far away from the image pickup device, the storage layer of the stored food material can also be determined by combining the weight detected by the weight sensor.
Optionally, in the embodiment of the application, after the controller is in the access state of the food material and the target storage layer of the accessed food material, the storage information of the food material may be displayed on a display provided on the refrigerator. For example, when the controller determines that the access state of the food material is the food material stocking state and determines that the target storage layer in which the food material is stocked is the first storage layer, the display screen may display storage information of "stocking of food material in the first storage layer".
Optionally, the controller may further identify the type of the food material accessed according to the image acquired by the image pickup device, and further display the identified type of the food material through the display. For example, the controller may recognize that the type of the obtained food material is apple, and the display screen may display storage information of "apple is stored in the first storage layer".
It should be noted that, in the related art, by setting a fixed depth value range corresponding to each storage layer, when it is detected that the depth of the food material is within the depth value range corresponding to which storage layer when the food material is stored and taken, it is determined which storage layer the storage position of the food material is. At present, the position of the uppermost shelf (i.e., the shelf closest to the top of the refrigerator) of many storage rooms of the refrigerator can be adjusted up and down, at this time, the size of the uppermost storage layer in the storage room changes, and the actual depth range of the storage area is different from the set depth value range corresponding to the storage layer. Therefore, the accuracy of determining the storage location of the food material using the related art is low, and the manner of determining the storage location of the food material is not adaptable.
In the embodiment of the application, the target storage area is determined by detecting the depth of the hand area, and then the target storage layer stored in the stored food material is determined according to the weight detected by the weight sensor corresponding to the shelf. Because even if the position of the shelf is changed, the weight sensor corresponding to the shelf can still normally detect the weight of the shelf and the objects carried by the shelf (namely, the objects in the storage layer above the shelf). Therefore, when the position of the shelf is changed, the storage layer of the stored food material can be accurately determined according to the weight detected by the weight sensor corresponding to the shelf, and therefore the method for determining the storage position of the food material in the embodiment of the application has good adaptability.
It should be understood that the terms "first," "second," "third," and the like in the description and in the claims of the present application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used are interchangeable under appropriate circumstances and can be implemented in sequences other than those illustrated or otherwise described herein with respect to the embodiments of the application, for example.
Furthermore, the terms "comprises" and "comprising," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a product or device that comprises a list of elements is not necessarily limited to those elements explicitly listed, but may include other elements not expressly listed or inherent to such product or device.
Embodiments of the present application also provide a computer program product containing instructions, which when run on a computer, cause the computer to execute the method provided by the embodiments of the present application. The above description is only exemplary of the present application and should not be taken as limiting, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.
Claims (12)
1. A refrigerator, characterized in that the refrigerator comprises:
a cabinet including a storage compartment having an opening;
the door is movably connected with the shell and used for shielding the opening;
a controller to:
when a first food material meeting the target food material condition is stored at a first moment, determining food material information corresponding to the target food material condition; wherein the food material information comprises at least one of shelf life and volume;
at a second moment after the first moment, if the updating condition is met, updating the food material information corresponding to the target food material condition;
wherein the update condition includes: storing or taking out food materials meeting the target food material conditions;
the quality guarantee period T corresponding to the updated target food material condition is T0- (T2-T1), T0 represents the quality guarantee period corresponding to the target food material condition before updating, T1 represents the first moment, and T2 represents the second moment;
when the food materials meeting the target food material condition are stored at the second moment, updating the volume V of the food material information corresponding to the target food material condition as V0 plus delta V1; when the food materials meeting the target food material condition are taken out at the second moment, the updated volume V of the food material information corresponding to the target food material condition is V0-delta V2; v0 represents the volume of the food material information corresponding to the target food material condition before updating, Δ V1 represents the volume of the food material satisfying the target food material condition stored at the second time, and Δ V2 represents the volume of the food material satisfying the target food material condition extracted at the second time.
2. The refrigerator of claim 1, wherein the target food material conditions comprise: conditions of food material types, or conditions of food material types and conditions of food material states; the food material state comprises a processed state or an unprocessed state.
3. The refrigerator according to claim 1 or 2, wherein when the food material satisfying the target food material condition is stored at the second time, the updating condition further comprises: and taking out the food materials meeting the target food material condition at a third moment before the second moment, wherein the time difference between the second moment and the third moment is smaller than a time difference threshold value.
4. The refrigerator according to claim 3, wherein the update condition further includes: the food materials which are stored at the second moment and meet the target food material condition are as follows: at least part of the food materials which are taken out at the third moment and meet the target food material condition, wherein the refrigerator further comprises:
the loudspeaker is used for playing first prompt voice, and the first prompt voice is used for confirming whether the food materials which are stored at the second moment and meet the target food material condition are: at least part of the food materials meeting the target food material conditions are taken out at the third moment;
the microphone is used for collecting first response voice within a first time length after the loudspeaker plays the first prompt voice;
the controller is further configured to determine whether the update condition is satisfied based on the first response voice.
5. The refrigerator of claim 2, wherein the target food material conditions comprise: a condition of the food material type and a condition of the food material status; the refrigerator further includes:
the camera equipment is used for acquiring an image at the opening;
the controller is further configured to:
inputting at least one frame of food material image including the first food material in the image acquired by the camera into a recognition model to obtain a recognition result of the food material image output by the recognition model, wherein the recognition model is used for outputting a food material type and a food material state based on the input food material image;
determining a target identification result according to the identification result of the at least one frame of food material image;
when the target identification result comprises the food material type of the first food material and does not comprise any food material state, determining a first similarity of the first food material and a first reference food material and a second similarity of the first food material and a second reference food material; the food material types of the first reference food material and the second reference food material are the food material types of the first food material, the food material state of the first reference food material is the processed state, and the food material state of the second reference food material is the unprocessed state;
determining the food material state of the food material corresponding to the greater similarity of the first similarity and the second similarity as the food material state of the first food material, wherein the condition for determining the food material type comprises the following steps: the food material type is the food material type of the first food material, and the conditions of the food material state comprise: the food material state is the food material state of the first food material.
6. The refrigerator of claim 5, wherein the controller is further configured to:
determining similarity between each food material image in the at least one frame of food material image and each first reference image in the at least one frame of first reference image to obtain at least one first reference similarity, wherein the first reference image comprises the first reference food material;
determining a maximum similarity among the at least one first reference similarity as the first similarity;
determining the similarity between each frame of food material image in the at least one frame of food material image and each frame of second reference image in the at least one frame of second reference image to obtain at least one second reference similarity, wherein the second reference image comprises the second reference food material;
determining a maximum similarity of the at least one second reference similarity as the second similarity.
7. The refrigerator according to claim 5, further comprising:
the loudspeaker is used for controlling the loudspeaker to play a second prompt voice for inputting the food material type of the first food material when the target identification result does not comprise any food material type;
the microphone is used for controlling the microphone to collect second response voice within a second time length after the loudspeaker plays the second prompt voice;
the controller is further used for determining the food material type of the first food material based on the second response voice.
8. The refrigerator according to claim 1 or 2, wherein the refrigerator further comprises:
and the loudspeaker is used for playing a third prompt voice which indicates the quality guarantee duration in the food material information corresponding to the target food material condition when the quality guarantee duration corresponding to the target food material condition is smaller than the duration limit value.
9. The refrigerator according to claim 1 or 2, wherein the refrigerator further comprises:
the loudspeaker is used for playing a fourth prompt voice for confirming whether the food meeting the target food material condition exists in the storage chamber or not when the food meeting the target food material condition is taken out at the second moment;
the microphone is used for collecting fourth response voice within a fourth time length after the fourth prompt voice is played by the loudspeaker;
the controller is further configured to delete the food material information corresponding to the target food material condition when it is determined that the food material satisfying the target food material condition does not exist in the storage chamber based on the fourth response voice.
10. The refrigerator of claim 5, wherein the controller is further configured to:
acquiring a retraining instruction, wherein the retraining instruction carries the food material type of the first food material, and the retraining instruction is used for indicating that the target recognition result does not include the food material type of the first food material;
training the recognition model with target training data based on the retraining instruction to update the recognition model, the target training data comprising: a target food material image in the at least one frame of food material image and a food material type of the first food material, wherein the recognition result of the target food material image does not include the food material type of the first food material.
11. The refrigerator of claim 5, wherein the controller is further configured to:
acquiring n frames of target images including a hand area in the images acquired by the camera equipment, wherein n is more than or equal to 1;
and determining the at least one frame of food material image according to the n frames of target images.
12. The refrigerator according to any one of claims 1, 10 and 11, further comprising:
the camera equipment is used for acquiring an image at the opening;
a shelf in the storage compartment for dividing the storage compartment into a plurality of storage layers;
the weight sensor is used for detecting the weight of the corresponding shelf and the object borne by the shelf;
the controller is further configured to:
determining a moving track of a hand according to a multi-frame target image including a hand area in an image acquired by the camera equipment;
determining a target storage area which is passed by the hand in a plurality of storage areas according to the moving track, wherein the storage areas comprise at least two storage layers, and the storage layers in the plurality of storage areas are different;
determining a target storage layer in which the food material stored in the target storage area is changed according to the weight detected by the weight sensor corresponding to at least one shelf located between the respective storage layers of the target storage area.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911416098.8A CN113124634B (en) | 2019-12-31 | 2019-12-31 | Refrigerator with a door |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911416098.8A CN113124634B (en) | 2019-12-31 | 2019-12-31 | Refrigerator with a door |
Publications (2)
Publication Number | Publication Date |
---|---|
CN113124634A true CN113124634A (en) | 2021-07-16 |
CN113124634B CN113124634B (en) | 2022-05-13 |
Family
ID=76769506
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201911416098.8A Active CN113124634B (en) | 2019-12-31 | 2019-12-31 | Refrigerator with a door |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN113124634B (en) |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2009180447A (en) * | 2008-01-31 | 2009-08-13 | Panasonic Corp | Refrigerator |
CN202734420U (en) * | 2012-06-21 | 2013-02-13 | 深圳市国显科技有限公司 | Intelligent refrigerator based on internet of things and application system thereof |
CN106642969A (en) * | 2016-11-11 | 2017-05-10 | 青岛海尔特种电冰箱有限公司 | Refrigerator food management method and refrigerator |
CN107763955A (en) * | 2016-08-23 | 2018-03-06 | 海信(山东)冰箱有限公司 | A kind of refrigerator and its control method |
CN108072230A (en) * | 2016-11-16 | 2018-05-25 | 九阳股份有限公司 | A kind of refrigerator defrosting control method |
CN108224895A (en) * | 2018-01-08 | 2018-06-29 | 合肥美的智能科技有限公司 | Article information input method and device based on deep learning, refrigerator and medium |
CN108766529A (en) * | 2018-05-16 | 2018-11-06 | 北京豆果信息技术有限公司 | A kind of intelligence recipe recommendation method and system |
-
2019
- 2019-12-31 CN CN201911416098.8A patent/CN113124634B/en active Active
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2009180447A (en) * | 2008-01-31 | 2009-08-13 | Panasonic Corp | Refrigerator |
CN202734420U (en) * | 2012-06-21 | 2013-02-13 | 深圳市国显科技有限公司 | Intelligent refrigerator based on internet of things and application system thereof |
CN107763955A (en) * | 2016-08-23 | 2018-03-06 | 海信(山东)冰箱有限公司 | A kind of refrigerator and its control method |
CN106642969A (en) * | 2016-11-11 | 2017-05-10 | 青岛海尔特种电冰箱有限公司 | Refrigerator food management method and refrigerator |
CN108072230A (en) * | 2016-11-16 | 2018-05-25 | 九阳股份有限公司 | A kind of refrigerator defrosting control method |
CN108224895A (en) * | 2018-01-08 | 2018-06-29 | 合肥美的智能科技有限公司 | Article information input method and device based on deep learning, refrigerator and medium |
CN108766529A (en) * | 2018-05-16 | 2018-11-06 | 北京豆果信息技术有限公司 | A kind of intelligence recipe recommendation method and system |
Also Published As
Publication number | Publication date |
---|---|
CN113124634B (en) | 2022-05-13 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN111444880B (en) | Food material identification method and refrigerator | |
US5170440A (en) | Perceptual grouping by multiple hypothesis probabilistic data association | |
CN113494792A (en) | Refrigerator with a door | |
CN113124635B (en) | Refrigerator with a door | |
CN109141395B (en) | Sweeper positioning method and device based on visual loopback calibration gyroscope | |
CN107705324A (en) | A kind of video object detection method based on machine learning | |
CN104408760B (en) | A kind of high-precision virtual assembly system algorithm based on binocular vision | |
CN110689560A (en) | Food material management method and equipment | |
Wang et al. | Towards cooperation in sequential prisoner's dilemmas: a deep multiagent reinforcement learning approach | |
CN107784663A (en) | Correlation filtering tracking and device based on depth information | |
CN105164481A (en) | Method for managing storage product in refrigerator using image recognition, and refrigerator for same | |
CN106052294A (en) | Refrigerator and method for judging change of objects in object storage area of refrigerator | |
CN107341442A (en) | Motion control method, device, computer equipment and service robot | |
KR101652261B1 (en) | Method for detecting object using camera | |
CN110674789A (en) | Food material management method and refrigerator | |
CN113139402B (en) | A kind of refrigerator | |
CN113124633B (en) | Refrigerator with a door | |
CN113124636B (en) | Refrigerator | |
CN113947770B (en) | Method for identifying object placed in different areas of intelligent cabinet | |
US20220325946A1 (en) | Selective image capture using a plurality of cameras in a refrigerator appliance | |
CN113124634B (en) | Refrigerator with a door | |
CN108765463A (en) | A kind of moving target detecting method calmodulin binding domain CaM extraction and improve textural characteristics | |
CN111488831A (en) | Food association identification method and refrigerator | |
Ju et al. | Video tracking to monitor turkey welfare | |
KR20210132998A (en) | Apparatus and method tracking object in image fames based on neural network |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |