CN110966833B - Method for detecting food material information in refrigerator and refrigerator - Google Patents
Method for detecting food material information in refrigerator and refrigerator Download PDFInfo
- Publication number
- CN110966833B CN110966833B CN201811161687.1A CN201811161687A CN110966833B CN 110966833 B CN110966833 B CN 110966833B CN 201811161687 A CN201811161687 A CN 201811161687A CN 110966833 B CN110966833 B CN 110966833B
- Authority
- CN
- China
- Prior art keywords
- data
- detection model
- refrigerator
- freshness
- hyperspectral
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
- F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
- F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
- F25D23/00—General constructional features
- F25D23/12—Arrangements of compartments additional to cooling compartments; Combinations of refrigerators with other equipment, e.g. stove
-
- F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
- F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
- F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
- F25D29/00—Arrangement or mounting of control or safety devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
- F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
- F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
- F25D2500/00—Problems to be solved
- F25D2500/06—Stock management
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Combustion & Propulsion (AREA)
- Mechanical Engineering (AREA)
- Thermal Sciences (AREA)
- Data Mining & Analysis (AREA)
- Chemical & Material Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Investigating Or Analysing Materials By Optical Means (AREA)
- Cold Air Circulating Systems And Constructional Details In Refrigerators (AREA)
Abstract
The invention provides a method for detecting food material information in a refrigerator and the refrigerator. Wherein, the hyperspectral imaging device is arranged in the storage compartment of the refrigerator. The method for detecting the food material information in the refrigerator comprises the following steps: acquiring hyperspectral data of food materials shot by a hyperspectral imaging device; respectively extracting image data and spectrum data from the hyperspectral data; detecting the type of the food material according to the extracted image data, acquiring a freshness detection model and a nutrition component detection model corresponding to the type of the food material, and inputting the extracted spectral data into the freshness detection model and the nutrition component detection model to obtain the information of the freshness and the nutrition component of the food material in the refrigerator. The food management is convenient for the user.
Description
Technical Field
The invention relates to the technical field of storage, in particular to a method for detecting food material information in a refrigerator and the refrigerator.
Background
With the improvement of society and the improvement of living standard of people, consumers pay attention to the nutrition value and safety of food materials when buying the food materials, and consider factors such as price, taste, appearance, freshness and the like, and the role of the refrigerator is gradually changed from pure storage and fresh keeping to a food material management center and a home nutrition center, so that new challenges are provided for the refrigerator, and meanwhile, a trigger is provided for various intelligent detection technology application to the refrigerator. The mode of knowing the type of food materials stored in the refrigerator is changed from opening the refrigerator door to intelligent detection. The automatic detection technology is utilized to realize the function of detecting the types of food materials on the domestic refrigerator, and the intelligent refrigerator has become a development trend.
The automatic detection technology is a technology which uses a specific detection device, enables detected objects to approach the detection device, automatically acquires related information of the detected objects, and provides the related information for a computer processing system to complete related subsequent processing. The automatic detection technology applied to the refrigerator at present comprises radio frequency detection, image detection and the like, radio frequency detection is carried out by pasting radio frequency detection codes on food materials put into the refrigerator and detecting the food materials by utilizing a radio frequency detection device arranged on the refrigerator, the technology needs that the purchased food materials contain the radio frequency detection codes, most food materials in the market at present do not contain the radio frequency detection codes, particularly vegetables and fruits, and the technology does not contain the detection codes, so that the technology is greatly limited in application. The image detection technology is also applied to the refrigerator, but the correct detection rate is lower, and because the technology mainly relies on detecting the difference of the color of food material images or the shape and texture of food materials, the technology has difficulty in correctly detecting food materials with similar colors and shapes, and can not realize the detection of the freshness of the food materials.
For the detection of the freshness of the existing food materials, a gas sensor array is generally adopted for realizing. The gas sensor array is arranged in the refrigerator compartment, various gases are continuously released along with the extension of the storage time of food materials stored in the refrigerator, and each gas sensor is respectively responsive to certain specific gases at the moment, so that the freshness and the change condition of the food materials are judged, and the freshness of the food materials is comprehensively judged.
When the gas sensor array detects the freshness of the food materials, the risk of misjudgment is relatively high. First, a typical gas sensor may be sensitive to a class of chemicals, and multiple gases may contain the same, so the sensor may not be able to truly detect what gas is responding, thus causing a false positive. Secondly, when a plurality of food materials are mixed and placed, the emitted gases are mixed together, so that the sensor responds, but the sensor cannot detect which food material is released, and misjudgment is caused.
For the existing detection of the nutritional ingredients of the food materials, different instruments such as a glycometer, a sclerometer and the like are used for respectively detecting the physical and chemical indexes, so that the information of the total nutritional ingredients of the food materials is obtained.
The rapid detection means-the spectroscopy which is gradually developed provides a rapid detection means, and the spectrum information is related to the chemical composition and the molecular structure of the object to be detected, so that the nutrition component information of food can be accurately carried out. During identification, firstly, the spectral information of the food to be detected is acquired, then the characteristic spectral information which can represent the food is extracted, a series of foods and the respective characteristic spectral information are modeled, and then the nutritional ingredients of the food to be detected can be detected. However, the method generally performs single-point detection, and the user experience is poor; meanwhile, the detection area is small, and when a certain food material is subjected to multipoint detection, the result is often inconsistent, so that the accuracy of the technology needs to be further improved.
Disclosure of Invention
An object of the present invention is to provide a method for automatically detecting food material information.
A further object of the present invention is to improve the accuracy of food material information detection.
The invention firstly provides a method for detecting food material information in a refrigerator, and a hyperspectral imaging device for shooting food materials in a storage room of the applicable refrigerator is arranged in the storage room. The method for detecting the food material information in the refrigerator comprises the following steps: acquiring hyperspectral data obtained by shooting by a hyperspectral imaging device; preprocessing hyperspectral data, and respectively extracting image data and spectrum data; detecting the type of food materials according to the image data; acquiring a freshness detection model and a nutritional ingredient detection model corresponding to the types of the food materials, wherein the freshness detection model and the nutritional ingredient detection model are respectively obtained by training according to spectral data of the food materials with different qualities in advance; the spectral data are respectively subjected to classification analysis and calculation by using a freshness detection model and a nutritional ingredient detection model, so that the freshness and the nutritional ingredient information of the food materials are determined; and outputting information of the types, freshness and nutritional ingredients of food materials in the refrigerator through a display screen of the refrigerator.
Optionally, the step of detecting the kind of food material from the image data includes: acquiring image data; obtaining a food material type detection model, wherein the food material type detection model is obtained by training according to hyperspectral data of different types of food materials in advance; inputting the image data into a food material type detection model; and performing mode detection by using a food material type detection model to obtain food material type information.
Optionally, the step of classifying the spectral data captured by the hyperspectral imaging device using the nutritional component detection model comprises: extracting image data required by a nutrient component detection model from hyperspectral data shot by a hyperspectral imaging device; inputting image data required by the nutrient detection model into the nutrient detection model; and performing mode detection by using a nutritional ingredient detection model to obtain nutritional ingredient information of the food material.
Optionally, the step of classifying the spectral data captured by the hyperspectral imaging device by the freshness detection model includes: extracting image data required by a freshness detection model from hyperspectral data shot by a hyperspectral imaging device; inputting image data required by the freshness detection model into the freshness detection model; and carrying out mode detection by using a freshness detection model to obtain the freshness information of the food materials.
Optionally, the hyperspectral data includes a set number of ternary data groups, each ternary data group includes two image pixel elements of one pixel point and one spectrum wavelength element, each pixel point has multiple sets of ternary data groups, the image data is extracted by analyzing and extracting data in the image pixel elements, and the spectrum data is extracted by analyzing and extracting data in the spectrum wavelength elements.
Optionally, the resolution of the spectral wavelength of each pixel in the hyperspectral data is less than or equal to 2nm.
Optionally, in the process of starting the hyperspectral imaging device, a light source system matched with the hyperspectral imaging device is also started simultaneously to provide light required by shooting of the hyperspectral imaging device, wherein the spectrum range of the light source system is 400nm to 1100nm.
Optionally, the step of determining the freshness and nutrient content information of the food material further comprises: and outputting information of the types, freshness and nutritional ingredients of food materials in the refrigerator through the mobile terminal bound with the refrigerator.
According to another aspect of the present invention, there is also provided a refrigerator. The refrigerator includes: a box body, in which a storage compartment is defined; the hyperspectral imaging device is arranged in the storage compartment and is configured to shoot food materials in the storage compartment; the controller comprises a memory and a processor, wherein the memory stores a computer program, and the computer program is used for realizing the method for detecting the food material information in the refrigerator according to the above when being executed by the processor.
Optionally, the refrigerator further includes: and the information output interface is configured to provide information of food materials for a display screen of the refrigerator or a mobile terminal bound with the refrigerator so as to output the information to a user.
According to the method for detecting the food material information in the refrigerator and the refrigerator, the hyperspectral imaging device is arranged in the refrigerator, hyperspectral data of the food material are obtained through shooting, the hyperspectral data are used for detecting the type, the freshness and the nutritional ingredients of the food material, the detection accuracy is high, and the requirements for rapidly and nondestructively obtaining the food material type, the freshness and the nutritional ingredient information are met.
Further, the method for detecting the food material information in the refrigerator and the refrigerator provided by the invention have the advantages that the characteristics of the spectral information of the food material, the type, the freshness and the nutritional ingredient degree are related, the mode detection technology is adopted, the freshness detection of the food material is carried out by means of the freshness detection model, the nutritional ingredient detection of the food material is carried out by means of the nutritional ingredient detection model, the accuracy of detecting the food material information is obviously improved, and the food material management is facilitated for users.
The above, as well as additional objectives, advantages, and features of the present invention will become apparent to those skilled in the art from the following detailed description of a specific embodiment of the present invention when read in conjunction with the accompanying drawings.
Drawings
Some specific embodiments of the invention will be described in detail hereinafter by way of example and not by way of limitation with reference to the accompanying drawings. The same reference numbers will be used throughout the drawings to refer to the same or like parts or portions. It will be appreciated by those skilled in the art that the drawings are not necessarily drawn to scale. In the accompanying drawings:
fig. 1 is a schematic view of a refrigerator according to an embodiment of the present invention;
fig. 2 is a functional schematic block diagram of a refrigerator according to an embodiment of the present invention;
fig. 3 is a schematic view of a refrigerator according to another embodiment of the present invention;
FIG. 4 is a schematic view of a method of detecting the type of food material in a refrigerator according to one embodiment of the present invention; and
fig. 5 is a schematic view of a method of detecting freshness and nutrients of food materials in a refrigerator according to an embodiment of the present invention.
Detailed Description
Fig. 1 is a schematic view of a refrigerator 10 according to one embodiment of the present invention. The refrigerator 10 of the present embodiment may generally include: a housing 110, a door 120, and a hyperspectral imaging device 210.
The case 110 defines therein at least one storage compartment 130, typically a plurality of compartments, such as a refrigerator compartment, a freezer compartment, a temperature change compartment, etc., which are open at a front side. The number and function of particular storage compartments 130 may be configured according to the needs in advance, and in some embodiments, the storage temperature of the refrigerated compartment may be 2-9 ℃, or may be 4-7 ℃; the storage temperature of the freezing chamber may be-22 to-14 ℃, or may be-20 to 16 ℃. The freezing chamber is arranged below the refrigerating chamber, and the temperature changing chamber is arranged between the freezing chamber and the refrigerating chamber. The temperature in the freezer compartment is typically in the range of-14 to-22 ℃. The temperature changing chamber can be adjusted according to the requirement to store proper food materials or be used as a fresh-keeping storage chamber.
The door 120 is disposed at the front side of the case 110, and is used for opening and closing the storage compartment 130. For example, the door 120 may be hinged to one side of the front of the case 110, and the storage compartments 130 may be opened and closed in a pivoting manner, and the number of the door 120 may be matched with the number of the storage compartments 130, so that the storage compartments 130 may be opened individually one by one. For example, a refrigerating chamber door, a freezing chamber door and a temperature changing chamber door may be provided for the refrigerating chamber, the freezing chamber and the temperature changing chamber, respectively. In some alternative embodiments, the door 120 may also take the form of a side-by-side door, a sliding door, or the like.
The storage compartment 130 is provided with cooling capacity by a refrigeration system to realize a refrigerated, frozen, and temperature-variable storage environment. The refrigeration system may be a refrigeration cycle system composed of a compressor, a condenser, a throttle device, an evaporator, and the like. The evaporator is configured to provide cooling directly or indirectly into the storage compartment 130. For example, in a compression type direct-cooling refrigerator, the evaporator may be disposed outside or inside the rear wall surface of the refrigerator liner. In the compressed air-cooled refrigerator, the interior of the refrigerator body 110 is also provided with an evaporator chamber, the evaporator chamber is communicated with the storage compartment 130 through an air path system, an evaporator is arranged in the evaporator chamber, and a fan is arranged at an outlet of the evaporator chamber so as to circularly refrigerate the storage compartment 130. Since the above-mentioned case 110, door 120, and refrigeration system are well known and easy to implement by those skilled in the art, the case 110, door 120, and refrigeration system are not described in detail in order to not obscure and obscure the invention of the present application.
The hyperspectral imaging device 210 is disposed in the storage compartment 130 of the refrigerator 10, and is configured to capture food materials in the storage compartment 130 and output hyperspectral data.
The hyperspectral data may be a series of triplets, each triplet including two image pixel elements and one spectral wavelength element for a pixel, each pixel having multiple sets of triplets. The hyperspectral data thus simultaneously obtain continuous spectral data for each pixel point and continuous image data for each spectral band. And the spectral data is obtained by analyzing and extracting the data in the pixel elements of the image, and the spectral data is obtained by analyzing and extracting the data in the spectral wavelength elements. The hyperspectral image is an optical image with continuous wavelength, the spectral range can be set to be 200nm to 2500nm, the higher spectral resolution is achieved, and the resolution can reach 2-3 nm. The hyperspectral data may be represented by a three-dimensional data block, where two dimensions are image pixel information (x, y) and the third dimension is wavelength information (λ). Image detector array with resolution x y pixels the data cube obtained at n wavelengths is a three-dimensional array of x y x lambda.
In the present embodiment, the spectral data in the spectral range of 400nm to 1100nm is preferably used, since the detection of the kind, freshness and nutrient information of the food material 300 is facilitated by a large amount of research on the spectral data in the above spectral range. The resolution requirement of the spectral wavelength of each pixel in the hyperspectral data photographed by the hyperspectral imaging device 210 is less than or equal to 2nm, thereby satisfying the requirement.
The variety detection model can be obtained by training hyperspectral data of a large amount of food materials, and a training algorithm which can be adopted by the variety detection model can comprise a BP neural network, a Support Vector Machine (SVM) and Adaboost. The method can train a plurality of different kinds of detection models at the position in advance according to the kinds of food materials, for example, train corresponding kinds of detection models for various meats, various fruits and various vegetables respectively.
The freshness detection model can be obtained by training hyperspectral data of a large number of food materials with different freshness, and a training algorithm which can be adopted by the freshness detection model can comprise a BP neural network, a Support Vector Machine (SVM) and Adaboost. A plurality of different freshness detection models can be trained in advance according to the types of food materials, for example, corresponding freshness detection models can be trained for various meats, various fruits and various vegetables.
The nutritional ingredient detection model can be obtained by training hyperspectral data of a large number of food materials with different qualities, and a training algorithm which can be adopted by the nutritional ingredient detection model can comprise a BP neural network, a Support Vector Machine (SVM) and Adaboost. A plurality of different nutrient component detection models can be trained in advance according to the types of food materials, for example, corresponding nutrient component detection models are trained for various meats, various fruits and various vegetables respectively.
In performing the category detection, the following steps may be performed: image data required for the type detection model is extracted from the hyperspectral data captured by the hyperspectral imaging device 210, the image data required for the type detection model is input into the type detection model, and the type detection model performs pattern detection to obtain the type of food.
In the detection of freshness and nutrients, the following steps may be performed: the method comprises the steps of acquiring a freshness detection model and a nutrition component detection model corresponding to the types of food materials, extracting spectrum data required by the freshness detection model and the nutrition component detection model from hyperspectral data shot by a hyperspectral imaging device 210, inputting the spectrum data into the freshness detection model and the nutrition component detection model respectively, and performing mode detection by the freshness detection model and the nutrition component detection model respectively to obtain the freshness and the nutrition component of the food materials.
The freshness and nutrition components can be detected by cloud technology, for example, after the hyperspectral data captured by the hyperspectral imaging device 210 is obtained, the data processing device of the refrigerator 10 performs preliminary processing, and then uploads the hyperspectral data after preliminary processing to the cloud, the cloud completes the mode detection steps of the freshness detection model and the nutrition components detection model, and then provides the freshness and the nutrition components of the food 300 to the refrigerator 10 or the mobile terminal bound to the refrigerator 10 for providing to the user. The freshness detection model and the nutritional ingredient detection model are stored in the cloud, and the data processing pressure of the refrigerator 10 is reduced.
The freshness may reflect the degree of rancidity, mold, dehydration, etc. of the food material. After the freshness exceeds the set degree, the user can be reminded in time. The nutrient components can reflect the information of the food material such as moisture, sugar content, and soluble solids. Interaction with the user may be achieved through a human-computer interaction system of the refrigerator 10, for example, outputting information of the kinds, freshness and nutritional ingredients of the food materials on a display screen of the refrigerator 10. In another embodiment, a message including information of the kind, freshness and nutritional ingredients of the above-described food materials may be transmitted to the mobile terminal bound to the refrigerator 10 and a message fed back by the user through the mobile terminal may be received.
Fig. 2 is a functional schematic block diagram of a refrigerator 10 according to another embodiment of the present invention. The following components can be flexibly added in the refrigerator 10 of this embodiment: a controller 270, a light source system 230, and an information output interface 250.
A controller 270, disposed in the refrigerator 10, for extracting image data and spectrum data from the hyperspectral data obtained by photographing the hyperspectral imaging device 210, detecting the type of the food material 300 according to the extracted image data, and obtaining a freshness detection model and a nutrition detection model corresponding to the type of the food material, wherein the freshness detection model and the nutrition detection model are respectively obtained by training according to the spectrum data of the food materials with different qualities in advance; carrying out classified analysis and calculation on the spectrum data by using a freshness detection model so as to determine the freshness of the food materials; and performing classification analysis calculation on the optical data by using a nutritional component detection model so as to determine the nutritional components of the food material; the controller 270 includes a memory 271 and a processor 272, and the memory 271 stores a computer program, and the processor 272 is configured to execute the computer program in the memory 271, and the computer program is configured to implement the method for detecting food material information in the refrigerator in this embodiment.
The light source system 230 is disposed in the storage compartment 130 and configured to provide the hyperspectral imaging device 210 with light required for photographing, wherein the spectral range of the light source system 230 is set to 400-1100 nm. The light source system 230 may be disposed at the rear of the top wall in the storage compartment 130 to provide photographing light obliquely downward. The light source system 230 may be activated simultaneously with the hyperspectral imaging device 210 to provide light to the enclosed storage compartment 130.
The information output interface 250 may be configured to provide the kind, freshness, and nutrient information of the food material 300 to a display screen of the refrigerator or a mobile terminal bound to the refrigerator to output to a user.
To ensure that the hyperspectral imaging device 210 is able to take an overview of the food material 300 placed in the storage compartment 130. The hyperspectral imaging device 210 preferably uses a wide-angle lens or a fisheye lens, and is disposed directly above the storage compartment 130.
Fig. 3 is a schematic view of a refrigerator 10 according to another embodiment of the present invention. In the refrigerator 10 of the present example, aiming at the problem that the internal space of the refrigerator 10 is small and the hyperspectral imaging device 210 is difficult to shoot the whole view of the storage compartment 130, hyperspectral data reflecting the whole view of the storage compartment 130 is obtained by arranging the reflecting mirror 260 and utilizing the way of shooting the reflected image.
The reflecting mirror 260 and the hyperspectral imaging device 210 are oppositely arranged inside the storage compartment 130. The hyperspectral imaging device 210 may be configured to capture the mirror 260 to obtain hyperspectral data of the image reflected by the mirror 260. Since the space inside the refrigerator 10 is relatively small and the storage compartment 130 is generally in a flat layered structure for convenience of storage, it is difficult for the conventional hyperspectral imaging apparatus 210 to take an overall view of the storage compartment 130 in such a flat region in the space, and thus this problem can be effectively solved by taking a reflected image of the reflecting mirror 260 in the present embodiment. In some alternative embodiments, the mirror 260 may optionally use a convex mirror to reflect the entire storage compartment 130.
The mirror 260 is provided on, for example, a top wall of the storage compartment, and the hyperspectral imaging device 210 is provided in, for example, a bottom wall of the storage compartment. The area where the hyperspectral imaging device 210 is located may be set as a blank area, preventing the user from placing the food material 300 to be detected above the hyperspectral imaging device 210, and shielding the lens.
Whether the hyperspectral imaging device 210 adopts a corner lens or a fisheye lens or adopts a reflecting mode of the reflecting mirror 260, the hyperspectral imaging device can obtain hyperspectral data reflecting the whole appearance of the storage compartment 130, so that the shooting requirement of the food material 300 is met.
In using the information detection function of the refrigerator 10, one specific example is: after an apple is placed in the storage compartment 130, the user issues a detection command via a button or mobile terminal on the refrigerator 10. The hyperspectral imaging device 210 photographs the storage compartment 130 to obtain hyperspectral data including apples. Image data and spectrum data of the food material 300 (apples) can be extracted from the hyperspectral data, the image data is obtained, the type of the food material 300 is determined by utilizing algorithms such as a neural network, a PLS/SVM and other chemometric methods, and a freshness detection model and a nutrition detection model corresponding to the type of the food material are obtained, wherein the freshness detection model and the nutrition detection model are respectively obtained by training according to the spectrum data of the food materials with different qualities in advance; and respectively carrying out classification analysis and calculation on the spectrum data by using a freshness detection model and a nutrient component detection model, so as to determine the freshness and the nutrient components of the food materials.
The type detection function of the refrigerator 10 may be automatically and periodically started to detect the type, freshness, and nutrient information of the food in the storage compartment 130.
The detection result can be used for further establishing food material type, freshness and nutrient information storage files in the refrigerator 10, recording the food material type, freshness and nutrient storage information and providing a data base for intelligent management of food materials.
The embodiment also provides a method for detecting the type of food in the refrigerator 10, which can be used for the refrigerator 10 in any embodiment, and can be used for detecting the type information of the storage compartment 130 in the refrigerator 10.
Fig. 4 is a schematic view of a method of detecting the kind of food material in the refrigerator 10 according to an embodiment of the present invention. The method of detecting the kind of food material in the refrigerator 10 may generally include:
step S402, obtaining hyperspectral data shot by the hyperspectral imaging device 210;
step S404, preprocessing the hyperspectral data to extract image data;
step S406, obtaining a food material type detection model;
step S408, inputting the image data into a food material type detection model;
in step S410, the food material type detection model performs mode detection to obtain the type information of the food material.
The variety detection model can be obtained by training hyperspectral data of a large amount of food materials, and a training algorithm which can be adopted by the variety detection model can comprise a BP neural network, a Support Vector Machine (SVM) and Adaboost. The method can train a plurality of different kinds of detection models at the position in advance according to the kinds of food materials, for example, train corresponding kinds of detection models for various meats, various fruits and various vegetables respectively.
In performing the category detection, the following steps may be performed: image data required for the type detection model is extracted from the hyperspectral data captured by the hyperspectral imaging device 210, the image data required for the type detection model is input into the type detection model, and the type detection model performs pattern detection to obtain the type of food.
In step S402, the hyperspectral data may include a set number of ternary data sets, each ternary data set including two image pixel elements and one spectral wavelength element of one pixel, each pixel having multiple sets of ternary data sets, and the resolution of the spectral wavelength of each pixel in the hyperspectral data is less than or equal to 2nm. The light source system 230 is also required to provide a light source in starting the hyperspectral imaging device 210 to shoot, so that the spectrum range of the spectrum data can be in the range of 400nm to 1100nm meeting the detection requirement of the food material type, and the spectrum range of the light source system 230 is required to be in the range of 400nm to 1100nm.
The embodiment also provides a method for detecting the freshness and the nutritional ingredients of the food materials in the refrigerator 10, which can be used for the refrigerator 10 in any embodiment, and can be used for detecting the freshness and the nutritional ingredient information of the food materials 300 in the refrigerator 10.
Fig. 5 is a schematic view of a method of detecting freshness and nutrients of food materials in a refrigerator according to an embodiment of the present invention. The method for detecting the freshness and the nutritional ingredients of the food materials in the refrigerator generally comprises the following steps:
step S502, acquiring hyperspectral data of the food material 300 photographed by the hyperspectral imaging device 210;
step S504, respectively extracting image data and spectrum data from the hyperspectral data;
step S506, detecting the type of food materials according to the image data;
step S508, a freshness detection model and a nutrient component detection model corresponding to the types of the food materials are obtained;
in step S510, the spectral data are classified by using the freshness detection model and the nutritional component model, so as to determine the freshness and the nutritional components of the food material 300.
The freshness detection model can be obtained by training hyperspectral data of a large number of food materials with different qualities, and a training algorithm which can be adopted by the freshness detection model can comprise a BP neural network, a Support Vector Machine (SVM) and Adaboost. The method can train a plurality of different freshness detection models at the position in advance according to the types of food materials, for example, train corresponding freshness detection models for various meats, various fruits and various vegetables respectively.
The nutritional ingredient detection model can be obtained by training hyperspectral data of a large number of food materials with different qualities, and a training algorithm which can be adopted by the nutritional ingredient detection model can comprise a BP neural network, a Support Vector Machine (SVM) and Adaboost. The method can train a plurality of different nutrient component detection models at the position in advance according to the types of food materials, for example, train corresponding nutrient component detection models for various meats, various fruits and various vegetables respectively.
One specific implementation of step S510 may include: spectral information required by the freshness detection model and the nutritional ingredient detection model is extracted from the hyperspectral data shot by the hyperspectral imaging device 210; the method comprises the steps of inputting spectral information required by a freshness detection model and a nutrient detection model into the freshness detection model and the nutrient detection model respectively; the freshness and the nutritional ingredients of the food material 300 are obtained by performing pattern recognition on the freshness detection model and the nutritional ingredient model, respectively.
The hyperspectral data may include a set number of ternary data sets, each ternary data set including two image pixel elements of one pixel point and one spectral wavelength element, each pixel point having a plurality of sets of ternary data sets, and the spectral data being extracted by analyzing data in the image pixel elements, the spectral data being extracted by analyzing data in the spectral wavelength elements. The resolution of the spectral wavelength of each pixel in the hyperspectral data is less than or equal to 2nm. To ensure that the spectral range of the spectral data can be in the range of 400nm to 1100nm, which meets the detection requirements, the spectral range of the light source system 230 needs to be 400nm to 1100nm.
In the method for detecting food material information in a refrigerator of the present embodiment, image data and spectrum data are extracted from hyperspectral data, the type of food material 300 in the refrigerator 10 is detected by using the image data, a corresponding freshness detection model and nutrition component model are selected according to the type of food material 300, the spectrum data is input into the freshness detection model and the nutrition component model, and freshness and nutrition component information of the food material 300 is detected. And the hyperspectral imaging technology combines the spectrum detection technology with the image recognition technology, so that the space image data of the food material and the spectrum data of each point can be acquired simultaneously. The type information of the food materials can be rapidly and nondestructively acquired by utilizing the image data; and then, by combining with spectrum data comprehensive analysis, the freshness and nutrition information of a plurality of pixel points on the image can be obtained at the same time, the freshness and nutrition information of the plurality of points is comprehensively calculated, and finally, the food freshness and nutrition information with high accuracy is obtained.
By now it should be appreciated by those skilled in the art that while a number of exemplary embodiments of the invention have been shown and described herein in detail, many other variations or modifications of the invention consistent with the principles of the invention may be directly ascertained or inferred from the present disclosure without departing from the spirit and scope of the invention. Accordingly, the scope of the present invention should be understood and deemed to cover all such other variations or modifications.
Claims (9)
1. A method for detecting food material information in a refrigerator, wherein a hyperspectral imaging device for shooting food materials in a storage compartment of the refrigerator is arranged in the storage compartment, and the method comprises the following steps:
acquiring hyperspectral data obtained by shooting by the hyperspectral imaging device;
preprocessing the hyperspectral data, and respectively extracting image data and spectrum data;
detecting the type of food materials according to the image data;
acquiring a freshness detection model and a nutritional ingredient detection model corresponding to the types of food materials, wherein the freshness detection model and the nutritional ingredient detection model are respectively obtained by training according to hyperspectral data of the food materials with different qualities in advance;
using the freshness detection model and the nutrient detection model to respectively carry out classification analysis and calculation on the spectrum data so as to determine the freshness and the nutrient information of the food materials; and
outputting information of the types, freshness and nutritional ingredients of food materials in the refrigerator through a display screen of the refrigerator;
wherein the step of detecting the kind of the food material from the image data includes:
acquiring the image data;
obtaining a food material type detection model, wherein the food material type detection model is obtained by training according to hyperspectral data of different types of food materials in advance;
inputting the image data into the food material type detection model;
and performing mode detection by the food material type detection model to obtain the type information of the food material.
2. The method of claim 1, wherein classifying spectral data captured by the hyperspectral imaging device using the nutritional component detection model comprises:
extracting spectral data required by the nutritional ingredient detection model from the hyperspectral data shot by the hyperspectral imaging device;
inputting spectral data required by the nutritional ingredient detection model into the nutritional ingredient detection model;
and performing mode detection by the nutritional ingredient detection model to obtain the nutritional ingredient information of the food material.
3. The method of claim 1, wherein classifying spectral data captured by the hyperspectral imaging device using the freshness detection model comprises:
extracting spectral data required by the freshness detection model from the hyperspectral data shot by the hyperspectral imaging device;
inputting spectral data required by the freshness detection model into the freshness detection model;
and performing mode detection by the freshness detection model to obtain the freshness information of the food materials.
4. The method of claim 1, wherein,
the hyperspectral data comprises a set number of ternary data groups, each ternary data group comprises two image pixel elements and one spectral wavelength element of one pixel point, each pixel point is provided with a plurality of sets of ternary data groups, and
the image data is obtained by analyzing and extracting data in the image pixel elements, and the spectrum data is obtained by analyzing and extracting data in the spectrum wavelength elements.
5. The method of claim 4, wherein,
the resolution of the spectral wavelength of each pixel point in the hyperspectral data is less than or equal to 2nm.
6. The method of claim 1, wherein,
in the process of starting the hyperspectral imaging device, a light source system matched with the hyperspectral imaging device is started at the same time to provide light required by shooting of the hyperspectral imaging device, wherein the spectrum range of the light source system is 400nm to 1100nm.
7. The method of claim 1, wherein the step of determining freshness, nutritional composition information of the food material further comprises:
and outputting information of the types, freshness and nutritional ingredients of food materials in the refrigerator through the mobile terminal bound with the refrigerator.
8. A refrigerator, comprising:
a box body, in which a storage compartment is defined;
the hyperspectral imaging device is arranged in the storage compartment and is configured to shoot food materials in the storage compartment;
a controller comprising a memory and a processor, the memory having a computer program stored therein and which, when executed by the processor, is adapted to carry out the method according to any one of claims 1 to 6.
9. The refrigerator of claim 8, further comprising:
and the information output interface is configured to provide information of the food materials for a display screen of the refrigerator or a mobile terminal bound with the refrigerator so as to output the information to a user.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811161687.1A CN110966833B (en) | 2018-09-30 | 2018-09-30 | Method for detecting food material information in refrigerator and refrigerator |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811161687.1A CN110966833B (en) | 2018-09-30 | 2018-09-30 | Method for detecting food material information in refrigerator and refrigerator |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110966833A CN110966833A (en) | 2020-04-07 |
| CN110966833B true CN110966833B (en) | 2023-06-02 |
Family
ID=70029171
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811161687.1A Active CN110966833B (en) | 2018-09-30 | 2018-09-30 | Method for detecting food material information in refrigerator and refrigerator |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110966833B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113932553A (en) * | 2020-07-14 | 2022-01-14 | 青岛海尔电冰箱有限公司 | Refrigerator and control method thereof |
| CN114252400A (en) | 2020-09-21 | 2022-03-29 | 三星电子株式会社 | Meat inspection device, meat inspection system, refrigerator, and meat inspection method |
| CN114322438A (en) * | 2020-09-30 | 2022-04-12 | 青岛海尔智能技术研发有限公司 | Method and device for managing refrigerator food materials and refrigerator |
| CN113409915A (en) * | 2021-05-10 | 2021-09-17 | 王蕴秋 | Safety recommendation system capable of preventing food from deteriorating and working method thereof |
| CN115371347B (en) * | 2021-05-18 | 2024-03-12 | 青岛海尔电冰箱有限公司 | Detection method and detection system for use state of storage compartment of refrigerator and refrigerator |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040211190A1 (en) * | 2002-09-17 | 2004-10-28 | The Furukawa Electric Co., Ltd. | Temperature adjustment device and laser module |
| CN102889727A (en) * | 2012-10-31 | 2013-01-23 | 昆山黄昆纺织机械有限公司 | Solar energy refrigerator |
| WO2016186301A1 (en) * | 2015-05-15 | 2016-11-24 | 피도연 | Food condition measuring device and food condition measuring system using same |
| CN106871568A (en) * | 2016-12-30 | 2017-06-20 | 青岛海尔智能技术研发有限公司 | The category identification method and refrigerator of refrigerator inside food |
| WO2018165605A1 (en) * | 2017-03-09 | 2018-09-13 | Northwestern University | Hyperspectral imaging sensor |
-
2018
- 2018-09-30 CN CN201811161687.1A patent/CN110966833B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040211190A1 (en) * | 2002-09-17 | 2004-10-28 | The Furukawa Electric Co., Ltd. | Temperature adjustment device and laser module |
| CN102889727A (en) * | 2012-10-31 | 2013-01-23 | 昆山黄昆纺织机械有限公司 | Solar energy refrigerator |
| WO2016186301A1 (en) * | 2015-05-15 | 2016-11-24 | 피도연 | Food condition measuring device and food condition measuring system using same |
| CN106871568A (en) * | 2016-12-30 | 2017-06-20 | 青岛海尔智能技术研发有限公司 | The category identification method and refrigerator of refrigerator inside food |
| WO2018165605A1 (en) * | 2017-03-09 | 2018-09-13 | Northwestern University | Hyperspectral imaging sensor |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110966833A (en) | 2020-04-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110966833B (en) | Method for detecting food material information in refrigerator and refrigerator | |
| CN108663331A (en) | Detect the method and refrigerator of food freshness in refrigerator | |
| CN108662842A (en) | The detecting system and refrigerator of food in refrigerator | |
| US11060735B2 (en) | Connected food preparation system and method of use | |
| CN110966834A (en) | Method for identifying food material types in refrigerator and refrigerator | |
| CN107873101B (en) | Imaging system for object recognition and evaluation | |
| CN106871568A (en) | The category identification method and refrigerator of refrigerator inside food | |
| CN106871569A (en) | The category identification method and refrigerator of refrigerator inside food | |
| Cubero et al. | Advances in machine vision applications for automatic inspection and quality evaluation of fruits and vegetables | |
| CN108662843A (en) | The detecting system and refrigerator of food in refrigerator | |
| US11425338B2 (en) | Refrigerator, and system and method for controlling same | |
| CN109655414B (en) | Electronic equipment, information pushing method and related product | |
| CN109869966A (en) | Refrigerator | |
| CN110966835B (en) | Method for detecting nutrient components of food materials in refrigerator and refrigerator | |
| KR102017980B1 (en) | Refrigerator with displaying image by identifying goods using artificial intelligence and method of displaying thereof | |
| CN115131783B (en) | Autonomous sensing method of user diet nutrition ingredient information based on machine vision | |
| CN110017659A (en) | Domestic refrigerator equipment | |
| CN109631484A (en) | Freezer | |
| CN108663333A (en) | High light spectrum image-forming component and refrigerator for detecting food in refrigerator | |
| CN115046363B (en) | Intelligent refrigerator based on machine vision and spectrum detection and operation method | |
| CN108663332A (en) | High light spectrum image-forming component and refrigerator for detecting food in refrigerator | |
| US11928769B2 (en) | Employing controlled illumination for hyperspectral or multispectral imaging of food in an appliance | |
| US20230076984A1 (en) | Inventory management system in a refrigerator appliance | |
| US20220414391A1 (en) | Inventory management system in a refrigerator appliance | |
| Argun et al. | Non-Destructive Prediction of Bread Staling Using Artificial Intelligence Methods |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20230518 Address after: 266101 Haier Industrial Park, 1 Haier Road, Laoshan District, Shandong, Qingdao Applicant after: QINGDAO HAIER SMART TECHNOLOGY R&D Co.,Ltd. Applicant after: Qingdao guochuang intelligent home appliance Research Institute Co.,Ltd. Address before: 266101 Haier Road, Laoshan District, Qingdao, Qingdao, Shandong Province, No. 1 Applicant before: QINGDAO HAIER SMART TECHNOLOGY R&D Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |