US20230169772A1 - Refrigeration system with imaging - Google Patents
Refrigeration system with imaging Download PDFInfo
- Publication number
- US20230169772A1 US20230169772A1 US18/073,552 US202218073552A US2023169772A1 US 20230169772 A1 US20230169772 A1 US 20230169772A1 US 202218073552 A US202218073552 A US 202218073552A US 2023169772 A1 US2023169772 A1 US 2023169772A1
- Authority
- US
- United States
- Prior art keywords
- rack
- refrigeration system
- item
- cameras
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000005057 refrigeration Methods 0.000 title claims abstract description 121
- 238000003384 imaging method Methods 0.000 title description 13
- 238000000034 method Methods 0.000 claims description 40
- 235000014101 wine Nutrition 0.000 claims description 38
- 238000010191 image analysis Methods 0.000 claims description 24
- 238000001816 cooling Methods 0.000 claims description 12
- 230000000977 initiatory effect Effects 0.000 claims description 11
- 230000004044 response Effects 0.000 claims description 5
- 235000013361 beverage Nutrition 0.000 description 35
- 238000004891 communication Methods 0.000 description 21
- 238000004458 analytical method Methods 0.000 description 20
- 238000001514 detection method Methods 0.000 description 14
- 230000008569 process Effects 0.000 description 10
- 238000003860 storage Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 238000012015 optical character recognition Methods 0.000 description 6
- 230000002441 reversible effect Effects 0.000 description 5
- 230000000153 supplemental effect Effects 0.000 description 5
- 239000011521 glass Substances 0.000 description 4
- 239000000463 material Substances 0.000 description 4
- 230000000717 retained effect Effects 0.000 description 4
- 239000011800 void material Substances 0.000 description 4
- 230000001413 cellular effect Effects 0.000 description 3
- 235000013305 food Nutrition 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 230000002452 interceptive effect Effects 0.000 description 3
- 238000004519 manufacturing process Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000000926 separation method Methods 0.000 description 3
- 230000003068 static effect Effects 0.000 description 3
- LFQSCWFLJHTTHZ-UHFFFAOYSA-N Ethanol Chemical compound CCO LFQSCWFLJHTTHZ-UHFFFAOYSA-N 0.000 description 2
- CDBYLPFSWZWCQE-UHFFFAOYSA-L Sodium Carbonate Chemical compound [Na+].[Na+].[O-]C([O-])=O CDBYLPFSWZWCQE-UHFFFAOYSA-L 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 239000002826 coolant Substances 0.000 description 2
- 239000007799 cork Substances 0.000 description 2
- 230000014759 maintenance of location Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 235000020095 red wine Nutrition 0.000 description 2
- 241000220317 Rosa Species 0.000 description 1
- 230000000712 assembly Effects 0.000 description 1
- 238000000429 assembly Methods 0.000 description 1
- 230000010267 cellular communication Effects 0.000 description 1
- 235000019993 champagne Nutrition 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 230000006866 deterioration Effects 0.000 description 1
- 238000009792 diffusion process Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 239000012530 fluid Substances 0.000 description 1
- 238000001746 injection moulding Methods 0.000 description 1
- 235000013372 meat Nutrition 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 239000002184 metal Substances 0.000 description 1
- 238000002156 mixing Methods 0.000 description 1
- 230000036961 partial effect Effects 0.000 description 1
- 238000003825 pressing Methods 0.000 description 1
- 235000021460 special beverage Nutrition 0.000 description 1
- 238000010408 sweeping Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 235000020097 white wine Nutrition 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
-
- F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
- F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
- F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
- F25D25/00—Charging, supporting, and discharging the articles to be cooled
- F25D25/02—Charging, supporting, and discharging the articles to be cooled by shelves
- F25D25/024—Slidable shelves
-
- F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
- F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
- F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
- F25D29/00—Arrangement or mounting of control or safety devices
-
- F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
- F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
- F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
- F25D29/00—Arrangement or mounting of control or safety devices
- F25D29/005—Mounting of control devices
-
- F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
- F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
- F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
- F25D31/00—Other cooling or freezing apparatus
- F25D31/006—Other cooling or freezing apparatus specially adapted for cooling receptacles, e.g. tanks
- F25D31/007—Bottles or cans
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/70—Labelling scene content, e.g. deriving syntactic or semantic representations
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/50—Constructional details
- H04N23/51—Housings
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/698—Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
-
- F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
- F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
- F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
- F25D2331/00—Details or arrangements of other cooling or freezing apparatus not provided for in other groups of this subclass
- F25D2331/80—Type of cooled receptacles
- F25D2331/803—Bottles
-
- F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
- F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
- F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
- F25D2500/00—Problems to be solved
- F25D2500/06—Stock management
-
- F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
- F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
- F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
- F25D2700/00—Means for sensing or measuring; Sensors therefor
- F25D2700/06—Sensors detecting the presence of a product
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/57—Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
Definitions
- Refrigeration units are commonplace in homes. In recent years, specialized refrigeration units have been developed for specific types of perishable and consumable goods. For example, freezers exist to store frozen items (e.g., meat). Beverage coolers allow people to maintain beverages in colder temperatures. Wine coolers referred to a general class of refrigeration assemblies which are designed specifically to maintain wine bottles and other beverages. Wine coolers typically include racks that can be extended outward to enable placement of bottles. This allows for the wine cooler to keep bottles on their side (as is customary).
- freezers exist to store frozen items (e.g., meat).
- Beverage coolers allow people to maintain beverages in colder temperatures.
- Wine coolers referred to a general class of refrigeration assemblies which are designed specifically to maintain wine bottles and other beverages.
- Wine coolers typically include racks that can be extended outward to enable placement of bottles. This allows for the wine cooler to keep bottles on their side (as is customary).
- FIG. 1 A is a frontal view of a refrigeration system in a closed orientation, according to one or more embodiments.
- FIG. 1 B is a frontal view of the refrigeration system of FIG. 1 A in the closed orientation, according to one or more embodiments.
- FIG. 1 C is a side cross-sectional view of the refrigeration system in the closed orientation, along lines A-A of FIG. 1 A , according to one or more embodiments.
- FIG. 1 D is a side cross-sectional view of the refrigeration system in the open orientation, along lines B-B of FIG. 1 B , according to one or more embodiments.
- FIG. 1 E is a simplified diagram of viewing angles used by a camera array of a refrigeration system shown by FIG. 1 A through FIG. 1 D , according to one or more embodiments.
- FIG. 1 F illustrates a variation in which an example refrigeration system is implemented to include a camera module as a retrofit, according to one or more embodiments.
- FIG. 1 G illustrates another variation in which an example refrigeration system is implemented coupled with the camera module 180 , where the camera array 160 is housed.
- FIG. 2 A is a hardware diagram of an example refrigeration system, according to one or more embodiments.
- FIG. 2 B illustrates a network communication system for providing information about items stored in a refrigeration system, according to one or more embodiments.
- FIG. 3 A through FIG. 3 C illustrate example methods for operating a refrigeration system, according to one or more embodiments.
- FIG. 4 illustrates a computer system on which one or more embodiments can be implemented.
- FIG. 5 is a block diagram illustrating a user device for use with embodiments as described.
- FIG. 6 A through FIG. 6 C illustrate an example beverage cooler, according to one or more embodiments.
- FIG. 6 D illustrates an example beverage cooler with a ribbed, light-diffusive door, according to one or more embodiments.
- FIG. 7 A and FIG. 7 B illustrate a reversible rack for a refrigeration system, according to one or more embodiments.
- Embodiments include a refrigeration assembly or system (referred herein as “refrigeration system” for convenience(that utilizes a set of cameras to image items when the items are placed on a rack that is extended out of an interior space of the refrigeration system.
- a refrigeration assembly or system referred herein as “refrigeration system” for convenience(that utilizes a set of cameras to image items when the items are placed on a rack that is extended out of an interior space of the refrigeration system.
- one or more cameras of the refrigeration system can image items on a rack, such as when the rack is extended or being pushed into a seated position.
- the refrigeration system can initiate image processing and analysis to determine information about the items on the rack. The information can then be transmitted to a user device and/or rendered on a display.
- examples enable a user to know information about items retained within a refrigeration system or appliance. Further, the processing and imaging that is performed can require minimal or no separate actions from a user. For example, the user does not have to handle the items in a particular manner, other than place the items on a rack (preferable label up, if applicable) and push the rack in—actions which the user would do in any case. Thus, the user can avoid tedious tasks such as holding items in front of a camera.
- examples generate content that enables the user to learn about items stored with a refrigeration system or appliance.
- the refrigeration system or appliance is implemented as a beverage cooler to hold wine, liquor, canned beverages and the like.
- type and category e.g., number of red wines, which vintage, attributes of wine, etc.
- users may also want to know which wines have been opened (e.g., so they can drink those first), and the fill level of bottles.
- Various embodiments enable the user to determine such information without the user having to open the unit, thereby saving user convenience and energy.
- a refrigeration system is operated by (i) detecting a given rack of multiple racks of the refrigeration system being moved from an extended position to a seated position within an interior of the refrigeration system; (ii) capturing one or more images of the rack; and (iii) initiating performance of, or performing, image analysis of the one or more images to determine information about one or more items that are placed on the rack.
- a refrigeration system implements operations for performing a first image scan of a first rack using a first set of cameras, where the first image scan is performed when the first rack is outside of an interior space of the refrigeration system, and where a portion of the first rack is a first distance from the first set of cameras.
- the operations further comprise performing a second image scan of a second rack using a second set of cameras, where the second image scan is performed when the second rack is outside of an interior space of the refrigeration system, and where the second rack is a second distance from the second set of cameras, the second distance being greater than the first distance.
- a camera module for a refrigeration system, where the camera module includes a set of one or more cameras and a housing that can attach to a cabinet or structure of a refrigeration system.
- the housing can be structured to secure to a refrigeration system, to position the camera array such that a corresponding viewing angle of one or more cameras of the camera array are positioned over a vertical space that coincides with a thickness that occupies a door of the refrigeration system when the door is in a closed position.
- the refrigeration system can also include one or more processors, that execute instructions to perform operations that include: (i) capturing one or more images of a rack of the refrigeration system when the rack is extended from an interior cabinet; and initiating performance of image analysis of the one or more images to determine information about one or more items that are placed on the rack.
- One or more embodiments described herein provide that methods, techniques, and actions performed by a computing device are performed programmatically, or as a computer-implemented method.
- Programmatically means through the use of code or computer-executable instructions. These instructions can be stored in one or more memory resources of the computing device.
- a programmatically performed step may or may not be automatic.
- a programmatic module, engine, or component can include a program, a sub-routine, a portion of a program, or a software component or a hardware component capable of performing one or more stated tasks or functions.
- a module or component can exist on a hardware component independently of other modules or components.
- a module or component can be a shared element or process of other modules, programs or machines.
- Some embodiments described herein can generally require the use of computing devices, including processing and memory resources.
- computing devices including processing and memory resources.
- one or more embodiments described herein may be implemented, in whole or in part, on computing devices such as servers, desktop computers, cellular or smartphones, tablets, wearable electronic devices, laptop computers, printers, digital picture frames, network equipment (e.g., routers) and tablet devices.
- Memory, processing, and network resources may all be used in connection with the establishment, use, or performance of any embodiment described herein (including with the performance of any method or with the implementation of any system).
- one or more embodiments described herein may be implemented through the use of instructions that are executable by one or more processors. These instructions may be carried on a computer-readable medium.
- Machines shown or described with figures below provide examples of processing resources and computer-readable mediums on which instructions for implementing embodiments of the invention can be carried and/or executed.
- the numerous machines shown with embodiments of the invention include processor(s) and various forms of memory for holding data and instructions.
- Examples of computer-readable mediums include permanent memory storage devices, such as hard drives on personal computers or servers.
- Other examples of computer storage mediums include portable storage units, such as CD or DVD units, flash memory (such as carried on smartphones, multifunctional devices or tablets), and magnetic memory.
- Computers, terminals, network enabled devices are all examples of machines and devices that utilize processors, memory, and instructions stored on computer-readable mediums. Additionally, embodiments may be implemented in the form of computer-programs, or a computer usable carrier medium capable of carrying such a program.
- FIG. 1 A is a frontal view of a refrigeration system (or assembly) in a closed orientation, according to one or more embodiments.
- FIG. 1 B is a frontal view of the refrigeration system of FIG. 1 A in the closed orientation, according to one or more embodiments.
- FIG. 1 C is a side cross-sectional view of the refrigeration system 100 in the closed orientation, along lines A-A of FIG. 1 A , according to one or more embodiments.
- FIG. 1 D is a side cross-sectional view of the refrigeration system 100 in the open orientation, along lines B-B of FIG. 1 B , according to one or more embodiments.
- a refrigeration system 100 as shown and described by examples of FIG. 1 A through FIG.
- the refrigeration system 100 can correspond to a beverage cooler, with structural and functional features that are optimized for maintaining wine bottles and beverages in a suitable environment.
- embodiments can be employed with other types of refrigeration systems.
- the refrigeration system 100 is configured to perform image processing and analysis to determine information about items and content stored within the refrigeration system 100 .
- the refrigeration system 100 includes a cabinet 110 , a door 120 and a cooling subsystem 130 .
- the door 120 When the refrigeration system 100 is closed, the door 120 is in the closed position, where an interior space of the cabinet is sealed by the door 120 .
- the door 120 includes an exterior facade and an optional handle 112 . To open the door 120 , a user can pull the handle 112 , so that the door swings outward, to provide the user with access to the interior space of the cabinet 110 .
- an interior space 114 of the refrigeration system 100 includes racks 118 that are movable laterally outwards (in the Z direction) when cabinet 110 is in the open position.
- the interior space 114 is maintained at a desired temperature for cooling items on racks 118 .
- the interior space 114 can be maintained at temperatures that range between, for example, 33° F. and 70° F.
- the interior space 114 can be maintained at temperatures that range between 44° F. and 65° F.
- the specific temperature can be selected by user preference and/or the type of item being stored.
- the temperature setting can be set automatically based at least in part on one or more items placed on the rack.
- the cameras 122 , 124 , 126 can be operated to determine information about the items, such as the type or item (e.g., type of beverage or wine, manufacturer or source of wine, fill state of beverage), etc.
- the refrigeration system 100 can include one or more processor(s) 220 (see FIG. 2 A ) that can process images of the items to determine the information, and automatically implement operations to control the operation of the refrigeration system 100 based on the detected information.
- the cabinet 110 includes a top segment 111 that integrates or otherwise provides a camera array, where the camera rate includes cameras 122 , 124 126 . While in examples shown, the camera rate includes three cameras, in variations more or fewer cameras may be used. Each camera can include a lens, or combination of lenses, combined with an optical sensor and processing resources for generating image data using light captured by the optical sensors. As further described, the camera 122 is proximate (i.e., closest to) to left side 101 of the cabinet 110 , and camera 126 is proximate to a right side 103 of the cabinet 110 . The cameras 122 , 126 can be of the same type, and the camera 124 may be of a different type.
- the cameras 122 , 126 can incorporate wide angle lenses, such as fisheye lenses, while the camera 124 includes a high definition lens.
- the cameras 122 , 126 can be selected to determine information about items and contents stored with racks 118 that are closest to the lenses of the camera array.
- the door 120 is shown in an open position, to provide access to an interior space 114 of the cabinet 110 .
- the rack subassembly 140 is mounted to an interior of the cabinet, where the rack subassembly 140 maintains multiple racks 118 that can be moved laterally (e.g., in Z direction) in and out of the cabinet (or between a seated position and an extended position) while the door 120 is in the open position.
- the rack assembly 140 enables individual racks 118 to be slid out.
- the user can place bottle(s) on the rack 118 , remove bottle(s) (e.g., for consumption) from the rack 118 , reposition bottles on the rack 118 , add bottles to the rack 118 , and/or remove a bottle for partial consumption and then return it to the rack 118 .
- the user can then slide the rack back in two the interior space 114 of the cabinet 110 .
- the array of cameras 122 , 124 , 126 operate to capture images of the items on any one of the racks that is being returned to the seated position.
- the cameras 122 , 124 (not visible in FIG. 1 C ), and 126 (not visible in FIG. 1 C ) a provided in the top segment of the cabinet 110 .
- Each camera 122 124 , 126 is oriented downward, such that the field of view of the respected lens(es) is blocked by the thickness of the door 120 .
- the cameras 122 , 124 , 126 are not positioned within the interior space, where, for example, the top rack 118 occludes other racks. Rather, the cameras are position over the door 120 , so that when the door is open and racks are slid in and out, the cameras 122 , 124 , 126 can capture images of the racks for processing.
- each rack 118 is movable laterally (in the Z direction) to provide a user with access to items that are held by that rack.
- the cameras 122 , 124 (not visible in FIG. 2 D ), 126 (not visible in FIG. 2 D ) are oriented downward such that the downward viewing angle coincides or overlaps with a space that is occupied by a thickness of the door 120 when the door is in the closed position.
- each camera 122 , 124 , 126 of the camera array is mounted within a frontward and top region 128 of the cabinet 110 .
- the top segment 128 can be an integral or unitary housing segment cabinet 110 . In other examples, the top segment 128 can be a void where the cameras 122 , 124 , 126 can be mounted and accessed.
- Each camera 122 , 124 , 126 of the camera array is mounted within the top segment 128 , with lens oriented downward.
- the cameras are aligned so that a vertical field of view (in Y direction) of the respective cameras includes, or otherwise coincides with a space 125 that occupies a thickness of the door 120 when the door is in the closed position.
- the cameras 122 , 124 , 126 can be mounted to be flush, or extend outward from a surface of the 121 .
- the 121 can be in the form of a surface with openings to accommodate the lenses of the individual cameras 122 , 124 , 126 . Alternatively, some or all of the lenses may protrude from the respective openings.
- the cameras 122 , 124 , 126 can be mounted to a swing arm and moved into and out of position.
- a user places a bottle 12 on a top rack 118 when the rack is slid outward into the extended position 119 .
- the bottle passes across the field of view of the cameras 122 , 124 , 126 before the rack is returned to the seated position 117 , and the cameras 122 , 124 , 126 , capture images (e.g., video frames) of the bottle 12 .
- the cameras 122 , 124 , 126 can be configured to capture images of the rack 118 when it static, or moved outward from the cabinet 110 .
- embodiments enable for image capture and processing of bottle 12 with minimal or no additional user interaction, other than the user placing the bottle 12 on the desired rack 118 .
- the camera array is able to capture recognizable images of the bottle even as the bottle is passed within a relatively short distance of the cameras 122 , 124 , 126 .
- the camera array can equally capture recognizable images of items position on other racks, as each rack can be slid into the interior space 114 of the cabinet 110 individually.
- the rack 118 can be configured to alternatively, or additionally, support items in an alternative orientation (e.g., widthwise along Z axis).
- FIG. 1 E is a simplified diagram of viewing angles used by a camera array of the refrigeration system 100 , according to one or more embodiments. While examples described provide for the camera array 160 to include 3 lenses, in variations, more or fewer cameras may be utilized.
- the camera array 160 includes multiple cameras 122 , 124 , 126 , each having a lens that is exposed at a bottom-facing plane 121 or opening of the cabinet 110 .
- the individual cameras of camera array 160 can be operated in sets of one or more cameras, where each set is a prioritized or primary source for obtaining recognizable images of items placed on corresponding racks 118 or regions thereof. Based on implementation, one camera of the camera array 160 can be used in multiple sets. In some variations, the individual cameras of each camera set are preconfigured, or dynamically controlled, to operate in a coordinated fashion, such as to obtain images simultaneously.
- the camera array 160 includes or otherwise provides alternative camera sets for imaging specific racks 118 based on the vertical distance of the respective racks and the lenses of the cameras (which for purpose of discussion here, are assumed to be coplanar with the 121 ).
- a distance between the 121 and the top-most rack can be in the range of 40-75 mm.
- a distance between the 121 and the second rack (from the top) may range between 120-180 mm, and a distance between the third from the top can be in the range of 200-250 mm.
- the operation of the camera sets including the selection, operation and/or configuration of cameras, can based on switching elements that are triggered by movement of the racks 118 .
- Embodiments recognize that different camera sets may enable or facilitate the operations of obtaining recognizable image data from each rack 118 when such racks are used.
- the proximity of the top rack with the 121 combined with a standard field-of-view of a camera lens, would otherwise limit a span of a camera's coverage in the X-direction. Further, the ability of the camera to focus on items in such proximity may be limited.
- items placed on the second rack can have similar challenges with respect to imaging by the camera array—the field-of-view from a single camera does not span (in the X-direction) the entire width of the rack, and the items placed on the second rack may be too close to the respective cameras to enable sufficient focus.
- the field-of-view for cameras may present a lesser requirement with respect to the field of view that would otherwise be required from an imaging camera.
- the vertical distance between the respective cameras and the racks may pose an additional challenge with respect to obtaining images with adequate definition of text that can be subsequently recognized using optical character recognition (OCR).
- OCR optical character recognition
- the camera array 160 can be configured to accommodate additional optimizations.
- the camera array 160 can include a minimal number of cameras to image a full span of each rack with sufficient definition to enable subsequent image recognition (e.g., OCR)and other image processing to be performed.
- image recognition e.g., OCR
- one optimal configuration for camera array 160 provides for placement of ultra-wide lens cameras 122 , 126 towards respective left and right lateral sides of the cabinet 110 , to capture images of items placed on the top two racks 118 .
- a camera e.g., HD camera
- an HD camera can be positioned towards the lateral ends of the camera array 160 (e.g., as cameras 122 , 126 ).
- the cameras 122 , 126 have a field of view that is greater than 180° (e.g., fisheye camera lenses), with video recording capability, while the mid-camera 124 can be selected to have a field-of-view between 70 and 100 degrees, with high definition video capture capabilities.
- 180° e.g., fisheye camera lenses
- the mid-camera 124 can be selected to have a field-of-view between 70 and 100 degrees, with high definition video capture capabilities.
- the cameras 122 , 124 , 126 positioned in a top segment or area of the refrigeration system 100 with a downward viewing angle
- the cameras 122 , 124 , 126 can be positioned in a bottom segment or area and configured so the viewing angle is upwards.
- FIG. 1 F illustrates a variation in which the refrigeration system 100 is implemented to include a camera module 180 , where the camera array 160 is housed.
- a refrigeration system 100 includes a camera module 180 having a housing 182 that is separate from a structure of the refrigeration system 100 .
- the camera module 180 can be affixed to a top surface of the refrigeration system, with the lens surface 185 of the camera array positioned to be in front of/or over the door 120 .
- camera module 180 can include alternative sets of cameras 122 , 124 , 126 , dispersed lengthwise (along X direction) of the cabinet 110 .
- the lens surface 185 positions the cameras 122 , 124 , 126 (e.g., see FIG.
- each camera includes a vertical thickness that is in front of the door.
- the field of views of the respective cameras can also overlap a vertical thickness that is occupied by the door 120 .
- the cameras of the camera can image the respective rack.
- FIG. 1 G illustrates another variation in which the refrigeration system 100 is implemented to include the camera module 180 , where the camera array 160 is housed.
- the door 120 of the refrigeration system 100 spans a height of the refrigeration system 100 .
- the housing 192 of the camera module 180 can be structured to position the camera array such that the lens surface 185 is substantially aligned or in front of a vertical thickness occupied by the door 120 .
- the camera module 180 can be partially positioned within the cabinet 110 , with an extension to house the camera array extending to the outside of the cabinet, such as in an orientation shown.
- FIG. 1 A through FIG. 1 G which provide for the cameras of the camera array to capture images when a corresponding rack is extended from the cabinet 110
- an item may be entered onto the rack 118 without moving the rack our of the cabinet 110 .
- the user can reach a bottle into a rack while the rack remains seated.
- the cameras can detect the item and capture the item as it is placed on the rack 118 .
- the camera module 180 can be manufactured as an accessory item for suitably configured refrigeration systems.
- FIG. 2 A is a hardware diagram of an example refrigeration system, according to one or more embodiments.
- a refrigeration system 200 of FIG. 2 A can be configured in accordance with examples such as described with FIG. 1 A through FIG. 1 E . Accordingly, the refrigeration system 200 provides an example of refrigeration system 100 , as described with other examples in this document, and reference made to elements of FIG. 1 A through FIG. 1 E illustrate suitable or like elements of the refrigeration system 100 , in context of refrigeration system 200 .
- the refrigeration system 200 includes a cooling subassembly 210 , one or more processors 220 , a camera interface 227 and/or a camera array 230 .
- the refrigeration system 200 can include one or multiple communication ports 240 , 242 .
- the communication ports can include wireless ports.
- one of the communication ports 240 can enable local device-to-device communications, using, for example, a Bluetooth protocol
- another of the communication ports 242 can enable network/Internet communications, such as through WiFi (e.g., wireless communications as promulgated under 802.11(g), 802.11(n), 802.11(ac), etc.), cellular communications or other wireless communication protocols.
- the cooling subassembly 210 includes a compressor unit, coolant handling mechanisms and other components for cooling an interior space 114 (see FIG. 1 A through FIG. 1 E ) of the cabinet 110 (see FIG. 1 A through FIG. 1 E ).
- the cooling subassembly 210 can include a compressor, evaporator, coolant lines, and temperature/power control to enable cooling of the interior space 114 of the refrigeration system 100 .
- the one or more processors (“processor(s) 220 ”) control the operations of the refrigeration system 200 , including control of the cooling subassembly 210 , while providing image control and/or processing functionality.
- the processor 220 receives, via the camera interface 227 , image data captured through individual cameras of the camera array 230 .
- the camera array 230 is an integrated component set of the refrigeration system 200 , meaning the camera array 230 is provided with the refrigeration system 200 at time of manufacturing or assembly.
- the camera array 230 may be provided as a separate module or component that is affixed to the housing of the refrigeration system 200 after manufacturing.
- the processor(s) 220 can also be distributed or located with the camera module 180 (see FIG. 1 F and FIG. 1 G ).
- the refrigeration system 200 includes or is connected to camera interface 227 to enable subsequent use of cameras, as described with various examples.
- the refrigeration system 200 can be manufactured to include ports for connecting the camera array 230 , or individual cameras. Subsequently, the camera array 230 can be added as an option during distribution, at point-of-sale, or by user (e.g., post-sale).
- the camera array 230 can be modularized, so as to include each camera of the camera array 230 .
- the individual cameras of camera array 230 can be connected or attached separately, and operated independently by the processor 220 .
- refrigeration system 200 can also include sensors or various types, such a (i) a door sensor 211 (to detect when the door is open, closed, in between, etc.); (ii) one or more rack sensors 212 , positioned to detect a position and/or movement by each of one or more corresponding racks; and (iii) a temperature sensor 213 to detect the temperature of the interior space, within the interior space 114 of the cabinet 110 .
- the processor(s) 220 receives temperature input from the temperature sensor 213 and implements temperature control operations 224 to cause the cooling subassembly 210 to maintain, increase, or decrease the temperature of the interior space 114 .
- the refrigeration system 200 includes a temperature control interface (not shown) to enable a user to adjust a desired temperature of the interior space 114 .
- the processor(s) 220 can control the cooling subassembly 210 to increase/decrease cooling of the interior space, and the temperature sensor 213 can generate output to enable the processor(s) 220 to implement the temperature control operations 224 to reach the target temperature.
- the processor(s) 220 implements image processing logic 222 to perform various types of image processing operations on racks when the racks are extended from the cabinet 110 (e.g., in the extended position).
- the image processing operations can include operations performed in connection with functionality that includes (i) presence detection (e.g., detecting whether items of a particular type are present on one of the racks 118 ); (ii) object detection, to determine a type of detected item (e.g., whether item placed on rack is a can, a bottle, or other type of beverage); (iii) hue, color contrast and/or opacity analysis (e.g., to determine a fill level or type of fluid within a container); (iv) label detection, to detect a label on a bottle; and/or (v) character and/or image recognition, to scan, for example, lettering and images of a label.
- presence detection e.g., detecting whether items of a particular type are present on one of the racks 118
- object detection to determine
- processor(s) 220 performs the operations for implementing the aforementioned functionality.
- the processor(s) 220 initiate the image processing operations by, for example, performing preliminary image processing steps.
- a normalized image of a label of an item, or alternatively, of the item itself is transmitted to another computing device, remote system or network service.
- the processor(s) 220 can implement operations to de-warp images captured by wide-eye (or ultra wide-eye) lens cameras. Dewarping operations can also be performed to normalize roundedness in images, caused by the shape of the bottles.
- the processor(s) 220 can implement operations to normalize and stitch the images of the different cameras.
- the images may be stitched by, for example, (i) dewarping images captured by multiple cameras, (ii) identifying an overlap region between two or more cameras that image the rack, (iii) removing the overlap region, and (iv) blending the remaining portion of the images.
- an image of the label of the item can be extracted and transmitted for further recognition and analysis.
- the processor(s) 220 performs select image processing operations, to enable or optimize performance of image processing functionality, as described with some examples.
- the processor(s) 220 may also initiate image processing and analysis by transmitting processed image data via one of the communication ports 240 , 242 to a remote computer system or service, to enable additional image processing to be performed by the remote system or service.
- the processor(s) 220 can capture and transmit image data to a remote service, computer, or computing device, where additional operations for presence detection, item detection, color/hue analysis, label detection and/or image/character recognition are performed.
- processor(s) 220 can implement image processing logic 222 to receive image data from the camera array 230 and initiate image processing and analysis by retransmitting the image data to another computer system for image processing.
- image processing logic 222 can be implemented by another computer system or systems, such as a network service, remote server and/or user mobile device.
- processor(s) 220 receives and processes images from camera array 230 . Further, processor(s) 220 performs different image processing operations for different sets of cameras of the camera array 230 . For example, for a first set of cameras that use ultra-wide angle lens cameras, processor(s) 220 implements operations to normalize the images. Still further, processor(s) 220 can normalize the images based on the distance of the rack where the images are captured and the respective lens of the corresponding camera. Still further, the processor(s) 220 can detect the rack in use (e.g., using rack sensor 212 , or through image processing), and perform a set of image processing operations based on the detected rack that is in use. Still further, the selected set of operations can also be based on the position of the items that are captured.
- processor(s) 220 receives and processes images from camera array 230 . Further, processor(s) 220 performs different image processing operations for different sets of cameras of the camera array 230 . For example, for a first set of cameras that use ultra-
- the processor(s) 220 implements image processing logic 222 to stitch images from multiple cameras together, to form a single image of items positioned on a given rack 118 .
- the processor(s) 220 can select images captured by particular combinations of the camera array 230 based on the rack 118 that is in use.
- the camera array 230 can implement image stitching operations, to stitch images of the rack captured by different cameras of the camera array 230 into a single image.
- the processor(s) 220 can selectively implement image stitching operations based on the particular rack in use and/or the proximity of the camera array 230 to the racks 118 .
- the camera array 230 can utilize ultra-wide lens cameras that are focused for a top rack 118 , or top set of racks.
- the processor(s) 220 implements image processing logic 222 to stitch images from, for example, a laterally positioned pair of ultra-wide lens cameras.
- the processor(s) 220 responds to the use of a rack (e.g., rack is pulled out, pushed in or left out in an extended position) by capturing multiple images of the rack 118 using individual cameras of the camera array 230 .
- the processor(s) 220 can implement the camera array 230 to select which image processing operations to perform, based on the rack that is detected in use. For example, in response to detecting the top rack 118 in use, the processor(s) 220 can select to use images from cameras that are optimized for the proximity of the top rack (e.g., ultra-wide eye lens cameras). In such implementations, the processor(s) 220 can implement image stitching to capture the entire width (X-direction) of rack 118 .
- the refrigeration system 200 includes a display 228 that displays an output that corresponds to or is based on rack information 225 .
- the display 228 can be touch-sensitive and/or provided with an interactive interface.
- the rack information 225 can be displayed as content to provide the user with information about the contents of the refrigeration system.
- the rack information 225 includes, for example, information about the contents of the interior space 114 .
- the rack information 225 identifies presence of items (e.g., bottles, beverage containers, etc.), one or more categories of items (e.g., type of beverage or container (e.g., bottle versus can)), sub-category of item (e.g., bottle of red wine versus white wine), state of the items (e.g., partially consumed, unopened, opened, etc.), orientation of item, and/or label information about individual items.
- the rack information 225 can identify information about an orientation of the item, such as a relative orientation of a label of a bottle.
- the orientation can reflect a determination of whether the label is, for example, (i) upward facing (e.g., so as to be visible to the camera array when the rack is pulled out), (ii) turned sideways, so that a portion of the label is skewed or not visible to the camera array, and/or (iii) turned so that a back label or surface of the item is visible, without much of the front panel.
- the rack information 225 can identify where individual items are located on a given rack (e.g., visually, by coordinates, by slot, etc.). Additional examples of rack information 225 , as well as other content which can be rendered with the display 228 , are provided with examples of FIG. 2 B .
- FIG. 2 B illustrates a network communication system for providing information about items stored in a refrigeration system, according to one or more embodiments.
- a network communication system 300 includes refrigeration system 200 and one or more user devices 272 , 274 , where user devices 272 , 274 communicate with the refrigeration system 200 using either a local wireless connection 242 (e.g., Bluetooth) or network connection (e.g., WiFi Direct).
- a local wireless connection 242 e.g., Bluetooth
- network connection e.g., WiFi Direct
- the refrigeration system 200 can communicate with user devices 272 , 274 indirectly, via a network service, server or remote computer.
- network communication system 300 includes one or more servers 280 or other remote computer systems.
- the server(s) 280 implement or otherwise provide a network service 282 that utilizes rack information transmitted by the refrigeration system 200 , via networks 292 .
- refrigeration system 200 transmits rack information 225 over one or more networks to servers 280 (or other remote computer systems) or network service 282 .
- the rack information 225 can include information obtained from performing image processing of images captured by camera array 230 , where the captured images depict items placed on individual racks 118 .
- the rack information 225 can be determined locally, on the refrigeration system 200 , by the processor(s) 220 . Alternatively, some or all of the rack information 225 can be determined by a remote computer, server or service. Accordingly, as described with an example of FIG.
- the rack information 225 includes (i) information that identifies a presence of items (e.g., bottles, beverage containers, etc.) on individual racks, (ii) information about one or more categories (e.g., type of beverage or container) or sub-categories of the individual items (e.g., type of wine (red, wine, rose, etc.), (iii) information about a state of individual items (e.g., open or uncorked, closed, previously opened, partially consumed, fill state, etc.); and/or (iv) label information for individual items (e.g., brand, vintage, trade name, information about source of origin, etc.).
- categories e.g., type of beverage or container
- sub-categories of the individual items e.g., type of wine (red, wine, rose, etc.
- a state of individual items e.g., open or uncorked, closed, previously opened, partially consumed, fill state, etc.
- label information for individual items e.g., brand, vintage
- An end user can access content that is based on rack information 225 using, for example, a mobile device that communicates or otherwise utilizes network service 282 .
- the mobile device can execute an application (“app”) that is dedicated for the network service 282 , in order to view content that is at least partially based on the rack information 225 .
- an application (“app”) that is dedicated for the network service 282 , in order to view content that is at least partially based on the rack information 225 .
- a user can operate a computer on which a browsing component is installed in order to view such content.
- refrigeration system 200 transmits rack information 225 to user device 272 through, for example, a local wireless connection.
- the user device 272 can execute an application (e.g., dedicated application, browsing component, etc.) to view content that is based at least in part on the rack information 225 .
- an application e.g., dedicated application, browsing component, etc.
- the user devices 272 , 274 generate rack content 235 that incorporates or is otherwise based on the rack information 225 . Accordingly, the user devices 272 , 274 can display content that is similar or the same as that provided on the display 228 of refrigeration system 200 .
- the rack content 335 includes an item/rack map that shows each detected item and a corresponding rack where the item is located. Additionally, the item/rack map can identify a position along the span of the rack (along X, Z directions) where the item is detected as being located. Additionally, the rack content 235 can include detailed information about each item, including information determined from performing image processing of the respective item, as well as supplemental information.
- the supplemental information can include information that is not determined from the label or item itself, but rather determined from information sources based on information obtained from the label or with the item.
- the maker and vintage can be cross reference with the information repository about the particular winemaker, and the vintage year, to obtain supplemental information that includes, for example, descriptive information about how the wine was made, and particular aspects or attributes of the wine, such as awards which the wine may have one, and taste attributes of the wine which are determined by experts and not necessarily included on the label of the wine.
- the rack content 235 can recommend types of food that may go well with the particular wine, by referencing identifiers (e.g., vintage, maker, trade label, source of origin, etc.), categories (e.g., “burgundy”, “pinot”, etc.) or other aspects that are determined about the wine through image processing (e.g., by image recognition of label information) against information repositories were food recommendations are maintained.
- identifiers e.g., vintage, maker, trade label, source of origin, etc.
- categories e.g., “burgundy”, “pinot”, etc.
- the rack content 235 is provided as an interactive interface.
- a user can interact with graphic representations of items identified as being present in the individual racks, in order to view supplemental information about the individual items.
- a user can view a listing of items which are present in the refrigeration system 200 , and through interaction with a given entry of the list, receive output that indicates a rack (or rack position) where the item is being stored.
- examples provide for additional information about the item to be provided.
- the additional information can include, for example, information about the state of the item (e.g., open or uncorked, closed, previously opened, partially consumed, fill state), recommendation about how the item can be consumed (e.g., with type of food, etc.), and supplemental information.
- the rack content 235 can include a list of recommendations.
- the list of recommendations can identify, for example, other items that may be of interest to the user based on their affinity for a particular item. For example, historical information can be maintained about items over the course of a duration, and information about preferences of the user (e.g., winemaker, wine type, vintage, aspects of the particular item, etc.) can be maintained and analyzed in order to determine preferences of the user. Based on user preferences, and or items which the user may have stored in the refrigeration system 200 , the rack content 235 can provide recommendations regarding additional items which the user can purchase.
- the rack content 235 can include an order interface that enables users to purchase items from a third-party source.
- a user can utilize the order interface to purchase a refill of a consumed item, such as a replacement wine bottle.
- the refrigeration system 200 can monitor items for consumption level, based on image processing that the text when items are taken offer rack in place back on it, as well as indications that an item was consumed or partially consumed.
- the rack content 335 can provide the order interface to enable the user to reorder the consumed item.
- the order interface can enable the user to purchase recommended items, or items which the user searches for or is otherwise provided information about.
- FIG. 3 A through FIG. 3 C illustrate example methods for operating a refrigeration system, according to one or more embodiments.
- a first scan or imaging processing is performed on a first rack 118 of a refrigeration system 100 , 200 when the rack 118 is positioned exterior to cabinet 110 of the refrigeration system.
- a second scan or imaging processing is performed on a second rack 118 of the refrigeration system 100 , 200 , when the second rack is positioned exterior to the cabinet 110 , where the second rack is different than the first rack.
- the first rack may correspond to a topmost rack 118 of the refrigeration system 100
- the second rack may correspond to a middle or bottom rack 118 .
- Each of the first and second image processing includes operations where image data, captured by corresponding cameras, is processed to enable recognition and other analysis.
- a result of each image processing can be to generate a normalized image of a label of an item, or of the item itself. Further, the normalized image can include optimizations to enhance recognition and subsequent analysis.
- the first and second imaging processes is performed on images captured as the respective first and second racks 118 , 118 .
- the first and second imaging processes can be performed when a rack is moved inward, outward, or positioned statically outside of the cabinet 110 .
- operations of the first imaging process are different than operations of the second imaging process.
- the first imaging process may be performed using a first set of cameras that are of a first type, while the second imaging process is performed using a second set of cameras that are of a second type.
- the first and second set of cameras may be different to accommodate the different distances between the individual racks and the respective cameras.
- the types of image processing that are performed may also be different.
- the image processing that is used (or prioritized) can be specifically configured for wide-lens cameras, including ultra-wide lens (or fisheye lens) cameras. Further, the image processing can use multiple cameras, where captured images are stitched into a single view.
- the image processing can accommodate higher resolution and alternative lighting conditions.
- the number and type of cameras which are used to image different racks 118 may differ based on the proximity of the respective rack to the corresponding cameras.
- the image processing can also differ based on the proximity of the cameras to the rack and/or the type of cameras in use.
- step 320 the rack of a refrigeration system 100 , 200 , is detected as being moved from an extended position to a seated position within interior space 114 of cabinet 110 .
- step 322 as the rack 118 is moved into the interior space 114 , the camera array 230 captures images of the contents of the rack 118 .
- the images correspond to a series of frames, such as may be captured by video.
- image analysis is initiated by the processor(s) 220 of the refrigeration system 100 , 200 .
- the processor(s) 220 can, for example, implement operations to select which images are to be analyzed for item detection, item recognition, label detection and recognition, etc.
- the operations can include selecting the image set to be analyzed, and more specifically, selecting which camera set of the camera array 230 to utilize for processing.
- the processor(s) 220 can normalize images, such as performing image stitching as needed, before transmitting the images to a remote computer or service for further processing. Still further, in other variations, the processor(s) 220 can implement operations to perform object detection and image/character recognition.
- the processor(s) 220 determines or otherwise obtains the results of the image analysis, and store the results in memory and/or generate content based on the results for the display. As an addition or variation, results are generated or otherwise provided to user devices.
- the results of the image analysis includes determination of the type of item.
- the item can be determined by type of container (e.g., bottle, can, box, etc.), and/or by type of beverage (e.g., soda versus hard alcohol).
- the results of the analysis identifies a position of the item.
- the position can associate an item with a particular rack 118 , as well as with a location of the item on the rack. For example, the coordinate of the item along a lateral axis (X or Z direction) can be determined. Furthermore, if the item is removed and repositioned on a rack, the new position of the item can be determined and recorded.
- step 327 information about the item is determined using recognition analysis.
- the processor(s) 220 capture images and perform label detection and normalizing. Subsequent OCR and image analysis can be used to determine additional information about the beverage.
- label detection can be used to generate a normalized image of a label of the item, and subsequent analysis is performed to determine information about the item.
- the additional information can identify a type of beverage (e.g., wine, champagne, soda, etc.), one or more sub-categories of the item (e.g., color of wine), and manufacturer or brand.
- the information can identify a maker, a year, source (e.g., country or geographic region), and descriptive terms (e.g., alcohol content) which may be provided on the label.
- step 330 the processor(s) 220 initiate and perform image processing using images captured by a camera set that is dedicated or prioritized for a rack that is in use.
- the rack in use can correspond to one that is in movement (e.g., moved from extended to seated position).
- step 332 provides that images are selected from a camera set that is optimized or otherwise configured for the particular rack.
- the configuration may include type of camera/lens, focus attributes, and subsequent image processing.
- the top-most rack 118 can be imaged using (i) multiple cameras, (ii) ultra-wide lens cameras, and/or (iii) image stitching, such that images from multiple cameras can be formed into a single image for processing.
- images from multiple camera sets can be processed and prioritized by weighting.
- the processor(s) 220 and subsequent processing/analysis of images can utilize multiple camera sets, with a non-prioritized camera set providing information for specific purposes (e.g., hue analysis), edge analysis and/or determination of confidence values and assessment.
- step 340 image analysis is performed to determine information about the items placed on the rack 118 .
- the processor(s) 220 of the refrigeration system 100 , 200 perform image analysis by implementing recognition algorithm locally.
- a remote computer system or device performs operations of the image analysis.
- the processor(s) 220 of the refrigeration system 100 , 200 and/or the remote computer system or device performs image analysis by transmitting image data to remote systems or services, including third-party services which return recognition results.
- step 342 information is determined about the type of the item. For example, the presence of a new item on a rack is determined. Further, the type of item can be detected by type, based on or corresponding to container type (e.g., can, box, bottle, etc.) or item shape.
- step 343 the position of the item on the rack 118 may be determined, where the position is based on, for example, coordinates for the rack and/or a particular slot position.
- step 344 additional information about individual items are determined from recognition operations.
- the recognition operations can include character image recognition, logo recognition, and/or pattern analysis of the imaged labels for each item.
- label detection and analysis can be performed.
- the analysis can implement, for example, label detection, logo detection, image analysis (of imagery on label), optical character recognition (OCR) and/or other operations for determining information such as vintage, maker, source, type of wine and other categories.
- opacity and/or hue analysis is performed.
- the opacity information can be used to determine the state of the item. For example, variations in the opacity/hue of the item can reflect the fill level of the item.
- cork or lid recognition can be performed to determine, for example, whether the item has been opened. For example, for corks, the height of the cork above the top of a bottle can reflect whether the bottle was uncorked. Likewise, if a lid is missing a bottom portion, the lid may be deemed as having been previously opened.
- the image analysis that is performed results in an identifier of the item, and processed images of the item are stored.
- the identifier of the item can be based in part on the label analysis.
- an image of the item may be stored in association with the identifier, along with determinations about the characteristics of the hue, opacity and/or lid.
- the characteristics of the hue, opacity and/or lid for the same item can be compared with the stored images to determine changes which are indicative of the item being opened (e.g., uncorked) and/or partially consumed.
- the determinations of the image analysis is transmitted and/or rendered.
- the determinations can be by the refrigeration system 100 , 200 , to another device using a local wireless port (step 352 ).
- the determinations can be by the refrigeration system 100 , 200 to a computing device of the user, using, for example, one of the wireless communication ports 240 , 242 (step 354 ).
- a remote computer system or service can make the determinations in whole or in part, and transmit the determinations to another device using a network connection.
- FIG. 4 illustrates a computer system on which one or more embodiments can be implemented.
- a computer system 400 can be implemented on, for example, a server or combination of servers.
- the computer system 400 may be implemented as part of a network communication system 300 , such as described with an example of FIG. 2 B .
- the computer system 400 can implement operations such as described with FIG. 2 A , and with steps or sub-steps of example methods such as described with FIG. 3 A through FIG. 3 C .
- the computer system 400 includes processing resources 410 , memory resources 420 (e.g., read-only memory (ROM) or random-access memory (RAM)), a storage device 440 , and a communication interface 450 .
- the computer system 400 includes at least one processor 410 for processing information stored in the memory resources 420 , such as provided by a random-access memory (RAM) or other dynamic storage device, for storing information and instructions which are executable by the processor 410 .
- the memory resources 420 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by the processor 410 .
- the computer system 400 may also include the memory resources 420 or other static storage device for storing static information and instructions for the processor 410 .
- the storage device 440 such as a magnetic disk or optical disk, is provided for storing information and instructions.
- the communication interface 450 enables the computer system 400 to communicate with one or more networks (e.g., cellular network) through use of the network link 480 (wireless or a wire).
- the computer system 400 can communicate a refrigeration system, user device or other computing system for implementing one or more embodiments as described.
- the executable instructions 442 stored in the memory resources 420 can include instructions 442 , to implement a remote computer system, network service and/or network communication system (e.g., see FIG. 2 B ), as described with one or more embodiments.
- the executable instructions stored in the memory resources 420 may also implement operations for implementing steps such as described with examples of FIG. 3 A through FIG. 3 C .
- examples described herein are related to the use of the computer system 400 for implementing the techniques described herein.
- techniques are performed by the computer system 400 in response to the processor 410 executing one or more sequences of one or more instructions contained in the memory resources 420 .
- Such instructions may be read into the memory resources 420 from another machine-readable medium, such as the storage device 440 .
- Execution of the sequences of instructions contained in the memory resources 420 causes the processor 410 to perform the process steps described herein.
- hard-wired circuitry may be used in place of or in combination with software instructions to implement examples described herein.
- the examples described are not limited to any specific combination of hardware circuitry and software.
- FIG. 5 is a block diagram illustrating a user device for use with examples as described.
- a user device 500 may execute a designated service application that utilizes rack information 525 to provide rack content 535 and other information.
- the user device 500 operates to communicate with a network service, through one or more networks.
- user device 500 includes a mobile computing device, such as a smartphone, tablet computer, laptop computer, VR or AR headset device, and the like.
- the user device 500 includes wireless communication interfaces 510 , 512 to communicate with external entities using any number of wireless communication protocols.
- the user device 500 can receive rack information 535 from the network system 502 (or other remote computer system or device) through, for example, a wireless network interface (e.g., cellular radio, WiFi, etc.).
- a wireless network interface e.g., cellular radio, WiFi, etc.
- the user device 500 can receive rack information 535 from the refrigeration system 100 , 200 using a local wireless communication protocol (e.g., Bluetooth).
- a local wireless communication protocol
- the user device 500 stores a designated application (e.g., a service app 532 ) in a local memory 530 .
- a designated application e.g., a service app 532
- the local memory 530 can store additional applications executable by one or more processors 520 of the user device 500 , enabling access and interaction with one or more host servers over one or more networks 580 .
- the service application 532 executes on the user device 500 to generate rack information 535 on a display screen 540 of the user device 500 .
- the rack information 535 can be displayed as part of a user interface 545 , where the user interface 545 includes interactive features (e.g., menus, selectable features, search functions, etc.), and enable the consumer view their inventory of beverages, purchase additional beverages, view recommendations relating to items in their inventory and other information.
- the user interface 545 can present a rack map, where items are shown by rack and, in some variations, by position on respective racks.
- FIG. 6 A through FIG. 6 C illustrate an example of a beverage cooler, according to one or more examples.
- a beverage cooler 600 can include configurations that enhance or optimize the user of the beverage cooler 600 .
- the beverage cooler can be structured with a camera array, processor and resources to determine rack information, as described with various examples.
- the beverage cooler 600 includes a housing 610 , having a door 620 and door handle 612 .
- the door inset can be mounted flush with a contoured top segment 615 of the housing 610 .
- the door 620 can be contoured and shaped to match the cabinet 610 , in such manner that the door 620 can be fully recessed and flush with respect to a facade of the top housing segment 629 .
- a housing 610 of the beverage cooler 600 includes an integrated tray 612 , positioned on a top surface 611 .
- the integrated tray 612 can be integrated with the top surface to enable the housing 610 to serve as a flat top surface to receive items, without affecting the design or function of the beverage cooler 600 .
- the tray 612 can include a lip or other protrusion to contain items that are received on it.
- the top surface 611 of the housing 610 can be shaped or structured to retain structures (e.g., outside of the tray 612 ).
- the top surface 611 can, for example, be contoured inwards or provided a lip or upward protrusion, separate from the tray 612 .
- the integrated tray 612 is unitarily formed (e.g., through a common manufacturing process, such as injection molding or metal pressing, with the remainder of the housing).
- the tray 612 is integrated, meaning the tray can be connected (e.g., using a fastener). In such examples, the integrated tray 612 can be attached and detached from the housing 610 .
- the beverage cooler 600 includes a compressor housing segment 630 positioned to extend from a bottom surface 613 of the housing 610 .
- the compressor housing segment 630 can include an inclined front facade 632 that tapers the 630 inward.
- the housing 610 can further be lifted from the ground by legs 636 .
- the legs 636 can telescope to enable the vertical height of the housing 610 to be adjustable.
- the inclined front facade 632 creates a void in a front region underneath the housing 610 .
- the void can allow for presence of footing, and enable sweeping and cleaning, without adding to the overall height of the housing.
- the front facade 632 can create a void under the compressor housing segment 630 , as well as to the sides of the compressor housing segment 630 . The provision for such voids allows for improved airflow from underneath the housing and to its sides.
- the compressor housing segment 630 can house a fan that blows warmed air outward, away from the compressor assembly.
- the air can blow downward into the ground, rather towards the feet of a person or in other directions where the warmed air may be trapped.
- the beverage cooler 600 can be pushed on its rear face 609 towards a wall, without need to accommodate an airgap for the blown air.
- the cabinet 610 can be fitted in tight spaces and flush-mounted under a counter or inside cabinetry, as a result of the blown air being directed forward and downward directly underneath the cabinet 610 .
- a rear surface 609 of the housing 610 can be mounted substantially flush to a wall or cabinet, with a back surface 609 being flat.
- the compressor housing segment 630 can also be flat on the back facade to eliminate or reduce separation between a backwall and the unit.
- FIG. 6 D illustrates a variation in which the beverage cooler 600 includes a ribbed door 628 to promote light diffusion.
- the ribbed door 628 or facade can be formed from ribbed or wavy glass or other translucent material.
- the door 628 can be partially constructed from ridged glass (or translucent material), such as bottle glass.
- the door can be formed from partially opaque glass or translucent material. The use of light-diffusive material can protect certain types of items, such as vintage wines, from deterioration by light.
- FIG. 7 A and FIG. 7 B illustrate a reversible rack for a refrigeration system, according to one or more examples.
- a rack 700 can be integrated with refrigeration systems as described with various examples, including FIG. 1 A through FIG. 1 G and FIG. 6 A through FIG. 6 D .
- the rack 700 includes a reversible bottom tray 720 that is retained by a rack structure 710 .
- the tray 720 can include first and second layers 712 , 714 can each include a corresponding arrangement of spaced support structures, where the spaced support structures can correspond to wires, rods, or similar typed structures.
- a spacing between the support structures of the first layer 712 can be greater than the spacing between the support structures of the second layer 714 .
- the variation in spacing can be implemented by, for example, varying the thickness of the support structures of the first layer 712 as compared to the second layer 714 .
- tray 720 can be reversible when positioned at the bottom of the area formed by the rack structure 710 .
- the first layer 712 is positioned on top (relative to the second layer 714 ).
- the second layer 714 is positioned on top.
- the tray 720 can be retained within the rack structure 710 by a combination of retention structures 716 (e.g., clips, shelf-support pins and bolts, etc.).
- retention structures 716 e.g., clips, shelf-support pins and bolts, etc.
- a user can manipulate the retention structures 716 on inside surfaces of the rack structure 710 to enable the tray 720 to be separated, flipped and inserted back again into the rack structure 710 .
- the first layer 712 of the tray structure 720 is top-facing.
- the first layer 712 can include spacing between support structures that are dimensioned to retain wine bottles.
- the second layer 714 can form a base structure to hold smaller bottles or items.
- the arrangement of the first layer 712 can retain wine bottles in a width-wise direction, meaning the wines can be provided on the tray 720 sideways relative to the direction of movement of the rack 700 (which can move in and out of the cabinet in the Z direction).
- the user can flip the tray 720 such that the second layer 714 is top-side.
- the second layer can support items of various dimensions, including cans or small bottles that would otherwise pass through conventional trays designed to hold wine bottles. Further, the second layer can also hold wine bottles or other similarly shaped items. However, the arrangement of the second layer 714 also enables wine bottles to be retained lengthwise relative to the movement of the rack (i.e., along Z direction).
- the user can choose which side of the tray 720 to orient topside based on preference with regards to orientation of wine bottle, and/or the size or dimension of items which the tray 720 may retain.
- the rack 700 can be provided as an accessory for a refrigeration system such as shown by various examples. Still further, a refrigeration system such as shown by various examples can include one or multiple racks 700 .
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Thermal Sciences (AREA)
- Mechanical Engineering (AREA)
- Combustion & Propulsion (AREA)
- Chemical & Material Sciences (AREA)
- Signal Processing (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Software Systems (AREA)
- Medical Informatics (AREA)
- General Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- Databases & Information Systems (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Cold Air Circulating Systems And Constructional Details In Refrigerators (AREA)
Abstract
In examples, a refrigeration system includes one or more cameras that are positioned to capture images of items before the items are positioned inside the cabinet. The refrigeration system can include one or more moveable racks on which items can be placed. When the rack is moved out, items can be placed on the rack, and the set of cameras can image the rack to determine information about the items placed on the rack.
Description
- This application claims benefit of priority to Provisional U.S. Patent Application No. 63/264,720; filed Dec. 1, 2021; the aforementioned priority application being hereby incorporated by reference in its entirety.
- Refrigeration units are commonplace in homes. In recent years, specialized refrigeration units have been developed for specific types of perishable and consumable goods. For example, freezers exist to store frozen items (e.g., meat). Beverage coolers allow people to maintain beverages in colder temperatures. Wine coolers referred to a general class of refrigeration assemblies which are designed specifically to maintain wine bottles and other beverages. Wine coolers typically include racks that can be extended outward to enable placement of bottles. This allows for the wine cooler to keep bottles on their side (as is customary).
-
FIG. 1A is a frontal view of a refrigeration system in a closed orientation, according to one or more embodiments. -
FIG. 1B is a frontal view of the refrigeration system ofFIG. 1A in the closed orientation, according to one or more embodiments. -
FIG. 1C is a side cross-sectional view of the refrigeration system in the closed orientation, along lines A-A ofFIG. 1A , according to one or more embodiments. -
FIG. 1D is a side cross-sectional view of the refrigeration system in the open orientation, along lines B-B ofFIG. 1B , according to one or more embodiments. -
FIG. 1E is a simplified diagram of viewing angles used by a camera array of a refrigeration system shown byFIG. 1A throughFIG. 1D , according to one or more embodiments. -
FIG. 1F illustrates a variation in which an example refrigeration system is implemented to include a camera module as a retrofit, according to one or more embodiments. -
FIG. 1G illustrates another variation in which an example refrigeration system is implemented coupled with thecamera module 180, where thecamera array 160 is housed. -
FIG. 2A is a hardware diagram of an example refrigeration system, according to one or more embodiments. -
FIG. 2B illustrates a network communication system for providing information about items stored in a refrigeration system, according to one or more embodiments. -
FIG. 3A throughFIG. 3C illustrate example methods for operating a refrigeration system, according to one or more embodiments. -
FIG. 4 illustrates a computer system on which one or more embodiments can be implemented. -
FIG. 5 is a block diagram illustrating a user device for use with embodiments as described. -
FIG. 6A throughFIG. 6C illustrate an example beverage cooler, according to one or more embodiments. -
FIG. 6D illustrates an example beverage cooler with a ribbed, light-diffusive door, according to one or more embodiments. -
FIG. 7A andFIG. 7B illustrate a reversible rack for a refrigeration system, according to one or more embodiments. - Embodiments include a refrigeration assembly or system (referred herein as “refrigeration system” for convenience(that utilizes a set of cameras to image items when the items are placed on a rack that is extended out of an interior space of the refrigeration system. For example, one or more cameras of the refrigeration system can image items on a rack, such as when the rack is extended or being pushed into a seated position. The refrigeration system can initiate image processing and analysis to determine information about the items on the rack. The information can then be transmitted to a user device and/or rendered on a display.
- Among other benefits, examples enable a user to know information about items retained within a refrigeration system or appliance. Further, the processing and imaging that is performed can require minimal or no separate actions from a user. For example, the user does not have to handle the items in a particular manner, other than place the items on a rack (preferable label up, if applicable) and push the rack in—actions which the user would do in any case. Thus, the user can avoid tedious tasks such as holding items in front of a camera.
- Additionally, examples generate content that enables the user to learn about items stored with a refrigeration system or appliance. In some examples, the refrigeration system or appliance is implemented as a beverage cooler to hold wine, liquor, canned beverages and the like. With wine, for example, users often have a need to know their inventory by type and category (e.g., number of red wines, which vintage, attributes of wine, etc.). Further, users may also want to know which wines have been opened (e.g., so they can drink those first), and the fill level of bottles. Various embodiments enable the user to determine such information without the user having to open the unit, thereby saving user convenience and energy.
- In some examples, a refrigeration system is operated by (i) detecting a given rack of multiple racks of the refrigeration system being moved from an extended position to a seated position within an interior of the refrigeration system; (ii) capturing one or more images of the rack; and (iii) initiating performance of, or performing, image analysis of the one or more images to determine information about one or more items that are placed on the rack.
- Still further, in examples, a refrigeration system implements operations for performing a first image scan of a first rack using a first set of cameras, where the first image scan is performed when the first rack is outside of an interior space of the refrigeration system, and where a portion of the first rack is a first distance from the first set of cameras. The operations further comprise performing a second image scan of a second rack using a second set of cameras, where the second image scan is performed when the second rack is outside of an interior space of the refrigeration system, and where the second rack is a second distance from the second set of cameras, the second distance being greater than the first distance.
- In examples, a camera module is provided for a refrigeration system, where the camera module includes a set of one or more cameras and a housing that can attach to a cabinet or structure of a refrigeration system. In examples, the housing can be structured to secure to a refrigeration system, to position the camera array such that a corresponding viewing angle of one or more cameras of the camera array are positioned over a vertical space that coincides with a thickness that occupies a door of the refrigeration system when the door is in a closed position. The refrigeration system can also include one or more processors, that execute instructions to perform operations that include: (i) capturing one or more images of a rack of the refrigeration system when the rack is extended from an interior cabinet; and initiating performance of image analysis of the one or more images to determine information about one or more items that are placed on the rack.
- One or more embodiments described herein provide that methods, techniques, and actions performed by a computing device are performed programmatically, or as a computer-implemented method. Programmatically, as used herein, means through the use of code or computer-executable instructions. These instructions can be stored in one or more memory resources of the computing device. A programmatically performed step may or may not be automatic.
- One or more embodiments described herein can be implemented using programmatic modules, engines, or components. A programmatic module, engine, or component can include a program, a sub-routine, a portion of a program, or a software component or a hardware component capable of performing one or more stated tasks or functions. As used herein, a module or component can exist on a hardware component independently of other modules or components. Alternatively, a module or component can be a shared element or process of other modules, programs or machines.
- Some embodiments described herein can generally require the use of computing devices, including processing and memory resources. For example, one or more embodiments described herein may be implemented, in whole or in part, on computing devices such as servers, desktop computers, cellular or smartphones, tablets, wearable electronic devices, laptop computers, printers, digital picture frames, network equipment (e.g., routers) and tablet devices. Memory, processing, and network resources may all be used in connection with the establishment, use, or performance of any embodiment described herein (including with the performance of any method or with the implementation of any system).
- Furthermore, one or more embodiments described herein may be implemented through the use of instructions that are executable by one or more processors. These instructions may be carried on a computer-readable medium. Machines shown or described with figures below provide examples of processing resources and computer-readable mediums on which instructions for implementing embodiments of the invention can be carried and/or executed. In particular, the numerous machines shown with embodiments of the invention include processor(s) and various forms of memory for holding data and instructions. Examples of computer-readable mediums include permanent memory storage devices, such as hard drives on personal computers or servers. Other examples of computer storage mediums include portable storage units, such as CD or DVD units, flash memory (such as carried on smartphones, multifunctional devices or tablets), and magnetic memory. Computers, terminals, network enabled devices (e.g., mobile devices, such as cell phones) are all examples of machines and devices that utilize processors, memory, and instructions stored on computer-readable mediums. Additionally, embodiments may be implemented in the form of computer-programs, or a computer usable carrier medium capable of carrying such a program.
-
FIG. 1A is a frontal view of a refrigeration system (or assembly) in a closed orientation, according to one or more embodiments.FIG. 1B is a frontal view of the refrigeration system ofFIG. 1A in the closed orientation, according to one or more embodiments.FIG. 1C is a side cross-sectional view of therefrigeration system 100 in the closed orientation, along lines A-A ofFIG. 1A , according to one or more embodiments.FIG. 1D is a side cross-sectional view of therefrigeration system 100 in the open orientation, along lines B-B ofFIG. 1B , according to one or more embodiments. Arefrigeration system 100, as shown and described by examples ofFIG. 1A throughFIG. 1D , can correspond to a beverage cooler, with structural and functional features that are optimized for maintaining wine bottles and beverages in a suitable environment. In variations, embodiments can be employed with other types of refrigeration systems. As described in greater detail, therefrigeration system 100 is configured to perform image processing and analysis to determine information about items and content stored within therefrigeration system 100. - With reference to
FIG. 1A , therefrigeration system 100 includes acabinet 110, adoor 120 and acooling subsystem 130. When therefrigeration system 100 is closed, thedoor 120 is in the closed position, where an interior space of the cabinet is sealed by thedoor 120. In examples, thedoor 120 includes an exterior facade and anoptional handle 112. To open thedoor 120, a user can pull thehandle 112, so that the door swings outward, to provide the user with access to the interior space of thecabinet 110. - To accommodate wine bottles, and more generally beverage containers, an
interior space 114 of therefrigeration system 100 includesracks 118 that are movable laterally outwards (in the Z direction) whencabinet 110 is in the open position. Theinterior space 114 is maintained at a desired temperature for cooling items onracks 118. As a beverage cooler, theinterior space 114 can be maintained at temperatures that range between, for example, 33° F. and 70° F. As a wine cooler, theinterior space 114 can be maintained at temperatures that range between 44° F. and 65° F. Thus, the specific temperature can be selected by user preference and/or the type of item being stored. Still further, in other examples, the temperature setting can be set automatically based at least in part on one or more items placed on the rack. For example, as described with examples, thecameras refrigeration system 100 can include one or more processor(s) 220 (seeFIG. 2A ) that can process images of the items to determine the information, and automatically implement operations to control the operation of therefrigeration system 100 based on the detected information. - The
cabinet 110 includes atop segment 111 that integrates or otherwise provides a camera array, where the camera rate includescameras camera 122 is proximate (i.e., closest to) to left side 101 of thecabinet 110, andcamera 126 is proximate to a right side 103 of thecabinet 110. Thecameras camera 124 may be of a different type. For example, thecameras camera 124 includes a high definition lens. Thecameras racks 118 that are closest to the lenses of the camera array. - With reference to
FIG. 1B , thedoor 120 is shown in an open position, to provide access to aninterior space 114 of thecabinet 110. Therack subassembly 140 is mounted to an interior of the cabinet, where therack subassembly 140 maintainsmultiple racks 118 that can be moved laterally (e.g., in Z direction) in and out of the cabinet (or between a seated position and an extended position) while thedoor 120 is in the open position. As described in greater detail, therack assembly 140 enablesindividual racks 118 to be slid out. When a given one of theracks 118 is slid out, the user can place bottle(s) on therack 118, remove bottle(s) (e.g., for consumption) from therack 118, reposition bottles on therack 118, add bottles to therack 118, and/or remove a bottle for partial consumption and then return it to therack 118. The user can then slide the rack back in two theinterior space 114 of thecabinet 110. As described in greater detail, the array ofcameras - With reference to a cross-sectional view shown
FIG. 1C (door in closed position), thecameras 122, 124 (not visible inFIG. 1C ), and 126 (not visible inFIG. 1C ) a provided in the top segment of thecabinet 110. Eachcamera 122 124, 126 is oriented downward, such that the field of view of the respected lens(es) is blocked by the thickness of thedoor 120. In examples as shown, thecameras top rack 118 occludes other racks. Rather, the cameras are position over thedoor 120, so that when the door is open and racks are slid in and out, thecameras - With reference to a cross-sectional view shown
FIG. 1D (door in open position), eachrack 118 is movable laterally (in the Z direction) to provide a user with access to items that are held by that rack. Thecameras 122, 124 (not visible inFIG. 2D ), 126 (not visible inFIG. 2D ) are oriented downward such that the downward viewing angle coincides or overlaps with a space that is occupied by a thickness of thedoor 120 when the door is in the closed position. - In examples, each
camera cabinet 110. The top segment 128 can be an integral or unitaryhousing segment cabinet 110. In other examples, the top segment 128 can be a void where thecameras camera space 125 that occupies a thickness of thedoor 120 when the door is in the closed position. In examples, thecameras individual cameras cameras - In an example shown by
FIG. 1D , a user places abottle 12 on atop rack 118 when the rack is slid outward into theextended position 119. As therack 118 is slid inward, the bottle passes across the field of view of thecameras position 117, and thecameras bottle 12. Additionally, thecameras rack 118 when it static, or moved outward from thecabinet 110. - As shown with an example of
FIG. 1D , embodiments enable for image capture and processing ofbottle 12 with minimal or no additional user interaction, other than the user placing thebottle 12 on the desiredrack 118. Further, because of the configuration of the camera array, the camera array is able to capture recognizable images of the bottle even as the bottle is passed within a relatively short distance of thecameras interior space 114 of thecabinet 110 individually. - While examples as described illustrate the
rack 118 supporting the item (e.g., bottle) in a first orientation (e.g., lengthwise along Z axis), in variations, therack 118 can be configured to alternatively, or additionally, support items in an alternative orientation (e.g., widthwise along Z axis). -
FIG. 1E is a simplified diagram of viewing angles used by a camera array of therefrigeration system 100, according to one or more embodiments. While examples described provide for thecamera array 160 to include 3 lenses, in variations, more or fewer cameras may be utilized. In some examples, thecamera array 160 includesmultiple cameras plane 121 or opening of thecabinet 110. The individual cameras ofcamera array 160 can be operated in sets of one or more cameras, where each set is a prioritized or primary source for obtaining recognizable images of items placed on correspondingracks 118 or regions thereof. Based on implementation, one camera of thecamera array 160 can be used in multiple sets. In some variations, the individual cameras of each camera set are preconfigured, or dynamically controlled, to operate in a coordinated fashion, such as to obtain images simultaneously. - Accordingly, in examples, the
camera array 160 includes or otherwise provides alternative camera sets for imagingspecific racks 118 based on the vertical distance of the respective racks and the lenses of the cameras (which for purpose of discussion here, are assumed to be coplanar with the 121). In examples, a distance between the 121 and the top-most rack can be in the range of 40-75 mm. Likewise, a distance between the 121 and the second rack (from the top) may range between 120-180 mm, and a distance between the third from the top can be in the range of 200-250 mm. Still further, in other variations, the operation of the camera sets, including the selection, operation and/or configuration of cameras, can based on switching elements that are triggered by movement of theracks 118. - Embodiments recognize that different camera sets may enable or facilitate the operations of obtaining recognizable image data from each
rack 118 when such racks are used. For items placed in thetop rack 118, the proximity of the top rack with the 121, combined with a standard field-of-view of a camera lens, would otherwise limit a span of a camera's coverage in the X-direction. Further, the ability of the camera to focus on items in such proximity may be limited. Similarly, items placed on the second rack can have similar challenges with respect to imaging by the camera array—the field-of-view from a single camera does not span (in the X-direction) the entire width of the rack, and the items placed on the second rack may be too close to the respective cameras to enable sufficient focus. - For middle or
bottom racks 118, the field-of-view for cameras may present a lesser requirement with respect to the field of view that would otherwise be required from an imaging camera. However, the vertical distance between the respective cameras and the racks may pose an additional challenge with respect to obtaining images with adequate definition of text that can be subsequently recognized using optical character recognition (OCR). - Additionally, the
camera array 160 can be configured to accommodate additional optimizations. Specifically, thecamera array 160 can include a minimal number of cameras to image a full span of each rack with sufficient definition to enable subsequent image recognition (e.g., OCR)and other image processing to be performed. For spacing as described, one optimal configuration forcamera array 160 provides for placement ofultra-wide lens cameras cabinet 110, to capture images of items placed on the top tworacks 118. Further, a camera (e.g., HD camera) can be positioned as the middle of the camera array to capture images of items placed on other racks. In other variations, an HD camera can be positioned towards the lateral ends of the camera array 160 (e.g., ascameras 122, 126). - In one implementation, the
cameras - While numerous examples illustrate the
cameras refrigeration system 100 with a downward viewing angle, in variations, thecameras -
FIG. 1F illustrates a variation in which therefrigeration system 100 is implemented to include acamera module 180, where thecamera array 160 is housed. In examples, arefrigeration system 100 includes acamera module 180 having ahousing 182 that is separate from a structure of therefrigeration system 100. For example, thecamera module 180 can be affixed to a top surface of the refrigeration system, with thelens surface 185 of the camera array positioned to be in front of/or over thedoor 120. As with other examples,camera module 180 can include alternative sets ofcameras cabinet 110. Thus, thelens surface 185 positions thecameras FIG. 1E ) so that he field of each camera includes a vertical thickness that is in front of the door. Further, based on orientation of the cameras and lens configuration, the field of views of the respective cameras can also overlap a vertical thickness that is occupied by thedoor 120. Thus, when thedoor 120 is opened and racks are moved in front of the cameras (underneath the lens surface 185), the cameras of the camera can image the respective rack. -
FIG. 1G illustrates another variation in which therefrigeration system 100 is implemented to include thecamera module 180, where thecamera array 160 is housed. In an example ofFIG. 1G , thedoor 120 of therefrigeration system 100 spans a height of therefrigeration system 100. In such examples, the housing 192 of thecamera module 180 can be structured to position the camera array such that thelens surface 185 is substantially aligned or in front of a vertical thickness occupied by thedoor 120. - Still further, in other variations, the
camera module 180 can be partially positioned within thecabinet 110, with an extension to house the camera array extending to the outside of the cabinet, such as in an orientation shown. - While numerous examples are described with
FIG. 1A throughFIG. 1G which provide for the cameras of the camera array to capture images when a corresponding rack is extended from thecabinet 110, in variations, an item may be entered onto therack 118 without moving the rack our of thecabinet 110. For example, the user can reach a bottle into a rack while the rack remains seated. In such examples, the cameras can detect the item and capture the item as it is placed on therack 118. Still further, in some variations, thecamera module 180 can be manufactured as an accessory item for suitably configured refrigeration systems. -
FIG. 2A is a hardware diagram of an example refrigeration system, according to one or more embodiments. Arefrigeration system 200 ofFIG. 2A can be configured in accordance with examples such as described withFIG. 1A throughFIG. 1E . Accordingly, therefrigeration system 200 provides an example ofrefrigeration system 100, as described with other examples in this document, and reference made to elements ofFIG. 1A throughFIG. 1E illustrate suitable or like elements of therefrigeration system 100, in context ofrefrigeration system 200. - With further reference to an example of
FIG. 2A , therefrigeration system 200 includes acooling subassembly 210, one ormore processors 220, acamera interface 227 and/or acamera array 230. Additionally, therefrigeration system 200 can include one ormultiple communication ports communication ports 240 can enable local device-to-device communications, using, for example, a Bluetooth protocol, and another of thecommunication ports 242 can enable network/Internet communications, such as through WiFi (e.g., wireless communications as promulgated under 802.11(g), 802.11(n), 802.11(ac), etc.), cellular communications or other wireless communication protocols. - The cooling
subassembly 210 includes a compressor unit, coolant handling mechanisms and other components for cooling an interior space 114 (seeFIG. 1A throughFIG. 1E ) of the cabinet 110 (seeFIG. 1A throughFIG. 1E ). In examples, the coolingsubassembly 210 can include a compressor, evaporator, coolant lines, and temperature/power control to enable cooling of theinterior space 114 of therefrigeration system 100. - In examples, the one or more processors (“processor(s) 220”) control the operations of the
refrigeration system 200, including control of the coolingsubassembly 210, while providing image control and/or processing functionality. Theprocessor 220 receives, via thecamera interface 227, image data captured through individual cameras of thecamera array 230. In some examples, thecamera array 230 is an integrated component set of therefrigeration system 200, meaning thecamera array 230 is provided with therefrigeration system 200 at time of manufacturing or assembly. In variations such as described with examples ofFIG. 1F andFIG. 1G , thecamera array 230 may be provided as a separate module or component that is affixed to the housing of therefrigeration system 200 after manufacturing. In such examples, the processor(s) 220 can also be distributed or located with the camera module 180 (seeFIG. 1F andFIG. 1G ). - In some examples, the
refrigeration system 200 includes or is connected tocamera interface 227 to enable subsequent use of cameras, as described with various examples. For example, therefrigeration system 200 can be manufactured to include ports for connecting thecamera array 230, or individual cameras. Subsequently, thecamera array 230 can be added as an option during distribution, at point-of-sale, or by user (e.g., post-sale). When manufactured separately, thecamera array 230 can be modularized, so as to include each camera of thecamera array 230. In variations, the individual cameras ofcamera array 230 can be connected or attached separately, and operated independently by theprocessor 220. - In examples,
refrigeration system 200 can also include sensors or various types, such a (i) a door sensor 211 (to detect when the door is open, closed, in between, etc.); (ii) one ormore rack sensors 212, positioned to detect a position and/or movement by each of one or more corresponding racks; and (iii) a temperature sensor 213 to detect the temperature of the interior space, within theinterior space 114 of thecabinet 110. In some examples, the processor(s) 220 receives temperature input from the temperature sensor 213 and implementstemperature control operations 224 to cause thecooling subassembly 210 to maintain, increase, or decrease the temperature of theinterior space 114. In examples, therefrigeration system 200 includes a temperature control interface (not shown) to enable a user to adjust a desired temperature of theinterior space 114. In response to a temperature input, the processor(s) 220 can control the coolingsubassembly 210 to increase/decrease cooling of the interior space, and the temperature sensor 213 can generate output to enable the processor(s) 220 to implement thetemperature control operations 224 to reach the target temperature. - Further, as described with various examples, the processor(s) 220 implements image processing logic 222 to perform various types of image processing operations on racks when the racks are extended from the cabinet 110 (e.g., in the extended position). For example, as described with other examples, the image processing operations can include operations performed in connection with functionality that includes (i) presence detection (e.g., detecting whether items of a particular type are present on one of the racks 118); (ii) object detection, to determine a type of detected item (e.g., whether item placed on rack is a can, a bottle, or other type of beverage); (iii) hue, color contrast and/or opacity analysis (e.g., to determine a fill level or type of fluid within a container); (iv) label detection, to detect a label on a bottle; and/or (v) character and/or image recognition, to scan, for example, lettering and images of a label.
- In some examples, processor(s) 220 performs the operations for implementing the aforementioned functionality. In variations, the processor(s) 220 initiate the image processing operations by, for example, performing preliminary image processing steps. As a result of initiating the image processing operations, a normalized image of a label of an item, or alternatively, of the item itself, is transmitted to another computing device, remote system or network service. For example, the processor(s) 220 can implement operations to de-warp images captured by wide-eye (or ultra wide-eye) lens cameras. Dewarping operations can also be performed to normalize roundedness in images, caused by the shape of the bottles. If a rack requires more than one camera to image the items of the rack, the processor(s) 220 can implement operations to normalize and stitch the images of the different cameras. The images may be stitched by, for example, (i) dewarping images captured by multiple cameras, (ii) identifying an overlap region between two or more cameras that image the rack, (iii) removing the overlap region, and (iv) blending the remaining portion of the images. As a result of a process, an image of the label of the item can be extracted and transmitted for further recognition and analysis.
- As an addition or variation, the processor(s) 220 performs select image processing operations, to enable or optimize performance of image processing functionality, as described with some examples. The processor(s) 220 may also initiate image processing and analysis by transmitting processed image data via one of the
communication ports - As another variation, processor(s) 220 can implement image processing logic 222 to receive image data from the
camera array 230 and initiate image processing and analysis by retransmitting the image data to another computer system for image processing. Thus, in variations, some or all of the operations for implementing image processing functionality can be performed by another computer system or systems, such as a network service, remote server and/or user mobile device. - In some examples, processor(s) 220 receives and processes images from
camera array 230. Further, processor(s) 220 performs different image processing operations for different sets of cameras of thecamera array 230. For example, for a first set of cameras that use ultra-wide angle lens cameras, processor(s) 220 implements operations to normalize the images. Still further, processor(s) 220 can normalize the images based on the distance of the rack where the images are captured and the respective lens of the corresponding camera. Still further, the processor(s) 220 can detect the rack in use (e.g., usingrack sensor 212, or through image processing), and perform a set of image processing operations based on the detected rack that is in use. Still further, the selected set of operations can also be based on the position of the items that are captured. - Additionally, in some examples, the processor(s) 220 implements image processing logic 222 to stitch images from multiple cameras together, to form a single image of items positioned on a given
rack 118. For example, the processor(s) 220 can select images captured by particular combinations of thecamera array 230 based on therack 118 that is in use. For a top rack or top set ofracks 118, thecamera array 230 can implement image stitching operations, to stitch images of the rack captured by different cameras of thecamera array 230 into a single image. The processor(s) 220 can selectively implement image stitching operations based on the particular rack in use and/or the proximity of thecamera array 230 to theracks 118. In examples as described, thecamera array 230 can utilize ultra-wide lens cameras that are focused for atop rack 118, or top set of racks. The processor(s) 220 implements image processing logic 222 to stitch images from, for example, a laterally positioned pair of ultra-wide lens cameras. - In some examples, the processor(s) 220 responds to the use of a rack (e.g., rack is pulled out, pushed in or left out in an extended position) by capturing multiple images of the
rack 118 using individual cameras of thecamera array 230. The processor(s) 220 can implement thecamera array 230 to select which image processing operations to perform, based on the rack that is detected in use. For example, in response to detecting thetop rack 118 in use, the processor(s) 220 can select to use images from cameras that are optimized for the proximity of the top rack (e.g., ultra-wide eye lens cameras). In such implementations, the processor(s) 220 can implement image stitching to capture the entire width (X-direction) ofrack 118. - In variations, the
refrigeration system 200 includes adisplay 228 that displays an output that corresponds to or is based onrack information 225. In examples, thedisplay 228 can be touch-sensitive and/or provided with an interactive interface. In some variations, therack information 225 can be displayed as content to provide the user with information about the contents of the refrigeration system. - In examples, the
rack information 225 includes, for example, information about the contents of theinterior space 114. In examples, therack information 225 identifies presence of items (e.g., bottles, beverage containers, etc.), one or more categories of items (e.g., type of beverage or container (e.g., bottle versus can)), sub-category of item (e.g., bottle of red wine versus white wine), state of the items (e.g., partially consumed, unopened, opened, etc.), orientation of item, and/or label information about individual items. Still further, therack information 225 can identify information about an orientation of the item, such as a relative orientation of a label of a bottle. In such case, the orientation can reflect a determination of whether the label is, for example, (i) upward facing (e.g., so as to be visible to the camera array when the rack is pulled out), (ii) turned sideways, so that a portion of the label is skewed or not visible to the camera array, and/or (iii) turned so that a back label or surface of the item is visible, without much of the front panel. Further, with respect to detected items, therack information 225 can identify where individual items are located on a given rack (e.g., visually, by coordinates, by slot, etc.). Additional examples ofrack information 225, as well as other content which can be rendered with thedisplay 228, are provided with examples ofFIG. 2B . -
FIG. 2B illustrates a network communication system for providing information about items stored in a refrigeration system, according to one or more embodiments. In examples, anetwork communication system 300 includesrefrigeration system 200 and one ormore user devices user devices refrigeration system 200 using either a local wireless connection 242 (e.g., Bluetooth) or network connection (e.g., WiFi Direct). In variations, therefrigeration system 200 can communicate withuser devices - Accordingly, in some examples,
network communication system 300 includes one ormore servers 280 or other remote computer systems. In variations, the server(s) 280 implement or otherwise provide anetwork service 282 that utilizes rack information transmitted by therefrigeration system 200, vianetworks 292. - With further reference to
FIG. 2B ,refrigeration system 200 transmitsrack information 225 over one or more networks to servers 280 (or other remote computer systems) ornetwork service 282. Therack information 225 can include information obtained from performing image processing of images captured bycamera array 230, where the captured images depict items placed onindividual racks 118. Therack information 225 can be determined locally, on therefrigeration system 200, by the processor(s) 220. Alternatively, some or all of therack information 225 can be determined by a remote computer, server or service. Accordingly, as described with an example ofFIG. 2A , therack information 225 includes (i) information that identifies a presence of items (e.g., bottles, beverage containers, etc.) on individual racks, (ii) information about one or more categories (e.g., type of beverage or container) or sub-categories of the individual items (e.g., type of wine (red, wine, rose, etc.), (iii) information about a state of individual items (e.g., open or uncorked, closed, previously opened, partially consumed, fill state, etc.); and/or (iv) label information for individual items (e.g., brand, vintage, trade name, information about source of origin, etc.). An end user can access content that is based onrack information 225 using, for example, a mobile device that communicates or otherwise utilizesnetwork service 282. For example, the mobile device can execute an application (“app”) that is dedicated for thenetwork service 282, in order to view content that is at least partially based on therack information 225. Alternatively, a user can operate a computer on which a browsing component is installed in order to view such content. - In variations,
refrigeration system 200 transmitsrack information 225 touser device 272 through, for example, a local wireless connection. Theuser device 272 can execute an application (e.g., dedicated application, browsing component, etc.) to view content that is based at least in part on therack information 225. - In examples, the
user devices rack content 235 that incorporates or is otherwise based on therack information 225. Accordingly, theuser devices display 228 ofrefrigeration system 200. In variations, the rack content 335 includes an item/rack map that shows each detected item and a corresponding rack where the item is located. Additionally, the item/rack map can identify a position along the span of the rack (along X, Z directions) where the item is detected as being located. Additionally, therack content 235 can include detailed information about each item, including information determined from performing image processing of the respective item, as well as supplemental information. The supplemental information can include information that is not determined from the label or item itself, but rather determined from information sources based on information obtained from the label or with the item. For example, in the case of wine, the maker and vintage can be cross reference with the information repository about the particular winemaker, and the vintage year, to obtain supplemental information that includes, for example, descriptive information about how the wine was made, and particular aspects or attributes of the wine, such as awards which the wine may have one, and taste attributes of the wine which are determined by experts and not necessarily included on the label of the wine. As another example, therack content 235 can recommend types of food that may go well with the particular wine, by referencing identifiers (e.g., vintage, maker, trade label, source of origin, etc.), categories (e.g., “burgundy”, “pinot”, etc.) or other aspects that are determined about the wine through image processing (e.g., by image recognition of label information) against information repositories were food recommendations are maintained. - In examples, the
rack content 235 is provided as an interactive interface. For example, a user can interact with graphic representations of items identified as being present in the individual racks, in order to view supplemental information about the individual items. As an addition or variation, a user can view a listing of items which are present in therefrigeration system 200, and through interaction with a given entry of the list, receive output that indicates a rack (or rack position) where the item is being stored. - When information about an item is provided (e.g., through selection of a graphic representation of the item), examples provide for additional information about the item to be provided. The additional information can include, for example, information about the state of the item (e.g., open or uncorked, closed, previously opened, partially consumed, fill state), recommendation about how the item can be consumed (e.g., with type of food, etc.), and supplemental information.
- Still further, in examples, the
rack content 235 can include a list of recommendations. The list of recommendations can identify, for example, other items that may be of interest to the user based on their affinity for a particular item. For example, historical information can be maintained about items over the course of a duration, and information about preferences of the user (e.g., winemaker, wine type, vintage, aspects of the particular item, etc.) can be maintained and analyzed in order to determine preferences of the user. Based on user preferences, and or items which the user may have stored in therefrigeration system 200, therack content 235 can provide recommendations regarding additional items which the user can purchase. - As another example, the
rack content 235 can include an order interface that enables users to purchase items from a third-party source. A user can utilize the order interface to purchase a refill of a consumed item, such as a replacement wine bottle. In examples, therefrigeration system 200 can monitor items for consumption level, based on image processing that the text when items are taken offer rack in place back on it, as well as indications that an item was consumed or partially consumed. In response to detecting that an item was consumed, the rack content 335 can provide the order interface to enable the user to reorder the consumed item. As an addition or variation, the order interface can enable the user to purchase recommended items, or items which the user searches for or is otherwise provided information about. -
FIG. 3A throughFIG. 3C illustrate example methods for operating a refrigeration system, according to one or more embodiments. In describing example methods, reference may be made to elements ofFIG. 1A throughFIG. 1G ,FIG. 2A and/orFIG. 2B for purpose of illustrating suitable components for performing a step or sub-step being described. - With reference to
FIG. 3A , instep 310, a first scan or imaging processing is performed on afirst rack 118 of arefrigeration system rack 118 is positioned exterior tocabinet 110 of the refrigeration system. Instep 312, a second scan or imaging processing is performed on asecond rack 118 of therefrigeration system cabinet 110, where the second rack is different than the first rack. For example, the first rack may correspond to atopmost rack 118 of therefrigeration system 100, while the second rack may correspond to a middle orbottom rack 118. Each of the first and second image processing includes operations where image data, captured by corresponding cameras, is processed to enable recognition and other analysis. Thus, for example, a result of each image processing can be to generate a normalized image of a label of an item, or of the item itself. Further, the normalized image can include optimizations to enhance recognition and subsequent analysis. - In some variations, the first and second imaging processes is performed on images captured as the respective first and
second racks cabinet 110. - Further, in examples, operations of the first imaging process are different than operations of the second imaging process. The first imaging process may be performed using a first set of cameras that are of a first type, while the second imaging process is performed using a second set of cameras that are of a second type. The first and second set of cameras may be different to accommodate the different distances between the individual racks and the respective cameras. Further, the types of image processing that are performed may also be different. To perform image analysis on racks which may have relatively short separation distance from the camera lens, the image processing that is used (or prioritized) can be specifically configured for wide-lens cameras, including ultra-wide lens (or fisheye lens) cameras. Further, the image processing can use multiple cameras, where captured images are stitched into a single view.
- To perform image processing on racks which have larger separations, the image processing can accommodate higher resolution and alternative lighting conditions. Thus, the number and type of cameras which are used to image
different racks 118 may differ based on the proximity of the respective rack to the corresponding cameras. Additionally, the image processing can also differ based on the proximity of the cameras to the rack and/or the type of cameras in use. - With reference to
FIG. 3B , instep 320, the rack of arefrigeration system interior space 114 ofcabinet 110. Instep 322, as therack 118 is moved into theinterior space 114, thecamera array 230 captures images of the contents of therack 118. In some implementations, the images correspond to a series of frames, such as may be captured by video. - In
step 324, image analysis is initiated by the processor(s) 220 of therefrigeration system camera array 230 to utilize for processing. In variations, the processor(s) 220 can normalize images, such as performing image stitching as needed, before transmitting the images to a remote computer or service for further processing. Still further, in other variations, the processor(s) 220 can implement operations to perform object detection and image/character recognition. - In some examples, the processor(s) 220 determines or otherwise obtains the results of the image analysis, and store the results in memory and/or generate content based on the results for the display. As an addition or variation, results are generated or otherwise provided to user devices.
- In
step 325, the results of the image analysis includes determination of the type of item. For example, the item can be determined by type of container (e.g., bottle, can, box, etc.), and/or by type of beverage (e.g., soda versus hard alcohol). - In
step 326, the results of the analysis identifies a position of the item. The position can associate an item with aparticular rack 118, as well as with a location of the item on the rack. For example, the coordinate of the item along a lateral axis (X or Z direction) can be determined. Furthermore, if the item is removed and repositioned on a rack, the new position of the item can be determined and recorded. - In step 327, information about the item is determined using recognition analysis. In some examples, the processor(s) 220 capture images and perform label detection and normalizing. Subsequent OCR and image analysis can be used to determine additional information about the beverage. In examples, label detection can be used to generate a normalized image of a label of the item, and subsequent analysis is performed to determine information about the item. The additional information can identify a type of beverage (e.g., wine, champagne, soda, etc.), one or more sub-categories of the item (e.g., color of wine), and manufacturer or brand. For wine, liquors and other specialty beverages, the information can identify a maker, a year, source (e.g., country or geographic region), and descriptive terms (e.g., alcohol content) which may be provided on the label.
- With reference to an example of
FIG. 3C , instep 330, the processor(s) 220 initiate and perform image processing using images captured by a camera set that is dedicated or prioritized for a rack that is in use. The rack in use can correspond to one that is in movement (e.g., moved from extended to seated position). In performing the image processing,step 332 provides that images are selected from a camera set that is optimized or otherwise configured for the particular rack. The configuration may include type of camera/lens, focus attributes, and subsequent image processing. For example, thetop-most rack 118 can be imaged using (i) multiple cameras, (ii) ultra-wide lens cameras, and/or (iii) image stitching, such that images from multiple cameras can be formed into a single image for processing. While in one implementation, the images are selected and subsequently processed, in variations, images from multiple camera sets can be processed and prioritized by weighting. Thus, the processor(s) 220 and subsequent processing/analysis of images can utilize multiple camera sets, with a non-prioritized camera set providing information for specific purposes (e.g., hue analysis), edge analysis and/or determination of confidence values and assessment. - Accordingly, in
step 340, image analysis is performed to determine information about the items placed on therack 118. In some examples, the processor(s) 220 of therefrigeration system refrigeration system - In
step 342, information is determined about the type of the item. For example, the presence of a new item on a rack is determined. Further, the type of item can be detected by type, based on or corresponding to container type (e.g., can, box, bottle, etc.) or item shape. Instep 343, the position of the item on therack 118 may be determined, where the position is based on, for example, coordinates for the rack and/or a particular slot position. - In
step 344, additional information about individual items are determined from recognition operations. The recognition operations can include character image recognition, logo recognition, and/or pattern analysis of the imaged labels for each item. Instep 345, label detection and analysis can be performed. The analysis can implement, for example, label detection, logo detection, image analysis (of imagery on label), optical character recognition (OCR) and/or other operations for determining information such as vintage, maker, source, type of wine and other categories. - As an addition or variation, in step 365, opacity and/or hue analysis is performed. The opacity information can be used to determine the state of the item. For example, variations in the opacity/hue of the item can reflect the fill level of the item. Still further in
step 347, cork or lid recognition can be performed to determine, for example, whether the item has been opened. For example, for corks, the height of the cork above the top of a bottle can reflect whether the bottle was uncorked. Likewise, if a lid is missing a bottom portion, the lid may be deemed as having been previously opened. - In some variations, the image analysis that is performed results in an identifier of the item, and processed images of the item are stored. For example, the identifier of the item can be based in part on the label analysis. When the item is initially detected, an image of the item may be stored in association with the identifier, along with determinations about the characteristics of the hue, opacity and/or lid. In subsequent instances when the
rack 118 is used, the characteristics of the hue, opacity and/or lid for the same item can be compared with the stored images to determine changes which are indicative of the item being opened (e.g., uncorked) and/or partially consumed. - According to examples, in
step 350, the determinations of the image analysis is transmitted and/or rendered. In some examples, the determinations can be by therefrigeration system refrigeration system wireless communication ports 240, 242 (step 354). Still further, a remote computer system or service can make the determinations in whole or in part, and transmit the determinations to another device using a network connection. -
FIG. 4 illustrates a computer system on which one or more embodiments can be implemented. Acomputer system 400 can be implemented on, for example, a server or combination of servers. For example, thecomputer system 400 may be implemented as part of anetwork communication system 300, such as described with an example ofFIG. 2B . Likewise, thecomputer system 400 can implement operations such as described withFIG. 2A , and with steps or sub-steps of example methods such as described withFIG. 3A throughFIG. 3C . - In one implementation, the
computer system 400 includes processingresources 410, memory resources 420 (e.g., read-only memory (ROM) or random-access memory (RAM)), astorage device 440, and acommunication interface 450. Thecomputer system 400 includes at least oneprocessor 410 for processing information stored in thememory resources 420, such as provided by a random-access memory (RAM) or other dynamic storage device, for storing information and instructions which are executable by theprocessor 410. Thememory resources 420 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by theprocessor 410. Thecomputer system 400 may also include thememory resources 420 or other static storage device for storing static information and instructions for theprocessor 410. Thestorage device 440, such as a magnetic disk or optical disk, is provided for storing information and instructions. - The
communication interface 450 enables thecomputer system 400 to communicate with one or more networks (e.g., cellular network) through use of the network link 480 (wireless or a wire). Using thenetwork link 480, thecomputer system 400 can communicate a refrigeration system, user device or other computing system for implementing one or more embodiments as described. Theexecutable instructions 442 stored in thememory resources 420 can includeinstructions 442, to implement a remote computer system, network service and/or network communication system (e.g., seeFIG. 2B ), as described with one or more embodiments. The executable instructions stored in thememory resources 420 may also implement operations for implementing steps such as described with examples ofFIG. 3A throughFIG. 3C . - As such, examples described herein are related to the use of the
computer system 400 for implementing the techniques described herein. According to an aspect, techniques are performed by thecomputer system 400 in response to theprocessor 410 executing one or more sequences of one or more instructions contained in thememory resources 420. Such instructions may be read into thememory resources 420 from another machine-readable medium, such as thestorage device 440. Execution of the sequences of instructions contained in thememory resources 420 causes theprocessor 410 to perform the process steps described herein. In alternative implementations, hard-wired circuitry may be used in place of or in combination with software instructions to implement examples described herein. Thus, the examples described are not limited to any specific combination of hardware circuitry and software. -
FIG. 5 is a block diagram illustrating a user device for use with examples as described. In an example, a user device 500 may execute a designated service application that utilizesrack information 525 to provide rack content 535 and other information. In some examples, the user device 500 operates to communicate with a network service, through one or more networks. In many implementations, user device 500 includes a mobile computing device, such as a smartphone, tablet computer, laptop computer, VR or AR headset device, and the like. The user device 500 includes wireless communication interfaces 510, 512 to communicate with external entities using any number of wireless communication protocols. The user device 500 can receive rack information 535 from the network system 502 (or other remote computer system or device) through, for example, a wireless network interface (e.g., cellular radio, WiFi, etc.). As an addition or variation, the user device 500 can receive rack information 535 from therefrigeration system - In certain aspects, the user device 500 stores a designated application (e.g., a service app 532) in a
local memory 530. In variations, thelocal memory 530 can store additional applications executable by one ormore processors 520 of the user device 500, enabling access and interaction with one or more host servers over one ormore networks 580. - As described with various examples, the
service application 532 executes on the user device 500 to generate rack information 535 on a display screen 540 of the user device 500. The rack information 535 can be displayed as part of a user interface 545, where the user interface 545 includes interactive features (e.g., menus, selectable features, search functions, etc.), and enable the consumer view their inventory of beverages, purchase additional beverages, view recommendations relating to items in their inventory and other information. Still further, in some examples, the user interface 545 can present a rack map, where items are shown by rack and, in some variations, by position on respective racks. -
FIG. 6A throughFIG. 6C illustrate an example of a beverage cooler, according to one or more examples. In examples shown, abeverage cooler 600 can include configurations that enhance or optimize the user of thebeverage cooler 600. Still further, in some variations, the beverage cooler can be structured with a camera array, processor and resources to determine rack information, as described with various examples. - With further reference to examples of
FIG. 6A throughFIG. 6C , thebeverage cooler 600 includes ahousing 610, having a door 620 anddoor handle 612. The door inset can be mounted flush with a contouredtop segment 615 of thehousing 610. The door 620 can be contoured and shaped to match thecabinet 610, in such manner that the door 620 can be fully recessed and flush with respect to a facade of the top housing segment 629. - In examples, a
housing 610 of thebeverage cooler 600 includes anintegrated tray 612, positioned on atop surface 611. Examples recognize that thebeverage cooler 600 can be waist high. Theintegrated tray 612 can be integrated with the top surface to enable thehousing 610 to serve as a flat top surface to receive items, without affecting the design or function of thebeverage cooler 600. Further, thetray 612 can include a lip or other protrusion to contain items that are received on it. Additionally, in examples, thetop surface 611 of thehousing 610 can be shaped or structured to retain structures (e.g., outside of the tray 612). Thetop surface 611 can, for example, be contoured inwards or provided a lip or upward protrusion, separate from thetray 612. In some examples, theintegrated tray 612 is unitarily formed (e.g., through a common manufacturing process, such as injection molding or metal pressing, with the remainder of the housing). In other examples, thetray 612 is integrated, meaning the tray can be connected (e.g., using a fastener). In such examples, theintegrated tray 612 can be attached and detached from thehousing 610. - In examples, the
beverage cooler 600 includes acompressor housing segment 630 positioned to extend from a bottom surface 613 of thehousing 610. Thecompressor housing segment 630 can include aninclined front facade 632 that tapers the 630 inward. Thehousing 610 can further be lifted from the ground bylegs 636. Optionally, thelegs 636 can telescope to enable the vertical height of thehousing 610 to be adjustable. With thehousing 610 lifted by thelegs 636, the inclinedfront facade 632 creates a void in a front region underneath thehousing 610. The void can allow for presence of footing, and enable sweeping and cleaning, without adding to the overall height of the housing. Further, thefront facade 632 can create a void under thecompressor housing segment 630, as well as to the sides of thecompressor housing segment 630. The provision for such voids allows for improved airflow from underneath the housing and to its sides. - Further, the
compressor housing segment 630 can house a fan that blows warmed air outward, away from the compressor assembly. By structuring thefront facade 632 to incline, the air can blow downward into the ground, rather towards the feet of a person or in other directions where the warmed air may be trapped. As the air can be blown forward (at an angle towards the ground), thebeverage cooler 600 can be pushed on itsrear face 609 towards a wall, without need to accommodate an airgap for the blown air. Moreover, thecabinet 610 can be fitted in tight spaces and flush-mounted under a counter or inside cabinetry, as a result of the blown air being directed forward and downward directly underneath thecabinet 610. - With reference to
FIG. 6C , arear surface 609 of thehousing 610 can be mounted substantially flush to a wall or cabinet, with aback surface 609 being flat. Thecompressor housing segment 630 can also be flat on the back facade to eliminate or reduce separation between a backwall and the unit. -
FIG. 6D illustrates a variation in which thebeverage cooler 600 includes aribbed door 628 to promote light diffusion. In some examples, theribbed door 628 or facade can be formed from ribbed or wavy glass or other translucent material. For example, thedoor 628 can be partially constructed from ridged glass (or translucent material), such as bottle glass. As an addition or variation, the door can be formed from partially opaque glass or translucent material. The use of light-diffusive material can protect certain types of items, such as vintage wines, from deterioration by light. -
FIG. 7A andFIG. 7B illustrate a reversible rack for a refrigeration system, according to one or more examples. Arack 700 can be integrated with refrigeration systems as described with various examples, includingFIG. 1A throughFIG. 1G andFIG. 6A throughFIG. 6D . Therack 700 includes areversible bottom tray 720 that is retained by arack structure 710. Thetray 720 can include first andsecond layers first layer 712 can be greater than the spacing between the support structures of thesecond layer 714. The variation in spacing can be implemented by, for example, varying the thickness of the support structures of thefirst layer 712 as compared to thesecond layer 714. - As described,
tray 720 can be reversible when positioned at the bottom of the area formed by therack structure 710. InFIG. 7A , thefirst layer 712 is positioned on top (relative to the second layer 714). InFIG. 7B , thesecond layer 714 is positioned on top. Thetray 720 can be retained within therack structure 710 by a combination of retention structures 716 (e.g., clips, shelf-support pins and bolts, etc.). For example, a user can manipulate theretention structures 716 on inside surfaces of therack structure 710 to enable thetray 720 to be separated, flipped and inserted back again into therack structure 710. - In an example of
FIG. 7A , thefirst layer 712 of thetray structure 720 is top-facing. Thefirst layer 712 can include spacing between support structures that are dimensioned to retain wine bottles. Thesecond layer 714 can form a base structure to hold smaller bottles or items. In an example shown, the arrangement of thefirst layer 712 can retain wine bottles in a width-wise direction, meaning the wines can be provided on thetray 720 sideways relative to the direction of movement of the rack 700 (which can move in and out of the cabinet in the Z direction). - With reference to an example of
FIG. 7B , the user can flip thetray 720 such that thesecond layer 714 is top-side. The second layer can support items of various dimensions, including cans or small bottles that would otherwise pass through conventional trays designed to hold wine bottles. Further, the second layer can also hold wine bottles or other similarly shaped items. However, the arrangement of thesecond layer 714 also enables wine bottles to be retained lengthwise relative to the movement of the rack (i.e., along Z direction). - In examples, the user can choose which side of the
tray 720 to orient topside based on preference with regards to orientation of wine bottle, and/or the size or dimension of items which thetray 720 may retain. In some examples, therack 700 can be provided as an accessory for a refrigeration system such as shown by various examples. Still further, a refrigeration system such as shown by various examples can include one ormultiple racks 700. - Although examples are described in detail herein with reference to the accompanying drawings, it is to be understood that the concepts are not limited to those precise examples. Accordingly, it is intended that the scope of the concepts be defined by the following claims and their equivalents. Furthermore, it is contemplated that a particular feature described either individually or as part of an example can be combined with other individually described features, or parts of other examples, even if the other features and examples make no mentioned of the particular feature. Thus, the absence of describing combinations should not preclude having rights to such combinations.
Claims (20)
1. A method for operating a refrigeration system, the method comprising:
detecting a given rack of multiple racks of the refrigeration system being moved from an extended position to a seated position within an interior of the refrigeration system; and
capturing one or more images of the rack, and initiating performance of image analysis of the one or more images to determine information about one or more items that are placed on the rack.
2. The method of claim 16 , wherein initiating performance of the image analysis includes determining, for each of the one or more items, information that identifies a type of the item.
3. The method of claim 2 , wherein determining the type of the item includes determining the item is a bottle or a can.
4. The method of claim 1 , wherein initiating performance of the image analysis includes determining a position of each of one or more items with respect to the rack.
5. The method of claim 1 , wherein initiating performance of the image analysis includes detecting a label of each of the one or more items, and performing image recognition on the label to determine information about a content of the bottle.
6. The method of claim 5 , wherein the determined information about the content of item includes at least one of a vintage, manufacturer, or type of the content.
7. The method of claim 6 , wherein the item is a wine bottle.
8. The method of claim 1 , further comprising performing the image analysis on the refrigeration system.
9. The method of claim 1 , further comprising:
selecting one of multiple sets of cameras of a camera array to capture the one or more images based on a vertical distance or position of the moving rack with respect to the camera array.
10. The method of claim 9 , wherein selecting one of multiple sets of cameras includes selecting a first set of cameras that include one or more wide angle lenses in response to detecting that the vertical distance is less than a first threshold.
11. The method of claim 9 , wherein the first set of cameras include at least a first wide angle lens positioned in proximity to at least a first lateral side of the interior space, and a second wide angle lens positioned in proximity to at least a second lateral side of the interior space.
12. The method of claim 11 , wherein initiating performance of the image analysis includes stitching an image captured using the first wide angle lens with an image captured using the second wide angle lens.
13. The method of claim 1 , further comprising transmitting information about the item to another device.
14. The method of claim 13 , wherein transmitting information about the item includes transmitting the information to a network service, using one or more networks.
15. The method of claim 13 , wherein transmitting information about the item includes transmitting the information to a mobile device of a user, using a local wireless connection.
16. The method of claim 1 , wherein initiating performance of the image analysis includes determining a type of the item, a position of the item with respect to the rack, and information about a content of the item.
17. The method of claim 1 , wherein determining the type of information includes estimating an amount of the content relative or a volume of the item.
18. A refrigeration system comprising:
a cabinet having an interior space;
a cooling sub-assembly to cool the interior space;
a door connected to a front side of the refrigeration cabinet to seal the interior space when in a closed position, and to provide access to the interior space when in an opened position;
a rack sub-assembly mounted to an interior of the cabinet, the rack sub-assembly including a set of racks, each rack being moveable from a seated position that is fully contained within the interior space, to an extended position where at least a segment of the rack extends outside of the interior space to receive one or more items;
a camera array including one or more cameras, each of the one or more cameras being oriented such that a corresponding viewing angle includes a vertical space that is in front of the interior space; and
one or more processors, wherein the one or more processors execute instructions to perform operations that include:
detecting a rack of the rack sub-assembly in the extended position;
capturing one or more images of the rack; and
initiating performance of image analysis of the one or more images to determine information about one or more items that are placed on the rack.
19. The refrigeration system of claim 18 , wherein the one or more processors execute instructions to perform the image analysis.
20. A camera module for a refrigeration system, the camera module comprising:
a camera array including one or more cameras;
a housing;
wherein the housing is structured to secure to a refrigeration system, to position the camera array such that a corresponding viewing angle of one or more cameras of the camera array a vertical space that coincides with a thickness that occupies a door of the refrigeration system when the door is in a closed position; and
one or more processors, wherein the one or more processors execute instructions to perform operations that include:
capturing one or more images of a rack of the refrigeration system when the rack is extended from an interior cabinet; and
initiating performance of image analysis of the one or more images to determine information about one or more items that are placed on the rack.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US18/073,552 US20230169772A1 (en) | 2021-12-01 | 2022-12-01 | Refrigeration system with imaging |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US202163264720P | 2021-12-01 | 2021-12-01 | |
US18/073,552 US20230169772A1 (en) | 2021-12-01 | 2022-12-01 | Refrigeration system with imaging |
Publications (1)
Publication Number | Publication Date |
---|---|
US20230169772A1 true US20230169772A1 (en) | 2023-06-01 |
Family
ID=86500483
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US18/073,545 Pending US20230168028A1 (en) | 2021-12-01 | 2022-12-01 | Refrigeration system with imaging |
US18/073,552 Pending US20230169772A1 (en) | 2021-12-01 | 2022-12-01 | Refrigeration system with imaging |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US18/073,545 Pending US20230168028A1 (en) | 2021-12-01 | 2022-12-01 | Refrigeration system with imaging |
Country Status (5)
Country | Link |
---|---|
US (2) | US20230168028A1 (en) |
CN (1) | CN118284780A (en) |
AU (1) | AU2022402004A1 (en) |
CA (1) | CA3240688A1 (en) |
WO (1) | WO2023102151A1 (en) |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5340209A (en) * | 1992-12-28 | 1994-08-23 | General Electric Company | Slide-out shelf assembly for refrigerators |
JP6498866B2 (en) * | 2013-03-12 | 2019-04-10 | 東芝ライフスタイル株式会社 | Refrigerator, camera device |
KR102026730B1 (en) * | 2013-04-08 | 2019-09-30 | 엘지전자 주식회사 | Refrigerator |
JP6550696B2 (en) * | 2014-07-18 | 2019-07-31 | パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカPanasonic Intellectual Property Corporation of America | Judgment apparatus and judgment method |
US10474987B2 (en) * | 2015-08-05 | 2019-11-12 | Whirlpool Corporation | Object recognition system for an appliance and method for managing household inventory of consumables |
EP3712852B1 (en) * | 2019-03-19 | 2021-12-15 | Carrier Corporation | Refrigerated sales cabinet |
US11335010B2 (en) * | 2019-08-15 | 2022-05-17 | Haier Us Appliance Solutions, Inc. | Methods for viewing and tracking stored items |
-
2022
- 2022-12-01 CA CA3240688A patent/CA3240688A1/en active Pending
- 2022-12-01 US US18/073,545 patent/US20230168028A1/en active Pending
- 2022-12-01 AU AU2022402004A patent/AU2022402004A1/en active Pending
- 2022-12-01 WO PCT/US2022/051590 patent/WO2023102151A1/en active Application Filing
- 2022-12-01 US US18/073,552 patent/US20230169772A1/en active Pending
- 2022-12-01 CN CN202280079779.3A patent/CN118284780A/en active Pending
Also Published As
Publication number | Publication date |
---|---|
US20230168028A1 (en) | 2023-06-01 |
CA3240688A1 (en) | 2023-06-08 |
CN118284780A (en) | 2024-07-02 |
AU2022402004A1 (en) | 2024-06-06 |
WO2023102151A1 (en) | 2023-06-08 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10809884B2 (en) | Paint color selection and display system and method | |
US20190279017A1 (en) | User Interface for Object Detection and Labeling | |
US9736361B2 (en) | Assisted text input for computing devices | |
US10330377B2 (en) | Refrigeration appliance comprising a camera module | |
CN105222518B (en) | Refrigerator and its control method | |
CN105222521B (en) | Refrigerator and its control method | |
US9342930B1 (en) | Information aggregation for recognized locations | |
US11017521B2 (en) | Bottle storage and system | |
US20180367732A1 (en) | Visual cues for managing image capture | |
US20130265232A1 (en) | Transparent display apparatus and method thereof | |
US20160156887A1 (en) | Projection processor for projective display system | |
US9934526B1 (en) | Text recognition for search results | |
KR102586911B1 (en) | Intelligent Marketing and Advertising Platform | |
US11763252B2 (en) | Intelligent marketing and advertising platform | |
US20230169772A1 (en) | Refrigeration system with imaging | |
KR20210036887A (en) | Refrigerator and Controlling Method for the same | |
KR20240116726A (en) | Refrigeration system with imaging capabilities | |
KR20200106474A (en) | Refrigerator and Controlling Method for the same | |
KR20200106473A (en) | Refrigerator and Controlling Method for the same | |
JP2018135152A (en) | Bottle storage apparatus and system | |
WO2023124088A1 (en) | Method for identifying information of articles in refrigerator, and refrigerator | |
US20240005271A1 (en) | Intelligent Marketing and Advertising Platform | |
TW201843606A (en) | Real-time information searching apparatus when playing image and operation method thereof |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
AS | Assignment |
Owner name: SNWC, INC., NEW YORK Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:NAPARSTEK, SAM;REEL/FRAME:067318/0355 Effective date: 20221129 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |