CN111310520A

CN111310520A - Dish identification method, cash registering method, dish order prompting method and related device

Info

Publication number: CN111310520A
Application number: CN201811513717.0A
Authority: CN
Inventors: 汪海洋
Original assignee: Alibaba Group Holding Ltd
Current assignee: Alibaba Group Holding Ltd
Priority date: 2018-12-11
Filing date: 2018-12-11
Publication date: 2020-06-19
Anticipated expiration: 2038-12-11
Also published as: CN111310520B

Abstract

The invention discloses a dish identification method, a cash registering method, a dish order urging method, a related device, a computing device and a medium, wherein the dish identification method comprises the following steps: inputting the dish image to be identified into a first dish type identification model for processing so as to obtain image characteristics output by a bottleneck layer in the first dish type identification model, wherein the bottleneck layer comprises all processing layers before the last processing layer in the first dish type identification model; and inputting the image characteristics into a second dish type identification model for identification so as to obtain the type of the dish in the dish image.

Description

Dish identification method, cash registering method, dish order prompting method and related device

Technical Field

The invention relates to the technical field of image processing, in particular to a dish identification method, a cash register method, a dish order urging method, a related device, a computing device and a medium.

Background

When a restaurant has meals, it is often desirable for a merchant to provide quick ordering, serving and checkout services to avoid unnecessary waste of time. However, in the peak of dining, especially for the dining hall, due to the fact that the current working strength cannot be loaded by human hands due to the fact that the stream of people is very large, the service is difficult to meet the requirements, various problems that ordered dishes are not placed, dishes are mistakenly placed, the time for waiting for the ordered dishes is too long, the settlement time of the bill amount is too long and the like easily occur, and poor dining experience is brought. Among them, the problem of waiting for the time to get on the dish and the time to settle the bill amount is too long is the most common.

In order to solve the problems, the settlement and invoicing efficiency can be improved through a self-service cashier robot, so that the conditions that the settlement speed of manually calculated prices is low, calculation errors are easy to occur, invoicing is forgotten due to busy conditions and the like are avoided. For the self-service cashier desk robot, if the speed and the accuracy of settlement and invoicing are to be ensured, the implanted dish identification algorithm is required to have higher accuracy and responsivity for identifying different dishes, and the dish identification algorithm is usually constructed by adopting a deep neural network.

However, in the existing dish identification method implemented by using a deep neural network, on one hand, a large number of dish samples actually used need to be collected so as to train the deep neural network, and on the other hand, because of too many dish samples, in order to meet the identification accuracy, the structure of the deep neural network is complex and the number of layers is large, in this case, a long training time is caused, even if the trained deep neural network can achieve a good identification effect, in the actual application of dish identification, the problem of slow identification response speed due to the complex network structure can also occur, and good user experience cannot be provided, so that a new dish identification method needs to be provided to optimize the processing process.

Disclosure of Invention

To this end, the present invention provides a dish identification scheme, and a cash register and dish order scheme based on dish identification, in an attempt to solve or at least alleviate the above-presented problems.

According to an aspect of the present invention, there is provided a method for identifying a dish, the method including the steps of: firstly, inputting a dish image to be identified into a first dish type identification model for processing so as to obtain image characteristics output by a bottleneck layer in the first dish type identification model, wherein the bottleneck layer comprises all processing layers before the last processing layer in the first dish type identification model; and inputting the image characteristics into a second dish type identification model for identification so as to obtain the type of the dish in the dish image.

Alternatively, in the dish recognition method according to the present invention, the first dish category recognition model is determined by performing a fine adjustment process based on a category recognition model trained in advance.

Optionally, in the dish identification method according to the present invention, the performing fine-tuning processing based on the pre-trained category identification model includes: modifying the output number of classifiers in a pre-trained class recognition model according to the number of the dish classes corresponding to the first dish image data set; correspondingly loading the parameters of all processing layers before the last processing layer in the pre-trained category identification model into the modified category identification model to generate a first dish category identification model; the first dish category identification model is trained based on the first set of dish image data such that an output of the first dish category identification model is indicative of a category of a dish in the input dish image.

Alternatively, in the dish identification method according to the present invention, the category identification model is model-trained based on a pre-acquired image data set, so that an output of the category identification model indicates a category of image content in the input image.

Optionally, in the dish identification method according to the present invention, the image data set includes a plurality of pieces of image data, each piece of image data includes a training image and a category of image content in the training image, and performing model training based on the pre-acquired image data set includes: for each piece of image data in the image data set, inputting a training image in the image data as input into a category recognition model to obtain a first category recognition result of the training image output by the category recognition model; based on the difference between the class of the image content in the training image and the first class recognition result, the parameters of the class recognition model are adjusted.

Optionally, in the dish identification method according to the present invention, the image data set is an ImageNet data set.

Optionally, in the dish identification method according to the present invention, the first dish image data set includes a plurality of pieces of dish image data, each piece of dish image data includes a dish training image and a category of a dish in the dish training image, and training the first dish category identification model based on the first dish image data set includes: for each item image data in the first item image data set, taking an item training image in the item image data as input, and inputting the input into a first item type identification model to obtain a first item type identification result of the item training image, which is output by the first item type identification model; and adjusting parameters of the first dish type identification model based on the difference between the dish type in the dish training image and the first dish type identification result.

Optionally, in the dish identification method according to the present invention, adjusting the parameter of the first dish category identification model includes: parameters of a plurality of processing layers in the first dish category identification model near the output end are adjusted.

Optionally, in the dish identification method according to the present invention, the second dish type identification model is model-trained based on a second dish image data set acquired in advance and the first dish type identification model, so that an output of the second dish type identification model indicates a type of a dish in the input dish image.

Optionally, in the dish identification method according to the present invention, the second dish image data set includes a plurality of pieces of specific dish image data, each piece of specific dish image data includes a specific dish training image and a category of a dish in the specific dish training image, and performing model training based on the second dish image data set acquired in advance and the first dish category identification model includes: inputting specific dish training images in the specific dish image data into the first dish type identification model for processing to obtain training image characteristics output by a bottleneck layer in the first dish type identification model for each specific dish image data in the second dish image data set; inputting the training image characteristics into a second dish type recognition model by taking the training image characteristics as input so as to obtain a second type recognition result of dishes in a specific dish training image corresponding to the training image characteristics and output by the second dish type recognition model; and adjusting parameters of the second dish type identification model based on the type of the dishes in the specific dish training image and the second type identification result.

Alternatively, in the dish identification method according to the present invention, the first dish category identification model includes a deep neural network including a plurality of processing layers.

Alternatively, in the dish identification method according to the present invention, the deep neural network is a convolutional neural network, and the processing layer is any one of a convolutional layer, a pooling layer, and a fully-connected layer.

Alternatively, in the dish identification method according to the present invention, the second dish category identification model includes a support vector machine model.

According to yet another aspect of the present invention, there is provided a method of cashing, the method comprising the steps of: firstly, acquiring one or more dish images corresponding to a current order, wherein the dish images contain corresponding dishes; inputting each dish image into a first dish type identification model respectively for processing so as to obtain image characteristics output by a bottleneck layer in the first dish type identification model, wherein the bottleneck layer comprises all processing layers before the last processing layer in the first dish type identification model; inputting the image characteristics into a second dish type identification model for identification so as to obtain the type of the dishes in the dish image; acquiring the price of the dish according to the category of the dish in the dish image; and calculating the bill amount corresponding to the current order based on the price of the dishes in each dish image and the quantity of the dish images.

Optionally, in the cash registering method according to the present invention, the first dish category identification model is determined by performing a fine adjustment process based on a category identification model trained in advance.

Optionally, in the cashier method according to the invention, the fine-tuning process based on the pre-trained class recognition model includes: modifying the output number of classifiers in a pre-trained class recognition model according to the number of the dish classes corresponding to the first dish image data set; correspondingly loading the parameters of all processing layers before the last processing layer in the pre-trained category identification model into the modified category identification model to generate a first dish category identification model; the first dish category identification model is trained based on the first set of dish image data such that an output of the first dish category identification model is indicative of a category of a dish in the input dish image.

Optionally, in the cashier method according to the invention, the class recognition model is model-trained on the basis of a pre-acquired image data set, such that an output of the class recognition model indicates a class of image content in the input image.

According to still another aspect of the present invention, there is provided a method for ordering dishes, comprising the steps of: firstly, acquiring one or more dish images corresponding to a current order, wherein the dish images contain corresponding dishes; inputting each dish image into a first dish type identification model respectively for processing so as to obtain image characteristics output by a bottleneck layer in the first dish type identification model, wherein the bottleneck layer comprises all processing layers before the last processing layer in the first dish type identification model; inputting the image characteristics into a second dish type identification model for identification so as to obtain the type of the dishes in the dish image; determining whether the dish is overtime according to the category of the dish in the dish image; and if the dish is out of service for a time, sending dish order-urging information to the corresponding client.

Optionally, in the dish order promoting method according to the present invention, the method further includes: counting the inventory of food materials corresponding to the dishes according to the categories of the dishes in the dish image; and if the stock quantity is lower than the preset food material consumption, sending a replenishment message to the client.

Optionally, in the dish order promoting method according to the present invention, the method further includes: and if the inventory is not lower than the preset food material consumption, sending a dish making message to the client.

Alternatively, in the dish order promotion method according to the present invention, the first dish category identification model is determined by performing a fine adjustment process based on a category identification model trained in advance.

Optionally, in the dish order promotion method according to the present invention, the performing fine-tuning processing based on the pre-trained category identification model includes: modifying the output number of classifiers in a pre-trained class recognition model according to the number of the dish classes corresponding to the first dish image data set; correspondingly loading the parameters of all processing layers before the last processing layer in the pre-trained category identification model into the modified category identification model to generate a first dish category identification model; the first dish category identification model is trained based on the first set of dish image data such that an output of the first dish category identification model is indicative of a category of a dish in the input dish image.

Optionally, in the dish order promotion method according to the present invention, the category identification model is model-trained based on a pre-acquired image data set, so that an output of the category identification model indicates a category of image content in the input image.

According to still another aspect of the present invention, there is provided a dish recognition apparatus including a feature extraction module and a recognition module. The characteristic extraction module is suitable for inputting the dish image to be identified into the first dish type identification model for processing so as to obtain the image characteristic output by a bottleneck layer in the first dish type identification model, wherein the bottleneck layer comprises all processing layers before the last processing layer in the first dish type identification model; the identification module is suitable for inputting the image characteristics into the second dish type identification model for identification so as to obtain the type of the dish in the dish image.

According to yet another aspect of the present invention, a cash register apparatus is provided, which includes a first acquisition module, a feature extraction module, a recognition module, a second acquisition module, and a calculation module. The first acquisition module is suitable for acquiring one or more dish images corresponding to the current order, and the dish images contain corresponding dishes; the characteristic extraction module is suitable for inputting each dish image into the first dish type identification model respectively for processing so as to obtain image characteristics output by a bottleneck layer in the first dish type identification model, wherein the bottleneck layer comprises all processing layers before the last processing layer in the first dish type identification model; the identification module is suitable for inputting the image characteristics into the second dish type identification model for identification so as to obtain the type of the dish in the dish image; the second acquisition module is suitable for acquiring the price of the dish according to the category of the dish in the dish image; the calculation module is suitable for calculating the bill amount corresponding to the current order based on the price of the dishes in each dish image and the quantity of the dish images.

According to still another aspect of the invention, a menu urging device is provided and comprises an acquisition module, a feature extraction module, an identification module, a determination module and a sending module. The acquisition module is suitable for acquiring one or more dish images corresponding to the current order, and the dish images contain corresponding dishes; the characteristic extraction module is suitable for inputting each dish image into the first dish type identification model respectively for processing so as to obtain image characteristics output by a bottleneck layer in the first dish type identification model, wherein the bottleneck layer comprises all processing layers before the last processing layer in the first dish type identification model; the identification module is suitable for inputting the image characteristics into the second dish type identification model for identification so as to obtain the type of the dish in the dish image; the determining module is suitable for determining whether the dish is out of the meal overtime according to the category of the dish in the dish image; the sending module is suitable for sending dish order-urging information to the corresponding client when the dish is out of the dish for a time.

According to yet another aspect of the invention, there is provided a computing device comprising one or more processors, memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing a dish identification method, a cash register method, and/or a dish order method according to the invention.

According to yet another aspect of the present invention, there is also provided a computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing device, cause the computing device to perform a dish identification method, a cash register method and/or a dish order method according to the present invention.

According to the dish identification scheme, the dish image to be identified is subjected to forward reasoning by using the first dish type identification model, and the image characteristics of the bottleneck layer are extracted and input into the second dish type identification model so as to determine the type of the dish. The first dish type identification model is determined by fine adjustment processing based on a pre-trained type identification model, and the type identification model is trained on a large-scale image data set, so that strong identification capability is ensured, and the first dish type identification model has a good initial network structure. Furthermore, a universal first dish image data set is adopted to complete the transfer training of the first dish type identification model, and the identification performance on dish identification is further improved. In consideration of the fact that the dishes obtained by processing the same raw material in different restaurants are likely to have larger difference, a small number of dish samples in each restaurant are collected to form a specific second dish image data set, and the set is combined with the first dish type identification model to train the second dish type identification model, so that more accurate identification capability is obtained, the expected dish identification effect can be achieved without collecting a large number of dish samples which are actually used, and the development time and the development cost are greatly saved.

Furthermore, based on the cashier scheme and the dish order-hastening scheme which are provided by the dish identification, on the premise that the identification precision and the identification speed of the dish type are guaranteed, the cashier scheme can achieve quick and accurate bill settlement, and the dish order-hastening scheme can timely inform a kitchen to accelerate the speed of making dishes aiming at the dishes with overtime dinner so as to improve the dining experience of diners.

Drawings

To the accomplishment of the foregoing and related ends, certain illustrative aspects are described herein in connection with the following description and the annexed drawings, which are indicative of various ways in which the principles disclosed herein may be practiced, and all aspects and equivalents thereof are intended to be within the scope of the claimed subject matter. The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description read in conjunction with the accompanying drawings. Throughout this disclosure, like reference numerals generally refer to like parts or elements.

FIG. 1 shows a schematic diagram of a dish identification system 100 according to one embodiment of the present invention;

fig. 2 shows a schematic diagram of a cashier system 200 according to an embodiment of the invention;

FIG. 3 shows a schematic diagram of a dish order system 300 according to one embodiment of the present invention;

FIG. 4 shows a block diagram of a computing device 400, according to an embodiment of the invention;

FIG. 5A shows a schematic diagram of a dish identification process according to one embodiment of the invention;

FIG. 5 illustrates a flow diagram of a dish identification method 500 according to one embodiment of the present invention;

FIG. 6A illustrates a schematic diagram of a type recognition model, according to one embodiment of the invention;

FIG. 6B illustrates a schematic diagram of a first dish category identification model, according to one embodiment of the invention;

FIG. 7 shows a flow diagram of a cashier method 600 according to one embodiment of the invention;

FIG. 8 shows a flow diagram of a dish order method 700 according to one embodiment of the invention

Fig. 9 shows a schematic diagram of a dish identification device 800 according to an embodiment of the invention;

fig. 10 shows a schematic view of a cashier device 900 according to an embodiment of the invention; and

fig. 11 shows a schematic view of a dish order taking device 1000 according to an embodiment of the present invention.

Detailed Description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

Fig. 1 shows a schematic diagram of a dish identification system 100 according to an embodiment of the invention. It should be noted that the dish identification system 100 in fig. 1 is only an example, in a specific practical situation, there may be different numbers of client devices and servers in the dish identification system 100, and the client devices are generally devices with a shooting function, and may be mobile terminals, such as smart phones, tablet computers, and the like, or may be computing devices, such as PCs, and the invention is not limited thereto.

As shown in fig. 1, the dish identification system 100 includes a client device 110 and a server 120, wherein a dish identification device (not shown) resides in the server 120. According to an embodiment of the present invention, the client device 110 is provided with a camera, and after the dish image is captured by the camera, the dish image is uploaded to the server 120.

The server 120 identifies the type of the received dish image by the dish identification device. Specifically, the dish identification device inputs an image of a dish to be identified into a first dish type identification model for processing so as to obtain image characteristics output by a bottleneck layer in the first dish type identification model, and then inputs the image characteristics into a second dish type identification model for identification so as to obtain the type of the dish in the dish image. Wherein the bottleneck layer comprises all processing layers before the last processing layer in the first dish category identification model. Further, in consideration of practical situations, the dish identification apparatus is not limited to be deployed in the server 120, but may also be deployed in the client device 110, so as to avoid dependency on a communication network, such as a 4G network, improve availability of the identification application in a network-less or weak signal network, and reduce operation and maintenance costs.

The dish identification system 100 will be described below in two specific application scenarios, that is, cash register and dish order promotion. In the cash register scenario, the dish identification system 100 is applied to the bill settlement of a restaurant, forming the cash register system 200. Fig. 2 shows a schematic view of a cashier system 200 according to an embodiment of the invention. It should be noted that the cashier system 200 in fig. 2 is only an example, in a specific practical situation, there may be different numbers of client devices and servers in the cashier system 200, and the client devices are generally devices with a shooting function, and may be mobile terminals, such as smart phones, tablet computers, and the like, or may be computing devices, such as PCs, and the invention is not limited thereto.

As shown in fig. 2, the cashier system 200 comprises a client device 210 and a server 220. A cash register (not shown) resides in the server 220. According to an embodiment of the present invention, the client device 210 is provided with a camera, and after the dish image is captured by the camera, the dish image is uploaded to the server 220. The server 220 performs bill settlement of the order corresponding to the dish through the cash register device for the received dish image.

After the diner selects the favorite dishes A1, A2 and A3, the dishes A1, A2 and A3 can be carried to the cashier through the tray for settlement. The cashier desk in the restaurant is provided with a settlement robot, the settlement robot is internally provided with a client device 210, and images of dishes A1, A2 and A3 can be shot through the client device 210.

It should be noted that the dishes a1, a2, and A3 may be photographed to form corresponding single dish images B1, B2, and B3, respectively, or may be photographed to form a dish image B4 including the dishes a1, a2, and A3 at the same time. If the dish images B1, B2 and B3 are obtained by shooting respectively, the three dish images B1, B2 and B3 are uploaded to the server 220 for dish identification, and then bill settlement is carried out on orders related to dishes, if only the dish image B4 is obtained by shooting, the dish image B4 is uploaded to the server 220, then the dish image B4 needs to be segmented, and then the images respectively containing the dishes A1, A2 and A3 are obtained and then subsequent processing is carried out. Of course, the invention is not limited thereto.

Taking the example that the server 220 receives the dish image a1 uploaded by the client device 210, the dish image a1 is input to the first dish type identification model through the cash register device to be processed, so as to obtain the image feature output by the bottleneck layer in the first dish type identification model, and then the image feature is input to the second dish type identification model to be identified, so as to obtain the dish type in the dish image a1 as the shredded pork with fish flavor. Further, dish images a2 and A3 were recognized, respectively, to obtain the corresponding categories of pan-fried rape and spare rib wax gourd soup. The price of the shredded pork with the fish flavor is 15 yuan, the price of the stir-fried rape is 9 yuan, the price of the spareribs and white gourd soup is 12 yuan, the current consumer is calculated to pay 36 yuan, and the result is fed back to the client device 210. Finally, the settlement robot feeds back the result that the 36 yuan meal fee should be paid to the diner for payment. Further, it is to be noted that, in consideration of practical circumstances, the cashier device is not limited to be disposed in the server 220, and may be disposed in the client device 210.

Under the menu ordering scene, the menu identification system 100 is applied to the menu ordering of a restaurant to form the menu ordering system 300. Fig. 3 shows a schematic diagram of a dish order system 300 according to an embodiment of the present invention. It should be noted that the dish order system 300 in fig. 3 is only exemplary, and in a specific practical situation, there may be different numbers of client devices and servers in the dish order system 300, and the client devices determine whether the shooting function should be provided according to the use situation, and may be mobile terminals, such as smart phones, tablet computers, and the like, or computing devices, such as PCs, and the invention is not limited thereto.

As shown in fig. 3, the menu system 300 includes a client device 310, a server 320, and a client device 330. The server 320 is resident with a dish order-ordering device (not shown). According to an embodiment of the present invention, the client device 310 is provided with a camera, through which images of dishes in the menu can be captured, and after a diner determines a dish with a good point, the client device 310 captures images of corresponding dishes in the menu to form a corresponding order, where the order is associated with images of dishes corresponding to one or more dishes selected by the diner. The client device 330 is usually disposed in the kitchen, and can receive various messages from the server 320, so that the kitchen staff can perform corresponding operations according to the messages, such as supplementing food materials, increasing the speed of making dishes, and the like.

After the client device 310 uploads the current order to the server 320, the server 320 obtains 3 dish images corresponding to the current order through a dish ordering device, wherein the 3 dish images are a dish image a1, a dish image a2 and a dish image A3. Taking the vegetable image a1 as an example, the vegetable image a1 is input into the first vegetable type identification model to be processed, so as to obtain the image characteristics output by the bottleneck layer in the first vegetable type identification model, and then the image characteristics are input into the second vegetable type identification model to be identified, so as to obtain the vegetable type in the vegetable image a1 as the shredded pork with fish flavor. Further, dish images a2 and A3 were recognized, respectively, to obtain the corresponding categories of pan-fried rape and spare rib wax gourd soup.

At this time, the dish ordering device in the server 320 has received the dish meal message fed back by the client device 330, and the dish meal message indicates that the two dishes, i.e., the shredded fish-flavor meat and the stir-fried rape, have been made and have been served, but the spareribs and white gourd soup has not yet been made, and the meal is out of time. Based on this, a dish order message is sent to the corresponding client (usually software such as a kitchen management system) in the client device 330, so as to prompt a kitchen worker to accelerate the speed of making the dish of the spare ribs and the white gourd soup. It should be noted that, in consideration of practical situations, the dish order urging device is not limited to be deployed in the server 320, but may be deployed in the client device 310, and if deployed in the client device 310, the client device 330 is directly connected to the client device 310 in a communication manner.

According to an embodiment of the present invention, the server 120 in the dish identification system 100, the server 220 in the cashier system 200, and the server 320 in the dish order system 300 may be implemented by the computing device 400 as described below. FIG. 4 shows a block diagram of a computing device 400, according to one embodiment of the invention.

As shown in FIG. 4, in a basic configuration 402, a computing device 400 typically includes a system memory 406 and one or more processors 404. A memory bus 408 may be used for communicating between the processor 404 and the system memory 406.

Depending on the desired configuration, processor 404 may be any type of processing, including but not limited to: a microprocessor (μ P), a microcontroller (μ C), a Digital Signal Processor (DSP), or any combination thereof. Processor 404 may include one or more levels of cache, such as a level one cache 410 and a level two cache 412, a processor core 414, and registers 416. The example processor core 414 may include an Arithmetic Logic Unit (ALU), a Floating Point Unit (FPU), a digital signal processing core (DSP core), or any combination thereof. The example memory controller 418 may be used with the processor 404, or in some implementations the memory controller 418 may be an internal part of the processor 404.

Depending on the desired configuration, system memory 406 may be any type of memory including, but not limited to: volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. The system memory 406 may include an operating system 420, one or more programs 422, and data 424. In some implementations, the program 422 can be arranged to execute instructions on an operating system with the data 424 by one or more processors 404.

Computing device 400 may also include an interface bus 440 that facilitates communication from various interface devices (e.g., output devices 442, peripheral interfaces 444, and communication devices 446) to the basic configuration 402 via bus/interface controller 430. The example output device 442 includes a graphics processing unit 448 and an audio processing unit 450. They may be configured to facilitate communication with various external devices, such as a display or speakers, via one or more a/V ports 452. Example peripheral interfaces 444 may include a serial interface controller 454 and a parallel interface controller 456, which may be configured to facilitate communications with external devices such as input devices (e.g., keyboard, mouse, pen, voice input device, touch input device) or other peripherals (e.g., printer, scanner, etc.) via one or more I/O ports 458. An example communication device 446 may include a network controller 460, which may be arranged to facilitate communications with one or more other computing devices 462 over a network communication link via one or more communication ports 464.

A network communication link may be one example of a communication medium. Communication media may typically be embodied by computer readable instructions, data structures, program modules, and may include any information delivery media, such as carrier waves or other transport mechanisms, in a modulated data signal. A "modulated data signal" may be a signal that has one or more of its data set or its changes made in such a manner as to encode information in the signal. By way of non-limiting example, communication media may include wired media such as a wired network or private-wired network, and various wireless media such as acoustic, Radio Frequency (RF), microwave, Infrared (IR), or other wireless media. The term computer readable media as used herein may include both storage media and communication media.

Computing device 400 may be implemented as a server, such as a file server, a database server, an application server, a WEB server, etc., or as part of a small-form factor portable (or mobile) electronic device, such as a cellular telephone, a Personal Digital Assistant (PDA), a personal media player device, a wireless WEB-watch device, a personal headset device, an application specific device, or a hybrid device that include any of the above functions. Computing device 400 may also be implemented as a personal computer including both desktop and notebook computer configurations.

In some embodiments, computing device 400 is implemented as server 120, server 220, and/or server 320 and is configured to perform dish identification method 500, cashier method 600, and/or dish order method 700 in accordance with the present invention. Program 222 of computing device 200 includes a plurality of program instructions for executing dish identification method 500, cash register method 600, and/or dish order method 700 according to the present invention, and data 224 may also store configuration information of dish identification system 100, cash register system 200, and/or dish order system 300, among other things.

Fig. 5A shows a schematic diagram of a dish identification process according to an embodiment of the invention. As shown in fig. 5A, an image of a dish to be recognized is input to the first dish type recognition model for processing, so as to obtain an image feature output by the bottleneck layer in the first dish type recognition model, and then the image feature is input to the second dish type recognition model for recognition, so as to obtain the type of the dish in the dish image. The first dish type recognition model is determined based on fine adjustment processing of a type recognition model which is trained through an image data set in advance, and is trained through the first dish image data set, and the second dish type recognition model is model training based on the second dish image data set and the first dish type recognition model.

Fig. 5 shows a flow diagram of a dish identification method 500 according to an embodiment of the invention. As shown in fig. 5, the method 500 begins at step S510. In step S510, the dish image to be recognized is input to the first dish type recognition model for processing, so as to obtain the image feature output by the bottleneck layer in the first dish type recognition model, where the bottleneck layer includes all processing layers before the last processing layer in the first dish type recognition model.

According to one embodiment of the invention, the first dish type identification model is determined by performing fine adjustment processing based on a pre-trained type identification model. Wherein the category identification model comprises a deep neural network comprising a plurality of processing layers. In this embodiment, the deep neural network is a convolutional neural network, and the processing layer is any one of a convolutional layer, a pooling layer, and a fully-connected layer. In other words, the class identification model is a model determined by taking a convolutional neural network as an architecture, and fig. 6A shows a schematic diagram of the class identification model according to an embodiment of the present invention.

As shown in fig. 6A, the class identification model includes M +1 processing layers and 1 classifier, and the classifier is used to output the final N1 identification results. The full connection layer is usually selected as the last processing layer, i.e. the (M + 1) th processing layer, and the first M processing layers may be convolutional layers, pooling layers, or full connection layers, and may be an activation function layer, a normalization layer, etc., without limiting the present invention. The network structure specifically adopted by the category identification model can be appropriately adjusted according to the actual application scenario, the network training situation, the system configuration, the performance requirement, and the like, which are easily conceivable by those skilled in the art who know the scheme of the present invention and are also within the protection scope of the present invention, and are not described herein again.

After the network structure of the class recognition model is determined, the class recognition model needs to be trained first for subsequent use. According to one embodiment of the invention, the class recognition model is model trained based on a pre-acquired image data set such that the output of the class recognition model is indicative of the class of image content in the input image. The image data set comprises a plurality of pieces of image data, and each piece of image data comprises a training image and a category of image content in the training image.

When training the class recognition model, first, a training image in image data is input to the class recognition model for each piece of image data in the image data set to obtain a first class recognition result of the training image output by the class recognition model, and then, based on a difference between a class of image content in the training image and the first class recognition result, a parameter of the class recognition model is adjusted, usually by using a back propagation algorithm to adjust the parameter.

In this embodiment, the image data set employs an ImageNet data set. The ImageNet dataset provides a large visual database for visual object recognition software research, providing a large number of public images. The ImageNet dataset contains 20000 classes, a typical class, such as "balloon" or "strawberry", contains hundreds of images, and is trained with models that facilitate the identification of objects or biological classes.

After the training of the category identification model is completed by utilizing the ImageNet data set, fine tuning can be performed through the trained category identification model to determine the first dish category identification model. According to an embodiment of the present invention, the fine-tuning process may be performed based on a class recognition model trained in advance in the following manner. Firstly, according to the dish category number corresponding to the first dish image data set, the output number of classifiers in a pre-trained category identification model is modified.

In this embodiment, the first dish image data set includes a plurality of pieces of dish image data, each piece of dish image data includes a dish training image and categories of dishes in the dish training image, the categories of the dishes in all the dish training images are counted, and the total number of different categories is accumulated as the number of the categories of the dishes. Generally, images of various dishes can be downloaded through the internet, and corresponding image preprocessing, such as cutting, rotating, denoising and the like, is performed to form dish training images with uniform size and resolution so as to train a model for dish category identification.

If the number of dish categories corresponding to the first dish image data set is N2, the number of output of the classifier in the modified and trained category identification model is N2. And finally, correspondingly loading the parameters of all the processing layers before the last processing layer in the pre-trained category identification model into the modified category identification model to generate a first dish category identification model. According to an embodiment of the present invention, referring to fig. 5, parameters of all processing layers before the M +1 th processing layer in the pre-trained category identification model, i.e. the first M processing layers, are correspondingly loaded into the modified category identification model to generate the first dish category identification model.

It is apparent that the first dish category identification model has a similar network structure to the category identification model, i.e. the first dish category identification model comprises a deep neural network comprising a plurality of processing layers. In this embodiment, the deep neural network is a convolutional neural network, and the processing layer is any one of a convolutional layer, a pooling layer, and a fully-connected layer. In other words, the first dish category identification model is also a model determined by taking a convolutional neural network as an architecture, and fig. 6B shows a schematic diagram of the first dish category identification model according to an embodiment of the present invention. As shown in fig. 6B, the first dish category identification model includes M +1 processing layers and 1 classifier, and the classifier will output N2 identification results. And M processing layers before the M +1 processing layer are bottleneck layers of the first dish type identification model. Of course, for the class identification model shown in fig. 6A, the M processing layers before the M +1 processing layer can also be regarded as bottleneck layers of the class identification model.

After the network structure of the first dish category identification model is preliminarily formed, the first dish category identification model is trained based on the first dish image data set so that the output of the first dish category identification model indicates the category of the dish in the input dish image. Specifically, each piece of dish image data in the first dish image data set is input into the first dish type identification model by taking a dish training image in the dish image data as input, so as to obtain a first dish type identification result of the dish training image output by the first dish type identification model, and then parameters of the first dish type identification model are adjusted based on the difference between the dish type in the dish training image and the first dish type identification result.

Considering that the trained category identification model has good identification capability, the parameters of the first M processing layers transplanted through the trained category identification model may remain unchanged for most of the processing layers near the input end, and the parameters of the processing layers near the output end in the first dish category identification model are usually adjusted. For example, the parameters of the M-1 th to M +1 th process layers are adjusted.

In step S510, after the dish image a to be recognized is input to the first dish type recognition model, the image feature output from the bottleneck layer in the first dish type recognition model is obtained, and then, the process proceeds to step S520. In step S520, the image features are input into the second dish type identification model for identification, so as to obtain the type of the dish in the dish image.

According to an embodiment of the present invention, the second dish type identification model is model-trained based on the second dish image data set acquired in advance and the first dish type identification model, so that an output of the second dish type identification model indicates a type of the dish in the input dish image. The second dish image data set comprises a plurality of pieces of specific dish image data, and each piece of specific dish image data comprises a specific dish training image and the category of dishes in the specific dish training image.

In this embodiment, the specific dish training image is usually a specific dish image provided by a dining place where dish identification is required, such as a restaurant, and since dishes of the same category cooked by different restaurants may be different from each other, the specific dish training image should be collected for the dish image of the restaurant, and the specific dish training image with uniform size and resolution is formed after corresponding image preprocessing, such as cutting, rotating, denoising, and the like, so as to train a model for identifying the specific dish category.

When the second dish type identification model is trained, firstly, for each piece of specific dish image data in the second dish image data set, a specific dish training image in the specific dish image data is input into the first dish type identification model for processing, so as to obtain training image characteristics output by a bottleneck layer in the first dish type identification model. Then, the training image features are used as input and input into the second dish type recognition model, so that a second type recognition result of dishes in the training image of the specific dish corresponding to the training image features and output by the second dish type recognition model is obtained. And finally, adjusting parameters of a second dish type identification model based on the type of the dishes in the specific dish training image and the second type identification result. In this embodiment, the second dish category identification model includes a Support Vector Machine (SVM) model.

The support vector machine is used as a classification algorithm, the generalization capability of the learning machine is improved by seeking the minimum structured risk, and the minimization of the experience risk and the confidence range is realized, so that the aim of obtaining a good statistical rule under the condition of less statistical sample quantity is fulfilled. In popular terms, the support vector machine model is a two-class classifier, and a basic model of the support vector machine model is defined as a linear classifier with the maximum interval on a feature space, namely, a learning strategy of the support vector machine is interval maximization, and finally, the learning strategy can be converted into the solution of a convex quadratic programming problem.

Obviously, the support vector machine model adopted in the second dish category identification model should be a multi-category classifier, and the current methods for constructing the multi-category classifier of the support vector machine mainly include two categories, namely a direct method and an indirect method. The direct method is to directly modify an objective function, solve and combine parameters of a plurality of classification surfaces into an optimization problem, and realize multi-class classification by solving the optimization problem in one step. The indirect method mainly realizes the construction of a multi-class classifier by combining a plurality of two classifiers, and common methods include a one-to-many (one-against-all) method and a one-to-one (one-against-one) method.

The one-to-many method is characterized in that samples of a certain class are sequentially classified into one class during training, other remaining samples are classified into another class, K two classes of classifiers are constructed by the samples of K classes, and unknown samples are classified into the class with the largest classification function value during classification. One-to-one rule is to design a two-class classifier between any two classes of samples, so that K (K-1)/2 two-class classifiers need to be designed for K classes of samples, and when an unknown sample is classified, the class with the most votes is the class of the unknown sample.

Specifically, which method is adopted to realize the support vector machine model of the multi-class classifier can be selected according to the actual situation, and the invention is not limited to this. In addition, the second dish type identification model may not only be constructed based on the above support vector machine model, but also be implemented by models such as Logistic Regression (LR) and GBDT (gradient boosting Decision Tree) algorithms, and the like, and the invention is not limited to which algorithm or model is used to construct the second dish type identification model, and may be selected according to actual application scenarios, network training conditions, system configurations, performance requirements, and the like, and the model construction process and corresponding parameters in the selected mode are appropriately adjusted, which are easily conceivable by those skilled in the art understanding the scheme of the present invention, and are also within the protection scope of the present invention, and are not described herein again.

According to an embodiment of the present invention, the image features of the dish image a obtained in step S510 are input into the second dish type identification model, and the type of the dish in the dish image a is identified as the shredded pork with fish flavor.

Fig. 7 shows a flow chart of a method 600 of cashing, according to one embodiment of the invention. As shown in fig. 7, the method 600 begins at step S610. In step S610, one or more dish images corresponding to the current order are obtained, where the dish images include corresponding dishes. According to an embodiment of the present invention, the current order corresponds to 3 menu images, menu image a1, menu image a2, and menu image A3, respectively.

Subsequently, step S620 is performed, and the dish images are respectively input into the first dish type identification model to be processed, so as to obtain image features output by a bottleneck layer in the first dish type identification model, where the bottleneck layer includes all processing layers before a last processing layer in the first dish type identification model. The first dish type identification model is determined by performing fine adjustment processing based on a pre-trained type identification model.

According to an embodiment of the present invention, the fine-tuning process may be performed based on a class recognition model trained in advance in the following manner. Firstly, according to the number of dish categories corresponding to a first dish image data set, modifying the output number of classifiers in a pre-trained category identification model, then correspondingly loading the parameters of all processing layers before the last processing layer in the pre-trained category identification model into the modified category identification model to generate a first dish category identification model, and then training the first dish category identification model based on the first dish image data set so that the output of the first dish category identification model indicates the categories of dishes input into the dish image. Wherein the class recognition model performs model training based on a pre-acquired image data set such that an output of the class recognition model indicates a class of image content in the input image.

In this embodiment, the dish images a1, a2, and A3 are respectively input to the first dish type identification model and processed, and the output obtained as the bottleneck layer in the first dish type identification model is image features C1, C2, and C3 in this order.

In step S630, the image features are input into the second dish type identification model for identification to obtain the type of the dish in the dish image. According to an embodiment of the invention, after the image features C1, C2 and C3 are input into the second dish type identification model for identification, the type of the dishes in the dish image a1 is fish-flavored shredded pork, the type of the dishes in the dish image a2 is stir-fried rape, and the type of the dishes in the dish image A3 is sparerib-white gourd soup.

Next, step S640 is executed to obtain the price of the dish according to the category of the dish in the dish image. According to one embodiment of the invention, the obtained shredded fish-flavor meat is 15 yuan, the fried rape is 9 yuan, and the sparerib and wax gourd soup is 12 yuan.

Finally, in step S650, the bill amount corresponding to the current order is calculated based on the price of the dishes in each dish image and the number of the dish images. According to one embodiment of the present invention, the current order corresponds to a bill amount of 15 × 1+9 × 1+12 × 1 ═ 36 dollars.

The processing procedure for identifying the dish type of the dish image in steps S620 and S630 is disclosed in detail in the description of the method 500, and is not described herein again.

Fig. 8 shows a flow diagram of a dish order method 700 according to an embodiment of the invention. As shown in fig. 8, the method 700 begins at step S710. In step S710, one or more dish images corresponding to the current order are obtained, where the dish images include corresponding dishes. According to an embodiment of the present invention, the current order corresponds to 3 menu images, menu image a1, menu image a2, and menu image A3, respectively.

Subsequently, step S720 is performed, and the dish images are respectively input into the first dish type identification model for processing, so as to obtain image features output by a bottleneck layer in the first dish type identification model, where the bottleneck layer includes all processing layers before the last processing layer in the first dish type identification model. The first dish type identification model is determined by performing fine adjustment processing based on a pre-trained type identification model.

In step S730, the image feature is input into the second dish type identification model for identification to obtain the type of the dish in the dish image. According to an embodiment of the invention, after the image features C1, C2 and C3 are input into the second dish type identification model for identification, the type of the dishes in the dish image a1 is fish-flavored shredded pork, the type of the dishes in the dish image a2 is stir-fried rape, and the type of the dishes in the dish image A3 is sparerib-white gourd soup.

After the category of the dish is determined, according to another embodiment of the present invention, the inventory amount of the food material corresponding to the dish is counted according to the category of the dish in the dish image, if the inventory amount is lower than the preset food material usage amount, a replenishment message is sent to the client, and if the inventory amount is not lower than the preset food material usage amount, a dish making message is sent to the corresponding client.

In this embodiment, taking a spare rib white gourd soup as an example, the food materials corresponding to the dish comprise spare ribs and white gourd, the inventory amounts of the spare ribs and the white gourd are counted as D1 and D2, when the spare rib white gourd soup is made, the preset food material usage amount of the spare ribs is Δ D1, the preset food material usage amount of the white gourd is Δ D2, and since D1> Δ D1 and D2> Δ D2, a dish making message corresponding to the spare rib white gourd soup is sent to the corresponding client.

Next, step S740 is executed to determine whether the dish is out of time according to the category of the dish in the dish image. According to one embodiment of the invention, the two dishes of the shredded pork with fish flavor and the stir-fried rape are already prepared and eaten, but the spareribs and white gourd soup is not prepared yet, and the eating is overtime.

Finally, in step S750, if the dish is out of service for a time, a dish order message is sent to the corresponding client. According to an embodiment of the invention, when the dish of the spareribs and white gourd soup is eaten for a long time, a dish order message is sent to a client (generally software such as a kitchen management system) to prompt a kitchen worker to accelerate the making speed of the dish of the spareribs and white gourd soup.

The processing procedure for identifying the dish type of the dish image in steps S720 and S730 is disclosed in detail in the description of the method 500, and is not described herein again.

Fig. 9 shows a schematic diagram of a dish identification device 800 according to an embodiment of the invention. As shown in fig. 9, the dish recognition apparatus 800 includes a feature extraction module 810 and a recognition module 820.

The feature extraction module 810 is adapted to input the dish image to be identified into the first dish type identification model for processing, so as to obtain image features output by a bottleneck layer in the first dish type identification model, wherein the bottleneck layer includes all processing layers before a last processing layer in the first dish type identification model.

According to one embodiment of the invention, the first dish type identification model is determined by performing fine adjustment processing based on a pre-trained type identification model. The feature extraction module 810 is further adapted to perform fine-tuning processing based on the pre-trained category identification model, and further adapted to modify the output number of the classifiers in the pre-trained category identification model according to the number of categories of dishes corresponding to the first dish image data set, correspondingly load the parameters of all processing layers before the last processing layer in the pre-trained category identification model into the modified category identification model to generate a first dish category identification model, and train the first dish category identification model based on the first dish image data set so that the output of the first dish category identification model indicates the category of the dish input into the dish image.

According to one embodiment of the present invention, the first dish category identification model includes a deep neural network including a plurality of processing layers. Wherein, the deep neural network is a convolutional neural network, and the processing layer is any one of a convolutional layer, a pooling layer and a full-link layer.

In other words, the first dish category identification model is also a model determined by taking a convolutional neural network as an architecture, and fig. 6B shows a schematic diagram of the first dish category identification model according to an embodiment of the present invention. As shown in fig. 6B, the first dish category identification model includes M +1 processing layers and 1 classifier, and the classifier will output N2 identification results. And M processing layers before the M +1 processing layer are bottleneck layers of the first dish type identification model.

According to one embodiment of the invention, the class recognition model is model trained based on a pre-acquired image data set such that the output of the class recognition model is indicative of the class of image content in the input image. The image data set comprises a plurality of pieces of image data, and each piece of image data comprises a training image and a category of image content in the training image. The feature extraction module 810 is further adapted to perform model training based on a pre-acquired image data set, further adapted to input a training image in the image data to the category recognition model as input for each piece of image data in the image data set to obtain a first category recognition result of the training image output by the category recognition model, and adjust a parameter of the category recognition model based on a difference between a category of image content in the training image and the first category recognition result. In this embodiment, the image dataset is an ImageNet dataset.

According to one embodiment of the present invention, the first set of dish image data includes a plurality of pieces of dish image data, each piece of dish image data including a dish training image and a category of a dish in the dish training image. Generally, images of various dishes can be downloaded through the internet, and corresponding image preprocessing, such as cutting, rotating, denoising and the like, is performed to form dish training images with uniform size and resolution so as to train a model for dish category identification.

The feature extraction module 810 is further adapted to train the first dish type identification model based on the first dish image data set, and further adapted to input, to the first dish type identification model, a dish training image in the dish image data as an input to obtain a first dish type identification result of the dish training image output by the first dish type identification model, and adjust parameters of the first dish type identification model based on a difference between a category of a dish in the dish training image and the first dish type identification result.

In this embodiment, the feature extraction module 810 is further adapted to adjust parameters of a plurality of processing layers in the first dish category identification model near the output.

The recognition module 820 is adapted to input the image features into the second dish type recognition model for recognition to obtain the type of the dish in the dish image.

According to an embodiment of the present invention, the second dish type identification model is model-trained based on the second dish image data set acquired in advance and the first dish type identification model, so that an output of the second dish type identification model indicates a type of the dish in the input dish image. In this embodiment, the second dish category identification model comprises a support vector machine model.

According to an embodiment of the present invention, the second dish image data set includes a plurality of pieces of specific dish image data, each piece of specific dish image data including a specific dish training image and a category of a dish in the specific dish training image. In this embodiment, the specific dish training image is usually a specific dish image provided by a dining place where dish identification is required, such as a restaurant, and since dishes of the same category cooked by different restaurants may be different from each other, the specific dish training image should be collected for the dish image of the restaurant, and the specific dish training image with uniform size and resolution is formed after corresponding image preprocessing, such as cutting, rotating, denoising, and the like, so as to train a model for identifying the specific dish category.

The identification module 820 is further adapted to perform model training based on a second dish image data set obtained in advance and the first dish type identification model, and further adapted to input a specific dish training image in the specific dish image data into the first dish type identification model for processing to obtain a training image feature output by a bottleneck layer in the first dish type identification model, input the training image feature as an input into the second dish type identification model to obtain a second type identification result output by the second dish type identification model and corresponding to a dish in the specific dish training image by the training image feature, and adjust parameters of the second dish type identification model based on a type of a dish in the specific dish training image and the second type identification result.

The specific steps and embodiments of dish identification are disclosed in detail in the description based on fig. 5A to 6B, and are not described herein again.

Fig. 10 shows a schematic view of a cashier device 900 according to an embodiment of the invention. As shown in fig. 10, the cashier device 900 includes a first obtaining module 910, a feature extracting module 920, a recognition module 930, a second obtaining module 940 and a calculating module 950.

The first obtaining module 910 is adapted to obtain one or more dish images corresponding to a current order, where the dish images include corresponding dishes.

The feature extraction module 920 is adapted to input each dish image into the first dish type identification model for processing, so as to obtain image features output by a bottleneck layer in the first dish type identification model, where the bottleneck layer includes all processing layers before a last processing layer in the first dish type identification model.

According to one embodiment of the invention, the first dish type identification model is determined by performing fine adjustment processing based on a pre-trained type identification model. The feature extraction module 920 is further adapted to perform fine-tuning processing based on the pre-trained category identification model, and further adapted to modify the output number of the classifiers in the pre-trained category identification model according to the number of categories of dishes corresponding to the first dish image data set, correspondingly load the parameters of all processing layers before the last processing layer in the pre-trained category identification model into the modified category identification model to generate a first dish category identification model, and train the first dish category identification model based on the first dish image data set, so that the output of the first dish category identification model indicates the category of the dish input into the dish image. Wherein the class recognition model performs model training based on a pre-acquired image data set such that an output of the class recognition model indicates a class of image content in the input image.

The recognition module 930 is adapted to input the image feature into the second dish type recognition model for recognition to obtain the type of the dish in the dish image.

The second obtaining module 940 is adapted to obtain the price of the dish according to the category of the dish in the dish image.

The calculating module 950 is adapted to calculate a billing amount corresponding to the current order based on the price of the dish in each dish image and the number of the dish images.

The specific steps and embodiments of cash registering have been disclosed in detail in the description based on fig. 2 and 7, and are not described herein again.

Fig. 11 shows a schematic view of a cashier device 1000 according to an embodiment of the invention. As shown in fig. 11, the menu urging apparatus 1000 includes an acquisition module 1010, a feature extraction module 1020, a recognition module 1030, a determination module 1040, and a transmission module 1050.

The obtaining module 1010 is adapted to obtain one or more dish images corresponding to the current order, where the dish images include corresponding dishes.

The feature extraction module 1020 is adapted to input each dish image into the first dish type identification model for processing, so as to obtain image features output by a bottleneck layer in the first dish type identification model, where the bottleneck layer includes all processing layers before a last processing layer in the first dish type identification model.

The identification module 1030 is suitable for inputting the image characteristics into the second dish type identification model for identification so as to obtain the type of the dish in the dish image;

the determining module 1040 is adapted to determine whether the dish has a meal timeout according to the category of the dish in the dish image;

the sending module 1050 is adapted to send a dish order message to the corresponding client when the dish is out for a timeout.

According to an embodiment of the present invention, the sending module 1050 is further adapted to count the stock quantity of food materials corresponding to the dishes according to the categories of the dishes in the dish image, send a replenishment message to the client when the stock quantity is lower than a preset food material usage amount, and send a dish making message to the client when the stock quantity is not lower than the preset food material usage amount.

The specific steps and embodiments of the dish order promotion are disclosed in detail in the description based on fig. 3 and 8, and are not described herein again.

The existing dish identification method is usually realized by adopting a conventional deep neural network, if the accuracy is required to be ensured, a large number of dish samples which are actually used need to be collected for network training, and the network structure is relatively complex, so that the identification efficiency is relatively low. According to the dish identification scheme provided by the embodiment of the invention, the dish image to be identified is subjected to forward reasoning by using the first dish type identification model, and the image characteristics of the bottleneck layer are extracted and input into the second dish type identification model so as to determine the type of the dish. The first dish type identification model is determined by fine adjustment processing based on a pre-trained type identification model, and the type identification model is trained on a large-scale image data set, so that strong identification capability is ensured, and the first dish type identification model has a good initial network structure. Furthermore, a universal first dish image data set is adopted to complete the transfer training of the first dish type identification model, and the identification performance on dish identification is further improved. In consideration of the fact that the dishes obtained by processing the same raw material in different restaurants are likely to have larger difference, a small number of dish samples in each restaurant are collected to form a specific second dish image data set, and the set is combined with the first dish type identification model to train the second dish type identification model, so that more accurate identification capability is obtained, the expected dish identification effect can be achieved without collecting a large number of dish samples which are actually used, and the development time and the development cost are greatly saved.

In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art will appreciate that the modules or units or groups of devices in the examples disclosed herein may be arranged in a device as described in this embodiment, or alternatively may be located in one or more devices different from the devices in this example. The modules in the foregoing examples may be combined into one module or may be further divided into multiple sub-modules.

Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. Modules or units or groups in embodiments may be combined into one module or unit or group and may furthermore be divided into sub-modules or sub-units or sub-groups. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

Furthermore, some of the described embodiments are described herein as a method or combination of method elements that can be performed by a processor of a computer system or by other means of performing the described functions. A processor having the necessary instructions for carrying out the method or method elements thus forms a means for carrying out the method or method elements. Further, the elements of the apparatus embodiments described herein are examples of the following apparatus: the apparatus is used to implement the functions performed by the elements for the purpose of carrying out the invention.

The various techniques described herein may be implemented in connection with hardware or software or, alternatively, with a combination of both. Thus, the methods and apparatus of the present invention, or certain aspects or portions thereof, may take the form of program code (i.e., instructions) embodied in tangible media, such as floppy diskettes, CD-ROMs, hard drives, or any other machine-readable storage medium, wherein, when the program is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the invention.

In the case of program code execution on programmable computers, the computing device will generally include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device. Wherein the memory is configured to store program code; the processor is configured to execute the dish identification method, the cash register method and/or the dish order method of the present invention according to instructions in the program code stored in the memory.

By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer-readable media includes both computer storage media and communication media. Computer storage media store information such as computer readable instructions, data structures, program modules or other data. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. Combinations of any of the above are also included within the scope of computer readable media.

As used herein, unless otherwise specified the use of the ordinal adjectives "first", "second", "third", etc., to describe a common object, merely indicate that different instances of like objects are being referred to, and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking, or in any other manner.

While the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of this description, will appreciate that other embodiments can be devised which do not depart from the scope of the invention as described herein. Furthermore, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. Accordingly, many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the appended claims. The present invention has been disclosed in an illustrative rather than a restrictive sense, and the scope of the present invention is defined by the appended claims.

Claims

1. A method of dish identification, comprising:

inputting a dish image to be identified into a first dish type identification model for processing so as to obtain image characteristics output by a bottleneck layer in the first dish type identification model, wherein the bottleneck layer comprises all processing layers before the last processing layer in the first dish type identification model;

and inputting the image characteristics into a second dish type identification model for identification so as to obtain the type of the dish in the dish image.

2. The method of claim 1, wherein the first dish category identification model is determined based on a pre-trained category identification model for a fine tuning process.

3. The method of claim 2, wherein the fine-tuning based on the pre-trained class recognition model comprises:

modifying the output number of classifiers in a pre-trained class recognition model according to the number of the dish classes corresponding to the first dish image data set;

correspondingly loading the parameters of all processing layers before the last processing layer in the pre-trained category identification model into the modified category identification model to generate the first dish category identification model;

training the first dish category identification model based on the first dish image data set so that an output of the first dish category identification model indicates a category of a dish in an input dish image.

4. The method of claim 2 or 3, wherein the class recognition model is model trained based on a pre-acquired image data set, such that an output of the class recognition model is indicative of a class of image content in the input image.

5. The method of claim 4, the set of image data comprising a plurality of pieces of image data, each piece of image data comprising a training image and a category of image content in the training image, the model training based on the pre-acquired set of image data comprising:

for each piece of image data in the image data set, inputting a training image in the image data as input into the class recognition model to obtain a first class recognition result of the training image output by the class recognition model;

adjusting parameters of the class recognition model based on a difference between the class of image content in the training image and the first class recognition result.

6. The method of claim 4 or 5, wherein the image dataset is an ImageNet dataset.

7. The method of claim 3, wherein the first set of dish image data includes a plurality of pieces of dish image data, each piece of dish image data including a dish training image and a category of a dish in the dish training image, the training of the first dish category identification model based on the first set of dish image data includes:

for each item image data in the first item image data set, taking an item training image in the item image data as input, and inputting the item training image into the first item type identification model to obtain a first item type identification result of the item training image, which is output by the first item type identification model;

and adjusting parameters of the first dish type identification model based on the difference between the type of the dish in the dish training image and the first dish type identification result.

8. The method of claim 7, wherein said adjusting parameters of said first dish category identification model comprises:

and adjusting parameters of a plurality of processing layers close to the output end in the first dish type identification model.

9. The method of claim 1, wherein the second dish category identification model is based on a second pre-acquired dish image data set and the first dish category identification model is model trained such that an output of the second dish category identification model is indicative of a category of a dish in the input dish image.

10. The method of claim 9, wherein the second set of dish image data includes a plurality of pieces of dish-specific image data, each piece of dish-specific image data includes a dish-specific training image and a category of a dish in the dish-specific training image, the model training based on the pre-acquired second set of dish image data, and the first dish category identification model includes:

inputting specific dish training images in the specific dish image data into the first dish type identification model for processing to obtain training image characteristics output by a bottleneck layer in the first dish type identification model for each specific dish image data in the second dish image data set;

inputting the training image characteristics into the second dish type recognition model by taking the training image characteristics as input so as to obtain a second type recognition result of dishes in the training image of the specific dish corresponding to the training image characteristics, which is output by the second dish type recognition model;

and adjusting parameters of the second dish type identification model based on the type of the dishes in the specific dish training image and the second type identification result.

11. The method of claim 1, wherein the first dish category identification model comprises a deep neural network comprising a plurality of processing layers.

12. The method of claim 11, wherein the deep neural network is a convolutional neural network, and the processing layer is any one of a convolutional layer, a pooling layer, and a fully-connected layer.

13. The method of claim 1, wherein the second dish category identification model comprises a support vector machine model.

14. A method of cashier, comprising:

acquiring one or more dish images corresponding to a current order, wherein the dish images contain corresponding dishes;

respectively inputting each dish image into a first dish type identification model for processing to obtain image characteristics output by a bottleneck layer in the first dish type identification model, wherein the bottleneck layer comprises all processing layers before the last processing layer in the first dish type identification model;

inputting the image characteristics into a second dish type identification model for identification so as to obtain the type of the dishes in the dish image;

acquiring the price of the dish according to the category of the dish in the dish image;

and calculating the bill amount corresponding to the current order based on the price of the dishes in each dish image and the quantity of the dish images.

15. The method of claim 14, wherein the first dish category identification model is determined based on a pre-trained category identification model for a fine tuning process.

16. The method of claim 15, wherein the fine-tuning based on the pre-trained class recognition model comprises:

17. The method of claim 15 or 16, wherein the class recognition model is model trained based on a pre-acquired image data set such that an output of the class recognition model is indicative of a class of image content in the input image.

18. A method of ordering dishes comprising:

determining whether the dish is overtime according to the category of the dish in the dish image;

and if the dish is out of service overtime, sending a dish order message to the corresponding client.

19. The method of claim 18, further comprising:

according to the category of the dish in the dish image, counting the inventory of the food material corresponding to the dish;

and if the stock quantity is lower than the preset food material consumption, sending a replenishment message to the client.

20. The method of claim 19, further comprising:

and if the inventory is not lower than the preset food material consumption, sending a dish making message to the client.

21. The method of any of claims 18-20, wherein the first dish category identification model is determined based on a pre-trained category identification model with a fine tuning process.

22. The method of claim 21, wherein the fine-tuning based on the pre-trained class recognition model comprises:

23. The method of claim 21, wherein the class recognition model is model trained based on a pre-acquired image data set such that an output of the class recognition model indicates a class of image content in the input image.

24. A dish identification device comprising:

the characteristic extraction module is suitable for inputting a dish image to be identified into a first dish type identification model to be processed so as to obtain image characteristics output by a bottleneck layer in the first dish type identification model, wherein the bottleneck layer comprises all processing layers before the last processing layer in the first dish type identification model;

and the identification module is suitable for inputting the image characteristics into a second dish type identification model for identification so as to obtain the type of the dish in the dish image.

25. A cash register apparatus comprising:

the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is suitable for acquiring one or more dish images corresponding to a current order, and the dish images contain corresponding dishes;

the characteristic extraction module is suitable for inputting each dish image into a first dish type identification model respectively for processing so as to obtain image characteristics output by a bottleneck layer in the first dish type identification model, wherein the bottleneck layer comprises all processing layers before the last processing layer in the first dish type identification model;

the identification module is suitable for inputting the image characteristics into a second dish type identification model for identification so as to obtain the type of the dish in the dish image;

the second acquisition module is suitable for acquiring the price of the dish according to the category of the dish in the dish image;

and the calculation module is suitable for calculating the bill amount corresponding to the current order based on the price of the dishes in each dish image and the quantity of the dish images.

26. A menu item ordering device comprising:

the acquisition module is suitable for acquiring one or more dish images corresponding to the current order, wherein the dish images contain corresponding dishes;

the determining module is suitable for determining whether the dish is out of time according to the category of the dish in the dish image;

and the sending module is suitable for sending a dish order-urging message to the corresponding client when the dish is overtime.

27. A computing device, comprising:

one or more processors;

a memory; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing any of the methods of claims 1-23.

28. A computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing device, cause the computing device to perform any of the methods of claims 1-23.