CN114819821A

CN114819821A - Goods warehouse-out checking method and device, computer equipment and storage medium

Info

Publication number: CN114819821A
Application number: CN202210414644.XA
Authority: CN
Inventors: 陈桂平; 李晓刚; 陈玮; 郭天文; 卢达辉; 吴洪亮; 郭剑华; 卢子奎; 林敬炬; 林慧; 俞登成; 季志鹏; 刘国旺; 罗亿辉; 庄自成; 潘晓权
Original assignee: Longyan Tobacco Industry Co Ltd
Current assignee: Longyan Tobacco Industry Co Ltd
Priority date: 2022-04-20
Filing date: 2022-04-20
Publication date: 2022-07-29

Abstract

The application relates to a goods warehouse-out verification method, a goods warehouse-out verification device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: when detecting that the goods reach the delivery station, acquiring delivery images of the goods, placing the goods on trays, and arranging identification marks on the outer surface of each tray; acquiring a warehousing image collected when goods are warehoused according to the identification mark of the identified pallet; carrying out target recognition on the warehousing image, extracting goods in the warehousing image to obtain a first goods image, carrying out target recognition on the ex-warehouse image, extracting goods in the ex-warehouse image to obtain a second goods image; and performing feature matching on the first goods image and the second goods image, and if the matching is successful, determining that the warehouse-out verification is successful. By adopting the method, the condition that the goods delivered from the warehouse are inconsistent due to the verification error can be effectively avoided, the normal operation of the automatic warehouse logistics system is maintained, and the production efficiency is improved.

Description

Goods warehouse-out checking method and device, computer equipment and storage medium

Technical Field

The present application relates to the field of warehouse logistics technology, and in particular, to a method and an apparatus for checking shipment of goods, a computer device, a storage medium, and a computer program product.

Background

With the development of warehouse logistics management technology, automated warehouse logistics management has become one of the basic management means of many industrial production enterprises. In an automatic warehouse logistics system, pallets with uniform specifications and unique identifiers are often used for bearing goods, the pallets and the goods are kept in a goods shelf as a whole, and a carrying mechanism generally adopts an AGV trolley, a manual transport vehicle, a conveying belt, a stacker and the like to carry and carry the pallets bearing the goods so as to realize the automatic transportation of the goods.

In industrial production, due to practical management requirements, goods carried on pallets need to be kept consistent when being put in and when being taken out of a warehouse. In the traditional technology, a WCS system is generally used for verifying the cargo information of warehoused cargo and warehoused cargo through the unique identification of a tray so as to ensure the consistency of the warehoused cargo and the warehoused cargo. However, in the actual production process, the goods may be not restored in time after being put in storage through manual operation, or the goods may be inconsistent in shape after being collided in the storage process, and if the WCS system is used only to check the goods information, a check error is likely to occur, so that the automatic warehouse logistics system may operate disorderly and the production efficiency may be reduced due to inconsistency between the goods taken out of the storage and the goods put in the storage.

Disclosure of Invention

In view of the foregoing, it is desirable to provide a goods warehouse-out verification method, a goods warehouse-out verification apparatus, a computer device, a computer readable storage medium, and a computer program product, which can effectively verify whether goods warehouse-in and warehouse-out are consistent.

In a first aspect, the present application provides a cargo ex-warehouse verification method, including:

when detecting that goods reach a warehouse-out station, acquiring a warehouse-out image of the goods, wherein the goods are placed on trays, and the outer surface of each tray is provided with an identification mark;

acquiring a warehousing image collected when the goods are warehoused according to the recognized identification mark of the tray;

carrying out target recognition on the warehousing image, extracting goods in the warehousing image to obtain a first goods image, carrying out target recognition on the ex-warehouse image, extracting goods in the ex-warehouse image to obtain a second goods image;

and performing feature matching on the first goods image and the second goods image, and if the matching is successful, determining that the warehouse-out verification is successful.

In one embodiment, the acquiring, according to the identified identification mark of the pallet, a warehousing image collected when the goods are warehoused includes:

according to the recognized identification mark of the tray, acquiring warehousing information stored when the goods are warehoused;

acquiring the delivery information of the goods according to the identification mark;

performing basic verification on the warehousing information and the ex-warehouse information;

and if the basic verification is successful, acquiring a warehousing image acquired when the goods are warehoused.

In one embodiment, the performing target identification on the warehousing image, extracting goods in the warehousing image to obtain a first goods image, performing target identification on the ex-warehouse image, and extracting goods in the ex-warehouse image to obtain a second goods image includes:

inputting the warehousing image into an image target detection model obtained through pre-training for target recognition, and extracting first key features of goods in the warehousing image to obtain a first goods image containing the first key features;

and inputting the ex-warehouse image into the image target detection model for target recognition, and extracting second key features of goods in the ex-warehouse image to obtain a second goods image containing the second key features.

In one embodiment, the key features include at least one of an appearance outline trademark, an icon, text, and a bar code of the goods in the image; the performing feature matching on the first cargo image and the second cargo image comprises:

performing image alignment operation on the first cargo image and the second cargo image to obtain a first cargo image and a second cargo image which are aligned;

and performing feature matching on the attribute of the first key feature in the aligned first image and the attribute of the second key feature in the aligned second image.

In one embodiment, the method further comprises:

if the matching is unsuccessful, obtaining unmatched key features in the first cargo image and the second cargo image;

and determining whether each unmatched key feature meets a preset feature exclusion rule, and if so, determining that the ex-warehouse verification is successful.

In one embodiment, the acquiring an ex-warehouse image of the goods when the goods arriving at the ex-warehouse station is detected comprises:

when detecting that the goods arrive at a delivery station, acquiring delivery time sequence images of the goods collected within a preset time period;

carrying out grading operation on the ex-warehouse time sequence images according to a preset grading rule;

and determining the image with the highest grade in the images of the time series of delivery as the image of the delivery of the goods.

In a second aspect, the present application further provides a cargo warehouse-out verification apparatus, the apparatus includes:

the warehouse-out image acquisition module is used for acquiring warehouse-out images of the goods when the goods are detected to reach a warehouse-out station, the goods are placed on the trays, and the outer surface of each tray is provided with an identification mark;

the warehousing image acquisition module is used for acquiring warehousing images acquired when the goods are warehoused according to the identified identification marks of the trays;

the image target identification module is used for carrying out target identification on the warehousing image, extracting goods in the warehousing image to obtain a first goods image, carrying out target identification on the ex-warehouse image, extracting goods in the ex-warehouse image to obtain a second goods image;

and the characteristic matching module is used for carrying out characteristic matching on the first goods image and the second goods image, and if the matching is successful, the successful warehouse-out verification is determined.

In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

In a fourth aspect, the present application also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the method described above.

In a fifth aspect, the present application also provides a computer program product comprising a computer program which, when executed by a processor, performs the steps of the method described above.

According to the goods ex-warehouse verification method, the goods ex-warehouse verification device, the computer equipment, the storage medium and the computer program product, when the goods are detected to reach the ex-warehouse station, the ex-warehouse images of the goods are collected, and the warehouse-in images collected when the goods are warehoused are obtained according to the identification marks of the goods containing trays. Carrying out target identification on the warehousing image, and extracting goods in the warehousing image to obtain a first goods image; and carrying out target recognition on the ex-warehouse image, and extracting goods in the ex-warehouse image to obtain a second goods image. The first goods image collected during warehousing and the second goods image collected during delivery are subjected to feature matching, when the goods in warehousing are consistent with the goods delivered from the warehouse, the goods are delivered from the warehouse after matching is successful, the condition that the delivered goods are inconsistent due to verification errors is effectively avoided, normal operation of an automatic warehouse storage logistics system is maintained, and production efficiency is improved.

Drawings

FIG. 1 is a diagram of an exemplary application environment for a cargo warehouse-out verification method;

FIG. 2 is a schematic flow chart of a cargo ex-warehouse verification method according to an embodiment;

fig. 3 is a schematic flow chart illustrating a step of acquiring warehousing images collected when goods are warehoused according to the identification marks of the identified pallets in one embodiment;

FIG. 4 is a schematic illustration of a first cargo image in one embodiment;

FIG. 5 is a flowchart illustrating the step of collecting images of the shipment as it is detected that the shipment arrives at the shipment terminal in one embodiment;

FIG. 6 is a schematic flow chart of a cargo ex-warehouse verification method in another embodiment;

FIG. 7 is a diagram of an application environment of the cargo ex-warehouse verification method in another embodiment;

FIG. 8 is a diagram illustrating an output of a cargo shipment validation failure in one embodiment;

FIG. 9 is a block diagram of an exemplary cargo checkout apparatus;

FIG. 10 is a diagram showing an internal structure of a computer device according to an embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of and not restrictive on the broad application.

The goods ex-warehouse verification method provided by the embodiment of the application can be applied to the application environment shown in fig. 1. Wherein, the cargo verification system 102 communicates with the image capturing device 104 through a network. Data storage system 106 may store data that cargo verification system 102 needs to process. The data storage system may be integrated on the cargo verification system 102, or may be located on the cloud or other network server. When detecting that the goods reach the delivery station, the goods verification system 102 acquires a delivery image of the goods collected by the image collection device 104, wherein the goods are placed on the pallets, and the outer surface of each pallet is provided with an identification mark. The goods checking system 102 collects warehousing images of goods warehoused from the data storage system 106 according to the identification marks of the identified pallets; carrying out target recognition on the warehousing image, extracting goods in the warehousing image to obtain a first goods image, carrying out target recognition on the ex-warehouse image, extracting goods in the ex-warehouse image to obtain a second goods image; and performing feature matching on the first goods image and the second goods image, and if the matching is successful, determining that the warehouse-out verification is successful. The cargo verification system 102 may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, and the like, and may also be implemented by an independent server or a server cluster formed by a plurality of servers. The image capturing apparatus 104 has a function of continuously capturing images.

In one embodiment, as shown in fig. 2, a cargo ex-warehouse verification method is provided, which is described by taking the method as an example applied to the cargo verification system 102 in fig. 1, and includes the following steps:

step 202, when detecting that the goods reach the warehouse-out station, acquiring warehouse-out images of the goods, placing the goods on the trays, and setting identification marks on the outer surface of each tray.

The delivery platform is provided for the delivery verification of goods when the goods are delivered. Necessary image acquisition equipment, goods ex-warehouse sensing equipment and tray identification equipment are arranged on the ex-warehouse platform. It can be understood that when the warehousing control system issues a goods ex-warehouse command, the transportation equipment conveys the goods to the ex-warehouse platform for ex-warehouse verification, and the goods can be ex-warehouse after the ex-warehouse verification is confirmed to be successful.

The delivery image of the goods is the image acquired by the image acquisition equipment arranged on the delivery platform when the goods are transported to the delivery platform. It can be understood that the warehouse-out image may be a cargo image acquired by the image acquisition device of the warehouse-out station controlled by the cargo verification system after the cargo arrives at the warehouse-out station, or a cargo image acquired by the image acquisition device within a preset time interval after the cargo verification system receives information that the cargo arrives at the warehouse-out station.

Pallets are vehicles for converting static goods into dynamic goods, a cargo platform, but also movable platforms, or movable floors. Even if the goods are put on the ground and lose flexibility, the goods can obtain the mobility immediately after being loaded with the tray, and become flexible and mobile goods. The outer surface, such as the side, of each pallet will be provided with a unique identification, such as a pallet code. The warehousing control system manages the automatic logistics operation of the goods according to the unique identification mark of the pallet. It can be understood that in the embodiment of the application, the goods are all placed on the tray for transportation and verification.

Specifically, the goods are placed on the trays, after a warehouse-out instruction sent by the warehousing control system is received, the trays bearing the goods are transported to a warehouse-out platform by the transporting equipment, and when the goods are detected to be transported to the warehouse-out platform by the transporting equipment, the goods checking system acquires warehouse-out images corresponding to the goods.

In one embodiment, the warehouse exit platform is provided with a sensor for detecting whether the goods arrive at the warehouse exit platform, when the sensor detects that the goods arrive at the warehouse exit platform, the sensor sends a goods arrival signal to the warehouse control system, and after receiving the signal, the warehouse control system sends a goods arrival at the warehouse exit platform signal to the goods checking system to inform the goods checking system that the goods arrive at the warehouse exit platform.

And 204, acquiring a warehousing image collected when the goods are warehoused according to the identified identification mark of the tray.

The warehousing image is an image acquired by image acquisition equipment of a warehousing station when goods which need to be delivered out of the warehouse are warehoused. It can be understood that after the warehousing images corresponding to the warehoused goods are acquired, the goods verification system can bind the warehousing images and the pallet codes bearing the goods pallets one by one and store the warehousing images and the pallet codes in the data storage system of the goods verification system, and the warehousing images and the pallet codes are convenient to search and use quickly in subsequent comparison. It is understood that the warehousing station and the ex-warehouse station are two different verification stations.

Specifically, according to the recognized identification mark of the tray, the goods verification system searches the warehousing image collected when the goods corresponding to the identification mark are warehoused from the data storage system.

In one embodiment, the warehouse-out platform is provided with a scanner for scanning and identifying the tray identification mark.

In one embodiment, since the pallets are recycled in the automatic material-handling system, one identification mark in the goods verification system may correspond to the warehousing image corresponding to a plurality of goods. At this moment, according to the identification mark of the identified tray, the acquired warehousing image when the goods are warehoused is acquired, and the method comprises the following steps: and acquiring the acquisition time of the corresponding warehousing image according to the identification mark of the identified tray, and determining the warehousing image with the acquisition time having the minimum time interval with the current time as the warehousing image of the goods.

And step 206, carrying out target recognition on the warehousing image, extracting goods in the warehousing image to obtain a first goods image, carrying out target recognition on the ex-warehouse image, extracting goods in the ex-warehouse image to obtain a second goods image.

Wherein the object recognition is an operation of recognizing and marking goods in the image.

Specifically, the goods verification system performs target identification on the warehousing image, identifies and extracts goods in the warehousing image, and obtains a first goods image; and simultaneously, carrying out the same target identification operation on the ex-warehouse images, identifying and extracting the goods in the ex-warehouse images to obtain second goods images. It will be appreciated that the target recognition process may use a specialized target recognition algorithm or a pre-trained target detection model.

And 208, performing feature matching on the first goods image and the second goods image, and if the matching is successful, determining that the warehouse-out verification is successful.

Specifically, the goods verification system performs feature matching on a first goods image obtained based on the warehousing image and a second goods image obtained based on the ex-warehouse image, if matching is successful, the goods needing to be ex-warehouse are consistent with the goods in warehousing, after the goods are warehoused, the situations of goods loss, transfer, dropping, collision damage and the like do not occur, at the moment, it is determined that goods ex-warehouse verification is successful, and subsequent ex-warehouse operation can be executed. It is understood that the feature matching may be feature point matching in the image, cargo contour feature matching, cargo key point area matching, or the like, which is not limited in this application.

According to the goods warehouse-out checking method, when the goods are detected to arrive at the warehouse-out station, warehouse-out images of the goods are collected, and warehouse-in images collected when the goods are warehoused are obtained according to the identification marks of the goods containing trays. Carrying out target identification on the warehousing image, and extracting goods in the warehousing image to obtain a first goods image; and carrying out target recognition on the ex-warehouse image, and extracting goods in the ex-warehouse image to obtain a second goods image. The first goods image collected during warehousing and the second goods image collected during delivery are subjected to feature matching, when the goods in warehousing are consistent with the goods delivered from the warehouse, the goods are delivered from the warehouse after matching is successful, the condition that the delivered goods are inconsistent due to verification errors is effectively avoided, normal operation of an automatic warehouse storage logistics system is maintained, and production efficiency is improved.

In one embodiment, to exclude the effect of accumulating items outside the stations, a region of interest (ROI) is set for each station. The ROI setting mode can be set manually or automatically.

Specifically, when the ROI is manually set, the maximum visual field range slightly larger than the shooting tray and the goods is selected as the ROI; when the ROI is set automatically, a large number of images of pallet goods shot by each platform are collected, the positions of the goods in the images are identified through an algorithm, after the images are integrated, the edge positions [ x _ ROI _ min, y _ ROI _ min, x _ ROI _ max, y _ ROI _ max ] of all the goods in the same platform are obtained, namely [ minimum x pixel value, minimum y pixel value, maximum x pixel value and maximum y pixel value ], then a redundancy threshold value alpha is set, and the final ROI can be set to be [ x _ ROI _ min-alpha, y _ ROI _ min-alpha, x _ ROI _ max + alpha, y _ ROI _ max + alpha ]. It will be appreciated that the identification of the good may be accomplished by using a recognition algorithm or recognition model.

In one embodiment, as shown in fig. 3, acquiring a warehousing image collected when goods are warehoused according to the identification mark of the identified pallet, includes:

and 302, acquiring warehousing information stored when goods are warehoused according to the identified identification marks of the trays.

The warehousing information is goods information uploaded when goods are warehoused, and the warehousing information can comprise tray identification marks of the goods, goods types, bar code information, warehousing time and the like.

Specifically, when goods are put in storage, the storage control system binds storage information of the goods, storage images of the goods and identification marks of the goods trays together, and sends the binding information to the goods verification system for storage. And when the goods are detected to reach the warehouse-out station, the goods checking system obtains warehouse-in information corresponding to the identification mark from the data storage system according to the identification mark of the tray.

And step 304, acquiring the delivery information of the goods according to the identification mark.

The delivery information is the goods information uploaded when the goods are delivered, and the delivery information can include goods types, bar code information, delivery destinations and the like of the goods.

Specifically, the goods verification system acquires the ex-warehouse information of the goods corresponding to the identification mark according to the identification mark. It can be understood that the warehouse-out information of the goods may be generated and sent to the goods verification system when the warehouse control system generates a goods warehouse-out instruction, or may be sent to the goods verification system in real time according to the identification when the warehouse control system determines that the goods arrive at the warehouse-out station.

And step 306, performing basic verification on the warehousing information and the ex-warehouse information.

The warehousing information and the ex-warehousing information of the same goods are different only in a few parameters such as ex-warehousing destinations and the like, and other parameters related to the goods are the same as the goods type and the bar code information, so that the goods verification system can perform basic verification according to the information with the same parameters in the ex-warehousing information and the ex-warehousing information, and judge whether the goods corresponding to the ex-warehousing goods and the acquired warehousing information are the same goods.

Specifically, the cargo verification system performs basic verification on the warehousing information and the ex-warehouse information according to the same parameters in the warehousing information and the ex-warehouse information.

And 308, if the basic verification is successful, acquiring a warehouse-in image acquired when the goods are warehoused.

Specifically, if the basic verification is successful, it is indicated that the currently ex-warehouse goods and the in-warehouse goods corresponding to the in-warehouse information are the same goods, so that the in-warehouse image corresponding to the in-warehouse information is the image acquired when the currently ex-warehouse goods are put in the warehouse, and the image is acquired as the in-warehouse image, so as to perform subsequent image matching verification. If the basic verification is unsuccessful, it is indicated that the current goods out of the warehouse and the warehoused goods corresponding to the warehousing information are not the same goods, which may be caused by interference of manual operation or control error of the warehousing control system, and at this time, the goods verification system directly determines that the ex-warehouse verification is failed and notifies the manual intervention processing.

In the embodiment, basic inspection is performed on the ex-warehouse information of the goods and the warehousing information of the goods, which are acquired in advance according to the tray identification marks, the warehousing image corresponding to the warehousing information is acquired only when the basic inspection is successful, and the operation step of matching the warehousing image with the ex-warehouse image of the goods is performed.

In one embodiment, the target recognition of the warehousing image, the extraction of goods in the warehousing image to obtain a first goods image, the target recognition of the ex-warehouse image, the extraction of goods in the ex-warehouse image to obtain a second goods image includes:

and inputting the ex-warehouse image into an image target detection model for target recognition, and extracting second key features of the goods in the ex-warehouse image to obtain a second goods image containing the second key features.

The target detection model is a model obtained by pre-training and used for identifying and marking targets contained in the image. Specifically, the target detection model performs target recognition on the input image and outputs an image marked with key features.

The key features are features which can be used for identifying the consistency of goods in the images, and whether the goods contained in the two images are the same goods can be determined through matching of the key features. It is understood that there may be one or more key features in the same image, and the number of the key features is set according to the actual situation, which is not limited in this application.

Taking goods to be delivered out of a warehouse as tobacco boxes storing tobacco as an example, each brand of tobacco corresponds to a respective tobacco box, the brand name, the bar code and other features corresponding to the brand contained in the surface of the tobacco box are determined as key features, and a deep learning model is trained through a large amount of training data to obtain a final image target detection model. The image target detection model can identify and mark the characteristics of brand names, bar codes and the like on the surfaces of the tobacco boxes in the images, and outputs the images marked with the key characteristics.

Specifically, the goods verification system inputs the acquired warehousing image of the goods into the image target detection model, the image target detection model performs target identification on the warehousing image, the key features of the goods in the warehousing image are identified and marked as first key features, and the first goods image containing the first key features is output. The obtained goods delivery image is input into the image target detection model, the image target detection model carries out target identification on the delivery image, key features of goods in the delivery image are identified and marked as second key features, and a second goods image containing the second key features is output.

In the embodiment, the key features in the goods in and out-of-warehouse images are respectively identified and marked through the pre-trained image target detection model, the warehouse-in image containing the first key feature is output as the first goods image, the warehouse-out image containing the second key feature is output as the second goods image, and when the first image is subsequently subjected to feature matching with the second image, the first key feature of the first image can be directly matched with the second key feature of the second image, so that the whole warehouse-out checking process is quicker and more accurate.

In one embodiment, the key features include at least one of an appearance outline, a trademark, an icon, text, and a barcode of the good in the image.

Performing feature matching on the first cargo image and the second cargo image, including: performing image alignment operation on the first cargo image and the second cargo image to obtain a first cargo image and a second cargo image which are aligned; and performing feature matching on the attributes of the first key features in the aligned first image and the attributes of the second key features in the aligned second image.

Wherein the attribute of the key feature comprises at least one of content and area of the key feature. For example, when the key features are patterns and characters, the attributes of the key features may be the contents of the key features, and whether the features are matched is determined according to whether the contents are consistent; and when the key features are the features marked by the rectangular frames, determining whether the features are matched according to the area Intersection ratio (IOU) of the feature matrix, wherein the attributes of the key features are the areas of the key features.

Specifically, image alignment operation is performed on the first cargo image and the second cargo image according to the key features identified and marked by the image target detection model, and the aligned first cargo image and second cargo image are obtained. And performing feature matching on the attributes of the first key features in the aligned first image and the attributes of the second key features in the aligned second image. It will be appreciated that the image alignment operation may be implemented by an image alignment algorithm or model.

In one embodiment, the key features include outlines of goods in the image and bar codes, and the key features are marked by rectangular boxes. Performing an image registration operation on the first cargo image and the second cargo image according to the key features includes: and performing alignment operation on the first cargo image and the second cargo image according to the coordinates of the key feature rectangular frame.

Taking the ex-warehouse goods as the shape of the box body as an example, inputting the shot in-warehouse images into an image target detection model, using rectangular frame marks for box body outline and bar codes as key features by the image target detection model, and outputting to obtain a first goods image, wherein the first goods image is shown in fig. 4, the rectangular frame mark corresponding to A, B, C, D in the image is the box body outline key feature, and the rectangular frame mark corresponding to E, F, G, H is the bar code key feature. Determining pixel positions of image areas occupied by cargos, namely [ a minimum x coordinate (in _ x _ min) of a warehousing image, a minimum y coordinate (in _ y _ min) of the warehousing image, a maximum x coordinate (in _ x _ max) of the warehousing image and a maximum y coordinate (in _ y _ max) of the warehousing image ], and similarly obtaining pixel positions of the areas occupied by cargos of a delivery image, namely [ a minimum x coordinate (out _ x _ min) of the delivery image, a minimum y coordinate (out _ y _ min) of the delivery image, a maximum x coordinate (out _ x _ max) of the delivery image and a maximum y coordinate (out _ y _ max) of the delivery image ]. Calculating the average value of two corresponding coordinates in the two groups of coordinates, obtaining an average value coordinate [ aligning with a minimum x coordinate (scaled _ x _ min), a minimum y coordinate (scaled _ y _ min), a maximum x coordinate (scaled _ x _ max) and a maximum y coordinate (scaled _ y _ max) ], and scaling the corresponding coordinate values of the first cargo image and the second cargo image to the average value coordinate. And obtaining the first goods image and the second goods image after alignment. And aligning all rectangular frames of the key targets in the first cargo image and the second cargo image to the size of the aligned rectangular frames.

Assume that the coordinates of any key feature rectangular box in the first cargo image are identified as the coordinates of the top-left corner point and the bottom-right corner point, [ in _ x1, in _ y1, in _ x2, in _ y2], whose coordinates projected onto the aligned key feature rectangular box are scaled as follows:

k_x＝(scaled_x_max–scaled_x_min)/(in_x_max–in_x_min)；

k_y＝(scaled_y_max–scaled_y_min)/(in_y_max–in_y_min)；

scaled_in_x1＝(in_x1–in_x_min)*k_x；

scaled_in_y1＝(in_y1–in_y_min)*k_y；

scaled_in_x2＝(in_x2–in_x_min)*k_x；

scaled_in_y2＝(in_y2–in_y_min)*k_y；

the coordinates of the key feature rectangular boxes in the first cargo image after scaling are represented as [ scaled _ in _ x1, scaled _ in _ y1, scaled _ in _ x2, scaled _ in _ y2], and similarly, the coordinates of the key feature rectangular boxes in the second cargo image can be calculated as [ scaled _ out _ x1, scaled _ out _ y1, scaled _ out _ x2, and scaled _ out _ y2 ].

At this time, the attribute of the key feature is the area of the rectangular frame. Performing feature matching on the attribute of the first key feature of the aligned first image and the attribute of the second key feature in the aligned second image, wherein the feature matching comprises the following steps: and calculating the area of each key feature rectangular frame according to the aligned key feature rectangular frame coordinates, and calculating the area intersection ratio of each first key feature rectangular frame and each second key feature rectangular frame. And arranging the obtained area intersection ratios in a descending order, selecting the key feature group with the largest intersection ratio and the intersection ratio meeting a preset threshold value, and determining the key features as matching.

In the above embodiment, by performing the alignment operation on the first cargo image and the second cargo image output by the image target detection model, the feature matching is performed on the attribute of the first key feature of the aligned first image and the attribute of the second key feature in the aligned second image. The influence of errors caused by external factors such as picture acquisition angles on the feature matching result can be effectively reduced, and the accuracy of the ex-warehouse checking process is improved.

In one embodiment, the cargo ex-warehouse verification method further comprises the following steps:

if the matching is unsuccessful, obtaining unmatched key features in the first cargo image and the second cargo image; and determining whether each unmatched key feature meets a preset feature exclusion rule, and if so, determining that the ex-warehouse verification is successful.

And the unmatched key features are key features which cannot correspond to each other in the first cargo image and the second cargo image. When the unmatched key features exist, the fact that the goods in the first goods image are warehoused goods at the moment is indicated, and the difference exists between the goods in the first goods image and the goods in the second goods image, namely the warehoused goods.

The preset feature exclusion rule is a feature exclusion rule set for reducing the condition of feature matching errors caused by target detection errors, and is used for excluding unmatched key features meeting the preset rule.

Specifically, the warehousing image acquisition device and the ex-warehouse image acquisition device are two different devices and are arranged at two different platforms, so that shooting angle errors are inevitable, and meanwhile, when the image target detection model outputs an image after target recognition, recognition errors occur at a certain probability. Therefore, in order to reduce errors in the feature matching result caused by target detection errors, a preset feature exclusion rule is preset according to the model and the actual situation of the image acquisition area. And when the unmatched key features which are not successfully matched exist, determining whether each unmatched key feature meets a preset feature exclusion rule, if so, indicating that the key features which are not successfully matched are caused by target detection errors, and excluding the key features from the unmatched key features. If all the unmatched key features meet the preset feature exclusion rule, it is indicated that the goods in the first goods image and the goods in the second goods image are consistent at the moment, namely the goods in the warehouse are consistent with the goods out of the warehouse, and the successful verification of the goods out of the warehouse is determined. If the unmatched key features have key features which cannot meet the preset feature exclusion rule, the situation that the warehoused goods are inconsistent with the warehoused goods at the moment is shown, and the warehoused goods are changed after being warehoused. At the moment, the goods verification system determines that the ex-warehouse verification fails, finally, unmatched key features or goods corresponding to the key features are identified in the input image, and a goods verification abnormal alarm is generated and returned to the warehousing control system.

In the embodiment, for the key features which are not successfully matched, the preset feature exclusion rule is used for screening again, so that the condition that errors occur in the feature matching result caused by target detection errors can be effectively avoided, and the accuracy of the ex-warehouse verification process is effectively improved.

In one embodiment, the preset feature exclusion rule includes at least one of the target detection confidence being below a preset confidence threshold or the cargo coverage area being above a preset area threshold.

Specifically, the target detection confidence is information of whether the recognition result given by the image target detection model is accurate, the closer the target detection confidence is to 1, the higher the accuracy of the recognition result obtained by the representative model is, and the closer the target detection confidence is to 0, the more inaccurate the recognition result obtained by the representative model is. Therefore, if the target detection confidence corresponding to the unmatched key feature is lower than the preset confidence threshold, the identification of the key feature at this time is considered to be inaccurate, and the unmatched key feature is most likely caused by the identification error, so that the key feature is excluded from the unmatched key features by the preset feature exclusion rule.

The cargo coverage area refers to an area in the image where a certain cargo is covered by other articles. In actual production, goods carried by one tray are generally multilayer, and due to the fact that image acquisition equipment and stations corresponding to in-out images are different, some goods can be acquired by the image acquisition equipment when being put in storage due to external factors such as shooting angles, and when the uncovered area of the goods reaches a certain value, the goods are identified as key features by the image target detection model. And when the goods are taken out of the warehouse, if the size of the goods which can be collected is smaller than a certain value, the goods cannot be identified by the image target detection model. The key feature will be classified as an unmatched key feature, which may result in an unsuccessful ex-warehouse check. Therefore, the key features are excluded from the unmatched key features through a preset feature exclusion rule, so that the accuracy of the ex-warehouse verification result is improved.

It can be understood that the preset confidence threshold and the preset area threshold may be set according to actual situations, in this embodiment, the preset confidence threshold is set to be 0.9, and the preset area threshold is set to be 0.5.

In one embodiment, as shown in fig. 5, when the goods arriving at the delivery station is detected, the delivery image of the goods is collected, which includes the following steps:

step 502, when detecting that the goods arrive at the delivery station, acquiring delivery time sequence images of the goods collected within a preset time period.

The preset time period is the time period for continuously acquiring the images of the goods out of the warehouse by the preset image acquisition equipment. The image acquisition equipment corresponds to a fixed image acquisition area, and in the operation process of the automatic logistics transportation system, the image acquisition equipment performs continuous image acquisition on the fixed image acquisition area to obtain a sequence image within a certain time period.

Specifically, due to the ubiquitous delay problem in the communication process, when the cargo verification system detects that the cargo arrives at the delivery platform, it is highly probable that the cargo actually passes through the delivery platform or passes through the optimal image acquisition point of the delivery platform. If the cargo verification system acquires the image acquired by the image acquisition device in real time, the image quality is likely not to meet the requirement of feature matching. Therefore, when the goods checking system detects that the goods reach the delivery station, the delivery time sequence images of the goods continuously collected by the image collecting device within the preset time period are obtained. Taking an example that the image acquisition device acquires 10 frames of images in 1 second, when the cargo verification system detects that the cargo arrives at the warehouse-out station, acquiring 10 frames of images acquired by the image acquisition device 1s before the current time as the warehouse-out time sequence images of the cargo.

And step 504, performing grading operation on the ex-warehouse time sequence images according to a preset grading rule.

Wherein the preset scoring rule is a rule for evaluating image capturing quality. And determining a corresponding score for the images of the time series of the ex-warehouse according to a preset scoring rule. It can be understood that the preset scoring rule can be preset according to the actual application scenario and stored in the data storage system of the cargo verification system.

Specifically, the goods verification system scores the obtained ex-warehouse time sequence images according to a preset scoring rule to obtain a score value corresponding to each ex-warehouse time sequence image.

In one embodiment, the preset scoring rule is set according to the center coordinates of the whole cargo area and the center coordinates of the frame. Specifically, the score of each ex-warehouse sequence image is determined according to the position distance between the center coordinate of the whole cargo region in each ex-warehouse sequence image and the center coordinate of the image picture, and the score is higher as the center coordinate of the whole cargo region is closer to the center coordinate of the image picture.

Step 506, determining the image with the highest grade in the images of the time series of delivery as the delivery image of the goods.

Specifically, the obtained image scores are arranged in a descending order, and the image with the highest corresponding score is selected to be determined as the delivery image of the goods.

In the embodiment, the continuously acquired time sequence images are scored, and the image with the highest score is selected to be determined as the ex-warehouse image of the goods, so that the quality of the subsequent image for target identification is ensured, the accuracy of target detection is further improved, and the accuracy of the ex-warehouse check result is improved.

In one embodiment, as shown in fig. 6, a cargo ex-warehouse verification method is provided, which may be applied in the application environment shown in fig. 7, where the cargo in this embodiment is, for example, boxed cargo.

A portal frame is assumed at the periphery of the warehouse entry and exit station, and a portal frame beam is positioned right above the middle of the warehouse entry and exit station and is vertical to the running direction of the conveyor belt. The candid photograph camera hoist and mount on the portal frame crossbeam, and the camera is installed directly over the platform central point, is higher than the tray and the superimposed maximum height of goods in order to avoid colliding with the transportation goods to set up the perpendicular shooting downwards of camera. The photoelectric sensor is arranged under the warehousing-in/out platform, vertically and upwards measures and is used for detecting whether the pallet goods enter the platform or not. It can be understood that a tray code scanning device is also arranged on the platform and used for scanning the tray code of the tray. The tray code scanning device can be arranged at any position of the platform as long as the tray code can be scanned, and the tray code scanning device is not shown in the figure in the embodiment.

In order to eliminate the influence of deposits outside the platform, the region of interest is set as a fixed image acquisition region for the platform. The camera continuously shoots the pallet goods passing through the fixed image acquisition area to acquire images.

Specifically, a WCS (warehousing control system) sends a goods warehousing instruction to a conveyor belt, the conveyor belt responds to the goods warehousing instruction, a tray bearing goods is transported to a warehousing platform, and when the goods pass through an image fixed acquisition area of a camera, the camera can capture the goods continuously to obtain warehousing time sequence images. Meanwhile, the photoelectric sensor sends a sensing signal to the WCS, the WCS receives the sensing signal, acquires a tray code scanned by a tray code scanning device, determines warehousing cargo information of the cargo according to the tray code, generates a warehousing cargo arrival instruction according to the warehousing cargo information of the cargo and the tray code, and sends the warehousing cargo arrival instruction to the cargo verification system.

And the goods checking system receives the warehousing goods arrival instruction and acquires warehousing time sequence images acquired by the camera within a preset time period. The warehousing time series images are a collection of images of the goods from entering the image fixed acquisition area to leaving the image fixed acquisition area. For example, if the camera 1s collects 10 frames of pictures, after receiving the warehousing goods arrival instruction, the goods verification system acquires 50 frames of images collected by the camera within 5s as warehousing time-series images.

And the goods checking system calculates the score of the goods entering the platform according to the positions of the goods detected in the warehousing time sequence images, and if the central coordinates of the whole goods region are (x0, y0), the score is higher as the (x0, y0) is closer to the position of the center of the picture. And selecting the goods image with the highest score, performing image preprocessing on the goods image, determining the goods image as a warehousing image of the goods, binding the warehousing image of the goods with warehousing goods information and a tray code, and storing the goods image in a data storage system so as to be used for subsequent goods ex-warehouse verification.

When the WCS system sends a goods delivery instruction to the conveyor belt, the conveyor belt responds to the goods delivery instruction and transports the tray loaded with goods to a delivery platform. The photoelectric sensor sends an induction signal to the WCS system, the WCS system obtains a tray code obtained by scanning of the tray code scanning equipment after receiving the induction signal, the delivery goods information of the goods is determined according to the tray code, a delivery goods arrival instruction is generated according to the delivery goods information of the goods and the tray code, and the delivery goods arrival instruction is sent to the goods checking system.

The goods checking system acquires warehousing goods information which is closest to the pallet code at the current time from the data storage system according to the pallet code in the delivery goods arrival instruction, checks the delivery goods information and goods types, bar code information, warehousing time and the like in the warehousing goods information, outputs delivery checking abnormal information if the checking fails, and sends the delivery checking abnormal information to the WCS system.

And if the verification is successful, acquiring the warehousing image bound with the warehousing goods information and acquiring the ex-warehouse image of the collected goods. It can be understood that the cargo delivery image acquisition process is substantially identical to the cargo storage image acquisition process, and is not described herein again.

The goods checking system respectively inputs the warehousing image and the ex-warehouse image into a pre-trained image target detection model, the image target detection model labels key features such as the outline of a goods box body, a bar code and the like in the warehousing image by using a rectangular frame and outputs a first goods image, and labels key features such as the outline of the goods box body, the bar code and the like in the ex-warehouse image by using the rectangular frame and outputs a second goods image. The method comprises the steps of carrying out size alignment operation on a first goods image and a second goods image, respectively carrying out feature matching on each key feature in the aligned first goods image and each key feature in the aligned second goods image, calculating area intersection and comparison between rectangular frames of the key features, determining two key features which have the largest area intersection and comparison ratio and meet a preset area intersection and comparison threshold value as matching key features, and determining the key features as unmatched key features if certain key feature in the first goods image or the second goods image cannot be matched with other key features to obtain the area intersection and comparison ratio meeting the preset area intersection and comparison threshold value. And determining whether the unmatched key features meet a preset exclusion rule, and if so, excluding the key features from the unmatched key features. After all unmatched key features are excluded and screened, if the unmatched key features exist, it is determined that the goods are failed in the warehouse-out verification, and finally the unmatched key features or the goods corresponding to the key features are identified in the input image, as shown in fig. 8, wherein the unmatched key features I, J exist in the warehouse-in image, and the unmatched key features K, L, M, N exist in the warehouse-out image. And generating a cargo checking abnormity alarm according to the output image and returning the cargo checking abnormity alarm to the warehousing control system. And if the unmatched key features do not exist after screening is eliminated, determining that the goods are successfully checked out of the warehouse, and executing the subsequent goods out-of-the-warehouse step.

It should be understood that, although the steps in the flowcharts related to the embodiments as described above are sequentially displayed as indicated by arrows, the steps are not necessarily performed sequentially as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a part of the steps in the flowcharts related to the embodiments described above may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the execution order of the steps or stages is not necessarily sequential, but may be rotated or alternated with other steps or at least a part of the steps or stages in other steps.

Based on the same inventive concept, the embodiment of the application also provides a goods ex-warehouse verification device for realizing the goods ex-warehouse verification method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the method, so that specific limitations in one or more embodiments of the cargo ex-warehouse verification device provided below can be referred to the limitations on the cargo ex-warehouse verification method in the foregoing, and details are not described herein again.

In one embodiment, as shown in fig. 9, there is provided a cargo ex-warehouse verification apparatus 900, including: an ex-warehouse image acquisition module 901, an in-warehouse image acquisition module 902, an image target recognition module 903 and a feature matching module 904, wherein:

the warehouse-out image acquiring module 901 is configured to acquire warehouse-out images of the goods when the goods are detected to reach the warehouse-out station, the goods are placed on the trays, and the outer surface of each tray is provided with an identification mark.

And the warehousing image acquisition module 902 is configured to acquire the warehousing image acquired when the goods are warehoused according to the identification mark of the identified pallet.

The image target identification module 903 is configured to perform target identification on the warehousing image, extract goods in the warehousing image to obtain a first goods image, perform target identification on the ex-warehouse image, extract goods in the ex-warehouse image, and obtain a second goods image.

And the feature matching module 904 performs feature matching on the first cargo image and the second cargo image, and if the matching is successful, determines that the ex-warehouse verification is successful.

The goods warehouse-out checking device acquires the warehouse-in image acquired when goods are warehoused according to the identification mark of the goods tray. Carrying out target identification on the warehousing image, and extracting goods in the warehousing image to obtain a first goods image; and carrying out target recognition on the ex-warehouse image, and extracting goods in the ex-warehouse image to obtain a second goods image. The first goods image acquired during warehousing and the second goods image acquired during delivery are subjected to feature matching, when the goods during warehousing are consistent with the goods during delivery, the goods are delivered out of the warehouse after successful matching, the condition that the goods during delivery are inconsistent due to check errors is effectively avoided, normal operation of the automatic warehouse storage logistics system is maintained, and production efficiency is improved.

In one embodiment, the binned image acquisition module is further configured to: according to the identified identification mark of the tray, warehousing information stored when goods are warehoused is obtained; acquiring the delivery information of the goods according to the identification mark; basic verification is carried out on warehousing information and ex-warehouse information; and if the basic verification is successful, acquiring the warehousing image acquired when the goods are warehoused.

In one embodiment, the image object recognition module is further to: inputting the warehousing image into an image target detection model obtained through pre-training for target recognition, and extracting first key features of goods in the warehousing image to obtain a first goods image containing the first key features; and inputting the ex-warehouse image into an image target detection model for target recognition, and extracting second key features of the goods in the ex-warehouse image to obtain a second goods image containing the second key features.

In one embodiment, the feature matching module is further to: performing image alignment operation on the first cargo image and the second cargo image to obtain a first cargo image and a second cargo image which are aligned; and performing feature matching on the attributes of the first key features in the aligned first image and the attributes of the second key features in the aligned second image.

In one embodiment, the warehouse-out verification apparatus further comprises: the unmatched key feature screening module is used for acquiring unmatched key features in the first cargo image and the second cargo image if the matching is unsuccessful; and determining whether each unmatched key feature meets a preset feature exclusion rule, and if so, determining that the ex-warehouse verification is successful.

In one embodiment, the ex-warehouse image acquisition module is further configured to: when detecting that the goods arrive at a delivery station, acquiring delivery time sequence images of the goods collected within a preset time period; performing grading operation on the ex-warehouse time sequence images according to a preset grading rule; and determining the image with the highest score in the images of the time series of the delivery as the delivery image of the goods.

All or part of the modules in the cargo ex-warehouse verification device can be realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in fig. 10. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer equipment is used for storing data such as warehousing images, warehousing information, ex-warehouse images, ex-warehouse information and the like. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a cargo ex-warehouse verification method.

Those skilled in the art will appreciate that the architecture shown in fig. 10 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is provided, which may be a cargo verification system in the present application, and includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the following steps when executing the computer program:

when detecting that the goods reach the delivery station, acquiring delivery images of the goods, placing the goods on trays, and arranging identification marks on the outer surface of each tray;

acquiring a warehousing image collected when goods are warehoused according to the identification mark of the identified pallet;

In one embodiment, the processor, when executing the computer program, further performs the steps of:

according to the identified identification mark of the tray, acquiring warehousing information stored when goods are warehoused;

basic verification is carried out on warehousing information and ex-warehouse information;

and if the basic verification is successful, acquiring the warehousing image acquired when the goods are warehoused.

and performing feature matching on the attributes of the first key features in the aligned first image and the attributes of the second key features in the aligned second image.

performing grading operation on the ex-warehouse time sequence images according to a preset grading rule;

and determining the image with the highest score in the images of the time series of the delivery as the delivery image of the goods.

In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of:

when the goods are detected to reach a delivery station, a delivery image of the goods is obtained, the goods are placed on trays, and the outer surface of each tray is provided with an identification mark;

In one embodiment, the computer program when executed by the processor further performs the steps of:

according to the identified identification mark of the tray, warehousing information stored when goods are warehoused is obtained;

if the matching is unsuccessful, acquiring unmatched key features in the first cargo image and the second cargo image;

In one embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, performs the steps of:

It should be noted that, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high-density embedded nonvolatile Memory, resistive Random Access Memory (ReRAM), Magnetic Random Access Memory (MRAM), Ferroelectric Random Access Memory (FRAM), Phase Change Memory (PCM), graphene Memory, and the like. Volatile Memory can include Random Access Memory (RAM), external cache Memory, and the like. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others. The databases referred to in various embodiments provided herein may include at least one of relational and non-relational databases. The non-relational database may include, but is not limited to, a block chain based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing based data processing logic devices, etc., without limitation.

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A cargo ex-warehouse verification method is characterized by comprising the following steps:

acquiring warehousing images collected when the goods are warehoused according to the identified identification marks of the trays;

2. The method according to claim 1, wherein the acquiring, according to the identified identification of the pallet, the warehousing image collected when the goods are warehoused comprises:

3. The method according to claim 1, wherein the performing the target recognition on the warehousing image, extracting the goods in the warehousing image to obtain a first goods image, performing the target recognition on the ex-warehouse image, extracting the goods in the ex-warehouse image to obtain a second goods image comprises:

and inputting the ex-warehouse image into the image target detection model for target recognition, and extracting a second key feature of the goods in the ex-warehouse image to obtain a second goods image containing the second key feature.

4. The method of claim 3, wherein the key features include at least one of appearance outlines, icons, text, and bar codes of goods in the image; the performing feature matching on the first cargo image and the second cargo image comprises:

5. The method of any of claim 4, further comprising:

6. The method of claim 1, wherein said capturing an outbound image of the cargo upon detecting arrival of the cargo at an outbound dock comprises:

7. A cargo ex-warehouse verification apparatus, comprising:

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor realizes the steps of the method of any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that the computer program realizes the steps of the method of any one of claims 1 to 6 when executed by a processor.