CN110910314B - Splicing method and device for shelf scene images - Google Patents
Splicing method and device for shelf scene images Download PDFInfo
- Publication number
- CN110910314B CN110910314B CN201911182321.7A CN201911182321A CN110910314B CN 110910314 B CN110910314 B CN 110910314B CN 201911182321 A CN201911182321 A CN 201911182321A CN 110910314 B CN110910314 B CN 110910314B
- Authority
- CN
- China
- Prior art keywords
- image
- spliced
- shelf
- images
- splicing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 230000009466 transformation Effects 0.000 claims description 26
- 239000011159 matrix material Substances 0.000 claims description 24
- 238000003702 image correction Methods 0.000 claims description 7
- 238000007499 fusion processing Methods 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 4
- 238000012937 correction Methods 0.000 description 3
- 238000003708 edge detection Methods 0.000 description 3
- 230000000877 morphologic effect Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000007500 overflow downdraw method Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4038—Scaling the whole image or part thereof for image mosaicing, i.e. plane images composed of plane sub-images
Abstract
The invention discloses a method and a device for splicing shelf scene images, wherein the method comprises the following steps: acquiring a plurality of shelf scene images to be spliced; all shelf scene images to be spliced are sequentially arranged to form an image queue, and an overlapping area exists between every two shelf scene images to be spliced; and image splicing is carried out on the shelf scene images to be spliced one by one along the direction from the two sides of the image queue to the middle. By implementing the embodiment of the invention, more image contents can be reserved in the image splicing process.
Description
Technical Field
The invention relates to the technical field of image splicing, in particular to a method and a device for splicing shelf scene images.
Background
In the fast-moving-away field, each large fast-moving-away enterprise needs to statistically analyze a large amount of terminal display data in order to accurately position and measure the excellence of the market. The key premise for effective statistics and analysis of the terminal display data is to acquire shelf images of off-line supermarkets or stores. Currently, the acquisition of shelf images is mainly shot by running a salesman. In a real shooting scene, due to the limited field or the overlong shelf, a salesperson often needs to record complete shelf information by shooting a plurality of images. Therefore, the plurality of shelf images are spliced effectively and quickly, and a complete shelf scene is presented. The existing image splicing technology is to splice all images by always taking the same direction as a splicing reference direction, and when a plurality of images (more than 4 pairs) are spliced by the splicing scheme, partial image content is easily lost. For example: if there are several images to be stitched, the leftmost image is taken as a reference image, and the left image is always taken as a reference direction to be stitched from left to right, an accumulated error is generated, so that the rightmost image is seriously deformed, and the content of the image on the right side is lost after the image is stitched. If more image contents are lost in image splicing in a long shelf scene, serious errors of terminal display data finally counted by a user can be caused directly.
Disclosure of Invention
The embodiment of the invention provides a method and a device for splicing shelf scene images, which can reduce the probability of image content loss and keep more image details during image splicing.
An embodiment of the invention provides a method for splicing shelf scene images, which comprises the following steps:
acquiring a plurality of shelf scene images to be spliced; all shelf scene images to be spliced are sequentially arranged to form an image queue, and an overlapping area exists between every two shelf scene images to be spliced;
and image splicing is carried out on the shelf scene images to be spliced one by one along the direction from the two sides of the image queue to the middle.
Further, the image queues are image queues arranged in the horizontal direction.
Further, along the direction in the middle of the image queue both sides side direction, carry out image stitching with each goods shelves scene image of waiting to splice one by one, specifically include:
when the number of shelf scene images to be spliced in the image queue is an even number, dividing all the shelf scene images to be spliced into a left image set and a right image set according to the arrangement sequence of the shelf scene images to be spliced;
splicing the images of shelf scenes to be spliced in the left image set by taking the left-to-right direction as a splicing reference direction, and obtaining a left spliced image;
splicing the images of the shelf scenes to be spliced in the right image set by taking the direction from right to left as a splicing reference direction to obtain a right spliced image;
performing image stitching on the left stitched image and the right stitched image;
when the number of shelf scene images to be spliced in the image queue is odd, taking the shelf scene image to be spliced in the middle as a reference image according to the arrangement sequence of the shelf scene images to be spliced;
splicing images of shelf scenes to be spliced positioned on the left side of the reference image by taking the left-to-right direction as a splicing reference direction to obtain a left spliced image;
splicing images of shelf scenes to be spliced positioned on the right side of the reference image by taking the direction from right to left as a splicing reference direction to obtain a right spliced image;
and performing image splicing on the left spliced image and the right spliced image and the reference image.
Further, the image stitching specifically includes:
extracting a feature point set of two images to be spliced by using an AKAZE algorithm to obtain a first feature point set and a second feature point set;
performing feature point matching on the first feature point set and the second feature point set through a FLANN algorithm to obtain a plurality of feature point matching pairs;
extracting feature point matching pairs which accord with a preset threshold value from the feature point matching pairs to serve as excellent feature point matching pairs, and then calculating a homography matrix between the two images to be spliced according to the excellent feature point matching pairs;
and carrying out perspective transformation on the two images to be spliced according to the homography matrix, and carrying out fusion processing on the overlapped areas in the two images to be spliced.
Further, before image stitching is performed on the shelf scene images to be stitched one by one along the direction between the two sides of the image queue, the method further comprises the following steps:
detecting a shelf reference line in the shelf scene image to be spliced, and calculating the degree of an included angle between the shelf reference line and a horizontal line;
and if the included angle degree exceeds a preset threshold value, judging that the shelf scene image to be spliced is an oblique shot image, and then carrying out image correction on the shelf scene image to be spliced.
Further, the image correction of the shelf scene image to be spliced specifically includes:
determining vanishing points of the shelf scene images to be spliced according to the shelf reference line;
determining an original reference point and a target reference point of the shelf scene image to be spliced according to the vanishing point and the edge of the shelf scene image to be spliced;
and calculating a perspective transformation matrix according to the original reference points and the target reference points, and then correcting the shelf scene image to be spliced according to the perspective transformation matrix.
Further, before image stitching is performed on the shelf scene images to be stitched one by one along the direction between the two sides of the image queue, extracting coordinate information of the SKU in the shelf scene images to be stitched.
Further, the method also comprises the following steps: according to the homography matrix, carrying out coordinate transformation on the coordinate information of the SKU in the spliced image;
calculating Euclidean distances between each SKU of one image to be spliced and each SKU of the other image to be spliced in the two images to be spliced;
and taking two SKUs with the Euclidean distance larger than a second preset threshold value as repeated SKUs, and removing any SKU in the repeated SKUs.
On the basis of the above method item embodiments, the present invention correspondingly provides apparatus item embodiments;
another embodiment of the invention provides a splicing device for shelf scene images, which comprises an image acquisition module and an image splicing module;
the image acquisition module is used for acquiring a plurality of shelf scene images to be spliced; all shelf scene images to be spliced are sequentially arranged to form an image queue, and an overlapping area exists between every two shelf scene images to be spliced;
and the image splicing module is used for splicing the images of the shelf scenes to be spliced one by one along the direction from two sides of the image queue to the middle.
The embodiment of the invention has the following beneficial effects:
the embodiment of the invention provides a splicing method and a splicing device of shelf scene images, wherein the method comprises the steps of firstly obtaining a plurality of shelf scene images to be spliced, arranging the obtained shelf scene images to be spliced into an image queue in sequence, and forming an overlapped area between two adjacent shelf scene images to be spliced; and then splicing the shelf scene images to be spliced in the image queue one by one along the direction from two sides of the image queue to the middle, namely splicing the images positioned at the outer sides of the image queue one by one from two sides of the image queue along the direction of the center of the image queue from outside to inside, and finally completing the splicing of all the shelf scene images to be spliced in the image queue. Compared with the prior art that images are spliced all the time along the same direction, in the splicing process, the images located in the middle are used as the reference, and the images are simultaneously spliced along the two sides to the middle, so that the image content of the original image can be better reserved, the loss probability of the image content is reduced, and further, the user can more accurately count the number of SKUs (commodities) on a shelf according to the spliced shelf scene images.
Drawings
Fig. 1 is a schematic flow chart of a method for stitching shelf scene images according to an embodiment of the present invention.
Fig. 2 is a schematic diagram illustrating a principle of a method for stitching shelf scene images according to an embodiment of the present invention.
Fig. 3 is a schematic diagram of another principle of a method for stitching shelf scene images according to an embodiment of the present invention.
Fig. 4 is a schematic diagram of a geometric principle of image correction according to an embodiment of the present invention.
Fig. 5 is a schematic structural diagram of a splicing device for shelf scene images according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
As shown in fig. 1, a schematic flow chart of a method for stitching shelf scene images according to an embodiment of the present invention includes:
s101, obtaining a plurality of shelf scene images to be spliced; all shelf scene images to be spliced are sequentially arranged to form an image queue, and an overlapping area exists between every two shelf scene images to be spliced.
And S102, splicing the images of the shelf scene to be spliced one by one along the direction from the two sides of the image queue to the middle.
For the step S101, sequentially acquiring scene images of each shelf to be spliced, which are transmitted by a user; in a preferred embodiment, after a user takes a shelf in a real scene in a sectional manner, the above shelf scene images to be spliced are obtained and are transmitted in the order of taking pictures, so that a plurality of shelf images to be spliced which are arranged in sequence can be obtained.
In a preferred embodiment, the image queues are image queues arranged in a horizontal direction. Namely, the shelf scene images to be spliced in the image queue are arranged from left to right or from right to left.
For step S102, two cases are specifically distinguished:
firstly, when the number of shelf scene images to be spliced in the image queue is an even number, dividing all shelf scene images to be spliced into a left image set and a right image set according to the arrangement sequence of the shelf scene images to be spliced;
splicing the images of shelf scenes to be spliced in the left image set by taking the left-to-right direction as a splicing reference direction, and obtaining a left spliced image;
splicing the images of the shelf scenes to be spliced in the right image set by taking the direction from right to left as a splicing reference direction to obtain a right spliced image;
and carrying out image splicing on the left spliced image and the right spliced image.
Specifically, as shown in fig. 2, when shelf scene images to be stitched are even numbers, 6 images to be stitched, namely img1, img2, img3, img4, img5 and img6, are sequentially arranged in an image queue, at this time, the img1, img2 and img3 are divided into a left image set, the img4 and img5 img6 are divided into a right image set, image stitching is performed on the left image set in a left-to-right manner, that is, image stitching is performed on img1 and img2, and then the stitched image obtained by stitching img1 and img2 is stitched with img3, so that the left stitched image is obtained. And image stitching is carried out on the right image set from right to left, namely image stitching is carried out on the img6 and the img5, and then a stitched image obtained by stitching the img6 and the img5 is stitched with the img4, so that the right stitched image is obtained. And finally, carrying out image splicing on the left spliced image and the right spliced image, and finally realizing the splicing of all shelf scene images to be spliced.
As shown in fig. 3, when the shelf scene images to be stitched are odd, 7 images to be stitched, namely img1, img2, img3, img4, img5, img6 and img7, are arranged in sequence in one image queue, and then the img4 is used as a reference image. And (3) splicing the img1, the img2 and the img3 in the order from left to right, namely, firstly splicing the images of the img1 and the img2, and then splicing the spliced image obtained by splicing the img1 and the img2 with the img3 to obtain the left spliced image. And (3) splicing the img5 and the img6 and the img7 from right to left, namely splicing the images of the img7 and the img6, and splicing the spliced image obtained by splicing the img7 and the img6 with the img5 to obtain the right-side spliced image. And finally, splicing the left spliced image and the right spliced image with img4, and finally splicing all shelf scene images to be spliced.
In other embodiments, of course, the image queue may also be a vertically arranged image queue (corresponding to the case of a shelf vertically placed in an actual scene), and then the splicing principle and the above-mentioned matching are all spliced from two sides of the queue to the middle, and the difference is that the original left-to-right and right-to-left splicing manner is changed to the top-to-bottom and bottom-to-top splicing manner, and the specific principle is the same as the above-mentioned one, and is not described herein again.
In the following, how to splice two images is further described, the current image splicing technology is generally based on SIFT algorithm to extract features, but the SIFT algorithm is not high in real-time; in a preferred embodiment, the two-image stitching method specifically includes:
extracting a feature point set of two images to be spliced by using an AKAZE algorithm to obtain a first feature point set and a second feature point set;
performing feature point matching on the first feature point set and the second feature point set through a FLANN algorithm to obtain a plurality of feature point matching pairs;
extracting feature point matching pairs which accord with a preset threshold value from the feature point matching pairs to serve as excellent feature point matching pairs, and then calculating a homography matrix between the two images to be spliced according to the excellent feature point matching pairs;
and carrying out perspective transformation on the two images to be spliced according to the homography matrix, and carrying out fusion processing on the overlapped areas in the two images to be spliced. When two images are stitched, one image is not transformed, and the other image is transformed according to the homography matrix calculated after the two images are registered. Example (c): and (4) two graphs A (left graph) and B (right graph), wherein if the left graph A is taken as a reference graph, the graph A is not changed, and only B is correspondingly transformed according to the homography matrix and then spliced.
The embodiment of the invention extracts the feature points based on the AKAZE algorithm, the robustness and the calculation efficiency of the algorithm are better than those of the classical feature matching algorithms such as SIFT, SURF and the like, and the condition of excessive mismatching points can be better improved.
In a preferred embodiment, after step S101, before step S102, the method further includes: detecting a shelf reference line in the shelf scene image to be spliced, and calculating the degree of an included angle between the shelf reference line and a horizontal line;
if the included angle degree exceeds a preset threshold value, judging that the shelf scene image to be spliced is an oblique shot image, and then correcting the shelf scene image to be spliced.
In the actual shooting process, because the waiters take pictures irregularly, there can be a large amount of images that incline and warp in the goods shelves scene image of waiting to splice that obtains, directly utilizes these images of shooing irregularity to splice, can lead to the concatenation effect extremely poor. Therefore, in the embodiment of the invention, before splicing the shelf scene images to be spliced, the shelf images need to be subjected to inclination detection and correction. Firstly, a shelf reference line in a shelf scene image to be spliced is detected through Canny edge detection, morphological operation and probability Hough transformation.
Specifically, the method comprises the following steps of,
step 1: detecting edge information in the shelf scene image to be spliced by using a Canny edge detection algorithm;
step 2: detecting a horizontal edge in step1 by using a sobel operator;
step3 carries out morphological operation on the result obtained in Step2 by setting a corresponding kernel template, so that the line segment in the horizontal direction is more prominent;
step4, determining parameters (the minimum line segment length, the maximum distance between line segments and the like) in probability Hough transformation according to the size of the image, and then detecting the straight line of the goods shelf in the image by utilizing the probability Hough transformation;
step5 specifies two shelf reference lines using the dimensional relationship between the position information of the shelf line obtained in Step4 and the image (two shelf reference lines are specified by setting a threshold value according to the image size)
If the shelf scene image to be spliced is not obliquely shot, that is, no oblique deformation exists, the detected shelf reference line and the horizontal line are parallel, and if the shelf scene image to be spliced is obliquely shot, the detected shelf reference line and the horizontal line form an included angle.
In the practical photographing process, the image with the shelf reference line completely parallel to the horizontal line is difficult to photograph, therefore, in the embodiment of the invention, a preset threshold value is set, when the included angle degree between the shelf reference line and the horizontal line exceeds the preset threshold value, the shelf scene image to be spliced is determined to be an oblique image, and then the image correction is carried out on the oblique image. The preset can be adjusted and set by a user according to the splicing effect of the actual image.
In a preferred embodiment, the image correction of the shelf scene image to be stitched specifically includes:
determining vanishing points of the shelf scene images to be spliced according to the shelf reference line;
determining an original reference point and a target reference point of the shelf scene image to be spliced according to the vanishing point and the edge of the shelf scene image to be spliced;
and calculating a perspective transformation matrix according to the original reference points and the target reference points, and then correcting the shelf scene image to be spliced according to the perspective transformation matrix.
As shown in fig. 4, four original reference points (P1, P2, P3, P4) are determined by using vanishing points and edges of the images, then coordinate information of target reference points (P1 ', P2', P3 ', P4') are determined according to the prior assumption that the shelf reference lines are parallel under the normal condition, and finally perspective transformation matrixes are calculated by using the original reference points and the target reference points, and corresponding perspective transformation is performed on the inclined shelf images, so that the correction of the inclined shelf images is realized.
The existing identification of the information of the shelf images SKU is generally to splice a plurality of shelf images and then identify the information of the SKU in the spliced shelf scene. However, the image is affected by deformation in the stitching process, so that the scheme of firstly stitching and then identifying is easy to cause the problem of missing identification of the SKU, and finally the problem of inaccurate statistical data is caused when the number of the SKU is counted.
In order to solve the above problem, in a preferred implementation, before image stitching is performed on the shelf scene images to be stitched one by one along the direction between the two sides of the image queue, the method further includes extracting coordinate information of SKUs in the shelf scene images to be stitched.
Detecting the coordinate information of the SKU before image splicing, and then carrying out coordinate transformation on the coordinate information of the SKU in the spliced images according to a homography matrix when the two images are spliced; calculating Euclidean distances between each SKU of one image to be spliced and each SKU of the other image to be spliced in the two images to be spliced; and taking two SKUs with the Euclidean distance larger than a second preset threshold value as repeated SKUs, and removing any SKU in the repeated SKUs.
When two images are merged, the reference map is not transformed, and the other map is transformed by a homography matrix calculated after the two images are registered. Example (c): if the left graph A is taken as a reference graph, the graph A is not changed, and only the graph B is correspondingly transformed according to the homography matrix and then spliced; therefore, similarly, in the change process of the SKU coordinate information, the SKU coordinate information in fig. a is not subjected to the coordinate transformation, but only the SKU coordinate information in fig. B is subjected to the coordinate transformation. Namely, the coordinate information of the SKU in the stitched image is transformed according to the homography matrix, and all SKU information in one of the images to be stitched is only required to be transformed.
Namely, SKU information is detected before image splicing, and then repeated SKUs are subjected to duplication elimination in the image splicing process. Therefore, the SKU information is detected accurately without any deformation on the original image before image splicing, omission does not occur, and some SKU information is possibly repeated due to the fact that the images to be spliced have overlapping parts, so that the SKU information needs to be removed in the image splicing process, and finally accurate SKU coordinate information of the actual shelf is obtained. It should be noted that reference herein to a SKU is to be understood as a product on a shelf.
In order to better understand the splicing method of shelf scene images disclosed by the invention, the following general description is provided, and the whole method comprises the following steps:
(1) and (3) introducing a plurality of shelf images to be spliced with overlapping areas into the image according to a photographing sequence (from left to right), and detecting shelf reference lines in the images by using Canny edge detection, morphological operation and probability Hough transformation. And then determining a vanishing point according to the detected shelf reference line, and carrying out oblique shooting detection on the transmitted shelf image by using the position information of the vanishing point.
(2) And if the image inclination angle is larger than the preset value, correcting the image inclination angle and splicing the image, otherwise, directly splicing the image. The specific process of correcting the shelf image comprises the following steps: the method comprises the steps of firstly determining four original reference points by utilizing vanishing points and edges of images, then determining coordinate information of a target reference point according to a priori assumption that shelf reference lines are parallel under a normal condition, finally calculating a perspective transformation matrix by utilizing the original reference points and the target reference points, and carrying out corresponding perspective transformation on inclined shelf images, thereby realizing the correction of the inclined shelf images.
(3) Before image splicing is carried out, SKU information identification is carried out on all images to be spliced, the conveyed shelf image to be spliced is divided into two parts (the condition that the number of the images to be spliced is even is listed here), wherein the left half part of the images are spliced by taking the right side as a reference direction, and the right half part of the images are spliced by taking the left side as a reference direction, so that all the images to be spliced are spliced towards the middle.
(4) The method comprises the steps of quickly extracting a feature point set of a shelf image to be spliced by using an AKAZE algorithm, matching feature points by using a FLANN (fast library for Approximate neighbor neighbors) algorithm, and screening excellent feature point matching pairs from the feature point matching pairs by setting a preset distance threshold.
(5) If the number of the excellent feature point matching pairs screened in the step (4) is larger than a preset minimum feature point pair threshold value, the two images to be spliced are considered to have an overlapping area, and the two shelf images are spliced; otherwise, if the two shelf images to be spliced do not have the overlapped area or the overlapped area is too small, the two shelf images are not spliced.
(6) And (4) calculating a homography matrix between the two images to be spliced according to the screened excellent characteristic point matching pairs, and carrying out perspective transformation on the corresponding images to be spliced by using the homography matrix according to the splicing reference direction determined in the step (3).
(7) And performing fusion processing on the overlapped area in the image to be spliced based on a gradual-in and gradual-out fusion method, splicing the image, and optimizing the problem of uneven splicing seams and brightness in a splicing result image.
(8) Performing corresponding coordinate transformation on the coordinate information of all SKUs in the corresponding shelf image by using the homography matrix calculated in the step (6); and calculating Euclidean distances of all the transformed SKUs, determining the SKUs as repeated SKUs when the distance between the two SKUs is larger than a set threshold value, and otherwise, not determining the SKUs as the repeated SKUs, and further performing SKU deduplication.
And finally, counting the number of the spliced shelf scene images.
Correspondingly, an embodiment of the apparatus item is provided on the basis of the embodiment of the method item;
as shown in fig. 5, another embodiment of the present invention provides a splicing apparatus for shelf scene images, including an image obtaining module and an image splicing module;
the image acquisition module is used for acquiring a plurality of shelf scene images to be spliced; all shelf scene images to be spliced are sequentially arranged to form an image queue, and an overlapping area exists between every two shelf scene images to be spliced;
and the image splicing module is used for splicing the images of the shelf scenes to be spliced one by one along the direction from two sides of the image queue to the middle.
It is understood that the above embodiment of the apparatus corresponds to an embodiment of the method of the present invention, and the splicing apparatus based on shelf scene images provided by any one of the above embodiments of the method of the present invention can be implemented.
It should be noted that the above-described device embodiments are merely illustrative, where the units/modules described as separate parts may or may not be physically separate, and the parts displayed as units/modules may or may not be physical units/modules, may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. In addition, in the drawings of the embodiment of the apparatus provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, and may be specifically implemented as one or more communication buses or signal lines. One of ordinary skill in the art can understand and implement it without inventive effort. The schematic diagram is merely an example of a splicing apparatus/terminal device based on shelf scene images, and does not constitute a limitation of the splicing apparatus/terminal device based on shelf scene images, and may include more or less components than those shown, or combine some components, or different components.
The embodiment of the invention has the following effects:
1. when a plurality of (more than 4) shelf images are spliced, especially when a deflection angle exists in the image shooting process, the method can better keep the content information in all the original images to be spliced, the splicing effect is better than that of the current algorithm which always splices in one fixed direction as the splicing reference direction, and the method is favorable for counting the number of the SKUs in the later period.
2. The method extracts the feature points based on the AKAZE algorithm, considers the robustness and the calculation efficiency of the algorithm comprehensively, and the AKAZE algorithm is superior to the SIFT, SURF and other classical feature matching algorithms, so that the condition of excessive mismatching points can be better improved, and the operation efficiency of the algorithm is higher. According to the invention, the total time of splicing and de-duplication of the two graphs (600 x 800) is about 0.5s on average, and the splicing result after de-duplication can be obtained in real time;
3. the invention can accurately count the number of the SKU, and effectively avoids the problem of missing identification in the SKU number counting process in the prior art by identifying and splicing.
While the foregoing is directed to the preferred embodiment of the present invention, it will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the invention.
Claims (8)
1. A splicing method of shelf scene images is characterized by comprising the following steps:
obtaining at least 4 shelf scene images to be spliced; all shelf scene images to be spliced are sequentially arranged to form an image queue, and an overlapping area exists between every two shelf scene images to be spliced;
when the number of shelf scene images to be spliced in the image queue is an even number, dividing all the shelf scene images to be spliced into a left image set and a right image set according to the arrangement sequence of the shelf scene images to be spliced; splicing the images of shelf scenes to be spliced in the left image set by taking the left-to-right direction as a splicing reference direction, and obtaining a left spliced image; splicing the images of the shelf scenes to be spliced in the right image set by taking the direction from right to left as a splicing reference direction to obtain a right spliced image; performing image stitching on the left stitched image and the right stitched image;
when the number of shelf scene images to be spliced in the image queue is odd, taking the shelf scene image to be spliced in the middle as a reference image according to the arrangement sequence of the shelf scene images to be spliced; splicing images of shelf scenes to be spliced positioned on the left side of the reference image by taking the left-to-right direction as a splicing reference direction to obtain a left spliced image; splicing images of shelf scenes to be spliced positioned on the right side of the reference image by taking the direction from right to left as a splicing reference direction to obtain a right spliced image; performing image splicing on the left spliced image and the right spliced image and the reference image;
when image splicing is carried out, another image to be spliced is spliced after perspective transformation is carried out on the image to be spliced by taking one image to be spliced as a reference.
2. The method for stitching shelf scene images according to claim 1, wherein the image queue is an image queue arranged in a horizontal direction.
3. The shelf scene image stitching method according to claim 1, wherein the image stitching specifically includes:
extracting a feature point set of two images to be spliced by using an AKAZE algorithm to obtain a first feature point set and a second feature point set;
performing feature point matching on the first feature point set and the second feature point set through a FLANN algorithm to obtain a plurality of feature point matching pairs;
extracting feature point matching pairs which accord with a preset threshold value from the feature point matching pairs to serve as excellent feature point matching pairs, and then calculating a homography matrix between the two images to be spliced according to the excellent feature point matching pairs;
and carrying out perspective transformation on the two images to be spliced according to the homography matrix, and carrying out fusion processing on the overlapped areas in the two images to be spliced.
4. The shelf scene image stitching method according to claim 1, further comprising, before the image stitching:
detecting a shelf reference line in the shelf scene image to be spliced, and calculating an included angle between the shelf reference line and a horizontal line;
and if the included angle degree exceeds a preset threshold value, judging that the shelf scene image to be spliced is an oblique shot image, and then carrying out image correction on the shelf scene image to be spliced.
5. The shelf scene image stitching method according to claim 4, wherein the image correction of the shelf scene image to be stitched specifically includes:
determining vanishing points of the shelf scene images to be spliced according to the shelf reference line;
determining an original reference point and a target reference point of the shelf scene image to be spliced according to the vanishing point and the edge of the shelf scene image to be spliced;
and calculating a perspective transformation matrix according to the original reference points and the target reference points, and then correcting the shelf scene image to be spliced according to the perspective transformation matrix.
6. The shelf scene image stitching method according to claim 3, further comprising, before the image stitching, extracting coordinate information of the SKU in each shelf scene image to be stitched.
7. The method for stitching shelf scene images according to claim 6, further comprising: carrying out coordinate transformation on the coordinate information of the SKU in the spliced image according to the homography matrix;
calculating Euclidean distances between each SKU of one image to be spliced and each SKU of the other image to be spliced in the two images to be spliced;
and taking two SKUs with the Euclidean distance larger than a second preset threshold value as repeated SKUs, and removing any SKU in the repeated SKUs.
8. A splicing device for shelf scene images is characterized by comprising: the system comprises an image acquisition module and an image splicing module;
the image acquisition module is used for acquiring at least 4 shelf scene images to be spliced; all shelf scene images to be spliced are sequentially arranged to form an image queue, and an overlapping area exists between every two shelf scene images to be spliced;
the image splicing module is used for dividing all shelf scene images to be spliced into a left image set and a right image set according to the arrangement sequence of the shelf scene images to be spliced when the number of the shelf scene images to be spliced in the image queue is an even number; splicing the images of shelf scenes to be spliced in the left image set by taking the left-to-right direction as a splicing reference direction, and obtaining a left spliced image; splicing the images of the shelf scenes to be spliced in the right image set by taking the direction from right to left as a splicing reference direction to obtain a right spliced image; performing image stitching on the left stitched image and the right stitched image;
when the number of shelf scene images to be spliced in the image queue is odd, taking the shelf scene image to be spliced in the middle as a reference image according to the arrangement sequence of the shelf scene images to be spliced; splicing images of shelf scenes to be spliced positioned on the left side of the reference image by taking the left-to-right direction as a splicing reference direction to obtain a left spliced image; splicing images of shelf scenes to be spliced positioned on the right side of the reference image by taking the direction from right to left as a splicing reference direction to obtain a right spliced image; performing image splicing on the left spliced image and the right spliced image and the reference image;
when image splicing is carried out, another image to be spliced is spliced after perspective transformation is carried out on the image to be spliced by taking one image to be spliced as a reference.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911182321.7A CN110910314B (en) | 2019-11-27 | 2019-11-27 | Splicing method and device for shelf scene images |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911182321.7A CN110910314B (en) | 2019-11-27 | 2019-11-27 | Splicing method and device for shelf scene images |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110910314A CN110910314A (en) | 2020-03-24 |
| CN110910314B true CN110910314B (en) | 2020-09-25 |
Family
ID=69818586
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911182321.7A Active CN110910314B (en) | 2019-11-27 | 2019-11-27 | Splicing method and device for shelf scene images |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110910314B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112188163B (en) * | 2020-09-29 | 2022-09-13 | 厦门汇利伟业科技有限公司 | Method and system for automatic de-duplication splicing of real-time video images |
| CN112037267B (en) * | 2020-11-06 | 2021-02-02 | 广州市玄武无线科技股份有限公司 | Method for generating panoramic graph of commodity placement position based on video target tracking |
| CN113592980B (en) * | 2021-06-29 | 2023-07-25 | 北京百度网讯科技有限公司 | Signboard topological relation construction method and device, electronic equipment and storage medium |
| TWI784605B (en) * | 2021-06-29 | 2022-11-21 | 倍利科技股份有限公司 | Image stitching method |
| CN115578441B (en) * | 2022-08-30 | 2023-07-28 | 感知信息科技(浙江)有限责任公司 | Vehicle side image stitching and vehicle size measuring method based on deep learning |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10547825B2 (en) * | 2014-09-22 | 2020-01-28 | Samsung Electronics Company, Ltd. | Transmission of three-dimensional video |
| CN107172361B (en) * | 2017-07-12 | 2019-11-15 | 维沃移动通信有限公司 | A kind of method and mobile terminal of pan-shot |
| CN108038826B (en) * | 2017-12-28 | 2021-12-07 | 上海小零网络科技有限公司 | Method and device for correcting perspective deformed shelf image |
| CN108648140B (en) * | 2018-04-12 | 2022-09-06 | 上海扩博智能技术有限公司 | Image splicing method, system, equipment and storage medium |
| CN110119675B (en) * | 2019-03-28 | 2021-07-27 | 广州广电运通金融电子股份有限公司 | Product identification method and device |
-
2019
- 2019-11-27 CN CN201911182321.7A patent/CN110910314B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN110910314A (en) | 2020-03-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110910314B (en) | Splicing method and device for shelf scene images | |
| KR101121034B1 (en) | System and method for obtaining camera parameters from multiple images and computer program products thereof | |
| EP3563345B1 (en) | Automatic detection, counting, and measurement of lumber boards using a handheld device | |
| US8699786B2 (en) | 3D model generating apparatus, method and CRM by line pattern imaging | |
| Zhang et al. | Robust metric reconstruction from challenging video sequences | |
| CN114119751A (en) | Method and system for large scale determination of RGBD camera poses | |
| CN105304002B (en) | The detection method and device of LED display stitching error | |
| Fan et al. | Shelf detection via vanishing point and radial projection | |
| US20200334650A1 (en) | Self-service checkout counter checkout | |
| JP2008064628A (en) | Object detector and detecting method | |
| CN113869343A (en) | Occluded item detection for vision-based checkout | |
| US11216905B2 (en) | Automatic detection, counting, and measurement of lumber boards using a handheld device | |
| Wanchaitanawong et al. | Multi-modal pedestrian detection with large misalignment based on modal-wise regression and multi-modal IoU | |
| CN115272446A (en) | Method and system for calculating head-stacking occupied area | |
| CN110120012B (en) | Video stitching method for synchronous key frame extraction based on binocular camera | |
| CN111462164A (en) | Foreground segmentation method and data enhancement method based on image synthesis | |
| CN109784227B (en) | image detection and identification method and device | |
| CN111160450A (en) | Fruit and vegetable weighing method based on neural network, storage medium and device | |
| JP6456244B2 (en) | Camera calibration method and apparatus | |
| CN115830604A (en) | Surface single image correction method, device, electronic apparatus, and readable storage medium | |
| CN112614176A (en) | Belt conveyor material volume measuring method and device and storage medium | |
| CN111783528A (en) | Method, computer and system for monitoring items on a shelf | |
| CN113840135B (en) | Color cast detection method, device, equipment and storage medium | |
| CN115619636A (en) | Image stitching method, electronic device and storage medium | |
| CN115205807A (en) | Lane line fusion method and device, automobile, electronic equipment and computer-readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: A method and device for splicing shelf scene image Effective date of registration: 20210308 Granted publication date: 20200925 Pledgee: Shanghai Pudong Development Bank Limited by Share Ltd. Guangzhou branch Pledgor: GUANGZHOU XUANWU WIRELESS TECHNOLOGY Co.,Ltd. Registration number: Y2021440000062 |
|
| CP02 | Change in the address of a patent holder |
Address after: 510000 room 23bd, No. 109, TIYU West Road, Tianhe District, Guangzhou City, Guangdong Province Patentee after: GUANGZHOU XUANWU WIRELESS TECHNOLOGY Co.,Ltd. Address before: 32B, no.103b, TianLiHe Road, Guangzhou, 510000 Patentee before: GUANGZHOU XUANWU WIRELESS TECHNOLOGY Co.,Ltd. |
|
| CP02 | Change in the address of a patent holder | ||
| PC01 | Cancellation of the registration of the contract for pledge of patent right | ||
| PC01 | Cancellation of the registration of the contract for pledge of patent right |
Date of cancellation: 20220525 Granted publication date: 20200925 Pledgee: Shanghai Pudong Development Bank Limited by Share Ltd. Guangzhou branch Pledgor: GUANGZHOU XUANWU WIRELESS TECHNOLOGY Co.,Ltd. Registration number: Y2021440000062 |
|
| PE01 | Entry into force of the registration of the contract for pledge of patent right | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: A Method and Device for Splicing Shelf Scene Images Effective date of registration: 20230331 Granted publication date: 20200925 Pledgee: Shanghai Pudong Development Bank Limited by Share Ltd. Guangzhou branch Pledgor: GUANGZHOU XUANWU WIRELESS TECHNOLOGY Co.,Ltd. Registration number: Y2023980037128 |
|
| PC01 | Cancellation of the registration of the contract for pledge of patent right |
Granted publication date: 20200925 Pledgee: Shanghai Pudong Development Bank Limited by Share Ltd. Guangzhou branch Pledgor: GUANGZHOU XUANWU WIRELESS TECHNOLOGY Co.,Ltd. Registration number: Y2023980037128 |