CN115601271B - Visual information anti-shake method, storage warehouse location state management method and system - Google Patents
Visual information anti-shake method, storage warehouse location state management method and system Download PDFInfo
- Publication number
- CN115601271B CN115601271B CN202211504539.1A CN202211504539A CN115601271B CN 115601271 B CN115601271 B CN 115601271B CN 202211504539 A CN202211504539 A CN 202211504539A CN 115601271 B CN115601271 B CN 115601271B
- Authority
- CN
- China
- Prior art keywords
- target object
- motion field
- moment
- field quantity
- state
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a visual information anti-shaking method, a warehouse location state management method and a warehouse location state management system, wherein the visual information anti-shaking method comprises the following steps: step S1: collecting scene image frames at each preset moment; identifying a target object in a scene image frame to acquire position and size information of the target object, establishing an enclosing frame and calculating a tangential vector of the target object so as to calculate the motion field amount of the target object corresponding to each preset moment; step S2: filtering and calculating the motion field quantity of a certain moment and the previous adjacent moment to obtain the final motion field quantity of the certain moment and the predicted motion field quantity of the future adjacent moment; and step S3: and fitting the motion field quantity obtained in the step S2 into a smooth piecewise curve equation through a curve so as to obtain the motion field quantity of the target object at any moment, thereby solving the problem of time delay jitter of visual information.
Description
Technical Field
The invention relates to a machine vision technology, in particular to a method for managing warehouse location states by adopting a target tracking technology and a management system thereof.
Background
With the rapid development of AGV/AMR technology and the logistics, warehousing and industrial industries, the need for intelligent logistics and warehouse management by the factories has increased significantly. During the early years, the main methods/techniques for dealing with library bit management include two types: manual recording and photoelectric sensor detection. The manual recording method is not only easy to make mistakes, but also not easy to enlarge the production scale (low efficiency), and is a huge challenge for reducing the cost, and the technology for detecting the cargo state on the warehouse by using the sensor is developed, so that the rapid development of the warehouse management application scene is greatly assisted due to the characteristics of high efficiency, rapidness and automation.
However, with the rapid development of machine vision technology, the disadvantages of the detection by the photoelectric sensor also appear: for example, the construction amount is large, the maintenance cost is high, the information amount is single, and the configuration is rigid. Compared with tens, dozens or even hundreds or thousands of photoelectric sensors, the method has the advantages of simple construction and simple maintenance but is more flexible.
Therefore, the method using the machine vision + deep learning means becomes a method and research hotspot for intelligent library bit detection. However, due to the inherent characteristics (100% accuracy cannot be guaranteed), when the technology is applied to detection, the problems of instability (errors) and jitter (sudden change) of the library position state can be encountered, so that information provided for the AGV/AMR task is wrong, and finally, the carrying task cannot be completed, even more serious article damage, traffic accidents and the like can be caused.
The underlying reason is that visual "real-time" detection is also discretized, which is limited to work at a fixed frame rate and therefore involves less delay jitter.
Disclosure of Invention
Therefore, the invention mainly aims to provide a visual information anti-shaking method, a warehouse location state management method and a warehouse location state management system, so as to solve the problem of time delay shaking of visual information.
In order to achieve the above object, according to a first aspect of the present invention, there is provided a visual information anti-shake method, comprising the steps of:
step S1: collecting scene image frames at each preset moment; identifying a target object in a scene image frame to acquire position and size information of the target object, establishing an enclosing frame and calculating a tangential vector of the target object so as to calculate the motion field amount of the target object corresponding to each preset moment;
step S2: filtering and calculating the motion field quantity of a certain moment and the previous adjacent moment to obtain the final motion field quantity of the certain moment and the predicted motion field quantity of the future adjacent moment;
and step S3: and fitting the motion field quantity obtained in the step S2 into a smooth piecewise curve equation through a curve so as to obtain the motion field quantity of the target object at any moment.
In order to achieve the above object and establish a stable state identification relationship between visual information and a bin state, there is also provided according to a second aspect of the present invention a bin state management method including the steps of:
step B1: collecting scene image frames at each preset moment; extracting a library position area and corner point coordinates of each library position in a scene image frame, identifying a target object in the scene image frame, acquiring position and size information of the target object, establishing a surrounding frame, calculating a tangential vector of the target object, and calculating motion field quantity of the target object corresponding to each preset moment;
and step B2: filtering and calculating the motion field quantity of a certain moment and the adjacent moment before the certain moment to obtain the final motion field quantity of the certain moment and the predicted motion field quantity of the near moment in the future;
and step B3: fitting the motion field quantity obtained in the step B2 into a smooth piecewise curve equation through a curve so as to obtain the motion field quantity of the target object at any moment;
and step B4: acquiring an enclosing frame corresponding to the motion field quantity of the target object at a certain moment according to the step B3, and respectively establishing an enveloping angle between the enclosing frame and all the library positions so as to judge the included angle relation formed by the enveloping line of each library position and the tangential vector of the target object;
and step B5: and D, judging the state of the storage position according to the relation between the storage position inventory information and the included angle obtained in the step B4.
In a possible preferred embodiment, the step B1 further includes, when the acquired image frame of the scene has distortion, correcting the bin region and the corner coordinates of each bin according to the internal parameters and distortion parameters of the camera.
In a possible preferred embodiment, in the step B1, the position information of the target object k at the time t is identified by using a target tracking network scheme
And size information S t To establish an enclosure frame (` H `)>
，S t ) And calculates the corresponding tangential vector pick>
To obtain the motion field quantity omega of the target object k at the time t k t Is prepared from (a)
) k 。
In a possible preferred embodiment, said filtering calculation in said step B2 employs a kalman filtering algorithm.
In a possible preferred embodiment, in the step B3, after at least 5 continuous motion field quantities are obtained through calculation in the step B2, a smooth piecewise curve equation is fitted through a third-order bezier curve, so as to obtain the motion field quantity of the target object at any time.
In a possible preferred embodiment, the step of establishing the envelope angle comprises: and respectively radiating lines from the central point of the target object surrounding frame to the angular point of each library position, selecting the maximum included angle formed by two radiation lines as an envelope angle, and taking the two radiation lines of the envelope angle as an envelope line.
In a possible preferred embodiment, in said step B5The judgment step of the included angle relation comprises the following steps: according to the included angle relation between the tangential vector of the target object k and the left and right envelope lines of each library position
、/>
In the following positional relationship>
Judging:
In a possible preferred embodiment, in the step B5, the step of determining the library bit state includes:
if there is goods in the storage location, and
is 0 and no other target object forms therewith>
Within the positive envelope angle, the bin position state is bin full;
if there is goods in the storage location, and
in the positive envelope angle, the storage position state is full;
if there is no cargo in the storage space, and
if the current position is within the positive envelope angle, the storage position state is a storage state;
if there is no cargo in the storage space, and
in the negative envelope angle, the storage position state is ex-storage;
if there is no cargo in the storage space, and
is 0 and no other cargo is formed therewith>
Within the envelope angle, the bin state is bin empty.
In order to achieve the above object and establish a stable state recognition relationship between visual information and a bin state, there is also provided according to a third aspect of the present invention a bin state management system comprising:
the storage unit is used for storing the program of any one of the warehouse location state management method steps so as to be used for the control unit, the identification unit, the processing unit and the information output unit to be called and executed in time;
wherein the control unit is configured to coordinate:
the camera is used for acquiring scene image frames at preset moments;
the identification unit is used for extracting the bin region and the corner coordinates of each bin from the scene image frame, and identifying the target object in the bin region to acquire the position and size information of the target object and establish a surrounding frame;
the processing unit is used for calculating a tangential vector of the target object according to the bounding box data so as to further calculate the motion field quantity of the target object at each preset moment; then, filtering calculation is carried out on the motion field quantity of a certain moment and the adjacent moment in front of the certain moment to obtain the final motion field quantity of the certain moment and the predicted motion field quantity of the near moment in the future, a smooth piecewise curve equation is fitted with a curve, and the motion field quantity of the target object at any moment and a corresponding surrounding frame are obtained to establish an envelope angle with all the library positions respectively;
the storage unit stores the judgment basis of the included angle relationship between the bin envelope curve and the tangential vector of the target object and the judgment basis of the corresponding bin state between the included angle relationship and the bin stock information; the processing unit calls the judgment basis of the storage unit, judges the state of the ex-warehouse bit and updates the state;
and the information output unit is used for transmitting the library position state information to the robot scheduling system.
According to the visual information anti-shake method, the warehouse location state management method and the system, a sub-pixel level motion model is theoretically established, continuous motion of goods is described through motion field quantity of a target object, a segmented multi-order curve is adopted for fitting and smoothing a goods path, path stability is achieved through a filtering algorithm, a continuous motion field of the sub-pixel level object is estimated through a fitting curve, the motion field quantity of the target object is accurate to the sub-pixel level, and therefore the problem of time delay shake of visual information is effectively solved.
On the other hand, in the related implementation scheme of the warehouse location state management method and system, a scheme capable of fully utilizing rich information provided by vision to establish a stable state relation between the position and the speed of the goods and the warehouse location is further established, so that the stability of the warehouse location state detection result is ensured, and non-contact type warehouse location detection is formed, thereby realizing a reliable warehouse location management function, avoiding the large-scale arrangement of sensors on site, and reducing the construction and maintenance cost for establishing warehouse location management.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments of the invention and, together with the description, serve to explain the invention and not to limit the invention. In the drawings:
FIG. 1 is a flowchart illustrating a method for anti-shaking visual information according to a first embodiment of the present invention;
FIG. 2 is a schematic diagram of a smooth piecewise curve fit between the motion field amount and a curve in the visual information anti-shaking method according to the first embodiment of the present invention;
FIG. 3 is a flowchart illustrating a visual information anti-shaking method according to a second embodiment of the present invention;
FIG. 4 is a schematic diagram illustrating a basic flow chart of a warehouse location status management method according to a second embodiment of the present invention;
fig. 5 is a schematic diagram illustrating an envelope angle and an envelope curve formed between a target object and a warehouse location in a warehouse location state management method according to a second embodiment of the present invention;
FIG. 6 is a schematic diagram illustrating a logical relationship between a position, a velocity vector and a storage location state of a target object in the method for managing a storage location state according to the second embodiment of the present invention;
fig. 7 is a schematic structural diagram of a warehouse location state management system according to a third embodiment of the present invention.
Detailed Description
In order to make those skilled in the art better understand the technical solution of the present invention, the following will clearly and completely describe the specific technical solution of the present invention with reference to the embodiments to help those skilled in the art to further understand the present invention. It should be apparent that the embodiments described herein are only a few embodiments of the present invention, and not all embodiments. It should be noted that the embodiments and features of the embodiments in the present application can be combined with each other without departing from the inventive concept and without conflicting therewith by those skilled in the art. All other embodiments based on the embodiments of the present invention, which can be obtained by a person of ordinary skill in the art without any creative effort, shall fall within the disclosure and the protection scope of the present invention.
Furthermore, the terms "first," "second," "S1," "S2," "B1," "B2," and the like in the description and claims of the present invention and in the drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those described herein. Also, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. Unless expressly stated or limited otherwise, the terms "disposed," "mounted," "connected," and "connected" are to be construed broadly and may, for example, be fixedly connected, detachably connected, or integrally connected; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meaning of the above terms in this case can be understood by those skilled in the art in combination with the prior art according to the specific situation.
Currently, a viewing device such as a camera is known as a visual acquisition device for machine vision, which usually performs image frame acquisition at a fixed frequency (or period), for example, once in 50ms, during the process of acquiring an image. The acquisition of the image frames is therefore scattered (discrete) in time points, so that only discrete point approximations can be used in order to achieve continuity of detection. However, the detection result is "jumpy" (jitter) "as a matter of course.
For this reason, when the library bit management is established by adopting the machine vision technology, the problem of time delay jitter exists inevitably in the visual detection information.
Referring to fig. 1 to 2, to solve the problem, a first aspect of the present invention provides a method for preventing visual information from shaking, comprising:
step S1: collecting scene image frames at each preset moment; and identifying the target object in the scene image frame to acquire the position and size information of the target object, establishing a surrounding frame and calculating a tangential vector of the target object so as to calculate the motion field amount of the target object corresponding to each preset moment.
Step S2: and filtering and calculating the motion field quantity of a certain moment and the previous adjacent moment to obtain the final motion field quantity of the certain moment and the predicted motion field quantity of the future adjacent moment.
And step S3: and fitting the motion field quantity obtained in the step S2 into a smooth piecewise curve equation through a curve so as to obtain the motion field quantity of the target object at any moment.
In particular toIn step S1, the present example assumes that scene image frames are acquired at time t1, t2, and t3, respectively, and the target tracking network (DeepSort) is used to identify the position and size information of the target object to obtain an object enclosure frame (b:)
，S t1 ) 1 , (/>
，S t1 ) 2 …(/>
，S t1 ) m ; (/>
，S t2 ) 1 , (/>
，S t2 ) 2 …(/>
，S t2 ) m ; (/>
，S t3 ) 1 , (/>
，S t3 ) 2 …(/>
，S t3 ) m (ii) a Wherein->
Is (x) t1 , y t1 ) I.e. at t 1 Pixel coordinates of a moment; s t1 Is (W, H), W is the width of the target object bounding box, H is the height of the target object bounding boxDegrees (the remaining bounding boxes are to be understood).
Then, the tangential vectors of the target objects at different moments, such as t, are calculated according to the obtained bounding box data 2 Time:
and the analogy is repeated, so that the measurement motion field omega of the kth target object at each moment is obtained k t ：(/>
) k . Wherein the amount of motion field omega k t And the vector representation is used for describing the position and the speed of the object in the image, wherein the position and the speed are both referred to the image coordinate system, and the pixel coordinate of the image is taken as a reference unit. When used as a real-time description variable, the right subscript t denotes the time t, and the right superscript k denotes the kth object, so that the entire notation can be understood as the amount of motion field of the kth object in the image frame at the time t.
Further, since the sampling period of the camera is discrete in the actual scene (usually, a frame is acquired every 50ms at a fixed frame rate, for example, 20 fps), there is a problem of instability (due to the discrete amount and instability of detection) when the object position is actually calculated, and therefore, it is necessary to apply a suitable filtering algorithm to filter and predict the object position.
For this purpose, in step S2, the present example employs a kalman filter algorithm. Predicting omega from t-1 time kp t ：(
) kp And the measured value omega at time t ks t ：(/>
) ks Obtaining the final motion field quantity omega at the time t by using a Kalman filtering algorithm k t ：(/>
) k And predicted value omega at time t +1 kp t+1 ：(/>
) kp The Kalman algorithm flow is as follows.
1. Calculating the equation of state transition
Based on the system having no control input, therefore->
The state transition equation is simplified to
Wherein the state variable is
Including the amount of motion field omega k t In addition, there is an acceleration variable->
And &>
；
2. Computational covariance transfer equation
，
Wherein
Wherein->
As variance of acceleration
3. Calculating a measurement equation
Since the measured quantity is directly the position of the object, accordingly->
Thus, therefore, it is
4. Calculating a measurement variance
Assuming that the x and y coordinates are independent of each other, then
In addition, for simplicity, the measurement error is assumed to be a stable quantity, and therefore
5. Computing kalman gain
6. To sum up, the state is updated with a known amount
Calculated in each subsequent time interval
I.e. the state variables at this moment, including position, velocity and acceleration (i.e. position and velocity constitute the motion field quantity omega) k t ) And acquiring the final motion field quantity of each moment and the predicted motion field quantity of the moment near to the future moment.
Further, as shown in fig. 2, in order to continue the motion field amount of the discrete target objects, in step S3, the motion field amount Ω of the kth target object at different time is illustrated t1 ，Ω t2 ，Ω t3 …Ω tn Fitting to a smooth piecewise curve equation through a multi-segment 3 rd order curve, such as solving the equation according to the following solution conditions:
the matrix equation, in its general form,
wherein n represents the nth curve (nth segment)
Solving the parameter space of each set of curves
Wherein
。
Then any time can be calculated
Is determined (is composed of a position vector and a tangential vector). As can be seen from the foregoing, a decision is made as to whether or not a decision is made to take a decision>
Represents the nth piecewise curve equation with a time range exemplified by [>
]The specific example is divided into 5 segments: [/>
]，[/>
]，[/>
]And [. Sup.>
]. According to a detection sequence>
、/>
、
…/>
At any time->
Is concerned with>
The interval is segmented, and the x coordinate of the moment is calculated by linear interpolation in the corresponding interval (the assumed moment belongs to [ -based on the judgment result of the judgment unit [ + ]>
]Interval):
From this, a position vector is obtained
And tangential vector
Therefore, by the above example, the continuous motion of the goods is described by adopting the motion field quantity of the target object, the path is stabilized by adopting the piecewise multi-order curve fitting and smoothing the goods path, and the continuous motion field of the object at the sub-pixel level is estimated by the fitting curve, so that the motion field quantity of the target object is accurate to the sub-pixel level, and the problem of time delay jitter of visual information is effectively solved.
In order to further establish a stable state identification relationship between the visual information and the bin position state and realize the bin position management function based on the implementation of the first object of the embodiment, as shown in fig. 3 to 6, according to a second aspect of the present invention, there is provided a bin position state management method, comprising the steps of:
step B1: collecting scene image frames at each preset moment; extracting a library position area and angular point coordinates of each library position in a scene image frame, identifying a target object in the scene image frame, acquiring position and size information of the target object, establishing a surrounding frame, calculating a tangential vector of the target object, and calculating motion field quantities of the target object at each preset moment;
and step B2: filtering and calculating the motion field quantity of a certain moment and the adjacent moment before the certain moment to obtain the final motion field quantity of the certain moment and the predicted motion field quantity of the near moment in the future;
and step B3: fitting the motion field quantity obtained in the step B2 into a smooth piecewise curve equation through a curve so as to obtain the motion field quantity of the target object at any moment;
and step B4: acquiring an enclosing frame corresponding to the motion field quantity of the target object at a certain moment according to the step B3, and respectively establishing an enveloping angle between the enclosing frame and all the library positions so as to judge the included angle relation formed by the enveloping line of each library position and the tangential vector of the target object;
and step B5: and D, judging the state of the storage according to the relation between the storage inventory information and the included angle acquired in the step B4.
Specifically, in step B1, the present example uses an image segmentation (color + shape segmentation) algorithm from a scene image frame T image The library position rectangle is extracted, and as shown in fig. 5 to 6, the library position rectangle comprises 4 corner points LC01-LC04 of the library position area rectangle and 4 corner points BC01-BC [ N ] of the library position]And N is an integer multiple of 4.
In addition, there is distortion due to problems in the camera manufacturing process. The distortion affects the true pixel coordinates of the target object and therefore needs to be corrected. As in the alternative embodiment, LC01-LC04 and N BC01-BC [ N ] corner points are corrected by monitoring the internal parameters and distortion parameters of the camera to obtain LC01'-LC04' and N BC01'-BC [ N ]', and the LC01'-LC04' and the N BC01'-BC [ N ]', and are stored in the system. The process of correcting the LC01 pixel coordinates (x, y) yields LC01':
distortion correction
Transformation of internal parameters
Wherein, (x, y) is the coordinate of the pixel point in the original image; (
) And (& lt & gt>
) Respectively representing radial and tangential distortion parameters; />
Representing pixel coordinates after distortion correction; (/>
) Representing internal parameters of the camera; (u, v) represents the pixel coordinates after final rectification, i.e. LC01' (the remaining corner points can be rectified by analogy).
Then, referring to the implementation of the exemplary embodiment of steps B1 to B3 in the first embodiment, to complete the calculation and prediction of the motion field amount at different time instants, to illustrate that the motion field amount at 5 time points and the multi-segment 3-order curve are fitted into a smooth piecewise curve equation, so as to obtain the equation for calculating any time instant
The motion field magnitude formula of (c).
And then, in order to establish a stable state relation scheme between the position, the speed and the storage position of the target object, and further ensure the stability of the detection result of the storage position state so as to form non-contact storage position state detection.
In step B4 of the present example, it is preferable to determine the actual position of the kth target object
:(/>
) Calculating the envelope angle formed by the kth target object corresponding to all the library bits at the time i (any time)>
，/>
，/>
…/>
Where m represents the total number of bin bits in a single frame image, and the envelope.
The calculation method is shown in fig. 5, the envelope angle is the largest angle of all radiation angles, the envelope curve corresponds to two sides of the envelope angle, and is a straight line (non-ray), and the two envelope curves are straight lines
And &>
Envelope angle is->
。
Further, in the step B5, the step of judging the relationship between the included angles includes: according to the included angle relation between the tangential vector of the target object k and the left and right envelope lines of each library position
、/>
In a positional relationship>
Judging:
is negative in the number of the positive lines,
is positive, then>
Within the envelope angle and in the opposite direction, i.e. within the negative envelope angle;
Then, according to the position relation between the kth target object and the library position
Jointly judge the status of the bank bit>
(as shown in FIG. 6): the step of judging the state of the storage position comprises the following steps:
if there is goods in the storage location, and
is 0 and no other target object forms therewith>
Within the positive envelope angle, the bin position state is bin full;
if there is goods in the storage location, and
in the positive envelope angle, the storage position state is full;
if there is no cargo in the storage space, and
if the current position is within the positive envelope angle, the storage position state is a storage state;
if there is no cargo in the storage space, and
in the negative envelope angle, the storage position state is ex-storage;
if there is no cargo in the storage space, and
is 0 and no other cargo is formed therewith>
Within the envelope angle, the bin state is bin empty.
Finally according to
The state of each bin bit is updated.
Therefore, the stable state relation among the position, the speed and the storage position of the target object goods is established, the stability of the detection result of the storage position state is further ensured, and the reliable storage position management function is realized.
As shown in fig. 7, according to a third aspect of the present invention, there is provided a storage space status management system, including:
and the storage unit is used for storing the program of the warehouse location state management method steps in the second embodiment, so that the control unit, the identification unit, the processing unit and the information output unit can be called and executed timely.
Wherein the control unit is configured to coordinate:
the camera is used for acquiring scene image frames at preset moments;
the identification unit is used for extracting the bin region and the corner coordinates of each bin from the scene image frame, and identifying the target object in the bin region to acquire the position and size information of the target object and establish a surrounding frame;
the processing unit is used for calculating a tangential vector of the target object according to the bounding box data so as to further calculate the motion field quantity of the target object at each preset moment; then, filtering calculation is carried out on the motion field quantity of a certain moment and the adjacent moment before the certain moment to obtain the final motion field quantity of the certain moment and the motion field quantity predicted at the near moment in the future, and a smooth piecewise curve equation is fitted with a curve to obtain the motion field quantity of the target object at any moment and a corresponding surrounding frame of the target object so as to establish an envelope angle with all the library positions respectively;
wherein the storage unit stores the judgment basis of the included angle relationship between the bin envelope line and the tangential vector of the target object and the judgment basis of the corresponding bin state between the included angle relationship and the bin stock information; the processing unit calls the judgment basis of the storage unit, judges the state of the ex-warehouse bit and updates the state;
and the information output unit is used for sending the storage position state information to the robot scheduling system so as to provide scheduling basis for the robot warehousing and freight tasks.
In the preferred embodiment, distortion is caused by a problem in the camera manufacturing process. The identification unit is further configured to correct the bin region and the corner coordinates of each bin according to the camera parameters and the distortion parameters.
In summary, according to the visual information anti-shake method, the warehouse location state management method and the warehouse location state management system provided by the invention, a sub-pixel level motion model is theoretically built, continuous motion of goods is described through the motion field quantity of a target object, a segmented multi-order curve is adopted to fit and smooth a goods path, path stability is realized through a filtering algorithm, and a continuous motion field of the sub-pixel level object is estimated through a fitting curve, so that the motion field quantity of the target object is accurate to the sub-pixel level, and the problem of time delay shake of visual information is effectively solved.
On the other hand, in the related implementation scheme of the warehouse location state management method and system, a scheme capable of fully utilizing rich information provided by vision to establish a stable state relation between the position and the speed of the goods and the warehouse location is further established, so that the stability of the warehouse location state detection result is ensured, and non-contact type warehouse location detection is formed, thereby realizing a reliable warehouse location management function, avoiding the large-scale arrangement of sensors on site, and reducing the construction and maintenance cost for establishing warehouse location management.
The preferred embodiments of the invention disclosed above are intended to be illustrative only. The preferred embodiments are not intended to be exhaustive or to limit the invention to the precise embodiments disclosed. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the invention and the practical application, to thereby enable others skilled in the art to best utilize the invention. The invention is limited only by the claims and the full scope and equivalents thereof, and any modification, equivalent replacement, or improvement made within the spirit and principle of the invention should be included in the protection scope of the invention.
It will be appreciated by those skilled in the art that, in addition to implementing the system, apparatus and various modules thereof provided by the present invention in the form of pure computer readable program code, the same procedures may be implemented entirely by logically programming method steps such that the system, apparatus and various modules thereof provided by the present invention are implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Therefore, the system, the device and the modules thereof provided by the present invention can be considered as a hardware component, and the modules included in the system, the device and the modules thereof for implementing various programs can also be considered as structures in the hardware component; modules for performing various functions may also be considered to be both software programs for performing the methods and structures within hardware components.
In addition, all or part of the steps of the method according to the above embodiments may be implemented by a program instructing related hardware, where the program is stored in a storage medium and includes several instructions to enable a single chip, a chip, or a processor (processor) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In addition, any combination of various different implementation manners of the embodiments of the present invention is also possible, and the embodiments of the present invention should be considered as disclosed in the embodiments of the present invention as long as the combination does not depart from the spirit of the embodiments of the present invention.
Claims (8)
1. A visual information anti-shake method is characterized by comprising the following steps:
step S1: collecting scene image frames at each preset moment; identifying a target object in a scene image frame to obtain position and size information of the target object, establishing an enclosing frame and calculating a tangential vector of the target object so as to calculate motion field quantity of the target object corresponding to each preset moment; wherein in the step S1, the position information of the target object k at the time t is identified by adopting a target tracking network scheme
And size information S t To create an enclosure frame (
，S t ) And calculating a corresponding tangential vector
To obtain the motion field quantity omega of the target object k at the time t k t Is prepared from (a)
) k ；
Step S2: filtering and calculating the motion field quantity of a certain moment and the previous adjacent moment to obtain the final motion field quantity of the certain moment and the predicted motion field quantity of the future adjacent moment;
and step S3: and fitting the motion field quantity obtained in the step S2 into a smooth piecewise curve equation through a curve so as to obtain the motion field quantity of the target object at any moment.
2. A warehouse bit state management method is characterized by comprising the following steps:
step B1: collecting scene image frames at each preset moment; extracting a library position area and angular point coordinates of each library position in a scene image frame, identifying a target object in the scene image frame, acquiring position and size information of the target object, establishing a surrounding frame, calculating a tangential vector of the target object, and calculating motion field quantities of the target object at each preset moment; wherein in the step B1, the position information of the target object k at the time t is identified by adopting a target tracking network scheme
And size information S t To create an enclosure frame (
，S t ) And calculating the corresponding tangential vector
To obtain the motion field quantity omega of the target object k at the time t k t Is prepared from (a)
) k ；
And step B2: filtering and calculating the motion field quantity of a certain moment and the adjacent moment before the certain moment to obtain the final motion field quantity of the certain moment and the predicted motion field quantity of the near moment in the future;
and step B3: fitting the motion field quantity obtained in the step B2 into a smooth piecewise curve equation through a curve so as to obtain the motion field quantity of the target object at any moment;
and step B4: acquiring an enclosing frame corresponding to the motion field quantity of the target object at a certain moment according to the step B3, and respectively establishing an enveloping angle between the enclosing frame and all the library positions so as to judge the included angle relation formed by the enveloping line of each library position and the tangential vector of the target object; wherein the step of establishing the envelope angle comprises: respectively radiating lines from the center point of the target object surrounding frame to the corner points of each library position, selecting the maximum included angle formed by two of the radiating lines as an envelope angle, and taking the two radiating lines of the envelope angle as envelope lines;
and step B5: and judging the state of the storage position according to the relation between the storage position inventory information and the included angle acquired in the step S4.
3. The warehouse storage location state management method according to claim 2, wherein the step B1 further comprises the step of correcting the location area and the corner coordinates of each storage location according to the internal parameters and distortion parameters of the camera when the acquired scene image frame has distortion.
4. The warehouse location state management method of claim 2, wherein the filtering calculation in step B2 employs a kalman filtering algorithm.
5. The warehouse storage space state management method according to claim 2, wherein in the step B3, after at least 5 continuous motion field quantities are calculated and obtained in the step B2, a smooth piecewise curve equation is fitted through a third-order bezier curve to obtain the motion field quantity of the target object at any time.
6. The warehouse storage space state management method according to claim 2, wherein in the step B5, the step of determining the included angle relationship comprises:
according to the included angle relation between the tangential vector of the target object k and the left and right envelope lines of each library position
、
Form the following positional relationship
Judging:
7. The warehouse location state management method according to claim 6, wherein in the step B5, the step of determining the location state comprises:
if there is goods in the storage location, and
is 0 and has no other target objectFormed therewith
Within the positive envelope angle, the bin position state is bin full;
if there is goods in the storage location, and
in the positive envelope angle, the storage position state is full;
if there is no cargo in the storage space, and
if the current position is within the positive envelope angle, the storage position state is a storage state;
if there is no cargo in the storage space, and
in the negative envelope angle, the storage position state is ex-storage;
if there is no cargo in the storage space, and
is 0 and no other goods are formed with it
Within the envelope angle, the bin state is bin empty.
8. A warehouse bit state management system, characterized by comprising:
a storage unit, for storing the program of the warehouse location state management method steps according to any one of claims 2 to 7, for the control unit, the identification unit, the processing unit, the information output unit to fetch and execute in due time;
wherein the control unit is configured to coordinate:
the camera is used for acquiring scene image frames at preset moments;
the identification unit is used for extracting the bin region and the corner coordinates of each bin from the scene image frame, and identifying the target object in the bin region to acquire the position and size information of the target object and establish a surrounding frame;
the processing unit is used for calculating a tangential vector of the target object according to the bounding box data so as to further calculate the motion field quantity of the target object at each preset moment; then, filtering calculation is carried out on the motion field quantity of a certain moment and the adjacent moment before the certain moment to obtain the final motion field quantity of the certain moment and the motion field quantity predicted at the near moment in the future, and a smooth piecewise curve equation is fitted with a curve to obtain the motion field quantity of the target object at any moment and a corresponding surrounding frame of the target object so as to establish an envelope angle with all the library positions respectively;
wherein the storage unit stores the judgment basis of the included angle relationship between the bin envelope line and the tangential vector of the target object and the judgment basis of the corresponding bin state between the included angle relationship and the bin stock information; the processing unit calls the judgment basis of the storage unit, judges the state of the ex-warehouse bit and updates the state;
and the information output unit is used for transmitting the library position state information to the robot scheduling system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211504539.1A CN115601271B (en) | 2022-11-29 | 2022-11-29 | Visual information anti-shake method, storage warehouse location state management method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211504539.1A CN115601271B (en) | 2022-11-29 | 2022-11-29 | Visual information anti-shake method, storage warehouse location state management method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115601271A CN115601271A (en) | 2023-01-13 |
| CN115601271B true CN115601271B (en) | 2023-03-24 |
Family
ID=84852224
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211504539.1A Active CN115601271B (en) | 2022-11-29 | 2022-11-29 | Visual information anti-shake method, storage warehouse location state management method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115601271B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115860642B (en) * | 2023-02-02 | 2023-05-05 | 上海仙工智能科技有限公司 | Visual identification-based warehouse-in and warehouse-out management method and system |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102721968A (en) * | 2012-06-21 | 2012-10-10 | 中国人民解放军电子工程学院 | Method for correcting tracking and aiming beam deviation in satellite-to-satellite SAL (synthetic aperture radar) imaging |
| CN104036137A (en) * | 2014-06-11 | 2014-09-10 | 沈阳东软医疗***有限公司 | Random coincidence event removing method and device |
| CN107851325A (en) * | 2016-05-09 | 2018-03-27 | 东莞市安零模具科技有限公司 | The detection method that falls down to the ground of shared bicycle |
| CN109631896A (en) * | 2018-07-23 | 2019-04-16 | 同济大学 | A kind of parking lot autonomous parking localization method based on vehicle vision and motion information |
| CN110634145A (en) * | 2018-06-22 | 2019-12-31 | 青岛日日顺物流有限公司 | Warehouse checking method based on image processing |
| CN112037265A (en) * | 2020-11-04 | 2020-12-04 | 天津天瞳威势电子科技有限公司 | Library bit tracking method and device |
| CN112802073A (en) * | 2021-04-08 | 2021-05-14 | 之江实验室 | Fusion registration method based on image data and point cloud data |
| WO2021093420A1 (en) * | 2019-11-12 | 2021-05-20 | 深圳创维数字技术有限公司 | Vehicle navigation method and apparatus, and computer readable storage medium |
| CN114693787A (en) * | 2022-03-18 | 2022-07-01 | 东风汽车集团股份有限公司 | Parking garage map building and positioning method and system and vehicle |
-
2022
- 2022-11-29 CN CN202211504539.1A patent/CN115601271B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102721968A (en) * | 2012-06-21 | 2012-10-10 | 中国人民解放军电子工程学院 | Method for correcting tracking and aiming beam deviation in satellite-to-satellite SAL (synthetic aperture radar) imaging |
| CN104036137A (en) * | 2014-06-11 | 2014-09-10 | 沈阳东软医疗***有限公司 | Random coincidence event removing method and device |
| CN107851325A (en) * | 2016-05-09 | 2018-03-27 | 东莞市安零模具科技有限公司 | The detection method that falls down to the ground of shared bicycle |
| CN110634145A (en) * | 2018-06-22 | 2019-12-31 | 青岛日日顺物流有限公司 | Warehouse checking method based on image processing |
| CN109631896A (en) * | 2018-07-23 | 2019-04-16 | 同济大学 | A kind of parking lot autonomous parking localization method based on vehicle vision and motion information |
| WO2021093420A1 (en) * | 2019-11-12 | 2021-05-20 | 深圳创维数字技术有限公司 | Vehicle navigation method and apparatus, and computer readable storage medium |
| CN112037265A (en) * | 2020-11-04 | 2020-12-04 | 天津天瞳威势电子科技有限公司 | Library bit tracking method and device |
| CN112802073A (en) * | 2021-04-08 | 2021-05-14 | 之江实验室 | Fusion registration method based on image data and point cloud data |
| CN114693787A (en) * | 2022-03-18 | 2022-07-01 | 东风汽车集团股份有限公司 | Parking garage map building and positioning method and system and vehicle |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115601271A (en) | 2023-01-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Chen et al. | AI-empowered speed extraction via port-like videos for vehicular trajectory analysis | |
| CN108458715B (en) | Robot positioning initialization method based on laser map | |
| Lippiello et al. | Adaptive extended Kalman filtering for visual motion estimation of 3D objects | |
| CN111798487B (en) | Target tracking method, apparatus and computer readable storage medium | |
| CN115601271B (en) | Visual information anti-shake method, storage warehouse location state management method and system | |
| CN115423846A (en) | Multi-target track tracking method and device | |
| CN112298194B (en) | Lane changing control method and device for vehicle | |
| CN110246160A (en) | Detection method, device, equipment and the medium of video object | |
| CN111382784B (en) | Moving target tracking method | |
| CN110726413A (en) | Multi-sensor fusion and data management mechanism facing large-scale SLAM | |
| EP4179401A1 (en) | Autonomous robotic navigation in storage site | |
| CN115008093A (en) | Multi-welding-point welding robot control system and method based on template identification | |
| CN116460843A (en) | Multi-robot collaborative grabbing method and system based on meta heuristic algorithm | |
| CN110702093A (en) | Particle filter-based positioning method and device, storage medium and robot | |
| CN112414444B (en) | Data calibration method, computer equipment and storage medium | |
| CN103363916A (en) | Information processing method and processing device | |
| CN114986051A (en) | Industrial robot welding control system and method based on template recognition | |
| Tran et al. | Robust uncalibrated rectification with low geometric distortion under unbalanced field of view circumstances | |
| Zhang et al. | DynPL-SLAM: A Robust Stereo Visual SLAM System for Dynamic Scenes Using Points and Lines | |
| CN104869285A (en) | Motion compensation method based on main motion direction identification | |
| Baek et al. | Deep learning-based automated productivity monitoring for on-site module installation in off-site construction | |
| Shi et al. | Multi-Sensor SLAM Assisted by 2D LiDAR Line Features | |
| CN115965899B (en) | Video segmentation-based unmanned sweeping robot anomaly detection method and system | |
| CN114862810B (en) | Fruit counting method, device and storage medium | |
| Diederichs | Hardware-software co-design tracking system for predictable high-speed mobile microrobot position control |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: A visual information anti shaking method, storage location status management method and system Effective date of registration: 20230828 Granted publication date: 20230324 Pledgee: Bank of Communications Ltd. Shanghai New District Branch Pledgor: Shanghai Xiangong Intelligent Technology Co.,Ltd. Registration number: Y2023310000491 |