CN111704036A - Lifting equipment alignment system and method - Google Patents
Lifting equipment alignment system and method Download PDFInfo
- Publication number
- CN111704036A CN111704036A CN202010631776.9A CN202010631776A CN111704036A CN 111704036 A CN111704036 A CN 111704036A CN 202010631776 A CN202010631776 A CN 202010631776A CN 111704036 A CN111704036 A CN 111704036A
- Authority
- CN
- China
- Prior art keywords
- pose
- data
- container
- inertial navigation
- lifting appliance
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B66—HOISTING; LIFTING; HAULING
- B66C—CRANES; LOAD-ENGAGING ELEMENTS OR DEVICES FOR CRANES, CAPSTANS, WINCHES, OR TACKLES
- B66C13/00—Other constructional features or details
- B66C13/18—Control systems or devices
- B66C13/40—Applications of devices for transmitting control pulses; Applications of remote control devices
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B66—HOISTING; LIFTING; HAULING
- B66C—CRANES; LOAD-ENGAGING ELEMENTS OR DEVICES FOR CRANES, CAPSTANS, WINCHES, OR TACKLES
- B66C13/00—Other constructional features or details
- B66C13/04—Auxiliary devices for controlling movements of suspended loads, or preventing cable slack
- B66C13/08—Auxiliary devices for controlling movements of suspended loads, or preventing cable slack for depositing loads in desired attitudes or positions
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B66—HOISTING; LIFTING; HAULING
- B66C—CRANES; LOAD-ENGAGING ELEMENTS OR DEVICES FOR CRANES, CAPSTANS, WINCHES, OR TACKLES
- B66C13/00—Other constructional features or details
- B66C13/16—Applications of indicating, registering, or weighing devices
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B66—HOISTING; LIFTING; HAULING
- B66C—CRANES; LOAD-ENGAGING ELEMENTS OR DEVICES FOR CRANES, CAPSTANS, WINCHES, OR TACKLES
- B66C13/00—Other constructional features or details
- B66C13/18—Control systems or devices
- B66C13/22—Control systems or devices for electric drives
Landscapes
- Engineering & Computer Science (AREA)
- Mechanical Engineering (AREA)
- Automation & Control Theory (AREA)
- Load-Engaging Elements For Cranes (AREA)
- Control And Safety Of Cranes (AREA)
Abstract
The invention discloses a system and a method for aligning hoisting equipment, wherein the system comprises: the data acquisition module comprises four groups of data acquisition equipment which are respectively arranged on four tail end execution mechanisms of the lifting appliance, and the four groups of data acquisition equipment are used for acquiring image data and inertial navigation data of the lifting appliance; the lifting appliance pose detection module is used for obtaining the real-time pose of the lifting appliance; the target position and pose detection module is used for obtaining the real-time pose of the container; and the alignment module is used for determining the real-time relative position relationship between the spreader and the container according to the real-time pose of the spreader and the real-time pose of the container, and performing low-point alignment on the tail end actuating mechanism closest to the container and the corresponding target lock hole, and then completing alignment of the other tail end actuating mechanism and the corresponding target lock hole. According to the system and the method for aligning the hoisting equipment, provided by the invention, the alignment success rate is improved by providing the low-delay high-precision position and posture information of the lifting appliance, so that the automatic loading and unloading of the container are realized, and the operation efficiency of the hoisting equipment is further improved.
Description
Technical Field
The invention relates to the field of crane loading and unloading, in particular to a system and a method for aligning hoisting equipment.
Background
The traditional crane operation adopts a manual operation mode to load and unload the container, the container is controlled by a manually operated crane, and when the alignment fails, the operation needs to be manually adjusted according to actual conditions. The mode has low operation efficiency, and the whole operation can be influenced by human because of manual intervention, so that the danger is greatly increased.
In the loading and unloading process of the container, the lifting appliance is a specific execution device for operating the container, and in the loading and unloading operation process by using the lifting appliance, due to the positioning error of the encoder, the deformation of the pull rope, the influence of weather environment and the like, the position and posture of the lifting appliance can be unexpectedly changed, so that the alignment failure of the lifting appliance and the target container is caused.
Therefore, the automatic loading and unloading of the container is a key technical link for realizing automation of the container terminal, and a system for aligning the hoisting equipment is needed to be provided, so that the automatic loading and unloading of the container can be realized.
Disclosure of Invention
The invention aims to solve the technical problem of providing a system and a method for aligning lifting equipment, and improving the success rate of alignment by providing low-delay high-precision hanger pose information, thereby realizing automatic loading and unloading of containers and further improving the operation efficiency of the lifting equipment.
The invention adopts a technical scheme for solving the technical problems that an aligning system of hoisting equipment is provided, which comprises:
the data acquisition module comprises four groups of data acquisition equipment which are respectively arranged on four tail end execution mechanisms of the lifting appliance, and the four groups of data acquisition equipment are used for acquiring image data and inertial navigation data of the lifting appliance;
the lifting appliance pose detection module is used for determining poses of the four groups of data acquisition equipment in a world coordinate system by using a pose detection algorithm, obtaining poses of four tail end execution mechanisms of the lifting appliance in the world coordinate system according to pose transformation of the four groups of data acquisition equipment in the world coordinate system, and fitting the poses of the four tail end execution mechanisms of the lifting appliance to a space rectangle so as to obtain a real-time pose of the lifting appliance;
the target position and pose detection module is used for obtaining poses of the four target lock holes relative to the four groups of data acquisition equipment through the image data and the inertial navigation data of the lifting appliance, and fitting the poses of the four target lock holes to a space rectangle so as to obtain real-time poses of the container;
and the alignment module is used for determining the real-time relative position relationship between the spreader and the container according to the real-time pose of the spreader and the real-time pose of the container, performing low-point alignment on the tail end actuating mechanism closest to the container and the corresponding target lock hole, and then completing container completion after the other tail end actuating mechanism closest to the container and positioned on the same long edge as the tail end actuating mechanism closest to the container and the corresponding target lock hole are aligned to each other.
Preferably, the data acquisition device comprises an image acquisition device for providing image data and an inertial navigation unit for providing angular velocity and linear acceleration data.
Preferably, the end effector is mounted on a lock knob of the spreader.
Preferably, the hanger pose detection module comprises a data preprocessing unit, an initialization unit, a rear end nonlinear optimization unit and a closed loop detection unit;
the preprocessing unit is used for synchronizing the image data and the inertial navigation data and abstracting and transforming the image data and the inertial navigation data;
the initialization unit is used for fusing the image data and the inertial navigation data to obtain the scale of the image acquisition equipment, the direction of a gravity vector and the bias of the inertial measurement unit;
the rear-end nonlinear optimization unit is used for performing combined optimization on a residual error item constructed based on the image data and a residual error item constructed based on the inertial navigation data, and obtaining an optimal state vector by adopting a nonlinear optimization mode;
the closed-loop detection unit is used for performing closed-loop detection by using a DBoW2 algorithm, filtering the detected result and adding the filtered result into the optimization queue again to participate in pose optimization so as to obtain an optimal pose result.
Preferably, external parameters of the coordinate system of the image acquisition device and the coordinate system of the inertial navigation unit are obtained by calibration using Kalibr calibration software before installation.
The invention also provides a hoisting equipment aligning method for solving the technical problems, which comprises the following steps:
acquiring image data and inertial navigation data of a lifting appliance;
determining poses of four groups of data acquisition equipment in a world coordinate system by using a pose detection algorithm, obtaining poses of four tail end execution mechanisms of the lifting appliance according to pose transformation of the four groups of data acquisition equipment in the world coordinate system, and fitting the poses of the four tail end execution mechanisms of the lifting appliance to a space rectangle so as to obtain a real-time pose of the lifting appliance;
the poses of the four target lock holes relative to the four groups of data acquisition equipment are obtained through the image data and the inertial navigation data of the lifting appliance, and the poses of the four target lock holes are fitted to a space rectangle, so that the real-time pose of the container is obtained;
and determining the real-time relative position relation between the spreader and the container according to the real-time pose of the spreader and the real-time pose of the container, and after the end actuating mechanism closest to the container is subjected to low point alignment with the corresponding target lock hole, finishing the container finishing work after the other end actuating mechanism closest to the container and positioned on the same long edge is subjected to high point alignment with the corresponding target lock hole.
Preferably, the image data is provided by an image acquisition device, the inertial navigation data comprises angular velocity and linear acceleration data, and the inertial navigation data is provided by an inertial navigation unit.
Preferably, the end effector is mounted on a lock knob of the spreader.
Preferably, the determining the poses of the four sets of data acquisition devices in the world coordinate system by using a pose detection algorithm comprises:
synchronizing the image data and the inertial navigation data, and abstracting and transforming the image data and the inertial navigation data;
fusing the image data and the inertial navigation data to obtain the scale of the image acquisition equipment, the direction of a gravity vector and the bias of an inertial measurement unit;
performing joint optimization on a residual error item constructed based on the image data and a residual error item constructed based on the inertial navigation data, and obtaining an optimal state vector by adopting a nonlinear optimization mode;
and performing closed-loop detection by using a DBoW2 algorithm, filtering the detected result, and adding the filtered result into the optimization queue again to participate in pose optimization so as to obtain an optimal pose result.
Preferably, external parameters of the coordinate system of the image acquisition device and the coordinate system of the inertial navigation unit are obtained by calibration using Kalibr calibration software before installation.
Compared with the prior art, the invention has the following beneficial effects: the invention provides a system and a method for aligning lifting equipment, which acquire image data and inertial navigation data of a lifting appliance through four groups of data acquisition equipment arranged on four tail end execution mechanisms of the lifting appliance, acquire poses of the four groups of data acquisition equipment under a world coordinate system, further acquire the poses of the four tail end execution mechanisms, acquire real-time poses of the lifting appliance through pose fitting a spatial rectangle, acquire the poses of four target lock holes through the image data and the inertial navigation data, acquire the real-time poses of a container through pose fitting the spatial rectangle, determine the real-time relative position relationship between the lifting appliance and the container according to the real-time poses of the lifting appliance and the container, complete low-point alignment firstly and then complete alignment between the other tail end execution mechanism and the corresponding target lock hole, and improve the alignment success rate through providing the real-time pose information of the lifting appliance and the container with low time delay, thereby realizing the automatic loading and unloading of the container and further improving the operation efficiency of the hoisting equipment;
furthermore, the image data of the lifting appliance is acquired through the data acquisition equipment, and then the real-time pose of the lifting appliance and the real-time pose of the target lock hole are acquired, so that the real-time pose of the container is acquired, the relative pose relation between the lifting appliance and the container is accurately acquired, and the alignment success rate and the alignment accuracy are improved.
Furthermore, the containers are placed at a low point first and then at a high point, so that the automatic dynamic placement of the containers is realized.
Drawings
Fig. 1 is a schematic structural diagram of an alignment system of a hoisting device in an embodiment of the invention;
FIG. 2 is a block diagram of an alignment system of the lifting device in the embodiment of the present invention;
FIG. 3 is a block diagram of an alignment system of a lifting device according to another embodiment of the present invention;
fig. 4 is a flowchart of a method for aligning a hoisting device in an embodiment of the present invention.
Detailed Description
The invention is further described below with reference to the figures and examples.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one of ordinary skill in the art that the present invention may be practiced without these specific details. Accordingly, the particular details set forth are merely exemplary, and the particular details may be varied from the spirit and scope of the present invention and still be considered within the spirit and scope of the present invention.
Referring now to fig. 1, fig. 1 is a schematic structural diagram of an alignment system of a hoisting device in an embodiment of the present invention, where the alignment system of the hoisting device includes a cart 1, a trolley 2, a spreader 3, a data acquisition device 4, a spreader low-point lock button 5, a container 6, and a target lock hole 7.
And when the cart 1 and the trolley 2 reach the target positions where the containers 6 can be operated and the lifting appliance 3 reaches the target height, starting lifting appliance pose detection. The target height is dynamically variable, determined by the height of the currently operating container 6, which is typically the height of the currently operating container 6 plus 3 meters. The target height is set so that the lifting appliance pose detection algorithm can be initialized normally.
The data acquisition device 4 of the positioning system of the lifting device generally adopts an image acquisition device, which may be a camera, and an Inertial navigation Unit (IMU) as data sources. As shown in fig. 1, the spreader has four vertexes on a horizontal plane, the vertex at which the spreader low-point lock knob 5 is located is a point with the lowest height among the four vertexes, that is, a low point, and adjacent points located on the same long side are high points. The data acquisition equipment 4, including image acquisition equipment, is placed downwards at the corresponding position of the low point, and is matched with and provided with an inertial navigation unit. The image acquisition device is used for providing image data, and the inertial navigation unit is used for providing angular velocity and linear acceleration data.
Fig. 2 is a block diagram of an alignment system of a hoisting device in an embodiment of the present invention, which shows an alignment system of a hoisting device, including: the data acquisition module 21 comprises four groups of data acquisition equipment which are respectively arranged on four tail end execution mechanisms of the lifting appliance and are used for acquiring image data and inertial navigation data of the lifting appliance; the hanger pose detection module 22 is used for determining poses of the four groups of data acquisition equipment in a world coordinate system by using a pose detection algorithm, obtaining poses of four tail end execution mechanisms of the hanger in the world coordinate system according to pose transformation of the four groups of data acquisition equipment in the world coordinate system, and fitting the poses of the four tail end execution mechanisms of the hanger to a space rectangle so as to obtain a real-time pose of the hanger; the target position and pose detection module 23 is configured to obtain poses of the four target lock holes relative to the four groups of data acquisition devices through the image data and the inertial navigation data of the spreader, and fit the poses of the four target lock holes to a spatial rectangle, so as to obtain a real-time pose of the container; and the alignment module 24 is configured to determine a real-time relative position relationship between the spreader and the container according to the real-time pose of the spreader and the real-time pose of the container, perform low-point alignment on the end actuator closest to the container and the corresponding target lock hole, and then perform alignment on the other end actuator located on the same long side as the end actuator closest to the container and the corresponding target lock hole.
In the specific implementation, four groups of data acquisition devices are used for image data and inertial navigation data of the lifting appliance, and the four groups of data acquisition devices are arranged on the four tail end execution mechanisms of the lifting appliance, so that after the poses of the four groups of data acquisition devices in a world coordinate system are determined through a pose detection algorithm, the poses of the four tail end execution mechanisms in the world coordinate system can be easily obtained, and the poses of the four data acquisition devices in the world coordinate system are respectively the same as the poses of the four corresponding tail end execution mechanisms in the world coordinate system.
In specific implementation, fitting the poses of the four end actuators of the spreader to a spatial rectangle comprises the following steps: the poses of the four end actuators of the spreader are known, i.e. the three-dimensional coordinates of four points are known. First, a plane is fitted to the four points by using a least square method, and then the four points are projected onto the plane respectively, so that two-dimensional coordinates of the four points on the plane are obtained. And (4) solving the minimum circumscribed rectangle of the four points according to the two-dimensional coordinates of the four points on the plane, so as to obtain the space rectangles of the four end actuating mechanisms. Because the plane equation and the representation of the rectangle in the plane are known, the spatial rectangles of the four target lock holes can be obtained, and the representation of the container in the world coordinate system can be further obtained. In specific implementation, the end actuating mechanisms are installed on the lock buttons of the lifting appliance, the number of the end actuating mechanisms is four, the number of the lock buttons of the lifting appliance is also four, and the four end actuating mechanisms are respectively installed on the four lock buttons of the lifting appliance.
For a single target lock hole, firstly, a rough position of the single target lock hole in each frame of image is framed in the image by using a deep learning method, and then, an accurate position of the single target lock hole in the image is obtained by using an edge segmentation algorithm. By analogy, the positions of the four target lock holes in the image can be obtained.
In specific implementation, after the lower point alignment is performed on the end executing mechanism closest to the container and the corresponding target lock hole, and then the upper point alignment is performed on the other end executing mechanism located on the same long side as the end executing mechanism closest to the container and the corresponding target lock hole, because the lifting appliance is a rigid body, after the long side on one side is successfully aligned, the long side on the other side is successfully aligned naturally, and the container operation is completed.
Fig. 3 is a block diagram of an aligning system of a hoisting device in another embodiment of the present invention, which shows an aligning system of a hoisting device, including: the data acquisition module 21 comprises four groups of data acquisition equipment which are respectively arranged on four tail end execution mechanisms of the lifting appliance and are used for acquiring image data and inertial navigation data of the lifting appliance; the hanger pose detection module 22 is used for determining poses of the four groups of data acquisition equipment in a world coordinate system by using a pose detection algorithm, obtaining poses of four tail end execution mechanisms of the hanger in the world coordinate system according to pose transformation of the four groups of data acquisition equipment in the world coordinate system, and fitting the poses of the four tail end execution mechanisms of the hanger to a space rectangle so as to obtain a real-time pose of the hanger; the target position and pose detection module 23 is configured to obtain poses of the four target lock holes relative to the four groups of data acquisition devices through the image data and the inertial navigation data of the spreader, and fit the poses of the four target lock holes to a spatial rectangle, so as to obtain a real-time pose of the container; and the alignment module 24 determines the real-time relative position relationship between the spreader and the container according to the real-time pose of the spreader and the real-time pose of the container, performs low-point alignment on the end actuator closest to the container and the corresponding target lock hole, and then performs high-point alignment on the other end actuator located on the same long side as the end actuator closest to the container and the corresponding target lock hole, so that when the long side of one side is successfully aligned, the long side of the other side is naturally aligned successfully, and the container operation is completed.
In the specific implementation, four groups of data acquisition devices are used for image data and inertial navigation data of the lifting appliance, and the four groups of data acquisition devices are arranged on the four tail end execution mechanisms of the lifting appliance, so that after the poses of the four groups of data acquisition devices in a world coordinate system are determined through a pose detection algorithm, the poses of the four tail end execution mechanisms in the world coordinate system can be easily obtained, and the poses of the four data acquisition devices in the world coordinate system are respectively the same as the poses of the four corresponding tail end execution mechanisms in the world coordinate system. The data acquisition module 21 includes an image acquisition device 211 and an inertial navigation unit 212. The spreader pose detection module 22 comprises a data preprocessing unit 221, an initialization unit 222, a back end nonlinear optimization unit 223 and a closed loop detection unit 224; the data preprocessing unit 221 is configured to synchronize the image data and the inertial navigation data, and abstract and transform the image data and the inertial navigation data; the initialization unit 222 is configured to fuse the image data and the inertial navigation data to obtain a scale of the image acquisition device 211, a direction of a gravity vector, and a bias of an inertial measurement unit; the back-end nonlinear optimization unit 223 is configured to perform joint optimization on the residual error items constructed based on the image data and the residual error items constructed based on the inertial navigation data, and obtain an optimal state vector by adopting a nonlinear optimization mode; the closed-loop detection unit 224 is configured to perform closed-loop detection by using a DBoW2 algorithm, filter a detected result, and add the detected result into the optimization queue again to participate in pose optimization, so as to obtain an optimal pose result.
In the data preprocessing unit 221, data inputs of the detection algorithm are image data of the image pickup device 211 and angular velocity and linear acceleration data from the inertial navigation unit 212, respectively. The synchronization module in the system provides the image acquisition device 211 and the inertial navigation unit 212 with the trigger signal and the clock that has been synchronized. The detection algorithm firstly synchronizes the two data according to the time stamp of the data, and corresponds each frame of image data to the inertial navigation data in the time interval of the two frames of image data one by one. The processing of image data often requires a significant amount of computation. In order to reduce the amount of calculation and improve the operation efficiency, feature points need to be extracted from the image first. Aiming at the characteristics of few objects and poor characteristic texture in a storage yard, considering the calculation amount, selecting and adopting ORB (ordered FAST and Rotated BRIEF) to extract characteristic points, tracking adjacent corner points by using an LK (Lucas-Kanade) optical flow method, filtering abnormal points by using a Random sampling consensus method (RANSAC, Random sampling consensus) and finally obtaining the target characteristic points of the frame. And pushing the target characteristic points serving as mathematical representations of the images to a data queue, and informing a back end of processing. For the data of the inertial navigation unit 212, the data of each frame of the inertial navigation unit 212 and the data of the inertial navigation unit 212 corresponding to each frame of the image which is synchronized before are integrated respectively, so as to obtain the corresponding speed, rotation and pose. At the same time, the pre-integration increment of the data of the inertial navigation unit 212 corresponding to the adjacent image frame is calculated.
In the initialization unit 222, the image information and the information of the inertial navigation unit 212 are fused in a loose coupling manner, so as to obtain the scale of the image acquisition device, the direction of the gravity vector, and the bias of the inertial navigation unit 212 in the system.
1) And maintaining a fixed number n +1 of sliding windows by the algorithm on the basis of the target feature point queue obtained in the first step. According to the frame rate of the image acquisition equipment and the processing speed of the computer, n is more ideal after being obtained through experiments, wherein n is 10. Through a visual SFM method, mainly PnP (Passive-n-Point) and triangulation, the image acquisition equipment pose at the current moment and the pose corresponding to the landmark Point are calculated for all frames of the sliding window. The pose is then translated to the coordinate system of inertial navigation unit 212, whose external parameters have been calibrated.
2) By the calculation in 1), the rotation between adjacent image frames can be obtained. While the rotation of adjacent image frames may also be obtained by pre-integration of the inertial navigation unit 212 as calculated in the first step. The difference between the two modes is minimized, and the bias, the gravity vector and the image acquisition device scale of the inertial navigation unit 212 can be calculated through an optimization method.
In the back-end nonlinear optimization unit 223, the residual items constructed based on the image information and the residual items constructed based on the information of the inertial navigation unit 212 are put together for joint optimization, and the optimal state vector is obtained by solving in a nonlinear optimization manner.
The state vector includes the states of n +1 image capture devices within the sliding window (X state vector), the external reference (X) of image capture device 211 to inertial navigation unit 212, and the inverse depths of m +1 waypoints.
An objective function of
The three residual terms, namely error terms, are marginalized prior information, the residual measured by the inertial navigation unit 212, and the visual re-projection residual, respectively.
In the closed-loop detection unit 224, DBoW2(Bags of Binary Words for Fast platform registration in Image Sequences) is used for closed-loop detection, and if the detected result is filtered and then added into the optimization queue again, the optimization is performed through the third step, so as to obtain the optimal pose result. The port itself is open and there are few objects except containers and equipment. For this feature of port, a generic bag of words is not applicable, which is a database for characterizing image features. The bag of words is then recreated using the port site picture and loaded before the algorithm is started. 500 FAST (features from accessed Segment test) corner points are extracted from the image, BRIEF descriptors are calculated for ORB (organized FAST and Rotated BRIEF) corner points extracted from the image in the first step, then DBoW2 is used for closed-loop detection, closed-loop frames are obtained, and the closed-loop frames are pushed to a data queue optimized at the rear end to participate in pose optimization.
On the basis of the pose output at the frequency of 10 Hz generated by the algorithm, the pose output of the lifting appliance with the highest frequency of 100 Hz can be obtained by superposing each frame of relative pose obtained by integrating the original data of the inertial navigation unit 212 in the first step of the algorithm. The accurate position appearance of hoist of high frequency can improve the success rate and the efficiency of counterpoint operation.
And (3) while the hanger pose detection algorithm is running, obtaining the coordinate value of the target lock hole in the initial frame image after the initialization is successful by using a deep learning method. Meanwhile, the depth of the target lock hole can be obtained through the image acquisition device 211, so that the relative position relationship between the target lock hole 7 and the lifting appliance 3 in the initial frame is restored. Since the container is fixed, the relative positional relationship between the target lock hole 7 and the spreader 3 can be obtained based on the movement of the container. The container and the lock hole structure are shown in fig. 1, and the target lock hole 7 is a target lock hole corresponding to a low point of the spreader. Therefore, the system can output the relative position relationship between the target lock hole 7 and the spreader 3 at 100 Hz, thereby providing support for the subsequent alignment operation and realizing the automation of the alignment of the spreader 3 and the container.
In specific implementation, for a single target lockhole, firstly, a rough position of the single target lockhole in each frame of image is framed in the image by using a deep learning method, then, the position of the single target lockhole in the image is accurately segmented by using an edge segmentation algorithm according to the framed result, and finally, the pixel coordinate of the central point of the single target lockhole, namely the pixel coordinate of the single target lockhole, is calculated. According to the pixel coordinates of the single target lock hole, the depth value of the single target lock hole in the coordinate system of the current image data acquisition device can be obtained from the depth map. The pixel coordinates and the depth values of the single target lock hole are known, the three-dimensional coordinates of the single target lock hole under the coordinate system of the current image data acquisition equipment can be obtained through calculation, and the relative position relation between the single target lock hole and the lifting appliance can be known. By analogy, the three-dimensional coordinates of the four target lock holes in the coordinate system of the current image data acquisition equipment can be obtained, and the relative position relation between the four target lock holes and the lifting appliance can be known.
In a specific implementation, the external parameters of the coordinate system of the image acquisition device 211 and of the coordinate system of the inertial navigation unit 212 are obtained by calibration using Kalibr calibration software before installation.
When the hanger pose detection algorithm is started, the image acquisition equipment 211 is started at the same time, and the lockhole or landmark information of the target position to be boxed is identified by using a deep learning method. Wherein, when the target position is a container, the lock button is identified. And when the target position is the ground, the landmark is identified. When the recognition of the target position reaches the confidence threshold, the pose information of the target position in the coordinate system of the image capturing device 211 is given.
Target position information p given by image acquisition equipmentDSelf pose information p at the same time with the visual inertia odometerVAligning to obtain a transfer matrix
Target position information pDAnd self pose information pVThe following formula is satisfied:
thereafter, with the movement of the visual inertial odometer system, it is possible to follow the transfer matrix
And obtaining the real-time position relation of the target position relative to the visual inertia.
Referring to fig. 4, in order to solve the above technical problem, the present invention further provides a method for aligning a hoisting device, including the following steps:
step 401: acquiring image data and inertial navigation data of a lifting appliance;
step 402: determining poses of four groups of data acquisition equipment in a world coordinate system by using a pose detection algorithm, obtaining poses of four tail end execution mechanisms of the lifting appliance according to pose transformation of the four groups of data acquisition equipment in the world coordinate system, and fitting the poses of the four tail end execution mechanisms of the lifting appliance to a space rectangle so as to obtain a real-time pose of the lifting appliance;
step 403: the poses of the four target lock holes relative to the four groups of data acquisition equipment are obtained through the image data and the inertial navigation data of the lifting appliance, and the poses of the four target lock holes are fitted to a space rectangle, so that the real-time pose of the container is obtained;
step 404: and determining the real-time relative position relation between the spreader and the container according to the real-time pose of the spreader and the real-time pose of the container, and after the end actuating mechanism closest to the container is subjected to low point alignment with the corresponding target lock hole, finishing the container finishing work after the other end actuating mechanism closest to the container and positioned on the same long edge is subjected to high point alignment with the corresponding target lock hole.
In the specific implementation, because the lifting appliance is a rigid body, after the long edge on one side is successfully aligned, the long edge on the other side is successfully aligned naturally. Preferably, the image data is provided by an image acquisition device, the inertial navigation data comprises angular velocity and linear acceleration data, and the inertial navigation data is provided by an inertial navigation unit.
Preferably, the end effector is mounted on a lock knob of the spreader.
Preferably, the determining the poses of the four sets of data acquisition devices in the world coordinate system by using a pose detection algorithm comprises:
synchronizing the image data and the inertial navigation data, and abstracting and transforming the image data and the inertial navigation data;
fusing the image data and the inertial navigation data to obtain the scale of the image acquisition equipment, the direction of the gravity vector and the bias of the inertial measurement unit;
performing joint optimization on a residual error item constructed based on the image data and a residual error item constructed based on the inertial navigation data, and obtaining an optimal state vector by adopting a nonlinear optimization mode;
and performing closed-loop detection by using a DBoW2 algorithm, filtering the detected result, and adding the filtered result into the optimization queue again to participate in pose optimization so as to obtain an optimal pose result.
Preferably, external parameters of the coordinate system of the image acquisition device and the coordinate system of the inertial navigation unit are obtained by calibration using Kalibr calibration software before installation.
In summary, the system and method for aligning a lifting device provided by this embodiment acquire image data and inertial navigation data of the lifting device through four sets of data acquisition devices disposed on four end actuators of the lifting device, acquire poses of the four sets of data acquisition devices in a world coordinate system, further acquire poses of the four end actuators, acquire real-time poses of the lifting device through pose fitting a spatial rectangle, acquire poses of four target lock holes through the image data and the inertial navigation data, acquire real-time poses of a container through pose fitting the spatial rectangle, determine a real-time relative position relationship between the lifting device and the container according to the real-time poses of the lifting device and the container, complete low-point alignment before completing alignment between the other end actuator and the corresponding target lock hole, and provide real-time pose information of the lifting device and the container with high accuracy through low-delay, the alignment success rate is improved, so that the automatic loading and unloading of the container are realized, and the operation efficiency of the hoisting equipment is improved;
furthermore, the image data of the lifting appliance is acquired through the data acquisition equipment, and then the real-time pose of the lifting appliance and the real-time pose of the target lock hole are acquired, so that the real-time pose of the container is acquired, the relative pose relation between the lifting appliance and the container is accurately acquired, and the alignment success rate and the alignment accuracy are improved.
Furthermore, the containers are placed at a low point first and then at a high point, so that the automatic dynamic placement of the containers is realized.
Although the present invention has been described with respect to the preferred embodiments, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (10)
1. The utility model provides a lifting device counterpoint system which characterized in that includes:
the data acquisition module comprises four groups of data acquisition equipment which are respectively arranged on four tail end execution mechanisms of the lifting appliance, and the four groups of data acquisition equipment are used for acquiring image data and inertial navigation data of the lifting appliance;
the lifting appliance pose detection module is used for determining poses of the four groups of data acquisition equipment in a world coordinate system by using a pose detection algorithm, obtaining poses of four tail end execution mechanisms of the lifting appliance in the world coordinate system according to pose transformation of the four groups of data acquisition equipment in the world coordinate system, and fitting the poses of the four tail end execution mechanisms of the lifting appliance to a space rectangle so as to obtain a real-time pose of the lifting appliance;
the target position and pose detection module is used for obtaining poses of the four target lock holes relative to the four groups of data acquisition equipment through the image data and the inertial navigation data of the lifting appliance, and fitting the poses of the four target lock holes to a space rectangle so as to obtain real-time poses of the container;
and the alignment module is used for determining the real-time relative position relationship between the spreader and the container according to the real-time pose of the spreader and the real-time pose of the container, performing low-point alignment on the tail end actuating mechanism closest to the container and the corresponding target lock hole, and then completing container completion after the other tail end actuating mechanism closest to the container and positioned on the same long edge as the tail end actuating mechanism closest to the container and the corresponding target lock hole are aligned to each other.
2. The lifting device aligning system of claim 1, wherein the data acquisition device comprises an image acquisition device for providing image data and an inertial navigation unit for providing angular velocity and linear acceleration data.
3. The lifting device alignment system as recited in claim 2, wherein the end effector is mounted on a lock knob of the spreader.
4. The lifting device alignment system according to claim 2, wherein the spreader pose detection module comprises a data preprocessing unit, an initialization unit, a rear-end nonlinear optimization unit and a closed-loop detection unit;
the preprocessing unit is used for synchronizing the image data and the inertial navigation data and abstracting and transforming the image data and the inertial navigation data;
the initialization unit is used for fusing the image data and the inertial navigation data to obtain the scale of the image acquisition equipment, the direction of a gravity vector and the bias of the inertial measurement unit;
the rear-end nonlinear optimization unit is used for performing combined optimization on a residual error item constructed based on the image data and a residual error item constructed based on the inertial navigation data, and obtaining an optimal state vector by adopting a nonlinear optimization mode;
the closed-loop detection unit is used for performing closed-loop detection by using a DBoW2 algorithm, filtering the detected result and adding the filtered result into the optimization queue again to participate in pose optimization so as to obtain an optimal pose result.
5. The lifting device aligning system of claim 2, wherein the coordinate system of the image capturing device and the external parameters of the coordinate system of the inertial navigation unit are obtained by calibration using Kalibr calibration software before installation.
6. A method for aligning hoisting equipment is characterized by comprising the following steps:
acquiring image data and inertial navigation data of a lifting appliance;
determining poses of four groups of data acquisition equipment in a world coordinate system by using a pose detection algorithm, obtaining poses of four tail end execution mechanisms of the lifting appliance according to pose transformation of the four groups of data acquisition equipment in the world coordinate system, and fitting the poses of the four tail end execution mechanisms of the lifting appliance to a space rectangle so as to obtain a real-time pose of the lifting appliance;
the poses of the four target lock holes relative to the four groups of data acquisition equipment are obtained through the image data and the inertial navigation data of the lifting appliance, and the poses of the four target lock holes are fitted to a space rectangle, so that the real-time pose of the container is obtained;
and determining the real-time relative position relation between the spreader and the container according to the real-time pose of the spreader and the real-time pose of the container, and after the end actuating mechanism closest to the container is subjected to low point alignment with the corresponding target lock hole, finishing the container finishing work after the other end actuating mechanism closest to the container and positioned on the same long edge is subjected to high point alignment with the corresponding target lock hole.
7. The method of claim 6, wherein the image data is provided by an image capture device, the inertial navigation data comprises angular velocity and linear acceleration data, and the inertial navigation data is provided by an inertial navigation unit.
8. The alignment method of hoisting equipment according to claim 7, wherein the end effector is mounted on a lock knob of the spreader.
9. The method of claim 7, wherein determining the poses of the four sets of data acquisition devices in the world coordinate system using a pose detection algorithm comprises:
synchronizing the image data and the inertial navigation data, and abstracting and transforming the image data and the inertial navigation data;
fusing the image data and the inertial navigation data to obtain the scale of the image acquisition equipment, the direction of a gravity vector and the bias of an inertial measurement unit;
performing joint optimization on a residual error item constructed based on the image data and a residual error item constructed based on the inertial navigation data, and obtaining an optimal state vector by adopting a nonlinear optimization mode;
and performing closed-loop detection by using a DBoW2 algorithm, filtering the detected result, and adding the filtered result into the optimization queue again to participate in pose optimization so as to obtain an optimal pose result.
10. The alignment method of hoisting equipment as recited in claim 7, wherein the coordinate system of the image capturing device and the external parameters of the coordinate system of the inertial navigation unit are calibrated before installation by using Kalibr calibration software.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010631776.9A CN111704036B (en) | 2020-07-03 | 2020-07-03 | Lifting equipment alignment system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010631776.9A CN111704036B (en) | 2020-07-03 | 2020-07-03 | Lifting equipment alignment system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111704036A true CN111704036A (en) | 2020-09-25 |
| CN111704036B CN111704036B (en) | 2022-03-01 |
Family
ID=72546412
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010631776.9A Active CN111704036B (en) | 2020-07-03 | 2020-07-03 | Lifting equipment alignment system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111704036B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113487672A (en) * | 2021-06-17 | 2021-10-08 | 北京控制工程研究所 | Object feature extraction method for behavior association in complex scene |
| CN113902814A (en) * | 2021-12-07 | 2022-01-07 | 武汉港迪智能技术有限公司 | Unified calibration method for multiple cameras on automatic crane sling |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| FI895892A0 (en) * | 1989-12-08 | 1989-12-08 | Kone Oy | Method and apparatus for locating container for lifting purpose |
| JP2006273532A (en) * | 2005-03-30 | 2006-10-12 | Mitsui Eng & Shipbuild Co Ltd | Crane for loading/unloading container |
| CN2890845Y (en) * | 2005-05-13 | 2007-04-18 | 高旭 | Automatic positioning device for container truck under quayside bridge |
| CN101251381A (en) * | 2007-12-29 | 2008-08-27 | 武汉理工大学 | Dual container positioning system based on machine vision |
| CN101289156A (en) * | 2008-05-30 | 2008-10-22 | 浙江工业大学 | Intelligent container sling based on omniberaing vision sensor |
| CN101934982A (en) * | 2010-09-29 | 2011-01-05 | 上海海事大学 | Positioning method of twin-lift spreader bridge crane based on image sensor |
| CN102452611A (en) * | 2010-10-21 | 2012-05-16 | 上海振华重工(集团)股份有限公司 | Detection method and detection device for space attitude of suspender of container crane |
| JP2014174021A (en) * | 2013-03-08 | 2014-09-22 | Mitsui Eng & Shipbuild Co Ltd | Center of gravity measuring system and center of gravity measuring method for a truck loading container |
| CN105712206A (en) * | 2016-04-27 | 2016-06-29 | 上海振华重工电气有限公司 | Container spreader pose detection system and method |
| CN105956619A (en) * | 2016-04-27 | 2016-09-21 | 浙江工业大学 | Container lockhole coarse positioning and tracking method |
| CN106096606A (en) * | 2016-06-07 | 2016-11-09 | 浙江工业大学 | A kind of container profile localization method based on fitting a straight line |
| CN106643661A (en) * | 2015-10-28 | 2017-05-10 | 上海振华重工电气有限公司 | Rail-mounted crane suspender pose detection system based on machine vision and method thereof |
| CN110902570A (en) * | 2019-11-25 | 2020-03-24 | 上海驭矩信息科技有限公司 | Dynamic measurement method and system for container loading and unloading operation |
-
2020
- 2020-07-03 CN CN202010631776.9A patent/CN111704036B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| FI895892A0 (en) * | 1989-12-08 | 1989-12-08 | Kone Oy | Method and apparatus for locating container for lifting purpose |
| JP2006273532A (en) * | 2005-03-30 | 2006-10-12 | Mitsui Eng & Shipbuild Co Ltd | Crane for loading/unloading container |
| CN2890845Y (en) * | 2005-05-13 | 2007-04-18 | 高旭 | Automatic positioning device for container truck under quayside bridge |
| CN101251381A (en) * | 2007-12-29 | 2008-08-27 | 武汉理工大学 | Dual container positioning system based on machine vision |
| CN101289156A (en) * | 2008-05-30 | 2008-10-22 | 浙江工业大学 | Intelligent container sling based on omniberaing vision sensor |
| CN101934982A (en) * | 2010-09-29 | 2011-01-05 | 上海海事大学 | Positioning method of twin-lift spreader bridge crane based on image sensor |
| CN102452611A (en) * | 2010-10-21 | 2012-05-16 | 上海振华重工(集团)股份有限公司 | Detection method and detection device for space attitude of suspender of container crane |
| JP2014174021A (en) * | 2013-03-08 | 2014-09-22 | Mitsui Eng & Shipbuild Co Ltd | Center of gravity measuring system and center of gravity measuring method for a truck loading container |
| CN106643661A (en) * | 2015-10-28 | 2017-05-10 | 上海振华重工电气有限公司 | Rail-mounted crane suspender pose detection system based on machine vision and method thereof |
| CN105712206A (en) * | 2016-04-27 | 2016-06-29 | 上海振华重工电气有限公司 | Container spreader pose detection system and method |
| CN105956619A (en) * | 2016-04-27 | 2016-09-21 | 浙江工业大学 | Container lockhole coarse positioning and tracking method |
| CN106096606A (en) * | 2016-06-07 | 2016-11-09 | 浙江工业大学 | A kind of container profile localization method based on fitting a straight line |
| CN110902570A (en) * | 2019-11-25 | 2020-03-24 | 上海驭矩信息科技有限公司 | Dynamic measurement method and system for container loading and unloading operation |
Non-Patent Citations (2)
| Title |
|---|
| 吴照阳: ""集装箱码头自动化轨道吊定位技术及参数设定"", 《水运工程》 * |
| 高飞等: "基于空间平面约束的视觉定位模型研究", 《仪器仪表学报》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113487672A (en) * | 2021-06-17 | 2021-10-08 | 北京控制工程研究所 | Object feature extraction method for behavior association in complex scene |
| CN113487672B (en) * | 2021-06-17 | 2024-02-20 | 北京控制工程研究所 | Object feature extraction method for behavior association under complex scene |
| CN113902814A (en) * | 2021-12-07 | 2022-01-07 | 武汉港迪智能技术有限公司 | Unified calibration method for multiple cameras on automatic crane sling |
| CN113902814B (en) * | 2021-12-07 | 2022-03-15 | 武汉港迪智能技术有限公司 | Unified calibration method for multiple cameras on automatic crane sling |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111704036B (en) | 2022-03-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111024066B (en) | Unmanned aerial vehicle vision-inertia fusion indoor positioning method | |
| CN109993113B (en) | Pose estimation method based on RGB-D and IMU information fusion | |
| CN109676604B (en) | Robot curved surface motion positioning method and motion positioning system thereof | |
| CN112634451B (en) | Outdoor large-scene three-dimensional mapping method integrating multiple sensors | |
| US11199414B2 (en) | Method for simultaneous localization and mapping | |
| WO2021180128A1 (en) | Rgbd sensor and imu sensor-based positioning method | |
| US9990726B2 (en) | Method of determining a position and orientation of a device associated with a capturing device for capturing at least one image | |
| CN111704036B (en) | Lifting equipment alignment system and method | |
| CN110726406A (en) | Improved nonlinear optimization monocular inertial navigation SLAM method | |
| US20220051031A1 (en) | Moving object tracking method and apparatus | |
| CN113763479B (en) | Calibration method of refraction and reflection panoramic camera and IMU sensor | |
| CN108225328A (en) | A kind of interior three dimensional data collection method | |
| CN110728716B (en) | Calibration method and device and aircraft | |
| Huttunen et al. | A monocular camera gyroscope | |
| CN109785428A (en) | A kind of handheld three-dimensional method for reconstructing based on polymorphic constrained Kalman filter | |
| CN108827287A (en) | A kind of robust vision SLAM system under complex environment | |
| Ling et al. | RGB-D inertial odometry for indoor robot via keyframe-based nonlinear optimization | |
| CN115574816B (en) | Bionic vision multi-source information intelligent perception unmanned platform | |
| CN116659490A (en) | Low cost vision-inertial fusion SLAM method | |
| Zhang et al. | Monocular visual-inertial and robotic-arm calibration in a unifying framework | |
| Lee et al. | A new localization method for mobile robot by data fusion of vision sensor data and motion sensor data | |
| CN115797490A (en) | Drawing construction method and system based on laser vision fusion | |
| CN109459046A (en) | The positioning and air navigation aid of suspending underwater autonomous navigation device | |
| CN115344033A (en) | Monocular camera/IMU/DVL tight coupling-based unmanned ship navigation and positioning method | |
| CN109542094A (en) | Mobile robot visual point stabilization without desired image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |