CN113129383B

CN113129383B - Hand-eye calibration method and device, communication equipment and storage medium

Info

Publication number: CN113129383B
Application number: CN202110276024.XA
Authority: CN
Inventors: 田璐璐; 苏世龙; 樊则森; 雷俊; 丁沛然; 马栓鹏
Original assignee: China Construction Science and Technology Group Co Ltd
Current assignee: China Construction Science and Technology Group Co Ltd
Priority date: 2021-03-15
Filing date: 2021-03-15
Publication date: 2024-06-14
Anticipated expiration: 2041-03-15
Also published as: CN113129383A

Abstract

The application is applicable to the technical field of computers, and provides a hand-eye calibration method, a device, communication equipment and a storage medium, comprising the following steps: obtaining a photo group obtained by shooting by a camera group, wherein the camera group at least comprises two cameras and is used for cooperatively shooting a target field of view area; the photo group comprises photos obtained by shooting each camera in the camera group respectively; each photo in the photo group is spliced to obtain a global photo corresponding to the target visual field area; and calibrating the corresponding relation between the pixel coordinate system of the global photo and the mechanical arm coordinate system according to the global photo to obtain a hand-eye calibration result. The embodiment of the application can conveniently and accurately realize hand-eye calibration based on multiple cameras.

Description

Hand-eye calibration method and device, communication equipment and storage medium

Technical Field

The application belongs to the technical field of computers, and particularly relates to a hand-eye calibration method, a hand-eye calibration device, communication equipment and a storage medium.

Background

Along with the increasing degree of automation of manufacturing industry and industrial production lines, the automatic production links of manual completion, classification, feeding, stacking and quality inspection are more and more replaced by using cameras and mechanical arms in factory production lines. Generally, a photo shot by a camera is firstly acquired for identification, and object positioning is completed; and then, according to the space coordinate relation between the camera and the mechanical arm, converting the pixel coordinates of the object on the photo into mechanical arm coordinates under the mechanical arm coordinate system pair so as to instruct the mechanical arm to finish object grabbing according to the mechanical arm coordinates.

In order to achieve accurate gripping of an object, the visual coordinates of a camera (specifically, the pixel coordinates in a photo taken by the camera) and the coordinates of a mechanical arm are usually required to be calibrated in advance, and this process is called hand-eye calibration. The existing hand-eye calibration method generally calibrates the visual coordinates of a single camera and the mechanical arm coordinates, but is difficult to calibrate the hand-eye conveniently and accurately based on a multi-camera visual system.

Disclosure of Invention

In view of the above, the embodiments of the present application provide a hand-eye calibration method, device, communication apparatus, and storage medium, so as to solve the problem in the prior art how to conveniently and accurately implement hand-eye calibration based on multiple cameras.

A first aspect of an embodiment of the present application provides a hand-eye calibration method, including:

obtaining a photo group obtained by shooting by a camera group, wherein the camera group at least comprises two cameras and is used for cooperatively shooting a target field of view area; the photo group comprises photos obtained by shooting each camera in the camera group respectively;

Each photo in the photo group is spliced to obtain a global photo corresponding to the target visual field area;

And calibrating the corresponding relation between the pixel coordinate system of the global photo and the mechanical arm coordinate system according to the global photo to obtain a hand-eye calibration result.

A second aspect of an embodiment of the present application provides a hand-eye calibration device, including:

the camera group acquisition unit is used for acquiring a photo group shot by the camera group, wherein the camera group at least comprises two cameras and is used for cooperatively shooting a target field of view area; the photo group comprises photos obtained by shooting each camera in the camera group respectively;

the splicing unit is used for splicing all the photos in the photo group to obtain global photos corresponding to the target visual field area;

And the calibration unit is used for calibrating the corresponding relation between the pixel coordinate system of the global photo and the mechanical arm coordinate system according to the global photo to obtain a hand-eye calibration result.

A third aspect of the embodiments of the present application provides a communication device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, which when executed by the processor causes the communication device to perform the steps of the hand-eye calibration method.

A fourth aspect of the embodiments of the present application provides a computer readable storage medium storing a computer program which, when executed by a processor, causes a communication device to implement the steps of the hand-eye calibration method.

A fifth aspect of an embodiment of the present application provides a computer program product for, when run on a communication device, causing the communication device to perform the hand-eye calibration method as described in the first aspect above.

Compared with the prior art, the embodiment of the application has the beneficial effects that: in the embodiment of the application, for a camera group comprising at least two cameras, a photo group obtained by shooting the camera group can be obtained, each photo in the photo group is spliced to obtain a corresponding global photo, and then the corresponding relation between the pixel coordinate system of the global photo and the mechanical arm coordinate system is calibrated according to the global photo to obtain a hand-eye calibration result. Because the hand-eye calibration can be carried out based on the global photo formed by splicing the plurality of photos shot by the multiple cameras, compared with the prior art which can only be limited to synchronous hand-eye calibration between a single camera and a mechanical arm, the hand-eye calibration based on the multiple cameras can be conveniently and effectively realized so as to meet the actual requirements in industrial production.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings used in the description of the embodiments or the prior art will be briefly described below.

Fig. 1 is a schematic diagram of an application scenario corresponding to a hand-eye calibration method according to an embodiment of the present application;

FIG. 2 is a schematic diagram of an implementation flow of a hand-eye calibration method according to an embodiment of the present application;

FIG. 3 is an overall view of a photo group prior to stitching provided by an embodiment of the present application;

FIG. 4 is a schematic diagram of a global photograph provided by an embodiment of the present application;

FIG. 5 is an exemplary diagram of a preset calibration object according to an embodiment of the present application;

fig. 6 is a schematic diagram of an application scenario corresponding to another hand-eye calibration method according to an embodiment of the present application;

FIG. 7 is a schematic diagram of a hand-eye calibration device according to an embodiment of the present application;

fig. 8 is a schematic diagram of a communication device according to an embodiment of the present application.

Detailed Description

In the following description, for purposes of explanation and not limitation, specific details are set forth such as the particular system architecture, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

In order to illustrate the technical scheme of the application, the following description is made by specific examples.

It should be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

It is also to be understood that the terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

It should be further understood that the term "and/or" as used in the present specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.

As used in this specification and the appended claims, the term "if" may be interpreted as "when..once" or "in response to a determination" or "in response to detection" depending on the context. Similarly, the phrase "if a determination" or "if a [ described condition or event ] is detected" may be interpreted in the context of meaning "upon determination" or "in response to determination" or "upon detection of a [ described condition or event ]" or "in response to detection of a [ described condition or event ]".

In addition, in the description of the present application, the terms "first," "second," "third," etc. are used merely to distinguish between descriptions and should not be construed as indicating or implying relative importance.

Currently, in industrial production lines, most of the industrial production lines use only one camera to deploy a visual frame, and the visual field range is usually limited, so that the size of an object to be identified is limited, and the large-size object cannot be accurately identified. In order to realize the identification of the large-size object, the embodiment of the application adopts a multi-camera to deploy the visual framework, thereby expanding the identification visual field range and realizing the identification of the large-size object. However, the existing hand-eye calibration method is generally limited to calibration between a single camera and a mechanical arm, even if calibration between a plurality of cameras and the mechanical arm exists, one camera is selected from a plurality of cameras, after the camera and the mechanical arm are calibrated, the relative positions of other cameras and the calibrated camera are further determined, namely, even if a vision system with a plurality of cameras exists, the calibration parameters of each camera are determined on the basis of the calibration between the single camera and the mechanical arm, synchronous calibration cannot be realized, the operation of the process is complex, and certain errors may exist in the relative position determination process between the plurality of cameras. Therefore, the technical problem that the hand-eye calibration based on multiple cameras is difficult to realize conveniently and accurately exists in the prior art. In order to solve the technical problem, the embodiment of the application provides a hand-eye calibration method, a device, communication equipment and a storage medium, wherein for a camera group comprising at least two cameras, a photo group obtained by shooting the camera group can be obtained, each photo in the photo group is spliced to obtain a corresponding global photo, and then a corresponding relation between a pixel coordinate system of the global photo and a mechanical arm coordinate system is calibrated according to the global photo to obtain a hand-eye calibration result. Because the hand-eye calibration can be carried out based on the global photo formed by splicing the plurality of photos shot by the multiple cameras, compared with the prior art which can only be limited to synchronous hand-eye calibration between a single camera and a mechanical arm, the hand-eye calibration based on the multiple cameras can be conveniently and effectively realized so as to meet the actual requirements in industrial production.

An exemplary schematic diagram of an application scenario corresponding to a hand-eye calibration method provided by the embodiment of the application is shown in fig. 1. Wherein a vision frame is deployed on a conveyor chain conveyor belt for conveying objects, the vision frame comprising at least two cameras (the vision frame shown in fig. 1 comprises 4 cameras) to form a camera group. Each camera of the camera set is mounted at a different position in the same plane on top of the vision frame (which plane is parallel to the plane of the conveyor belt) for cooperatively capturing a field of view, i.e., a target field of view, within the vision frame. Furthermore, an independent light supplementing lamp can be arranged in the visual frame so as to ensure the brightness of shooting by the camera.

Embodiment one:

Fig. 2 is a schematic flow chart of a hand-eye calibration method according to an embodiment of the present application, where an execution subject of the hand-eye calibration method may be a communication device, for example, a computer, a server, etc., and the details are as follows:

In S201, a group of photographs taken by a camera group is obtained, where the camera group includes at least two cameras for cooperatively taking a target field of view; the photo group comprises photos obtained by shooting each camera in the camera group respectively.

The camera set in the embodiment of the application is used for cooperatively shooting a target field of view area, such as an inner area of a visual frame shown in fig. 1, and comprises at least two cameras. The collaborative shooting target field of view region refers to that different cameras in the camera set are respectively responsible for shooting sub-regions at different positions in the target field of view region. Furthermore, an overlapping area exists between adjacent subareas, the overlapping area enables a common view area existing when the camera group shoots, and information of the common view area can facilitate subsequent photo splicing. The photo group in the embodiment of the application specifically comprises photos taken by each camera in the camera group. For example, if the camera group includes four cameras, camera a, camera B, camera C, and camera D, the photo group includes four photos, photo a, photo B, photo C, and photo D, respectively.

Specifically, in the embodiment of the application, each camera in the camera group can be synchronously triggered to take a picture through the infrared sensor arranged at the entrance of the target visual field area, and each photo generated by each camera is acquired, so that the photo group is obtained. In another embodiment, a group of photographs taken by a camera group may be stored in a designated memory space, and the group of photographs may be obtained by accessing the designated memory space.

Further, in order to ensure consistent shooting quality of each photo in the photo group, the exposure degree, signal jitter removal and other software and hardware parameters of each camera in the camera group in the embodiment of the application are kept consistent. Further, in order to ensure the synchronism of multi-camera shooting in the camera set, synchronous starting of the cameras can be performed through an external hard trigger mode, and photo storage of each camera can be performed through a multithreaded picture container, so that the synchronism of camera shooting is ensured, photos generated and stored at almost the same moment (only microsecond order) in the photo set are ensured, and the calibration accuracy is ensured. In one example, if the camera set of the embodiment of the present application uses the multithreaded picture container for storage, the maximum value of the running speed of the conveying chain in the application scenario as shown in fig. 1 is limited to 45000 rpm in the subsequent practical use process, that is, when the running speed of the conveying chain is less than or equal to 45000 rpm, the picture will not generate dislocation ghosts, and when the rotating speed is higher than 45000 rpm, the picture will generate dislocation ghosts.

In S202, each photo in the photo group is spliced to obtain a global photo corresponding to the target field of view region.

In the embodiment of the application, each photo in the photo group is spliced, namely, the photos corresponding to different subregions in the target visual field area are spliced, and the images of the common visual field area are fused and displayed to obtain a global photo corresponding to the target visual field area, wherein the global photo is a photo capable of completely presenting the image information in the target visual field area. Specifically, during calibration, a plurality of characteristic pictures are placed in the common view area between the cameras, so that each picture in a picture group obtained by shooting of the camera group contains information of the characteristic pictures, and then the information of the characteristic pictures is used as a reference standard, so that the splicing between the pictures can be realized.

For example, if four photographs taken by four cameras in the visual frame shown in fig. 1 are sequentially arranged according to the positional relationship between each other, an overall view of the four photographs (photograph a, photograph B, photograph C, photograph D) before they are spliced is shown in fig. 3, wherein a photograph area of each photograph is divided into two parts according to a broken line as shown in fig. 3, and an area including information of a feature photograph is the above-mentioned common view area. And according to the information of the characteristic pictures in the common area, splicing the four pictures shown in the figure 3, and fusing the images of the common view area to obtain the global picture which is obtained by splicing as shown in the figure 4. In one embodiment, for each photo in the group of photos, the common view area is at least one third of the total area of the photo, thereby ensuring a subsequent stitching effect.

In S203, according to the global photo, the corresponding relation between the pixel coordinate system of the global photo and the mechanical arm coordinate system is calibrated, so as to obtain a hand-eye calibration result.

After the global photo is obtained, the corresponding relation between the pixel coordinate system of the global photo and the mechanical arm coordinate system can be calibrated based on the global photo, and the corresponding relation is used as a hand-eye calibration result, so that the corresponding mechanical arm coordinate can be determined according to the corresponding relation after the pixel coordinate of the identified object to be grabbed is determined, and the object to be grabbed can be accurately grabbed. In one embodiment, a plurality of calibration points may be preset in the target field of view area, and a coordinate conversion equation is established to calculate according to the pixel coordinates of the calibration points corresponding to the global photo and the mechanical arm coordinates output when the mechanical arm moves the points to the calibration points, so as to obtain the corresponding relationship between the pixel coordinate system of the global photo and the mechanical arm coordinate system.

Optionally, the step S202 specifically includes:

S20201: performing feature point matching and image registration on every two adjacent photos in the photo group, and determining a transformation matrix between the adjacent photos in the photo group;

s20202: and splicing all the photos in the photo group according to the transformation matrix between the adjacent photos in the photo group to obtain a global photo.

In the embodiment of the application, the characteristic points refer to points with sharp change of the gray value of the image or points with larger curvature on the edge of the image; intuitively, feature points are points that can be robustly extracted from each photo by photographing the same scene from different angles. The feature points in the embodiment of the present application are pixel points corresponding to the feature pictures contained in the common region as shown in fig. 3.

In S20201, for each photo in the photo group, feature points may be extracted from the respective common areas, respectively. The feature point extraction can be realized by a common feature point extraction algorithm, such as a Scale-invariant feature transform (Scale-INVARIANT FEATURE TRANSFORM, SIFT) algorithm, an acceleration robust feature (Speeded Up Robust Features, SURF) algorithm, a fast and rotating-oriented binary robust independent basic feature (Oriented FAST and Rotated BRIEF, ORB) algorithm, and the like. After extracting the characteristic points, carrying out characteristic point matching on two adjacent photos in the photo group to obtain characteristic point pairs; then, image registration is performed according to the feature point pairs, and a transformation matrix between the two photos is determined. For every two adjacent photos in the photo group, the transformation matrix between the two photos can be determined through the characteristic point matching and the image registration processing, and finally, the transformation matrix between all the adjacent photos in the photo group can be determined.

Specifically, the feature point matching and image registration process of two adjacent photos can be realized through the following steps:

A1: and carrying out feature point matching according to the feature points respectively extracted from the two photos to obtain the feature point pairs with the specified number. Specifically, feature point matching refers to matching extracted feature points by using feature descriptors, so as to match the same feature points extracted from different photos. For example, in the SIFI algorithm, each feature point may be described by a 128-dimensional feature vector, and by comparing feature vectors of two feature points respectively from two photos, and calculating the euclidean distance between them, it can be determined whether the two feature points can be matched. After the feature point matching obtains the preliminary feature point pairs, the feature point pairs may be further screened to obtain a specified number (e.g., 16 pairs) of valid feature point pairs. In the screening process of the feature point pairs, some feature point pairs with poor quality can be specifically removed, or only one group of feature point pairs is reserved in a plurality of feature point pairs with short distances.

A2: and carrying out image registration according to the appointed number of feature point pairs to obtain a transformation matrix between the two photos. After the feature point pairs are obtained, the relative positions of the images need to be obtained according to the feature point pairs, and the process is called image registration. Specifically, in the embodiment of the present application, the relative position between two photographs is represented by a transformation matrix between two photographs, and the transformation matrix can be obtained by the following formula:

Wherein, A homogeneous representation of pixel coordinates representing the first photograph; /(I)A homogeneous representation of pixel coordinates representing the second photograph; the first photo and the second photo are the two photos respectively. H is a transformation matrix, also called homography matrix, which is a 3×3 matrix, and where H _3,3 =1, the transformation matrix is thus an eight-parameter matrix. The pixel coordinates of the feature points in each group of feature point pairs are substituted into the formula, and the equation is solved, so that a transformation matrix between two photos can be obtained respectively, and registration between images is realized. Specifically, since each feature point has x and y two pixel coordinates, four points are needed to solve the solution of the eight-parameter matrix, and meanwhile, the fact that the feature point pair obtained by utilizing the feature vector matching is possibly mismatched is considered, so that a random sample consistency (Random Sample Consensus, RANSAC) algorithm can be used for solving, and an optimal transformation matrix is obtained.

In S20202, the process of step S20201 obtains the transformation matrices between all adjacent photos in the photo group, that is, each photo in the photo group is sequentially spliced after position conversion according to the transformation matrices, so as to obtain a global photo. In one embodiment, a photo may be designated in a photo group as a reference photo, and other photos in the photo group are sequentially transformed into the reference photo pixel coordinate system according to a transformation matrix between adjacent photos to be spliced, so as to obtain the global photo.

Further, in order to reduce error accumulation in the photo stitching process, the estimated fixed matrix may be jointly optimized by a beam adjustment method. Further, after the global photos are obtained through splicing, horizontal correction or vertical correction of the global photos can be performed according to the actual splicing effect.

In the embodiment of the application, the transformation matrix between the adjacent photos in the photo group can be accurately determined through the feature point matching and the image registration, so that the photos can be accurately spliced according to the transformation matrix to obtain an accurate global photo, namely, an accurate basis is provided for the subsequent calibration process, and the accuracy of the hand-eye calibration method is further improved.

Optionally, step S20202 described above includes:

B1: according to the transformation matrix between adjacent photos in the photo group, performing projection transformation on each photo in the photo group to obtain a preliminary spliced photo;

b2: and performing joint fusion processing on the preliminary spliced photos to obtain global photos.

In B1, after the transformation matrix between adjacent photos in the photo group is obtained, the relative positional relationship of each camera in the camera group can be obtained. If the pictures are spliced directly according to the transformation matrices, the consistency of the view field may be damaged, so that the global pictures obtained by splicing are incoherent. Further, if obvious brightness change exists in the obtained preliminary spliced photo, exposure compensation treatment can be performed on the preliminary spliced photo, so that the overall brightness of the processed preliminary spliced photo is consistent.

In the B2, in the preliminary stitched photo obtained by projective transformation, there may be an obvious transition trace in a region opposite to the common view region of the photo before stitching (i.e., a photo overlapping region), and the transition trace is a stitching seam. In the embodiment of the application, a fusion algorithm is adopted for a plurality of pixels near the seam so as to remove repeated characteristic points in the primary spliced photo, namely seam fusion processing, namely pixel seam beautifying processing. Through the joint fusion processing, defects such as image dislocation, artifacts and the like in the global photo can be effectively removed, and the global photo with low feature point repetition rate and high splicing quality is obtained.

Further, the above-mentioned seam fusion process may be composed of a seam determination algorithm and a fusion algorithm. The seam determining algorithm is an algorithm for searching the seam in the preliminary spliced photo, such as a point-by-point method, a dynamic programming method, a graph cutting method and the like. By means of the method and the device, the edge joint is determined point by point, the point by point method is relatively simple, and processing efficiency can be improved. The fusion algorithm is an algorithm for carrying out fusion calculation on pixels near the seam on the basis of determining the seam. The fusion algorithm can be an eclosion fusion algorithm, a Laplacian fusion algorithm and the like. The eclosion fusion is to calculate the weight of the position near the seam according to the distance between the seam and the position, and the Laplace fusion algorithm is equivalent to calculating the components of different frequencies of the image, and then fusing according to the frequencies.

In the embodiment of the application, the picture splicing quality can be further improved through projection transformation and seam fusion processing, and the accuracy of the global picture is further improved, so that the accuracy of subsequent calibration is further improved.

In some embodiments, a virtual focal length and a translational rotation matrix (i.e., internal and external parameters of each camera) corresponding to each camera in the camera group may be estimated according to a transformation matrix between adjacent photos obtained by image registration (or further combined with parameters used in a stitching optimization process such as a stitching fusion process), so as to form a fixed matrix, and the fixed matrix is stored. So that when the global photo needs to be acquired again under the condition that the camera group is not changed, each photo in the photo group can be directly transformed and spliced according to the stored fixed matrix to obtain the global photo.

Optionally, the target field of view area includes a preset calibration object, where the preset calibration object includes a preset number of calibration points, and step S203 includes:

C1: acquiring global camera parameters corresponding to the global photo; the global camera parameters at least comprise estimated internal parameters of the global camera;

c2: determining global camera coordinates corresponding to each calibration point according to the pixel coordinates of each calibration point in the pixel coordinate system of the global photo and the global camera parameters; the global camera coordinates are estimated camera coordinates of the global camera;

and C3: determining the corresponding relation between the global camera coordinate system and the mechanical arm coordinate system according to the global camera coordinate of each calibration point and the read mechanical arm coordinate corresponding to each calibration point; the mechanical arm coordinate is a coordinate of the calibration point in the mechanical arm coordinate system, and the global camera coordinate system is a coordinate system where the global camera coordinate is located;

And C4: and determining the corresponding relation between the pixel coordinate system of the global photo and the mechanical arm coordinate system according to the global camera parameters and the corresponding relation between the global camera coordinate system and the mechanical arm coordinate system, and obtaining a hand-eye calibration result.

In the embodiment of the application, before the hand-eye calibration method of the embodiment of the application is executed, a preset calibration object is placed in the target visual field area in advance, and the preset calibration object comprises a preset number of calibration points. In one embodiment, the predetermined calibration object may be a sheet of paper or a calibration plate containing a predetermined number of calibration points. In another embodiment, the preset calibration object may include n sheets or n calibration plates, where m calibration points exist in each sheet or each calibration plate, so that a calibration object formed by combining n sheets or n calibration plates includes n×m calibration points, where n and m are positive integers greater than 0, and n×m is a preset number. For example, as shown in fig. 5, 16 sheets of paper may be tiled in the target field of view to form a preset calibration object, and if 2 cross points exist in each sheet of paper as 2 calibration points, 16×2=32 calibration points exist in the target field of view.

In C1, global camera parameters corresponding to the global photo are obtained, where the global camera parameters at least include predicted internal parameters of the global camera, and may further include the predicted internal and external parameters of the global camera. The global camera is a camera under the estimated global photo view, specifically a virtual camera suspended at a certain spatial position in the visual frame, and does not coincide with any actual camera position in the camera set. Because the internal and external parameters of each camera in the photo group have slight differences and different installation precision, the global photo can be directly used as the calibration basis of the internal and external parameters of the camera under the global visual field, and the photo which is independently shot by a single camera can be used as the calibration basis, so that the accuracy of the hand-eye calibration method is improved. The internal and external parameters of the camera in the embodiment of the application specifically comprise the internal parameters and the external parameters of the camera. Wherein the in-camera parameters are parameters related to the characteristics of the camera itself, such as the focal length of the camera, the pixel size, etc.; the off-camera parameters are parameters in the world coordinate system, such as the position, rotation direction, etc. of the camera.

In the embodiment of the application, the global camera parameters, that is, the predicted internal and external parameters of the global camera, can be specifically obtained by placing a calibration object (such as a checkerboard or the preset calibration object) in a target field area under the current installation condition of the camera set, shooting the target field area, splicing to obtain a specified number (for example, 80) of global photos, and calibrating according to the specified number of global photos. Specifically, given the coordinates (X _w,Y_w,Z_w) of the world coordinate system of the calibration object or the calibration point in the other calibration plate, and the coordinates (X _c,Y_c,Z_c) of the global camera coordinate system, the following relationship exists between the two:

Wherein the rotational displacement matrix I.e. the external parameters of the global camera, r ₁₁～r₃₃ is the element for describing the rotational movement and t _x,t_y,t_z is the element for describing the translational movement. Specifically, the rotational displacement matrix can be used for obtaining external parameters of the global camera, namely the rotational displacement matrix T by selecting a global photo of a calibration object positioned at the center of the pixel as a calibration reference.

Assuming that the coordinates on the pixel coordinate system of the global photograph are (u, v), the following relationship (simply referred to as a pixel-camera relationship) exists between the coordinates of the global camera coordinate system and the pixel coordinate system of the global photograph:

Wherein Z _deep is the depth of field parameter of the global camera, f is the focal length of the global camera, dy is the imaging size of the pixels in the vertical direction, dx is the imaging size of the pixels in the horizontal direction, cx represents the optical center coordinate of the imaging of the global camera in the horizontal direction, cy represents the optical center coordinate of the imaging of the global camera in the vertical direction, and these parameters are the internal parameters of the global camera. The two formulas can be combined to obtain the parameters in the global camera by obtaining the pixel coordinates of the specified points in the global photos, the coordinates of the specified points corresponding to the world coordinate system and the determined external parameters, so that the corresponding relation between the pixel coordinate system of the global photos and the global camera coordinate system is determined. As a possible implementation, the internal and external parameters of the global camera may be calibrated by means of halcon (a standard machine vision algorithm package) calibration assistant. When the internal and external parameters of the global camera are specifically obtained through halcon calibration assistant calibration, the accuracy of the global camera internal and external parameter calibration can be estimated through the residual projection parameters in halcon.

In C2, after the global camera parameters calibrated in advance are obtained, the coordinates of each calibration point in the estimated global camera coordinate system, that is, the global camera coordinates, can be determined according to the internal parameters of the calibrated global camera and the pixel coordinates of each calibration point in the pixel coordinate system of the global photo by the above pixel-camera relational expression. Namely, according to the calibrated parameters in the global camera, the corresponding relation between the pixel coordinate system of the global photo and the global camera coordinate system can be determined, so that the conversion from the pixel coordinate to the global camera coordinate is completed, and the global camera coordinate corresponding to each calibration point is obtained.

In C3, the robot arm coordinates are coordinates of the calibration point in the robot arm coordinate system. The following relational expression (simply referred to as a camera-mechanical arm relational expression) exists between the global camera coordinate system corresponding to the global photo and the mechanical arm coordinate system:

Wherein, (X _c,Y_c,Z_c) is the coordinates of the global camera coordinate system, (X _r,Y_r,Z_r) is the coordinates of the robot arm coordinate system, and R ₁ ^-1 is the inverse of the 4*4 rotational translation matrix between the global camera coordinate system and the robot arm coordinate system.

In the embodiment of the application, after the global camera coordinate of each calibration point in the global camera coordinate system is determined and the mechanical arm coordinate corresponding to each calibration point output by the mechanical arm is read, the two coordinates of each calibration point are substituted into the camera-mechanical arm relational expression to carry out equation solving operation, so that a transformation matrix R ₁ ^-1 between the global camera coordinate system and the mechanical arm self coordinate system can be calibrated, and the corresponding relation between the global camera coordinate system and the mechanical arm coordinate system is determined.

Substituting the internal parameters (Z _deep, f, dy, dx, cx, cy) in the calibrated global camera parameters obtained in the step C1 as known coefficients into the pixel-camera relational expression, so as to determine the corresponding relation from the pixel coordinate system of the current global photo to the global camera coordinate system; substituting the matrix R ₁ ^-1 obtained in the step C3 as a known coefficient into the camera-mechanical arm relation equation, so as to determine the corresponding relation from the current global camera coordinate system to the mechanical arm coordinate system; and combining the two corresponding relations, the corresponding relation from the pixel coordinate system of the global photo to the mechanical arm coordinate system can be determined. In the embodiment of the application, the pixel-camera relation and the camera-mechanical arm relation which are substituted into the determined coefficients can be specifically used as hand-eye calibration results.

In the embodiment of the application, the global camera coordinate of the calibration point can be determined according to the pixel coordinate of the calibration point and the global camera parameter of the global photo by means of the preset calibration object, the corresponding relation between the global camera coordinate system and the mechanical arm coordinate system is obtained according to the global camera coordinate and the mechanical arm coordinate of the calibration point, and finally the corresponding relation between the pixel coordinate system of the global photo and the global camera coordinate system and the corresponding relation between the global camera coordinate system and the mechanical arm coordinate system are determined, so that the two corresponding relations are combined, the corresponding relation between the pixel coordinate system of the global photo and the mechanical arm coordinate system is accurately determined, and the hand and eye calibration is accurately completed.

Further, after determining the correspondence between the pixel coordinate system of the global photo and the global camera coordinate system, and the correspondence between the global camera coordinate system and the mechanical arm coordinate system, the accuracy of the hand-eye calibration result can be verified by placing a fixed object in the application scene as shown in fig. 1, and starting the camera, the conveyor belt and the mechanical arm. Further, if the hand-eye calibration result is determined to have a certain height error through verification, the calibration precision and error can be adjusted by fine-tuning the depth-of-field parameter Z _deep in the global camera internal parameters.

Optionally, in the above-mentioned preset calibration object, the interval between the calibration points is smaller than or equal to a preset threshold.

In the embodiment of the application, the preset number of calibration points are specifically and uniformly distributed in the target visual field area, the target visual field area is paved, and the interval between each calibration point is smaller than a preset threshold value. For example, the lateral distance between calibration points is less than or equal to 15 cm and the vertical distance is less than or equal to 12 cm, thereby improving calibration accuracy. Specifically, the above-described preset number is determined specifically according to the size of the target field of view area on condition that the interval between the calibration points is smaller than or equal to a preset threshold value is satisfied (for example, 5 to 8 calibration points may be set in an area per square meter). Specifically, under the condition that the interval between the calibration points is smaller than or equal to a preset threshold value, the preset number is generally larger than 9, so that the accuracy of the hand-eye calibration method of the embodiment of the application is higher than that of the traditional nine-point calibration method, and the influence of distortion errors possibly existing in the global photo on the accuracy can be compensated.

Optionally, after the step S203, the method further includes:

d1: if the target object is detected to enter the target visual field area, starting the camera group to shoot, and obtaining a target photo group;

d2: splicing the target photo groups to obtain a target global photo containing the complete image information of the target object;

D3: determining a target mechanical arm coordinate corresponding to the target object according to the target global photo and the hand-eye calibration result; the target mechanical arm coordinates are coordinates of grabbing points of the target object corresponding to the mechanical arm coordinates.

In the step D1, when the target object is detected to enter the target field of view area, all cameras in the camera group are synchronously started to shoot, and the target photo group is obtained. Specifically, the target object in the embodiment of the present application is a large-sized object to be grabbed (i.e., the size of the large-sized object is larger than a preset size that can be identified by a single camera, for example, the large-sized object may be a steel bar), and each photo in the target photo group includes a part of image information of the target object. In one embodiment, when a trigger signal exists in an infrared sensor at the entrance of the target field of view area, it is determined that the target object is currently detected to enter the target field of view area, and all cameras in the camera set are synchronously started to take pictures, so that a target photo set is obtained.

In D2, each photo in the target photo group is spliced to obtain a target global photo, and the target global photo includes the complete image information of the target object. Specifically, the target photo group may be directly spliced in the calibration process, that is, the relevant splicing parameters (for example, the parameters used in the transformation matrix, the projection transformation, and the seam fusion process between adjacent photos) determined in step S202 may be obtained.

In D3, after the target global photograph is obtained, an object recognition model (for example, a neural network model obtained by training in advance) may be preset, and the pixel coordinates of the grabbing point of the target object in the target global photograph are determined. And then, according to the hand-eye calibration result (namely the corresponding relation between the pixel coordinate system of the global photo and the mechanical arm coordinate system), converting the pixel coordinate corresponding to the grabbing point into the coordinate corresponding to the grabbing point in the mechanical arm coordinate system, namely the target mechanical arm coordinate. And then, the target mechanical arm coordinates are sent to the mechanical arm, so that the mechanical arm can be instructed to move according to the target mechanical arm coordinates, the grabbing points of the target object are accurately positioned, and the target object is accurately grabbed.

According to the embodiment of the application, after the hand-eye calibration result is obtained, when the target object is detected, the target global photo can be obtained by splicing the target photo group obtained by shooting the camera group, and the coordinates of the target mechanical arm corresponding to the target object can be accurately determined according to the target global photo and the hand-eye calibration result, so that the mechanical arm can be accurately instructed to accurately grasp the target object.

Optionally, before the step C3, the method further includes:

indicating a calibration tool with a preset length carried by the mechanical arm to sequentially point through each calibration point, and reading the coordinates output by the mechanical arm when the calibration tool points are at the positions of the calibration points to obtain the mechanical arm coordinates corresponding to the calibration points; the coordinates output by the mechanical arm are coordinates under a mechanical arm base coordinate system;

correspondingly, after the step D3, the method further includes:

The target mechanical arm coordinates are sent to the mechanical arm to instruct the mechanical arm to grasp the target object according to the target mechanical arm coordinates and preset parameters of the mechanical arm; the mechanical arm preset parameters comprise information of the calibration tool and coordinates of a starting point of a conveying chain.

In the embodiment of the application, a certain distance exists between the mechanical arm base and the visual frame, namely, a certain distance exists between the mechanical arm base and the target visual field area, so that when the hand and the eye are marked, the mechanical arm needs to be provided with a marking tool with a preset length to contact a marking point in a preset object in the target visual field area. For example, the calibration tool of the preset length may be a long spike tool of 1 meter (i.e., a tool of one meter in length and ending in a spike probe), which may be mounted on a flange of a robotic arm.

After the calibration tool with the preset length is installed on the mechanical arm, the mechanical arm is instructed to carry the calibration tool with the preset length, each calibration point on a preset calibration object is sequentially clicked, and when the calibration tool point is positioned at the calibration point, the coordinate output by the mechanical arm is used as the mechanical arm coordinate corresponding to the calibration point. Specifically, the coordinates output by the mechanical arm are mechanical arm base coordinates output to the demonstrator by the mechanical arm, and the coordinates output by the mechanical arm can be obtained by establishing connection with the demonstrator of the mechanical arm.

Correspondingly, in step D3 of the embodiment of the present application, according to the global photograph of the target and the calibration result of the hand and the eye, the determined coordinates of the target mechanical arm are the coordinates of the grabbing point of the target object corresponding to the coordinates in the mechanical arm base coordinate system. After step D3, the target manipulator coordinates are sent to the manipulator, the manipulator is instructed to calculate and obtain the coordinate position (simply referred to as the conveyor chain target coordinates) on the conveyor chain coordinate system to which the target manipulator actually needs to move according to the target manipulator coordinates and the manipulator preset parameters set in advance by the manipulator, and the manipulator moves the conveyor chain target coordinates to achieve grabbing of the target object. Specifically, the preset parameters of the mechanical arm include information of the calibration tool and coordinates of a start point of the conveying chain. The information of the calibration tool may include indication information indicating that the mechanical arm has carried the calibration tool, indication information indicating that the mechanical arm has not carried the calibration tool, and information such as a type and a length of the calibration tool. The starting point coordinates of the conveying chain are specifically coordinates of the visual frame in a conveying chain coordinate system, and the coordinates can be used for calibrating the coordinates when the objects in the visual frame do not start conveying.

Specifically, in the embodiment of the application, the internal conversion relation between the mechanical arm base coordinate system, the mechanical arm tool coordinate system and the conveying chain coordinate system is determined in the mechanical arm according to the preset parameters of the mechanical arm, so that after the mechanical arm acquires the target mechanical arm coordinate under the mechanical arm base coordinate system, the coordinate of the grabbing point of the target object corresponding to the coordinate under the conveying chain coordinate system can be calculated through the internal conversion relation, and the position of the grabbing point of the target object can be accurately moved, and the accurate grabbing of the target object can be realized. Specifically, the internal conversion relationship between the "robot arm base coordinate system-robot arm tool coordinate system-conveyor chain coordinate system" can be represented by an inverse matrix R ₂ ^-1 matrix of the 4*4 rotational-translational matrix between the robot arm coordinate system and the conveyor chain coordinate system, and then:

Wherein, (X _{Conveying device},Y_{Conveying device},Z_{Conveying device}) is the coordinates of the conveyor chain coordinate system, (X _r,Y_r,Z_r) is the coordinates of the robot arm coordinate system, (X _c,Y_c,Z_c) is the coordinates of the global camera coordinate system, R ₁ ^-1 is the inverse of the 4*4 rotational translation matrix between the global camera coordinate system and the robot arm coordinate system, and R ₂ ^-1 is the 4*4 rotational translation matrix between the robot arm coordinate system and the conveyor chain coordinate system. For convenience of description, the above formula is referred to as a robot arm-conveyor chain relation.

That is, in the embodiment of the present application, after the target photo group is obtained and the target global photo is spliced, the pixel coordinates of the grabbing point identified by the target global photo can be calculated and obtained based on the above pixel-camera relational expression and the camera-mechanical arm relational expression, which correspond to the target mechanical arm coordinates on the mechanical arm coordinate system; after the coordinates of the target mechanical arm are sent to the mechanical arm, the mechanical arm further combines the mechanical arm-conveying chain relation, and the coordinates of the grabbing points on the conveying chain coordinate system are calculated, so that the space coordinate chain conversion of a pixel coordinate system-a global camera coordinate system-a mechanical arm base coordinate system-a conveying chain coordinate system is jointly realized, the mechanical arm can accurately position the grabbing points of the target object, and the accurate grabbing of the target object is realized.

According to the embodiment of the application, when the distance between the visual frame and the mechanical arm is far, the hand-eye calibration can be conveniently and accurately realized through the calibration tool with the preset length. And correspondingly, in a scene with a far distance between the visual frame and the mechanical arm, after the target mechanical arm coordinate is obtained subsequently, the target mechanical arm coordinate can be sent to the mechanical arm, and the mechanical arm is instructed to accurately position the grabbing point of the target object according to the target mechanical arm coordinate and the mechanical arm preset parameters comprising the information of the calibration tool and the starting point coordinate of the conveying chain, so that the accurate grabbing of the target object is realized.

It should be understood that the sequence number of each step in the foregoing embodiment does not mean that the execution sequence of each process should be determined by the function and the internal logic, and should not limit the implementation process of the embodiment of the present application.

Embodiment two:

Fig. 7 is a schematic structural diagram of a hand-eye calibration device according to an embodiment of the present application, and for convenience of explanation, only the portions related to the embodiment of the present application are shown:

The hand-eye calibration device comprises: a photo group acquisition unit 71, a splicing unit 72 and a calibration unit 73.

Wherein:

A photo group obtaining unit 71, configured to obtain a photo group obtained by capturing a photo by a camera group, where the camera group includes at least two cameras for cooperatively capturing a target field of view; the photo group comprises photos obtained by shooting each camera in the camera group respectively.

And the stitching unit 72 is configured to stitch each photo in the photo group to obtain a global photo corresponding to the target field of view region.

And the calibration unit 73 is configured to calibrate a corresponding relationship between a pixel coordinate system of the global photo and a mechanical arm coordinate system according to the global photo, so as to obtain a hand-eye calibration result.

Optionally, the stitching unit 72 includes a transformation matrix determining module and a stitching module:

The transformation matrix determining module is used for carrying out characteristic point matching and image registration on every two adjacent photos in the photo group and determining a transformation matrix between the adjacent photos in the photo group;

And the splicing module is used for splicing all the photos in the photo group according to the transformation matrix between the adjacent photos in the photo group to obtain global photos.

Optionally, the splicing module is specifically configured to perform projective transformation on each photo in the photo group according to a transformation matrix between adjacent photos in the photo group, so as to obtain a preliminary spliced photo; and performing joint fusion processing on the preliminary spliced photos to obtain global photos.

Optionally, the target field of view area includes a preset calibration object, where the preset calibration object includes a preset number of calibration points, and the calibration unit 73 is specifically configured to obtain global camera parameters corresponding to the global photo; the global camera parameters at least comprise estimated internal parameters of the global camera; determining global camera coordinates corresponding to each calibration point according to the pixel coordinates of each calibration point in the pixel coordinate system of the global photo and the global camera parameters; the global camera coordinates are coordinates in a global camera coordinate system, and the global camera coordinate system is a coordinate system in which the global camera is located; determining the corresponding relation between the global camera coordinate system and the mechanical arm coordinate system according to the global camera coordinate of each calibration point and the read mechanical arm coordinate corresponding to each calibration point; the mechanical arm coordinates are coordinates of the calibration points in the mechanical arm coordinate system; and determining the corresponding relation between the pixel coordinate system of the global photo and the mechanical arm coordinate system according to the global camera parameters and the corresponding relation between the global camera coordinate system and the mechanical arm coordinate system, and obtaining a hand-eye calibration result.

Optionally, the interval between each of the calibration points in the target field of view is less than or equal to a preset threshold.

Optionally, the hand-eye calibration device further comprises:

The target photo group acquisition unit is used for starting the camera group to shoot if the target object is detected to enter the target visual field area, so as to obtain a target photo group;

The target global photo determining unit is used for splicing the target photo groups to obtain target global photos containing the complete image information of the target object;

The target mechanical arm coordinate determining unit is used for determining the target mechanical arm coordinate corresponding to the target object according to the target global photo and the hand-eye calibration result; the target mechanical arm coordinates are coordinates of grabbing points of the target object corresponding to the mechanical arm coordinates.

Optionally, the hand-eye calibration device further comprises:

The first indicating unit is used for indicating the mechanical arm to carry a calibration tool with preset length to sequentially point through each calibration point, and reading the coordinates output by the mechanical arm when the calibration tool point is at the position of the calibration point to obtain the mechanical arm coordinates corresponding to the calibration point; the coordinates output by the mechanical arm are coordinates under a mechanical arm base coordinate system;

The second indicating unit is used for sending the target mechanical arm coordinates to the mechanical arm so as to indicate the mechanical arm to grasp the target object according to the target mechanical arm coordinates and the mechanical arm preset parameters; the mechanical arm preset parameters comprise information of the calibration tool and coordinates of a starting point of a conveying chain.

It should be noted that, because the content of information interaction and execution process between the above devices/units is based on the same concept as the method embodiment of the present application, specific functions and technical effects thereof may be referred to in the method embodiment section, and will not be described herein.

It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions. The functional units and modules in the embodiment may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of a software functional unit. In addition, the specific names of the functional units and modules are only for distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working process of the units and modules in the above system may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.

Embodiment III:

Fig. 8 is a schematic diagram of a communication device according to an embodiment of the present application. As shown in fig. 8, the communication device 8 of this embodiment includes: a processor 80, a memory 81 and a computer program 82, such as a hand-eye calibration program, stored in the memory 81 and executable on the processor 80. The processor 80, when executing the computer program 82, implements the steps of the various hand-eye calibration method embodiments described above, such as steps S201 to S203 shown in fig. 2. Or the processor 80, when executing the computer program 82, performs the functions of the modules/units in the above-described device embodiments, such as the functions of the photo group acquisition unit 71 to the calibration unit 73 shown in fig. 7.

By way of example, the computer program 82 may be partitioned into one or more modules/units that are stored in the memory 81 and executed by the processor 80 to complete the present application. The one or more modules/units may be a series of computer program instruction segments capable of performing the specified functions describing the execution of the computer program 82 in the communication device 8.

The communication device 8 may be a computing device such as a desktop computer, a notebook computer, a palm computer, a cloud server, etc. The communication device may include, but is not limited to, a processor 80, a memory 81. It will be appreciated by those skilled in the art that fig. 8 is merely an example of communication device 8 and is not limiting of communication device 8, and may include more or fewer components than shown, or may combine certain components, or different components, e.g., the communication device may also include input-output devices, network access devices, buses, etc.

The Processor 80 may be a central processing unit (Central Processing Unit, CPU), other general purpose Processor, digital signal Processor (DIGITAL SIGNAL Processor, DSP), application SPECIFIC INTEGRATED Circuit (ASIC), field-Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 81 may be an internal storage unit of the communication device 8, such as a hard disk or a memory of the communication device 8. The memory 81 may also be an external storage device of the communication device 8, such as a plug-in hard disk, a smart memory card (SMART MEDIA CARD, SMC), a Secure Digital (SD) card, a flash memory card (FLASH CARD) or the like, which are provided on the communication device 8. Further, the memory 81 may also include both an internal storage unit and an external storage device of the communication device 8. The memory 81 is used for storing the computer program and other programs and data required by the communication device. The memory 81 may also be used to temporarily store data that has been output or is to be output.

In the foregoing embodiments, the descriptions of the embodiments are emphasized, and in part, not described or illustrated in any particular embodiment, reference is made to the related descriptions of other embodiments.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus/communication device and method may be implemented in other manners. For example, the apparatus/communication device embodiments described above are merely illustrative, e.g., the division of the modules or units is merely a logical function division, and there may be additional divisions in actual implementation, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection via interfaces, devices or units, which may be in electrical, mechanical or other forms.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated modules/units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the present application may implement all or part of the flow of the method of the above embodiment, or may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, and when the computer program is executed by a processor, the computer program may implement the steps of each of the method embodiments described above. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the computer readable medium contains content that can be appropriately scaled according to the requirements of jurisdictions in which such content is subject to legislation and patent practice, such as in certain jurisdictions in which such content is subject to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

The above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application, and are intended to be included in the scope of the present application.

Claims

1. A hand-eye calibration method, comprising:

according to the global photo, calibrating the corresponding relation between a pixel coordinate system of the global photo and a mechanical arm coordinate system to obtain a hand-eye calibration result;

The target visual field area comprises a preset calibration object, the preset calibration object comprises a preset number of calibration points, and the corresponding relation between a pixel coordinate system of the global photo and a mechanical arm coordinate system is calibrated according to the global photo to obtain a hand-eye calibration result, wherein the method comprises the following steps of:

acquiring global camera parameters corresponding to the global photo; the global camera parameters at least comprise estimated internal parameters of the global camera and estimated internal and external parameters of the global camera;

Determining global camera coordinates corresponding to each calibration point according to the pixel coordinates of each calibration point in the pixel coordinate system of the global photo and the global camera parameters; the global camera coordinates are coordinates in a global camera coordinate system, and the global camera coordinate system is a coordinate system in which the global camera is located;

Determining the corresponding relation between the global camera coordinate system and the mechanical arm coordinate system according to the global camera coordinate of each calibration point and the read mechanical arm coordinate corresponding to each calibration point; the mechanical arm coordinates are coordinates of the calibration points in the mechanical arm coordinate system;

According to the global camera parameters and the corresponding relation between the global camera coordinate system and the mechanical arm coordinate system, determining the corresponding relation between the pixel coordinate system of the global photo and the mechanical arm coordinate system, and obtaining a hand-eye calibration result;

The step of splicing the photos in the photo group to obtain the global photo corresponding to the target field of view area, comprising the following steps: performing feature point matching and image registration on every two adjacent photos in the photo group, and determining a transformation matrix between the adjacent photos in the photo group; splicing all the photos in the photo group according to the transformation matrix between the adjacent photos in the photo group to obtain a global photo;

The splicing of each photo in the photo group according to the transformation matrix between adjacent photos in the photo group to obtain a global photo comprises the following steps: according to the transformation matrix between adjacent photos in the photo group, performing projection transformation on each photo in the photo group to obtain a preliminary spliced photo; performing joint fusion processing on the preliminary spliced photos to obtain global photos;

The global camera is a camera under the estimated global photo field of view, specifically a virtual camera suspended at a certain spatial position in the visual frame, and does not coincide with any actual camera position in the camera set.

2. The hand-eye calibration method according to claim 1, wherein a spacing between each of the calibration points in the target field of view is less than or equal to a preset threshold.

3. The hand-eye calibration method according to claim 1, further comprising, after the obtaining of the hand-eye calibration result:

If the target object is detected to enter the target visual field area, starting the camera group to shoot, and obtaining a target photo group;

splicing the target photo groups to obtain a target global photo containing the complete image information of the target object;

Determining a target mechanical arm coordinate corresponding to the target object according to the target global photo and the hand-eye calibration result; the target mechanical arm coordinates are coordinates of grabbing points of the target object corresponding to the mechanical arm coordinates.

4. The hand-eye calibration method according to claim 3, further comprising, before the determining the correspondence between the global camera coordinate system and the robot coordinate system according to the global camera coordinate of each calibration point and the read robot coordinate corresponding to each calibration point:

correspondingly, after determining the target mechanical arm coordinates corresponding to the target object according to the target global photo and the hand-eye calibration result, the method further comprises:

5. A hand-eye calibration device, comprising:

the calibration unit is used for calibrating the corresponding relation between the pixel coordinate system of the global photo and the mechanical arm coordinate system according to the global photo to obtain a hand-eye calibration result; the target visual field area comprises a preset calibration object, the preset calibration object comprises a preset number of calibration points, and the corresponding relation between a pixel coordinate system of the global photo and a mechanical arm coordinate system is calibrated according to the global photo to obtain a hand-eye calibration result, wherein the method comprises the following steps of:

According to the global camera parameters and the corresponding relation between the global camera coordinate system and the mechanical arm coordinate system, determining the corresponding relation between the pixel coordinate system of the global photo and the mechanical arm coordinate system, and obtaining a hand-eye calibration result; the global camera is a camera under the estimated global photo view, specifically a virtual camera suspended at a certain spatial position in the visual frame and is not overlapped with any actual camera position in the camera set;

The splicing unit comprises a transformation matrix determining module and a splicing module;

The transformation matrix determining module is used for carrying out characteristic point matching and image registration on every two adjacent photos in the photo group, and determining a transformation matrix between the adjacent photos in the photo group;

The splicing module is used for splicing all the photos in the photo group according to the transformation matrix between the adjacent photos in the photo group to obtain global photos;

The splicing module is also used for carrying out projection transformation on each photo in the photo group according to the transformation matrix between the adjacent photos in the photo group to obtain a preliminary spliced photo; and performing joint fusion processing on the preliminary spliced photos to obtain global photos.

6. A communication device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the computer program, when executed by the processor, causes the communication device to carry out the steps of the method according to any one of claims 1 to 4.

7. A computer readable storage medium storing a computer program, which, when executed by a processor, causes a communication device to carry out the steps of the method according to any one of claims 1 to 4.