CN112598752B

CN112598752B - Calibration method and operation method based on visual recognition

Info

Publication number: CN112598752B
Application number: CN202011545117.XA
Authority: CN
Inventors: 李家清; 石金博; 陈晓聪; 邬荣飞
Original assignee: QKM Technology Dongguan Co Ltd
Current assignee: QKM Technology Dongguan Co Ltd
Priority date: 2020-12-24
Filing date: 2020-12-24
Publication date: 2024-02-27
Anticipated expiration: 2040-12-24
Also published as: CN112598752A

Abstract

The invention discloses a calibration method and an operation method based on visual identification, wherein the calibration method based on visual identification comprises the following steps: acquiring a camera TOOL _C The method comprises the steps of carrying out a first treatment on the surface of the Determining that the end flange is in a first fixed angular attitude alpha ₁ The conversion relation between the pixel coordinate system and the camera coordinate system; shooting at least one characteristic point on the marker; obtaining the position relation between the characteristic points and the center of the camera during shooting; angle ROLL-V for obtaining the mark of a marker ₁ The method comprises the steps of carrying out a first treatment on the surface of the Acquiring the position P of the end flange when the camera center is aligned with the feature points or the set relative positions among the feature points ₁ The method comprises the steps of carrying out a first treatment on the surface of the Acquiring position P of end flange when work tool reaches work position in expected posture ₁ According to P ₁ 、α ₁ 、P _T 、α _T The transform relation offset is obtained. The invention can enable the operation equipment with the visual identification function to adapt to the operation requirement under the condition that the photographing position and the operation position are not coincident in the operation process.

Description

Calibration method and operation method based on visual recognition

Technical Field

The invention relates to the technical field of industrial automation, in particular to a calibration method and an operation method based on visual identification.

Background

In industrial automation equipment, robots or other motion mechanisms with visual recognition function, a camera and a working tool are fixed on a rotating shaft and rotate along with the rotating shaft, a camera is often required to photograph a product or a working point in a plane to obtain the position or the working position of the product, and then the working tool is used to go to the working position for working, such as grabbing, placing or welding of the product. In the common demands, the photographing position and the working position of the product or the working point are overlapped sometimes, but in many cases, the photographing position and the working position of the product or the working point are not overlapped often due to different conditions of the product or the working point.

Disclosure of Invention

The present invention aims to solve at least one of the technical problems existing in the prior art. Therefore, the invention provides a calibration method based on visual identification, so that in the operation process, the operation equipment with the visual identification function can adapt to the operation requirement under the condition that the photographing position and the operation position are not coincident.

The invention further provides an operation method based on the visual identification calibration method.

According to the visual identification-based calibration method, the embodiment of the invention is applied to operation equipment, the operation equipment comprises an end effector and a movement mechanism, the end effector comprises an end flange, a camera and an operation tool, the camera and the operation tool are arranged on the end flange, and the movement mechanism can drive the end flange to freely move on an XY plane and can drive the end flange to rotate around an axis of the end flange; the calibration method comprises the following steps:

the TOOL of the camera is obtained by determining the position relationship between the center and the tail end flange of the camera _C The method comprises the steps of carrying out a first treatment on the surface of the According to the TOOL of the camera _C Determining that the end flange is in a first fixed angular attitude alpha ₁ At this time, the conversion relation T between the pixel coordinates and the physical position with reference to the camera center _PC The method comprises the steps of carrying out a first treatment on the surface of the At the end flange in a first fixed angular attitude alpha ₁ At least one characteristic point on the marker is shot, wherein the relative position between the marker and the working position is fixed, and each characteristic is shotRecording the position of the current tail end flange when the point is reached; according to the image obtained by shooting each characteristic point and the conversion relation T _PC Acquiring the position relationship between each characteristic point and the center of the camera during shooting; according to the position relationship between each characteristic point and the center of the camera during shooting, the current end flange position and the TOOL of the camera during shooting each characteristic point _C When the camera center is aligned with the feature points or the set relative positions among the feature points are obtained, the position P of the tail end flange ₁ The method comprises the steps of carrying out a first treatment on the surface of the Acquiring an angle ROLL-V of the marked object when the marked object is marked according to the image obtained by shooting each characteristic point ₁ The method comprises the steps of carrying out a first treatment on the surface of the Acquiring the position P of the end flange when the work tool reaches the work position in a desired posture _T And angular attitude alpha _T The method comprises the steps of carrying out a first treatment on the surface of the According to P ₁ 、α ₁ 、P _T 、α _T Obtain (P) ₁ 、α ₁ ) And (P) _T 、α _T ) Is a transform of the relation offset.

The calibration method based on visual recognition according to the embodiment of the first aspect of the invention has at least the following beneficial effects:

the embodiment of the invention firstly obtains the TOOL of the camera _C Then according to the TOOL of the camera _C Acquiring a first fixed angle attitude alpha of any one end flange ₁ The transformation relation between the pixel coordinate system and the physical position taking the camera center as a reference is adopted, then the characteristic points are photographed, the relative position relation between the characteristic points and the camera center can be obtained according to the coordinates of the characteristic points in the pixel coordinate system, and then the TOOL of the camera is adopted _C And acquiring a relative positional relationship between the acquired feature point and the camera center while maintaining the first fixed angular attitude α ₁ When the camera center is aligned with the feature points or the set relative positions among the feature points, the position P of the center of the flange tail end ₁ Meanwhile, the angle ROLL-V when the marker is marked can be obtained according to the image obtained by shooting the feature points ₁ Then, when the working tool reaches the working position in the expected posture, the position P of the tail end flange is obtained _T And end flange angular attitude alpha _T I.e. according to P ₁ 、α ₁ 、P _T 、α _T The method can obtainP ₁ 、α ₁ ) And (P) _T 、α _T ) Transforming the relation offset so as to complete the calibration process; according to the transformation relation offset obtained during calibration and the angle ROLL-V of the calibration of the marker ₁ In the subsequent actual operation process, after the characteristic points fixed at the relative positions with the markers are photographed, the position and the angle posture of the tail end flange can be known according to the photographed image when the operation tool reaches the operation position in the expected posture, and the operation tool can be driven to come to the operation position to finish the operation in the proper posture; therefore, after the operation equipment is calibrated according to the calibration method based on visual identification, the operation equipment can adapt to the operation requirement under the condition that the photographing position and the operation position are not coincident.

According to some embodiments of the invention, the calibration method based on visual recognition comprises determining the positional relationship between the center and the end flange of the camera to obtain the TOOL of the camera _C The method comprises the following steps:

at the end flange at a certain angular attitude alpha ₃ Respectively photographing the calibration object from at least three different position points, and obtaining an angle posture alpha according to the position of the tail end flange and the pixel coordinates of the calibration object in the image when photographing each time ₃ The transformation relation between the lower pixel coordinates and the flange end position, and the position P required to be reached by the end flange when the calibration object is positioned in the center of the image is obtained according to the transformation relation ₃ ；

At the end flange at a certain angular attitude alpha ₄ Respectively photographing the calibration object from at least three different position points, and obtaining an angle posture alpha according to the position of the tail end flange and the pixel coordinates of the calibration object in the image when photographing each time ₄ The transformation relation between the lower pixel coordinates and the flange end position, and the position P required to be reached by the end flange when the calibration object is positioned in the center of the image is obtained according to the transformation relation ₄ The method comprises the steps of carrying out a first treatment on the surface of the Wherein alpha is ₄ Not equal to alpha ₃ ；

According to alpha ₃ And P ₃ Alpha and alpha ₄ And P ₄ Calculate the TOOL of the camera _C 。

According toThe invention relates to a calibration method based on visual recognition, which is based on alpha ₃ And P ₃ Alpha and alpha ₄ And P ₄ Calculate the TOOL of the camera _C The method comprises the following steps:

will be alpha ₃ 、P ₃ 、α ₄ 、P ₄ The related numerical values in the map f are substituted into the mapping relation f _TOOL Calculating to obtain the TOOL of the camera _C The method comprises the steps of carrying out a first treatment on the surface of the Wherein,

(X ₃ ，Y ₃ ) Is P ₃ Position coordinates of (X) ₄ ，Y ₄ ) Is P ₄ Is used for the position coordinates of the object.

According to some embodiments of the invention, a calibration method based on visual recognition is provided, wherein, according to alpha ₃ And P ₃ Alpha and alpha ₄ And P ₄ Calculate the TOOL of the camera _C The method comprises the following steps:

acquiring the position P of the end flange ₃ And the angle posture is alpha ₃ In the state of (2), the camera center coordinates with respect to the camera TOOL in the motion mechanism coordinate system _C Expression T of medium parameter ₃ ；

Acquiring the position P of the end flange ₄ And the angle posture is alpha ₄ In the state of (2), the camera center coordinates with respect to the camera TOOL in the motion mechanism coordinate system _C Expression T of medium parameter ₄ ；

According to the relationship that the coordinate position of the camera center in the motion mechanism is unchanged in the two states, an equation T is established ₃ ＝T ₄ Obtaining the TOOL of the camera _C ，

Wherein, (X ₃ ，Y ₃ ) Is P ₃ Position coordinates of (X) ₄ ，Y ₄ ) Is P ₄ Is used for the position coordinates of the object.

According to some embodiments of the invention, the calibration method based on visual recognition is implemented according to the TOOL of the camera _C Determining that the end flange is in a first fixed angular attitude alpha ₁ At this time, the conversion relation T between the pixel coordinate system and the physical position with the camera center as the reference _PC The method comprises the following steps: at the end flange in a first fixed angular attitude alpha ₁ Photographing the standard points from at least 3 different positions respectively, so that the corresponding graph of the standard points in the image is not collinear among at least three points in a pixel coordinate system; when photographing the standard points at each position, recording the current position of the tail end flange respectively; according to the terminal flange position recorded when each shooting of the calibration point and the TOOL of the camera _C Converting to obtain the position of the camera center of each shooting calibration point, and acquiring the conversion relation T between the pixel coordinate and the physical position taking the camera center as a reference according to the pixel coordinate corresponding to the calibration point under each shooting position and the position of the camera center of each shooting calibration point _PC 。

According to the calibration method based on visual recognition of some embodiments of the present invention, if there are 2 feature points, the number of feature points is 2, wherein the number of feature points is determined according to the positional relationship between each feature point and the center of the camera during shooting, the current end flange position during shooting each feature point, and the TOOL of the camera _C When the camera center is aligned with the feature points or the set relative positions among the feature points are obtained, the position P of the tail end flange ₁ The method comprises the following steps: according to the position relation between each characteristic point and the center of the camera during shooting, the TOOL of the camera _C Shooting the current end flange position of each characteristic point, and acquiring the positions P of the two camera centers when the camera centers are respectively aligned with the two characteristic points _O1 And P _O2 The method comprises the steps of carrying out a first treatment on the surface of the According to the obtained P _O1 And P _O2 Obtaining P _O1 P _O2 The position of the central point O of the connecting line; based on the position of the center point O and the camera TOOL _C Acquiring a terminal flange position P when a camera center is aligned with a center point O ₁ 。

Calibration methods based on visual recognition according to some embodiments of the present inventionThe number of the feature points is multiple, wherein the angle ROLL-V of the marked article when the marked article is marked is obtained according to the image obtained by shooting each feature point ₁ The method comprises the following steps: according to the obtained P _O1 And P _O2 Obtaining P _O1 P _O2 The angle of the line to the horizontal center line of the camera, i.e. the angle ROLL-V at which the marker is calibrated ₁ 。

According to the calibration method based on visual recognition of some embodiments of the present invention, if the feature points are one, the current end flange position when each feature point is photographed, and the camera TOOL are based on the positional relationship between each feature point and the camera center when photographing _C When the camera center is aligned with the feature points or the set relative positions among the feature points are obtained, the position P of the tail end flange ₁ The method comprises the following steps: according to the TOOL of the camera _C The position relation between the characteristic points and the camera center during shooting and the current end flange position during shooting the characteristic points are obtained, and the position of the camera center when the camera center is aligned to the characteristic points is obtained; according to the position of the camera center when the camera center is aligned with the feature point and the camera TOOLC, the position P of the tail end flange is obtained when the camera center is aligned with the feature point ₁ 。

According to some embodiments of the invention, the calibration method based on visual recognition includes acquiring a position P of the end flange when the work tool reaches the work position in a desired posture _T And angular attitude alpha _T The method comprises the following steps: the motion mechanism is controlled to drive the tail end flange to move and enable the working tool to reach the working position in a desired gesture by means of manual teaching, and the current position P of the tail end flange is recorded _T And angular attitude alpha _T 。

According to the calibration method based on visual recognition of some embodiments of the present invention, if the marker is a carrier or a product carrying a product, the position P of the end flange is obtained when the working tool reaches the working position in a desired posture _T And angular attitude alpha _T The method comprises the following steps: at the end flange in a first fixed angular attitude alpha ₁ When the product is photographed by a camera, a product image is obtained; from the product image Conversion relation T between pixel coordinates and physical position referenced to camera center _PC TOOL of camera _C Acquiring the operation position of a product; acquiring the position P of the tail end flange when the working tool reaches the working position in a desired posture according to the working position of the product _T And angular attitude alpha _T 。

According to the calibration method based on visual recognition of some embodiments of the present invention, if the marker is a carrier for receiving a product, and a placement structure for placing the product is provided on the carrier, the position P of the end flange is obtained when the working tool reaches the working position in a desired posture _T And angular attitude alpha _T The method comprises the following steps: at the end flange in a first fixed angular attitude alpha ₁ When the camera is used, the camera is used for photographing the placement structure, so that an image of the placement structure is obtained; based on the conversion relation T between the image of the placement structure, the pixel coordinates and the physical position referenced to the camera center _PC TOOL of camera _C Acquiring the operation position of a product; acquiring the position P of the tail end flange when the working tool reaches the working position in a desired posture according to the working position of the product _T And angular attitude alpha _T 。

According to the calibration method based on visual recognition of some embodiments of the present invention, the marker corresponds to N operation positions, N being greater than or equal to 1; wherein, when the working tool reaches the working position in a desired posture, the position P of the tail end flange is obtained _T And angular attitude alpha _T The method comprises the following steps: for each working position, the position P of the end flange 20 is obtained when the working tool 40 reaches each working position in a desired posture _Tn And angular attitude alpha _Tn Wherein N is greater than or equal to N is greater than or equal to 1.

According to the calibration method based on visual recognition according to some embodiments of the invention, according to P ₁ 、α ₁ 、P _T 、α _T Obtain (P) ₁ 、α ₁ ) And (P) _T 、α _T ) The conversion relation offset of (2) includes the following steps: according to P ₁ 、α ₁ And the recorded position P of each set of end flanges _Tn And powderEnd flange angular attitude alpha _Tn Obtain (P) ₁ 、α ₁ ) And each group (P _Tn 、α _Tn ) Is a transform of offset-n.

According to a second aspect of the invention, an operation method based on visual recognition is applied to an operation device, the operation device comprises an end effector and a movement mechanism, the end effector comprises an end flange, a camera and an operation tool, the camera and the operation tool are arranged on the end flange, and the movement mechanism can drive the end flange to freely move on an XY plane and can drive the end flange to rotate around an axis of the end flange; the operation method comprises the following steps:

the TOOL of the camera is obtained by determining the position relationship between the center and the tail end flange of the camera _C The method comprises the steps of carrying out a first treatment on the surface of the According to the TOOL of the camera _C Determining that the end flange is in a first fixed angular attitude alpha ₁ At this time, the conversion relation T between the pixel coordinate system and the physical position with the camera center as the reference _PC The method comprises the steps of carrying out a first treatment on the surface of the At the end flange in a first fixed angular attitude alpha ₁ Shooting a single characteristic point on the marker, wherein the relative position between the marker and the operation position is fixed, and recording the position of the current tail end flange when shooting the characteristic point; image obtained according to shooting characteristic points and conversion relation T _PC Acquiring the position relationship between the characteristic points and the center of the camera during shooting; according to the position relation between the characteristic points and the center of the camera during shooting, the current end flange position during shooting the characteristic points and the TOOL of the camera _C When the camera center alignment feature point is acquired, the position P of the tail end flange ₁ The method comprises the steps of carrying out a first treatment on the surface of the Acquiring an angle ROLL-V of the marked object when the marked object is marked according to the image obtained by shooting the characteristic points ₁ The method comprises the steps of carrying out a first treatment on the surface of the Acquiring the position P of the end flange when the work tool reaches the work position in a desired posture _T And angular attitude alpha _T The method comprises the steps of carrying out a first treatment on the surface of the According to P ₁ 、α ₁ 、P _T 、α _T Obtain (P) ₁ 、α ₁ ) And (P) _T 、α _T ) Is a transform of the offset;

at the end flange in a first fixed angular attitude alpha ₁ Then, shooting the characteristic points on the marker again,recording the position coordinates of the current tail end flange; based on the image obtained by photographing the marker again and the conversion relation T _PC Obtaining the position relation between the feature points and the center of the camera during re-shooting; according to the position relation between the characteristic points and the camera center during the re-shooting, the current terminal flange position during the re-shooting of the characteristic points and the camera TOOL _C Acquiring the position P of the camera center when the camera center is aligned with the feature point again ₅ The method comprises the steps of carrying out a first treatment on the surface of the Acquiring an angle ROLL-V of the marker during operation according to the image obtained by shooting the marker again ₂ The method comprises the steps of carrying out a first treatment on the surface of the According to ROLL-V ₂ And ROLL-V ₁ Deviation between camera TOOL _C Acquiring the center of the camera to be at P ₅ Position and end flange in angular attitude alpha ₁ +(ROLL-V ₂ -ROLL-V ₁ ) Lower end flange position P ₆ The method comprises the steps of carrying out a first treatment on the surface of the According to P ₆ Angle posture alpha ₁ +(ROLL-V ₂ -ROLL-V ₁ ) Converting the relation offset to obtain the end flange position P in the working position ₇ And angular attitude alpha ₇ The method comprises the steps of carrying out a first treatment on the surface of the Moving the end flange to P ₇ And rotating the end flange to an angular attitude alpha ₇ The work is performed at the work position by the work tool.

The operation method based on visual recognition according to the embodiment of the second aspect of the invention has at least the following beneficial effects: the operation method comprises the calibration method of the embodiment of the first aspect and corresponding operation steps when the characteristic points are one; in the working step, the position P of the end flange is found by finding the position P of the end flange when the camera center is also aligned with the feature point and the angle of the marker in the camera field of view is consistent with the angle of the marker in the camera field of view 60 during calibration ₆ And angular attitude alpha ₁ +(ROLL-V ₂ -ROLL-V ₁ ) Then, by means of the conversion relation offset obtained in the calibration process, the position P of the end flange can be obtained when the working tool arrives at the actual working position in the expected posture ₇ And angular attitude alpha ₇ Therefore, the tail end flange can be driven to reach the position and the angle posture through the movement mechanism, and the working tool can be ensured to smoothly finish the operation; thus, embodiments of the second aspect of the inventionThe operation method based on visual recognition can finish the operation under the condition that the photographing position and the operation position are not coincident.

According to a third aspect of the invention, the visual recognition-based working method is applied to a working device, the working device comprises an end effector and a movement mechanism, the end effector comprises an end flange, a camera and a working tool, the camera and the working tool are arranged on the end flange, and the movement mechanism can drive the end flange to freely move on an XY plane and can drive the end flange to rotate around an axis of the end flange; the operation method comprises the following steps:

the TOOL of the camera is obtained by determining the position relationship between the center and the tail end flange of the camera _C The method comprises the steps of carrying out a first treatment on the surface of the According to the TOOL of the camera _C Determining that the end flange is in a first fixed angular attitude alpha ₁ At this time, the conversion relation T between the pixel coordinate system and the physical position with the camera center as the reference _PC The method comprises the steps of carrying out a first treatment on the surface of the At the end flange in a first fixed angular attitude alpha ₁ Shooting a plurality of characteristic points on the marker, wherein the relative positions between the marker and the operation position are fixed, and recording the position of the current tail end flange when shooting each characteristic point; according to the image obtained by shooting each characteristic point and the conversion relation T _PC Acquiring the position relationship between each characteristic point and the center of the camera during shooting; according to the position relationship between each characteristic point and the center of the camera during shooting, the current end flange position and the TOOL of the camera during shooting each characteristic point _C When the set relative position between the camera center alignment feature points is obtained, the position P of the tail end flange ₁ The method comprises the steps of carrying out a first treatment on the surface of the Acquiring an angle ROLL-V of the marked object when the marked object is marked according to the image obtained by shooting the characteristic points ₁ The method comprises the steps of carrying out a first treatment on the surface of the Acquiring the position P of the end flange when the work tool reaches the work position in a desired posture _T And angular attitude alpha _T The method comprises the steps of carrying out a first treatment on the surface of the According to P ₁ 、α ₁ 、P _T 、α _T Obtain (P) ₁ 、α ₁ ) And (P) _T 、α _T ) Is a transform of the offset;

at the end flange in a first fixed angular attitude alpha ₁ Next, for each of the markers againA feature point is photographed, and when each feature point is photographed, the position of the current tail end flange is recorded; according to the image obtained by shooting each characteristic point again and the conversion relation T _PC Obtaining the position relation between each characteristic point and the center of the camera during secondary shooting; according to the position relation between each characteristic point and the center of the camera during re-shooting, the current end flange position and the camera TOOL during re-shooting each characteristic point _C Acquiring a position P of a camera center in which the camera center is aligned again with a set relative position between feature points ₅ The method comprises the steps of carrying out a first treatment on the surface of the Acquiring an angle ROLL-V of the marker during operation according to the image obtained by shooting each characteristic point again ₂ The method comprises the steps of carrying out a first treatment on the surface of the According to ROLL-V ₂ And ROLL-V ₁ Deviation between camera TOOL _C Acquiring the center of the camera to be at P ₅ Position and end flange in angular attitude alpha ₁ +(ROLL-V ₂ -ROLL-V ₁ ) In the state of (2) the position P of the end flange ₆ The method comprises the steps of carrying out a first treatment on the surface of the According to P ₆ Angle posture alpha ₁ +(ROLL-V ₂ -ROLL-V ₁ ) Converting the relation offset to obtain the end flange position P in the working position ₇ And angular attitude alpha ₇ The method comprises the steps of carrying out a first treatment on the surface of the Moving the end flange to P ₇ And rotating the end flange to an angular attitude alpha ₇ The work is performed at the work position by the work tool.

According to the operation method based on visual identification, which is an embodiment of the third aspect of the invention, the operation method has at least the following beneficial effects: the operation method comprises the calibration method of the embodiment of the first aspect and corresponding operation steps when the characteristic points are a plurality of; in the working step, the position P of the end flange is found when the camera center is also aligned with the set relative position between the feature points and the angle of the marker in the camera view is consistent with the angle of the marker in the camera view in the calibration process ₆ And angular attitude alpha ₁ +(ROLL-V ₂ -ROLL-V ₁ ) Then, by means of the conversion relation offset obtained in the calibration process, the position P of the end flange can be obtained when the working tool arrives at the actual working position in the expected posture ₇ And angular attitude alpha ₇ Thereby can be driven by the movement mechanismThe movable end flange comes to the position and the angle posture, so that the working tool can smoothly finish the operation; thus, the visual recognition-based working method according to the embodiment of the third aspect of the present invention can complete the work in the case where the photographing position and the working position do not coincide.

Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.

Drawings

The foregoing and/or additional aspects and advantages of the invention will become apparent and may be better understood from the following description of embodiments taken in conjunction with the accompanying drawings in which:

FIG. 1 is a flow chart of a calibration method based on visual recognition in an embodiment of the first aspect of the present invention;

FIG. 2 is a schematic diagram of a working apparatus according to an embodiment of the present invention;

fig. 3 is a schematic diagram of the movement paths of the camera center and the corresponding nine photo spots of the camera center in step S210;

Fig. 4 is a schematic diagram of a 3×3 dot matrix in which the dot corresponding patterns are arranged in the pixel coordinate system in step S210;

fig. 5 is a process diagram of steps S300 to S700 when there are two feature points;

fig. 6 is a process diagram of steps S300 to S700 when the feature points are single;

FIG. 7 is a flow chart of a visual recognition-based job method of a second aspect embodiment;

FIG. 8 is a process diagram of S900b to S1400b in FIG. 7;

FIG. 9 is a flow chart of a visual recognition-based job method in accordance with an embodiment of the third aspect;

fig. 10 is a process diagram of S900b to S1400b in fig. 9.

Reference numerals:

motion mechanism 10, end flange 20, camera 30, work tool 40, calibration point 50, camera field of view 60, marker 70, work position 710, product 80, feature point 90.

Detailed Description

Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are illustrative only and are not to be construed as limiting the invention.

In the description of the present invention, it should be understood that the direction or positional relationship indicated with respect to the description of the orientation, such as up, down, left, right, front, rear, etc., is based on the direction or positional relationship shown in the drawings, is merely for convenience of describing the present invention and simplifying the description, and does not indicate or imply that the apparatus or element to be referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as limiting the present invention.

In the description of the present invention, unless explicitly defined otherwise, terms such as arrangement, installation, connection, etc. should be construed broadly and the specific meaning of the terms in the present invention can be reasonably determined by a person skilled in the art in combination with the specific contents of the technical scheme.

A visual recognition-based calibration method according to an embodiment of the first aspect of the present invention, which is mainly applied to the following working apparatuses, will be described below with reference to fig. 1 to 6.

Referring to fig. 1 and 2, the working device includes an end effector and a movement mechanism 10, the end effector includes an end flange 20, a camera 30 and a working tool 40, the camera 30 and the working tool 40 are disposed on the end flange 20, and the movement mechanism 10 can drive the end flange 20 to freely move on an XY plane and can drive the end flange 20 to rotate around its own axis.

Depending on the type of the work tool 40, for example, the work tool 40 may be a clamping jaw, a welding gun, a detection device, or the like, and correspondingly, the work equipment may be a loading/unloading and material handling equipment for taking and placing the product 80, a welding equipment for welding, a detection equipment for detecting, or the like, and the work tool 40 and the work equipment are not particularly limited.

Likewise, the movement mechanism 10 may be of various types, such as an articulated robot, or other automated device having a rotary driving module and a plurality of linear driving modules, and the like, and the movement mechanism 10 is not particularly limited herein, as long as it can drive the end flange 20 to freely move on the XY plane and can drive the end flange 20 to rotate about its own axis.

The calibration method based on visual identification comprises the following steps:

s100, determining the position relationship between the center of the camera and the end flange 20 to obtain the TOOL of the camera _C ；

S200, according to the TOOL of the camera _C The end flange 20 is determined to be in the first fixed angular attitude α ₁ At this time, the conversion relation T between the pixel coordinate system and the physical position with the camera center as the reference _PC ；

S300, at the end flange 20, in the first fixed angular attitude alpha ₁ Photographing at least one characteristic point 90 on the marker 70, wherein the relative position between the marker 70 and the working position 710 is fixed, and recording the position of the current end flange 20 when photographing each characteristic point 90;

s400, according to the image obtained by shooting each feature point 90, the conversion relation T _PC Acquiring a positional relationship between the feature points 90 and the camera center at the time of shooting;

S500, according to the position relationship between each feature point 90 and the center of the camera during shooting, the current end flange position during shooting each feature point 90, and the TOOL of the camera _C When the camera center is aligned with the feature points 90 or the set relative positions between the feature points 90 are obtained, the position P of the end flange 20 ₁ ；

S600, acquiring the angle ROLL-V of the marker 70 when in calibration according to the image obtained by shooting the feature points 90 ₁ ；

S700, when the working tool 40 reaches the working position 710 in a desired posture, the position P of the end flange 20 is obtained _T And angular attitude alpha _T ；

S800 according to P ₁ 、α ₁ 、P _T 、α _T Obtain (P) ₁ 、α ₁ ) And (P) _T 、α _T ) Is a transform of the relation offset.

It will be appreciated that the tag 70 may be a product 80, a carrier already carrying the product 80, a carrier waiting to receive the product 80, etc.; for example, when the marker 70 is a product 80, the working location 710 is located on the product 80, and may be, for example, a welding location on the product 80, a detection location on the product 80, a grasping location of the product 80, and the like; for example, when the marker 70 is a carrier that already carries the product 80, the working position 710 is located on the product 80 in the carrier, and may be a welding position on the product 80, a detection position on the product 80, a gripping position of the product 80, or the like; for another example, when the tag 70 is a carrier waiting to receive the product 80, the operation position 710 may be a placement position of the corresponding product 80 on the carrier, and the corresponding operation process is to place the product 80 on the carrier.

It will be appreciated that the angle ROLL-V is marked with respect to the marker 70 ₁ And an angle ROLL-V at the time of operation of the tag 70 described later ₂ Since the change value of the corresponding angle of the marker 70, namely ROLL-V, is finally obtained during the operation ₂ -ROLL-V ₁ Thus ROLL-V ₁ And ROLL-V ₂ The angle of the marker 70 in the image coordinate system may be obtained, or the angle of the marker 70 in the movement mechanism coordinate system may be obtained, or the first fixed angular attitude α of the end flange 20 may be obtained ₁ In this state, the angle of the marker 70 in the camera coordinate system may be, for example, an angle between a straight line with a constant relative position on the marker 70 and a certain coordinate axis in the pixel coordinate system, an angle between a straight line with a constant relative position on the marker 70 and a certain coordinate axis in the motion mechanism coordinate system, or an angle between a straight line with a constant relative position on the marker 70 and a horizontal center line of the camera.

It can be understood that the feature points 90 may be MARK points manually set on the product 80 or the carrier, or may be some existing structural features on the product 80 or the carrier, for example, may be existing holes, bosses or boundaries, vertices, etc. on the product 80 or the carrier, where the corresponding graphic after photographing is convenient for recognition in the image.

The calibration method of the embodiment comprises the steps of firstly obtaining TOOL of a camera _C Then according to the TOOL of the camera _C Acquiring a first fixed angle attitude alpha of any one end flange ₁ The transformation relation between the pixel coordinate system and the physical position taking the camera center as the reference is adopted, then the characteristic points 90 are photographed, the relative position relation between the characteristic points 90 and the camera center can be obtained according to the coordinates of the characteristic points 90 in the pixel coordinate system, and then the relative position relation between the characteristic points 90 and the camera center can be obtained according to the TOOL of the camera _C And the acquired relative positional relationship between the feature point 90 and the camera center is known while maintaining the first fixed angular attitude α ₁ When the camera center is aligned with the feature point 90 or the set relative position between the feature points 90, the position P of the flange end center ₁ At the same time, the angle ROLL-V when the marker is marked can be known according to the image obtained by shooting the feature point 90 ₁ Then, when the work tool 40 reaches the work position 710 in a desired posture, the position P of the end flange 20 is obtained _T And end flange angular attitude alpha _T I.e. according to P ₁ 、α ₁ 、P _T 、α _T Thus, the product (P) ₁ 、α ₁ ) And (P) _T 、α _T ) Transforming the relation offset so as to complete the calibration process; according to the transformation relation offset obtained during calibration and the angle ROLL-V of the calibration of the marker ₁ In the subsequent actual operation process, after photographing the feature point 90 fixed relative to the marker 70, the position and the angular posture of the end flange 20 can be known according to the photographed image when the operation tool 40 reaches the operation position 710 in the expected posture, so that the operation tool 40 can be driven to come to the operation position 710 to complete the operation in the proper posture; therefore, after the operation equipment is calibrated according to the calibration method based on visual recognition, the operation equipment can adapt to the operation requirement in the case that the photographing position and the operation position 710 are not coincident. How to obtain the transformation relation offset according to the calibration and the angle ROLL-V of the calibration of the marker 70 ₁ Bringing the work tool 40 to the work position 710 to perform work in a proper posture will be described in detail in the second and third embodiments of the present invention.

It will be appreciated that, unless specifically stated otherwise, the position of the end flange 20 refers to the coordinates of the end flange 20 in the motion mechanism coordinate system; it is to be understood that, unless specifically described otherwise, the position of the camera center refers to the coordinates of the camera center in the movement mechanism coordinate system; the motion mechanism coordinate system may be established with any one of the fixed points on the base of the motion mechanism 10 as the origin, and the XY plane in the motion mechanism 10 is parallel to the plane in which the marker 70 is located.

It will be appreciated that in step S100, the positional relationship between the center of the camera and the end flange 20 is determined, i.e. the camera TOOL is obtained _C The method comprises the following steps:

s110, the end flange 20 is positioned at a certain angle posture alpha ₃ Then, photographing the calibration object from at least three different position points respectively, and obtaining an angle posture alpha according to the position of the end flange 20 and the pixel coordinates of the calibration object in the image when photographing each time ₁ The transformation relation between the lower pixel coordinates and the flange end position, and the position P required to be reached by the end flange 20 when the calibration object is positioned in the center of the image is obtained according to the transformation relation ₃ ；

It will be appreciated that α ₃ The angle may be any angle, for example, 0 degrees, 45 degrees, 90 degrees, 180 degrees, or the like.

Specifically, the end flange 20 is first driven to move to a preset angular posture α by the movement mechanism 10 ₃ Before photographing, the movement of the tail end flange 20 is controlled by the movement mechanism 10, and the camera 30 is made to come to a position near the calibration object; then selecting at least three points, such as three points, four points or more than four points, near the current position, then driving the three points to automatically move to each point one by one through a movement mechanism 10, photographing through a camera 30 after moving to each point, and ensuring that a calibration object is in the visual field of the camera 30 when photographing each time; at each point of the photograph, record The position of the lower current end flange 20; the current position and angular attitude α of the end flange 20 ₃ All can be obtained in real time through the bottom layer of the operation equipment; according to the shot image, the coordinates of the corresponding graph of the calibration object in the image, namely the pixel coordinates of the calibration object in the image, can be obtained through an image processing technology. The angular attitude alpha of the end flange 20 can be obtained through the obtained coordinates of the positions of the at least three end flanges 20 and the obtained coordinates of the three pixels in one-to-one correspondence ₃ The transformation relationship between the pixel coordinates and the coordinates of the center position of the end flange 20; then according to the pixel coordinates of the image center and the transformation relation, the position P of the end flange 20 required to be reached when the calibration object is positioned at the image center can be obtained ₃ The method comprises the steps of carrying out a first treatment on the surface of the The calibration object is positioned at the center of the image to indicate that the camera center is aligned with the calibration object at the moment, namely, the angle posture alpha of the flange 20 at the tail end is obtained ₃ When the camera is centered on the calibration object, the position P of the end flange 20 ₃ 。

S120, the end flange 20 is positioned at a certain angle posture alpha ₄ Then, photographing the calibration object from at least three different position points respectively, and obtaining an angle posture alpha according to the position of the end flange 20 and the pixel coordinates of the calibration object in the image when photographing each time ₂ The transformation relation between the lower pixel coordinates and the flange end position, and the position P required to be reached by the end flange 20 when the calibration object is positioned in the center of the image is obtained according to the transformation relation ₄ The method comprises the steps of carrying out a first treatment on the surface of the Wherein alpha is ₄ Not equal to alpha ₃ ；

It will be appreciated that α ₄ May be at any angle, such as 0 degrees, 45 degrees, 90 degrees, 180 degrees, etc., but α ₄ Not equal to alpha ₃ The method comprises the steps of carrying out a first treatment on the surface of the Alpha can also be selected for ease of calculation ₃ 0 degrees, and alpha ₄ 180 degrees.

Specifically, the step of S320 is substantially the same as the step of S310, except that at the beginning, the end flange 20 is first driven to move to a preset angular attitude α by the movement mechanism 10 ₄ . And, correspondingly, the angular attitude alpha of the flange 20 at the end will be obtained ₄ When the camera is centered on the calibration object, the position of the end flange 20P4。

S130 according to alpha ₃ And P ₃ Alpha and alpha ₄ And P ₄ Calculate the TOOL of the camera _C 。

It should be appreciated that due to the camera TOOL _C And the position of the calibration object under the coordinate of the motion mechanism are fixed, so that the TOOL of the camera can be obtained _C . The camera TOOL will be given below _C Is a specific calculation mode.

In the first calculation mode, step S330, according to α ₃ And P ₃ Alpha and alpha ₄ And P ₄ Calculate the TOOL of the camera _C The method comprises the following steps:

s131a, acquiring the position P of the end flange 20 ₃ And the angle posture is alpha ₃ In the state of (2), the camera center coordinates with respect to the camera TOOL in the motion mechanism coordinate system _C Expression T of medium parameter ₃ ；

As can be appreciated, the camera TOOL _C For the purpose of describing the deviation between the center of the camera and the center of the end flange 20 in the X direction and Y direction under the coordinate system of the movement mechanism in the state that the end flange 20 is in the angular posture of 0 degree, the deviations in the two directions are respectively denoted as X for convenience of description _C And Y _C TOOL of camera _C Denoted as P _C ，P _C ＝[X _C Y _C ] ^T ，P ₃ Position sitting at (X) ₃ ，Y ₃ )，P ₄ Position sitting at (X) ₄ ，Y ₄ ) The method comprises the steps of carrying out a first treatment on the surface of the For ease of computation, the spatial coordinates are converted into a matrix representation consisting of a 3×3 rotation matrix R and a 1×3 offset matrix M, expressed as follows:

will P ₃ 、P ₄ TOOL of camera _C Conversion to matrix for operation, where P ₃ The corresponding matrix is:

P ₃ the corresponding matrix is:

P _C the corresponding matrix is:

since the camera center is thus co-ordinate in the motion mechanism coordinate system with respect to the camera TOOL _C Expression T of medium parameter ₃ The method comprises the following steps:

s132a, acquiring the position P of the end flange 20 ₄ And the angle posture is alpha ₄ In the state of (2), the camera center coordinates with respect to the camera TOOL in the motion mechanism coordinate system _C Expression T of medium parameter ₄ ；

It will be appreciated that, substantially the same as in S131a described above, the expression T3 of the coordinates of the camera center in the motion mechanism coordinate system with respect to the parameters in the camera tollc is given as follows:

S132a, establishing an equation T according to the relationship that the coordinate position of the camera center in the motion mechanism is unchanged in the two states ₃ ＝T ₄ Obtaining the TOOL of the camera _C ；

It can be understood that since the camera centers in the two states are aligned with the calibration object, i.e. the coordinate positions of the camera centers in the motion mechanism are unchanged, equation T can be established ₃ ＝T ₄ ：

From the above equation, it can be derived that:

since only the offset value X is required here _C And Y _C The rotation matrix can therefore be ignored to obtain:

R ₃ M _C +M ₃ ＝R ₄ M _C +M ₄

the simplification can be obtained:

(R ₃ -R ₄ )M _C ＝M ₄ -M ₃

finally, it can be obtained that:

M _C ＝(R ₃ -R ₄ ) ^-1 ×(M ₄ -M ₃ )

ignoring the height-brought-in expression may yield:

in the second calculation mode, step S330, according to α ₃ And P ₃ Alpha and alpha ₄ And P ₄ Calculate the TOOL of the camera _C The method comprises the following steps:

s130b, will be alpha ₃ 、P ₃ 、α ₄ 、P ₄ The related numerical values in the map f are substituted into the mapping relation f _TOOL Calculating to obtain the TOOL of the camera _C 。

It can be appreciated that the mapping relation f _TOOL Can preset the internal program of the working equipment in advance and then make alpha in the teaching process ₃ 、P ₃ 、α ₄ 、P ₄ The related numerical values in the map f are substituted into the mapping relation f _TOOL Is directly calculated.

Wherein,

Wherein the mapping relation f _TOOL The principle and detailed calculation procedure of (a) are basically the same as those of the first calculation method, and will not be described in detail here.

Referring to fig. 3 and 4, it can be understood that in step S200, according to the camera TOOL _C The end flange 20 is determined to be in the first fixed angular attitude α ₁ At this time, the conversion relation T between the pixel coordinate system and the physical position with the camera center as the reference _PC The method comprises the following steps:

s210, at the end flange 20, in a first fixed angular attitude alpha ₁ Photographing the calibration points 50 from at least 3 different positions respectively, so that the corresponding graph of the calibration points 50 in the image is not collinear among at least three points in a pixel coordinate system; recording the current end flange positions respectively when photographing the calibration points 50 at each position;

it will be appreciated that α ₁ The angle may be any angle, for example, 0 degrees, 45 degrees, 90 degrees, 180 degrees, or the like.

And, it can be understood that the more the position points are selected, the more accurate the result is, for example, in order to ensure the accuracy, the following way may be selected to perform shooting: the movement mechanism 10 drives the camera center to move by a fixed offset to 9 different positions along the X direction and the Y direction to take pictures respectively, so that corresponding patterns of the standard points 50 in the images are arranged into a 3X 3 lattice in a pixel coordinate system; the current end flange 20 positions were recorded separately at 9 position shots. Specifically, the end flange 20 is first driven to rotate by the movement mechanism 10 by the angular posture α ₁ Then the movement mechanism 10 drives the camera center to come near the single calibration point 50 to ensure that the single calibration point 50 can both be used in the subsequent photographing processInto the camera field of view 60; then the movement mechanism 10 drives the camera center to reach one of preset points, photographing is started, and the current position of the tail end flange 20 is recorded; then, the movement mechanism 10 drives the camera center to move a fixed distance along the X direction or the Y direction each time, and after each movement, photographing and recording the current position of the tail end flange 20 are carried out; the track of the movement of the camera center and the position of the calibration point 50 driven by the movement mechanism 10 can be set in a manner shown in fig. 3; it will be understood, of course, that the initial camera point, the camera center, may be selected to be aligned with the calibration point 50, or may be selected to be misaligned with the calibration point 50. Moreover, it will be appreciated that the locus of movement of the camera center by the movement mechanism 10 may be selected in other ways than the one shown in fig. 3, as long as it is ensured that the corresponding pattern of the calibration point 50 in the image is not collinear with at least three points in the pixel coordinate system.

S220, according to the terminal flange position and the TOOL of the camera recorded when the calibration point 50 is shot each time _C The position of the camera center of each shooting calibration point 50 is obtained through conversion, and the conversion relation T between the pixel coordinates and the physical position taking the camera center as a reference is obtained according to the pixel coordinates corresponding to the calibration point 50 under each shooting position and the position of the camera center of each shooting calibration point 50 _PC 。

It will be appreciated that, by image processing, the pixel coordinates corresponding to the lower calibration point 50 of each photographing position can be obtained, and the position of each end flange 20 corresponding to the photographing position can be directly obtained from the bottom of the moving mechanism 10 during photographing, and then the camera TOOL is used _C The position of the center of each photographing point camera, that is, the coordinates of the center of the camera under the coordinate system of the movement mechanism can be converted. Then the conversion relation T between the pixel coordinate system and the physical position taking the camera center as the reference can be obtained through the pixel coordinate corresponding to the lower calibration point of each photographing position and the position of the camera center of each photographing calibration point _PC 。

Specifically, for example, on the basis of the above 9 different photographing positions, the subscripts of the 9 photographing positions can be obtained by an image processing techniqueThe 9 pixel coordinates corresponding to the fixed point 50, and the 9 terminal flange 20 positions corresponding to the photographing point can be directly obtained from the bottom of the movement mechanism 10 during photographing, and then the camera TOOL is used _C The position of the camera center of each photographing point, that is, the coordinates of the camera center under the coordinate system of the movement mechanism during each photographing point, can be converted. Then through the 9 position coordinates of the camera center and the corresponding 9 pixel coordinates, the conversion relation T between the pixel coordinate system and the physical position taking the camera center as the reference can be obtained _PC 。

By the conversion relation T between the acquired pixel coordinate system and the physical position taking the camera center as the reference _PC The relative position between the object and the center of the camera, that is, the distance of the object with respect to the center of the camera in the X direction and the Y direction under the motion mechanism coordinates can be obtained by means of the pixel coordinates of the corresponding pattern of the object in the image. It will be appreciated that if the relative distance between the working position 710 and the feature point 90 is large, for example, the marker is a large table, and the feature point 90 is disposed at one corner of the table, and the working position 710 is located at another diagonal or center of the table, when the positioning is performed by only a single feature point 90, there may be a small deviation in image recognition due to the single feature point 90, and thus, there is a large deviation between the actually arrived position of the working tool 40 and the working position 710 in actual working. In order to solve this problem, a plurality of feature points 90 may be used for positioning, for example, two feature points 90, three feature points 90, or three or more feature points 90, or the like.

Referring to fig. 5, if there are two or more feature points 90, in step S300, the end flange 20 is in the first fixed angular attitude α ₁ Next, at least one feature point 90 on the marker 70 is photographed, wherein the relative position between the marker 70 and the working position 710 is fixed, and when each feature point 90 is photographed, the position of the current end flange 20 is recorded, which includes the following steps:

s310a, the end flange 20 maintains a first fixed angular attitude α ₁ Or the movement mechanism 10 drives the end flange 20 to rotate to reach the first fixed angle posture alpha ₁ Lower part;

s320a, aiming at each characteristic point 90, the movement mechanism 10 drives the tail end flange 20 to drive the camera 30 to move to the vicinity of each characteristic point 90, and photographing is respectively carried out at the vicinity of each characteristic point 90; and, when each of the feature points 90 is photographed, the position of the front end flange 20 is recorded.

It will be understood that if the number of feature points 90 is 2 or more, in step S400, the conversion relationship T is determined based on the image obtained by capturing each feature point 90 _PC The method for acquiring the position relationship between each feature point 90 and the center of the camera during shooting comprises the following steps:

s410a, after photographing near each feature point 90, acquiring the coordinates of the pattern corresponding to each feature point 90 in a pixel coordinate system by an image processing technology according to the image acquired after each photographing;

S420a, according to the conversion relation T between the pixel coordinate system obtained in step S200 and the physical position with the camera center as the reference _PC The coordinate value of each feature point corresponding to the graph in the pixel coordinate system obtained in step S410a can be converted to obtain the positional relationship between each feature point 90 and the center of the camera during photographing, that is, the distance Δx between each feature point and the center of the camera in the X direction and the distance Δy between each feature point and the center of the camera in the Y direction under the coordinate system of the motion mechanism during photographing.

Referring to fig. 5, it can be understood that if there are 2 feature points 90, in step S500, according to the positional relationship between each feature point 90 and the center of the camera at the time of photographing, the current end flange position at the time of photographing each feature point 90, and the camera TOOL _C When the camera center is aligned with the feature points 90 or the set relative positions between the feature points 90 are obtained, the position P of the end flange 20 ₁ The method comprises the following steps:

s510a, acquiring the positions P of the two camera centers when the camera centers are respectively aligned with the two feature points 90 according to the position relationship between each feature point 90 and the camera center during shooting, the TOOL of the camera, and the position of the current end flange 20 during shooting each feature point 90 _O1 And P _O2 ；

Specifically, since in step S400, it is acquired The distance DeltaX between each characteristic point 90 and the center of the camera in the X direction and the distance DeltaY between the distances DeltaX and Y in the coordinate system of the moving mechanism are the same, and the camera 30 is mounted on the end flange 20, so that on the basis of the current position of the end flange 20 when each characteristic point 90 is shot, if the end flange 20 is driven to move the distances DeltaX and DeltaY, the center of the camera is aligned with the corresponding characteristic point 90, and then the camera TOOL is combined _C The positions P of the two camera centers when the camera centers are aligned with the two feature points 90 can be calculated _O1 And P _O2 . It should be understood, of course, that it is not necessary to actually move the end flange 20 by Δx and Δy as described above, and it may be obtained by calculation only.

S520a, according to the obtained P _O1 And P _O2 Obtaining P _O1 P _O2 The position of the central point O of the connecting line;

it can be appreciated that P is acquired _O1 P _O2 The position of the center point O of the connection line may represent the position when the center of the camera is obtained and aligned with the midpoint of the connection line between the two feature points 90, and at this time, the set relative position between the feature points 90 is the midpoint of the connection line between the two feature points 90.

S530a, according to the position of the center point O and the camera TOOLc, obtaining the position P of the end flange 20 when the center of the camera is aligned with the center point O ₁ 。

Specifically, since the position of the center point O may represent the position when the camera center is aligned with the midpoint of the line between the two feature points 90, the position P of the end flange 20 can be obtained by the camera toloc when the camera center is aligned with the relative position between the feature points 90 (i.e., the midpoint of the line between the two feature points 90) ₁ 。

It will be appreciated that when dual feature points 90 are used, the camera center is required to be aligned with the set relative position between the feature points 90, except for P as described above _O1 P _O2 The central point O of the connection line can be P _O1 P _O2 Other types of points such as the three-point of the line, as long as the relative position between the two feature points 90 is fixed. It will be appreciated that the detailed procedure of step S500 is given above only for the dual feature points 90However, in the calibration method of the present invention, the case of the multiple feature points 90 is not limited to the double feature points 90, and three feature points 90 or more than three feature points 90 may be selected, for example, the set relative positions between the feature points 90 whose camera centers are to be aligned may be set in the case of using three feature points 90, so long as the relative positions between the three feature points 90 and the center point, the inner center, the vertical center, etc. of the triangle structure are fixed.

It will be appreciated that if the dual feature points 90 are used, in step S600, the angle rol-V at which the marker 70 is marked is obtained based on the image obtained by capturing the feature points 90 ₁ The method comprises the following steps:

s600a, according to P obtained in step S500 _O1 And P _O2 Obtaining P _O1 P _O2 The angle of the line to the horizontal center line of the camera, i.e. the angle ROLL-V at which the marker is calibrated ₁ 。

Specifically, according to P _O1 And P _O2 Can obtain P _O1 P _O2 Angle gamma between connecting line and X axis under moving mechanism coordinate system ₁ While the end flange is in the first fixed angular attitude alpha ₁ So the included angle between the horizontal center line of the camera and the X axis under the coordinate system of the movement mechanism is fixed, and P can be obtained by calculation _O1 P _O2 An angle of the line to the horizontal center line of the camera; for example, assume that when the angular pose of the end flange is 0 degrees, the camera horizontal center line forms an angle γ with the X-axis in the motion mechanism coordinate system ₂ The end flange is positioned at a first fixed angle posture alpha ₁ When the camera is in use, an angle gamma is formed between the horizontal central line of the camera and the X axis under the coordinate system of the movement mechanism ₂ +α ₁ Thereby the angle ROLL-V of the marked object can be calculated ₁ Is gamma ₁ -(γ ₂ +α ₁ )。

It will be appreciated that when three or more feature points 90 are used, the step S600 is substantially the same as the step S600a, and only any two feature points among the feature points 90 are required to be used as the dual feature points 90 in the step S600a, and then P is obtained correspondingly _O1 And P _O2 Then get P again _O1 P _O2 The angle between the connecting line and the horizontal center line of the camera is only needed.

Referring to fig. 6, it can be understood that if the feature points 90 are one, in step S300, at least one feature point 90 on the marker 70 is photographed in the first fixed angle posture of the end flange 20, and when each feature point 90 is photographed, the current end flange position is recorded, which includes the following steps:

s310b, the end flange 20 maintains a first fixed angular attitude α ₁ Or the movement mechanism 10 drives the end flange 20 to rotate to reach the first fixed angle posture alpha ₁ Lower part;

s320b, the movement mechanism 10 drives the end flange 20 to drive the camera 30 to move to the vicinity of the feature point 90, and then the camera 30 photographs the feature point 90, and records the current position of the end flange 20.

Moving the camera 30 near the feature point 90 ensures that the feature point 90 is within the camera field of view 60 during photographing to ensure that the captured image has a pattern corresponding to the feature point 90 so that the coordinates of the pattern corresponding to the feature point 90 in the pixel coordinate system can be obtained in a subsequent step.

It will be appreciated that if the feature points 90 are one, in step S400, the conversion relationship T is determined based on the image obtained by capturing each feature point 90 _PC The method for acquiring the position relationship between each feature point 90 and the center of the camera during shooting comprises the following steps:

S410b, acquiring coordinate values of the graph corresponding to the feature points 90 in a pixel coordinate system through an image processing technology according to the image shot in the step S300;

s420b, according to the conversion relation T between the pixel coordinate system obtained in the step S200 and the physical position with the camera center as the reference _PC The coordinate values of the feature points 90 corresponding to the graphics in the pixel coordinate system obtained in step S410b can be converted to obtain the positional relationship between the feature points and the camera center, i.e. the distances Δx and Δy between the feature points 90 and the camera center in the X direction and the Y direction under the motion coordinate system are obtained.

Referring to fig. 6, it can be understood that if the feature points 90 are one, in step S500, according to the positional relationship between each feature point and the center of the camera at the time of photographing, the current end flange position at the time of photographing each of the feature points 90, and the camera TOOL _C When the camera center is aligned with the feature points 90 or the set relative positions between the feature points 90 are obtained, the position P of the end flange 20 ₁ The method comprises the following steps:

s510b, acquiring the position of the camera center when the camera center is aligned with the feature point 90 according to the position relation between the feature point 90 and the camera center acquired in the step S400 and the position of the end flange 20 when photographing in the step S400;

It is to be understood that the position of the camera center is acquired here in the same manner as described in step S510a, with respect to the camera center alignment feature point 90.

S520b, based on the position of the camera center obtained in step 510b and the camera TOOLC, the position P of the end flange 20 can be calculated when the camera center is aligned with the feature point 90 ₁ 。

It should be understood that, similarly, the position of the camera center in the alignment feature point 90 is obtained without actually moving the camera center and aligning the feature point 90, which simply requires calculating the position of the camera center in the alignment feature point 90, and finally obtaining the position P of the end flange 20 in the corresponding state ₁ And (3) obtaining the product.

It will be appreciated that if a single feature 90 is used, the line between the features 90 cannot be relied upon to obtain the angle ROLL-V at which the marker 70 is marked ₁ The method comprises the steps of carrying out a first treatment on the surface of the To solve this problem, when the feature point 90 is one, the feature point 90 is provided in a non-circular and other asymmetric structure, for example, the feature point 90 is provided in a non-isosceles triangle or a non-isosceles trapezoid, etc.; when the feature points 90 are set to the triangular form, specifically, in step S600, the image angle rol-V of the product 80 is acquired from the photographed image ₁ The method comprises the following steps:

s600b, obtaining characteristic points according to the image processing technology90, the angle between the connecting line between two pixel points on the boundary of the feature point 90 and the horizontal center line of the camera (namely the u axis of the pixel coordinate system) in the pixel coordinate system is obtained, namely the angle ROLL-V when the marker 70 is marked ₁ 。

It can be understood that, during photographing, the boundary of the feature point 90 can be obtained by image processing technology by means of the difference of color, brightness and the like between the figure of the feature point 90 and the background figure in the image, and then the coordinates of two pixel points on the boundary of the feature point 90 can be obtained, and then the calibration angle ROLL-V of the marker can be obtained in a similar form to S600a ₁ 。

It is understood that the above step S600b is equally applicable to the case where the feature points 90 are in an asymmetric structure.

It will be appreciated that in step S700, the position P of the end flange 20 is obtained when the work tool 40 reaches the work position 710 in a desired posture _T And angular attitude alpha _T This may be accomplished in a number of different ways, a few of which are listed below.

In the first mode, in step S700, the position P of the end flange 20 is acquired when the work tool 40 reaches the work position 710 in a desired posture _T And angular attitude alpha _T The method comprises the following steps:

s710a, controlling the movement mechanism 10 to drive the tail end flange 20 to move in a manual teaching mode, so that the working tool 40 reaches the working position 710 in a desired posture;

s720a, recording the current end flange position P _T And angular attitude alpha _T 。

It can be understood that in the first mode, the first manual teaching is mainly implemented, and when the first mode is adopted in step S700, the position relationship between the working tool 40 and the end flange 20 is not required to be acquired or calculated, i.e. the working tool is not required to be calculated, throughout the whole calibration method; and because the working tool 40 is irregular in some working scenarios, the working tool is difficult to obtain effectively, and the working tool and the image processing technology are adopted for obtainingTaking P _T And alpha _T The acquisition process is complex, the operation is complex, relatively complex offset calculation such as component calculation of the XY direction caused by angle offset is needed, and meanwhile, the adjustment of the angle is needed to be tried manually to obtain the correct working angle posture. Step S700 is therefore simpler in its operation by employing the first approach, and is equally applicable even if work tool 40 is irregular.

In the second mode, if the marker 70 is a carrier or a product carrying a product, step S700 is performed to obtain the position P of the end flange when the work tool reaches the work position in a desired posture _T And angular attitude alpha _T The method comprises the following steps:

s710b at the end flange 20 in a first fixed angular position α ₁ At the time, the product 80 is photographed by a camera, so as to obtain a product image;

s720b, converting the conversion relation T between the product image, the pixel coordinates and the physical position with the camera center as the reference _PC TOOL of camera _C Acquiring the operation position of the product 80;

it will be appreciated that, specifically, assuming that the operation to be performed on the product 80 is a grabbing product or a welding operation, the characteristics of the boundary or the weld of the product 80 can be obtained by an image processing technique, and the pixel coordinates of the grabbing position or the welding position of the product 80 in the pixel coordinate system can be deduced and calculated according to the characteristics of the boundary or the weld of the product 80, and then according to the conversion relation T between the pixel coordinates, the pixel coordinates and the physical position with the camera center as a reference _PC TOOL of camera _C The position coordinates of the camera center, namely the operation position of the product under the coordinate system of the movement mechanism, can be obtained when the camera center is aligned to the grabbing position or the welding position (namely the operation position);

S730b, according to the working position of the product 80, acquiring the position P of the end flange 20 when the working tool 40 reaches the working position 710 in a desired posture _T And angular attitude alpha _T 。

It can be appreciated that the calibration method described in connection with the embodiments of the present inventionThe calculated tool can calculate P based on the coordinate of the working position 710 of the product 80 in the coordinate system of the motion mechanism _T And alpha _T 。

In the third aspect, if the marker 70 is a carrier for receiving the product 80 and a placement structure for placing the product 80 is provided on the carrier, step S700 is performed to obtain the position P of the end flange 20 when the working tool 40 reaches the working position 710 in the desired posture _T And angular attitude alpha _T The method comprises the following steps:

acquiring the position P of the end flange when the work tool 40 reaches the work position 710 in a desired attitude _T And angular attitude alpha _T The method comprises the following steps:

s710c at the end flange 20 in a first fixed angular position α ₁ At the time, the placement structure is photographed by the camera 30, and an image of the placement structure is obtained;

s720c, according to the conversion relation T between the image of the placement structure, the pixel coordinates and the physical position with the camera center as the reference _PC TOOL of camera _C Acquiring a working position 710 of the product 80;

it will be appreciated that, assuming that the placement structure is a containing cavity of the product 80, the image of the placement structure is processed by image processing technology to obtain the characteristics of the boundary of the containing cavity, and the pixel coordinates of the placement position of the product 80 in the pixel coordinate system can be deduced and calculated according to the characteristics of the boundary of the containing cavity, and then the conversion relation T between the pixel coordinates, the pixel coordinates and the physical position with the camera center as the reference is calculated _PC TOOL of camera _C The position coordinates of the camera center, that is, the working position 710 of the product under the coordinate system of the moving mechanism, can be obtained when the camera center is aligned to the grabbing position or the welding position (that is, the working position 710);

s730c, according to the working position 710 of the product, acquiring the position P of the end flange 20 when the working tool 40 reaches the working position 710 in a desired posture _T And angular attitude alpha _T 。

It will be appreciated that embodiments of the invention are incorporated thereinThe calculated tool, etc. can calculate P based on the coordinate of the working position 710 of the product 80 in the coordinate system of the motion mechanism _T And alpha _T 。

If the operation is performed only for a single product 80 or a single operation position 710 on a carrier, in step S700, the operation is performed only in the first to third manners described above, however, in a real industrial production process, there is often a case where the operation is required for a plurality of products 80 at the same time, or there is a need to perform the operation for a plurality of operation positions 710 on the same product 80. To solve this need, the following methods may be combined on the basis of the first to third modes described above.

Referring to FIG. 5, it can be appreciated that when the marker 70 corresponds to N job positions 710, N is greater than or equal to 1; step S700 of acquiring the position P of the end flange 20 when the work tool 40 reaches the work position 710 in a desired posture _T And angular attitude alpha _T The method comprises the following steps:

s700d, for each work position 710, the position P of the end flange 20 is obtained when the work tool 40 reaches each work position 710 in the desired posture _Tn And angular attitude alpha _Tn Wherein N is greater than or equal to N is greater than or equal to 1.

It will be appreciated that, at S700d, for any one of the job positions 710, the corresponding P may be acquired by selecting any one of the first to third modes described above _Tn And alpha _Tn . For example, the movement mechanism 10 may be controlled to drive the end flange 20 to move by manual teaching, and cause the work tool 40 to arrive at each work position 710 in a desired posture, and record the position P of the current end flange 20 when the work tool 40 arrives at each work position 710 _Tn And end flange 20 angular attitude alpha _Tn Wherein N is greater than or equal to N is greater than or equal to 1.

In addition to step S700d, step S800 is performed according to P ₁ 、α ₁ 、P _T 、α _T Obtain (P) ₁ 、α ₁ ) And (P) _T 、α _T ) The transformation relation offset of (1) includes The steps are as follows:

s800d according to P ₁ 、α ₁ And the recorded position P of each set of end flanges _Tn And end flange angular attitude alpha _Tn Obtain (P) ₁ 、α ₁ ) And each group (P _Tn 、α _Tn ) Is a transform of offset-n.

It can be understood that, when the marker 70 is a carrier carrying the products 80, the N operation positions 710 corresponding to the marker 70 may be distributed on N different products 80, so that in the operation process after the teaching is completed through the steps S700d and S800d, according to the transformation relationship offset-N corresponding to each product 80, the operation can be ensured to be completed with the desired gesture and position for each product 80 in the subsequent operation process, the operation accuracy of each product 80 is ensured, and the requirement of simultaneously operating a plurality of products 80 is satisfied.

It will be appreciated that, when the tag 70 is a carrier waiting to receive the product 80, the N operation positions 710 corresponding to the tag 70 may be N different placement structures of the product 80 distributed on the carrier, so that in the operation process after the completion of teaching, according to the transformation relationship offset-N of the placement position (operation position) of each product 80, in the subsequent operation process, the product 80 can be placed one by one in a desired posture and position for each product 80 and the like, so as to meet the requirement of placing the plurality of products 80 simultaneously.

It will be appreciated that when the tag 70 is the product 80, the N job positions 710 corresponding to the tag 70 may be N different job positions 710 distributed on the same product 80, so that through the above steps S700d and S800d, the job can be guaranteed to be completed in a desired posture and position for each job position 710 on the product 80 during the job process after the teaching is completed, the accuracy of the job at each job position 710 is guaranteed, and the requirement of simultaneously performing the job on multiple job positions 710 on the same product 80 is satisfied.

Embodiments of the second aspect of the present invention and the visual recognition-based working method implemented in the third aspect of the present invention are described below, both being equally applicable to the working apparatus described in the embodiments of the first aspect.

Referring to fig. 7, the visual recognition-based operation method according to the second embodiment of the present invention includes two major method steps, wherein the first major step is a calibration step, and the second major step is an operation step, wherein the calibration step adopts the visual recognition-based calibration method according to the first embodiment of the present invention, and is a case that a single feature point 90 is adopted in the visual recognition-based operation method according to the first embodiment of the present invention, and therefore, a detailed description will not be given for the calibration step portion in the first major step.

Referring to fig. 8, the visual recognition-based operation method according to the second aspect of the present invention further includes the following second major steps:

s900b at the end flange 20 in a first fixed angular position α ₁ Photographing the feature points 90 on the marker 70 again, and recording the position coordinates of the current end flange 20;

s1000b, converting the relationship T according to the image obtained by photographing the marker 70 again in S900b _PC The positional relationship between the feature points 90 and the camera center at the time of re-shooting is obtained;

s1100b, according to the position relation between the feature point 90 and the camera center during re-shooting, the current end flange position during re-shooting the feature point 90, and the camera TOOL _C Acquiring the position P of the camera center of the camera with the camera center aligned with the feature point 90 again ₅ ；

S1200b, acquiring the angle ROLL-V of the marker 70 during operation according to the image obtained by shooting the marker 70 again in S900b ₂ ；

It will be appreciated that the detailed steps of S900b to S1200b are substantially identical to the steps of S300 to S600 under the single feature point 90 in the embodiment of the first aspect;

s1300b, according to ROLL-V ₂ And ROLL-V ₁ Deviation between camera TOOL _C Acquiring the center of the camera to be at P ₅ Position and end flange in angular attitude alpha ₁ +(ROLL-V ₂ -ROLL-V ₁ ) The lower part of the upper part is provided with a lower part,position P of the end flange ₆ ；

It can be appreciated that when the camera center is at P ₅ Positioned with the end flange 20 in the angular attitude alpha ₁ +(ROLL-V ₂ -ROLL-V ₁ ) Indicating the position and angle of the marker 70 in the camera field of view 60, and the end flange 20 at calibration at P ₁ And alpha ₁ The position and angle of the marker 70 in the camera view 60 are consistent in the state according to P ₅ Position coordinates of (a) and camera TOOL _C The position P of the end flange 20 in this state can be deduced ₆ ；

S1400b according to P ₆ Angle posture alpha ₁ +(ROLL-V ₂ -ROLL-V ₁ ) The relation offset is transformed to obtain the end flange position P at the working position 710 ₇ And alpha ₇ The method comprises the steps of carrying out a first treatment on the surface of the Moving the end flanges 20 to P ₇ And rotates the end flange 20 to the angular attitude alpha ₇ The work is performed at the work position by the work tool.

It will be appreciated that when the end flange is at P ₆ Position and angle posture alpha ₁ +(ROLL-V ₂ -ROLL-V ₁ ) The position and angular orientation of end flange 20 and work tool 40 in the marker coordinate system established with reference to marker 70 itself is then determined as compared to the position and angular orientation of end flange at the calibration method step (i.e., at P1 and alpha with the end flange ₁ In the state of (2) is consistent, and thus passes P at this time ₆ Angle posture alpha ₁ +(ROLL-V ₂ -ROLL-V ₁ ) By shifting the relation offset, the end flange position P of the work tool 40 at the desired posture at the work position 710 can be obtained ₇ And angular attitude alpha ₇ Then the end flange 20 is driven to move by the moving mechanism 10, and the end flange 20 is positioned at the position P ₇ And angular attitude alpha ₇ I.e., work may be performed directly at work location 710 by work tool 40.

Referring to fig. 9 and 10, the visual recognition-based operation method according to the third embodiment of the present invention includes two main method steps, wherein the first main step is a calibration step, and the second main step is an operation step, wherein the calibration step adopts the visual recognition-based calibration method according to the first embodiment of the present invention, and is a case that a plurality of feature points are adopted in the visual recognition-based operation method according to the first embodiment of the present invention, so that a detailed description will not be given for the calibration step portion in the first main step.

Referring to fig. 10, in a second major step of the visual recognition-based operation method according to the third aspect of the present invention, the operation step includes the steps of:

s900a is in a first fixed angular position alpha at the end flange 20 ₁ Photographing each feature point 90 on the marker 70 again, and recording the position coordinates of the current end flange 20 when photographing each feature point 90;

S1000a, capturing the image obtained by each feature point 90 again according to the transformation relation T in S900a _PC Obtaining the position relationship between each feature point 90 and the center of the camera during re-shooting;

s1100a, according to the position relation between each feature point 90 and the center of the camera when shooting again in S1000a, the current end flange position when shooting again each feature point 90, and the TOOL of the camera _C The position P of the camera center is obtained in which the camera center is again aligned with the position P of the camera center between the feature points 90 set relative to each other ₅ ；

S1200a, acquiring the angle ROLL-V of the marker 70 during operation according to the image obtained by shooting each feature point 90 again in S900a ₂ ；

It will be appreciated that the detailed steps of S900a to S1200a are substantially identical to the steps of S300 to S600 under the plurality of feature points 90 in the embodiment of the first aspect;

s1300a, according to ROLL-V ₂ And ROLL-V ₁ Deviation between camera TOOL _C Acquiring the center of the camera to be at P ₅ Positioned with the end flange 20 in the angular attitude alpha ₁ +(ROLL-V ₂ -ROLL-V ₁ ) Position P of end flange 20 in the state of (2) ₆ ；

S1400a according to P ₆ Angle posture alpha ₁ +(ROLL-V ₂ -ROLL-V ₁ ) Switch with changeable functionIs offset to obtain the end flange position P at the working position 710 ₇ And alpha ₇ The method comprises the steps of carrying out a first treatment on the surface of the Moving the end flange to P ₇ And rotating the end flange to an angular attitude alpha ₇ Work is performed at work position 710 by work tool 40.

In the description of the present specification, reference to the terms "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

While embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that: many changes, modifications, substitutions and variations may be made to the embodiments without departing from the spirit and principles of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. The calibration method based on visual identification is characterized by being applied to operation equipment, wherein the operation equipment comprises an end effector and a movement mechanism, the end effector comprises an end flange, a camera and an operation tool, the camera and the operation tool are arranged on the end flange, and the movement mechanism can drive the end flange to freely move on an XY plane and can drive the end flange to rotate around an axis of the end flange;

The calibration method comprises the following steps:

the TOOL of the camera is obtained by determining the position relationship between the center and the tail end flange of the camera _C ；

According to the TOOL of the camera _C Determining that the end flange is in a first fixed angular attitude alpha ₁ At this time, the conversion relation T between the pixel coordinates and the physical position with reference to the camera center _PC ；

At the end flange in a first fixed angular attitude alpha ₁ Shooting at least one characteristic point on the marker, wherein the relative position between the marker and the working position is fixed, and recording the position of the current tail end flange when shooting each characteristic point;

according to the image obtained by shooting each characteristic point and the conversion relation T _PC Acquiring the position relationship between each characteristic point and the center of the camera during shooting;

according to the position relationship between each characteristic point and the center of the camera during shooting, the current end flange position and the TOOL of the camera during shooting each characteristic point _C When the camera center is aligned with the feature points or the set relative positions among the feature points are obtained, the position P of the tail end flange ₁ ；

Acquiring an angle ROLL-V of the marked object when the marked object is marked according to the image obtained by shooting the characteristic points ₁ ；

Acquiring the position P of the end flange when the work tool reaches the work position in a desired posture _T And angular attitude alpha _T ；

According to P ₁ 、α ₁ 、P _T 、α _T Obtain (P) ₁ 、α ₁ ) And (P) _T 、α _T ) Is a transform of the relation offset.

2. The visual recognition-based calibration method according to claim 1, wherein determining the positional relationship between the center of the camera and the end flange obtains the TOOL of the camera _C The method comprises the following steps:

At the end flange at a certain angular attitude alpha ₄ The lower part of the upper part is provided with a lower part,photographing the calibration object from at least three different position points respectively, and obtaining an angle posture alpha according to the position of the tail end flange and the pixel coordinates of the calibration object in the image when photographing each time ₄ The transformation relation between the lower pixel coordinates and the flange end position, and the position P required to be reached by the end flange when the calibration object is positioned in the center of the image is obtained according to the transformation relation ₄ The method comprises the steps of carrying out a first treatment on the surface of the Wherein alpha is ₄ Not equal to alpha ₃ ；

3. The visual recognition-based calibration method according to claim 2, wherein the method is based on a ₃ And P ₃ Alpha and alpha ₄ And P ₄ Calculate the TOOL of the camera _C The method comprises the following steps:

will be alpha ₃ 、P ₃ 、α ₄ 、P ₄ The related numerical values in the map f are substituted into the mapping relation f _TOOL The camera TOOL, wherein,

4. The visual recognition-based calibration method according to claim 2, wherein the method is based on a ₃ And P ₃ Alpha and alpha ₄ And P ₄ Calculate the TOOL of the camera _C The method comprises the following steps:

Acquiring the position P of the end flange ₄ And the angle posture is alpha ₄ State of (2)The camera center coordinates in the motion mechanism coordinate system are related to the TOOL of the camera _C Expression T of medium parameter ₄ ；

5. The visual recognition-based calibration method according to claim 1, wherein the method is based on camera TOOL _C Determining that the end flange is in a first fixed angular attitude alpha ₁ At this time, the conversion relation T between the pixel coordinates and the physical position with reference to the camera center _PC The method comprises the following steps:

at the end flange in a first fixed angular attitude alpha ₁ Photographing the standard points from at least 3 different positions respectively, so that the corresponding graph of the standard points in the image is not collinear among at least three points in a pixel coordinate system; when photographing the standard points at each position, recording the current position of the tail end flange respectively;

according to the terminal flange position recorded when each shooting of the calibration point and the TOOL of the camera _C Converting to obtain the position of the camera center of each shooting calibration point, and acquiring the conversion relation T between the pixel coordinate and the physical position taking the camera center as a reference according to the pixel coordinate corresponding to the calibration point under each shooting position and the position of the camera center of each shooting calibration point _PC 。

6. The visual recognition-based calibration method according to claim 1, wherein if the number of feature points is 2,

wherein,according to the position relationship between each characteristic point and the center of the camera during shooting, the current end flange position and the TOOL of the camera during shooting each characteristic point _C When the camera center is aligned with the feature points or the set relative positions among the feature points are obtained, the position P of the tail end flange ₁ The method comprises the following steps:

according to the position relationship between each feature point and the center of the camera and the TOOL of the camera during shooting _C Shooting the current end flange position of each characteristic point, and acquiring the positions P of the two camera centers when the camera centers are respectively aligned with the two characteristic points _O1 And P _O2 ；

According to the obtained P _O1 And P _O2 Obtaining P _O1 P _O2 The position of the central point O of the connecting line;

based on the position of the center point O and the camera TOOL _C Acquiring a terminal flange position P when a camera center is aligned with a center point O ₁ 。

7. The visual recognition-based calibration method according to claim 6, wherein, if the number of feature points is two,

wherein, according to the image obtained by shooting each characteristic point, the angle ROLL-V of the marked object when the marked object is marked is obtained ₁ The method comprises the following steps:

according to the obtained P _O1 And P _O2 Obtaining P _O1 P _O2 The angle between the line and the horizontal center line of the camera, i.e. the angle ROLL-V when the marker is calibrated ₁ 。

8. The visual recognition-based calibration method according to claim 1, wherein, if the feature point is one,

wherein, according to the position relationship between each characteristic point and the center of the camera during shooting, the current end flange position and the TOOL of the camera during shooting each characteristic point _C When the camera center is aligned with the feature points or the set relative positions among the feature points are obtained, the position P of the tail end flange ₁ The method comprises the following steps:

according to the phaseTOOL machine _C The position relation between the characteristic points and the camera center during shooting and the current end flange position during shooting the characteristic points are obtained, and the position of the camera center when the camera center is aligned to the characteristic points is obtained;

according to the position of the camera center when the camera center is aligned with the feature point and the camera TOOLC, the position P of the tail end flange is obtained when the camera center is aligned with the feature point ₁ 。

9. The visual recognition-based calibration method according to claim 1, wherein the position P of the end flange is obtained when the work tool reaches the work position in a desired posture _T And angular attitude alpha _T The method comprises the following steps:

the motion mechanism is controlled to drive the tail end flange to move and enable the working tool to reach the working position in a desired gesture by means of manual teaching, and the current position P of the tail end flange is recorded _T And angular attitude alpha _T 。

10. The method of claim 1, wherein, if the marker is a carrier or a product carrying the product,

acquiring the position P of the end flange when the work tool reaches the work position in a desired posture _T And angular attitude alpha _T The method comprises the following steps:

at the end flange in a first fixed angular attitude alpha ₁ When the product is photographed by a camera, a product image is obtained;

based on the conversion relation T between the product image, the pixel coordinates and the physical position with the camera center as the reference _PC TOOL of camera _C Acquiring the operation position of a product;

acquiring the position P of the tail end flange when the working tool reaches the working position in a desired posture according to the working position of the product _T And angular attitude alpha _T 。

11. The method of claim 1, wherein if the marker is a carrier for receiving the product and the carrier is provided with a placement structure for placing the product,

at the end flange in a first fixed angular attitude alpha ₁ When the camera is used, the camera is used for photographing the placement structure, so that an image of the placement structure is obtained;

based on the conversion relation T between the image of the placement structure, the pixel coordinates and the physical position referenced to the camera center _PC TOOL of camera _C Acquiring the operation position of a product;

12. The visual recognition-based calibration method according to claim 1, wherein;

the marker corresponds to N operation positions, and N is greater than or equal to 1;

wherein, when the working tool reaches the working position in a desired posture, the position P of the tail end flange is obtained _T And angular attitude alpha _T The method comprises the following steps:

for each working position, the position P of the end flange 20 is obtained when the working tool 40 reaches each working position in a desired posture _Tn And angular attitude alpha _Tn Wherein N is greater than or equal to N is greater than or equal to 1.

13. The visual recognition-based calibration method of claim 12, wherein; according to P ₁ 、α ₁ 、P _T 、α _T Obtain (P) ₁ 、α ₁ ) And (P) _T 、α _T ) The conversion relation offset of (2) includes the following steps:

according to P ₁ 、α ₁ And the recorded position P of each set of end flanges _Tn And end flange angular attitude alpha _Tn Obtain (P) ₁ 、α ₁ ) And each group (P _Tn 、α _Tn ) Is a transform of offset-n.

14. The operation method based on visual identification is characterized by being applied to operation equipment, wherein the operation equipment comprises an end effector and a movement mechanism, the end effector comprises an end flange, a camera and an operation tool, the camera and the operation tool are arranged on the end flange, and the movement mechanism can drive the end flange to freely move on an XY plane and can drive the end flange to rotate around an axis of the end flange;

The operation method comprises the following steps:

At the end flange is in a first fixed stateAngular attitude alpha ₁ Shooting the feature points on the marker again, and recording the position coordinates of the current tail end flange; based on the image obtained by shooting the feature points again, the conversion relation T _PC Obtaining the position relation between the feature points and the center of the camera during re-shooting; according to the position relation between the characteristic points and the camera center during the re-shooting, the current terminal flange position during the re-shooting of the characteristic points and the camera TOOL _C Acquiring the position P of the camera center when the camera center is aligned with the feature point again ₅ The method comprises the steps of carrying out a first treatment on the surface of the Acquiring an angle ROLL-V of the marker during operation according to the image obtained by shooting the characteristic points again ₂ The method comprises the steps of carrying out a first treatment on the surface of the According to ROLL-V ₂ And ROLL-V ₁ Deviation between camera TOOL _C Acquiring the center of the camera to be at P ₅ Position and end flange in angular attitude alpha ₁ +(ROLL-V ₂ -ROLL-V ₁ ) Lower end flange position P ₆ The method comprises the steps of carrying out a first treatment on the surface of the According to P ₆ Angle posture alpha ₁ +(ROLL-V ₂ -ROLL-V ₁ ) Converting the relation offset to obtain the end flange position P in the working position ₇ And angular attitude alpha ₇ The method comprises the steps of carrying out a first treatment on the surface of the Moving the end flange to P ₇ And rotating the end flange to an angular attitude alpha ₇ The work is performed at the work position by the work tool.

15. The operation method based on visual identification is characterized by being applied to operation equipment, wherein the operation equipment comprises an end effector and a movement mechanism, the end effector comprises an end flange, a camera and an operation tool, the camera and the operation tool are arranged on the end flange, and the movement mechanism can drive the end flange to freely move on an XY plane and can drive the end flange to rotate around an axis of the end flange;

The operation method comprises the following steps:

the TOOL of the camera is obtained by determining the position relationship between the center and the tail end flange of the camera _C The method comprises the steps of carrying out a first treatment on the surface of the According to the TOOL of the camera _C Determining that the end flange is in a first fixed angular attitude alpha ₁ At this time, the conversion relation T between the pixel coordinate system and the physical position with the camera center as the reference _PC The method comprises the steps of carrying out a first treatment on the surface of the At the end flange in a first fixed angular attitude alpha ₁ Shooting a plurality of characteristic points on the marker, wherein the relative positions between the marker and the operation position are fixed, and recording the position of the current tail end flange when shooting each characteristic point; according to the image obtained by shooting each characteristic point and the conversion relation T _PC Acquiring the position relationship between each characteristic point and the center of the camera during shooting; according to the position relationship between each characteristic point and the center of the camera during shooting, the current end flange position and the TOOL of the camera during shooting each characteristic point _C When the set relative position between the camera center alignment feature points is obtained, the position P of the tail end flange ₁ The method comprises the steps of carrying out a first treatment on the surface of the Acquiring the angle ROLL-V of the marked object when the marked object is marked according to the image obtained by shooting the marked point ₁ The method comprises the steps of carrying out a first treatment on the surface of the Acquiring the position P of the end flange when the work tool reaches the work position in a desired posture _T And angular attitude alpha _T The method comprises the steps of carrying out a first treatment on the surface of the According to P ₁ 、α ₁ 、P _T 、α _T Obtain (P) ₁ 、α ₁ ) And (P) _T 、α _T ) Is a transform of the offset;

at the end flange in a first fixed angular attitude alpha ₁ Photographing each characteristic point on the marker again, and recording the position of the current tail end flange when photographing each characteristic point; according to the image obtained by shooting each characteristic point again and the conversion relation T _PC Obtaining the position relation between each characteristic point and the center of the camera during secondary shooting; according to the position relation between each characteristic point and the center of the camera during re-shooting, the current end flange position and the camera TOOL during re-shooting each characteristic point _C Acquiring a position P of a camera center in which the camera center is aligned again with a set relative position between feature points ₅ The method comprises the steps of carrying out a first treatment on the surface of the Acquiring an angle ROLL-V of the marker during operation according to the image obtained by shooting each characteristic point again ₂ The method comprises the steps of carrying out a first treatment on the surface of the According to ROLL-V ₂ And ROLL-V ₁ Deviation between camera TOOL _C Acquiring the center of the camera to be at P ₅ Position and end flange in angular attitude alpha ₁ +(ROLL-V ₂ -ROLL-V ₁ ) In the state of (2) the position P of the end flange ₆ The method comprises the steps of carrying out a first treatment on the surface of the According to P ₆ Angle posture alpha ₁ +(ROLL-V ₂ -ROLL-V ₁ ) Converting the relation offset to obtain the end flange position P in the working position ₇ And angular attitude alpha ₇ The method comprises the steps of carrying out a first treatment on the surface of the Moving the end flange to P ₇ And rotating the end flange to an angular attitude alpha ₇ The work is performed at the work position by the work tool.