CN118081778A

CN118081778A - Machine vision screw screwing system, control method thereof, screw screwing equipment and medium

Info

Publication number: CN118081778A
Application number: CN202410156173.6A
Authority: CN
Inventors: 陈丹
Original assignee: Midea Welling Motor Technology Shanghai Co Ltd; Guangdong Midea Intelligent Technologies Co Ltd
Current assignee: Midea Welling Motor Technology Shanghai Co Ltd; Guangdong Midea Intelligent Technologies Co Ltd
Priority date: 2024-02-04
Filing date: 2024-02-04
Publication date: 2024-05-28

Abstract

The application discloses a machine vision screw screwing system, a control method thereof, screw screwing equipment and a medium, wherein the method comprises the following steps: acquiring image information of a workpiece to be screwed by a first shooting mechanism and a second shooting mechanism; determining first pose information of a workpiece to be screwed according to the image information acquired by the first shooting mechanism, and determining second pose information of the workpiece to be screwed according to the image acquired by the second shooting mechanism; controlling the executing mechanism according to the pose information of the executing mechanism, the pose information of the conveying mechanism and the first pose information of the workpiece to be screwed so that the speeds of the executing mechanism and the workpiece to be screwed are synchronous, and the pose error of the screw screwing mechanism and the workpiece to be screwed is smaller than a first preset error threshold; and controlling the executing mechanism according to the first pose information, the second pose information and the pose information of the screw screwing mechanism so that the pose error of the screw screwing mechanism and the workpiece to be screwed is smaller than a second preset error threshold. Thereby improving the efficiency and positioning accuracy of screwing the screw.

Description

Machine vision screw screwing system, control method thereof, screw screwing equipment and medium

Technical Field

The application relates to the technical field of visual positioning, in particular to a control method of a machine vision screw screwing system, the machine vision screw screwing system, a computer readable storage medium and screw screwing equipment.

Background

At present, the vision guiding screw screwing scenes are stepping scenes, the conveyor belt needs to be frequently stopped, screw screwing efficiency is low, and the vision guiding screw screwing device is only suitable for static scenes or scenes with few disturbance factors, and the positioning accuracy requirement is low.

Disclosure of Invention

The present application aims to solve at least one of the technical problems in the related art to some extent. Therefore, a first object of the present application is to provide a control method for a machine vision screw tightening system, which obtains first pose information and second pose information of a screw workpiece to be screwed through a first shooting mechanism and a second shooting mechanism, and combines the pose information of an executing mechanism and the pose information of a transmission mechanism to synchronize the executing mechanism and a threaded hole of the screw workpiece to be screwed, so that screw tightening can be completed without stopping the transmission mechanism, efficiency of vision guiding screw tightening is optimized, and automation level is improved.

A second object of the present application is to propose a machine vision screwing system.

A third object of the present application is to propose a computer readable storage medium.

A fourth object of the present application is to propose a screwing device.

To achieve the above object, an embodiment of a first aspect of the present application provides a control method of a machine vision screw tightening system, the system including an actuator, a first photographing mechanism, a second photographing mechanism, a transmission mechanism, and a screw tightening mechanism, the first photographing mechanism being fixedly installed at a position having a constant vertical distance from the transmission mechanism, the second photographing mechanism and the screw tightening mechanism being detachably installed on the actuator, the first photographing mechanism and the second photographing mechanism being configured to acquire image information, the transmission mechanism being configured to transmit a workpiece to be screwed, the method comprising: acquiring image information of a workpiece to be screwed by a first shooting mechanism and a second shooting mechanism; determining first pose information of a threaded hole of a workpiece to be screwed according to the image information acquired by the first shooting mechanism, and determining second pose information of the threaded hole of the workpiece to be screwed according to the image acquired by the second shooting mechanism; controlling the executing mechanism according to the pose information of the executing mechanism, the pose information of the conveying mechanism and the first pose information of the threaded hole of the workpiece to be screwed, so that the speeds of the executing mechanism and the workpiece to be screwed are synchronous, and the pose error of the screw screwing mechanism and the threaded hole of the workpiece to be screwed is smaller than a first preset error threshold; and controlling the executing mechanism according to the first pose information, the second pose information, the pose information of the executing mechanism and the pose information of the conveying mechanism so that the pose error of the screw screwing mechanism and the threaded hole of the workpiece to be screwed is smaller than a second preset error threshold.

In some embodiments, controlling the actuator according to pose information of the actuator, pose information of the transmission mechanism, and first pose information of a threaded hole of a screw workpiece to be screwed, includes: determining the first pose error information of the threaded hole of the workpiece to be screwed and the screw screwing mechanism according to the pose information of the executing mechanism, the pose information of the conveying mechanism and the pose information of the threaded hole of the workpiece to be screwed; generating a first motion control instruction according to the first pose error information, the pose information of the executing mechanism and the pose information of the conveying mechanism; and controlling the executing mechanism according to the first motion control instruction.

In some embodiments, controlling the actuator based on the first pose information, the second pose information, the pose information of the actuator, and the pose information of the transfer mechanism includes: fusing the first pose information and the second pose information to obtain fused pose information; determining a threaded hole of a workpiece to be screwed and second pose error information of the screw screwing mechanism according to the fusion pose information, the pose information of the transmission mechanism and the pose information of the execution mechanism; generating a second motion control instruction according to the second pose error information, the pose information of the executing mechanism and the pose information of the conveying mechanism; and controlling the executing mechanism according to the second motion control instruction.

In some embodiments, fusing the first pose information and the second pose information to obtain fused pose information includes: acquiring a first weight coefficient corresponding to the first pose information and a second weight coefficient corresponding to the second pose information; and determining the fusion pose information according to the first pose information, the first weight coefficient, the second pose information and the second weight coefficient.

In some embodiments, the sum of the first weight coefficient and the second weight coefficient is 1, and the first weight coefficient gradually decreases with movement of the actuator and the second weight coefficient gradually increases with movement of the actuator.

In some embodiments, the first weight coefficient and the second weight coefficient are determined by the following formulas:

Wherein Wc represents a first weight coefficient, ws represents a second weight coefficient, p _i represents a weight polynomial coefficient, and Y represents pose information of a workpiece to be screwed.

In some embodiments, fusing the first pose information and the second pose information to obtain fused pose information includes: and fusing the first pose information and the second pose information by adopting a Kalman filtering algorithm to obtain fused pose information.

In some embodiments, the second pose error information is determined using the following formula:

err_init＝f(L₀,P_c,l_2c,l_1c)

Wherein err _init represents second pose error information, L ₀ represents pose deviation between the second photographing mechanism and the screw screwing mechanism, P _c represents fusion pose information, L _2c represents pose information of the executing mechanism, and L _1c represents pose information of the conveying mechanism.

In some embodiments, further comprising: extracting features of the image information to obtain coordinates of feature points; and determining the first pose information and the second pose information by adopting a PNP clustering algorithm or a binocular triangulation positioning algorithm based on the feature points.

In some embodiments, further comprising: acquiring second pose information according to a first preset servo frequency, and acquiring pose information of an executing mechanism and pose information of a conveying mechanism according to the second preset servo frequency; wherein the first preset servo frequency is smaller than the second preset servo frequency.

In order to achieve the above object, a second aspect of the present application provides a machine vision screw tightening system, including an executing mechanism, a first shooting mechanism, a second shooting mechanism, a transmission mechanism, a screw tightening mechanism and a controller, wherein the first shooting mechanism is fixedly installed at a position with a constant vertical distance from the transmission mechanism, the second shooting mechanism and the screw tightening mechanism are detachably installed on the executing mechanism, the transmission mechanism is configured to transmit a workpiece to be screwed, and the first shooting mechanism and the second shooting mechanism are used for acquiring image information of the workpiece to be screwed; the controller is used for determining first pose information of the threaded hole of the workpiece to be screwed according to the image information obtained by the first shooting mechanism, determining second pose information of the threaded hole of the workpiece to be screwed according to the image obtained by the second shooting mechanism, and controlling the executing mechanism according to the pose information of the executing mechanism, the pose information of the conveying mechanism and the first pose information of the threaded hole of the workpiece to be screwed so that the speeds of the executing mechanism and the workpiece to be screwed are synchronous, the pose error of the screw screwing mechanism and the threaded hole of the workpiece to be screwed is smaller than a first preset error threshold value, and controlling the executing mechanism according to the first pose information, the second pose information, the pose information of the executing mechanism and the pose information of the conveying mechanism so that the pose error of the screw screwing mechanism and the threaded hole of the workpiece to be screwed is smaller than a second preset error threshold value.

To achieve the above object, a third aspect of the present application provides a computer-readable storage medium having stored thereon a control program of a machine vision screw tightening system, which when executed by a processor, implements the control method of the machine vision screw tightening system described above.

To achieve the above object, a fourth aspect of the present application provides a screw driving apparatus including the machine vision screw driving system described above.

According to the control method of the machine vision screw screwing system, the first shooting mechanism and the second shooting mechanism are used for acquiring the first pose information and the second pose information of the screw workpiece to be screwed, and the pose information of the executing mechanism and the pose information of the conveying mechanism are combined to enable the executing mechanism to be synchronous with the threaded holes of the screw workpiece to be screwed, screw screwing can be completed on the premise that the conveying mechanism does not stop, screw screwing efficiency of vision guiding is optimized, and automation level is improved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

FIG. 1 is a schematic diagram of a machine vision screw tightening system provided in accordance with one embodiment of the present application;

FIG. 2 is a flow chart of a control method of a machine vision screw tightening system provided according to one embodiment of the present application;

FIG. 3 is a flow chart of a method for locating a threaded hole in a workpiece to be screwed according to one embodiment of the present application;

fig. 4 is a flow chart illustrating a control method of a machine vision screwing system according to another embodiment of the present application.

Detailed Description

It should be noted that, without conflict, the embodiments of the present application and features of the embodiments may be combined with each other. The application will be described in detail below with reference to the drawings in connection with embodiments.

In order that those skilled in the art will better understand the present application, a technical solution in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, shall fall within the scope of the present application.

It should be noted that the terms "first," "second," and the like in the description and the claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate in order to describe the embodiments of the application herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

According to an embodiment of the present application, a control method of a machine vision screw tightening system is provided.

Fig. 1 is a schematic structural view of a machine vision screw tightening system according to an embodiment of the present application, referring to fig. 1, the machine vision screw tightening system 1 includes an actuator 11, a first photographing mechanism 12, a second photographing mechanism 13, a conveying mechanism 14, and a screw tightening mechanism 15, the first photographing mechanism 12 is fixedly installed at a position having a constant vertical distance from the conveying mechanism 14, the second photographing mechanism 13 and the screw tightening mechanism 15 are detachably installed on the actuator 11, the first photographing mechanism 12 and the second photographing mechanism 13 are configured to collect image information, and the conveying mechanism 14 is configured to convey a screw workpiece to be screwed. The actuator 11 may include an actuator body and an actuator end effector. The actuator 11 may include an actuator body 111 and an actuator end effector 112.

In some embodiments, with continued reference to fig. 1, the conveyor 14 includes a conveyor belt, and the machine vision screw tightening system may further include at least one photoelectric sensor 18 disposed on the conveyor belt, and when the number of photoelectric sensors 18 is two, the photoelectric sensors may be symmetrically disposed on two sides of the conveyor belt, for detecting whether a workpiece to be screwed is present on the conveyor 14. The machine vision screw-down system may further comprise: a visual motion controller 16 and a servo driver 17.

Specifically, the conveying mechanism 14 is configured to convey a screw workpiece to be screwed, such as a conveyor belt, and is smoothly movable in the horizontal direction after being opened. The conveyor 14 is also provided with a photoelectric sensor 18, which may be one or more in number. The actuator 11, such as a co-operating robot, a six-axis robot, etc., is used for dynamically tracking the workpiece to be screwed, the tracking direction being the same as the movement direction of the conveyor 14. The first shooting mechanism 12 is a 3D vision system, is fixedly installed at a position with a constant vertical distance from the conveying mechanism 14, has a large visual angle range, and is used for collecting image information of a workpiece to be screwed. The second shooting mechanism 13 is a binocular vision system, is detachably mounted on the executing mechanism 11, has a smaller visual angle range, can move along with a workpiece to be screwed, and is used for collecting image information of the workpiece to be screwed in real time. The screw screwing mechanism 15 is detachably mounted on the actuator 11 and can vertically move up and down, and is controlled by the actuator 11 to perform screw screwing operation when the screw screwing mechanism 15 is synchronized with the threaded hole of the workpiece to be screwed. Wherein there is a height difference between the second photographing mechanism 13 and the screw tightening mechanism 15, the screw tightening mechanism 15 is installed closer to the conveying mechanism 14.

The actuating mechanism 11 is in a servo state at the starting position, waits for the arrival of a workpiece to be screwed, enters a coarse tracking stage after the photoelectric sensor 18 detects the workpiece to be screwed, outputs a trigger signal to the visual motion controller 16 at the photoelectric sensor 18, and the visual motion controller 16 generates a periodic trigger signal to trigger the first shooting mechanism 12 to shoot and acquire image information of the workpiece to be screwed, and counts the number of the workpiece to be screwed. The vision motion controller 16 acquires an image stream from the first photographing mechanism 12 and performs operations such as processing on the acquired image to determine pose information of the screw hole of the screw workpiece to be screwed. The visual motion controller 16 obtains pose information of the executing mechanism 11 and the conveying mechanism 14 from the servo driver 17 through an EtherCAT (EtherNet Control Automation Technology, a real-time Ethernet technology) bus, determines the type of a threaded hole on the workpiece to be screwed and the condition of the threaded hole on the surface of the workpiece to be screwed based on the pose information of the threaded hole of the workpiece to be screwed, generates motion control information according to the pose information of the executing mechanism 11, the pose information of the conveying mechanism 14 and the pose information of the threaded hole of the workpiece to be screwed, drives the end effector 112 of the executing mechanism to drive the screw screwing mechanism 15 to perform rough tracking on the workpiece to be screwed through the servo driver 17, enables the executing mechanism 11 to reach the moving speed of the workpiece to be screwed in a short time, enables the center of the screw screwing mechanism 15 to be primarily aligned with the center of the threaded hole of the workpiece to be screwed, and ends the rough tracking stage to enter the fine tracking stage.

In the fine tracking stage, the visual motion controller 16 triggers the second shooting mechanism 13 to shoot an image of the workpiece to be screwed, the visual motion controller 16 acquires an image stream from the second shooting mechanism 13, processes the acquired image and the like, continuously determines pose information of a threaded hole of the workpiece to be screwed, generates motion control information according to the pose information of the threaded hole of the workpiece to be screwed, pose information of the actuating mechanism 11 and pose information of the conveying mechanism 14, drives the end effector 112 of the actuating mechanism through the servo driver 17 to drive the screwing mechanism 15 to carry out fine tracking on the workpiece to be screwed, and enables the center of the screw mechanism 15 to be controlled to be aligned with the center of the threaded hole of the workpiece to be screwed, namely, the screw mechanism 15 is synchronous with the threaded hole of the workpiece to be screwed, and carries out screwing operation on the workpiece to be screwed.

Fig. 2 is a flow chart of a control method of a machine vision screwing system according to an embodiment of the present application. Referring to fig. 2, the control method of the machine vision screwing system may include:

s110, acquiring image information of a workpiece to be screwed through a first shooting mechanism and a second shooting mechanism.

Specifically, the first shooting mechanism is a 3D vision system, specifically a 3D camera, and can accurately acquire 3D image information of a workpiece to be screwed, and can be used for identifying the type of a threaded hole on the workpiece to be screwed, such as a round threaded hole, a polygonal threaded hole and the like, and can also be used for identifying the condition of the threaded hole on the surface of the workpiece to be screwed, such as inclination or height difference and the like. The second shooting mechanism is a binocular vision system, and is specifically composed of two 2D cameras, the image precision is inferior to that of a 3D camera, but the 2D image information of the screw workpiece to be screwed can be acquired in real time in the moving process of the screw workpiece to be screwed.

S120, determining first pose information of the threaded hole of the workpiece to be screwed according to the image information acquired by the first shooting mechanism, and determining second pose information of the threaded hole of the workpiece to be screwed according to the image acquired by the second shooting mechanism.

Specifically, image processing is performed on the image information of the workpiece to be screwed obtained by the first photographing mechanism, first pose information, such as 6D pose information, of the threaded hole of the workpiece to be screwed can be obtained according to the image processing result, the type of the threaded hole on the workpiece to be screwed, such as a circular threaded hole, a polygonal threaded hole and the like, can be preliminarily determined based on the first pose information, and the condition that the threaded hole is on the surface of the workpiece to be screwed, such as inclination or height difference and the like, can be determined. And carrying out image processing on the image information of the to-be-screwed screw workpiece acquired by the second shooting mechanism, acquiring second pose information, such as 6D pose information, of the threaded hole of the to-be-screwed screw workpiece according to an image processing result, and further determining whether the screw screwing mechanism is synchronous with the threaded hole of the to-be-screwed screw workpiece or not based on the second pose information on the basis of the first pose information.

S130, controlling the executing mechanism according to the pose information of the executing mechanism, the pose information of the conveying mechanism and the first pose information of the threaded hole of the workpiece to be screwed, so that the speeds of the executing mechanism and the workpiece to be screwed are synchronous, and the pose error of the threaded hole of the screw screwing mechanism and the threaded hole of the workpiece to be screwed is smaller than a first preset error threshold.

Specifically, when the threaded hole is not coplanar with the workpiece to be screwed, for example, the screw screwing mechanism is required to be controlled to roughly trace the threaded hole of the workpiece to be screwed, so that the speeds of the actuating mechanism and the workpiece to be screwed are synchronous, the pose error of the screw screwing mechanism and the threaded hole of the workpiece to be screwed is smaller than a first preset error threshold value, even if the actuating mechanism reaches the speed of movement of the workpiece to be screwed in a short time, and the center of the screw screwing mechanism is primarily aligned with the center of the threaded hole of the workpiece to be screwed, so that the tracing speed can be increased.

For example, a corresponding encoder may be disposed on the conveyor belt side, and the pose information of the conveyor belt may be obtained by reading the encoder, and similarly, a corresponding encoder may be disposed on the end effector of the execution structure, and the pose information of the execution structure may be obtained by reading the encoder. The pose information (such as 6D pose information) of the executing mechanism and the pose information (such as 6D pose information) of the conveying mechanism can be obtained in real time through the EtherCAT bus, the executing mechanism is controlled according to the pose information of the executing mechanism, the pose information of the conveying mechanism and the first pose information, for example, the executing mechanism is controlled to adjust the pose in a 3D space until the speeds of the executing mechanism and the workpiece to be screwed are synchronous, the pose error of the screw screwing mechanism and the threaded hole of the workpiece to be screwed is smaller than a first preset error threshold value, even if the executing mechanism reaches the movement speed of the workpiece to be screwed in a short time, and the center of the screw screwing mechanism is primarily aligned with the center of the threaded hole of the workpiece to be screwed. Thus, the actuator completes coarse tracking of the threaded hole of the workpiece to be screwed.

And S140, controlling the executing mechanism according to the first pose information, the second pose information, the pose information of the executing mechanism and the pose information of the conveying mechanism so that the pose error of the screw screwing mechanism and the threaded hole of the workpiece to be screwed is smaller than a second preset error threshold.

Specifically, when the executing mechanism completes coarse tracking of the threaded hole of the workpiece to be screwed, the center of the screw screwing mechanism is only initially aligned with the center of the threaded hole of the workpiece to be screwed, the pose errors of the screw screwing mechanism and the threaded hole of the workpiece to be screwed are still larger, the precise tracking stage is required to be entered, the pose errors of the screw screwing mechanism and the threaded hole of the workpiece to be screwed are further reduced until the pose errors of the screw screwing mechanism and the threaded hole of the workpiece to be screwed are smaller than a second preset error threshold value, the center of the screw screwing mechanism and the center of the threaded hole of the workpiece to be screwed are basically aligned, namely the screw screwing mechanism and the threaded hole of the workpiece to be screwed are synchronous, and tracking precision is improved.

Illustratively, the actuator is controlled according to the first pose information, the second pose information, the pose information of the actuator, and the pose information of the transmission mechanism, for example, the actuator is controlled to adjust the pose in the 3D space, so as to reduce the pose error of the screw screwing mechanism and the threaded hole of the workpiece to be screwed until the pose error is smaller than a second preset error threshold value, even if the center of the screw screwing mechanism is substantially aligned with the center of the threaded hole of the workpiece to be screwed. Thus, the execution mechanism completes fine tracking of the threaded hole of the workpiece to be screwed.

Therefore, screw screwing can be completed on the premise that the conveying mechanism does not stop, the efficiency of visual guiding screw screwing is optimized, and the automation level is improved.

Specifically, the type of the screw hole of the screw workpiece to be screwed, such as a circular screw hole, a polygonal screw hole, etc., may be determined based on the first pose information of the screw hole of the screw workpiece to be screwed, and the surface condition of the screw hole of the screw workpiece to be screwed and the screw workpiece to be screwed, such as inclination or height difference, etc., may be determined. Therefore, the screw screwing mechanism needs to be controlled to roughly track the threaded hole of the workpiece to be screwed according to the first pose information of the threaded hole of the workpiece to be screwed, the pose information of the executing mechanism and the pose information of the conveying mechanism until the center of the screw screwing mechanism is primarily aligned with the center of the threaded hole of the workpiece to be screwed.

Illustratively, in the rough tracking stage, pose information of the executing mechanism, pose information of the conveying mechanism and first pose information of a threaded hole of a workpiece to be screwed are fused, and the threaded hole of the workpiece to be screwed and first pose error information of the screw screwing mechanism are determined. Based on the first pose error information, generating a first motion control instruction by combining pose information of an executing mechanism and pose information of a transmission mechanism, controlling the executing mechanism to adjust the pose in a 3D space based on the type of a threaded hole on a workpiece to be screwed and the specific condition of the threaded hole on the surface of the workpiece to be screwed, until the speeds of the executing mechanism and the workpiece to be screwed are synchronous, and the pose error of the threaded hole of the workpiece to be screwed and the screw screwing mechanism is smaller than a first preset error threshold value, even if the executing mechanism reaches the moving speed of the workpiece to be screwed in a short time, the center of the screw screwing mechanism is primarily aligned with the center of the threaded hole of the workpiece to be screwed, and completing coarse tracking of the threaded hole of the workpiece to be screwed.

Therefore, the type of the threaded hole on the workpiece to be screwed and the condition that the threaded hole is on the surface of the workpiece to be screwed are determined in the rough tracking stage, so that the actuating mechanism can reach the movement speed of the workpiece to be screwed in a short time, the center of the screw screwing mechanism is primarily aligned with the center of the threaded hole of the workpiece to be screwed, and the tracking speed is increased.

Specifically, when the rough tracking of the workpiece to be screwed is completed, the center of the screw screwing mechanism is primarily aligned with the center of the threaded hole of the workpiece to be screwed, and then when the fine tracking stage is entered, the screw screwing mechanism is controlled to precisely track the threaded hole of the workpiece to be screwed on the basis of the first pose information of the threaded hole of the workpiece to be screwed, combined with the second pose information, the pose information of the transmission mechanism and the pose information of the execution mechanism until the center of the screw screwing mechanism is substantially aligned with the center of the threaded hole of the workpiece to be screwed, namely the screw screwing mechanism and the threaded hole of the workpiece to be screwed are synchronous.

In the fine tracking stage, the first pose information and the second pose information are fused to obtain fused pose information, and the threaded hole of the workpiece to be screwed and the second pose error information of the screw screwing mechanism are determined according to the fused pose information, the pose information of the actuating mechanism and the pose information of the conveying mechanism. Based on the second pose error information, generating a second motion control instruction by combining pose information of the executing mechanism and pose information of the conveying mechanism, and controlling the executing mechanism to adjust the pose in a 3D space until the pose error of the threaded hole of the workpiece to be screwed and the screw screwing mechanism is smaller than a second preset error threshold value, so that the center of the screw screwing mechanism is basically aligned with the center of the threaded hole of the workpiece to be screwed, namely the screw screwing mechanism and the threaded hole of the workpiece to be screwed are synchronous, and finishing precise tracking of the threaded hole of the workpiece to be screwed.

Thus, the execution mechanism is controlled to further track the threaded hole of the workpiece to be screwed in the fine tracking stage, so that the center of the screw screwing mechanism is basically aligned with the center of the threaded hole of the workpiece to be screwed, and the tracking precision is improved.

Specifically, the first shooting mechanism is fixedly arranged at a position with a constant vertical distance from the conveying mechanism, the visual angle range is larger, the obtained image information of the to-be-screwed workpiece is more accurate, and the first pose information of the threaded hole of the to-be-screwed workpiece can be determined according to the image information obtained by the first shooting mechanism. The second shooting mechanism is detachably arranged on the executing mechanism and can move along with the workpiece to be screwed, the visual angle range is smaller, the image information of the workpiece to be screwed can be fed back in real time, and the second pose information of the threaded hole of the workpiece to be screwed can be determined according to the image acquired by the second shooting mechanism. Then, along with the movement of the executing mechanism, the shooting range of the first shooting mechanism is gradually reduced under the influence of the shooting visual angle, so that the acquired image information is gradually reduced, and the second shooting mechanism can move along with the workpiece to be screwed, so that the acquired image information is gradually relatively stable. Therefore, in the fine tracking stage, when the first pose information and the second pose information are fused, a weight coefficient is required to be set, for example, the first pose information is correspondingly provided with the first weight coefficient, the second pose information is correspondingly provided with the second weight coefficient, and the fused pose information is determined according to the first pose information, the first weight coefficient, the second pose information and the second weight coefficient.

Specifically, as the actuator moves, the image information acquired by the first photographing mechanism gradually decreases, and the image information acquired by the second photographing mechanism gradually becomes relatively stable. Therefore, the first pose information is correspondingly provided with a first weight coefficient which gradually decreases along with the movement of the executing mechanism, and the second pose information is correspondingly provided with a second weight coefficient which gradually increases along with the movement of the executing mechanism, wherein the sum of the first weight coefficient and the second weight coefficient is 1.

Specifically, in the fine tracking stage, when the first pose information and the second pose information are fused, fused pose information is obtained, including: and fusing the first pose information and the second pose information by adopting a weighted average algorithm to obtain fused pose information.

Illustratively, the fused pose information is determined from the first pose information, the first weight coefficient, the second pose information, and the second weight coefficient by the following formula:

S_m＝Wc*S_c+Ws*S_s

wherein S _m represents the fusion pose information, wc represents the first weight coefficient, S _c represents the first pose information of the coarse tracking stage, ws represents the second weight coefficient, and S _s represents the second pose information of the fine tracking stage.

Specifically, besides the fusion of the first pose information and the second pose information by adopting a weighted average algorithm, the fusion of the first pose information and the second pose information can be realized by adopting a Kalman filtering algorithm, recursive calculation can be performed, interference can be effectively restrained and compensated, the fusion value can be optimized by the prior information of the state, and the fusion precision and instantaneity are improved.

Illustratively, the fused pose information is determined from the first pose information and the second pose information by the following formula:

Wherein x represents the state of the system, An estimated value indicating the fusion pose information after error correction at time k+1, and x (k+1) indicating the second pose information,/>Represents the estimated value of x (k+1) at the kth time, W (k+1) represents the Kalman gain,/>An observed estimated error value representing the first pose information at time k+1. Therefore, based on the first pose information and the second pose information, iterative calculation is continuously performed, the fusion pose information is corrected, and fusion accuracy and instantaneity are improved.

err_init＝f(L₀,P_c,l_2c,l_1c)

Specifically, pose deviation L ₀ between the second shooting mechanism and the screw screwing mechanism, fusion pose information P _c, pose information L _2c of the executing mechanism and pose information L _1c of the conveying mechanism are input into a pose error information calculation model to be calculated, and second pose error information err _init can be obtained, wherein the second pose error information err _init is a pose vector of a 3D space, and the vector direction is the direction of pointing to a threaded hole of a workpiece to be screwed by the screw screwing mechanism.

In some embodiments, the first pose error information is determined using the following formula:

err_init'＝f(L₁,P₁,l_2c,l_1c)

Wherein err _init' represents first pose error information, L ₁ represents pose deviation between the first photographing mechanism and the screw screwing mechanism, P ₁ represents first pose information, L _2c represents pose information of the executing mechanism, and L _1c represents pose information of the transmitting mechanism.

Specifically, pose deviation L ₁ between the first shooting mechanism and the screw screwing mechanism, first pose information P ₁, pose information L _2c of the executing mechanism and pose information L _1c of the conveying mechanism are input into a pose error information calculation model to be calculated, and first pose error information err _init 'can be obtained, wherein the first pose error information err _init' is a pose vector in a 3D space, and the vector direction is the direction of pointing to a threaded hole of a workpiece to be screwed by the screw screwing mechanism.

In some embodiments, further comprising: extracting features of the image information to obtain coordinates of feature points; and determining the first pose information and the second pose information by adopting a PNP (PESPECTIVE-N-Point) clustering algorithm or a binocular triangulation positioning algorithm based on the feature points.

Specifically, the image information of the to-be-screwed screw workpiece, which is acquired by the first shooting mechanism and the second shooting mechanism, is processed, so that the first pose information and the second pose information of the threaded hole of the to-be-screwed screw workpiece can be acquired.

Referring to fig. 3, fig. 3 is a schematic flow chart of a method for positioning a threaded hole of a workpiece to be screwed according to an embodiment of the present application. The method for positioning the threaded hole of the screw workpiece to be screwed comprises the following steps:

s201, extracting feature points.

First, the first frame image of the workpiece to be screwed is obtained through shooting by the first shooting mechanism and the second shooting mechanism, for example, the 2D image and the 3D image are standardized and normalized, interference factors are discharged, and characteristic information is highlighted. Then, feature extraction is performed, for example, feature point coordinates of the screw hole image of the screw workpiece to be screwed are extracted using SIFT (SCALE INVARIANT Feature Transform, scale-invariant feature transform), a corner extraction algorithm, or the like.

S202, clustering characteristic points.

A clustering algorithm is used to obtain target feature points, such as circle feature points or line feature points, of the threaded holes of the workpieces to be screwed, and the number of the target feature points is generally required to be greater than or equal to 4.

S203, template matching.

For the rest frame images except the first frame image, only a template matching mode is needed to match and track the following frame images.

S204, pose positioning.

And determining first pose information and second pose information of the threaded hole of the workpiece to be screwed by adopting a PNP clustering algorithm or a binocular triangulation positioning algorithm based on the target feature points.

Therefore, the subsequent frame images are matched and tracked based on the template matching method, preprocessing of the subsequent frame images is not needed, calculation efficiency is improved, and tracking speed is further improved.

In some embodiments, the control method further includes: acquiring second pose information according to a first preset servo frequency, and acquiring pose information of an executing mechanism and pose information of a conveying mechanism according to the second preset servo frequency; wherein the first preset servo frequency is smaller than the second preset servo frequency.

Specifically, the first preset servo frequency is the frequency of the second shooting mechanism shooting images, the sampling frequency is low, the acquired images are accurate, the second preset servo frequency is the data acquired by the encoder, the sampling frequency is fast, the measured data are less, and after the two data are fused, the second pose error information with higher frequency and more accuracy can be obtained. Typically, the first preset servo frequency is 30Hz and the second preset servo frequency is 1kHz.

Specifically, in the fine tracking stage, the second shooting mechanism shoots and acquires the moving image information of the to-be-screwed screw workpiece in the tracking process, processes the image information of the to-be-screwed screw workpiece, acquires the second pose information of the threaded hole of the to-be-screwed screw workpiece at a first preset servo frequency, fuses the first pose information and the second pose information of the threaded hole of the to-be-screwed screw workpiece to obtain fused pose information, and simultaneously acquires the pose information of the executing mechanism and the pose information of the conveying mechanism in real time at the second preset servo frequency through the EtherCAT bus. And fusing the fused pose information, the pose information of the executing mechanism and the pose information of the transmitting mechanism at a second preset servo frequency to obtain the threaded hole of the workpiece to be screwed and the second pose error information of the screwing mechanism. At the moment, based on the second pose error information, the pose information of the executing mechanism and the pose information of the conveying mechanism, a second motion control instruction is generated at a second preset servo frequency, the executing mechanism adjusts the pose in a 3D space based on the second motion control instruction, and the threaded hole of the workpiece to be screwed is accurately tracked until the pose error of the threaded hole of the workpiece to be screwed and the pose error of the screw screwing mechanism are smaller than a second preset error threshold.

It should be noted that, the first preset servo frequency is smaller than the second preset servo frequency, so that the pose information of the executing mechanism and the pose information of the transmission mechanism obtained by the second preset servo frequency are shorter than the second pose information of the threaded hole of the workpiece to be screwed obtained by the first preset servo frequency. The second pose information of the threaded hole of the workpiece to be screwed at the moment t can be obtained at a first preset servo frequency, and the second pose information of the threaded hole of the workpiece to be screwed at the moment t is fused with the first pose information to obtain fused pose information. And simultaneously, acquiring pose information of the executing mechanism at the moment t and pose information of the conveying mechanism at the moment k and pose information of the executing mechanism at the moment k by using a second preset servo frequency, wherein the moment k is between the moment t and the moment t+1. Therefore, the fusion of the pose information, the pose information of the executing mechanism and the pose information of the conveying mechanism is carried out at a second preset servo frequency, so that the threaded hole of the workpiece to be screwed and the second pose error information of the screw screwing mechanism at the moment k can be obtained, and the threaded hole of the workpiece to be screwed is aligned with the pose information acquisition time stamp of the conveying mechanism and the executing mechanism. In addition, when external disturbance occurs at the current moment, the disturbance is compensated based on the fusion pose acquired by the image at the next moment, so that the robustness of the system is improved.

According to the embodiment, the dynamic screw screwing process is divided into three stages, namely a dynamic tracking stage, a synchronous screw screwing tracking stage and a reset zero-returning stage. Wherein the dynamic tracking phase comprises a coarse tracking phase and a fine tracking phase.

Specifically, in the rough tracking stage, the executing mechanism is controlled to carry out gesture adjustment according to the gesture information of the executing mechanism, the gesture information of the conveying mechanism and the first gesture information of the threaded hole of the workpiece to be screwed, until the executing mechanism and the speed of the workpiece to be screwed are synchronous, the gesture error of the threaded hole of the workpiece to be screwed and the screw screwing mechanism is smaller than a first preset error threshold, the executing mechanism can reach the movement speed of the workpiece to be screwed in a short time, the center of the screw screwing mechanism is primarily aligned with the center of the threaded hole of the workpiece to be screwed, and the tracking speed is accelerated.

And when the coarse tracking stage is finished, entering a fine tracking stage, and controlling the executing mechanism to adjust the pose according to the pose information of the executing mechanism, the pose information of the conveying mechanism and the fusion pose information until the pose error of the screw screwing mechanism and the threaded hole of the workpiece to be screwed is smaller than a second preset error threshold value, so that the center of the screw screwing mechanism is basically aligned with the center of the threaded hole of the workpiece to be screwed, namely the screw screwing mechanism and the threaded hole of the workpiece to be screwed are synchronous, and the tracking precision is improved.

And at the end of the fine tracking stage, synchronizing the screw screwing mechanism with the threaded hole of the workpiece to be screwed, entering the synchronous tracking screw screwing stage, executing screw screwing operation by the screw screwing mechanism, then entering the reset zero-returning stage, returning the executing mechanism to the initial position, waiting for the next workpiece to be screwed, and screwing the screw by the workpiece to be screwed on the conveying mechanism in the circulation.

When the number of the threaded holes on the workpiece to be screwed is a plurality of, in the coarse tracking stage, the executing mechanism sequentially performs coarse tracking on the threaded holes on the workpiece to be screwed, and enters the fine tracking stage when the speeds of the executing mechanism and the workpiece to be screwed are synchronous and the pose errors of the screw screwing mechanism and the threaded holes on the workpiece to be screwed are smaller than a first preset error threshold. In the fine tracking stage, the executing mechanism sequentially carries out fine tracking on threaded holes on the workpiece to be screwed, when the pose errors of the screw screwing mechanism and the threaded holes on the workpiece to be screwed are smaller than a second preset error threshold value, the synchronous tracking screw screwing stage is carried out, the screw screwing mechanism sequentially finishes all the threaded holes on the workpiece to be screwed in a specified sequence, after the screw screwing operation is finished, the resetting and zeroing stage is carried out, and the next workpiece to be screwed is waited for.

In summary, the first shooting mechanism and the second shooting mechanism are arranged to respectively acquire the first pose information and the second pose information of the workpiece to be screwed, the type of the threaded hole on the workpiece to be screwed and the condition that the threaded hole is on the surface of the workpiece to be screwed are acquired based on the first pose information, the executing mechanism is controlled to perform rough tracking on the workpiece to be screwed, so that the screw screwing mechanism is primarily aligned with the threaded hole of the workpiece to be screwed, and the tracking speed is increased; and fusing the first pose information and the second pose information, controlling the execution mechanism to precisely track the workpiece to be screwed based on the fused information, enabling the screw screwing mechanism to be basically aligned with the threaded hole of the workpiece to be screwed, and improving tracking precision. Therefore, dynamic tracking labeling is completed on the premise that the conveying mechanism is not stopped, and the efficiency and the automation level of visual guiding labeling are improved. In addition, when external disturbance occurs in the tracking stage, compensation of the external disturbance can be achieved, and robustness of the system is improved.

In some embodiments, with continued reference to fig. 1, there is also provided a machine vision screw tightening system 1 including an actuator 11, a first photographing mechanism 12, a second photographing mechanism 13, a transfer mechanism 14, a screw tightening mechanism 15, and a controller 16. The first shooting mechanism 12 is fixedly arranged at a position with a constant vertical distance from the conveying mechanism 14, the second shooting mechanism 13 and the screw screwing mechanism 15 are detachably arranged on the executing mechanism 11, the conveying mechanism 14 is configured to convey a workpiece to be screwed, and the first shooting mechanism 12 and the second shooting mechanism 13 are used for acquiring image information of the workpiece to be screwed; and a controller 16, configured to determine first pose information of the threaded hole of the workpiece to be screwed according to the image information acquired by the first photographing mechanism 12, determine second pose information of the threaded hole of the workpiece to be screwed according to the image acquired by the second photographing mechanism 13, and control the actuator 11 according to the pose information of the actuator 11, the pose information of the transmission mechanism 14, and the first pose information of the threaded hole of the workpiece to be screwed, so that the speeds of the actuator 11 and the workpiece to be screwed are synchronized and the pose error of the screw screwing mechanism 15 and the threaded hole of the workpiece to be screwed is smaller than a first preset error threshold, and control the actuator 11 according to the first pose information, the second pose information, the pose information of the actuator, and the pose information of the transmission mechanism, so that the pose error of the screw screwing mechanism 15 and the threaded hole of the workpiece to be screwed is smaller than a second preset error threshold.

Specifically, the conveying mechanism 14, such as a conveyor belt, is configured to convey the screw workpiece to be screwed, and is smoothly movable in the horizontal direction after being opened. The conveying mechanism 14 is also provided with photoelectric sensors 18, the number of which can be one or more, for detecting whether a workpiece to be screwed exists on the conveying mechanism 14. The actuator 11, such as a co-operating robot, a six-axis robot, etc., comprises an actuator body and an actuator end effector for dynamically tracking the screw workpiece to be screwed in the same direction as the movement of the conveyor 14. The first photographing mechanism 12 is a 3D vision system, is fixedly installed at a position with a constant vertical distance from the conveying mechanism 14, for example, is fixedly installed on the actuator body, has a large viewing angle range, and is used for collecting 3D image information of a workpiece to be screwed and uploading the 3D image information to the controller 16. The second shooting mechanism 13 is a binocular vision system, is detachably mounted on the actuating mechanism 11, for example, mounted on an end effector of the actuating mechanism, has a smaller visual angle range, can move along with a workpiece to be screwed, is used for acquiring 2D image information of the workpiece to be screwed, and is uploaded to the controller 16. The screwing mechanism 15 is detachably mounted on the actuator 11, for example on an end effector of the actuator, and can move vertically up and down, controlled by the actuator 11, for performing screwing operations when the screwing mechanism 15 is synchronized with a workpiece to be screwed. Wherein there is a height difference between the second photographing mechanism 13 and the screw tightening mechanism 15, the screw tightening mechanism 15 is installed closer to the conveying mechanism 14.

Specifically, the controller 16 integrates a vision processing functional block and a motion control functional block, and the two functional blocks perform real-time data interaction through a shared memory, wherein the vision processing functional block comprises a pose fusion module, and the motion control functional block comprises a dynamic tracking module. The controller 16 may also obtain the pose information of the actuator 11 and the pose information of the transfer mechanism 14 fed back in real time from the servo driver 17 via the EtherCAT bus at the second servo frequency.

The vision processing functional block is used for acquiring corresponding first pose information and second pose information of a threaded hole of a screw workpiece to be screwed according to image information of the screw workpiece to be screwed uploaded by the first shooting mechanism 12 and the second shooting mechanism 13, wherein the pose fusion module is used for fusing the pose information of the executing mechanism 11, the pose information of the conveying mechanism 14 and the first pose information of the threaded hole of the screw workpiece to be screwed in a rough tracking stage, and outputting first pose error information of the threaded hole of the screw workpiece to be screwed and the screw screwing mechanism 15; the pose fusion module is used for fusing pose information of the executing mechanism 11, pose information of the conveying mechanism 14 and first pose information and second pose information of a threaded hole of a workpiece to be screwed in a fine tracking stage, and outputting error information of the threaded hole of the workpiece to be screwed in the fine tracking stage and second pose information of the screw screwing mechanism 15.

The dynamic tracking module in the motion control functional block is used for controlling the executing mechanism 11 to perform dynamic tracking, including coarse tracking and fine tracking. In the rough tracking stage, the dynamic tracking module outputs a first motion control instruction to the servo driver 17 according to the pose information of the actuating mechanism 11, the pose information of the conveying mechanism 14 and the first pose error information of the screw hole of the screw workpiece to be screwed and the screw screwing mechanism 15, the actuating mechanism 11 performs rough tracking based on the first motion control instruction issued by the servo driver 17 until the actuating mechanism 11 and the screw workpiece to be screwed are synchronous in speed and the pose error of the screw screwing mechanism 15 and the screw hole of the screw workpiece to be screwed is smaller than a first preset error threshold value, even if the actuating mechanism 11 reaches the movement speed of the screw workpiece to be screwed in a short time and the center of the screw screwing mechanism 15 is primarily aligned with the center of the screw hole of the screw workpiece to be screwed, and the rough tracking is ended to enter the fine tracking stage. In the fine tracking stage, the dynamic tracking module can adopt a PID (Proportional-integral-derivative) control method to realize high-precision dynamic tracking. Inputting a second pose error of the threaded hole of the workpiece to be screwed and the screw screwing mechanism 15, a proportional coefficient K _p, a differential coefficient K _d and an integral coefficient K _i into a dynamic tracking module, and calculating a control amount of the current control period by the dynamic tracking module based on each coefficient and the second pose error of the threaded hole of the workpiece to be screwed and the screw screwing mechanism 15 in the current control period through the following formula:

Where u (t) is the control amount of the current control cycle, err _init is the second pose error curve of the threaded hole of the workpiece to be screwed and the screw screwing mechanism 15. The dynamic tracking module outputs a second motion control instruction to the servo driver 17 according to the pose information of the executing mechanism 11, the pose information of the transmitting mechanism 14 and the control quantity of the current control period, and the executing mechanism 11 dynamically tracks based on the second motion control instruction issued by the servo driver 17 until the pose error of the screw screwing mechanism 15 and the threaded hole of the workpiece to be screwed is smaller than a second preset error threshold, namely the center of the screw screwing mechanism 15 is basically aligned with the center of the threaded hole of the workpiece to be screwed, and the fine tracking stage is ended.

As a specific example, referring to fig. 4, a control method of a machine vision screw tightening system may include:

S301, starting.

S302, the actuator is in a servo state at the initial position.

The actuating mechanism is in a servo state at the initial position and waits for the arrival of a workpiece to be screwed.

S303, detecting by a photoelectric sensor.

S304, judging whether a workpiece to be screwed is detected, if so, executing S305, otherwise, executing S302.

After the photoelectric sensor detects the workpieces to be screwed, a trigger signal is output to the controller, the controller generates a periodic hardware trigger signal and transmits the periodic hardware trigger signal to the first shooting mechanism, the first shooting mechanism is triggered to shoot and acquire 3D image information, the image information is uploaded to the controller, and meanwhile the number of the workpieces to be screwed is counted.

S305, tracking is started.

S306, coarse tracking.

The controller obtains and processes the 3D image information of the workpiece to be screwed, and obtains first pose information of the threaded hole of the workpiece to be screwed by moving the workpiece to be screwed at a first preset servo frequency. Meanwhile, the controller acquires the pose information of the executing mechanism and the pose information of the conveying mechanism which are fed back in real time from the servo driver through the EtherCAT bus at a second servo frequency. And transmitting the pose information of the threaded hole of the workpiece to be screwed obtained through calculation at the first preset servo frequency, the pose information of the actuating mechanism obtained at the second servo frequency and the pose information of the transmission mechanism to a pose fusion module for fusion, and outputting the first pose error information of the threaded hole of the workpiece to be screwed and the screw screwing mechanism. The dynamic tracking module outputs a first motion control instruction to the servo driver according to the first pose error information of the screw hole of the workpiece to be screwed and the screw screwing mechanism, the pose information of the executing mechanism and the pose information of the conveying mechanism. The actuating mechanism is based on a first motion control instruction issued by the servo driver so as to roughly track the workpiece to be screwed.

S307, judging whether the rough tracking motion parameters are synchronous, if so, executing S308, otherwise, continuing to execute S306.

When the speed of the executing mechanism is synchronous with that of the workpiece to be screwed, and the pose error of the screw screwing mechanism and the threaded hole of the workpiece to be screwed is smaller than a first preset error threshold value, the center of the screw screwing mechanism is primarily aligned with the center of the threaded hole of the workpiece to be screwed, and the coarse tracking is finished to enter the fine tracking stage.

S308, fine tracking.

The controller generates a periodic hardware trigger signal and transmits the periodic hardware trigger signal to the second shooting mechanism, the second shooting mechanism is triggered to shoot and acquire 2D image information, and the image information is uploaded to the controller. After the controller acquires the image information of the workpiece to be screwed, the controller acquires second pose information of the threaded hole of the workpiece to be screwed by moving the workpiece to be screwed at a first preset servo frequency. Meanwhile, the controller acquires pose information of the executing mechanism and pose information of the conveying mechanism, which are fed back in real time, from the servo driver through the EtherCAT bus at a second servo frequency, and transmits the first pose information and the second pose information of the threaded hole of the workpiece to be screwed, which are acquired at the first preset servo frequency, the pose information of the executing mechanism and the pose information of the conveying mechanism, which are acquired at the second servo frequency, to the pose fusion module for fusion, and outputs the error information of the threaded hole of the workpiece to be screwed and the second pose of the screw screwing mechanism. The dynamic tracking module outputs a second motion control instruction to the servo driver according to the second pose error information of the threaded hole of the workpiece to be screwed and the screw screwing mechanism, the pose information of the executing mechanism and the pose information of the conveying mechanism, and the executing mechanism carries out fine tracking on the threaded hole of the workpiece to be screwed based on the second motion control instruction issued by the servo driver.

S309, judging whether the fine tracking motion parameters are synchronous, if so, executing S310, otherwise, continuing to execute S308.

And when the pose error of the screw screwing mechanism and the threaded hole of the workpiece to be screwed is smaller than a second preset error threshold value, synchronizing the screw screwing mechanism and the threaded hole of the workpiece to be screwed, enabling the center of the screw screwing mechanism to be basically aligned with the center of the threaded hole of the workpiece to be screwed, and finishing the fine tracking to enter a synchronous tracking screw screwing stage.

S310, synchronously tracking the workpieces to be screwed.

S311, executing a screw screwing operation.

S312, ending.

It should be noted that the above explanation of the embodiments and advantageous effects of the control method of the machine vision screw driving system is also applicable to the machine vision screw driving system of the embodiment of the present invention, and is not developed in detail here to avoid redundancy.

In some embodiments, a computer readable storage medium having stored thereon a control program for a machine vision screw tightening system, which when executed by a processor, implements the aforementioned control method for a machine vision screw tightening system.

It should be noted that the above explanation of the embodiments and advantageous effects of the control method of the machine vision screwing system is also applicable to the computer readable storage medium of the embodiment of the present invention, and is not developed in detail here for avoiding redundancy.

In some embodiments, there is also provided a screw-driving apparatus comprising the machine vision screw-driving system described above.

It should be noted that the above explanation of the embodiments and advantageous effects of the control method of the machine vision screwing system is also applicable to the screwing apparatus of the embodiment of the present invention, and is not developed in detail here to avoid redundancy.

It should be noted that the logic and/or steps represented in the flowcharts or otherwise described herein, for example, may be considered as a ordered listing of executable instructions for implementing logical functions, and may be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer readable medium may even be paper or other suitable medium on which the program is printed, as the program may be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

It is to be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.

In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

Furthermore, the terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In the description of the present invention, the meaning of "plurality" means at least two, for example, two, three, etc., unless specifically defined otherwise.

While embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and not to be construed as limiting the invention, and that variations, modifications, alternatives and variations may be made to the above embodiments by one of ordinary skill in the art within the scope of the invention.

Claims

1. A control method of a machine vision screw tightening system, the system comprising an actuator, a first photographing mechanism fixedly mounted at a constant vertical distance from the conveyor, a second photographing mechanism detachably mounted on the actuator, the first photographing mechanism and the second photographing mechanism being configured to acquire image information, a conveyor configured to convey a screw workpiece to be screwed, the method comprising:

acquiring image information of the to-be-screwed screw workpiece through the first shooting mechanism and the second shooting mechanism;

Determining first pose information of the threaded hole of the to-be-screwed screw workpiece according to the image information acquired by the first shooting mechanism, and determining second pose information of the threaded hole of the to-be-screwed screw workpiece according to the image acquired by the second shooting mechanism;

Controlling the executing mechanism according to the pose information of the executing mechanism, the pose information of the conveying mechanism and the first pose information of the threaded hole of the to-be-screwed bolt workpiece, so that the speeds of the executing mechanism and the to-be-screwed bolt workpiece are synchronous, and the pose errors of the screw screwing mechanism and the threaded hole of the to-be-screwed bolt workpiece are smaller than a first preset error threshold;

And controlling the executing mechanism according to the first pose information, the second pose information, the pose information of the executing mechanism and the pose information of the conveying mechanism so that the pose error of the screw screwing mechanism and the threaded hole of the workpiece to be screwed is smaller than a second preset error threshold.

2. The control method according to claim 1, characterized in that controlling the actuator according to the pose information of the actuator, the pose information of the transfer mechanism, and the first pose information of the threaded hole of the screw workpiece to be screwed, comprises:

Determining first pose error information of the threaded hole of the screw workpiece to be screwed and the screw screwing mechanism according to pose information of the executing mechanism, pose information of the conveying mechanism and pose information of the threaded hole of the screw workpiece to be screwed;

generating a first motion control instruction according to the first pose error information, the pose information of the executing mechanism and the pose information of the conveying mechanism;

And controlling the executing mechanism according to the first motion control instruction.

3. The control method according to claim 1, characterized in that controlling the actuator based on the first pose information, the second pose information, the pose information of the actuator, and the pose information of the transmission mechanism, includes:

Fusing the first pose information and the second pose information to obtain fused pose information;

Determining a threaded hole of the workpiece to be screwed and second pose error information of the screw screwing mechanism according to the fusion pose information, the pose information of the transmission mechanism and the pose information of the execution mechanism;

generating a second motion control instruction according to the second pose error information, the pose information of the executing mechanism and the pose information of the conveying mechanism;

And controlling the executing mechanism according to the second motion control instruction.

4. The control method according to claim 3, wherein fusing the first pose information and the second pose information to obtain fused pose information includes:

Acquiring a first weight coefficient corresponding to the first pose information and a second weight coefficient corresponding to the second pose information;

And determining the fusion pose information according to the first pose information, the first weight coefficient, the second pose information and the second weight coefficient.

5. The control method according to claim 4, characterized in that a sum of the first weight coefficient and the second weight coefficient is 1, and the first weight coefficient gradually decreases with the movement of the actuator, and the second weight coefficient gradually increases with the movement of the actuator.

6. The control method according to claim 5, characterized in that the first weight coefficient and the second weight coefficient are determined by the following formula:

Wherein Wc represents the first weight coefficient, ws represents the second weight coefficient, p _i represents a weight polynomial coefficient, and Y represents pose information of the workpiece to be screwed.

7. The control method according to claim 3, wherein fusing the first pose information and the second pose information to obtain fused pose information includes:

and fusing the first pose information and the second pose information by adopting a Kalman filtering algorithm to obtain the fused pose information.

8. A control method according to claim 3, characterized in that the second pose error information is determined using the following formula:

err_init＝f(L₀,P_c,l_2c,l_1c)

Wherein err _init represents the second pose error information, L ₀ represents pose deviation between the second photographing mechanism and the screw screwing mechanism, P _c represents the fused pose information, L _2c represents pose information of the executing mechanism, and L _1c represents pose information of the transmitting mechanism.

9. The control method according to claim 1, characterized by further comprising:

Extracting features of the image information to obtain coordinates of feature points;

and determining the first pose information and the second pose information by adopting a PNP clustering algorithm or a binocular triangulation positioning algorithm based on the feature points.

10. The control method according to any one of claims 1 to 9, characterized by further comprising:

Acquiring the second pose information with a first preset servo frequency, and acquiring the pose information of the executing mechanism and the pose information of the conveying mechanism with a second preset servo frequency;

wherein the first preset servo frequency is smaller than the second preset servo frequency.

11. The machine vision screw screwing system is characterized by comprising an executing mechanism, a first shooting mechanism, a second shooting mechanism, a conveying mechanism, a screw screwing mechanism and a controller, wherein the first shooting mechanism is fixedly arranged at a position with a constant vertical distance from the conveying mechanism, the second shooting mechanism and the screw screwing mechanism are detachably arranged on the executing mechanism, the conveying mechanism is configured to convey a screw workpiece to be screwed, and the screw workpiece screwing mechanism comprises a screw screwing mechanism, a screw screwing mechanism and a screw screwing mechanism,

The first shooting mechanism and the second shooting mechanism are used for acquiring image information of the to-be-screwed screw workpiece;

The controller is used for determining first pose information of the threaded hole of the screw workpiece to be screwed according to the image information obtained by the first shooting mechanism, determining second pose information of the threaded hole of the screw workpiece to be screwed according to the image obtained by the second shooting mechanism, and controlling the executing mechanism according to the pose information of the executing mechanism, the pose information of the conveying mechanism and the first pose information of the threaded hole of the screw workpiece to be screwed so that the speeds of the executing mechanism and the screw workpiece to be screwed are synchronous, the pose errors of the screw mechanism and the threaded hole of the screw workpiece to be screwed are smaller than a first preset error threshold value, and controlling the executing mechanism according to the first pose information, the second pose information, the pose information of the executing mechanism and the pose information of the conveying mechanism so that the pose errors of the screw mechanism and the threaded hole of the screw workpiece to be screwed are smaller than a second preset error threshold value.

12. A computer-readable storage medium, characterized in that a control program of a machine vision screw tightening system is stored thereon, which control program, when executed by a processor, implements the control method of a machine vision screw tightening system according to any one of claims 1-10.

13. A screw driving apparatus comprising the machine vision screw driving system of claim 11.