WO2024128092A1 - Assembly system, control method, and program - Google Patents

Assembly system, control method, and program Download PDF

Info

Publication number
WO2024128092A1
WO2024128092A1 PCT/JP2023/043647 JP2023043647W WO2024128092A1 WO 2024128092 A1 WO2024128092 A1 WO 2024128092A1 JP 2023043647 W JP2023043647 W JP 2023043647W WO 2024128092 A1 WO2024128092 A1 WO 2024128092A1
Authority
WO
WIPO (PCT)
Prior art keywords
nut
unit
attitude angle
end effector
position coordinates
Prior art date
Application number
PCT/JP2023/043647
Other languages
French (fr)
Japanese (ja)
Inventor
洋音 小松
伸 浅野
篤 杉浦
満久 川部
哲也 尾形
裕紀 森
佑樹 菅
Original Assignee
三菱重工業株式会社
学校法人早稲田大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 三菱重工業株式会社, 学校法人早稲田大学 filed Critical 三菱重工業株式会社
Publication of WO2024128092A1 publication Critical patent/WO2024128092A1/en

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J13/00—Controls for manipulators
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J13/00—Controls for manipulators
    • B25J13/08—Controls for manipulators by means of sensing devices, e.g. viewing or touching devices

Definitions

  • the present disclosure relates to an assembly system, a control method, and a program.
  • This application claims priority to Japanese Patent Application No. 2022-199698, filed in Japan on December 14, 2022, the contents of which are incorporated herein by reference.
  • Patent Document 1 discloses a robot system in which a sensor detects changes in the contact state of an end effector equipped to a robot with a workpiece, and based on the change in contact state detected by the sensor, estimates and corrects the amount of misalignment of the end effector with respect to the workpiece.
  • Patent Document 1 only corrects the amount of misalignment based on changes in the contact state of the socket with the bolt head caused by the feel-and-see action of the end effector, making it difficult to reproduce the fastening action performed by a human. As a result, for example, it may take a long time for the bolt head to fit into the socket. Therefore, there is a demand for an assembly technology that allows for smoother fastening.
  • the present disclosure provides an assembly system, control method, and program that allows for smoother fastening.
  • the assembly system includes a manipulator having an arm with multiple joints, an end effector that is moved by the arm and holds a nut to rotate the nut around an axis, a first sensor that acquires the joint angles of each of the joints, and a second sensor that acquires the contact force and torque transmitted from the end effector when the nut contacts a bolt provided on a construction object, and a control device that controls the manipulator.
  • the control device includes an acquisition unit that acquires the joint angles of each of the joints and the contact force and torque at a first predetermined period, a calculation unit that calculates the position coordinates and attitude angle of the nut based on the acquired joint angles of each of the joints, and a control unit that controls the end effector using a first trained model that has been trained to output command values indicating the position coordinates and attitude angle of the nut at a second time after the first time when the position coordinates and attitude angle of the nut approach the target position coordinates and target attitude angle of the nut that indicate a state in which the nut is fastened to the bolt, when the position coordinates and attitude angle of the nut at a first time and the contact force and torque at the first time are input.
  • the control method is a method for controlling a manipulator having an arm with multiple joints, an end effector that is moved by the arm and holds a nut to rotate the nut around an axis, a first sensor that acquires the joint angle of each of the joints, and a second sensor that acquires the contact force and torque transmitted from the end effector, and includes the steps of acquiring the joint angle of each of the joints and the contact force and torque at a first predetermined period, calculating the position coordinates and attitude angle of the nut based on the acquired joint angles of each of the joints, and controlling the end effector using a first trained model that has been trained to output, when the position coordinates and attitude angle of the nut at a first time and the contact force and torque at the first time are input, command values indicating the position coordinates and attitude angle of the nut at a second time after the first time that approach the target position coordinates and target attitude angle of the nut that indicate a state in which the nut is fastened to the bolt.
  • the program according to the present disclosure causes a computer of a control device that controls a manipulator having an arm with multiple joints, an end effector that is moved by the arm and holds a nut and rotates the nut around an axis, a first sensor that acquires the joint angle of each of the joints, and a second sensor that acquires the contact force and torque transmitted from the end effector to execute the steps of acquiring the joint angle of each of the joints and the contact force and torque at a first predetermined period, calculating the position coordinates and attitude angle of the nut based on the acquired joint angles of each of the joints, and controlling the end effector using a first trained model that has been trained to output command values indicating the position coordinates and attitude angle of the nut at a second time after the first time when the position coordinates and attitude angle of the nut approach the target position coordinates and target attitude angle of the nut that indicate a state in which the nut is fastened to the bolt, when the position coordinates and attitude angle of the nut at a first time and the contact force and
  • the present disclosure provides an assembly system, control method, and program that allows for smoother fastening.
  • FIG. 1 is a schematic configuration diagram showing an entire assembly system according to a first embodiment of the present disclosure.
  • FIG. 2 is a diagram showing an end effector, a nut, and a bolt arranged in a line according to a first embodiment of the present disclosure.
  • FIG. 2 is a view showing the inside of the nut runner according to the first embodiment of the present disclosure as viewed from the axial direction.
  • 13A to 13C are diagrams for explaining the movement of a nut holding member when the end effector according to the first embodiment of the present disclosure holds a nut.
  • 13A to 13C are views for explaining the movement of the nut retaining member when removing the collar from the end effector according to the first embodiment of the present disclosure.
  • FIG. 2 is a diagram illustrating an example of an image captured by the imaging device according to the first embodiment of the present disclosure.
  • FIG. 2 is a functional block diagram of a control device according to the first embodiment of the present disclosure.
  • FIG. 2 is a diagram for conceptually explaining a coordinate system in which the position coordinates of a nut according to the first embodiment of the present disclosure are expressed.
  • 1A to 1C are diagrams for conceptually explaining a contact force received from a bolt when a nut held by the end effector according to the first embodiment of the present disclosure comes into contact with the bolt.
  • 5 is a diagram for conceptually explaining a correction value used by a correction unit according to the first embodiment of the present disclosure to correct a contact force.
  • FIG. 1 is a diagram illustrating an example of an image captured by the imaging device according to the first embodiment of the present disclosure.
  • FIG. 2 is a functional block diagram of a control device according to the first embodiment of the present disclosure.
  • FIG. 2 is a diagram for conceptually explaining a
  • FIG. 1 is a flowchart illustrating an example of a learning step of a first trained model according to the first embodiment of the present disclosure.
  • 5 is a flowchart showing an example of an operation of the control device according to the first embodiment of the present disclosure.
  • FIG. 11 is a schematic configuration diagram showing an entire assembly system according to a second embodiment of the present disclosure.
  • FIG. 13 is a schematic configuration diagram showing an entire assembly system according to a third embodiment of the present disclosure.
  • FIG. 13 is a schematic configuration diagram showing an entire assembly system according to a modified example of the third embodiment of the present disclosure.
  • FIG. 2 is a hardware configuration diagram illustrating a configuration of a computer according to an embodiment of the present disclosure.
  • the assembly system is a part of an automated system that operates in, for example, an assembly factory, and assembles a plurality of parts so as to be integrated by controlling a manipulator.
  • the parts to be assembled by the assembly system are referred to as "construction target".
  • the construction target 200 is divided into, for example, a first part 210 and a second part 220, and is placed on, for example, a workbench 300 (work desk).
  • the assembly system 100 integrates the first part 210 and the second part 220 by controlling the manipulator 10.
  • Examples of the construction target 200 include a panel, which is one of aircraft parts, and a panel, which is one of ship parts.
  • the assembly system 100 includes, for example, the manipulator 10 and a control device 30.
  • the manipulator 10 is a robot that moves the nut 202 as a fastener by changing the overall posture while holding the nut 202, and automatically fastens the nut 202 to a bolt 201 as a fastener provided on the construction target 200.
  • the manipulator 10 is a multi-joint robot having a plurality of joints 110. In this embodiment, a six-axis multi-joint robot having six joints 110 will be described as an example of the manipulator 10.
  • the manipulator 10 fastens the nut 202 to the bolt 201 that is temporarily fixed to the construction target 200 with an adhesive or the like, thereby integrating the first part 210 and the second part 220.
  • the nut 202 handled by the manipulator 10 is a collar nut having a nut body 202a and a collar 202b integrally formed with the nut body 202a.
  • the nut 202 has a nut hole 202h as a female threaded portion formed through both the nut body 202a and the collar 202b.
  • the collar 202b has an outer shell formed in a hexagonal column shape. In the process of fastening the nut 202 to the bolt 201 by the manipulator 10, only the collar 202b is removed (collar cut), and the nut body 202a is fastened to the bolt 201.
  • the bolt 201 is inserted through holes (not shown) provided in the first part 210 and the second part 220 as the construction object 200, and protrudes from the second part 220, exposing the male threaded portion 201a.
  • the bolt 201 is, for example, a stud bolt (implanted bolt) without a head.
  • the manipulator 10 fastens the exposed male threaded portion 201a of the bolt 201 by screwing it into the nut hole 202h of the nut 202.
  • the manipulator 10 has, for example, an arm 11, an encoder 130, a force sensor 140, an end effector 12, and a base portion 15.
  • the arm 11 constitutes the main body of the manipulator 10.
  • the arm 11 is placed on a pedestal 15 placed on the ground. Specifically, one end of the arm 11 is fixed to the pedestal 15.
  • the arm 11 can move three-dimensionally around the pedestal 15 by changing its posture with the pedestal 15 as a base point.
  • the arm 11 has, for example, a plurality of (six) joints 110, a first connection portion 118a, a second connection portion 118b, and a third connection portion 118c.
  • Each joint 110 is, for example, cylindrical, and has a motor 117 therein for making the joint 110 rotatable.
  • the motor 117 in the joint 110 may be, for example, a servo motor.
  • the motor 117 rotates to rotate the entire joint 110 around a rotation axis around which the motor 117 rotates.
  • the six joints 110 provided on the arm 11 will be referred to as the "first joint 111,” “second joint 112,” “third joint 113,” “fourth joint 114,” “fifth joint 115,” and “sixth joint 116,” in order from the side closest to the base 15.
  • the motor 117 of the first joint 111 is referred to as the "first motor 117a”
  • the motor 117 of the second joint 112 is referred to as the “second motor 117b”
  • the motor 117 of the third joint 113 is referred to as the “third motor 117c”
  • the motor 117 of the fourth joint 114 is referred to as the "fourth motor 117d”
  • the motor 117 of the fifth joint 115 is referred to as the "fifth motor 117e”
  • the motor 117 of the sixth joint 116 is referred to as the “sixth motor 117f”.
  • the rotation of each motor 117 (first motor 117a to sixth motor 117f) is controlled by a control device 30 (described below).
  • the first joint 111 is fixed to the upper surface of the base 15.
  • the first joint 111 corresponds to the one end of the arm 11.
  • the first motor 117a can rotate around a first rotation axis O1 as a pan axis extending in the vertical direction, for example, and rotates the entire first joint 111 around the first rotation axis O1.
  • the second joint 112 is provided so as to extend from the outer surface of the first joint 111 in a state integral with the first joint 111.
  • the second motor 117b can rotate around a second rotation axis O2 as a tilt axis extending in the horizontal direction, for example, and rotates the entire second joint 112 around the second rotation axis O2.
  • the second joint 112 is provided with a first connection portion 118a extending in a direction intersecting the second rotation axis O2.
  • the first connecting portion 118a is cylindrical and extends from the outer surface of the second joint portion 112.
  • the third joint portion 113 is provided integrally at the end of the first connecting portion 118a extending from the second joint portion 112.
  • the third motor 117c is rotatable about a third rotation axis O3 as a tilt axis extending in the horizontal direction, for example, and rotates the entire third joint portion 113 about the third rotation axis O3.
  • the third joint portion 113 is provided with a second connecting portion 118b extending in a direction intersecting the third rotation axis O3.
  • the second connecting portion 118b is cylindrical and extends from the outer surface of the third joint portion 113.
  • the fourth joint portion 114 is provided integrally at the end of the second connecting portion 118b extending from the third joint portion 113.
  • the fourth motor 117d is rotatable around a fourth rotation axis O4 as a tilt axis extending in the horizontal direction, for example, and rotates the entire fourth joint 114 around the fourth rotation axis O4.
  • the fifth joint 115 is provided integrally with the fourth joint 114.
  • the fifth motor 117e is rotatable around a fifth rotation axis O5 as a pan axis extending in a direction intersecting with the fourth rotation axis O4, for example, and rotates the entire fifth joint 115 around the fifth rotation axis O5.
  • the fifth joint 115 is provided with a third connection portion 118c extending in a direction intersecting with the fifth rotation axis O5.
  • the third connection portion 118c is cylindrical and extends from the outer surface of the fifth joint 115.
  • the sixth joint 116 is provided integrally with the third connection portion 118c at the end extending from the fifth joint 115.
  • the sixth joint 116 corresponds to the other end of the arm 11 opposite to the one end.
  • the sixth motor 117f can rotate around a sixth rotation axis O6 as a pan axis extending parallel to the fifth rotation axis O5.
  • the posture of the arm 11 changes three-dimensionally.
  • the sixth rotation axis O6 is simply referred to as the "axis Ar1".
  • the direction in which the axis Ar1 extends is referred to as the "axial direction Da”
  • one side in the axial direction Da is simply referred to as the "one side Da1”
  • the side opposite the one side Da1 is simply referred to as the "other side Da2”.
  • the motor 117 of each joint 110 is an example of a controlled device S.
  • the encoder 130 is a sensor capable of acquiring the rotation angle of the joint 110.
  • the rotation angles ( ⁇ 1 to ⁇ 6) of each of the joints 110 (the first joint 111 to the sixth joint 116) will be referred to as the “joint angle.”
  • one encoder 130 is provided for each of the motors 117 of the joints 110.
  • the encoder 130 provided in the first motor 117a is referred to as the "first encoder 131”
  • the encoder 130 provided in the second motor 117b is referred to as the “second encoder 132”
  • the encoder 130 provided in the third motor 117c is referred to as the “third encoder 133”
  • the encoder 130 provided in the fourth motor 117d is referred to as the "fourth encoder 134”
  • the encoder 130 provided in the fifth motor 117e is referred to as the "fifth encoder 135"
  • the encoder 130 provided in the sixth motor 117f is referred to as the "sixth encoder 136".
  • Each encoder 130 transmits the joint angle (sensor data) acquired from the motor 117 to the control device 30.
  • the encoder 130 is an example of the first sensor 13.
  • the first sensor 13 is connected to the control device 30 by wired or wireless communication.
  • the force sensor 140 is a load cell connected to the arm 11. Specifically, the force sensor 140 is fixed to the sixth joint 116 from one side Da1 in the axial direction Da. The force sensor 140 is provided with an end effector 12 (described later) on the opposite side (one side Da1 in the axial direction Da) of the sixth joint 116. The force sensor 140 can acquire data related to the force transmitted from the end effector 12.
  • the force sensor 140 can acquire the magnitude of the contact force transmitted to the force sensor 140 from the one side Da1 on the end effector 12 side for each of the three directional components (Fx, Fy, Fz) of the three axes, and can acquire the magnitude of the torque (Mx, My, Mz) acting around each axis of the contact force acting on each axis.
  • the contact force (unit: N) and torque (unit: Nm) acquired by the force sensor data may be collectively referred to as "force data.” That is, the force sensor 140 is capable of acquiring six-axis force data.
  • the force sensor 140 transmits the acquired force data (sensor data) to the control device 30.
  • the force sensor 140 is an example of the second sensor 14.
  • the second sensor 14 is connected to the control device 30 by wired or wireless communication.
  • the end effector 12 holds the nut 202 and rotates the nut 202 around the rotation axis Ar.
  • the rotation axis Ar is, for example, a virtual axis extending in the axial direction Da at a position offset from the axis Ar1.
  • the rotation axis Ar may coincide with the axis Ar1.
  • the end effector 12 is provided on one side Da1 of the force sensor 140. Therefore, the end effector 12 is moved by the arm 11 and fastens the held nut 202 to the bolt 201 provided on the construction target 200. As shown in FIG. 1 and FIG.
  • the end effector 12 has, for example, an imaging device 20, a main body 120, a nut runner 121, a nut holding member 124, and an elastic member 128.
  • the socket portion 123 of the nut runner 121 is shown in cross section so that the inside can be seen.
  • the imaging device 20 is provided in the end effector 12 and generates an image of the construction target 200.
  • the imaging device 20 is arranged inside the socket part 123 of the nut runner 121 of the end effector 12 described later in a state where it cannot rotate with respect to the socket part 123.
  • the imaging device 20 is arranged in the socket part 123 (the storage space R described later) closer to the base part 122 than the storage part 123r described later.
  • the imaging device 20 has, for example, an imaging unit 21 and an illumination unit 22.
  • the imaging unit 21 generates an image of the construction target 200 through a nut hole 202h of a nut 202 held by a nut holding member 124 described later.
  • the imaging unit 21 can be exemplified by a sensor module in which a semiconductor element (solid-state imaging element) such as a CCD image sensor or a CMOS image sensor is incorporated. Note that the imaging unit 21 may be a fiberscope or the like instead of this.
  • the illumination unit 22 irradiates light toward the construction target 200 through the nut hole 202h.
  • the illumination unit 22 may be, for example, an element such as an LED (Light Emitting Diode).
  • FIG. 6 is a diagram showing an example of an image captured by the imaging device 20.
  • FIG. 6 is a diagram showing an example of an image captured by the imaging device 20.
  • FIG. 6 shows an example of an image of a bolt 201 provided on the construction target 200 captured from a slightly obliquely upward direction.
  • FIG. 6 shows an example of a bolt 201 in which a hole is formed with a center line Ar2 (not shown) of the bolt 201 as the center, but the bolt 201 may not have a hole.
  • the imaging device 20 can image the surface of the construction target 200, the male thread portion 201a of the bolt 201, and the end surface 201e of the bolt 201 while irradiating the construction target 200 and the bolt 201 with light through the nut hole 202h.
  • the imaging device 20 is connected to the control device 30 by wired or wireless communication.
  • the imaging device 20 transmits the captured image to the control device 30 .
  • the main body 120 is fixed to the force sensor 140.
  • a socket 123 (described later, see FIG. 2) of a nut runner 121 is inserted into the main body 120, and the main body 120 has therein a fastening motor 120a (see FIG. 1) for rotating the socket 123 about a rotation axis Ar.
  • the rotation of the fastening motor 120a of the main body 120 is controlled by the control device 30.
  • the fastening motor 120a of the main body 120 is an example of a device S to be controlled.
  • the nut runner 121 has, for example, a base 122 and a socket 123.
  • the base 122 has, for example, a disk shape centered on the rotation axis Ar, and is fixed to the main body 120 from one side Da1.
  • the base 122 has a front surface 122a facing the one side Da1.
  • the socket 123 is provided so as to extend toward the one side Da1 beyond the front surface 122a of the base 122, and has a cylindrical shape centered on the rotation axis Ar.
  • the socket 123 is inserted into the base 122 and the main body 120, and is rotatably held by a fastening motor 120a provided in the main body 120 (see FIG. 1).
  • the socket 123 has a cylindrical inner circumferential surface 123i that defines the interior, and a cylindrical outer circumferential surface 123o that faces outward.
  • the socket portion 123 has an opening 123e at a tip portion located on one side Da1.
  • the opening 123e is formed in a perfect circle perpendicular to the rotation axis Ar.
  • the socket portion 123 also has an accommodating portion 123r that includes the opening 123e and is capable of accommodating at least a portion of the nut 202.
  • the accommodating portion 123r is located on one side Da1 in the accommodation space R.
  • the accommodation portion 123r is composed of, for example, a plurality of tapered surfaces 123a, a plurality of fastening surfaces 123b, and a positioning surface 123c.
  • Each tapered surface 123a is arranged at an incline with respect to the direction in which the opening 123e widens.
  • each tapered surface 123a is arranged at an incline with respect to the direction in which the opening 123e widens so as to approach the rotation axis Ar as it approaches the other side Da2.
  • six tapered surfaces 123a are arranged side by side around the rotation axis Ar, and adjacent tapered surfaces 123a around the rotation axis Ar are connected to each other.
  • the area defined by the plurality of tapered surfaces 123a in the accommodation portion 123r has a cross section perpendicular to the rotation axis Ar that forms a regular hexagon, and is tapered toward the other side Da2.
  • Each fastening surface 123b is connected to the tapered surface 123a from the other side Da2 so as to correspond to one of the tapered surfaces 123a, and is arranged perpendicular to the direction in which the opening 123e widens.
  • the fastening surfaces 123b are arranged in a row of six around the rotation axis Ar, and the fastening surfaces 123b adjacent to each other around the rotation axis Ar are connected to each other.
  • the area defined by the multiple fastening surfaces 123b in the accommodating portion 123r forms a hexagonal prism whose cross section perpendicular to the rotation axis Ar is constant in the axial direction Da.
  • the area defined by the multiple fastening surfaces 123b is, for example, large enough to fit the collar 202b of the nut 202.
  • the positioning surface 123c is a surface that connects the multiple (six) fastening surfaces 123b and the inner peripheral surface 123i of the socket portion 123 to each other, and faces one side Da1.
  • the positioning surface 123c is arranged parallel to the direction in which the opening 123e widens. In other words, the positioning surface 123c is arranged perpendicular to each fastening surface 123b.
  • the positioning surface 123c is a surface for positioning the nut 202 when the nut 202 is accommodated in the accommodation portion 123r. As shown in FIG. 3, the positioning surface 123c has a shape obtained by removing a perfect circle from the center part of a regular hexagon when viewed from one side Da1.
  • the collar 202b of the nut 202 abuts against this positioning surface 123c from one side Da1, thereby positioning the nut 202.
  • the position of the nut 202 is determined by the positioning surface 123c, so that the nut 202 does not enter the accommodation space R on the other side Da2 beyond the accommodation portion 123r.
  • the nut holding member 124 holds the nut 202 so that the nut 202 does not fall out of the housing portion 123r to one side Da1 when the collar 202b of the nut 202 is housed in the housing portion 123r of the nut runner 121.
  • the nut holding member 124 is provided on the outer periphery side of the socket portion 123.
  • two nut holding members 124 are provided on the outer periphery side of the socket portion 123 so as to sandwich the socket portion 123 therebetween.
  • three or more nut holding members 124 may be disposed at equal intervals around the rotation axis Ar with the socket portion 123 as the center.
  • the nut holding member 124 has, for example, a rotating shaft 125, a first portion 126, and a second portion 127.
  • the pivot shaft 125 is rotatably provided on the outer peripheral surface 123o of the socket portion 123.
  • the first portion 126 is fixed to the pivot shaft 125 and is a rod-shaped member extending in a direction intersecting the rotation axis Ar.
  • the second portion 127 has an extension portion 127a extending from the first portion 126 toward the tip side of the socket portion 123, and a claw portion 127b provided at the tip of the extension portion 127a and protruding from the extension portion 127a so as to approach the rotation axis Ar.
  • the extension portion 127a is a rod-shaped member extending along the rotation axis Ar.
  • the extension portion 127a forms an L-shape together with the first portion 126.
  • the claw portion 127b extends from the extension portion 127a toward the other side Da2 as it approaches the rotation axis Ar, and has a tapered shape as it approaches the rotation axis Ar.
  • the claw portion 127b is disposed on one side Da1 of the opening 123e of the socket portion 123. Also, as shown in FIG. 2 and FIG. 3, the tip of the claw portion 127b is located closer to the rotation axis Ar than the fastening surface 123b.
  • the elastic member 128 is a spring provided between the base 122 of the nut runner 121 and the first portion 126 of the nut holding member 124. Specifically, one end of the elastic member 128 is fixed to the front surface 122a of the base 122, and the other end is fixed to the first portion 126. Thus, the elastic member 128 connects the base 122 and the first portion 126 to each other. The elastic member 128 applies an elastic force toward the tip side (one side Da1) of the socket portion 123 to the first portion 126. In this embodiment, one elastic member 128 is provided between the base 122 and the first portion 126 so as to correspond to one nut holding member 124.
  • a plurality of elastic members 128 may be provided between the base 122 and the first portion 126 so as to correspond to one nut holding member 124.
  • the elastic member 128 is not limited to a spring, so long as it is capable of applying an elastic force to the first portion 126 in the direction toward the tip side (one side Da1) of the socket portion 123.
  • the collar 202b of the nut 202 that has passed between the claw portions 127b enters the accommodating portion 123r and is positioned in the area defined by the fastening surface 123b and the positioning surface 123c while being guided to the other side Da2 by the tapered surface 123a.
  • the elastic member 128 presses the first portion 126 to one side Da1
  • the claw portion 127b integral with the extension portion 127a extending from the first portion 126 abuts against the side surface of the nut body 202a.
  • the nut body 202a is gripped (restrained) by the two claw portions 127b.
  • the nut 202 may be held by the nut holding member 124 of the end effector 12 using a jig, actuator, or the like instead of the operator's hand.
  • the first part 126 is pressed to the other side Da2 by the hand of an operator or the like against the elastic force of the elastic member 128, and the first part 126 and the second part 127 rotate around the rotation shaft 125.
  • the gap between the claw portions 127b widens to an extent that the collar 202b can pass through.
  • the collar 202b held by the claw portions 127b falls from the gap between the claw portions 127b.
  • the first portion 126 functions as a lever to allow the collar 202b held by the nut holding member 124 to drop through the gap between the claws 127b after the nut body 202a and the bolt 201 have been tightened.
  • the control device 30 controls each joint angle of the manipulator 10 and the fastening motor 120a of the main body 120 of the end effector 12 so that the nut 202 held by the end effector 12 is fastened to the bolt 201 provided on the construction target 200.
  • the control device 30 includes, for example, an acquisition unit 31, a calculation unit 32, a coordinate conversion unit 33, a correction unit 34, a determination unit 35, a control unit 36, a learning unit 37, a counter unit 38, and a storage unit 39.
  • the acquisition unit 31 acquires sensor data over time by receiving the sensor data transmitted from the encoder 130 (first sensor 13) and the force sensor 140 (second sensor 14) in real time.
  • the acquisition unit 31 acquires the joint angles of the joints 110 (first joint 111 to sixth joint 116) from the encoder 130 at a first predetermined period, and acquires force data from the force sensor 140 at the first predetermined period.
  • the first predetermined period is determined based on, for example, a sampling rate of the encoder 130 or the force sensor 140.
  • the acquisition unit 31 also acquires images over time by receiving images transmitted from the imaging device 20 in real time.
  • the acquisition unit 31 acquires images from the imaging device 20 at a second predetermined period.
  • the second predetermined period is determined based on, for example, a frame rate (fps: flames per second) of the imaging device 20.
  • the second predetermined period may be the same as the first predetermined period, or may be different.
  • the acquisition unit 31 also acquires the central position coordinates of the bolt 201 to be immediately executed from CAD data 395 (Computer-Aided Design) of the construction target 200 stored in advance in the storage unit 39.
  • the CAD data 395 of the construction target 200 includes, for example, point cloud data that three-dimensionally shows the contour of the construction target 200. Each point constituting the point cloud data is assigned a three-dimensional coordinate (Xw, Yw, Zw) expressed in a world coordinate system based on the origin (0, 0, 0) of the manipulator 10.
  • the acquisition unit 31 sends the acquired joint angles of each joint 110 to the calculation unit 32 and stores them in the memory unit 39, sends the acquired force data to the correction unit 34 and stores them in the memory unit 39, sends the acquired center position coordinates of the bolt 201 to the coordinate conversion unit 33 and stores them in the memory unit 39, and sends the acquired images to the judgment unit 35 and the control unit 36 and stores them in the memory unit 39.
  • the acquisition unit 31 sends the acquired center position coordinates of the bolt 201 to the coordinate conversion unit 33, it also sends an acquisition flag to the counter unit 38 indicating that it has acquired the center position coordinates of the bolt 201 that is the target of the most recent construction.
  • the acquisition flag is, for example, 1.
  • the counter unit 38 increments the acquisition flag each time it receives it from the acquisition unit 31, and sends the incremented acquisition flag to the sixth judgment unit 356.
  • the calculation unit 32 calculates the position coordinates and the attitude angle of the nut 202 based on the joint angles of each joint 110 (the first joint 111 to the sixth joint 116) received from the acquisition unit 31. Specifically, as shown in FIG. 8, the calculation unit 32 calculates the position coordinates (Xn, Yn, Zn) of the center part of the end face 202e of the nut 202 based on the origin of the manipulator 10, and the attitude angle ( ⁇ x, ⁇ y, ⁇ z) which is the inclination angle with respect to each of the three coordinate planes (XY plane, YZ plane, ZX plane) of the world coordinate system in which the position coordinates are expressed, based on the joint angles of each joint 110.
  • the position coordinates (Xn, Yn, Zn) of the center part of the end face 202e of the nut 202 based on the origin of the manipulator 10
  • the attitude angle ( ⁇ x, ⁇ y, ⁇ z) which is the inclination angle with respect to each of the three coordinate
  • the origin of the manipulator 10 may be any one point set in the manipulator 10, or may be any one point in the construction object 200 set after calibration or the like is performed to adjust the positional relationship between the manipulator 10 and the construction object 200.
  • FIG. 8 shows an example in which the origin of the manipulator 10 is located in the construction target object 200 .
  • the calculation unit 32 sends the calculated position coordinates of the nut 202 to the coordinate conversion unit 33 and stores them in the memory unit 39, and sends the calculated attitude angle of the nut 202 to the determination unit 35 and control unit 36 and stores them in the memory unit 39.
  • the calculation unit 32 may input each joint angle into correspondence information (such as a function) that is stored in advance in the memory unit 39 and associates the joint angles of each joint unit 110 with the position coordinates and attitude angle of the nut 202, and obtain the output position coordinates and attitude angle as the position coordinates and attitude angle of the nut 202.
  • the coordinate conversion unit 33 converts the position coordinates of the nut 202 received from the calculation unit 32 into position coordinates expressed by the bolt-centered coordinate system based on the central position coordinates of the bolt 201 received from the acquisition unit 31. Specifically, the coordinate conversion unit 33 converts the position coordinates of the nut 202 into the bolt-centered coordinate system (Xb, Yb, Zb) as shown by the arrow A (vector) in FIG. 8 by subtracting the vector indicating the central position coordinates of the bolt 201 to be the most recent construction target from the vector indicating the position coordinates of the nut 202 expressed in the world coordinate system.
  • the central position coordinates of the bolt 201 (the origin of the bolt-centered coordinate system) are, for example, located somewhere on the center line Ar2 of the bolt 201, and are, for example, located at a point where the surface of the construction target 200 and the center line Ar2 of the bolt 201 virtually intersect.
  • the coordinate conversion unit 33 sends the position coordinates of the nut 202 converted into the bolt-centered coordinate system to the determination unit 35 and the control unit 36 and stores them in the memory unit 39.
  • FIG. 9 is a diagram for conceptually explaining the contact force that the nut 202 held by the end effector 12 receives from the bolt 201 when the nut 202 contacts the bolt 201.
  • the contact force acting in the axial direction Da is indicated by an arrow Fb
  • the torque acting around the rotation axis Ar is indicated by an arrow Ft.
  • FIG. 10 is a diagram for conceptually explaining the correction value (arrow Fg1) used by the correction unit 34 to correct the contact force (arrow Fb).
  • Dv indicates the vertical direction
  • CG indicates the center of gravity position of the end effector 12.
  • the correction unit 34 corrects the contact force (arrow Fb) received from the acquisition unit 31 by adding a correction value (arrow Fg1) based on the weight (arrow Fg) of the end effector 12 to the contact force.
  • the correction value (arrow Fg1) indicates a negative value, for example, when the weight (arrow Fg) of the end effector 12 due to gravity has a directional component on one side Da1 in the axial direction Da ((a) in FIG. 10).
  • the correction value (arrow Fg1) indicates a positive value, for example, when the weight (arrow Fg) of the end effector 12 has a directional component on the other side Da2 in the axial direction Da ((b) in FIG. 10).
  • the correction value is stored in advance in the storage unit 39 as, for example, correspondence information (such as a function) associated with the joint angle of each joint unit 110.
  • the correction unit 34 acquires the correction value by referring to the correspondence information stored in the storage unit 39 at appropriate times.
  • the correction unit 34 sends the force data including the corrected contact force to the determination unit 35 and the control unit 36 and stores it in the storage unit 39.
  • the determination unit 35 performs various determination processes based on the image received from the acquisition unit 31, the attitude angle of the nut 202 received from the calculation unit 32, the position coordinates of the nut 202 received from the coordinate conversion unit 33, and the force data received from the correction unit 34.
  • the determination unit 35 has, for example, a first determination unit 351, a second determination unit 352, a third determination unit 353, a fourth determination unit 354, a fifth determination unit 355, and a sixth determination unit 356.
  • the first determination unit 351 determines whether the nut 202 is in contact with the bolt 201 based on the force data received from the correction unit 34.
  • the determination by the first determination unit 351 is referred to as the "first determination". For example, when the contact force included in the received force data is less than a first threshold value indicating the magnitude of a predetermined contact force, the first determination unit 351 determines that the nut 202 is not in contact with the bolt 201. On the other hand, when the contact force included in the acquired force data is equal to or greater than the first threshold value, the first determination unit 351 determines that the nut 202 is in contact with the bolt 201.
  • the position of the nut 202 when the nut 202 is not in contact with the bolt 201 is referred to as the "first position”
  • the position of the nut 202 when the nut 202 is in contact with the bolt 201 is referred to as the "second position”.
  • the state in which the nut 202 and the bolt 201 are screwed together, which will be described later, is included in the state in which the nut 202 is located at the second position.
  • the first threshold value is stored in advance in, for example, the storage unit 39.
  • the first determination unit 351 makes the first determination by timely referring to the first threshold value stored in the memory unit 39.
  • the first determination unit 351 sends the result of the first determination, which indicates whether the nut 202 is in contact with the bolt 201, to the control unit 36.
  • the second determination unit 352 determines whether or not there is an abnormality in the state of the nut 202 based on the attitude angle of the nut 202 received from the calculation unit 32 and the position coordinates of the nut 202 received from the coordinate conversion unit 33.
  • the determination by the second determination unit 352 is referred to as the "second determination”.
  • the second determination unit 352 determines that there is a problem with the position coordinates of the nut 202.
  • the second determination unit 352 determines that there is no problem with the position coordinates of the nut 202.
  • the second determination unit 352 determines that there is a problem with the attitude angle of the nut 202.
  • the second determination unit 352 determines that there is no problem with the attitude angle of the nut 202.
  • the second determination unit 352 determines that there is a problem with one or more of the position coordinates and the attitude angle of the nut 202, it determines that there is an abnormality in the state of the nut 202.
  • the second determination unit 352 determines that there is no problem with either the position coordinates or the attitude angle of the nut 202, it determines that there is no abnormality in the state of the nut 202.
  • the above coordinate area and attitude angle threshold value are stored in advance in, for example, the storage unit 39.
  • the second determination unit 352 performs the above second determination by referring to the coordinate area and the attitude angle threshold value stored in the storage unit 39 at appropriate times.
  • the second determination unit 352 sends the result of the second determination to the control unit 36.
  • the third determination unit 353 determines whether or not the nut 202 has been successfully screwed based on the image received from the acquisition unit 31, the attitude angle of the nut 202 received from the calculation unit 32, the position coordinates of the nut 202 received from the coordinate conversion unit 33, and the force data received from the correction unit 34. That is, the third determination unit 353 determines whether or not the nut 202 and the bolt 201 have been screwed together.
  • screwed together means, for example, a state in which the first thread closest to the end face 202e in the nut hole 202h as the female thread portion of the nut 202 is engaged (engaged) with the first thread closest to the end face 201e in the exposed male thread portion 201a of the bolt 201.
  • the third determination unit 353 performs a determination using a trained model that has been trained to output a determination result indicating whether or not the nut 202 and the bolt 201 have been screwed together.
  • the judgment by the third judgment unit 353 is referred to as the "third judgment”
  • the trained model used by the third judgment unit 353 for the third judgment is referred to as the "third trained model 393". That is, the third trained model 393 uses a data set including an image, the position coordinates and attitude angle of the nut 202, and force data as input elements, and uses a judgment result indicating whether the nut 202 and the bolt 201 are screwed together as an output element.
  • the third trained model 393 is stored in advance in, for example, the storage unit 39.
  • the third judgment unit 353 inputs the above data set into the third trained model 393 stored in the storage unit 39, and acquires the output judgment result as the result of the third judgment.
  • the third trained model 393 is, for example, a deep learning model (supervised learning model) such as a deep neural network (DNN).
  • the third trained model 393 is generated (learned) by repeating a learning step multiple times (for example, several hundred times) in which the above dataset is input and the successful or unsuccessful screwing of the nut 202 for the input dataset (correct answer data that is accurately determined by a human as being screwed or not screwed) is taught.
  • the third trained model 393 may be, for example, a deep learning model such as a convolutional neural network (CNN) or a recurrent neural network (RNN).
  • CNN convolutional neural network
  • RNN recurrent neural network
  • the retry flag output by the third judgment unit 353 indicates the result of the third judgment and has a value of 1 or 0.
  • the third judgment unit 353 sends the retry flag, which is the result of the third judgment, to the control unit 36 and the counter unit 38 and stores it in the memory unit 39.
  • the counter unit 38 increments the retry flag each time it receives a retry flag from the third determination unit 353, and sends the incremented retry flag to the sixth determination unit 356.
  • the fourth determination unit 354 determines whether the fastening of the nut 202 has been successful or not based on the image received from the acquisition unit 31, the attitude angle of the nut 202 received from the calculation unit 32, the position coordinates of the nut 202 received from the coordinate conversion unit 33, and the force data received from the correction unit 34. That is, the fourth determination unit 354 determines whether the fastening of the nut 202 and the bolt 201 has been successful or not.
  • fastening means, for example, a state in which the last thread farthest from the end face 202e in the nut hole 202h as the female thread portion of the nut 202 is fitted into the last thread farthest from the end face 201e in the exposed male thread portion 201a of the bolt 201.
  • the fourth determination unit 354 performs the determination using a trained model trained to output a determination result indicating whether the fastening of the nut 202 and the bolt 201 has been successful or not.
  • the judgment by the fourth judgment unit 354 is referred to as the "fourth judgment", and the trained model used by the fourth judgment unit 354 for the fourth judgment is referred to as the "fourth trained model 394". That is, the fourth trained model 394 uses a data set including an image, the position coordinates and attitude angle of the nut 202, and force data as input elements, and uses a judgment result indicating whether the nut 202 and the bolt 201 are fastened as an output element.
  • the fourth trained model 394 is stored in advance in, for example, the storage unit 39.
  • the fourth judgment unit 354 inputs the above data set into the fourth trained model 394 stored in the storage unit 39, and acquires the output judgment result as the result of the fourth judgment.
  • the fourth trained model 394 is, for example, a deep learning model (supervised learning model) such as a deep neural network (DNN).
  • the fourth trained model 394 is generated (learned) by repeating a learning step multiple times (for example, several hundred times) in which the above dataset is input and the success or failure of fastening the nut 202 for the input dataset (correct answer data that is accurately determined by a human as being fastened or not fastened) is taught.
  • the fourth trained model 394 may be, for example, a deep learning model such as a convolutional neural network (CNN) or a recurrent neural network (RNN).
  • CNN convolutional neural network
  • RNN recurrent neural network
  • the retry flag output by the fourth judgment unit 354 indicates the result of the fourth judgment and has a value of 1 or 0.
  • the fourth judgment unit 354 sends the retry flag, which is the result of the fourth judgment, to the fifth judgment unit 355, the control unit 36, and the counter unit 38, and stores it in the memory unit 39.
  • the counter unit 38 increments the retry flag each time it receives a retry flag from the fourth determination unit 354, and sends the incremented retry flag to the sixth determination unit 356.
  • the fifth judgment unit 355 judges that the collar 202b has been cut when the contact force included in the received force data is less than a second threshold value indicating the magnitude of a predetermined contact force. On the other hand, the fifth judgment unit 355 judges that the collar 202b has not been cut when the contact force included in the acquired force data is equal to or greater than the second threshold value.
  • the second threshold value is stored in advance in, for example, the storage unit 39.
  • the fifth judgment unit 355 performs the fifth judgment by referring to the second threshold value stored in the storage unit 39 at appropriate times. Instead of the above judgment operation, the fifth judgment unit 355 may determine whether or not the fastening of the nut 202 has been successful based on the image received from the acquisition unit 31, the attitude angle of the nut 202 received from the calculation unit 32, the position coordinates of the nut 202 received from the coordinate conversion unit 33, and the force data received from the correction unit 34.
  • the fifth judgment unit 355 may perform the judgment using a fifth trained model 396 that has been trained to output a judgment result indicating whether or not the collar 202b has been cut when a data set including the image, the position coordinates and attitude angle of the nut 202, and the force data is input.
  • the fifth trained model 396 is stored in advance in, for example, the storage unit 39.
  • the fifth judgment unit 355 sends the result of the fifth judgment to the control unit 36 and stores it in the storage unit 39.
  • the sixth determination unit 356 performs a predetermined determination based on the addition results of various flags received from the counter unit 38. Hereinafter, the determination by the sixth determination unit 356 is referred to as the "sixth determination”. The sixth determination unit 356 determines whether or not the nuts 202 have been fastened to all the bolts 201 based on the addition results of the acquisition flags received from the counter unit 38. Specifically, the sixth determination unit 356 determines that the nuts 202 have been fastened to all the bolts 201 when the addition result of the acquisition flags received from the counter unit 38 is the number of bolts 201 to be installed on the construction target 200.
  • the sixth determination unit 356 determines that the nuts 202 have not been fastened to all the bolts 201 when the addition result of the acquisition flags received from the counter unit 38 is less than the number of bolts 201 installed on the construction target 200.
  • the number of bolts 201 provided on the construction target object 200 is stored in advance, for example, in the CAD data 395 of the memory unit 39, and is an integer (for example, several thousand to tens of thousands).
  • the sixth determination unit 356 also determines whether or not a predetermined number of retries have been performed based on the result of the retry flag addition received from the counter unit 38. Specifically, the sixth determination unit 356 determines that a predetermined number of retries have been performed when the result of the retry flag addition received from the counter unit 38 is equal to or greater than a predetermined retry threshold. On the other hand, the sixth determination unit 356 determines that a predetermined number of retries have not been performed when the result of the retry flag addition received from the counter unit 38 is less than the retry threshold.
  • the retry threshold is, for example, stored in advance in the memory unit 39 and is an integer. The sixth determination unit 356 performs the sixth determination by referring to the retry threshold stored in the memory unit 39 at appropriate times. The sixth determination unit 356 sends the result of the sixth determination to the learning unit 37 and the control unit 36.
  • the control unit 36 controls the movement of the end effector 12 based on the results of various determinations received from the determination unit 35.
  • the control unit 36 has, for example, a first control unit 361, a second control unit 362, and a third control unit 363.
  • the first control unit 361 executes a fastening operation of the nut 202 with respect to the bolt 201 based on a data set including the attitude angle of the nut 202 received from the calculation unit 32, the position coordinates of the nut 202 received from the coordinate conversion unit 33, and the force data received from the correction unit 34.
  • the "fastening operation” here includes changing the position coordinates and attitude angle of the nut 202 so that the nut 202 and the bolt 201 are screwed together when the nut 202 is located at the second position, and rotating the nut 202 so that the nut 202 and the bolt 201 are fastened together when the nut 202 and the bolt 201 are screwed together (when the nut 202 is located at the second position).
  • the first control unit 361 controls the movement and rotation of the end effector 12 using the learned model.
  • movement of the end effector 12 means, for example, control of the joint angle of the motor 117 of each joint unit 110 of the arm 11, resulting in movement of the end effector 12 and the nut 202 held by the end effector 12, and "rotation of the end effector 12” means, for example, control of the fastening motor 120a of the main body unit 120 of the end effector 12, resulting in rotation of the nut 202.
  • the trained model used by the first control unit 361 for the above-mentioned control of the end effector 12 will be referred to as the "first trained model 391.”
  • the first trained model 391 is stored in advance in, for example, the storage unit 39.
  • the first trained model 391 is trained to output a command value indicating the position coordinates and the posture angle of the nut 202 at a second time after the first time when the nut 202 approaches the target position coordinates and the target posture angle of the nut 202, which indicates the state in which the nut 202 is fastened to the bolt 201, when a data set including the position coordinates and the posture angle of the nut 202 at the first time and the contact force and the torque at the first time is input.
  • the first trained model 391 uses the data set including the position coordinates and the posture angle of the nut 202 at the first time and the contact force and the torque at the first time as input elements, and uses the command value indicating the position coordinates and the posture angle of the nut 202 at the second time as output elements.
  • the time interval between the first time and the second time is, for example, the first predetermined period.
  • the target position coordinates and the target posture angle of the nut 202 mean, for example, the position coordinates and the posture angle of the nut 202, which indicate the state in which the female thread portion of the nut 202 and the male thread portion 201a of the bolt 201 to be fastened are accurately meshed with each other and the fastening is completed.
  • the first control unit 361 controls the joint angles of the motors 117 of the joints 110 as the controlled device S and the fastening motor 120a of the end effector 12 based on the command values output by inputting the above data set into the first trained model 391 stored in the storage unit 39, thereby controlling the position coordinates and attitude angle of the nut 202.
  • the first trained model 391 is, for example, a deep learning model (supervised learning model) such as a deep neural network (DNN).
  • DNN deep neural network
  • a deep learning model such as a convolutional neural network (CNN) or a regression neural network (RNN) may be used for the first trained model 391.
  • the first control unit 361 sequentially stores the joint angles of the joints 110 that have changed over time, the rotation speed of the fastening motor 120a of the end effector 12 (the screw feed speed associated with the rotation of the nut 202), and the like in the storage unit 39 at every time.
  • the second control unit 362 controls the movement and rotation of the end effector 12 based on a data set including the image received from the acquisition unit 31, the attitude angle of the nut 202 received from the calculation unit 32, and the position coordinates of the nut 202 received from the coordinate conversion unit 33. Specifically, the second control unit 362 controls the movement and rotation of the end effector 12 using the trained model to position the nut 202 located at a first position where the nut 202 is not in contact with the bolt 201 to a second position where the nut 202 is in contact with the bolt 201. That is, the second control unit 362 makes the nut 202 located at the first position approach the bolt 201 to be worked from the outside.
  • the trained model used by the second control unit 362 to control the end effector 12 is referred to as the "second trained model 392".
  • the second trained model 392 is stored in advance in the storage unit 39, for example.
  • the second trained model 392 is trained to output a command value indicating the position coordinates and attitude angle of the nut 202 at a fourth time, which is after the third time and when the nut 202 approaches the second position, when a data set including an image at the third time acquired at the first position and the position coordinates and attitude angle of the nut 202 at the third time is input. That is, the second trained model 392 uses an image at the third time acquired at the first position and a data set including the position coordinates and attitude angle of the nut 202 at the third time as input elements, and uses a command value indicating the position coordinates and attitude angle of the nut 202 at the fourth time as output elements.
  • the time interval between the third time and the fourth time is, for example, the second predetermined period.
  • the second control unit 362 controls the joint angle of the motor 117 of each joint unit 110 as the controlled device S based on the command value output by inputting the above data set into the second trained model 392 stored in the memory unit 39, to bring the nut 202 closer to (approach) the bolt 201.
  • the second trained model 392 is, for example, a deep learning model (supervised learning model) such as a deep neural network (DNN).
  • DNN deep neural network
  • a deep learning model such as a convolutional neural network (CNN) or a recurrent neural network (RNN) may be used as the second trained model 392.
  • the "predetermined time in the past” here means, for example, the time when the control of the end effector 12 was started (the time when the fastening operation steps S106, S4, etc. described later were started).
  • the third control unit 363 controls the movement and rotation of the end effector 12 in reverse order while timely referring to the time-series data of the joint angles of each joint unit 110 stored in the memory unit 39, the rotational speed of the fastening motor 120a of the end effector 12 (the screw feed speed accompanying the rotation of the nut 202), etc.
  • the learning unit 37 modifies the weight values of the parameters in the first trained model 391 based on various data stored in the memory unit 39.
  • the learning unit 37 modifies (learns) the weight values of the parameters in the first trained model 391, for example, by using an error backpropagation method or the like.
  • the learning unit 37 modifies the parameters of the first trained model 391, it sends a modification flag indicating that the modification has been made to the counter unit 38.
  • the modification flag is, for example, 1.
  • the counter unit 38 adds each time it receives a correction flag from the learning unit 37, and sends (returns) the addition result of the correction flag to the sixth judgment unit 356.
  • the sixth judgment unit 356 compares the addition result of the correction flag received from the counter unit 38 with a predetermined value. If the addition result of the correction flag is equal to or greater than the predetermined value, the sixth judgment unit 356 determines that the predetermined number of corrections has been made. On the other hand, if the addition result of the correction flag is less than the predetermined value, the sixth judgment unit 356 determines that the predetermined number of corrections has not been made.
  • the predetermined value is, for example, stored in advance in the memory unit 39, and is an integer. The sixth judgment unit 356 makes the sixth judgment by referring to the predetermined value stored in the memory unit 39 at the appropriate time.
  • Fig. 11 is a flowchart showing an example of the learning steps of the first trained model 391.
  • the model before learning will be referred to as the "trained model” without distinction.
  • the acquisition unit 31 acquires data on the bolt 201 to be installed immediately from the CAD data 395 pre-stored in the memory unit 39, etc. (step S101).
  • the acquisition unit 31 acquires the known position coordinates and known attitude angle of the nut 202 (step S102).
  • the "known position coordinates and known attitude angle" of the nut 202 acquired by the acquisition unit 31 here is acquired, for example, from a variation generation model 397 (see FIG. 7) pre-stored in the memory unit 39.
  • the variation generation model 397 When referenced by the acquisition unit 31, the variation generation model 397 varies any position coordinates (second position) and attitude angle on the end face 201e of the bolt 201 according to a probability distribution such as a normal distribution (a state in which errors are intentionally given randomly), and outputs them to the acquisition unit 31.
  • the position coordinates that the variation generation model 397 averages are, for example, the central position coordinates of the end face 201e of the bolt 201
  • the attitude angle that the variation generation model 397 averages are, for example, the attitude angle when the rotation axis Ar and the center line Ar2 of the bolt 201 coincide in a straight line.
  • the central position coordinates of the end face 201e of the bolt 201 are expressed, for example, in the bolt-center coordinate system (Xb, Yb, Zb). That is, the acquisition unit 31 acquires the known position coordinates and known attitude angle of the nut 202 by referring to the variation generation model 397.
  • the second control unit 362 uses the second trained model 392 to make the nut 202 approach the target bolt 201 (step S103). Note that, in step S103, the second trained model 392 does not need to be used to make the nut 202 approach the bolt 201.
  • the control unit 36 may cause the nut 202 to approach the target bolt 201 according to, for example, a predetermined approach operation stored in advance in the storage unit 39.
  • the first determination unit 351 performs a first determination to determine whether the nut 202 has contacted the bolt 201 (step S104). If the result of the first determination indicates that the nut 202 has contacted the bolt 201 (step S104: YES), the position coordinates and attitude angle of the nut 202 are corrected to the known position coordinates and known attitude angle of the nut 202 acquired in step S102 under the control of the control unit 36 or manually by the operator (step S105). On the other hand, if the result of the first determination indicates that the nut 202 has not contacted the bolt 201 (step S104: NO), the process returns to step S103. Following step S105, a fastening operation step S106 is executed.
  • the control unit 36 starts the fastening operation of the nut 202 according to a predetermined procedure.
  • the fastening operation of the nut 202 according to the predetermined procedure is called a "teaching operation".
  • the "predetermined procedure” here means data showing command values indicating the position coordinates and attitude angle of the nut 202 in a time series for converting the known position coordinates (second position) and known attitude angle of the nut 202 to the target position coordinates and target attitude angle of the nut 202.
  • the data is stored in advance in the storage unit 39, for example.
  • the predetermined procedure may be, for example, correspondence information (such as a function) in which the known position coordinates and known attitude angle of the nut 202 at the start time of the fastening operation, multiple times after the start time of the fastening operation, and the command values of the position coordinates and attitude angles of the nut 202 corresponding to each time are associated with each other.
  • the control unit 36 controls the movement and rotation of the end effector 12 according to the predetermined procedure.
  • the acquisition unit 31, the calculation unit 32, the coordinate conversion unit 33, and the correction unit 34 sequentially store the results of the processing in the memory unit 39.
  • the results of the processing by the acquisition unit 31, the calculation unit 32, the coordinate conversion unit 33, and the correction unit 34 stored in the memory unit 39 during the teaching operation are used as a teacher data set used for learning the first trained model 391.
  • the control unit 36 finishes controlling the end effector 12 based on the final command value included in the predetermined procedure the fastening operation step S106 ends.
  • the fourth judgment unit 354 performs a fourth judgment to judge whether or not the fastening of the nut 202 was successful (step S107). If the result of the fourth judgment indicates that the fastening of the nut 202 was successful (step S107: YES), the sixth judgment unit 356 performs a judgment as to whether or not the nuts 202 have been fastened to all of the bolts 201 (step S108). On the other hand, if the result of the fourth judgment indicates that the fastening of the nut 202 was unsuccessful (step S107: NO), a retry operation step S111 is executed. In the retry operation step S111, the third control unit 363 performs the retry operation described above. When the retry operation is completed, the process returns to step S105.
  • step S108 if the result of the sixth judgment indicates that the nuts 202 have been fastened to all of the bolts 201 (step S108: YES), the learning unit 37 modifies the weighting value of the parameters of the first trained model 391 based on the various data acquired in the fastening operation step S106 (step S109). On the other hand, if the result of the sixth judgment indicates that the nuts 202 have not been fastened to all of the bolts 201 (step S108: NO), the process proceeds to step S101. Following the process of step S109, the sixth judgment unit 356 judges whether the learning unit 37 has modified the weighting value of the parameters of the first trained model 391 a predetermined number of times (step S110).
  • step S110: YES If the result of the judgment of the sixth judgment unit 356 indicates that the weighting value has been modified a predetermined number of times (step S110: YES), the learning step is completed. On the other hand, if the result of the judgment by the sixth judgment unit 356 indicates that the correction has not been performed the predetermined number of times (step S110: NO), the process returns to step S109.
  • the learning unit 37 repeatedly modifies the weighting values of the parameters based on various data acquired by the learning step that undergoes the series of processing operations described above, so that the first trained model 391 is taught to output a command value according to the input. That is, the first trained model 391 is trained to output a command value according to the input by repeatedly executing a learning step in which a teacher data set including the position coordinates and attitude angles of multiple nuts 202 acquired during a teaching operation that controls the end effector 12 so that the known position coordinates and known attitude angles of the nuts 202 that deviate from the target position coordinates and target attitude angle of the nuts 202 by a randomly given amount become the target position coordinates and target attitude angles, and the contact force and torque corresponding to the time when the position coordinates and attitude angles of each nut 202 were acquired, is input.
  • the second trained model 392 used by the second control unit 362 has been trained (reverse taught) to output a command value according to the input by repeatedly executing a learning step in which a teacher data set including a plurality of images acquired during a direct teaching operation to move the end effector 12 from the first position to the second position and the position coordinates and attitude angle of the nut 202 corresponding to the time when each image was acquired is input.
  • the reverse teaching in this embodiment is a conversion so that the time series of the image acquired by the acquisition unit 31 when the end effector 12 is moved from the second position to the first position (direct teaching operation) and the time series of the data set of the position coordinates and attitude angle of the nut 202 are reversed, and the data set converted into the reverse time series is trained by the second trained model 392 as the teacher data set.
  • the "direct teaching operation” here means an operation of moving the end effector 12 so that the nut 202 moves from the second position to the first position by the operator's manual operation, as shown as an example in FIG. 12.
  • the above learning step for learning (reverse teaching) the second trained model 392 is repeated, for example, tens to hundreds of times.
  • the second control unit 362 uses the second trained model 392 to make the nut 202 approach the bolt 201 to be installed (step S1).
  • the second judgment unit 352 performs a second judgment to judge whether or not there is an abnormality in the state of the nut 202 (step S2). If the result of the second judgment indicates that there is an abnormality in the state of the nut 202 (step S2: YES), for example, the control unit 36 operates to move the nut 202 away from the bolt 201 once, and then the process returns to step S1.
  • step S3 the first judgment unit 351 performs a first judgment to judge whether or not the nut 202 has come into contact with the bolt 201 (step S3). If the result of the first judgment indicates that the nut 202 is not in contact with the bolt 201 (step S3: NO), for example, the control unit 36 operates to move the nut 202 away from the bolt 201 once, and then the process returns to step S1. On the other hand, if the result of the first judgment indicates that the nut 202 has contacted the bolt 201 (step S3: YES), the fastening operation step S4 is executed.
  • the first control unit 361 executes the fastening operation of the nut 202 on the target bolt 201 by controlling the movement and rotation of the end effector 12 using the first learned model 391.
  • the third judgment unit 353 may perform a third judgment to determine whether or not the screwing of the nut 202 has been successful. If the result of the third judgment indicates that the screwing of the nut 202 has been successful, the first control unit 361 continues to execute the fastening operation. On the other hand, if the result of the third judgment indicates that the screwing of the nut 202 has failed, the first control unit 361 stops the fastening operation. If the first control unit 361 stops the fastening operation, for example, the retry operation step S8 is executed.
  • the fourth judgment unit 354 performs a fourth judgment to judge whether or not the fastening of the nut 202 was successful (step S5).
  • the fifth judgment unit 355 judges whether or not the collar 202b was broken and cut (step S6).
  • the retry operation step S8 is executed. In the retry operation step S8, the third control unit 363 performs the above-mentioned retry operation.
  • step S9 the sixth judgment unit 356 performs a sixth judgment to judge whether or not the retry has been performed a predetermined number of times (step S9).
  • step S9: YES the result of the sixth judgment indicates that the retry has been performed a predetermined number of times
  • step S9: NO the control device 30 ends the operation.
  • step S9: NO the result of the sixth judgment indicates that the retry has not been performed a predetermined number of times
  • step S9: NO the process proceeds to step S1.
  • step S6 if the result of the fifth judgment indicates that the collar 202b has not been cut (step S6: NO), the process returns to the fastening operation step S4.
  • step S7 judges whether or not the nuts 202 have been fastened to all of the bolts 201 (step S7). If the result of the sixth judgment indicates that the nuts 202 have not been fastened to all of the bolts 201 (step S7: NO), the process proceeds to step S1. On the other hand, if the result of the sixth judgment indicates that the nuts 202 have been fastened to all of the bolts 201 (step S7: YES), for example, the control device 30 ends its operation.
  • control device 30 is repeatedly performed during the assembly process of the construction object 200.
  • the movement and rotation of the end effector 12 are controlled based on the command values of the position coordinates and attitude angle of the nut 202 at the second time output by the first trained model 391, which receives as input the position coordinates and attitude angle of the nut 202 based on the joint angles of each joint 110 at the first time and the contact force and torque transmitted from the end effector 12 at the first time.
  • the position coordinates and attitude angle of the nut 202 of the command values at the second time output by the first trained model 391 are approaching the target position coordinates and target attitude angle of the nut 202 indicating the state in which the nut 202 is fastened to the bolt 201.
  • the state of the nut 202 can be transitioned toward the state in which the nut 202 is fastened to the bolt 201. Therefore, the nut 202 can be more smoothly fastened to the bolt 201 provided on the construction target 200.
  • the first trained model 391 is trained in advance so that the above input and output are performed by repeatedly executing instructions to fasten the nut 202, which indicates known position coordinates and known attitude angles that are previously determined and deviate from the target position coordinates and target attitude angle, to the bolt 201. Therefore, for example, even if a deviation in the position coordinates or attitude angle of the nut 202 relative to the bolt 201 occurs when fastening the nut 202 and bolt 201 during actual operation, it is possible to prevent the time actually required for fastening from deviating from the predicted time required for fastening. As a result, for example, it is possible to reduce the man-hours required for assembly.
  • the movement and rotation of the end effector 12 are controlled based on a command value indicating the position coordinates and attitude angle of the nut 202 at the fourth time output by the second trained model 392, which receives as input an image of the construction target 200 at the third time acquired at the first position where the nut 202 is not in contact with the bolt 201 and the position coordinates and attitude angle of the nut 202 at the third time.
  • the position coordinates and attitude angle of the nut 202 of the command value at the fourth time output by the second trained model 392 are approaching the position coordinates and attitude angle of the second position where the nut 202 is in contact with the bolt 201.
  • the nut 202 located at the first position can be transitioned toward the second position. Therefore, the nut 202 can be made to approach the bolt 201 provided on the construction target 200 more smoothly.
  • the above action can be realized with higher accuracy compared to, for example, a case where the image is not used as an input element.
  • the second trained model 392 is trained in advance (reverse taught) so that the above input and output are performed by repeatedly executing a learning step in which a teacher data set including a plurality of images acquired during a direct teaching operation to move the end effector 12 from a first position to a second position and the position coordinates and attitude angle of the nut 202 corresponding to the time when each image was acquired is input.
  • the inventors have found that when teaching the second trained model 392 using a direct teaching operation to manually move the end effector 12 by an operator using the assembly system 100, the operation of moving the end effector 12 so that the position of the nut 202 moves from the second position to the first position is easier than the operation of moving the end effector 12 so that the position of the nut 202 moves from the first position to the second position, and that shaking of the end effector 12 during operation is less likely to occur. As a result, a higher quality teacher data set with a lower proportion of noise can be acquired.
  • the end effector 12 is controlled using a third trained model 393 that has been trained to output a determination result indicating whether the nut 202 and bolt 201 have been screwed together.
  • a third trained model 393 that has been trained to output a determination result indicating whether the nut 202 and bolt 201 have been screwed together.
  • the coordinate system in which the position coordinates of the nut 202 are expressed is converted from the world coordinate system to the bolt-centered coordinate system.
  • the position coordinates of the nut 202 expressed in the world coordinate system are unique position coordinates that arise for each bolt 201
  • the relative position coordinates of the nut 202 with respect to the bolt 201 can be expressed on the same scale, such as the bolt 201-centered coordinates.
  • there is no need to teach each bolt 201 which increases generalizability and reduces the effort and time required to teach the first trained model 391.
  • the acquired contact force is corrected using a correction value related to the weight of the end effector 12, which is based on the joint angle of the joint unit 110. This allows a contact force that is closer to the actual contact force than the contact force transmitted from the end effector 12 to be used as an input element for the first trained model 391, thereby improving generalizability.
  • the elastic member 128 applies an elastic force toward the tip side of the socket portion 123 to the first portion 126, so that the claw portion 127b of the nut holding member 124 grips the nut 202 whose collar 202b is accommodated in the accommodation portion 123r of the nut runner 121.
  • This allows the end effector 12 to hold the nut 202 with a simpler configuration than, for example, an end effector 12 equipped with a mechanism for suctioning and holding the nut 202.
  • the imaging device 20 is disposed in the socket portion 123 of the nut runner 121, and the imaging device 20 irradiates light toward the construction target 200 through the nut hole 202h and generates an image of the construction target 200.
  • an image can be generated while intensively illuminating the imaging range of the imaging device 20. Therefore, it is less susceptible to the influence of the lighting environment at the construction site.
  • the assembly system 100 does not include a force sensor 140, and instead includes a current sensor 141.
  • the force data described in the first embodiment corresponds to the current value acquired by the current sensor 141, and the above-mentioned force data may be read by appropriately replacing it with the current value.
  • the current sensor 141 is a sensor capable of acquiring a current value of the motor 117 (first motor 117a to sixth motor 117f) of each joint 110 (first joint 111 to sixth joint 116).
  • the current sensor 141 is connected to the motor 117 of each joint 110 by wired or wireless communication.
  • the current sensor 141 is an example of the second sensor 14.
  • the current sensor 141 transmits the acquired current value of the motor 117 to the control device 30.
  • the current sensor 141 may be provided in the manipulator 10 or in the control device 30, for example.
  • the acquisition unit 31 acquires the current value of the current flowing through each motor 117 from the current sensor 141 at a first predetermined period, sends the acquired current value to the judgment unit 35 and the control unit 36, and stores it in the storage unit 39.
  • the first judgment unit 351 performs a first judgment based on the current value received from the acquisition unit 31.
  • the third judgment unit 353 performs a third judgment based on the image received from the acquisition unit 31, the attitude angle of the nut 202 received from the calculation unit 32, the position coordinates of the nut 202 received from the coordinate conversion unit 33, and the current value received from the acquisition unit 31.
  • the third trained model 393 is trained to output a judgment result indicating whether the nut 202 and the bolt 201 are screwed together when a data set including the image, the position coordinates and attitude angle of the nut 202, and the current value is input.
  • the fourth determination unit 354 performs a fourth determination based on the image received from the acquisition unit 31, the attitude angle of the nut 202 received from the calculation unit 32, the position coordinates of the nut 202 received from the coordinate conversion unit 33, and the current value received from the acquisition unit 31.
  • the fourth trained model 394 is trained to output a determination result indicating whether the nut 202 and the bolt 201 have been fastened, when a data set including the image, the position coordinates and attitude angle of the nut 202, and the current value is input.
  • the fifth determination unit 355 determines whether the collar 202b of the nut 202 has been separated from the nut body 202a, based on the current value received from the acquisition unit 31.
  • the first control unit 361 executes a fastening operation of the nut 202 on the bolt 201 based on a data set including the attitude angle of the nut 202 received from the calculation unit 32, the position coordinates of the nut 202 received from the coordinate conversion unit 33, and the current value received from the acquisition unit 31.
  • the first trained model 391 is trained to output a command value indicating the position coordinates and attitude angle of the nut 202 at a second time after the first time that approaches the target position coordinates and target attitude angle of the nut 202 that indicate a state in which the nut 202 is fastened to the bolt 201.
  • the current value of the motor 117 of each joint 110 acquired by the current sensor 141 can be substituted for the force data described in the first embodiment. That is, the current value of each motor 117 indicates the contact force and torque transmitted from the end effector 12 when the nut 202 contacts the bolt 201.
  • the assembly system 100 further includes a moving mechanism 16 and a calibration mechanism 17.
  • the moving mechanism 16 allows the base 15 to move around the construction target 200.
  • the moving mechanism 16 is a plurality of wheels 160 provided on the base 15.
  • the moving mechanism 16 can travel around the work platform 300.
  • FIG. 15 shows an example in which four wheels 160 are provided on the base 15.
  • the wheels 160 are provided on the base 15 so as to be rotatable while in contact with the floor surface.
  • the driving (rotation) of the wheels 160 may be controlled by the control device 30, or may be manually driven by an operator using the assembly system 100.
  • the calibration mechanism 17 determines the relative position between the construction target 200 and the moving mechanism 16. Specifically, the calibration mechanism 17, for example, matches (calibrates) an arbitrary point on the construction target 200 expressed in a world coordinate system with an arbitrary point on the world coordinate system based on the origin of the manipulator 10. In other words, the calibration mechanism 17 calibrates the position of the moving mechanism 16 with respect to the construction target 200.
  • the calibration mechanism 17 is provided in, for example, the manipulator 10.
  • the base 15 is provided with wheels 160 that enable the base 15 to move around the construction target 200, so that the manipulator 10 can operate over a wider range. Therefore, for example, when constructing a large construction target 200, it is unlikely that the arrangement or posture of the construction target 200 needs to be changed.
  • the moving mechanism 16 is a linear-acting electric slider 161 arranged between the base 15 and the floor surface.
  • the electric slider 161 extends long in one direction.
  • the electric slider 161 is fixed on the floor surface adjacent to the workbench 300, for example.
  • the electric slider 161 is connected to the base 15 while supporting the base 15 from below, and moves the base 15 in the one direction.
  • the driving of the electric slider 161 (linear movement of the base 15) may be controlled by the control device 30 or may be manually performed by an operator.
  • the electric slider 161 described in this modification may be arranged separately from the floor surface instead of being fixed on the floor surface, and may be mounted on a moving mechanism 16 different from the electric slider 161, for example.
  • the first control unit 361 described in the first embodiment may receive an image from the acquisition unit 31.
  • the first trained model 391 used for control by the first control unit 361 may add the received image to the above-mentioned input elements.
  • the first trained model 391 may generate and output, as a subtask, a predicted image at the second time, in addition to the above-mentioned command value, which is the output for the input.
  • the first trained model 391 described in the first embodiment may be trained to generate and output, as a subtask, a predicted value of the contact force and a predicted value of the torque at a second time in addition to the above-mentioned command value, which is an output for an input.
  • the predicted value of the contact force and the predicted value of the torque are values indicating the contact force and the torque predicted by the first trained model 391 when acquired by the acquisition unit 31 at the second time. These predicted values of the contact force and the torque are not used for control by the first control unit 361.
  • the inventors have found that by having the first trained model 391 output the predicted value of the contact force and the predicted value of the torque at the second time as a subtask, the accuracy of the command value, which is another output element, is improved.
  • the second trained model 392 used for control by the second control unit 362 described in the first embodiment may be trained to generate and output, as a subtask, a predicted image at a fourth time, in addition to the above-mentioned command value, which is an output for an input.
  • the predicted image is an image predicted by the second trained model 392 to be acquired by the acquisition unit 31 at the fourth time. This predicted image is not used for control by the second control unit 362.
  • the inventors have found that by having the second trained model 392 output the predicted value of the contact force and the predicted value of the torque at the fourth time as a subtask, the accuracy of the command value, which is the other output element, is improved.
  • each of the first trained model 391, the second trained model 392, the third trained model 393, the fourth trained model 394, and the fifth trained model 396 described in the first embodiment is not limited to a deep learning model, and may be another machine learning model (e.g., a supervised learning model such as an SVM (Support Vector Machine)).
  • a supervised learning model such as an SVM (Support Vector Machine)
  • the control device 30 described in the first embodiment may not include the coordinate conversion unit 33 and the correction unit 34.
  • the acquisition unit 31 may send the acquired force data to the determination unit 35 and the control unit 36.
  • the acquisition unit 31 may send an acquisition flag indicating that the center position coordinates of the bolt 201 that is the target of the most recent construction have been acquired to the counter unit 38.
  • the calculation unit 32 may send the calculated position coordinates and attitude angle of the nut 202 to the determination unit 35 and the control unit 36.
  • the first judgment unit 351 makes a first judgment based on the force data received from the acquisition unit 31
  • the second judgment unit 352 makes a second judgment based on the position coordinates and attitude angle of the nut 202 received from the calculation unit 32
  • the third judgment unit 353 makes a third judgment based on the image received from the acquisition unit 31, the position coordinates and attitude angle of the nut 202 received from the calculation unit 32
  • the force data received from the acquisition unit 31
  • the fourth judgment unit 354 makes a fourth judgment based on the image received from the acquisition unit 31, the position coordinates and attitude angle of the nut 202 received from the calculation unit 32, and the force data received from the acquisition unit 31,
  • the fifth judgment unit 355 makes a fifth judgment based on the force data received from the acquisition unit 31 when the result of the fourth judgment received from the fourth judgment unit 354 indicates that the tightening of the nut 202 has been successful.
  • the first control unit 361 executes the tightening operation of the nut 202 on the bolt 201 based on a data set including the position coordinates and attitude angle of the nut 202 received from the calculation unit 32 and the force data received from the acquisition unit 31, and the second control unit 362 controls the movement and rotation of the end effector 12 based on a data set including the image received from the acquisition unit 31 and the position coordinates and attitude angle of the nut 202 received from the calculation unit 32.
  • the control device 30 described in the second embodiment may further include a current value conversion unit (not shown) that receives a current value from the acquisition unit 31 and converts the received current value into force data including a contact force and a torque.
  • the current value conversion unit may send force data including a contact force and a torque converted from the current value to the correction unit 34, for example.
  • the current value conversion unit may convert the current value into force data based on correspondence information (such as a function) that associates the current value of the current flowing through each motor 117 (first motor 117a to sixth motor 117f) with the contact force and the torque.
  • correspondence information is stored in advance in, for example, the storage unit 39, and the current value conversion unit may perform the above conversion by referring to the correspondence information stored in the storage unit 39 at appropriate times.
  • the first judgment unit 351 may further use the image received from the acquisition unit 31. That is, the first judgment unit 351 may make the first judgment based on the image and the force data. In this case, the first judgment unit 351 makes the judgment using a predetermined trained model that has been trained to output the result of the first judgment when a data set including the image and the force data is input. That is, the trained model has the data set including the image and the force data as input elements and the result of the first judgment as output elements.
  • the trained model is stored in advance in the storage unit 39, for example, and the first judgment unit 351 may obtain the result of the first judgment output by inputting the above data set into the trained model stored in the storage unit 39.
  • the second judgment unit 352 may further use the image received from the acquisition unit 31. That is, the second judgment unit 352 may make the second judgment based on the image and the position coordinates and attitude angle of the nut 202. In this case, the second judgment unit 352 makes the judgment using a predetermined trained model that has been trained to output the result of the second judgment when a data set including the image and the position coordinates and attitude angle of the nut 202 is input. That is, the trained model uses the image and the data set including the position coordinates and attitude angle of the nut 202 as input elements, and the result of the second judgment as output elements.
  • the trained model is stored in advance in the storage unit 39, for example, and the second judgment unit 352 may acquire the result of the second judgment output by inputting the above data set into the trained model stored in the storage unit 39.
  • the third determination unit 353 does not need to use the image received from the acquisition unit 31 when determining whether the nut 202 is screwed in.
  • the third trained model 393 uses a data set including the position coordinates and attitude angle of the nut 202 and the force data as input elements, and uses a determination result indicating whether the nut 202 and the bolt 201 are screwed in as output elements.
  • the fifth determination unit 355 may also perform a determination regarding cutting, based on the force data received from the correction unit 34, of whether the third threshold value indicating the torque required for cutting has been exceeded, and a determination regarding the drive of the end effector 12, of whether the fastening motor 120a has changed from a stopped state to a rotating state, before performing the fifth determination.
  • the fifth determination unit 355 may perform the fifth determination when the determination regarding cutting indicates that the data (contact force) has exceeded the third threshold value and the determination regarding the drive of the end effector 12 indicates that the fastening motor 120a is in a rotating state.
  • the third threshold value is stored in advance in, for example, the storage unit 39, and the fifth determination unit 355 performs the determination regarding cutting by referring to the third threshold value stored in the storage unit 39 as appropriate.
  • the computer 1100 includes, for example, a processor 1110, a main memory 1120, a storage 1130, and an interface 1140.
  • the above-mentioned control device 30 is implemented in, for example, one or more computers 1100.
  • the operation of each of the above-mentioned processing units is stored in the storage 1130 in the form of a program.
  • the processor 1110 reads the program from the storage 1130, expands it in the main memory 1120, and executes the above-mentioned processing according to the program.
  • the processor 1110 also secures a memory area in the main memory 1120 corresponding to the above-mentioned memory unit 39 according to the program.
  • the program may be for realizing part of the function to be performed by the computer 1100.
  • the program may be for performing a function by combining it with other programs already stored in the storage 1130 or by combining it with other programs implemented in other devices.
  • the computer 1100 may also be provided with a custom LSI (Large Scale Integrated Circuit) such as a PLD (Programmable Logic Device) in addition to or instead of the above configuration.
  • PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Array).
  • PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Array).
  • PLDs Programmable Logic Device
  • GAL Generic Array Logic
  • CPLD Complex Programmable Logic Device
  • FPGA Field Programmable Gate Array
  • storage 1130 include a magnetic disk, a magneto-optical disk, and a semiconductor memory. Storage 1130 may be an internal medium directly connected to the bus of computer 1100, or an external medium connected to computer 1100 via interface 1140 or
  • computer 1100 that receives the program may expand the program in main memory 1120 and execute the above-mentioned processing.
  • storage 1130 is a non-transitory tangible storage medium.
  • the program may be for realizing some of the above-mentioned functions.
  • the program may be a so-called differential file (differential program) that realizes the above-mentioned functions in combination with other programs already stored in storage 1130.
  • the assembly system 100 includes a manipulator 10 having an arm 11 having a plurality of joints 110, an end effector 12 that is moved by the arm 11 and holds a nut 202 to rotate the nut 202 around an axis (rotation axis Ar or axis Ar1), a first sensor 13 (encoder 130) that acquires the joint angle of each of the joints 110, and a second sensor 14 (force sensor 140, current sensor 141) that acquires the contact force and torque transmitted from the end effector 12 when the nut 202 comes into contact with a bolt 201 provided on a construction object 200, and a control device 30 that controls the manipulator 10, and the control device 30 controls each of the related
  • the system includes an acquisition unit 31 that acquires the joint angle of the joint portion 110 and the contact force and torque at a first predetermined period, a calculation unit 32 that calculates the position coordinates and attitude angle of the nut 202 based on the acquired joint angles of each of the joint portions 110, and a control unit 36 that controls the end effector 12
  • the assembly system 100 according to the second aspect is the assembly system 100 according to (1), and the first trained model 391 may be trained to output the command value according to the input by repeatedly executing a learning step in which a teacher data set including position coordinates and attitude angles of the nuts 202 acquired during a teaching operation for controlling the end effector 12 so that known position coordinates and known attitude angles of the nuts 202 that deviate from the target position coordinates and target attitude angles of the nuts 202 by a randomly given amount become the target position coordinates and target attitude angles, and the contact force and torque corresponding to the time when the position coordinates and attitude angles of each of the nuts 202 were acquired, are input.
  • the assembly system 100 according to the third aspect is the assembly system 100 according to (1), further comprising an imaging device 20 provided on the end effector 12 and configured to generate an image of the construction target 200, the acquisition unit 31 further acquires the image at a second predetermined period, and the control unit 36 may control the end effector 12 using a second trained model 392 trained to output command values indicating the position coordinates and attitude angle of the nut 202 at a fourth time after the third time when the nut 202 approaches the second position where the nut 202 contacts the bolt 201 when the image acquired at a third time when the nut 202 is not in contact with the bolt 201 and the position coordinates and attitude angle of the nut 202 at the third time are input.
  • the assembly system 100 according to the fourth aspect is the assembly system 100 according to (3), in which the second trained model 392 may be trained to output the command value according to the input by repeatedly executing a learning step in which a teacher data set including a plurality of images acquired during a direct teaching operation that moves the end effector 12 from the second position to the first position and the position coordinates and attitude angle of the nut 202 corresponding to the time when each of the images was acquired is input.
  • the assembly system 100 according to the fifth aspect is the assembly system 100 according to (3) or (4), and the control unit 36 may control the end effector 12 using a third trained model 393 trained to output a judgment result indicating whether the nut 202 and the bolt 201 are screwed together when the image acquired at the second position, the position coordinates and attitude angle of the nut 202 corresponding to the time when the image was acquired, and the contact force and torque corresponding to the time when the image was acquired are input.
  • the assembly system 100 according to the sixth aspect is any one of the assembly systems 100 of (1) to (4), in which the position coordinates of the nut 202 are expressed in a world coordinate system based on the origin of the manipulator 10, the control device 30 further includes a coordinate conversion unit 33 that converts the calculated position coordinates of the nut 202 from the world coordinate system to a bolt-centered coordinate system, and the control unit 36 may input the converted position coordinates of the nut 202 to the first trained model 391.
  • the position coordinates of the nut 202 expressed in the world coordinate system are unique position coordinates that arise for each bolt 201
  • the relative position coordinates of the nut 202 with respect to the bolt 201 can be expressed on the same scale, such as the center coordinates of the bolt 201.
  • the assembly system 100 according to the seventh aspect is any one of the assembly systems 100 of (1) to (4), and the control device 30 further includes a correction unit 34 that corrects the acquired contact force using a correction value related to the weight of the end effector 12 based on the acquired joint angles of each of the joint units 110, and the control unit 36 may input the corrected contact force to the first trained model 391.
  • the assembly system 100 according to the eighth aspect is any one of the assembly systems 100 of (1) to (4), and the second sensor 14 may be a current sensor 141 capable of acquiring the current value of the motor 117 of each of the joints 110.
  • the assembly system 100 may be any one of the assembly systems 100 of (1) to (4), and may further include a base 15 on which the manipulator 10 is placed, and a movement mechanism 16 (wheels 160, electric slider 161) that enables the base 15 to move around the construction target 200.
  • a movement mechanism 16 wheels 160, electric slider 161 that enables the base 15 to move around the construction target 200.
  • the assembly system 100 according to the tenth aspect is the assembly system 100 according to (3) or (4), and the end effector 12 includes a nut runner 121 having a base 122, a socket portion 123 extending from the base 122 and having a cylindrical shape centered on the axis, the socket portion 123 having an accommodating portion 123r formed at the tip end capable of accommodating at least a portion of the nut 202, and a second nut runner 121 fixed to the outer periphery of the socket portion 123 and extending in a direction intersecting the axis.
  • the nut holding member 124 may have a first part 126, an extension part 127a extending from the first part 126 toward the tip side, and a second part 127 having a claw part 127b provided at the tip of the extension part 127a and protruding from the extension part 127a so as to approach the axis and abutting against the nut 202, and an elastic member 128 provided between the base 122 and the first part 126 and applying an elastic force to the first part 126 toward the tip side.
  • the assembly system 100 according to the eleventh aspect is the assembly system 100 according to (10), in which the imaging device 20 is disposed in the socket portion 123 closer to the base portion 122 than the storage portion 123r, and the imaging device 20 may have an imaging unit 21 that generates the image of the construction object 200 captured through a hole (nut hole 202h) of the nut 202, and an illumination unit 22 that irradiates light toward the construction object 200 through the hole (nut hole 202h) of the nut 202.
  • a control method is a control method for controlling a manipulator 10 having an arm 11 having a plurality of joints 110, an end effector 12 that is moved by the arm 11 and holds a nut 202 to rotate the nut 202 around an axis, a first sensor 13 that acquires joint angles of each of the joints 110, and a second sensor 14 that acquires a contact force and torque transmitted from the end effector 12 when the nut 202 contacts a bolt 201 provided on a construction target 200, the control method including: a switch that acquires the joint angle of each of the joints 110 and the contact force and torque at a first predetermined period; a step of calculating a position coordinate and an attitude angle of the nut 202 based on the acquired joint angles of each of the joint parts 110; and a step of controlling the end effector 12 using a first trained model 391 that has been trained to output command values indicating the position coordinate and attitude angle of the nut 202 at a second time after the first time when the position coordinate
  • the program relating to the thirteenth aspect includes an arm 11 having a plurality of joints 110, an end effector 12 that is moved by the arm 11 and holds a nut 202 to rotate the nut 202 around an axis, a first sensor 13 that acquires the joint angles of each of the joints 110, and a second sensor 14 that acquires the contact force and torque transmitted from the end effector 12 when the nut 202 contacts a bolt 201 provided on a construction object 200.
  • the program instructs a computer 1100 of a control device 30 that controls a manipulator 10 having the arm 11 having a plurality of joints 110, a first sensor 13 that acquires the joint angles of each of the joints 110, and the contact force and torque transmitted from the end effector 12 when the nut 202 contacts a bolt 201 provided on a construction object 200 to transmit the joint angles of each of the joints 110 and the contact force and torque at a first predetermined period.
  • the method executes the steps of: acquiring the position coordinates and the attitude angle of the nut 202 based on the acquired joint angles of each of the joint parts 110; calculating the position coordinates and the attitude angle of the nut 202 based on the acquired joint angles of each of the joint parts 110; and controlling the end effector 12 using a first trained model 391 that has been trained to output command values indicating the position coordinates and the attitude angle of the nut 202 at a second time after the first time when the position coordinates and the attitude angle of the nut 202 approach the target position coordinates and the target attitude angle of the nut 202 that indicate the state in which the nut 202 is fastened to the bolt 201 when the position coordinates and the attitude angle of the nut 202 at a first time and the contact force and the torque at the first time are input.
  • the present disclosure provides an assembly system, control method, and program that allows for smoother fastening.

Landscapes

  • Engineering & Computer Science (AREA)
  • Robotics (AREA)
  • Mechanical Engineering (AREA)
  • Human Computer Interaction (AREA)
  • Manipulator (AREA)

Abstract

This assembly system comprises: an end effector that is moved by an arm having a plurality of joints to hold and rotate a nut; a manipulator having a first sensor and a second sensor; and a control device. The control device calculates position coordinates and the attitude angle of the nut on the basis of a joint angle of each joint, and controls the end effector by using a trained model that has been trained to output a command value indicating the position coordinates and attitude angle of the nut at a first time and the position coordinates and attitude angle of the nut at a second time after the first time, which are close to the target position coordinates and target attitude angle indicating a state where the nut is fastened to a bolt.

Description

組立システム、制御方法、およびプログラムASSEMBLY SYSTEM, CONTROL METHOD, AND PROGRAM
　本開示は、組立システム、制御方法、およびプログラムに関する。
　本願は、２０２２年１２月１４日に日本に出願された特願２０２２－１９９６９８号について優先権を主張し、その内容をここに援用する。 The present disclosure relates to an assembly system, a control method, and a program.
This application claims priority to Japanese Patent Application No. 2022-199698, filed in Japan on December 14, 2022, the contents of which are incorporated herein by reference.
　特許文献１には、ロボットが備えるエンドエフェクタのワークに対する接触状態の変化をセンサが検知し、当該センサが検知した接触状態の変化に基づき、ワークに対するエンドエフェクタのずれ量を推定して補正するロボットシステムが開示されている。 Patent Document 1 discloses a robot system in which a sensor detects changes in the contact state of an end effector equipped to a robot with a workpiece, and based on the change in contact state detected by the sensor, estimates and corrects the amount of misalignment of the end effector with respect to the workpiece.
特許第６８７０１２２号公報Patent No. 6870122
　ところで、上記特許文献１に記載の技術では、エンドエフェクタの手探り動作によって生じたボルトの頭部に対するソケットの接触状態の変化に基づいてずれ量を補正しているに留まるため、人間が行うような締結動作の再現が難しい。そのため、例えば、ソケットにボルトの頭部がはまるまでに時間がかかる場合がある。したがって、より円滑に締結することができる組立技術が期待されている。 However, the technology described in the above Patent Document 1 only corrects the amount of misalignment based on changes in the contact state of the socket with the bolt head caused by the feel-and-see action of the end effector, making it difficult to reproduce the fastening action performed by a human. As a result, for example, it may take a long time for the bolt head to fit into the socket. Therefore, there is a demand for an assembly technology that allows for smoother fastening.
　本開示は、より円滑に締結することができる組立システム、制御方法、およびプログラムを提供する。 The present disclosure provides an assembly system, control method, and program that allows for smoother fastening.
　本開示に係る組立システムは、複数の関節部を有するアームと、前記アームによって移動されるとともに、ナットを保持して前記ナットを軸線回りに回転させるエンドエフェクタと、各々の前記関節部の関節角を取得する第１センサと、施工対象物に設けられたボルトに前記ナットが接触した場合に、前記エンドエフェクタから伝達された接触力およびトルクを取得する第２センサと、を有するマニピュレータと、前記マニピュレータを制御する制御装置と、を備え、前記制御装置は、各々の前記関節部の関節角と、前記接触力およびトルクとを第１所定周期で取得する取得部と、前記取得された各々の前記関節部の関節角に基づき、前記ナットの位置座標および姿勢角を算出する算出部と、第１時刻における前記ナットの位置座標および姿勢角と、前記第１時刻における前記接触力およびトルクとが入力されると、前記ナットが前記ボルトに締結された状態を示す前記ナットの目標位置座標および目標姿勢角に近づいた、前記第１時刻よりも後の第２時刻における前記ナットの位置座標および姿勢角を示す指令値を出力するように学習された第１学習済みモデルを用いて、前記エンドエフェクタを制御する制御部と、を備える。 The assembly system according to the present disclosure includes a manipulator having an arm with multiple joints, an end effector that is moved by the arm and holds a nut to rotate the nut around an axis, a first sensor that acquires the joint angles of each of the joints, and a second sensor that acquires the contact force and torque transmitted from the end effector when the nut contacts a bolt provided on a construction object, and a control device that controls the manipulator. The control device includes an acquisition unit that acquires the joint angles of each of the joints and the contact force and torque at a first predetermined period, a calculation unit that calculates the position coordinates and attitude angle of the nut based on the acquired joint angles of each of the joints, and a control unit that controls the end effector using a first trained model that has been trained to output command values indicating the position coordinates and attitude angle of the nut at a second time after the first time when the position coordinates and attitude angle of the nut approach the target position coordinates and target attitude angle of the nut that indicate a state in which the nut is fastened to the bolt, when the position coordinates and attitude angle of the nut at a first time and the contact force and torque at the first time are input.
　本開示に係る制御方法は、複数の関節部を有するアームと、前記アームによって移動されるとともに、ナットを保持して前記ナットを軸線回りに回転させるエンドエフェクタと、各々の前記関節部の関節角を取得する第１センサと、前記エンドエフェクタから伝達された接触力およびトルクを取得する第２センサと、を有するマニピュレータを制御する制御方法であって、各々の前記関節部の関節角と、前記接触力およびトルクとを第１所定周期で取得するステップと、前記取得された各々の前記関節部の関節角に基づき、前記ナットの位置座標および姿勢角を算出するステップと、第１時刻における前記ナットの位置座標および姿勢角と、前記第１時刻における前記接触力およびトルクとが入力されると、前記ナットが前記ボルトに締結された状態を示す前記ナットの目標位置座標および目標姿勢角に近づいた、前記第１時刻よりも後の第２時刻における前記ナットの位置座標および姿勢角を示す指令値を出力するように学習された第１学習済みモデルを用いて、前記エンドエフェクタを制御するステップと、を実行する。 The control method according to the present disclosure is a method for controlling a manipulator having an arm with multiple joints, an end effector that is moved by the arm and holds a nut to rotate the nut around an axis, a first sensor that acquires the joint angle of each of the joints, and a second sensor that acquires the contact force and torque transmitted from the end effector, and includes the steps of acquiring the joint angle of each of the joints and the contact force and torque at a first predetermined period, calculating the position coordinates and attitude angle of the nut based on the acquired joint angles of each of the joints, and controlling the end effector using a first trained model that has been trained to output, when the position coordinates and attitude angle of the nut at a first time and the contact force and torque at the first time are input, command values indicating the position coordinates and attitude angle of the nut at a second time after the first time that approach the target position coordinates and target attitude angle of the nut that indicate a state in which the nut is fastened to the bolt.
　本開示に係るプログラムは、複数の関節部を有するアームと、前記アームによって移動されるとともに、ナットを保持して前記ナットを軸線回りに回転させるエンドエフェクタと、各々の前記関節部の関節角を取得する第１センサと、前記エンドエフェクタから伝達された接触力およびトルクを取得する第２センサと、を有するマニピュレータを制御する制御装置のコンピュータに、各々の前記関節部の関節角と、前記接触力およびトルクとを第１所定周期で取得するステップと、前記取得された各々の前記関節部の関節角に基づき、前記ナットの位置座標および姿勢角を算出するステップと、第１時刻における前記ナットの位置座標および姿勢角と、前記第１時刻における前記接触力およびトルクとが入力されると、前記ナットが前記ボルトに締結された状態を示す前記ナットの目標位置座標および目標姿勢角に近づいた、前記第１時刻よりも後の第２時刻における前記ナットの位置座標および姿勢角を示す指令値を出力するように学習された第１学習済みモデルを用いて、前記エンドエフェクタを制御するステップと、を実行させる。 The program according to the present disclosure causes a computer of a control device that controls a manipulator having an arm with multiple joints, an end effector that is moved by the arm and holds a nut and rotates the nut around an axis, a first sensor that acquires the joint angle of each of the joints, and a second sensor that acquires the contact force and torque transmitted from the end effector to execute the steps of acquiring the joint angle of each of the joints and the contact force and torque at a first predetermined period, calculating the position coordinates and attitude angle of the nut based on the acquired joint angles of each of the joints, and controlling the end effector using a first trained model that has been trained to output command values indicating the position coordinates and attitude angle of the nut at a second time after the first time when the position coordinates and attitude angle of the nut approach the target position coordinates and target attitude angle of the nut that indicate a state in which the nut is fastened to the bolt, when the position coordinates and attitude angle of the nut at a first time and the contact force and torque at the first time are input.
　本開示によれば、より円滑に締結することができる組立システム、制御方法、およびプログラムを提供することができる。 The present disclosure provides an assembly system, control method, and program that allows for smoother fastening.
本開示の第１実施形態に係る組立システムの全体を示す概略構成図である。1 is a schematic configuration diagram showing an entire assembly system according to a first embodiment of the present disclosure. 本開示の第１実施形態に係るエンドエフェクタ、ナット、およびボルトを一列に並べて表示した図である。FIG. 2 is a diagram showing an end effector, a nut, and a bolt arranged in a line according to a first embodiment of the present disclosure. 本開示の第１実施形態に係るナットランナの内部を軸線方向から見た図である。FIG. 2 is a view showing the inside of the nut runner according to the first embodiment of the present disclosure as viewed from the axial direction. 本開示の第１実施形態に係るエンドエフェクタにナットを保持させる際のナット保持部材の動きを説明するための図である。13A to 13C are diagrams for explaining the movement of a nut holding member when the end effector according to the first embodiment of the present disclosure holds a nut. 本開示の第１実施形態に係るエンドエフェクタからカラーを取り外す際のナット保持部材の動きを説明するための図である。13A to 13C are views for explaining the movement of the nut retaining member when removing the collar from the end effector according to the first embodiment of the present disclosure. 本開示の第１実施形態に係る撮像装置が撮像した画像の一例を模式的に示す図である。FIG. 2 is a diagram illustrating an example of an image captured by the imaging device according to the first embodiment of the present disclosure. 本開示の第１実施形態に係る制御装置の機能ブロック図である。FIG. 2 is a functional block diagram of a control device according to the first embodiment of the present disclosure. 本開示の第１実施形態に係るナットの位置座標が表現される座標系を概念的に説明するための図である。FIG. 2 is a diagram for conceptually explaining a coordinate system in which the position coordinates of a nut according to the first embodiment of the present disclosure are expressed. 本開示の第１実施形態に係るエンドエフェクタが保持したナットがボルトに接触した際にボルトから受ける接触力を概念的に説明するための図である。1A to 1C are diagrams for conceptually explaining a contact force received from a bolt when a nut held by the end effector according to the first embodiment of the present disclosure comes into contact with the bolt. 本開示の第１実施形態に係る補正部が接触力の補正に用いる補正値を概念的に説明するための図である。5 is a diagram for conceptually explaining a correction value used by a correction unit according to the first embodiment of the present disclosure to correct a contact force. FIG. 本開示の第１実施形態に係る第１学習済みモデルの学習ステップの一例を示すフローチャートである。1 is a flowchart illustrating an example of a learning step of a first trained model according to the first embodiment of the present disclosure. 本開示の第１実施形態に係る第２学習済みモデルを教示する際のダイレクトティーチング動作の一例を示す図である。A figure showing an example of a direct teaching operation when teaching a second learned model according to the first embodiment of the present disclosure. 本開示の第１実施形態に係る制御装置の動作の一例を示すフローチャートである。5 is a flowchart showing an example of an operation of the control device according to the first embodiment of the present disclosure. 本開示の第２実施形態に係る組立システムの全体を示す概略構成図である。FIG. 11 is a schematic configuration diagram showing an entire assembly system according to a second embodiment of the present disclosure. 本開示の第３実施形態に係る組立システムの全体を示す概略構成図である。FIG. 13 is a schematic configuration diagram showing an entire assembly system according to a third embodiment of the present disclosure. 本開示の第３実施形態の変形例に係る組立システムの全体を示す概略構成図である。FIG. 13 is a schematic configuration diagram showing an entire assembly system according to a modified example of the third embodiment of the present disclosure. 本開示の実施形態に係るコンピュータの構成を示すハードウェア構成図である。FIG. 2 is a hardware configuration diagram illustrating a configuration of a computer according to an embodiment of the present disclosure.
　以下、添付図面を参照しながら、組立システムを実施するための形態を説明する。 Below, we will explain how to implement the assembly system with reference to the attached drawings.
＜組立システムの第１実施形態＞
　組立システムは、例えば組立工場などで稼動する自動化システムの一部であり、マニピュレータを制御することで複数の部品が一体化するように組み付ける。以下、組立システムが組立の対象とする部品を「施工対象物」と称する。図１に示すように、施工対象物２００は、例えば第１部品２１０と第２部品２２０とに分かれており、例えば作業台３００（作業用机）の上に据え置かれている。組立システム１００は、マニピュレータ１０を制御することで、第１部品２１０および第２部品２２０を一体化させる。施工対象物２００の例として、航空機部品の１つであるパネルや、船舶部品の１つであるパネルなどが挙げられる。組立システム１００は、例えば、マニピュレータ１０と、制御装置３０とを備えている。 First embodiment of assembly system
The assembly system is a part of an automated system that operates in, for example, an assembly factory, and assembles a plurality of parts so as to be integrated by controlling a manipulator. Hereinafter, the parts to be assembled by the assembly system are referred to as "construction target". As shown in FIG. 1, the construction target 200 is divided into, for example, a first part 210 and a second part 220, and is placed on, for example, a workbench 300 (work desk). The assembly system 100 integrates the first part 210 and the second part 220 by controlling the manipulator 10. Examples of the construction target 200 include a panel, which is one of aircraft parts, and a panel, which is one of ship parts. The assembly system 100 includes, for example, the manipulator 10 and a control device 30.
（マニピュレータの構成）
　本実施形態では、マニピュレータ１０は、結合具としてのナット２０２を保持した状態で、全体の姿勢を変えることでナット２０２を移動させるとともに、当該ナット２０２を施工対象物２００に設けられた結合具としてのボルト２０１に自動で締結させるロボットである。マニピュレータ１０は、複数の関節部１１０を備える多関節ロボットである。本実施形態では、６つの関節部１１０を備える６軸の多関節ロボットをマニピュレータ１０の一例として説明する。マニピュレータ１０は、施工対象物２００に接着剤などで仮固定されたボルト２０１にナット２０２を締結することで、第１部品２１０および第２部品２２０を一体化させている。 (Manipulator configuration)
In this embodiment, the manipulator 10 is a robot that moves the nut 202 as a fastener by changing the overall posture while holding the nut 202, and automatically fastens the nut 202 to a bolt 201 as a fastener provided on the construction target 200. The manipulator 10 is a multi-joint robot having a plurality of joints 110. In this embodiment, a six-axis multi-joint robot having six joints 110 will be described as an example of the manipulator 10. The manipulator 10 fastens the nut 202 to the bolt 201 that is temporarily fixed to the construction target 200 with an adhesive or the like, thereby integrating the first part 210 and the second part 220.
　図２に示すように、本実施形態では、マニピュレータ１０が扱うナット２０２は、ナット本体２０２ａと、ナット本体２０２ａに一体に設けられたカラー２０２ｂとを有するカラーナットである。ナット２０２には、雌ねじ部としてのナット孔２０２ｈがナット本体２０２ａおよびカラー２０２ｂの両方に亘って形成されている。本実施形態では、カラー２０２ｂは、外殻が六角柱状に形成されている。ナット２０２は、マニピュレータ１０によってボルト２０１に締結される過程で、カラー２０２ｂのみが外され（カラー切り）、ナット本体２０２ａがボルト２０１に締結された状態となる。ボルト２０１は、施工対象物２００としての第１部品２１０および第２部品２２０のそれぞれに設けられた孔（図示省略）などに挿通されるとともに第２部品２２０から突出することで、雄ねじ部２０１ａがむき出しとされている。本実施形態では、ボルト２０１は、例えば頭部を有さないスタッドボルト（植え込みボルト）である。マニピュレータ１０は、むき出しとされたボルト２０１の雄ねじ部２０１ａにナット２０２のナット孔２０２ｈを螺合させて締結させる。図１に戻り、マニピュレータ１０は、例えば、アーム１１と、エンコーダ１３０と、力覚センサ１４０と、エンドエフェクタ１２と、台座部１５とを有している。 As shown in FIG. 2, in this embodiment, the nut 202 handled by the manipulator 10 is a collar nut having a nut body 202a and a collar 202b integrally formed with the nut body 202a. The nut 202 has a nut hole 202h as a female threaded portion formed through both the nut body 202a and the collar 202b. In this embodiment, the collar 202b has an outer shell formed in a hexagonal column shape. In the process of fastening the nut 202 to the bolt 201 by the manipulator 10, only the collar 202b is removed (collar cut), and the nut body 202a is fastened to the bolt 201. The bolt 201 is inserted through holes (not shown) provided in the first part 210 and the second part 220 as the construction object 200, and protrudes from the second part 220, exposing the male threaded portion 201a. In this embodiment, the bolt 201 is, for example, a stud bolt (implanted bolt) without a head. The manipulator 10 fastens the exposed male threaded portion 201a of the bolt 201 by screwing it into the nut hole 202h of the nut 202. Returning to FIG. 1, the manipulator 10 has, for example, an arm 11, an encoder 130, a force sensor 140, an end effector 12, and a base portion 15.
（アーム）
　アーム１１は、マニピュレータ１０の本体部分を成している。アーム１１は、地面に配置された台座部１５の上に載置されている。具体的には、アーム１１の一端は、台座部１５に固定されている。アーム１１は、台座部１５を基点に姿勢を変えることで台座部１５の周囲を３次元的に移動可能である。アーム１１は、例えば、複数（６つ）の関節部１１０と、第１接続部１１８ａと、第２接続部１１８ｂと、第３接続部１１８ｃとを有している。 (arm)
The arm 11 constitutes the main body of the manipulator 10. The arm 11 is placed on a pedestal 15 placed on the ground. Specifically, one end of the arm 11 is fixed to the pedestal 15. The arm 11 can move three-dimensionally around the pedestal 15 by changing its posture with the pedestal 15 as a base point. The arm 11 has, for example, a plurality of (six) joints 110, a first connection portion 118a, a second connection portion 118b, and a third connection portion 118c.
　各々の関節部１１０は、例えば円筒状を成しており、当該関節部１１０を回動可能にするためのモータ１１７を内部に有している。関節部１１０が有するモータ１１７には、例えばサーボモータが挙げられる。モータ１１７は、回転駆動することで、当該モータ１１７が回動の中心とする回動軸線回りに関節部１１０の全体を回動させる。以下、アーム１１が備える６つの関節部１１０を台座部１５に近い側から順に、「第１関節部１１１」、「第２関節部１１２」、「第３関節部１１３」、「第４関節部１１４」、「第５関節部１１５」、「第６関節部１１６」と称する。また、第１関節部１１１のモータ１１７を「第１モータ１１７ａ」と称し、第２関節部１１２のモータ１１７を「第２モータ１１７ｂ」と称し、第３関節部１１３のモータ１１７を「第３モータ１１７ｃ」と称し、第４関節部１１４のモータ１１７を「第４モータ１１７ｄ」と称し、第５関節部１１５のモータ１１７を「第５モータ１１７ｅ」と称し、第６関節部１１６のモータ１１７を「第６モータ１１７ｆ」と称する。各々のモータ１１７（第１モータ１１７ａ～第６モータ１１７ｆ）の回動は、制御装置３０（後述）によって制御されている。 Each joint 110 is, for example, cylindrical, and has a motor 117 therein for making the joint 110 rotatable. The motor 117 in the joint 110 may be, for example, a servo motor. The motor 117 rotates to rotate the entire joint 110 around a rotation axis around which the motor 117 rotates. Hereinafter, the six joints 110 provided on the arm 11 will be referred to as the "first joint 111," "second joint 112," "third joint 113," "fourth joint 114," "fifth joint 115," and "sixth joint 116," in order from the side closest to the base 15. In addition, the motor 117 of the first joint 111 is referred to as the "first motor 117a", the motor 117 of the second joint 112 is referred to as the "second motor 117b", the motor 117 of the third joint 113 is referred to as the "third motor 117c", the motor 117 of the fourth joint 114 is referred to as the "fourth motor 117d", the motor 117 of the fifth joint 115 is referred to as the "fifth motor 117e", and the motor 117 of the sixth joint 116 is referred to as the "sixth motor 117f". The rotation of each motor 117 (first motor 117a to sixth motor 117f) is controlled by a control device 30 (described below).
　第１関節部１１１は、台座部１５の上面に固定されている。第１関節部１１１は、アーム１１の上記一端に相当している。第１モータ１１７ａは、例えば鉛直方向に延びるパン軸としての第１回動軸線Ｏ１を中心に回動可能であり、第１関節部１１１の全体を当該第１回動軸線Ｏ１回りに回動させる。第２関節部１１２は、第１関節部１１１と一体の状態で第１関節部１１１の外面から延びるように設けられている。第２モータ１１７ｂは、例えば水平方向に延びるチルト軸としての第２回動軸線Ｏ２を中心に回動可能であり、第２関節部１１２の全体を当該第２回動軸線Ｏ２回りに回動させる。第２関節部１１２には、第２回動軸線Ｏ２に交差する方向に延びる第１接続部１１８ａが設けられている。第１接続部１１８ａは、円筒状を成しており、第２関節部１１２の外面から延びている。第３関節部１１３は、第１接続部１１８ａの第２関節部１１２から延びた先に一体に設けられている。第３モータ１１７ｃは、例えば水平方向に延びるチルト軸としての第３回動軸線Ｏ３を中心に回動可能であり、第３関節部１１３の全体を当該第３回動軸線Ｏ３回りに回動させる。第３関節部１１３には、第３回動軸線Ｏ３に交差する方向に延びる第２接続部１１８ｂが設けられている。第２接続部１１８ｂは、円筒状を成しており、第３関節部１１３の外面から延びている。第４関節部１１４は、第２接続部１１８ｂの第３関節部１１３から延びた先に一体に設けられている。第４モータ１１７ｄは、例えば水平方向に延びるチルト軸としての第４回動軸線Ｏ４を中心に回動可能であり、第４関節部１１４の全体を当該第４回動軸線Ｏ４回りに回動させる。第５関節部１１５は、第４関節部１１４と一体に設けられている。第５モータ１１７ｅは、例えば第４回動軸線Ｏ４と交差する方向に延びるパン軸としての第５回動軸線Ｏ５を中心に回動可能であり、第５関節部１１５の全体を当該第５回動軸線Ｏ５回りに回動させる。第５関節部１１５には、第５回動軸線Ｏ５に交差する方向に延びる第３接続部１１８ｃが設けられている。第３接続部１１８ｃは、円筒状を成しており、第５関節部１１５の外面から延びている。第６関節部１１６は、第３接続部１１８ｃの第５関節部１１５から延びた先に一体に設けられている。第６関節部１１６は、アーム１１の上記一端とは反対側の他端に相当している。第６モータ１１７ｆは、例えば第５回動軸線Ｏ５と平行に延びるパン軸としての第６回動軸線Ｏ６を中心に回動可能である。上述した各々の関節部１１０がモータ１１７によって回動することで、アーム１１の姿勢が３次元的に変化する。以下、第６回動軸線Ｏ６を単に「軸線Ａｒ１」と称する。また、当該軸線Ａｒ１が延びる方向を「軸線方向Ｄａ」と称し、軸線方向Ｄａにおける一方側を単に「一方側Ｄａ１」と称し、一方側Ｄａ１とは反対の側を「他方側Ｄａ２」と称する。各々の関節部１１０のモータ１１７は、制御対象装置Ｓの一例である。 The first joint 111 is fixed to the upper surface of the base 15. The first joint 111 corresponds to the one end of the arm 11. The first motor 117a can rotate around a first rotation axis O1 as a pan axis extending in the vertical direction, for example, and rotates the entire first joint 111 around the first rotation axis O1. The second joint 112 is provided so as to extend from the outer surface of the first joint 111 in a state integral with the first joint 111. The second motor 117b can rotate around a second rotation axis O2 as a tilt axis extending in the horizontal direction, for example, and rotates the entire second joint 112 around the second rotation axis O2. The second joint 112 is provided with a first connection portion 118a extending in a direction intersecting the second rotation axis O2. The first connecting portion 118a is cylindrical and extends from the outer surface of the second joint portion 112. The third joint portion 113 is provided integrally at the end of the first connecting portion 118a extending from the second joint portion 112. The third motor 117c is rotatable about a third rotation axis O3 as a tilt axis extending in the horizontal direction, for example, and rotates the entire third joint portion 113 about the third rotation axis O3. The third joint portion 113 is provided with a second connecting portion 118b extending in a direction intersecting the third rotation axis O3. The second connecting portion 118b is cylindrical and extends from the outer surface of the third joint portion 113. The fourth joint portion 114 is provided integrally at the end of the second connecting portion 118b extending from the third joint portion 113. The fourth motor 117d is rotatable around a fourth rotation axis O4 as a tilt axis extending in the horizontal direction, for example, and rotates the entire fourth joint 114 around the fourth rotation axis O4. The fifth joint 115 is provided integrally with the fourth joint 114. The fifth motor 117e is rotatable around a fifth rotation axis O5 as a pan axis extending in a direction intersecting with the fourth rotation axis O4, for example, and rotates the entire fifth joint 115 around the fifth rotation axis O5. The fifth joint 115 is provided with a third connection portion 118c extending in a direction intersecting with the fifth rotation axis O5. The third connection portion 118c is cylindrical and extends from the outer surface of the fifth joint 115. The sixth joint 116 is provided integrally with the third connection portion 118c at the end extending from the fifth joint 115. The sixth joint 116 corresponds to the other end of the arm 11 opposite to the one end. The sixth motor 117f can rotate around a sixth rotation axis O6 as a pan axis extending parallel to the fifth rotation axis O5. When each of the above-mentioned joints 110 rotates by the motor 117, the posture of the arm 11 changes three-dimensionally. Hereinafter, the sixth rotation axis O6 is simply referred to as the "axis Ar1". In addition, the direction in which the axis Ar1 extends is referred to as the "axial direction Da", one side in the axial direction Da is simply referred to as the "one side Da1", and the side opposite the one side Da1 is simply referred to as the "other side Da2". The motor 117 of each joint 110 is an example of a controlled device S.
（エンコーダ）
　エンコーダ１３０は、関節部１１０の回転角を取得可能なセンサである。以下、各々の関節部１１０（第１関節部１１１～第６関節部１１６）の回転角（θ１～θ６）を「関節角」と称する。エンコーダ１３０は、例えば、各々の関節部１１０のモータ１１７に１つずつ設けられている。以下、第１モータ１１７ａに設けられたエンコーダ１３０を「第１エンコーダ１３１」と称し、第２モータ１１７ｂに設けられたエンコーダ１３０を「第２エンコーダ１３２」と称し、第３モータ１１７ｃに設けられたエンコーダ１３０を「第３エンコーダ１３３」と称し、第４モータ１１７ｄに設けられたエンコーダ１３０を「第４エンコーダ１３４」と称し、第５モータ１１７ｅに設けられたエンコーダ１３０を「第５エンコーダ１３５」と称し、第６モータ１１７ｆに設けられたエンコーダ１３０を「第６エンコーダ１３６」と称する。各々のエンコーダ１３０は、モータ１１７から取得した関節角（センサデータ）を制御装置３０に送信する。エンコーダ１３０は、第１センサ１３の一例である。第１センサ１３は、有線または無線通信によって制御装置３０に接続されている。 (Encoder)
The encoder 130 is a sensor capable of acquiring the rotation angle of the joint 110. Hereinafter, the rotation angles (θ1 to θ6) of each of the joints 110 (the first joint 111 to the sixth joint 116) will be referred to as the “joint angle.” For example, one encoder 130 is provided for each of the motors 117 of the joints 110. Hereinafter, the encoder 130 provided in the first motor 117a is referred to as the "first encoder 131", the encoder 130 provided in the second motor 117b is referred to as the "second encoder 132", the encoder 130 provided in the third motor 117c is referred to as the "third encoder 133", the encoder 130 provided in the fourth motor 117d is referred to as the "fourth encoder 134", the encoder 130 provided in the fifth motor 117e is referred to as the "fifth encoder 135", and the encoder 130 provided in the sixth motor 117f is referred to as the "sixth encoder 136". Each encoder 130 transmits the joint angle (sensor data) acquired from the motor 117 to the control device 30. The encoder 130 is an example of the first sensor 13. The first sensor 13 is connected to the control device 30 by wired or wireless communication.
（力覚センサ）
　力覚センサ１４０は、アーム１１に接続されたロードセルである。具体的には、力覚センサ１４０は、第６関節部１１６に軸線方向Ｄａにおける一方側Ｄａ１から固定されている。力覚センサ１４０には、第６関節部１１６とは反対側（軸線方向Ｄａにおける一方側Ｄａ１）にエンドエフェクタ１２（後述）が設けられている。力覚センサ１４０は、エンドエフェクタ１２から伝達された力に関するデータを取得可能である。具体的には、力覚センサ１４０は、エンドエフェクタ１２が保持したナット２０２がボルト２０１に接触した場合に、当該力覚センサ１４０に対してエンドエフェクタ１２の側である一方側Ｄａ１から伝達された接触力の大きさを３軸の方向成分（Ｆｘ，Ｆｙ，Ｆｚ）ごとに取得可能であるとともに、軸ごとに働く各々の接触力の軸回りに作用するトルク（Ｍｘ，Ｍｙ，Ｍｚ）の大きさを取得可能である。以下、力覚データが取得した接触力（単位：Ｎ）およびトルク（単位：Ｎ・ｍ）をまとめて「力データ」と称する場合がある。すなわち、力覚センサ１４０は、６軸の力データを取得可能である。力覚センサ１４０は、取得した力データ（センサデータ）を制御装置３０に送信する。力覚センサ１４０は、第２センサ１４の一例である。第２センサ１４は、有線または無線通信によって制御装置３０に接続されている。 (Force sensor)
The force sensor 140 is a load cell connected to the arm 11. Specifically, the force sensor 140 is fixed to the sixth joint 116 from one side Da1 in the axial direction Da. The force sensor 140 is provided with an end effector 12 (described later) on the opposite side (one side Da1 in the axial direction Da) of the sixth joint 116. The force sensor 140 can acquire data related to the force transmitted from the end effector 12. Specifically, when the nut 202 held by the end effector 12 contacts the bolt 201, the force sensor 140 can acquire the magnitude of the contact force transmitted to the force sensor 140 from the one side Da1 on the end effector 12 side for each of the three directional components (Fx, Fy, Fz) of the three axes, and can acquire the magnitude of the torque (Mx, My, Mz) acting around each axis of the contact force acting on each axis. Hereinafter, the contact force (unit: N) and torque (unit: Nm) acquired by the force sensor data may be collectively referred to as "force data." That is, the force sensor 140 is capable of acquiring six-axis force data. The force sensor 140 transmits the acquired force data (sensor data) to the control device 30. The force sensor 140 is an example of the second sensor 14. The second sensor 14 is connected to the control device 30 by wired or wireless communication.
（エンドエフェクタ）
　エンドエフェクタ１２は、ナット２０２を保持してナット２０２を回転軸線Ａｒ回りに回転させる。回転軸線Ａｒは、例えば、軸線Ａｒ１に対してオフセットされた位置で軸線方向Ｄａに延びる仮想軸線である。なお、回転軸線Ａｒは、軸線Ａｒ１と一致していてもよい。本実施形態では、エンドエフェクタ１２は、力覚センサ１４０に一方側Ｄａ１から設けられている。したがって、エンドエフェクタ１２は、アーム１１によって移動されるとともに、施工対象物２００に設けられたボルト２０１に対して保持したナット２０２を締結する。図１および図２に示すように、エンドエフェクタ１２は、例えば、撮像装置２０と、本体部１２０と、ナットランナ１２１と、ナット保持部材１２４と、弾性部材１２８とを有している。図２中では、図示の便宜上、ナットランナ１２１のソケット部１２３を断面で内部が見えるように示している。 (End effector)
The end effector 12 holds the nut 202 and rotates the nut 202 around the rotation axis Ar. The rotation axis Ar is, for example, a virtual axis extending in the axial direction Da at a position offset from the axis Ar1. The rotation axis Ar may coincide with the axis Ar1. In this embodiment, the end effector 12 is provided on one side Da1 of the force sensor 140. Therefore, the end effector 12 is moved by the arm 11 and fastens the held nut 202 to the bolt 201 provided on the construction target 200. As shown in FIG. 1 and FIG. 2, the end effector 12 has, for example, an imaging device 20, a main body 120, a nut runner 121, a nut holding member 124, and an elastic member 128. In FIG. 2, for convenience of illustration, the socket portion 123 of the nut runner 121 is shown in cross section so that the inside can be seen.
　（撮像装置）
　図１、図２、および図３に示すように、撮像装置２０は、エンドエフェクタ１２に設けられ、施工対象物２００を撮像した画像を生成する。本実施形態では、撮像装置２０は、後述のエンドエフェクタ１２が有するナットランナ１２１のソケット部１２３の内部に、ソケット部１２３に対して回転不能の状態で配置されている。具体的には、図２に示すように、撮像装置２０は、後述の収容部１２３ｒよりも基部１２２に近い側のソケット部１２３内（後述の収容空間Ｒ）に配置されている。撮像装置２０は、例えば、撮像部２１と、照明部２２とを有している。撮像部２１は、後述のナット保持部材１２４によって保持されたナット２０２のナット孔２０２ｈを通じて施工対象物２００を撮像した画像を生成する。撮像部２１には、例えば、ＣＣＤイメージセンサやＣＭＯＳイメージセンサなどの半導体素子（固体撮像素子）が組み込まれたセンサモジュールを例示することができる。なお、撮像部２１は、これに代えて、ファイバスコープなどであってもよい。照明部２２は、ナット孔２０２ｈを通じて施工対象物２００に向けて光を照射する。照明部２２には、例えば、ＬＥＤ（Light Emitting Diode）などの素子を例示することができる。図６は、撮像装置２０が撮像した画像の一例を模式的に示す図である。図６中では、施工対象物２００に設けられたボルト２０１をやや斜め上方から撮像した画像の一例を模式的に示している。なお、図６中では、ボルト２０１の中心線Ａｒ２（符号の図示は省略）を中心とした穴が形成されたボルト２０１を一例として示しているが、ボルト２０１には穴が形成されていなくてもよい。図６に示すように、撮像装置２０は、ナット孔２０２ｈを通じて施工対象物２００およびボルト２０１に光を照射した状態で、施工対象物２００の表面や、ボルト２０１の雄ねじ部２０１ａおよびボルト２０１の端面２０１ｅなどを撮像することができる。撮像装置２０は、有線または無線通信によって制御装置３０に接続されている。撮像装置２０は、撮像した画像を制御装置３０に送信する。 (Imaging device)
As shown in FIG. 1, FIG. 2, and FIG. 3, the imaging device 20 is provided in the end effector 12 and generates an image of the construction target 200. In this embodiment, the imaging device 20 is arranged inside the socket part 123 of the nut runner 121 of the end effector 12 described later in a state where it cannot rotate with respect to the socket part 123. Specifically, as shown in FIG. 2, the imaging device 20 is arranged in the socket part 123 (the storage space R described later) closer to the base part 122 than the storage part 123r described later. The imaging device 20 has, for example, an imaging unit 21 and an illumination unit 22. The imaging unit 21 generates an image of the construction target 200 through a nut hole 202h of a nut 202 held by a nut holding member 124 described later. The imaging unit 21 can be exemplified by a sensor module in which a semiconductor element (solid-state imaging element) such as a CCD image sensor or a CMOS image sensor is incorporated. Note that the imaging unit 21 may be a fiberscope or the like instead of this. The illumination unit 22 irradiates light toward the construction target 200 through the nut hole 202h. The illumination unit 22 may be, for example, an element such as an LED (Light Emitting Diode). FIG. 6 is a diagram showing an example of an image captured by the imaging device 20. FIG. 6 shows an example of an image of a bolt 201 provided on the construction target 200 captured from a slightly obliquely upward direction. Note that FIG. 6 shows an example of a bolt 201 in which a hole is formed with a center line Ar2 (not shown) of the bolt 201 as the center, but the bolt 201 may not have a hole. As shown in FIG. 6, the imaging device 20 can image the surface of the construction target 200, the male thread portion 201a of the bolt 201, and the end surface 201e of the bolt 201 while irradiating the construction target 200 and the bolt 201 with light through the nut hole 202h. The imaging device 20 is connected to the control device 30 by wired or wireless communication. The imaging device 20 transmits the captured image to the control device 30 .
　（本体部）
　本体部１２０は、力覚センサ１４０に固定されている。本体部１２０には、ナットランナ１２１のソケット部１２３（後述、図２参照）が挿通されており、本体部１２０は、当該ソケット部１２３を回転軸線Ａｒ回りに回転させるための締結用モータ１２０ａ（図１参照）を内部に有している。本体部１２０の締結用モータ１２０ａの回転は、制御装置３０によって制御されている。本体部１２０の締結用モータ１２０ａは、制御対象装置Ｓの一例である。 (Main body)
The main body 120 is fixed to the force sensor 140. A socket 123 (described later, see FIG. 2) of a nut runner 121 is inserted into the main body 120, and the main body 120 has therein a fastening motor 120a (see FIG. 1) for rotating the socket 123 about a rotation axis Ar. The rotation of the fastening motor 120a of the main body 120 is controlled by the control device 30. The fastening motor 120a of the main body 120 is an example of a device S to be controlled.
　（ナットランナ）
　図２に示すように、ナットランナ１２１は、例えば、基部１２２と、ソケット部１２３とを有している。基部１２２は、例えば回転軸線Ａｒを中心とした円板状を成しており、本体部１２０に一方側Ｄａ１から固定されている。基部１２２は、一方側Ｄａ１を向く前面１２２ａを有している。ソケット部１２３は、基部１２２の前面１２２ａよりも一方側Ｄａ１に延びるように設けられて、回転軸線Ａｒを中心とした円筒状を成している。ソケット部１２３は、基部１２２および本体部１２０内に挿通されており、本体部１２０内に設けられた締結用モータ１２０ａによって回動可能に保持されている（図１参照）。ソケット部１２３は、内部を画定する円筒面状の内周面１２３ｉと、外側を向く円筒面状の外周面１２３ｏとを有している。以下、ソケット部１２３の内部の空間を「収容空間Ｒ」と称する。ソケット部１２３は、一方側Ｄａ１に配置された先端部分に開口１２３ｅを有している。本実施形態では、開口１２３ｅは、回転軸線Ａｒに対して直交する正円形に形成されている。また、ソケット部１２３には、当該開口１２３ｅを含み、ナット２０２の少なくとも一部を収容可能な収容部１２３ｒが形成されている。収容部１２３ｒは、収容空間Ｒにおける一方側Ｄａ１に配置されている。 (Nutrunner)
As shown in FIG. 2, the nut runner 121 has, for example, a base 122 and a socket 123. The base 122 has, for example, a disk shape centered on the rotation axis Ar, and is fixed to the main body 120 from one side Da1. The base 122 has a front surface 122a facing the one side Da1. The socket 123 is provided so as to extend toward the one side Da1 beyond the front surface 122a of the base 122, and has a cylindrical shape centered on the rotation axis Ar. The socket 123 is inserted into the base 122 and the main body 120, and is rotatably held by a fastening motor 120a provided in the main body 120 (see FIG. 1). The socket 123 has a cylindrical inner circumferential surface 123i that defines the interior, and a cylindrical outer circumferential surface 123o that faces outward. Hereinafter, the space inside the socket 123 is referred to as an "accommodation space R". The socket portion 123 has an opening 123e at a tip portion located on one side Da1. In this embodiment, the opening 123e is formed in a perfect circle perpendicular to the rotation axis Ar. The socket portion 123 also has an accommodating portion 123r that includes the opening 123e and is capable of accommodating at least a portion of the nut 202. The accommodating portion 123r is located on one side Da1 in the accommodation space R.
　図２および図３に示すように、収容部１２３ｒは、例えば、複数のテーパ面１２３ａと、複数の締結面１２３ｂと、位置決め面１２３ｃとで構成されている。各々のテーパ面１２３ａは、開口１２３ｅが広がる方向に対して傾斜して配置されている。具体的には、各々のテーパ面１２３ａは、他方側Ｄａ２に向かうにしたがって回転軸線Ａｒに近づくように、開口１２３ｅが広がる方向に対して傾斜して配置されている。本実施形態では、６つのテーパ面１２３ａが回転軸線Ａｒ回りに並んで配置されており、回転軸線Ａｒ回りに隣り合うテーパ面１２３ａ同士は、互いに接続されている。収容部１２３ｒにおける複数のテーパ面１２３ａによって画定された領域は、回転軸線Ａｒに直交する断面が正六角形を成すとともに、他方側Ｄａ２に向かうにしたがって小さくなるテーパ状を成している。 2 and 3, the accommodation portion 123r is composed of, for example, a plurality of tapered surfaces 123a, a plurality of fastening surfaces 123b, and a positioning surface 123c. Each tapered surface 123a is arranged at an incline with respect to the direction in which the opening 123e widens. Specifically, each tapered surface 123a is arranged at an incline with respect to the direction in which the opening 123e widens so as to approach the rotation axis Ar as it approaches the other side Da2. In this embodiment, six tapered surfaces 123a are arranged side by side around the rotation axis Ar, and adjacent tapered surfaces 123a around the rotation axis Ar are connected to each other. The area defined by the plurality of tapered surfaces 123a in the accommodation portion 123r has a cross section perpendicular to the rotation axis Ar that forms a regular hexagon, and is tapered toward the other side Da2.
　各々の締結面１２３ｂは、１つのテーパ面１２３ａに対応するように他方側Ｄａ２からテーパ面１２３ａに接続されており、開口１２３ｅが広がる方向に対して垂直に配置されている。本実施形態では、締結面１２３ｂは、６つの締結面１２３ｂが回転軸線Ａｒ回りに並んで配置されており、回転軸線Ａｒ回りに隣り合う締結面１２３ｂ同士は、互いに接続されている。収容部１２３ｒにおける複数の締結面１２３ｂによって画定された領域は、回転軸線Ａｒに直交する断面が軸線方向Ｄａに一定とされた六角柱状を成している。複数の締結面１２３ｂによって画定された領域は、例えば、ナット２０２のカラー２０２ｂがちょうど収まる大きさとされている。 Each fastening surface 123b is connected to the tapered surface 123a from the other side Da2 so as to correspond to one of the tapered surfaces 123a, and is arranged perpendicular to the direction in which the opening 123e widens. In this embodiment, the fastening surfaces 123b are arranged in a row of six around the rotation axis Ar, and the fastening surfaces 123b adjacent to each other around the rotation axis Ar are connected to each other. The area defined by the multiple fastening surfaces 123b in the accommodating portion 123r forms a hexagonal prism whose cross section perpendicular to the rotation axis Ar is constant in the axial direction Da. The area defined by the multiple fastening surfaces 123b is, for example, large enough to fit the collar 202b of the nut 202.
　位置決め面１２３ｃは、複数（６つ）の締結面１２３ｂと、ソケット部１２３の上記内周面１２３ｉとを互いに接続する面であり、一方側Ｄａ１を向いている。本実施形態では、位置決め面１２３ｃは、開口１２３ｅが広がる方向に対して平行に配置されている。すなわち、位置決め面１２３ｃは、各々の締結面１２３ｂに対して垂直に配置されている。位置決め面１２３ｃは、収容部１２３ｒにナット２０２が収容された際、当該ナット２０２を位置決めさせるための面である。図３に示すように、位置決め面１２３ｃは、一方側Ｄａ１から見て、正六角形の中央部分から正円形を取り除いた形状を成している。この位置決め面１２３ｃにナット２０２のカラー２０２ｂが一方側Ｄａ１から当接することで、ナット２０２が位置決めされる。つまり、位置決め面１２３ｃによってナット２０２の位置が位置決めされることで、当該ナット２０２が収容部１２３ｒよりも他方側Ｄａ２の収容空間Ｒに入り込むことがない。 The positioning surface 123c is a surface that connects the multiple (six) fastening surfaces 123b and the inner peripheral surface 123i of the socket portion 123 to each other, and faces one side Da1. In this embodiment, the positioning surface 123c is arranged parallel to the direction in which the opening 123e widens. In other words, the positioning surface 123c is arranged perpendicular to each fastening surface 123b. The positioning surface 123c is a surface for positioning the nut 202 when the nut 202 is accommodated in the accommodation portion 123r. As shown in FIG. 3, the positioning surface 123c has a shape obtained by removing a perfect circle from the center part of a regular hexagon when viewed from one side Da1. The collar 202b of the nut 202 abuts against this positioning surface 123c from one side Da1, thereby positioning the nut 202. In other words, the position of the nut 202 is determined by the positioning surface 123c, so that the nut 202 does not enter the accommodation space R on the other side Da2 beyond the accommodation portion 123r.
　（ナット保持部材）
　ナット保持部材１２４は、ナットランナ１２１の収容部１２３ｒにナット２０２のカラー２０２ｂが収容された際、当該ナット２０２が収容部１２３ｒから一方側Ｄａ１へ抜け落ちることがないようにナット２０２を保持する。図２に戻り、ナット保持部材１２４は、ソケット部１２３の外周側に設けられている。本実施形態では、２つのナット保持部材１２４がソケット部１２３の外周側でソケット部１２３を間に挟むように設けられている。なお、３つ以上のナット保持部材１２４がソケット部１２３を中心にして回転軸線Ａｒ回りに等間隔に配置されてもよい。ナット保持部材１２４は、例えば、回動軸１２５と、第１部分１２６と、第２部分１２７とを有している。 (Nut holding member)
The nut holding member 124 holds the nut 202 so that the nut 202 does not fall out of the housing portion 123r to one side Da1 when the collar 202b of the nut 202 is housed in the housing portion 123r of the nut runner 121. Returning to FIG. 2, the nut holding member 124 is provided on the outer periphery side of the socket portion 123. In this embodiment, two nut holding members 124 are provided on the outer periphery side of the socket portion 123 so as to sandwich the socket portion 123 therebetween. Note that three or more nut holding members 124 may be disposed at equal intervals around the rotation axis Ar with the socket portion 123 as the center. The nut holding member 124 has, for example, a rotating shaft 125, a first portion 126, and a second portion 127.
　回動軸１２５は、ソケット部１２３の外周面１２３ｏに回動可能に設けられている。第１部分１２６は、回動軸１２５に固定されており、回転軸線Ａｒに交差する方向に延びる棒状の部材である。第２部分１２７は、第１部分１２６からソケット部１２３の先端側に向かって延びる延在部１２７ａと、当該延在部１２７ａの先端に設けられて回転軸線Ａｒに近づくように延在部１２７ａから突出する爪部１２７ｂとを有している。延在部１２７ａは、回転軸線Ａｒに沿って延びる棒状の部材である。延在部１２７ａは、第１部分１２６と共にＬ字状を成している。爪部１２７ｂは、延在部１２７ａから回転軸線Ａｒに近づくにしたがって他方側Ｄａ２に向かっており、回転軸線Ａｒに近づくにしたがって先細りした形状を成している。爪部１２７ｂは、ソケット部１２３の開口１２３ｅよりも一方側Ｄａ１に配置されている。また、図２および図３に示すように、爪部１２７ｂの先端は、締結面１２３ｂよりも回転軸線Ａｒに近い側に位置している。爪部１２７ｂの先端は、ナット２０２のカラー２０２ｂがソケット部１２３の収容部１２３ｒに収容された際、弾性部材１２８（後述）の弾性力によってナット２０２の側面に当接する。具体的には、ナット２０２が収容部１２３ｒに収容された際、当該ナット２０２は、２つの爪部１２７ｂによって把持される（拘束される）。 The pivot shaft 125 is rotatably provided on the outer peripheral surface 123o of the socket portion 123. The first portion 126 is fixed to the pivot shaft 125 and is a rod-shaped member extending in a direction intersecting the rotation axis Ar. The second portion 127 has an extension portion 127a extending from the first portion 126 toward the tip side of the socket portion 123, and a claw portion 127b provided at the tip of the extension portion 127a and protruding from the extension portion 127a so as to approach the rotation axis Ar. The extension portion 127a is a rod-shaped member extending along the rotation axis Ar. The extension portion 127a forms an L-shape together with the first portion 126. The claw portion 127b extends from the extension portion 127a toward the other side Da2 as it approaches the rotation axis Ar, and has a tapered shape as it approaches the rotation axis Ar. The claw portion 127b is disposed on one side Da1 of the opening 123e of the socket portion 123. Also, as shown in FIG. 2 and FIG. 3, the tip of the claw portion 127b is located closer to the rotation axis Ar than the fastening surface 123b. When the collar 202b of the nut 202 is accommodated in the accommodation portion 123r of the socket portion 123, the tip of the claw portion 127b abuts against the side of the nut 202 by the elastic force of the elastic member 128 (described later). Specifically, when the nut 202 is accommodated in the accommodation portion 123r, the nut 202 is gripped (restrained) by the two claw portions 127b.
　（弾性部材）
　弾性部材１２８は、ナットランナ１２１の基部１２２と、ナット保持部材１２４の第１部分１２６との間に設けられたばねである。具体的には、弾性部材１２８は、一端が基部１２２の前面１２２ａに固定されており、他端が第１部分１２６に固定されている。したがって、弾性部材１２８は、基部１２２と第１部分１２６とを互いに接続している。弾性部材１２８は、第１部分１２６に対してソケット部１２３の先端側（一方側Ｄａ１）に向かう弾性力を付与している。本実施形態では、１つの弾性部材１２８が１つのナット保持部材１２４に対応するように、基部１２２と第１部分１２６との間に設けられている。なお、複数の弾性部材１２８が１つのナット保持部材１２４に対応するように、基部１２２と第１部分１２６との間に設けられてもよい。また、弾性部材１２８は、第１部分１２６に対してソケット部１２３の先端側（一方側Ｄａ１）に向かう弾性力を付与することが可能であれば、ばねに限定されることはない。 (Elastic member)
The elastic member 128 is a spring provided between the base 122 of the nut runner 121 and the first portion 126 of the nut holding member 124. Specifically, one end of the elastic member 128 is fixed to the front surface 122a of the base 122, and the other end is fixed to the first portion 126. Thus, the elastic member 128 connects the base 122 and the first portion 126 to each other. The elastic member 128 applies an elastic force toward the tip side (one side Da1) of the socket portion 123 to the first portion 126. In this embodiment, one elastic member 128 is provided between the base 122 and the first portion 126 so as to correspond to one nut holding member 124. Note that a plurality of elastic members 128 may be provided between the base 122 and the first portion 126 so as to correspond to one nut holding member 124. Furthermore, the elastic member 128 is not limited to a spring, so long as it is capable of applying an elastic force to the first portion 126 in the direction toward the tip side (one side Da1) of the socket portion 123.
　以下、図４および図５を参照して、エンドエフェクタ１２のナット保持部材１２４にナット２０２を保持させる際のナット保持部材１２４の動き、および、エンドエフェクタ１２のナット保持部材１２４からカラー２０２ｂを取り外す際のナット保持部材１２４の動きを説明する。 Below, with reference to Figures 4 and 5, the movement of the nut holding member 124 when the nut 202 is held by the nut holding member 124 of the end effector 12, and the movement of the nut holding member 124 when the collar 202b is removed from the nut holding member 124 of the end effector 12 will be described.
　図４中の（ａ）に示すように、組立システム１００を利用するオペレータ（熟練作業員）などの手によって扱われたナット２０２のカラー２０２ｂが一方側Ｄａ１から爪部１２７ｂに当接した状態で、弾性部材１２８の弾性力に抗しながら他方側Ｄａ２に押圧されることで、ナット保持部材１２４の第２部分１２７および第１部分１２６は、回動軸１２５と共に回動する。第２部分１２７および第１部分１２６が回動軸１２５と共に回動することで、爪部１２７ｂ同士の間隔は、ナット２０２が通ることができる程度に広がる。そして、図４中の（ｂ）に示すように、爪部１２７ｂの間を通り抜けたナット２０２のカラー２０２ｂは、収容部１２３ｒ内に入り込み、テーパ面１２３ａによって他方側Ｄａ２に案内されながら、締結面１２３ｂおよび位置決め面１２３ｃによって画定された領域で位置決めされる。この際、弾性部材１２８が第１部分１２６を一方側Ｄａ１に押圧しているため、第１部分１２６から延びる延在部１２７ａと一体の爪部１２７ｂは、ナット本体２０２ａの側面に当接する。つまり、ナット本体２０２ａは、カラー２０２ｂが収容部１２３ｒに収容された際、２つの爪部１２７ｂによって把持される（拘束される）。なお、オペレータの手に代えて、治具やアクチュエータなどを用いてナット２０２をエンドエフェクタ１２のナット保持部材１２４に保持させてもよい。 As shown in (a) of FIG. 4, when the collar 202b of the nut 202 is handled by an operator (skilled worker) using the assembly system 100, the collar 202b is pressed against the claw portion 127b from one side Da1 against the elastic force of the elastic member 128 toward the other side Da2, causing the second part 127 and the first part 126 of the nut holding member 124 to rotate together with the pivot shaft 125. As the second part 127 and the first part 126 rotate together with the pivot shaft 125, the gap between the claw portions 127b widens to an extent that the nut 202 can pass through. Then, as shown in (b) of FIG. 4, the collar 202b of the nut 202 that has passed between the claw portions 127b enters the accommodating portion 123r and is positioned in the area defined by the fastening surface 123b and the positioning surface 123c while being guided to the other side Da2 by the tapered surface 123a. At this time, since the elastic member 128 presses the first portion 126 to one side Da1, the claw portion 127b integral with the extension portion 127a extending from the first portion 126 abuts against the side surface of the nut body 202a. In other words, when the collar 202b is accommodated in the accommodation portion 123r, the nut body 202a is gripped (restrained) by the two claw portions 127b. Note that the nut 202 may be held by the nut holding member 124 of the end effector 12 using a jig, actuator, or the like instead of the operator's hand.
　図５中の（ａ）に示すように、ナットランナ１２１のソケット部１２３の回転によってナット２０２がボルト２０１に締結された際、ナット２０２のナット本体２０２ａのみがボルト２０１に締結された状態でカラー２０２ｂがナット本体２０２ａから外れる（破断する）。そして、ナット本体２０２ａから分離されたカラー２０２ｂは、ナット保持部材１２４の爪部１２７ｂによって保持される。つまり、ナット２０２がカラー切りされる。そして、図５中の（ｂ）に示すように、オペレータなどの手によって第１部分１２６が弾性部材１２８の弾性力に抗しながら他方側Ｄａ２に押圧されることで、第１部分１２６および第２部分１２７が回動軸１２５を中心に回動した結果、爪部１２７ｂ同士の間隔は、カラー２０２ｂが通ることができる程度に広がる。これによって、爪部１２７ｂによって保持されていたカラー２０２ｂが爪部１２７ｂ同士の隙間から落下する。すなわち、第１部分１２６は、ナット本体２０２ａとボルト２０１との締結が終わった後にナット保持部材１２４によって保持されたカラー２０２ｂを、爪部１２７ｂ同士の隙間から落下させるためのレバーとして機能している。 As shown in (a) of FIG. 5, when the nut 202 is fastened to the bolt 201 by the rotation of the socket portion 123 of the nut runner 121, the collar 202b comes off (breaks off) from the nut body 202a with only the nut body 202a of the nut 202 fastened to the bolt 201. Then, the collar 202b separated from the nut body 202a is held by the claw portion 127b of the nut holding member 124. In other words, the nut 202 is collar-cut. Then, as shown in (b) of FIG. 5, the first part 126 is pressed to the other side Da2 by the hand of an operator or the like against the elastic force of the elastic member 128, and the first part 126 and the second part 127 rotate around the rotation shaft 125. As a result, the gap between the claw portions 127b widens to an extent that the collar 202b can pass through. As a result, the collar 202b held by the claw portions 127b falls from the gap between the claw portions 127b. In other words, the first portion 126 functions as a lever to allow the collar 202b held by the nut holding member 124 to drop through the gap between the claws 127b after the nut body 202a and the bolt 201 have been tightened.
（制御装置の構成）
　制御装置３０は、エンドエフェクタ１２に保持されたナット２０２が施工対象物２００に設けられたボルト２０１に締結されるように、マニピュレータ１０の各々の関節角、およびエンドエフェクタ１２の本体部１２０が有する締結用モータ１２０ａを制御する。図７に示すように、制御装置３０は、例えば、取得部３１と、算出部３２と、座標変換部３３と、補正部３４と、判定部３５と、制御部３６と、学習部３７と、カウンタ部３８と、記憶部３９とを備えている。 (Configuration of the control device)
The control device 30 controls each joint angle of the manipulator 10 and the fastening motor 120a of the main body 120 of the end effector 12 so that the nut 202 held by the end effector 12 is fastened to the bolt 201 provided on the construction target 200. As shown in Fig. 7, the control device 30 includes, for example, an acquisition unit 31, a calculation unit 32, a coordinate conversion unit 33, a correction unit 34, a determination unit 35, a control unit 36, a learning unit 37, a counter unit 38, and a storage unit 39.
（取得部）
　取得部３１は、エンコーダ１３０（第１センサ１３）および力覚センサ１４０（第２センサ１４）から送信されるセンサデータをリアルタイムで受信することで経時的に取得する。取得部３１は、エンコーダ１３０から各々の関節部１１０（第１関節部１１１～第６関節部１１６）の関節角を第１所定周期で取得し、力覚センサ１４０から力データを第１所定周期で取得する。第１所定周期は、例えば、エンコーダ１３０または力覚センサ１４０のサンプリングレートなどに基づき決定される。また、取得部３１は、撮像装置２０から送信される画像をリアルタイムで受信することで経時的に取得する。取得部３１は、画像を第２所定周期で撮像装置２０から取得する。第２所定周期は、例えば、撮像装置２０のフレームレート（ｆｐｓ：flames per second）などに基づき決定される。なお、第２所定周期は、第１所定周期と同じであってもよく、異なっていてもよい。また、取得部３１は、直近の施工の対象となるボルト２０１の中心位置座標を、記憶部３９によってあらかじめ記憶されている施工対象物２００のＣＡＤデータ３９５（Computer-Aided Design）などから取得する。施工対象物２００のＣＡＤデータ３９５は、例えば、施工対象物２００の輪郭を３次元的に示した点群データなどを含む。当該点群データを構成する各々の点には、マニピュレータ１０が持つ原点（０，０，０）に準拠した世界座標系で表現される３次元の座標（Ｘｗ，Ｙｗ，Ｚｗ）が割り当てられている。 (Acquisition Department)
The acquisition unit 31 acquires sensor data over time by receiving the sensor data transmitted from the encoder 130 (first sensor 13) and the force sensor 140 (second sensor 14) in real time. The acquisition unit 31 acquires the joint angles of the joints 110 (first joint 111 to sixth joint 116) from the encoder 130 at a first predetermined period, and acquires force data from the force sensor 140 at the first predetermined period. The first predetermined period is determined based on, for example, a sampling rate of the encoder 130 or the force sensor 140. The acquisition unit 31 also acquires images over time by receiving images transmitted from the imaging device 20 in real time. The acquisition unit 31 acquires images from the imaging device 20 at a second predetermined period. The second predetermined period is determined based on, for example, a frame rate (fps: flames per second) of the imaging device 20. The second predetermined period may be the same as the first predetermined period, or may be different. The acquisition unit 31 also acquires the central position coordinates of the bolt 201 to be immediately executed from CAD data 395 (Computer-Aided Design) of the construction target 200 stored in advance in the storage unit 39. The CAD data 395 of the construction target 200 includes, for example, point cloud data that three-dimensionally shows the contour of the construction target 200. Each point constituting the point cloud data is assigned a three-dimensional coordinate (Xw, Yw, Zw) expressed in a world coordinate system based on the origin (0, 0, 0) of the manipulator 10.
　取得部３１は、取得した各々の関節部１１０の関節角を算出部３２に送るとともに記憶部３９に記憶させ、取得した力データを補正部３４に送るとともに記憶部３９に記憶させ、取得したボルト２０１の中心位置座標を座標変換部３３に送るとともに記憶部３９に記憶させ、取得した画像を判定部３５および制御部３６に送るとともに記憶部３９に記憶させる。また、取得部３１は、取得したボルト２０１の中心位置座標を座標変換部３３に送った際、直近の施工の対象となるボルト２０１の中心位置座標を取得した旨を示す取得フラグをカウンタ部３８に送る。取得フラグは、例えば１である。なお、カウンタ部３８は、取得部３１から受け付けた取得フラグを受け付けるたびに順次加算していき、取得フラグの加算結果を第６判定部３５６に送る。 The acquisition unit 31 sends the acquired joint angles of each joint 110 to the calculation unit 32 and stores them in the memory unit 39, sends the acquired force data to the correction unit 34 and stores them in the memory unit 39, sends the acquired center position coordinates of the bolt 201 to the coordinate conversion unit 33 and stores them in the memory unit 39, and sends the acquired images to the judgment unit 35 and the control unit 36 and stores them in the memory unit 39. When the acquisition unit 31 sends the acquired center position coordinates of the bolt 201 to the coordinate conversion unit 33, it also sends an acquisition flag to the counter unit 38 indicating that it has acquired the center position coordinates of the bolt 201 that is the target of the most recent construction. The acquisition flag is, for example, 1. The counter unit 38 increments the acquisition flag each time it receives it from the acquisition unit 31, and sends the incremented acquisition flag to the sixth judgment unit 356.
（算出部）
　算出部３２は、取得部３１から受け付けた各々の関節部１１０（第１関節部１１１～第６関節部１１６）の関節角に基づき、ナット２０２の位置座標および姿勢角を算出する。具体的には、図８に示すように、算出部３２は、各々の関節部１１０の関節角に基づき、マニピュレータ１０が持つ原点に準拠したナット２０２の端面２０２ｅの中心部分の位置座標（Ｘｎ，Ｙｎ，Ｚｎ）、および当該位置座標が表現される世界座標系の３座標面（ＸＹ平面、ＹＺ平面、ＺＸ平面）のそれぞれに対する傾斜角度である姿勢角（θｘ，θｙ，θｚ）を算出する。図８中では、図示の便宜上、ナット２０２の位置座標（Ｘｎ，Ｙｎ，Ｚｎ）のみを示している。マニピュレータ１０が持つ原点は、マニピュレータ１０中に設定された任意の一点であってもよいし、マニピュレータ１０と施工対象物２００との位置関係を調整するためのキャリブレーションなどがなされた後に設定された施工対象物２００中の任意の一点であってもよい。図８中では、施工対象物２００中にマニピュレータ１０が持つ原点が配置された場合を一例として示している。 (Calculation section)
The calculation unit 32 calculates the position coordinates and the attitude angle of the nut 202 based on the joint angles of each joint 110 (the first joint 111 to the sixth joint 116) received from the acquisition unit 31. Specifically, as shown in FIG. 8, the calculation unit 32 calculates the position coordinates (Xn, Yn, Zn) of the center part of the end face 202e of the nut 202 based on the origin of the manipulator 10, and the attitude angle (θx, θy, θz) which is the inclination angle with respect to each of the three coordinate planes (XY plane, YZ plane, ZX plane) of the world coordinate system in which the position coordinates are expressed, based on the joint angles of each joint 110. In FIG. 8, for convenience of illustration, only the position coordinates (Xn, Yn, Zn) of the nut 202 are shown. The origin of the manipulator 10 may be any one point set in the manipulator 10, or may be any one point in the construction object 200 set after calibration or the like is performed to adjust the positional relationship between the manipulator 10 and the construction object 200. FIG. 8 shows an example in which the origin of the manipulator 10 is located in the construction target object 200 .
　算出部３２は、算出したナット２０２の位置座標を座標変換部３３に送るとともに記憶部３９に記憶させ、算出したナット２０２の姿勢角を判定部３５および制御部３６に送るとともに記憶部３９に記憶させる。なお、算出部３２は、上記の算出動作に代えて、記憶部３９にあらかじめ記憶されている、各々の関節部１１０の関節角と、ナット２０２の位置座標および姿勢角とが互いに関連付いた対応関係情報（関数など）に各々の関節角を入力することで出力された位置座標および姿勢角をナット２０２の位置座標および姿勢角として取得してもよい。 The calculation unit 32 sends the calculated position coordinates of the nut 202 to the coordinate conversion unit 33 and stores them in the memory unit 39, and sends the calculated attitude angle of the nut 202 to the determination unit 35 and control unit 36 and stores them in the memory unit 39. Note that instead of the above calculation operation, the calculation unit 32 may input each joint angle into correspondence information (such as a function) that is stored in advance in the memory unit 39 and associates the joint angles of each joint unit 110 with the position coordinates and attitude angle of the nut 202, and obtain the output position coordinates and attitude angle as the position coordinates and attitude angle of the nut 202.
（座標変換部）
　座標変換部３３は、取得部３１から受け付けたボルト２０１の中心位置座標に基づき、算出部３２から受け付けたナット２０２の位置座標をボルト中心座標系によって表現された位置座標に変換する。具体的には、座標変換部３３は、世界座標系で表現されたナット２０２の位置座標を示すベクトルから、受け付けた直近の施工の対象とするボルト２０１の中心位置座標を示すベクトルを減算することによって、図８中に矢印Ａ（ベクトル）で示すように、ナット２０２の位置座標をボルト中心座標系（Ｘｂ，Ｙｂ，Ｚｂ）に変換する。本実施形態では、ボルト２０１の中心位置座標（ボルト中心座標系の原点）は、例えば、ボルト２０１の中心線Ａｒ２上のどこかに配置されており、例えば、施工対象物２００の表面とボルト２０１の中心線Ａｒ２とが仮想的に交わる個所に配置されている。座標変換部３３は、ボルト中心座標系に変換したナット２０２の位置座標を判定部３５および制御部３６に送るとともに記憶部３９に記憶させる。 (Coordinate conversion section)
The coordinate conversion unit 33 converts the position coordinates of the nut 202 received from the calculation unit 32 into position coordinates expressed by the bolt-centered coordinate system based on the central position coordinates of the bolt 201 received from the acquisition unit 31. Specifically, the coordinate conversion unit 33 converts the position coordinates of the nut 202 into the bolt-centered coordinate system (Xb, Yb, Zb) as shown by the arrow A (vector) in FIG. 8 by subtracting the vector indicating the central position coordinates of the bolt 201 to be the most recent construction target from the vector indicating the position coordinates of the nut 202 expressed in the world coordinate system. In this embodiment, the central position coordinates of the bolt 201 (the origin of the bolt-centered coordinate system) are, for example, located somewhere on the center line Ar2 of the bolt 201, and are, for example, located at a point where the surface of the construction target 200 and the center line Ar2 of the bolt 201 virtually intersect. The coordinate conversion unit 33 sends the position coordinates of the nut 202 converted into the bolt-centered coordinate system to the determination unit 35 and the control unit 36 and stores them in the memory unit 39.
（補正部）
　補正部３４は、取得部３１から受け付けた力データに含まれる接触力を補正する。図９は、エンドエフェクタ１２が保持したナット２０２がボルト２０１に接触した際にナット２０２がボルト２０１から受ける接触力を概念的に説明するための図である。図９中では、図示の便宜上、軸線方向Ｄａに働く接触力を矢印Ｆｂで示し、回転軸線Ａｒ回りに作用するトルクを矢印Ｆｔで示している。図１０は、補正部３４が接触力（矢印Ｆｂ）の補正に用いる補正値（矢印Ｆｇ１）を概念的に説明するための図である。図１０中では、Ｄｖが鉛直方向を示しており、ＣＧがエンドエフェクタ１２の重心位置を示している。補正部３４は、エンドエフェクタ１２の自重（矢印Ｆｇ）に基づく補正値（矢印Ｆｇ１）を接触力に加算することで、取得部３１から受け付けた接触力（矢印Ｆｂ）を補正する。補正値（矢印Ｆｇ１）は、例えば、重力に伴うエンドエフェクタ１２の自重（矢印Ｆｇ）が軸線方向Ｄａにおける一方側Ｄａ１に方向成分を有する場合（図１０中の（ａ））に負の値を示す。一方、補正値（矢印Ｆｇ１）は、例えば、エンドエフェクタ１２の自重（矢印Ｆｇ）が軸線方向Ｄａにおける他方側Ｄａ２に方向成分を有する場合（図１０中の（ｂ））に正の値を示す。補正値は、例えば、各々の関節部１１０の関節角に関連付いた対応関係情報（関数など）として、記憶部３９にあらかじめ記憶されている。補正部３４は、記憶部３９に記憶されている当該対応関係情報を適時に参照することで補正値を取得する。補正部３４は、補正後の接触力を含む力データを判定部３５および制御部３６に送るとともに記憶部３９に記憶させる。 (Correction unit)
The correction unit 34 corrects the contact force included in the force data received from the acquisition unit 31. FIG. 9 is a diagram for conceptually explaining the contact force that the nut 202 held by the end effector 12 receives from the bolt 201 when the nut 202 contacts the bolt 201. In FIG. 9, for convenience of illustration, the contact force acting in the axial direction Da is indicated by an arrow Fb, and the torque acting around the rotation axis Ar is indicated by an arrow Ft. FIG. 10 is a diagram for conceptually explaining the correction value (arrow Fg1) used by the correction unit 34 to correct the contact force (arrow Fb). In FIG. 10, Dv indicates the vertical direction, and CG indicates the center of gravity position of the end effector 12. The correction unit 34 corrects the contact force (arrow Fb) received from the acquisition unit 31 by adding a correction value (arrow Fg1) based on the weight (arrow Fg) of the end effector 12 to the contact force. The correction value (arrow Fg1) indicates a negative value, for example, when the weight (arrow Fg) of the end effector 12 due to gravity has a directional component on one side Da1 in the axial direction Da ((a) in FIG. 10). On the other hand, the correction value (arrow Fg1) indicates a positive value, for example, when the weight (arrow Fg) of the end effector 12 has a directional component on the other side Da2 in the axial direction Da ((b) in FIG. 10). The correction value is stored in advance in the storage unit 39 as, for example, correspondence information (such as a function) associated with the joint angle of each joint unit 110. The correction unit 34 acquires the correction value by referring to the correspondence information stored in the storage unit 39 at appropriate times. The correction unit 34 sends the force data including the corrected contact force to the determination unit 35 and the control unit 36 and stores it in the storage unit 39.
（判定部）
　判定部３５は、取得部３１から受け付けた画像、算出部３２から受け付けたナット２０２の姿勢角、座標変換部３３から受け付けたナット２０２の位置座標、ならびに、補正部３４から受け付けた力データに基づき、各種の判定の処理を行う。図７に戻り、判定部３５は、例えば、第１判定部３５１と、第２判定部３５２と、第３判定部３５３と、第４判定部３５４と、第５判定部３５５と、第６判定部３５６とを有している。 (Determination unit)
The determination unit 35 performs various determination processes based on the image received from the acquisition unit 31, the attitude angle of the nut 202 received from the calculation unit 32, the position coordinates of the nut 202 received from the coordinate conversion unit 33, and the force data received from the correction unit 34. Returning to Fig. 7, the determination unit 35 has, for example, a first determination unit 351, a second determination unit 352, a third determination unit 353, a fourth determination unit 354, a fifth determination unit 355, and a sixth determination unit 356.
　（第１判定部）
　第１判定部３５１は、補正部３４から受け付けた力データに基づき、ナット２０２がボルト２０１に接触しているか否かを判定する。以下、第１判定部３５１による判定を「第１判定」と称する。第１判定部３５１は、例えば、受け付けた力データに含まれる接触力が所定の接触力の大きさを示す第１しきい値未満である場合に、ナット２０２がボルト２０１に接触していないと判定する。一方、第１判定部３５１は、取得した力データに含まれる接触力が第１しきい値以上である場合に、ナット２０２がボルト２０１に接触していると判定する。以下、ナット２０２がボルト２０１に接触していない時のナット２０２の位置を「第１位置」と称し、ナット２０２がボルト２０１に接触している時のナット２０２の位置を「第２位置」と称する。後述のナット２０２およびボルト２０１が螺合した状態は、ナット２０２が第２位置に位置した状態に含まれる。なお、第１しきい値は、例えば記憶部３９にあらかじめ記憶されている。第１判定部３５１は、記憶部３９に記憶されている第１しきい値を適時に参照することで上記第１判定を行う。第１判定部３５１は、ナット２０２がボルト２０１に接触している否かを示す第１判定の結果を制御部３６に送る。 (First Determination Unit)
The first determination unit 351 determines whether the nut 202 is in contact with the bolt 201 based on the force data received from the correction unit 34. Hereinafter, the determination by the first determination unit 351 is referred to as the "first determination". For example, when the contact force included in the received force data is less than a first threshold value indicating the magnitude of a predetermined contact force, the first determination unit 351 determines that the nut 202 is not in contact with the bolt 201. On the other hand, when the contact force included in the acquired force data is equal to or greater than the first threshold value, the first determination unit 351 determines that the nut 202 is in contact with the bolt 201. Hereinafter, the position of the nut 202 when the nut 202 is not in contact with the bolt 201 is referred to as the "first position", and the position of the nut 202 when the nut 202 is in contact with the bolt 201 is referred to as the "second position". The state in which the nut 202 and the bolt 201 are screwed together, which will be described later, is included in the state in which the nut 202 is located at the second position. The first threshold value is stored in advance in, for example, the storage unit 39. The first determination unit 351 makes the first determination by timely referring to the first threshold value stored in the memory unit 39. The first determination unit 351 sends the result of the first determination, which indicates whether the nut 202 is in contact with the bolt 201, to the control unit 36.
　（第２判定部）
　第２判定部３５２は、算出部３２から受け付けたナット２０２の姿勢角、および座標変換部３３から受け付けたナット２０２の位置座標に基づき、ナット２０２の状態に異常があるか否かを判定する。以下、第２判定部３５２による判定を「第２判定」と称する。第２判定部３５２は、受け付けたナット２０２の位置座標がボルト中心座標系で表現された所定の座標領域（３次元領域）を逸脱している場合に、ナット２０２の位置座標に問題があると判定する。一方、第２判定部３５２は、受け付けたナット２０２の位置座標が座標領域内である場合に、ナット２０２の位置座標に問題がないと判定する。また、第２判定部３５２は、受け付けたナット２０２の姿勢角が所定の姿勢角しきい値以上である場合に、ナット２０２の姿勢角に問題があると判定する。一方、第２判定部３５２は、受け付けたナット２０２の姿勢角が姿勢角しきい値未満である場合に、ナット２０２の姿勢角に問題がないと判定する。第２判定部３５２は、ナット２０２の位置座標および姿勢角のうち１つ以上に問題があると判定した場合に、ナット２０２の状態に異常があると判定する。一方、第２判定部３５２は、ナット２０２の位置座標および姿勢角のいずれにも問題がないと判定した場合に、ナット２０２の状態に異常がないと判定する。なお、上記の座標領域および姿勢角しきい値は、例えば記憶部３９にあらかじめ記憶されている。第２判定部３５２は、記憶部３９に記憶されている座標領域および姿勢角しきい値を適時に参照することで上記第２判定を行う。第２判定部３５２は、第２判定の結果を制御部３６に送る。 (Second Judgment Unit)
The second determination unit 352 determines whether or not there is an abnormality in the state of the nut 202 based on the attitude angle of the nut 202 received from the calculation unit 32 and the position coordinates of the nut 202 received from the coordinate conversion unit 33. Hereinafter, the determination by the second determination unit 352 is referred to as the "second determination". When the position coordinates of the received nut 202 deviate from a predetermined coordinate area (three-dimensional area) expressed in the bolt-centered coordinate system, the second determination unit 352 determines that there is a problem with the position coordinates of the nut 202. On the other hand, when the position coordinates of the received nut 202 are within the coordinate area, the second determination unit 352 determines that there is no problem with the position coordinates of the nut 202. In addition, when the attitude angle of the received nut 202 is equal to or greater than a predetermined attitude angle threshold, the second determination unit 352 determines that there is a problem with the attitude angle of the nut 202. On the other hand, when the attitude angle of the received nut 202 is less than the attitude angle threshold, the second determination unit 352 determines that there is no problem with the attitude angle of the nut 202. When the second determination unit 352 determines that there is a problem with one or more of the position coordinates and the attitude angle of the nut 202, it determines that there is an abnormality in the state of the nut 202. On the other hand, when the second determination unit 352 determines that there is no problem with either the position coordinates or the attitude angle of the nut 202, it determines that there is no abnormality in the state of the nut 202. Note that the above coordinate area and attitude angle threshold value are stored in advance in, for example, the storage unit 39. The second determination unit 352 performs the above second determination by referring to the coordinate area and the attitude angle threshold value stored in the storage unit 39 at appropriate times. The second determination unit 352 sends the result of the second determination to the control unit 36.
　（第３判定部）
　第３判定部３５３は、取得部３１から受け付けた画像、算出部３２から受け付けたナット２０２の姿勢角、座標変換部３３から受け付けたナット２０２の位置座標、ならびに、補正部３４から受け付けた力データに基づき、ナット２０２の螺合が成功したか否かを判定する。すなわち、第３判定部３５３は、ナット２０２およびボルト２０１の螺合の成否を判定する。本明細書中における「螺合」とは、例えば、ナット２０２の雌ねじ部としてのナット孔２０２ｈ内における、端面２０２ｅに最も近い最初の一山が、ボルト２０１の露出した雄ねじ部２０１ａにおける、端面２０１ｅに最も近い最初の一山にはまった（かかった）状態などを意味する。本実施形態では、第３判定部３５３は、画像と、ナット２０２の位置座標および姿勢角と、力データとを含むデータセットが入力されると、ナット２０２およびボルト２０１の螺合の成否を示す判定結果を出力するように学習された学習済みモデルを用いて判定を行う。以下、第３判定部３５３による判定を「第３判定」と称し、第３判定部３５３が第３判定に用いる学習済みモデルを「第３学習済みモデル３９３」と称する。すなわち、第３学習済みモデル３９３は、画像と、ナット２０２の位置座標および姿勢角と、力データとを含むデータセットを入力要素とし、ナット２０２およびボルト２０１の螺合の成否を示す判定結果を出力要素としている。第３学習済みモデル３９３は、例えば記憶部３９にあらかじめ記憶されている。第３判定部３５３は、記憶部３９に記憶されている第３学習済みモデル３９３に上記データセットを入力することで出力された判定結果を第３判定の結果として取得する。 (Third Judgment Unit)
The third determination unit 353 determines whether or not the nut 202 has been successfully screwed based on the image received from the acquisition unit 31, the attitude angle of the nut 202 received from the calculation unit 32, the position coordinates of the nut 202 received from the coordinate conversion unit 33, and the force data received from the correction unit 34. That is, the third determination unit 353 determines whether or not the nut 202 and the bolt 201 have been screwed together. In this specification, "screwed together" means, for example, a state in which the first thread closest to the end face 202e in the nut hole 202h as the female thread portion of the nut 202 is engaged (engaged) with the first thread closest to the end face 201e in the exposed male thread portion 201a of the bolt 201. In this embodiment, when a data set including an image, the position coordinates and attitude angle of the nut 202, and the force data is input, the third determination unit 353 performs a determination using a trained model that has been trained to output a determination result indicating whether or not the nut 202 and the bolt 201 have been screwed together. Hereinafter, the judgment by the third judgment unit 353 is referred to as the "third judgment", and the trained model used by the third judgment unit 353 for the third judgment is referred to as the "third trained model 393". That is, the third trained model 393 uses a data set including an image, the position coordinates and attitude angle of the nut 202, and force data as input elements, and uses a judgment result indicating whether the nut 202 and the bolt 201 are screwed together as an output element. The third trained model 393 is stored in advance in, for example, the storage unit 39. The third judgment unit 353 inputs the above data set into the third trained model 393 stored in the storage unit 39, and acquires the output judgment result as the result of the third judgment.
　第３学習済みモデル３９３は、例えば、ディープニューラルネットワーク（ＤＮＮ：Deep Neural Network）などの深層学習モデル（教師あり学習モデル）である。第３学習済みモデル３９３は、上記データセットが入力されるとともに、入力されたデータセットに対するナット２０２の螺合の成否（螺合できている、あるいは螺合できていないと人間によって正確に判断された正解データ）が教示される学習ステップが、複数回（例えば数百回）繰り返されることで生成される（学習する）。なお、第３学習済みモデル３９３には、例えば、畳み込みニューラルネットワーク（ＣＮＮ）や回帰型ニューラルネットワーク（ＲＮＮ）などの深層学習モデルが用いられてよい。 The third trained model 393 is, for example, a deep learning model (supervised learning model) such as a deep neural network (DNN). The third trained model 393 is generated (learned) by repeating a learning step multiple times (for example, several hundred times) in which the above dataset is input and the successful or unsuccessful screwing of the nut 202 for the input dataset (correct answer data that is accurately determined by a human as being screwed or not screwed) is taught. Note that the third trained model 393 may be, for example, a deep learning model such as a convolutional neural network (CNN) or a recurrent neural network (RNN).
　第３判定部３５３は、ナット２０２およびボルト２０１の螺合が失敗したことを第３判定の結果が示す場合に、リトライ動作（後述）が必要であることを示すフラグ（ｉ＝１）を出力する。一方、第３判定部３５３は、ナット２０２およびボルト２０１の螺合が成功したことを第３判定の結果が示す場合に、リトライ動作が必要でないことを示すフラグ（ｉ＝０）を出力する。以下、第３判定部３５３が出力するリトライ動作の要否に関するフラグ（ｉ＝１または０）を「リトライフラグ」と称する。すなわち、第３判定部３５３が出力するリトライフラグは、第３判定の結果を示し、１または０の値である。第３判定部３５３は、第３判定の結果であるリトライフラグを制御部３６およびカウンタ部３８に送るとともに記憶部３９に記憶させる。なお、カウンタ部３８は、第３判定部３５３から受け付けたリトライフラグを受け付けるたびに順次加算していき、リトライフラグの加算結果を第６判定部３５６に送る。 If the result of the third judgment indicates that the screwing of the nut 202 and the bolt 201 has failed, the third judgment unit 353 outputs a flag (i=1) indicating that a retry operation (described later) is necessary. On the other hand, if the result of the third judgment indicates that the screwing of the nut 202 and the bolt 201 has been successful, the third judgment unit 353 outputs a flag (i=0) indicating that a retry operation is not necessary. Hereinafter, the flag (i=1 or 0) regarding the necessity of a retry operation output by the third judgment unit 353 is referred to as a "retry flag". In other words, the retry flag output by the third judgment unit 353 indicates the result of the third judgment and has a value of 1 or 0. The third judgment unit 353 sends the retry flag, which is the result of the third judgment, to the control unit 36 and the counter unit 38 and stores it in the memory unit 39. The counter unit 38 increments the retry flag each time it receives a retry flag from the third determination unit 353, and sends the incremented retry flag to the sixth determination unit 356.
　（第４判定部）
　第４判定部３５４は、取得部３１から受け付けた画像、算出部３２から受け付けたナット２０２の姿勢角、座標変換部３３から受け付けたナット２０２の位置座標、ならびに、補正部３４から受け付けた力データに基づき、ナット２０２の締結が成功したか否かを判定する。すなわち、第４判定部３５４は、ナット２０２およびボルト２０１の締結の成否を判定する。本明細書中における「締結」とは、例えば、ナット２０２の雌ねじ部としてのナット孔２０２ｈ内における、端面２０２ｅから最も遠い最後の一山が、ボルト２０１の露出した雄ねじ部２０１ａにおける、端面２０１ｅから最も遠い最後の一山にはまった状態などを意味する。本実施形態では、第４判定部３５４は、画像と、ナット２０２の位置座標および姿勢角と、力データとを含むデータセットが入力されると、ナット２０２およびボルト２０１の締結の成否を示す判定結果を出力するように学習された学習済みモデルを用いて判定を行う。以下、第４判定部３５４による判定を「第４判定」と称し、第４判定部３５４が第４判定に用いる学習済みモデルを「第４学習済みモデル３９４」と称する。すなわち、第４学習済みモデル３９４は、画像と、ナット２０２の位置座標および姿勢角と、力データとを含むデータセットを入力要素とし、ナット２０２およびボルト２０１の締結の成否を示す判定結果を出力要素としている。第４学習済みモデル３９４は、例えば記憶部３９にあらかじめ記憶されている。第４判定部３５４は、記憶部３９に記憶されている第４学習済みモデル３９４に上記データセットを入力することで出力された判定結果を第４判定の結果として取得する。 (Fourth Judgment Unit)
The fourth determination unit 354 determines whether the fastening of the nut 202 has been successful or not based on the image received from the acquisition unit 31, the attitude angle of the nut 202 received from the calculation unit 32, the position coordinates of the nut 202 received from the coordinate conversion unit 33, and the force data received from the correction unit 34. That is, the fourth determination unit 354 determines whether the fastening of the nut 202 and the bolt 201 has been successful or not. In this specification, "fastening" means, for example, a state in which the last thread farthest from the end face 202e in the nut hole 202h as the female thread portion of the nut 202 is fitted into the last thread farthest from the end face 201e in the exposed male thread portion 201a of the bolt 201. In this embodiment, when a data set including an image, the position coordinates and attitude angle of the nut 202, and the force data is input, the fourth determination unit 354 performs the determination using a trained model trained to output a determination result indicating whether the fastening of the nut 202 and the bolt 201 has been successful or not. Hereinafter, the judgment by the fourth judgment unit 354 is referred to as the "fourth judgment", and the trained model used by the fourth judgment unit 354 for the fourth judgment is referred to as the "fourth trained model 394". That is, the fourth trained model 394 uses a data set including an image, the position coordinates and attitude angle of the nut 202, and force data as input elements, and uses a judgment result indicating whether the nut 202 and the bolt 201 are fastened as an output element. The fourth trained model 394 is stored in advance in, for example, the storage unit 39. The fourth judgment unit 354 inputs the above data set into the fourth trained model 394 stored in the storage unit 39, and acquires the output judgment result as the result of the fourth judgment.
　第４学習済みモデル３９４は、例えば、ディープニューラルネットワーク（ＤＮＮ）などの深層学習モデル（教師あり学習モデル）である。第４学習済みモデル３９４は、上記データセットが入力されるとともに、入力されたデータセットに対するナット２０２の締結の成否（締結できている、あるいは締結できていないと人間によって正確に判断された正解データ）が教示される学習ステップが、複数回（例えば数百回）繰り返されることで生成される（学習する）。なお、第４学習済みモデル３９４には、例えば、畳み込みニューラルネットワーク（ＣＮＮ）や回帰型ニューラルネットワーク（ＲＮＮ）などの深層学習モデルが用いられてよい。 The fourth trained model 394 is, for example, a deep learning model (supervised learning model) such as a deep neural network (DNN). The fourth trained model 394 is generated (learned) by repeating a learning step multiple times (for example, several hundred times) in which the above dataset is input and the success or failure of fastening the nut 202 for the input dataset (correct answer data that is accurately determined by a human as being fastened or not fastened) is taught. Note that the fourth trained model 394 may be, for example, a deep learning model such as a convolutional neural network (CNN) or a recurrent neural network (RNN).
　第４判定部３５４は、ナット２０２およびボルト２０１の締結が失敗したことを第４判定の結果が示す場合に、リトライ動作が必要であることを示すフラグ（ｉ＝１）を出力する。一方、第４判定部３５４は、ナット２０２およびボルト２０１の締結が成功したことを第４判定の結果が示す場合に、リトライ動作が必要でないことを示すフラグ（ｉ＝０）を出力する。以下、第４判定部３５４が出力するリトライ動作の要否に関するフラグ（ｉ＝１または０）を、第３判定部３５３が出力するリトライフラグと区別することなく「リトライフラグ」と称する。すなわち、第４判定部３５４が出力するリトライフラグは、第４判定の結果を示し、１または０の値である。第４判定部３５４は、第４判定の結果であるリトライフラグを第５判定部３５５、制御部３６、およびカウンタ部３８に送るとともに記憶部３９に記憶させる。なお、カウンタ部３８は、第４判定部３５４から受け付けたリトライフラグを受け付けるたびに順次加算していき、リトライフラグの加算結果を第６判定部３５６に送る。 The fourth judgment unit 354 outputs a flag (i=1) indicating that a retry operation is necessary when the result of the fourth judgment indicates that the tightening of the nut 202 and the bolt 201 has failed. On the other hand, the fourth judgment unit 354 outputs a flag (i=0) indicating that a retry operation is not necessary when the result of the fourth judgment indicates that the tightening of the nut 202 and the bolt 201 has been successful. Hereinafter, the flag (i=1 or 0) regarding the necessity of a retry operation output by the fourth judgment unit 354 will be referred to as a "retry flag" without distinguishing it from the retry flag output by the third judgment unit 353. In other words, the retry flag output by the fourth judgment unit 354 indicates the result of the fourth judgment and has a value of 1 or 0. The fourth judgment unit 354 sends the retry flag, which is the result of the fourth judgment, to the fifth judgment unit 355, the control unit 36, and the counter unit 38, and stores it in the memory unit 39. The counter unit 38 increments the retry flag each time it receives a retry flag from the fourth determination unit 354, and sends the incremented retry flag to the sixth determination unit 356.
　（第５判定部）
　第５判定部３５５は、第４判定部３５４から受け付けた第４判定の結果がナット２０２の締結が成功したことを示す場合（ｉ＝０）に、カラー２０２ｂが破断して切断（カラー切り）されたか否かを判定する。具体的には、第５判定部３５５は、例えば補正部３４から受け付けた力データに基づき、ナット２０２のカラー２０２ｂがナット本体２０２ａから分離しているか否かを判定する。以下、第５判定部３５５による判定を「第５判定」と称する。第５判定部３５５は、例えば、受け付けた力データに含まれる接触力が所定の接触力の大きさを示す第２しきい値未満である場合に、カラー２０２ｂが切断されたと判定する。一方、第５判定部３５５は、取得した力データに含まれる接触力が第２しきい値以上である場合に、カラー２０２ｂが切断されていないと判定する。 (Fifth Judgment Unit)
The fifth judgment unit 355 judges whether or not the collar 202b has been broken and cut (collar cut) when the result of the fourth judgment received from the fourth judgment unit 354 indicates that the fastening of the nut 202 has been successful (i=0). Specifically, the fifth judgment unit 355 judges whether or not the collar 202b of the nut 202 has been separated from the nut body 202a based on, for example, the force data received from the correction unit 34. Hereinafter, the judgment by the fifth judgment unit 355 is referred to as the "fifth judgment". For example, the fifth judgment unit 355 judges that the collar 202b has been cut when the contact force included in the received force data is less than a second threshold value indicating the magnitude of a predetermined contact force. On the other hand, the fifth judgment unit 355 judges that the collar 202b has not been cut when the contact force included in the acquired force data is equal to or greater than the second threshold value.
　なお、第２しきい値は、例えば記憶部３９にあらかじめ記憶されている。第５判定部３５５は、記憶部３９に記憶されている第２しきい値を適時に参照することで上記第５判定を行う。また、第５判定部３５５は、上記の判定動作に代えて、取得部３１から受け付けた画像、算出部３２から受け付けたナット２０２の姿勢角、座標変換部３３から受け付けたナット２０２の位置座標、ならびに、補正部３４から受け付けた力データに基づき、ナット２０２の締結が成功したか否かを判定してもよい。この場合、第５判定部３５５は、画像と、ナット２０２の位置座標および姿勢角と、力データとを含むデータセットが入力されると、カラー２０２ｂが切断されたか否かを示す判定結果を出力するように学習された第５学習済みモデル３９６などを用いて判定を行ってもよい。第５学習済みモデル３９６は、例えば記憶部３９にあらかじめ記憶されている。第５判定部３５５は、第５判定の結果を制御部３６に送るとともに記憶部３９に記憶させる。 The second threshold value is stored in advance in, for example, the storage unit 39. The fifth judgment unit 355 performs the fifth judgment by referring to the second threshold value stored in the storage unit 39 at appropriate times. Instead of the above judgment operation, the fifth judgment unit 355 may determine whether or not the fastening of the nut 202 has been successful based on the image received from the acquisition unit 31, the attitude angle of the nut 202 received from the calculation unit 32, the position coordinates of the nut 202 received from the coordinate conversion unit 33, and the force data received from the correction unit 34. In this case, the fifth judgment unit 355 may perform the judgment using a fifth trained model 396 that has been trained to output a judgment result indicating whether or not the collar 202b has been cut when a data set including the image, the position coordinates and attitude angle of the nut 202, and the force data is input. The fifth trained model 396 is stored in advance in, for example, the storage unit 39. The fifth judgment unit 355 sends the result of the fifth judgment to the control unit 36 and stores it in the storage unit 39.
　（第６判定部）
　第６判定部３５６は、カウンタ部３８から受け付けた各種のフラグの加算結果に基づき、所定の判定を行う。以下、第６判定部３５６による判定を「第６判定」と称する。第６判定部３５６は、カウンタ部３８から受け付けた取得フラグの加算結果に基づき、すべてのボルト２０１にナット２０２を締結したか否かを判定する。具体的には、第６判定部３５６は、カウンタ部３８から受け付けた取得フラグの加算結果が施工対象物２００に設けられた施工の対象となるボルト２０１の数である場合に、すべてのボルト２０１にナット２０２を締結したと判定する。一方、第６判定部３５６は、カウンタ部３８から受け付けた取得フラグの加算結果が施工対象物２００に設けられたボルト２０１の数未満である場合に、すべてのボルト２０１にナット２０２を締結していないと判定する。施工対象物２００に設けられたボルト２０１の数は、例えば記憶部３９のＣＡＤデータ３９５などにあらかじめ記憶されており、整数（例えば数千から数万）である。 (Sixth Judgment Unit)
The sixth determination unit 356 performs a predetermined determination based on the addition results of various flags received from the counter unit 38. Hereinafter, the determination by the sixth determination unit 356 is referred to as the "sixth determination". The sixth determination unit 356 determines whether or not the nuts 202 have been fastened to all the bolts 201 based on the addition results of the acquisition flags received from the counter unit 38. Specifically, the sixth determination unit 356 determines that the nuts 202 have been fastened to all the bolts 201 when the addition result of the acquisition flags received from the counter unit 38 is the number of bolts 201 to be installed on the construction target 200. On the other hand, the sixth determination unit 356 determines that the nuts 202 have not been fastened to all the bolts 201 when the addition result of the acquisition flags received from the counter unit 38 is less than the number of bolts 201 installed on the construction target 200. The number of bolts 201 provided on the construction target object 200 is stored in advance, for example, in the CAD data 395 of the memory unit 39, and is an integer (for example, several thousand to tens of thousands).
　また、第６判定部３５６は、カウンタ部３８から受け付けたリトライフラグの加算結果に基づき、所定回数リトライしたか否かを判定する。具体的には、第６判定部３５６は、カウンタ部３８から受け付けたリトライフラグの加算結果が所定のリトライしきい値以上である場合に、所定回数リトライしたと判定する。一方、第６判定部３５６は、カウンタ部３８から受け付けたリトライフラグの加算結果がリトライしきい値未満である場合に、所定回数リトライしていないと判定する。リトライしきい値は、例えば記憶部３９にあらかじめ記憶されており、整数である。第６判定部３５６は、記憶部３９に記憶されているリトライしきい値を適時に参照することで上記第６判定を行う。第６判定部３５６は、第６判定の結果を学習部３７および制御部３６に送る。 The sixth determination unit 356 also determines whether or not a predetermined number of retries have been performed based on the result of the retry flag addition received from the counter unit 38. Specifically, the sixth determination unit 356 determines that a predetermined number of retries have been performed when the result of the retry flag addition received from the counter unit 38 is equal to or greater than a predetermined retry threshold. On the other hand, the sixth determination unit 356 determines that a predetermined number of retries have not been performed when the result of the retry flag addition received from the counter unit 38 is less than the retry threshold. The retry threshold is, for example, stored in advance in the memory unit 39 and is an integer. The sixth determination unit 356 performs the sixth determination by referring to the retry threshold stored in the memory unit 39 at appropriate times. The sixth determination unit 356 sends the result of the sixth determination to the learning unit 37 and the control unit 36.
（制御部）
　制御部３６は、判定部３５から受け付けた各種の判定の結果に基づき、エンドエフェクタ１２の動きを制御する。制御部３６は、例えば、第１制御部３６１と、第２制御部３６２と、第３制御部３６３とを有している。 (Control Unit)
The control unit 36 controls the movement of the end effector 12 based on the results of various determinations received from the determination unit 35. The control unit 36 has, for example, a first control unit 361, a second control unit 362, and a third control unit 363.
　（第１制御部）
　第１制御部３６１は、ナット２０２がボルト２０１に接触していると第１判定の結果が示す場合（ナット２０２が第２位置に位置する場合）に、算出部３２から受け付けたナット２０２の姿勢角と、座標変換部３３から受け付けたナット２０２の位置座標と、補正部３４から受け付けた力データとを含むデータセットに基づき、ボルト２０１に対するナット２０２の締結動作を実行する。ここでいう「締結動作」は、ナット２０２が第２位置に位置した状態で、ナット２０２およびボルト２０１が螺合するようにナット２０２の位置座標および姿勢角を変化させること、ならびに、ナット２０２およびボルト２０１が螺合した状態（ナット２０２が第２位置に位置した状態）で、ナット２０２およびボルト２０１が締結するようにナット２０２を回転させることなどを含む。具体的には、第１制御部３６１は、学習済みモデルを用いて、エンドエフェクタ１２の移動および回転を制御する。ここでいう「エンドエフェクタ１２の移動」は、例えばアーム１１が有する各々の関節部１１０のモータ１１７の関節角を制御した結果、エンドエフェクタ１２およびエンドエフェクタ１２に保持されたナット２０２が移動することを意味し、「エンドエフェクタ１２の回転」は、例えばエンドエフェクタ１２の本体部１２０の締結用モータ１２０ａを制御した結果、ナット２０２が回転することを意味する。以下、第１制御部３６１がエンドエフェクタ１２の上記制御に用いる学習済みモデルを「第１学習済みモデル３９１」と称する。第１学習済みモデル３９１は、例えば記憶部３９にあらかじめ記憶されている。 (First control unit)
When the result of the first determination indicates that the nut 202 is in contact with the bolt 201 (when the nut 202 is located at the second position), the first control unit 361 executes a fastening operation of the nut 202 with respect to the bolt 201 based on a data set including the attitude angle of the nut 202 received from the calculation unit 32, the position coordinates of the nut 202 received from the coordinate conversion unit 33, and the force data received from the correction unit 34. The "fastening operation" here includes changing the position coordinates and attitude angle of the nut 202 so that the nut 202 and the bolt 201 are screwed together when the nut 202 is located at the second position, and rotating the nut 202 so that the nut 202 and the bolt 201 are fastened together when the nut 202 and the bolt 201 are screwed together (when the nut 202 is located at the second position). Specifically, the first control unit 361 controls the movement and rotation of the end effector 12 using the learned model. Here, "movement of the end effector 12" means, for example, control of the joint angle of the motor 117 of each joint unit 110 of the arm 11, resulting in movement of the end effector 12 and the nut 202 held by the end effector 12, and "rotation of the end effector 12" means, for example, control of the fastening motor 120a of the main body unit 120 of the end effector 12, resulting in rotation of the nut 202. Hereinafter, the trained model used by the first control unit 361 for the above-mentioned control of the end effector 12 will be referred to as the "first trained model 391." The first trained model 391 is stored in advance in, for example, the storage unit 39.
　第１学習済みモデル３９１は、第１時刻におけるナット２０２の位置座標および姿勢角と、第１時刻における接触力およびトルクとを含むデータセットが入力されると、ナット２０２がボルト２０１に締結された状態を示すナット２０２の目標位置座標および目標姿勢角に近づいた、第１時刻よりも後の第２時刻におけるナット２０２の位置座標および姿勢角を示す指令値を出力するように学習されている。すなわち、第１学習済みモデル３９１は、第１時刻におけるナット２０２の位置座標および姿勢角と、第１時刻における接触力およびトルクとを含むデータセットを入力要素とし、第２時刻におけるナット２０２の位置座標および姿勢角を示す指令値を出力要素としている。第１時刻と第２時刻との間の時間間隔は、例えば、上記第１所定周期である。ナット２０２の目標位置座標および目標姿勢角は、例えば、ナット２０２の雌ねじ部と、施工の対象となるボルト２０１の雄ねじ部２０１ａとが互いに正確に噛み合って締結が完了した状態を示すナット２０２の位置座標および姿勢角を意味する。本実施形態では、第１制御部３６１は、記憶部３９に記憶されている第１学習済みモデル３９１に上記データセットを入力することで出力された指令値に基づき、制御対象装置Ｓとしての各々の関節部１１０のモータ１１７の関節角、およびエンドエフェクタ１２の締結用モータ１２０ａを制御して、ナット２０２の位置座標および姿勢角を制御する。第１学習済みモデル３９１は、例えば、ディープニューラルネットワーク（ＤＮＮ）などの深層学習モデル（教師あり学習モデル）である。なお、第１学習済みモデル３９１には、例えば、畳み込みニューラルネットワーク（ＣＮＮ）や回帰型ニューラルネットワーク（ＲＮＮ）などの深層学習モデルが用いられてよい。また、第１制御部３６１は、経時的に変化させた各々の関節部１１０の関節角、およびエンドエフェクタ１２の締結用モータ１２０ａの回転速度（ナット２０２の回転に伴うねじ送り速度）などを毎時刻で記憶部３９に逐次記憶させる。 The first trained model 391 is trained to output a command value indicating the position coordinates and the posture angle of the nut 202 at a second time after the first time when the nut 202 approaches the target position coordinates and the target posture angle of the nut 202, which indicates the state in which the nut 202 is fastened to the bolt 201, when a data set including the position coordinates and the posture angle of the nut 202 at the first time and the contact force and the torque at the first time is input. That is, the first trained model 391 uses the data set including the position coordinates and the posture angle of the nut 202 at the first time and the contact force and the torque at the first time as input elements, and uses the command value indicating the position coordinates and the posture angle of the nut 202 at the second time as output elements. The time interval between the first time and the second time is, for example, the first predetermined period. The target position coordinates and the target posture angle of the nut 202 mean, for example, the position coordinates and the posture angle of the nut 202, which indicate the state in which the female thread portion of the nut 202 and the male thread portion 201a of the bolt 201 to be fastened are accurately meshed with each other and the fastening is completed. In this embodiment, the first control unit 361 controls the joint angles of the motors 117 of the joints 110 as the controlled device S and the fastening motor 120a of the end effector 12 based on the command values output by inputting the above data set into the first trained model 391 stored in the storage unit 39, thereby controlling the position coordinates and attitude angle of the nut 202. The first trained model 391 is, for example, a deep learning model (supervised learning model) such as a deep neural network (DNN). Note that, for example, a deep learning model such as a convolutional neural network (CNN) or a regression neural network (RNN) may be used for the first trained model 391. In addition, the first control unit 361 sequentially stores the joint angles of the joints 110 that have changed over time, the rotation speed of the fastening motor 120a of the end effector 12 (the screw feed speed associated with the rotation of the nut 202), and the like in the storage unit 39 at every time.
　（第２制御部）
　第２制御部３６２は、取得部３１から受け付けた画像と、算出部３２から受け付けたナット２０２の姿勢角と、座標変換部３３から受け付けたナット２０２の位置座標とを含むデータセットに基づき、エンドエフェクタ１２の移動および回転を制御する。具体的には、第２制御部３６２は、学習済みモデルを用いて、エンドエフェクタ１２の移動および回転を制御して、ナット２０２がボルト２０１に接触していない第１位置に位置するナット２０２を、ナット２０２がボルト２０１に接触した第２位置に位置させる。すなわち、第２制御部３６２は、第１位置に位置するナット２０２を、施工の対象となるボルト２０１に外部からアプローチさせる。以下、第２制御部３６２がエンドエフェクタ１２の制御に用いる学習済みモデルを「第２学習済みモデル３９２」と称する。第２学習済みモデル３９２は、例えば記憶部３９にあらかじめ記憶されている。 (Second control unit)
The second control unit 362 controls the movement and rotation of the end effector 12 based on a data set including the image received from the acquisition unit 31, the attitude angle of the nut 202 received from the calculation unit 32, and the position coordinates of the nut 202 received from the coordinate conversion unit 33. Specifically, the second control unit 362 controls the movement and rotation of the end effector 12 using the trained model to position the nut 202 located at a first position where the nut 202 is not in contact with the bolt 201 to a second position where the nut 202 is in contact with the bolt 201. That is, the second control unit 362 makes the nut 202 located at the first position approach the bolt 201 to be worked from the outside. Hereinafter, the trained model used by the second control unit 362 to control the end effector 12 is referred to as the "second trained model 392". The second trained model 392 is stored in advance in the storage unit 39, for example.
　第２学習済みモデル３９２は、第１位置で取得された第３時刻における画像と、第３時刻におけるナット２０２の位置座標および姿勢角とを含むデータセットが入力されると、第２位置にナット２０２が近づいた、第３時刻よりも後の第４時刻におけるナット２０２の位置座標および姿勢角を示す指令値を出力するように学習されている。すなわち、第２学習済みモデル３９２は、第１位置で取得された第３時刻における画像と、第３時刻におけるナット２０２の位置座標および姿勢角とを含むデータセットを入力要素とし、第４時刻におけるナット２０２の位置座標および姿勢角を示す指令値を出力要素としている。第３時刻と第４時刻との間の時間間隔は、例えば、上記第２所定周期である。本実施形態では、第２制御部３６２は、記憶部３９に記憶されている第２学習済みモデル３９２に上記データセットを入力することで出力された指令値に基づき、制御対象装置Ｓとしての各々の関節部１１０のモータ１１７の関節角を制御して、ナット２０２をボルト２０１に近づける（アプローチさせる）。第２学習済みモデル３９２は、例えば、ディープニューラルネットワーク（ＤＮＮ）などの深層学習モデル（教師あり学習モデル）である。なお、第２学習済みモデル３９２には、例えば、畳み込みニューラルネットワーク（ＣＮＮ）や回帰型ニューラルネットワーク（ＲＮＮ）などの深層学習モデルが用いられてよい。 The second trained model 392 is trained to output a command value indicating the position coordinates and attitude angle of the nut 202 at a fourth time, which is after the third time and when the nut 202 approaches the second position, when a data set including an image at the third time acquired at the first position and the position coordinates and attitude angle of the nut 202 at the third time is input. That is, the second trained model 392 uses an image at the third time acquired at the first position and a data set including the position coordinates and attitude angle of the nut 202 at the third time as input elements, and uses a command value indicating the position coordinates and attitude angle of the nut 202 at the fourth time as output elements. The time interval between the third time and the fourth time is, for example, the second predetermined period. In this embodiment, the second control unit 362 controls the joint angle of the motor 117 of each joint unit 110 as the controlled device S based on the command value output by inputting the above data set into the second trained model 392 stored in the memory unit 39, to bring the nut 202 closer to (approach) the bolt 201. The second trained model 392 is, for example, a deep learning model (supervised learning model) such as a deep neural network (DNN). Note that, for example, a deep learning model such as a convolutional neural network (CNN) or a recurrent neural network (RNN) may be used as the second trained model 392.
　（第３制御部）
　第３制御部３６３は、ナット２０２およびボルト２０１の螺合が失敗したことを示す第３判定の結果（ｉ＝１）を受け付けた場合、または、ナット２０２およびボルト２０１の締結が失敗したことを示す第４判定の結果（ｉ＝１）を受け付けた場合に、リトライ動作を実行する。具体的には、第３制御部３６３は、過去の所定時刻から第３判定または第４判定の結果を受け付けた時刻に至るまでに行われたエンドエフェクタ１２の制御（後述の締結動作ステップＳ１０６，Ｓ４など）をさかのぼるように逆順に実施する。ここでいう「過去の所定時刻」は、例えば、エンドエフェクタ１２の制御が開始された時刻（後述の締結動作ステップＳ１０６，Ｓ４などが開始された時刻）を意味する。第３制御部３６３は、記憶部３９に記憶された各々の関節部１１０の関節角の時系列上のデータ、および、エンドエフェクタ１２の締結用モータ１２０ａの回転速度（ナット２０２の回転に伴ったねじ送り速度）などを適時に参照しながら、エンドエフェクタ１２の移動および回転を逆順に制御する。 (Third control unit)
The third control unit 363 executes a retry operation when it receives a result of the third judgment (i=1) indicating that the screwing of the nut 202 and the bolt 201 has failed, or when it receives a result of the fourth judgment (i=1) indicating that the fastening of the nut 202 and the bolt 201 has failed. Specifically, the third control unit 363 executes the control of the end effector 12 (fastening operation steps S106, S4, etc. described later) performed from a predetermined time in the past until the time when the result of the third judgment or the fourth judgment is received in reverse order. The "predetermined time in the past" here means, for example, the time when the control of the end effector 12 was started (the time when the fastening operation steps S106, S4, etc. described later were started). The third control unit 363 controls the movement and rotation of the end effector 12 in reverse order while timely referring to the time-series data of the joint angles of each joint unit 110 stored in the memory unit 39, the rotational speed of the fastening motor 120a of the end effector 12 (the screw feed speed accompanying the rotation of the nut 202), etc.
（学習部）
　学習部３７は、第６判定部３５６から受け付けた第６判定の結果が、すべてのボルト２０１にナット２０２を締結したことを示す場合に、記憶部３９に記憶されている各種のデータに基づき、第１学習済みモデル３９１におけるパラメータの重み値を修正する。学習部３７は、例えば誤差逆伝播法などを用いて、第１学習済みモデル３９１におけるパラメータの重み値を修正する（学習させる）。学習部３７は、第１学習済みモデル３９１が持つパラメータを修正した際、修正した旨を示す修正フラグをカウンタ部３８に送る。修正フラグは、例えば１である。 (Learning Department)
When the result of the sixth judgment received from the sixth judgment unit 356 indicates that the nuts 202 have been fastened to all the bolts 201, the learning unit 37 modifies the weight values of the parameters in the first trained model 391 based on various data stored in the memory unit 39. The learning unit 37 modifies (learns) the weight values of the parameters in the first trained model 391, for example, by using an error backpropagation method or the like. When the learning unit 37 modifies the parameters of the first trained model 391, it sends a modification flag indicating that the modification has been made to the counter unit 38. The modification flag is, for example, 1.
　なお、カウンタ部３８は、学習部３７から受け付けた修正フラグを受け付けるたびに順次加算していき、修正フラグの加算結果を第６判定部３５６に送る（返す）。第６判定部３５６は、カウンタ部３８から受け付けた修正フラグの加算結果と所定数値とを比較する。第６判定部３５６は、修正フラグの加算結果が所定数値以上である場合に、所定回数修正したと判定する。一方、第６判定部３５６は、修正フラグの加算結果が所定数値未満である場合に、所定回数修正していないと判定する。所定数値は、例えば記憶部３９にあらかじめ記憶されており、整数である。第６判定部３５６は、記憶部３９に記憶されている所定数値を適時に参照することで当該第６判定を行う。 Note that the counter unit 38 adds each time it receives a correction flag from the learning unit 37, and sends (returns) the addition result of the correction flag to the sixth judgment unit 356. The sixth judgment unit 356 compares the addition result of the correction flag received from the counter unit 38 with a predetermined value. If the addition result of the correction flag is equal to or greater than the predetermined value, the sixth judgment unit 356 determines that the predetermined number of corrections has been made. On the other hand, if the addition result of the correction flag is less than the predetermined value, the sixth judgment unit 356 determines that the predetermined number of corrections has not been made. The predetermined value is, for example, stored in advance in the memory unit 39, and is an integer. The sixth judgment unit 356 makes the sixth judgment by referring to the predetermined value stored in the memory unit 39 at the appropriate time.
（第１学習済みモデルの教示方法）
　以下、第１制御部３６１が用いる第１学習済みモデル３９１の学習ステップ（教示のされ方）について図１１を参照して説明する。図１１は、第１学習済みモデル３９１の学習ステップの一例を示すフローチャートである。なお、以下では説明の便宜上、学習がなされる前のモデルであっても区別することなく「学習済みモデル」と称する。 (Method of teaching first trained model)
Hereinafter, the learning steps (how the model is taught) of the first trained model 391 used by the first control unit 361 will be described with reference to Fig. 11. Fig. 11 is a flowchart showing an example of the learning steps of the first trained model 391. For ease of explanation, the model before learning will be referred to as the "trained model" without distinction.
　はじめに、取得部３１は、直近の施工の対象となるボルト２０１のデータを、記憶部３９にあらかじめ記憶されているＣＡＤデータ３９５などから取得する（ステップＳ１０１）。次いで、取得部３１は、ナット２０２の既知位置座標および既知姿勢角を取得する（ステップＳ１０２）。ここでいう取得部３１が取得するナット２０２の「既知位置座標および既知姿勢角」は、例えば記憶部３９によってあらかじめ記憶されているばらつき生成モデル３９７（図７参照）から取得される。ばらつき生成モデル３９７は、取得部３１によって参照された場合に、ボルト２０１の端面２０１ｅにおける任意の位置座標（第２位置）および姿勢角を、例えば正規分布などの確率分布にしたがってばらつかせ（誤差をあえてランダムに与えた状態）、取得部３１に出力する。ばらつき生成モデル３９７が平均とする位置座標は、例えば、ボルト２０１の端面２０１ｅの中心位置座標であり、ばらつき生成モデル３９７が平均とする姿勢角は、例えば、回転軸線Ａｒとボルト２０１の中心線Ａｒ２とが一直線に一致した時の姿勢角である。ボルト２０１の端面２０１ｅの中心位置座標は、例えば、ボルト中心座標系（Ｘｂ，Ｙｂ，Ｚｂ）で表現されている。つまり、取得部３１は、ばらつき生成モデル３９７を参照することでナット２０２の既知位置座標および既知姿勢角を取得する。次いで、第２制御部３６２は、第２学習済みモデル３９２を用いて、対象となるボルト２０１にナット２０２をアプローチさせる（ステップＳ１０３）。なお、ステップＳ１０３では、ボルト２０１にナット２０２をアプローチさせるに当たり、第２学習済みモデル３９２を用いなくてもよい。この場合、制御部３６は、例えば記憶部３９にあらかじめ記憶されている所定のアプローチ動作などにしたがって、対象となるボルト２０１にナット２０２をアプローチさせてもよい。次いで、第１判定部３５１は、ナット２０２がボルト２０１に接触したか否かを判定する第１判定を行う（ステップＳ１０４）。ナット２０２がボルト２０１に接触したことを第１判定の結果が示す場合（ステップＳ１０４：ＹＥＳ）、制御部３６による制御またはオペレータの手動などによって、ナット２０２の位置座標および姿勢角をステップＳ１０２で取得されたナット２０２の既知位置座標および既知姿勢角に修正する（ステップＳ１０５）。一方、ナット２０２がボルト２０１に接触していないことを第１判定の結果が示す場合（ステップＳ１０４：ＮＯ）、ステップＳ１０３の処理に戻る。ステップＳ１０５に次いで、締結動作ステップＳ１０６を実行する。 First, the acquisition unit 31 acquires data on the bolt 201 to be installed immediately from the CAD data 395 pre-stored in the memory unit 39, etc. (step S101). Next, the acquisition unit 31 acquires the known position coordinates and known attitude angle of the nut 202 (step S102). The "known position coordinates and known attitude angle" of the nut 202 acquired by the acquisition unit 31 here is acquired, for example, from a variation generation model 397 (see FIG. 7) pre-stored in the memory unit 39. When referenced by the acquisition unit 31, the variation generation model 397 varies any position coordinates (second position) and attitude angle on the end face 201e of the bolt 201 according to a probability distribution such as a normal distribution (a state in which errors are intentionally given randomly), and outputs them to the acquisition unit 31. The position coordinates that the variation generation model 397 averages are, for example, the central position coordinates of the end face 201e of the bolt 201, and the attitude angle that the variation generation model 397 averages are, for example, the attitude angle when the rotation axis Ar and the center line Ar2 of the bolt 201 coincide in a straight line. The central position coordinates of the end face 201e of the bolt 201 are expressed, for example, in the bolt-center coordinate system (Xb, Yb, Zb). That is, the acquisition unit 31 acquires the known position coordinates and known attitude angle of the nut 202 by referring to the variation generation model 397. Next, the second control unit 362 uses the second trained model 392 to make the nut 202 approach the target bolt 201 (step S103). Note that, in step S103, the second trained model 392 does not need to be used to make the nut 202 approach the bolt 201. In this case, the control unit 36 may cause the nut 202 to approach the target bolt 201 according to, for example, a predetermined approach operation stored in advance in the storage unit 39. Next, the first determination unit 351 performs a first determination to determine whether the nut 202 has contacted the bolt 201 (step S104). If the result of the first determination indicates that the nut 202 has contacted the bolt 201 (step S104: YES), the position coordinates and attitude angle of the nut 202 are corrected to the known position coordinates and known attitude angle of the nut 202 acquired in step S102 under the control of the control unit 36 or manually by the operator (step S105). On the other hand, if the result of the first determination indicates that the nut 202 has not contacted the bolt 201 (step S104: NO), the process returns to step S103. Following step S105, a fastening operation step S106 is executed.
　締結動作ステップＳ１０６では、制御部３６は、あらかじめ定められた手順にしたがったナット２０２の締結動作を開始する。本実施形態では、あらかじめ定められた手順にしたがったナット２０２の締結動作を「教示動作」と称する。ここでいう「あらかじめ定められた手順」とは、ナット２０２の既知位置座標（第２位置）および既知姿勢角から、ナット２０２の目標位置座標および目標姿勢角に転じさせるためのナット２０２の位置座標および姿勢角を示す指令値を時系列で示したデータなどを意味している。当該データは、例えば記憶部３９にあらかじめ記憶されている。なお、あらかじめ定められた手順は、例えば、締結動作の開始時刻におけるナット２０２の既知位置座標および既知姿勢角と、締結動作の開始時刻よりも後の複数の時刻と、各々の時刻に対応するナット２０２の位置座標および姿勢角の指令値とが互いに関連付いた対応関係情報（関数など）であってもよい。制御部３６は、あらかじめ定められた手順にしたがって、エンドエフェクタ１２の移動および回転を制御する。制御部３６がエンドエフェクタ１２を制御する教示動作中、取得部３１、算出部３２、座標変換部３３、および補正部３４は、処理の結果を記憶部３９に逐次記憶させる。教示動作中に記憶部３９に記憶された取得部３１、算出部３２、座標変換部３３、および補正部３４の処理の結果は、第１学習済みモデル３９１の学習に用いる教師データセットとされる。あらかじめ定められた手順に含まれる最後の指令値に基づき、制御部３６がエンドエフェクタ１２を制御し終えた場合に、締結動作ステップＳ１０６が終了する。 In the fastening operation step S106, the control unit 36 starts the fastening operation of the nut 202 according to a predetermined procedure. In this embodiment, the fastening operation of the nut 202 according to the predetermined procedure is called a "teaching operation". The "predetermined procedure" here means data showing command values indicating the position coordinates and attitude angle of the nut 202 in a time series for converting the known position coordinates (second position) and known attitude angle of the nut 202 to the target position coordinates and target attitude angle of the nut 202. The data is stored in advance in the storage unit 39, for example. The predetermined procedure may be, for example, correspondence information (such as a function) in which the known position coordinates and known attitude angle of the nut 202 at the start time of the fastening operation, multiple times after the start time of the fastening operation, and the command values of the position coordinates and attitude angles of the nut 202 corresponding to each time are associated with each other. The control unit 36 controls the movement and rotation of the end effector 12 according to the predetermined procedure. During the teaching operation in which the control unit 36 controls the end effector 12, the acquisition unit 31, the calculation unit 32, the coordinate conversion unit 33, and the correction unit 34 sequentially store the results of the processing in the memory unit 39. The results of the processing by the acquisition unit 31, the calculation unit 32, the coordinate conversion unit 33, and the correction unit 34 stored in the memory unit 39 during the teaching operation are used as a teacher data set used for learning the first trained model 391. When the control unit 36 finishes controlling the end effector 12 based on the final command value included in the predetermined procedure, the fastening operation step S106 ends.
　締結動作ステップＳ１０６に次いで、第４判定部３５４は、ナット２０２の締結が成功したか否かを判定する第４判定を行う（ステップＳ１０７）。ナット２０２の締結が成功したことを第４判定の結果が示す場合（ステップＳ１０７：ＹＥＳ）、第６判定部３５６は、すべてのボルト２０１にナット２０２を締結したか否かの判定を行う（ステップＳ１０８）。一方、ナット２０２の締結が失敗したことを第４判定の結果が示す場合（ステップＳ１０７：ＮＯ）、リトライ動作ステップＳ１１１を実行する。リトライ動作ステップＳ１１１では、第３制御部３６３は、上述のリトライ動作を行う。リトライ動作が終了した場合、ステップＳ１０５の処理に戻る。 Following the fastening operation step S106, the fourth judgment unit 354 performs a fourth judgment to judge whether or not the fastening of the nut 202 was successful (step S107). If the result of the fourth judgment indicates that the fastening of the nut 202 was successful (step S107: YES), the sixth judgment unit 356 performs a judgment as to whether or not the nuts 202 have been fastened to all of the bolts 201 (step S108). On the other hand, if the result of the fourth judgment indicates that the fastening of the nut 202 was unsuccessful (step S107: NO), a retry operation step S111 is executed. In the retry operation step S111, the third control unit 363 performs the retry operation described above. When the retry operation is completed, the process returns to step S105.
　ここで、ステップＳ１０８の判定の処理の説明に戻り、すべてのボルト２０１にナット２０２を締結したことを第６判定の結果が示す場合（ステップＳ１０８：ＹＥＳ）、締結動作ステップＳ１０６で取得された各種のデータに基づき、学習部３７は、第１学習済みモデル３９１が持つパラメータの重み値を修正する（ステップＳ１０９）。一方、すべてのボルト２０１にナット２０２を締結していないと第６判定の結果が示す場合（ステップＳ１０８：ＮＯ）、ステップＳ１０１の処理に進む。ステップＳ１０９の処理に次いで、第６判定部３５６は、第１学習済みモデル３９１が持つパラメータの重み値を学習部３７が所定回数に亘って修正したか否かを判定する（ステップＳ１１０）。第６判定部３５６の判定の結果が、所定回数修正したことを示す場合（ステップＳ１１０：ＹＥＳ）、学習ステップが完了する。一方、第６判定部３５６の判定の結果が、所定回数修正していないことを示す場合（ステップＳ１１０：ＮＯ）、ステップＳ１０９の処理に戻る。 Returning now to the explanation of the judgment process of step S108, if the result of the sixth judgment indicates that the nuts 202 have been fastened to all of the bolts 201 (step S108: YES), the learning unit 37 modifies the weighting value of the parameters of the first trained model 391 based on the various data acquired in the fastening operation step S106 (step S109). On the other hand, if the result of the sixth judgment indicates that the nuts 202 have not been fastened to all of the bolts 201 (step S108: NO), the process proceeds to step S101. Following the process of step S109, the sixth judgment unit 356 judges whether the learning unit 37 has modified the weighting value of the parameters of the first trained model 391 a predetermined number of times (step S110). If the result of the judgment of the sixth judgment unit 356 indicates that the weighting value has been modified a predetermined number of times (step S110: YES), the learning step is completed. On the other hand, if the result of the judgment by the sixth judgment unit 356 indicates that the correction has not been performed the predetermined number of times (step S110: NO), the process returns to step S109.
　以上説明した一連の処理動作を経る学習ステップにより取得された各種のデータに基づき、学習部３７がパラメータの重み値を繰り返し修正するため、第１学習済みモデル３９１は、入力に応じた指令値を出力するように教示される。すなわち、第１学習済みモデル３９１は、ナット２０２の目標位置座標および目標姿勢角から、ランダムに与えられた量外れたナット２０２の既知位置座標および既知姿勢角が、目標位置座標および目標姿勢角になるようにエンドエフェクタ１２を制御する教示動作中に取得された複数のナット２０２の位置座標および姿勢角と、各々のナット２０２の位置座標および姿勢角が取得された時刻に対応する接触力およびトルクとを含む教師データセットが入力される学習ステップが繰り返し実行されたことで、入力に応じた指令値を出力するように学習されている。 The learning unit 37 repeatedly modifies the weighting values of the parameters based on various data acquired by the learning step that undergoes the series of processing operations described above, so that the first trained model 391 is taught to output a command value according to the input. That is, the first trained model 391 is trained to output a command value according to the input by repeatedly executing a learning step in which a teacher data set including the position coordinates and attitude angles of multiple nuts 202 acquired during a teaching operation that controls the end effector 12 so that the known position coordinates and known attitude angles of the nuts 202 that deviate from the target position coordinates and target attitude angle of the nuts 202 by a randomly given amount become the target position coordinates and target attitude angles, and the contact force and torque corresponding to the time when the position coordinates and attitude angles of each nut 202 were acquired, is input.
（第２学習済みモデルの教示方法）
　また、第２制御部３６２が用いる第２学習済みモデル３９２は、第１位置から第２位置にエンドエフェクタ１２を移動させるダイレクトティーチング動作中に取得された複数の画像と、各々の画像が取得された時刻に対応するナット２０２の位置座標および姿勢角とを含む教師データセットが入力される学習ステップが繰り返し実行されたことで、入力に応じた指令値を出力するように学習（逆教示）されている。具体的には、本実施形態における逆教示は、エンドエフェクタ１２を第２位置から第１位置へ移動（ダイレクトティーチング動作）させた際に取得部３１により取得された画像と、ナット２０２の位置座標および姿勢角の時系列上のデータセットの時系列が逆になるように変換し、逆時系列に変換されたデータセットを上記教師データセットとして第２学習済みモデル３９２に学習させることである。また、ここでいう「ダイレクトティーチング動作」とは、図１２に一例として示すように、オペレータの手動などによって、第２位置から第１位置にナット２０２が移動するようにエンドエフェクタ１２を移動させる動作のことを意味する。なお、第２学習済みモデル３９２を学習（逆教示）させるための上記学習ステップは、例えば数十回から数百回に亘って繰り返し行われる。 (Method of teaching second trained model)
The second trained model 392 used by the second control unit 362 has been trained (reverse taught) to output a command value according to the input by repeatedly executing a learning step in which a teacher data set including a plurality of images acquired during a direct teaching operation to move the end effector 12 from the first position to the second position and the position coordinates and attitude angle of the nut 202 corresponding to the time when each image was acquired is input. Specifically, the reverse teaching in this embodiment is a conversion so that the time series of the image acquired by the acquisition unit 31 when the end effector 12 is moved from the second position to the first position (direct teaching operation) and the time series of the data set of the position coordinates and attitude angle of the nut 202 are reversed, and the data set converted into the reverse time series is trained by the second trained model 392 as the teacher data set. The "direct teaching operation" here means an operation of moving the end effector 12 so that the nut 202 moves from the second position to the first position by the operator's manual operation, as shown as an example in FIG. 12. The above learning step for learning (reverse teaching) the second trained model 392 is repeated, for example, tens to hundreds of times.
（制御装置の動作）
　続いて、本実施形態における制御装置３０の動作の一例について図１３を参照して説明する。ただし、以下に説明する処理の順番は、以下の例に限定されず、適宜入れ替えられてもよい。 (Operation of the control device)
Next, an example of the operation of the control device 30 in this embodiment will be described with reference to Fig. 13. However, the order of the processes described below is not limited to the following example, and may be changed as appropriate.
　はじめに、第２制御部３６２は、第２学習済みモデル３９２を用いて、施工の対象となるボルト２０１にナット２０２をアプローチさせる（ステップＳ１）。次いで、第２判定部３５２は、ナット２０２の状態に異常があるか否かを判定する第２判定を行う（ステップＳ２）。ナット２０２の状態に異常があることを第２判定の結果が示す場合（ステップＳ２：ＹＥＳ）、例えば制御部３６の動作によりナット２０２をボルト２０１から一度離間させた後、ステップＳ１の処理に戻る。一方、ナット２０２の状態に異常がないことを第２判定の結果が示す場合（ステップＳ２：ＮＯ）、第１判定部３５１は、ナット２０２がボルト２０１に接触したか否かを判定する第１判定を行う（ステップＳ３）。ナット２０２がボルト２０１に接触していないことを第１判定の結果が示す場合（ステップＳ３：ＮＯ）、例えば制御部３６の動作によりナット２０２をボルト２０１から一度離間させた後、ステップＳ１の処理に戻る。一方、ナット２０２がボルト２０１に接触したことを第１判定の結果が示す場合（ステップＳ３：ＹＥＳ）、締結動作ステップＳ４を実行する。締結動作ステップＳ４では、第１制御部３６１は、第１学習済みモデル３９１を用いてエンドエフェクタ１２の移動および回転を制御することで対象となるボルト２０１に対するナット２０２の締結動作を実行する。締結動作ステップＳ４では、第１制御部３６１が締結動作を実行している際、第３判定部３５３は、ナット２０２の螺合が成功したか否かを判定する第３判定を行ってよい。ナット２０２の螺合が成功したことを第３判定の結果が示す場合、第１制御部３６１は、締結動作を実行し続ける。一方、ナット２０２の螺合が失敗したことを第３判定の結果が示す場合、第１制御部３６１は、締結動作を停止する。第１制御部３６１が締結動作を停止した場合、例えば、リトライ動作ステップＳ８を実行する。 First, the second control unit 362 uses the second trained model 392 to make the nut 202 approach the bolt 201 to be installed (step S1). Next, the second judgment unit 352 performs a second judgment to judge whether or not there is an abnormality in the state of the nut 202 (step S2). If the result of the second judgment indicates that there is an abnormality in the state of the nut 202 (step S2: YES), for example, the control unit 36 operates to move the nut 202 away from the bolt 201 once, and then the process returns to step S1. On the other hand, if the result of the second judgment indicates that there is no abnormality in the state of the nut 202 (step S2: NO), the first judgment unit 351 performs a first judgment to judge whether or not the nut 202 has come into contact with the bolt 201 (step S3). If the result of the first judgment indicates that the nut 202 is not in contact with the bolt 201 (step S3: NO), for example, the control unit 36 operates to move the nut 202 away from the bolt 201 once, and then the process returns to step S1. On the other hand, if the result of the first judgment indicates that the nut 202 has contacted the bolt 201 (step S3: YES), the fastening operation step S4 is executed. In the fastening operation step S4, the first control unit 361 executes the fastening operation of the nut 202 on the target bolt 201 by controlling the movement and rotation of the end effector 12 using the first learned model 391. In the fastening operation step S4, while the first control unit 361 is executing the fastening operation, the third judgment unit 353 may perform a third judgment to determine whether or not the screwing of the nut 202 has been successful. If the result of the third judgment indicates that the screwing of the nut 202 has been successful, the first control unit 361 continues to execute the fastening operation. On the other hand, if the result of the third judgment indicates that the screwing of the nut 202 has failed, the first control unit 361 stops the fastening operation. If the first control unit 361 stops the fastening operation, for example, the retry operation step S8 is executed.
　第１制御部３６１による締結動作が終了した場合、第４判定部３５４は、ナット２０２の締結が成功したか否かを判定する第４判定を行う（ステップＳ５）。ナット２０２の締結が成功したことを第４判定の結果が示す場合（ステップＳ５：ＹＥＳ）、第５判定部３５５は、カラー２０２ｂが破断して切断されたか否かを判定する（ステップＳ６）。一方、ナット２０２の締結が失敗したことを第４判定の結果が示す場合（ステップＳ５：ＮＯ）、リトライ動作ステップＳ８を実行する。リトライ動作ステップＳ８では、第３制御部３６３は、上述のリトライ動作を行う。リトライ動作ステップＳ８が終了した場合、第６判定部３５６は、所定回数リトライしたか否かを判定する第６判定を行う（ステップＳ９）。所定回数リトライしたことを第６判定の結果が示す場合（ステップＳ９：ＹＥＳ）、例えば、制御装置３０は動作を終了する。一方、所定回数リトライしていないことを第６判定の結果が示す場合（ステップＳ９：ＮＯ）、ステップＳ１の処理に進む。ここで、ステップＳ６の判定の処理の説明に戻り、カラー２０２ｂが切断されていないことを第５判定の結果が示す場合（ステップＳ６：ＮＯ）、締結動作ステップＳ４に戻る。一方、カラー２０２ｂが切断されたことを第５判定の結果が示す場合（ステップＳ６：ＹＥＳ）、第６判定部３５６は、すべてのボルト２０１にナット２０２を締結したか否かを判定する（ステップＳ７）。すべてのボルト２０１にナット２０２を締結していないことを第６判定の結果が示す場合（ステップＳ７：ＮＯ）、ステップＳ１の処理に進む。一方、すべてのボルト２０１にナット２０２を締結したことを第６判定の結果が示す場合（ステップＳ７：ＹＥＳ）、例えば、制御装置３０は動作を終了する。 When the fastening operation by the first control unit 361 is completed, the fourth judgment unit 354 performs a fourth judgment to judge whether or not the fastening of the nut 202 was successful (step S5). When the result of the fourth judgment indicates that the fastening of the nut 202 was successful (step S5: YES), the fifth judgment unit 355 judges whether or not the collar 202b was broken and cut (step S6). On the other hand, when the result of the fourth judgment indicates that the fastening of the nut 202 failed (step S5: NO), the retry operation step S8 is executed. In the retry operation step S8, the third control unit 363 performs the above-mentioned retry operation. When the retry operation step S8 is completed, the sixth judgment unit 356 performs a sixth judgment to judge whether or not the retry has been performed a predetermined number of times (step S9). When the result of the sixth judgment indicates that the retry has been performed a predetermined number of times (step S9: YES), for example, the control device 30 ends the operation. On the other hand, when the result of the sixth judgment indicates that the retry has not been performed a predetermined number of times (step S9: NO), the process proceeds to step S1. Returning now to the explanation of the judgment process of step S6, if the result of the fifth judgment indicates that the collar 202b has not been cut (step S6: NO), the process returns to the fastening operation step S4. On the other hand, if the result of the fifth judgment indicates that the collar 202b has been cut (step S6: YES), the sixth judgment unit 356 judges whether or not the nuts 202 have been fastened to all of the bolts 201 (step S7). If the result of the sixth judgment indicates that the nuts 202 have not been fastened to all of the bolts 201 (step S7: NO), the process proceeds to step S1. On the other hand, if the result of the sixth judgment indicates that the nuts 202 have been fastened to all of the bolts 201 (step S7: YES), for example, the control device 30 ends its operation.
　以上説明した制御装置３０の動作は、施工対象物２００の組立工程で繰り返し実行される。 The above-described operation of the control device 30 is repeatedly performed during the assembly process of the construction object 200.
（作用・効果）
　本実施形態では、第１時刻における各々の関節部１１０の関節角に基づくナット２０２の位置座標および姿勢角と、第１時刻におけるエンドエフェクタ１２から伝達された接触力およびトルクとを入力とする第１学習済みモデル３９１により出力された第２時刻におけるナット２０２の位置座標および姿勢角の指令値に基づき、エンドエフェクタ１２の移動および回転が制御される。この際、第１学習済みモデル３９１が出力する第２時刻における指令値のナット２０２の位置座標および姿勢角は、ナット２０２がボルト２０１に締結された状態を示すナット２０２の目標位置座標および目標姿勢角に近づいている。つまり、第１学習済みモデル３９１に対しての入力が行われるたびに、ナット２０２の状態を、ナット２０２がボルト２０１に締結された状態に向けて遷移させることができる。したがって、施工対象物２００に設けられたボルト２０１にナット２０２をより円滑に締結することができる。 (Action and Effects)
In this embodiment, the movement and rotation of the end effector 12 are controlled based on the command values of the position coordinates and attitude angle of the nut 202 at the second time output by the first trained model 391, which receives as input the position coordinates and attitude angle of the nut 202 based on the joint angles of each joint 110 at the first time and the contact force and torque transmitted from the end effector 12 at the first time. At this time, the position coordinates and attitude angle of the nut 202 of the command values at the second time output by the first trained model 391 are approaching the target position coordinates and target attitude angle of the nut 202 indicating the state in which the nut 202 is fastened to the bolt 201. In other words, each time an input is made to the first trained model 391, the state of the nut 202 can be transitioned toward the state in which the nut 202 is fastened to the bolt 201. Therefore, the nut 202 can be more smoothly fastened to the bolt 201 provided on the construction target 200.
　また、本実施形態では、第１学習済みモデル３９１は、目標位置座標および目標姿勢角から外れたあらかじめ把握されている既知位置座標および既知姿勢角を示すナット２０２をボルト２０１に締結される教示が繰り返し実行されることで上記入出力がなされるようにあらかじめ学習されている。そのため、例えば、実運用時にナット２０２およびボルト２０１の締結を行う際にナット２０２のボルト２０１に対する位置座標のずれや姿勢角のずれが生じても、実際に締結にかかった時間が、予測された締結にかかる時間から乖離することを抑制することができる。その結果、例えば、組立にかかる工数などを低減させることができる。 In addition, in this embodiment, the first trained model 391 is trained in advance so that the above input and output are performed by repeatedly executing instructions to fasten the nut 202, which indicates known position coordinates and known attitude angles that are previously determined and deviate from the target position coordinates and target attitude angle, to the bolt 201. Therefore, for example, even if a deviation in the position coordinates or attitude angle of the nut 202 relative to the bolt 201 occurs when fastening the nut 202 and bolt 201 during actual operation, it is possible to prevent the time actually required for fastening from deviating from the predicted time required for fastening. As a result, for example, it is possible to reduce the man-hours required for assembly.
　また、本実施形態では、ナット２０２がボルト２０１に接触していない第１位置で取得された第３時刻における施工対象物２００を撮像した画像と、第３時刻におけるナット２０２の位置座標および姿勢角とを入力とする第２学習済みモデル３９２により出力された第４時刻におけるナット２０２の位置座標および姿勢角を示す指令値に基づき、エンドエフェクタ１２の移動および回転が制御される。この際、第２学習済みモデル３９２が出力する第４時刻における指令値のナット２０２の位置座標および姿勢角は、ナット２０２がボルト２０１に接触した第２位置の位置座標および姿勢角に近づいている。つまり、第２学習済みモデル３９２に対しての入力が行われるたびに、第１位置に位置するナット２０２を、第２位置に向けて遷移させることができる。したがって、ナット２０２を施工対象物２００に設けられたボルト２０１により円滑にアプローチさせることができる。また、画像を第２学習済みモデル３９２の入力要素とするため、例えば画像を入力要素に採用しない場合と比較して、上記の作用をより高い精度で実現することができる。 In addition, in this embodiment, the movement and rotation of the end effector 12 are controlled based on a command value indicating the position coordinates and attitude angle of the nut 202 at the fourth time output by the second trained model 392, which receives as input an image of the construction target 200 at the third time acquired at the first position where the nut 202 is not in contact with the bolt 201 and the position coordinates and attitude angle of the nut 202 at the third time. At this time, the position coordinates and attitude angle of the nut 202 of the command value at the fourth time output by the second trained model 392 are approaching the position coordinates and attitude angle of the second position where the nut 202 is in contact with the bolt 201. In other words, each time an input is made to the second trained model 392, the nut 202 located at the first position can be transitioned toward the second position. Therefore, the nut 202 can be made to approach the bolt 201 provided on the construction target 200 more smoothly. In addition, since the image is used as an input element of the second trained model 392, the above action can be realized with higher accuracy compared to, for example, a case where the image is not used as an input element.
　また、本実施形態では、第２学習済みモデル３９２は、第１位置から第２位置にエンドエフェクタ１２を移動させるダイレクトティーチング動作中に取得された複数の画像と、各々の画像が取得された時刻に対応するナット２０２の位置座標および姿勢角とを含む教師データセットが入力される学習ステップが繰り返し実行されることで上記入出力がなされるようにあらかじめ学習されている（逆教示されている）。発明者らは、組立システム１００を利用するオペレータの手動によってエンドエフェクタ１２を移動させるダイレクトティーチング動作を用いて第２学習済みモデル３９２を教示させるに当たって、ナット２０２の位置が第１位置から第２位置に移動するようにエンドエフェクタ１２を移動させる動作よりも、第２位置から第１位置に移動するようにエンドエフェクタ１２を移動させる動作の方が容易であること、および、動作中のエンドエフェクタ１２のブレなどが発生しにくいことを見出した。その結果、ノイズなどの割合が少ないより高品質な教師データセットを取得することができる。 In addition, in this embodiment, the second trained model 392 is trained in advance (reverse taught) so that the above input and output are performed by repeatedly executing a learning step in which a teacher data set including a plurality of images acquired during a direct teaching operation to move the end effector 12 from a first position to a second position and the position coordinates and attitude angle of the nut 202 corresponding to the time when each image was acquired is input. The inventors have found that when teaching the second trained model 392 using a direct teaching operation to manually move the end effector 12 by an operator using the assembly system 100, the operation of moving the end effector 12 so that the position of the nut 202 moves from the second position to the first position is easier than the operation of moving the end effector 12 so that the position of the nut 202 moves from the first position to the second position, and that shaking of the end effector 12 during operation is less likely to occur. As a result, a higher quality teacher data set with a lower proportion of noise can be acquired.
　また、本実施形態では、第２位置で取得された画像と、画像が取得された時刻に対応するナット２０２の位置座標および姿勢角と、画像が取得された時刻に対応する接触力およびトルクとが入力されると、ナット２０２およびボルト２０１の螺合の成否を示す判定結果を出力するように学習された第３学習済みモデル３９３を用いて、エンドエフェクタ１２を制御する。これにより、例えば、ナット２０２およびボルト２０１の螺合が誤った状態で締結に向かうことを回避できる。その結果、例えば、組立における歩留まりの低下などを抑制することができる。 In addition, in this embodiment, when an image captured at the second position, the position coordinates and attitude angle of the nut 202 corresponding to the time the image was captured, and the contact force and torque corresponding to the time the image was captured are input, the end effector 12 is controlled using a third trained model 393 that has been trained to output a determination result indicating whether the nut 202 and bolt 201 have been screwed together. This makes it possible to avoid, for example, the nut 202 and bolt 201 being screwed together in an incorrect state before being fastened. As a result, for example, it is possible to suppress a decrease in yield during assembly.
　また、本実施形態では、ナット２０２の位置座標が表現される座標系が、世界座標系からボルト中心座標系に変換される。これにより、世界座標系で表現されるナット２０２の位置座標がボルト２０１ごとに生じる固有の位置座標であるのに対して、ボルト２０１中心座標といった同一の尺度でナット２０２のボルト２０１に対する相対的な位置座標を表すことができる。その結果、ボルト２０１ごとに教示する必要が生じないため、汎化性が高まるとともに、第１学習済みモデル３９１の教示にかかる労力および時間を削減することができる。 In addition, in this embodiment, the coordinate system in which the position coordinates of the nut 202 are expressed is converted from the world coordinate system to the bolt-centered coordinate system. As a result, while the position coordinates of the nut 202 expressed in the world coordinate system are unique position coordinates that arise for each bolt 201, the relative position coordinates of the nut 202 with respect to the bolt 201 can be expressed on the same scale, such as the bolt 201-centered coordinates. As a result, there is no need to teach each bolt 201, which increases generalizability and reduces the effort and time required to teach the first trained model 391.
　また、本実施形態では、関節部１１０の関節角に基づくエンドエフェクタ１２の自重に関する補正値を用いて取得された接触力を補正する。これにより、エンドエフェクタ１２から伝達された接触力よりも実際に生じた接触力に近づいた接触力を第１学習済みモデル３９１の入力要素とすることができるため、汎化性を高めることができる。 In addition, in this embodiment, the acquired contact force is corrected using a correction value related to the weight of the end effector 12, which is based on the joint angle of the joint unit 110. This allows a contact force that is closer to the actual contact force than the contact force transmitted from the end effector 12 to be used as an input element for the first trained model 391, thereby improving generalizability.
　また、本実施形態におけるエンドエフェクタ１２によれば、弾性部材１２８が第１部分１２６に対してソケット部１２３の先端側に向かう弾性力を付与することで、ナットランナ１２１の収容部１２３ｒにカラー２０２ｂが収容されたナット２０２を、ナット保持部材１２４の爪部１２７ｂが把持する機構である。これにより、例えばナット２０２を吸引して保持する機構を備えたエンドエフェクタ１２などと比較して、より簡易な構成でエンドエフェクタ１２がナット２０２を保持することができる。また、撮像装置２０がナットランナ１２１のソケット部１２３内に配置されており、当該撮像装置２０は、ナット孔２０２ｈを通じて施工対象物２００に向けて光を照射するとともに、施工対象物２００を撮像した画像を生成する。すなわち、撮像装置２０の撮像範囲を集中的に照らしながら画像を生成することができる。したがって、施工現場の照明環境などの影響を受けにくい。 In addition, according to the end effector 12 in this embodiment, the elastic member 128 applies an elastic force toward the tip side of the socket portion 123 to the first portion 126, so that the claw portion 127b of the nut holding member 124 grips the nut 202 whose collar 202b is accommodated in the accommodation portion 123r of the nut runner 121. This allows the end effector 12 to hold the nut 202 with a simpler configuration than, for example, an end effector 12 equipped with a mechanism for suctioning and holding the nut 202. In addition, the imaging device 20 is disposed in the socket portion 123 of the nut runner 121, and the imaging device 20 irradiates light toward the construction target 200 through the nut hole 202h and generates an image of the construction target 200. In other words, an image can be generated while intensively illuminating the imaging range of the imaging device 20. Therefore, it is less susceptible to the influence of the lighting environment at the construction site.
＜組立システムの第２実施形態＞
　次に、本開示に係る組立システム１００の第２実施形態について図１４を参照して説明する。なお、以下に説明する第２実施形態では、上記の第１実施形態と共通する構成については図中に同符号を付してその説明を省略する。第２実施形態では、組立システム１００は、力覚センサ１４０を備えておらず、当該力覚センサ１４０に代えて、電流センサ１４１を備えている。本実施形態では、第１実施形態で説明した力データが、電流センサ１４１が取得する電流値に対応しており、上述した力データを電流値に適宜置き換えて読めばよい。 Second embodiment of assembly system
Next, a second embodiment of the assembly system 100 according to the present disclosure will be described with reference to FIG. 14. In the second embodiment described below, components common to the first embodiment will be denoted by the same reference numerals in the drawings, and description thereof will be omitted. In the second embodiment, the assembly system 100 does not include a force sensor 140, and instead includes a current sensor 141. In this embodiment, the force data described in the first embodiment corresponds to the current value acquired by the current sensor 141, and the above-mentioned force data may be read by appropriately replacing it with the current value.
（電流センサ）
　電流センサ１４１は、各々の関節部１１０（第１関節部１１１～第６関節部１１６）が有するモータ１１７（第１モータ１１７ａ～第６モータ１１７ｆ）の電流値を取得可能なセンサである。電流センサ１４１は、有線または無線通信によって各々の関節部１１０が有するモータ１１７に接続されている。電流センサ１４１は、第２センサ１４の一例である。電流センサ１４１は、取得したモータ１１７の電流値を制御装置３０に送信する。なお、電流センサ１４１は、例えば、マニピュレータ１０に設けられてもよく、制御装置３０に設けられてもよい。 (Current sensor)
The current sensor 141 is a sensor capable of acquiring a current value of the motor 117 (first motor 117a to sixth motor 117f) of each joint 110 (first joint 111 to sixth joint 116). The current sensor 141 is connected to the motor 117 of each joint 110 by wired or wireless communication. The current sensor 141 is an example of the second sensor 14. The current sensor 141 transmits the acquired current value of the motor 117 to the control device 30. The current sensor 141 may be provided in the manipulator 10 or in the control device 30, for example.
（制御装置）
　本実施形態では、取得部３１は、電流センサ１４１から各々のモータ１１７を流れる電流の電流値を第１所定周期で取得し、取得した電流値を判定部３５および制御部３６に送るとともに記憶部３９に記憶させる。第１判定部３５１は、取得部３１から受け付けた電流値に基づき、第１判定を行う。第３判定部３５３は、取得部３１から受け付けた画像、算出部３２から受け付けたナット２０２の姿勢角、座標変換部３３から受け付けたナット２０２の位置座標、ならびに、取得部３１から受け付けた電流値に基づき、第３判定を行う。この際、第３学習済みモデル３９３は、画像と、ナット２０２の位置座標および姿勢角と、電流値とを含むデータセットが入力されると、ナット２０２およびボルト２０１の螺合の成否を示す判定結果を出力するように学習されている。第４判定部３５４は、取得部３１から受け付けた画像、算出部３２から受け付けたナット２０２の姿勢角、座標変換部３３から受け付けたナット２０２の位置座標、ならびに、取得部３１から受け付けた電流値に基づき、第４判定を行う。この際、第４学習済みモデル３９４は、画像と、ナット２０２の位置座標および姿勢角と、電流値とを含むデータセットが入力されると、ナット２０２およびボルト２０１の締結の成否を示す判定結果を出力するように学習されている。第５判定部３５５は、第４判定部３５４から受け付けた第４判定の結果がナット２０２の締結が成功したことを示す場合に、取得部３１から受け付けた電流値に基づき、ナット２０２のカラー２０２ｂがナット本体２０２ａから分離しているか否かを判定する。第１制御部３６１は、ナット２０２がボルト２０１に接触していると第１判定の結果が示す場合に、算出部３２から受け付けたナット２０２の姿勢角と、座標変換部３３から受け付けたナット２０２の位置座標と、取得部３１から受け付けた電流値とを含むデータセットに基づき、ボルト２０１に対するナット２０２の締結動作を実行する。この際、第１学習済みモデル３９１は、第１時刻におけるナット２０２の位置座標および姿勢角と、第１時刻における電流値とを含むデータセットが入力されると、ナット２０２がボルト２０１に締結された状態を示すナット２０２の目標位置座標および目標姿勢角に近づいた、第１時刻よりも後の第２時刻におけるナット２０２の位置座標および姿勢角を示す指令値を出力するように学習されている。 (Control device)
In this embodiment, the acquisition unit 31 acquires the current value of the current flowing through each motor 117 from the current sensor 141 at a first predetermined period, sends the acquired current value to the judgment unit 35 and the control unit 36, and stores it in the storage unit 39. The first judgment unit 351 performs a first judgment based on the current value received from the acquisition unit 31. The third judgment unit 353 performs a third judgment based on the image received from the acquisition unit 31, the attitude angle of the nut 202 received from the calculation unit 32, the position coordinates of the nut 202 received from the coordinate conversion unit 33, and the current value received from the acquisition unit 31. At this time, the third trained model 393 is trained to output a judgment result indicating whether the nut 202 and the bolt 201 are screwed together when a data set including the image, the position coordinates and attitude angle of the nut 202, and the current value is input. The fourth determination unit 354 performs a fourth determination based on the image received from the acquisition unit 31, the attitude angle of the nut 202 received from the calculation unit 32, the position coordinates of the nut 202 received from the coordinate conversion unit 33, and the current value received from the acquisition unit 31. At this time, the fourth trained model 394 is trained to output a determination result indicating whether the nut 202 and the bolt 201 have been fastened, when a data set including the image, the position coordinates and attitude angle of the nut 202, and the current value is input. When the result of the fourth determination received from the fourth determination unit 354 indicates that the fastening of the nut 202 has been successful, the fifth determination unit 355 determines whether the collar 202b of the nut 202 has been separated from the nut body 202a, based on the current value received from the acquisition unit 31. When the result of the first determination indicates that the nut 202 is in contact with the bolt 201, the first control unit 361 executes a fastening operation of the nut 202 on the bolt 201 based on a data set including the attitude angle of the nut 202 received from the calculation unit 32, the position coordinates of the nut 202 received from the coordinate conversion unit 33, and the current value received from the acquisition unit 31. At this time, when a data set including the position coordinates and attitude angle of the nut 202 at the first time and the current value at the first time is input, the first trained model 391 is trained to output a command value indicating the position coordinates and attitude angle of the nut 202 at a second time after the first time that approaches the target position coordinates and target attitude angle of the nut 202 that indicate a state in which the nut 202 is fastened to the bolt 201.
（作用・効果）
　本実施形態では、電流センサ１４１が取得した各々の関節部１１０が有するモータ１１７の電流値に、第１実施形態で説明した力データを代替させることができる。すなわち、各々のモータ１１７の電流値は、ナット２０２がボルト２０１に接触した際にエンドエフェクタ１２から伝達される接触力およびトルクを示している。これにより、マニピュレータ１０が力覚センサ１４０を備える必要がない。したがって、例えば、マニピュレータ１０全体のサイズおよび重量を抑えることができる。 (Action and Effects)
In this embodiment, the current value of the motor 117 of each joint 110 acquired by the current sensor 141 can be substituted for the force data described in the first embodiment. That is, the current value of each motor 117 indicates the contact force and torque transmitted from the end effector 12 when the nut 202 contacts the bolt 201. This eliminates the need for the manipulator 10 to include the force sensor 140. Therefore, for example, the size and weight of the entire manipulator 10 can be reduced.
＜組立システムの第３実施形態＞
　次に、本開示に係る組立システム１００の第３実施形態について図１５を参照して説明する。なお、以下に説明する第３実施形態では、上記の第１実施形態と共通する構成については図中に同符号を付してその説明を省略する。第３実施形態では、組立システム１００は、移動機構１６と、キャリブレーション機構１７とを更に備えている。 Third embodiment of assembly system
Next, a third embodiment of the assembly system 100 according to the present disclosure will be described with reference to Fig. 15. In the third embodiment described below, components common to the first embodiment will be denoted by the same reference numerals in the drawings and will not be described. In the third embodiment, the assembly system 100 further includes a moving mechanism 16 and a calibration mechanism 17.
（移動機構）
　移動機構１６は、台座部１５を施工対象物２００の周囲で移動可能にさせる。本実施形態では、移動機構１６は、台座部１５に設けられた複数の車輪１６０である。移動機構１６は、作業台３００の周囲を走行可能である。図１５中では、４つの車輪１６０が台座部１５に設けられている場合を一例として示している。車輪１６０は、床面に当接した状態で回動可能に台座部１５に設けられている。車輪１６０の駆動（回転）は、制御装置３０によって制御されてもよいし、組立システム１００を利用するオペレータによる手動などであってもよい。 (Moving mechanism)
The moving mechanism 16 allows the base 15 to move around the construction target 200. In this embodiment, the moving mechanism 16 is a plurality of wheels 160 provided on the base 15. The moving mechanism 16 can travel around the work platform 300. FIG. 15 shows an example in which four wheels 160 are provided on the base 15. The wheels 160 are provided on the base 15 so as to be rotatable while in contact with the floor surface. The driving (rotation) of the wheels 160 may be controlled by the control device 30, or may be manually driven by an operator using the assembly system 100.
（キャリブレーション機構）
　キャリブレーション機構１７は、施工対象物２００と、移動機構１６との相対的な位置を決定する。具体的には、キャリブレーション機構１７は、例えば、世界座標系で表現された施工対象物２００中の任意の一点と、マニピュレータ１０が持つ原点に準拠した世界座標系の任意の一地点とを一致させる（キャリブレーションする）。つまり、キャリブレーション機構１７により、施工対象物２００に対する移動機構１６の位置がキャリブレーションされる。キャリブレーション機構１７は、例えば、マニピュレータ１０などに設けられている。 (Calibration mechanism)
The calibration mechanism 17 determines the relative position between the construction target 200 and the moving mechanism 16. Specifically, the calibration mechanism 17, for example, matches (calibrates) an arbitrary point on the construction target 200 expressed in a world coordinate system with an arbitrary point on the world coordinate system based on the origin of the manipulator 10. In other words, the calibration mechanism 17 calibrates the position of the moving mechanism 16 with respect to the construction target 200. The calibration mechanism 17 is provided in, for example, the manipulator 10.
（作用・効果）
　本実施形態では、台座部１５に当該台座部１５を施工対象物２００の周囲で移動可能にする車輪１６０が設けられているため、より広い範囲でマニピュレータ１０が動作を行うことができる。したがって、例えば、サイズが大きい施工対象物２００を施工する場合に、施工対象物２００の配置や姿勢などを変更させる必要が生じにくい。 (Action and Effects)
In this embodiment, the base 15 is provided with wheels 160 that enable the base 15 to move around the construction target 200, so that the manipulator 10 can operate over a wider range. Therefore, for example, when constructing a large construction target 200, it is unlikely that the arrangement or posture of the construction target 200 needs to be changed.
＜第３実施形態の変形例＞
　次に、本開示に係る組立システム１００の第３実施形態の変形例について図１６を参照して説明する。本変形例では、上記第３実施形態に対して移動機構１６の構成が異なっている。 <Modification of the third embodiment>
Next, a modified example of the third embodiment of the assembly system 100 according to the present disclosure will be described with reference to Fig. 16. In this modified example, the configuration of the moving mechanism 16 is different from that of the third embodiment.
（移動機構）
　本変形例では、移動機構１６は、台座部１５と床面との間に配置された直動式の電動スライダ１６１である。電動スライダ１６１は、一方向に長尺に延びている。電動スライダ１６１は、例えば作業台３００に隣接して床面上に固定されている。電動スライダ１６１は、台座部１５を下方から支持した状態で当該台座部１５に接続されており、台座部１５を上記一方向に移動させる。電動スライダ１６１の駆動（台座部１５の直動）は、制御装置３０によって制御されてもよいし、オペレータによる手動などであってもよい。なお、本変形例で説明した電動スライダ１６１は、床面上に固定されている構成に代えて、当該床面とは別体として配置され、例えば電動スライダ１６１とは異なる移動機構１６に搭載されていてもよい。 (Moving mechanism)
In this modification, the moving mechanism 16 is a linear-acting electric slider 161 arranged between the base 15 and the floor surface. The electric slider 161 extends long in one direction. The electric slider 161 is fixed on the floor surface adjacent to the workbench 300, for example. The electric slider 161 is connected to the base 15 while supporting the base 15 from below, and moves the base 15 in the one direction. The driving of the electric slider 161 (linear movement of the base 15) may be controlled by the control device 30 or may be manually performed by an operator. Note that the electric slider 161 described in this modification may be arranged separately from the floor surface instead of being fixed on the floor surface, and may be mounted on a moving mechanism 16 different from the electric slider 161, for example.
（作用・効果）
　当該構成によっても、上記の第３実施形態で説明した作用・効果と同様の作用・効果が奏される。 (Action and Effects)
This configuration also provides the same effects and advantages as those described in the third embodiment above.
（その他の実施形態）
　以上、本開示の実施形態について図面を参照して詳述したが、具体的な構成は各実施形態の構成に限られるものではなく、本開示の要旨を逸脱しない範囲内での構成の付加、省略、置換、およびその他の変更が可能である。 Other Embodiments
Although the embodiments of the present disclosure have been described in detail above with reference to the drawings, the specific configurations are not limited to those of the embodiments, and additions, omissions, substitutions, and other modifications of the configurations are possible without departing from the gist of the present disclosure.
　なお、第１実施形態で説明した第１制御部３６１は、取得部３１から画像を受け付けてもよい。この場合、第１制御部３６１が制御に用いる第１学習済みモデル３９１は、受け付けた画像を上述の入力要素に加えてもよい。なおこの場合、第１学習済みモデル３９１は、入力に対する出力である上記指令値に加えて、サブタスクとして、第２時刻における予測画像を生成して出力してもよい。 The first control unit 361 described in the first embodiment may receive an image from the acquisition unit 31. In this case, the first trained model 391 used for control by the first control unit 361 may add the received image to the above-mentioned input elements. In this case, the first trained model 391 may generate and output, as a subtask, a predicted image at the second time, in addition to the above-mentioned command value, which is the output for the input.
　また、第１実施形態で説明した第１学習済みモデル３９１は、入力に対する出力である上記指令値に加えて、サブタスクとして、第２時刻における接触力の予測値およびトルクの予測値を生成して出力するように学習されてもよい。接触力の予測値およびトルクの予測値は、第２時刻で取得部３１によって取得されると第１学習済みモデル３９１により予測された接触力およびトルクを示す値である。これら接触力の予測値およびトルクの予測値は、第１制御部３６１による制御に用いられることはない。発明者らは、第１学習済みモデル３９１が第２時刻における接触力の予測値およびトルクの予測値をサブタスクとして出力することで、他の出力要素である指令値の精度が向上することを見出した。 Furthermore, the first trained model 391 described in the first embodiment may be trained to generate and output, as a subtask, a predicted value of the contact force and a predicted value of the torque at a second time in addition to the above-mentioned command value, which is an output for an input. The predicted value of the contact force and the predicted value of the torque are values indicating the contact force and the torque predicted by the first trained model 391 when acquired by the acquisition unit 31 at the second time. These predicted values of the contact force and the torque are not used for control by the first control unit 361. The inventors have found that by having the first trained model 391 output the predicted value of the contact force and the predicted value of the torque at the second time as a subtask, the accuracy of the command value, which is another output element, is improved.
　また、第１実施形態で説明した第２制御部３６２が制御に用いる第２学習済みモデル３９２は、入力に対する出力である上記指令値に加えて、サブタスクとして、第４時刻における予測画像を生成して出力するように学習されてもよい。予測画像は、第４時刻で取得部３１によって取得されると第２学習済みモデル３９２により予測された画像である。当該予測画像は、第２制御部３６２による制御に用いられることはない。発明者らは、第２学習済みモデル３９２が第４時刻における接触力の予測値およびトルクの予測値をサブタスクとして出力することで、他の出力要素である指令値の精度が向上することを見出した。 The second trained model 392 used for control by the second control unit 362 described in the first embodiment may be trained to generate and output, as a subtask, a predicted image at a fourth time, in addition to the above-mentioned command value, which is an output for an input. The predicted image is an image predicted by the second trained model 392 to be acquired by the acquisition unit 31 at the fourth time. This predicted image is not used for control by the second control unit 362. The inventors have found that by having the second trained model 392 output the predicted value of the contact force and the predicted value of the torque at the fourth time as a subtask, the accuracy of the command value, which is the other output element, is improved.
　また、第１実施形態で説明した第１学習済みモデル３９１、第２学習済みモデル３９２、第３学習済みモデル３９３、第４学習済みモデル３９４、および第５学習済みモデル３９６のそれぞれは、深層学習モデルに限定されることはなく、他の機械学習モデル（例えばＳＶＭ（Support Vector Machine）などの教師あり学習モデル）であってもよい。 Furthermore, each of the first trained model 391, the second trained model 392, the third trained model 393, the fourth trained model 394, and the fifth trained model 396 described in the first embodiment is not limited to a deep learning model, and may be another machine learning model (e.g., a supervised learning model such as an SVM (Support Vector Machine)).
　また、第１実施形態で説明した制御装置３０は、座標変換部３３および補正部３４を備えていなくてもよい。この場合、取得部３１は、取得した力データを判定部３５および制御部３６に送ればよい。またこの場合、取得部３１は、ボルト２０１の中心位置座標を取得した際、直近の施工の対象となるボルト２０１の中心位置座標を取得した旨を示す取得フラグをカウンタ部３８に送ればよい。そして、算出部３２は、算出したナット２０２の位置座標および姿勢角を判定部３５および制御部３６に送ればよい。そして、第１判定部３５１は、取得部３１から受け付けた力データに基づき第１判定を行い、第２判定部３５２は、算出部３２から受け付けたナット２０２の位置座標および姿勢角に基づき第２判定を行い、第３判定部３５３は、取得部３１から受け付けた画像、算出部３２から受け付けたナット２０２の位置座標および姿勢角、ならびに、取得部３１から受け付けた力データに基づき第３判定を行い、第４判定部３５４は、取得部３１から受け付けた画像、算出部３２から受け付けたナット２０２の位置座標および姿勢角、ならびに、取得部３１から受け付けた力データに基づき第４判定を行い、第５判定部３５５は、第４判定部３５４から受け付けた第４判定の結果がナット２０２の締結が成功したことを示す場合に、取得部３１から受け付けた力データに基づき、第５判定を行えばよい。そして、第１制御部３６１は、ナット２０２がボルト２０１に接触していると第１判定の結果が示す場合に、算出部３２から受け付けたナット２０２の位置座標および姿勢角と、取得部３１から受け付けた力データとを含むデータセットに基づき、ボルト２０１に対するナット２０２の締結動作を実行し、第２制御部３６２は、取得部３１から受け付けた画像と、算出部３２から受け付けたナット２０２の位置座標および姿勢角とを含むデータセットに基づき、エンドエフェクタ１２の移動および回転を制御すればよい。 The control device 30 described in the first embodiment may not include the coordinate conversion unit 33 and the correction unit 34. In this case, the acquisition unit 31 may send the acquired force data to the determination unit 35 and the control unit 36. In this case, when the acquisition unit 31 acquires the center position coordinates of the bolt 201, it may send an acquisition flag indicating that the center position coordinates of the bolt 201 that is the target of the most recent construction have been acquired to the counter unit 38. Then, the calculation unit 32 may send the calculated position coordinates and attitude angle of the nut 202 to the determination unit 35 and the control unit 36. Then, the first judgment unit 351 makes a first judgment based on the force data received from the acquisition unit 31, the second judgment unit 352 makes a second judgment based on the position coordinates and attitude angle of the nut 202 received from the calculation unit 32, the third judgment unit 353 makes a third judgment based on the image received from the acquisition unit 31, the position coordinates and attitude angle of the nut 202 received from the calculation unit 32, and the force data received from the acquisition unit 31, the fourth judgment unit 354 makes a fourth judgment based on the image received from the acquisition unit 31, the position coordinates and attitude angle of the nut 202 received from the calculation unit 32, and the force data received from the acquisition unit 31, and the fifth judgment unit 355 makes a fifth judgment based on the force data received from the acquisition unit 31 when the result of the fourth judgment received from the fourth judgment unit 354 indicates that the tightening of the nut 202 has been successful. Then, when the result of the first judgment indicates that the nut 202 is in contact with the bolt 201, the first control unit 361 executes the tightening operation of the nut 202 on the bolt 201 based on a data set including the position coordinates and attitude angle of the nut 202 received from the calculation unit 32 and the force data received from the acquisition unit 31, and the second control unit 362 controls the movement and rotation of the end effector 12 based on a data set including the image received from the acquisition unit 31 and the position coordinates and attitude angle of the nut 202 received from the calculation unit 32.
　また、第２実施形態で説明した制御装置３０は、取得部３１から電流値を受け付けるとともに、受け付けた電流値を接触力およびトルクを含む力データに変換する電流値変換部（図示省略）などを更に備えてもよい。この場合、電流値変換部は、例えば、電流値から変換した接触力およびトルクを含む力データを補正部３４などに送ればよい。電流値変換部は、例えば、各々のモータ１１７（第１モータ１１７ａ～第６モータ１１７ｆ）を流れる電流の電流値と、接触力およびトルクとが互いに関連付いた対応関係情報（関数など）に基づき、電流値を力データに変換してもよい。この場合、対応関係情報は、例えば記憶部３９にあらかじめ記憶されており、電流値変換部は、記憶部３９に記憶されている対応関係情報を適時に参照することで上記の変換を行えばよい。 The control device 30 described in the second embodiment may further include a current value conversion unit (not shown) that receives a current value from the acquisition unit 31 and converts the received current value into force data including a contact force and a torque. In this case, the current value conversion unit may send force data including a contact force and a torque converted from the current value to the correction unit 34, for example. The current value conversion unit may convert the current value into force data based on correspondence information (such as a function) that associates the current value of the current flowing through each motor 117 (first motor 117a to sixth motor 117f) with the contact force and the torque. In this case, the correspondence information is stored in advance in, for example, the storage unit 39, and the current value conversion unit may perform the above conversion by referring to the correspondence information stored in the storage unit 39 at appropriate times.
　また、第１判定部３５１は、第１判定を行うに当たり、取得部３１から受け付けた画像を更に用いてもよい。すなわち、第１判定部３５１は、画像と、力データとに基づき、第１判定を行ってよい。この場合、第１判定部３５１は、画像と、力データとを含むデータセットが入力されると、第１判定の結果を出力するように学習された所定の学習済みモデルを用いて判定を行う。すなわち、当該学習済みモデルは、画像と、力データとを含むデータセットを入力要素とし、第１判定の結果を出力要素とする。当該学習済みモデルは、例えば記憶部３９にあらかじめ記憶されており、第１判定部３５１は、記憶部３９に記憶されている当該学習済みモデルに上記データセットを入力することで出力された第１判定の結果を取得してもよい。 Furthermore, when making the first judgment, the first judgment unit 351 may further use the image received from the acquisition unit 31. That is, the first judgment unit 351 may make the first judgment based on the image and the force data. In this case, the first judgment unit 351 makes the judgment using a predetermined trained model that has been trained to output the result of the first judgment when a data set including the image and the force data is input. That is, the trained model has the data set including the image and the force data as input elements and the result of the first judgment as output elements. The trained model is stored in advance in the storage unit 39, for example, and the first judgment unit 351 may obtain the result of the first judgment output by inputting the above data set into the trained model stored in the storage unit 39.
　また、第２判定部３５２は、第２判定を行うに当たり、取得部３１から受け付けた画像を更に用いてもよい。すなわち、第２判定部３５２は、画像と、ナット２０２の位置座標および姿勢角とに基づき、第２判定を行ってよい。この場合、第２判定部３５２は、画像と、ナット２０２の位置座標および姿勢角とを含むデータセットが入力されると、第２判定の結果を出力するように学習された所定の学習済みモデルを用いて判定を行う。すなわち、当該学習済みモデルは、画像と、ナット２０２の位置座標および姿勢角とを含むデータセットを入力要素とし、第２判定の結果を出力要素とする。当該学習済みモデルは、例えば記憶部３９にあらかじめ記憶されており、第２判定部３５２は、記憶部３９に記憶されている当該学習済みモデルに上記データセットを入力することで出力された第２判定の結果を取得してもよい。 Furthermore, when making the second judgment, the second judgment unit 352 may further use the image received from the acquisition unit 31. That is, the second judgment unit 352 may make the second judgment based on the image and the position coordinates and attitude angle of the nut 202. In this case, the second judgment unit 352 makes the judgment using a predetermined trained model that has been trained to output the result of the second judgment when a data set including the image and the position coordinates and attitude angle of the nut 202 is input. That is, the trained model uses the image and the data set including the position coordinates and attitude angle of the nut 202 as input elements, and the result of the second judgment as output elements. The trained model is stored in advance in the storage unit 39, for example, and the second judgment unit 352 may acquire the result of the second judgment output by inputting the above data set into the trained model stored in the storage unit 39.
　また、第３判定部３５３は、ナット２０２の螺合の成否を判定するに当たり、取得部３１から受け付けた画像を用いなくてもよい。この場合、第３学習済みモデル３９３は、ナット２０２の位置座標および姿勢角と、力データとを含むデータセットを入力要素とし、ナット２０２およびボルト２０１の螺合の成否を示す判定結果を出力要素とする。 The third determination unit 353 does not need to use the image received from the acquisition unit 31 when determining whether the nut 202 is screwed in. In this case, the third trained model 393 uses a data set including the position coordinates and attitude angle of the nut 202 and the force data as input elements, and uses a determination result indicating whether the nut 202 and the bolt 201 are screwed in as output elements.
　また、第５判定部３５５は、補正部３４から受け付けた力データに基づき、切断に必要なトルクを示す第３しきい値を超えたか否かの切断に関する判定、および締結用モータ１２０ａが停止状態から回転状態になったか否かのエンドエフェクタ１２の駆動に関する判定を、第５判定を行う前に併せて行ってもよい。つまり、第５判定部３５５は、上記切断に関する判定が、データ（接触力）が第３しきい値を超えたことを示し、かつ、エンドエフェクタ１２の駆動に関する判定が、締結用モータ１２０ａが回転状態であることを示した場合に、上記第５判定を行ってもよい。なお、第３しきい値は、例えば記憶部３９にあらかじめ記憶されており、第５判定部３５５は、記憶部３９に記憶されている第３しきい値を適時に参照することで上記切断に関する判定を行う。 The fifth determination unit 355 may also perform a determination regarding cutting, based on the force data received from the correction unit 34, of whether the third threshold value indicating the torque required for cutting has been exceeded, and a determination regarding the drive of the end effector 12, of whether the fastening motor 120a has changed from a stopped state to a rotating state, before performing the fifth determination. In other words, the fifth determination unit 355 may perform the fifth determination when the determination regarding cutting indicates that the data (contact force) has exceeded the third threshold value and the determination regarding the drive of the end effector 12 indicates that the fastening motor 120a is in a rotating state. The third threshold value is stored in advance in, for example, the storage unit 39, and the fifth determination unit 355 performs the determination regarding cutting by referring to the third threshold value stored in the storage unit 39 as appropriate.
（コンピュータの構成）
　図１７は、本実施形態に係るコンピュータ１１００の構成を示すハードウェア構成図である。コンピュータ１１００は、例えば、プロセッサ１１１０と、メインメモリ１１２０と、ストレージ１１３０と、インターフェース１１４０とを備えている。 (Computer Configuration)
17 is a hardware configuration diagram showing the configuration of a computer 1100 according to this embodiment. The computer 1100 includes, for example, a processor 1110, a main memory 1120, a storage 1130, and an interface 1140.
　上述の制御装置３０は、例えば１以上のコンピュータ１１００に実装される。そして、上述した各処理部の動作は、プログラムの形式でストレージ１１３０に記憶されている。プロセッサ１１１０は、プログラムをストレージ１１３０から読み出してメインメモリ１１２０に展開し、当該プログラムにしたがって上記処理を実行する。また、プロセッサ１１１０は、プログラムにしたがって、上述した記憶部３９に対応する記憶領域をメインメモリ１１２０に確保する。プログラムは、コンピュータ１１００に発揮させる機能の一部を実現するためのものであってもよい。例えば、プログラムは、ストレージ１１３０に既に記憶されている他のプログラムとの組み合わせ、または他の装置に実装された他のプログラムとの組み合わせによって機能を発揮させるものであってもよい。また、コンピュータ１１００は、上記構成に加えて、または上記構成に代えてＰＬＤ（Programmable Logic Device）などのカスタムＬＳＩ（Large Scale Integrated Circuit）を備えてもよい。ＰＬＤの例としては、ＰＡＬ(Programmable Array Logic)、ＧＡＬ(Generic Array Logic)、ＣＰＬＤ(Complex Programmable Logic Device)、ＦＰＧＡ（Field Programmable Gate Array）が挙げられる。この場合、プロセッサ１１１０によって実現される機能の一部またはすべてが当該集積回路によって実現されてよい。ストレージ１１３０の例としては、磁気ディスク、光磁気ディスク、半導体メモリなどが挙げられる。ストレージ１１３０は、コンピュータ１１００のバスに直接的に接続された内部メディアであってもよいし、インターフェース１１４０または通信回線を介してコンピュータ１１００に接続される外部メディアであってもよい。また、このプログラムが通信回線によってコンピュータ１１００に配信される場合、配信を受けたコンピュータ１１００が当該プログラムをメインメモリ１１２０に展開し、上記処理を実行してもよい。上記実施形態では、ストレージ１１３０は、一時的でない有形の記憶媒体である。また、当該プログラムは、前述した機能の一部を実現するためのものであってもよい。さらに、当該プログラムは、前述した機能をストレージ１１３０に既に記憶されている他のプログラムとの組み合わせで実現するもの、いわゆる差分ファイル（差分プログラム）であってもよい。 The above-mentioned control device 30 is implemented in, for example, one or more computers 1100. The operation of each of the above-mentioned processing units is stored in the storage 1130 in the form of a program. The processor 1110 reads the program from the storage 1130, expands it in the main memory 1120, and executes the above-mentioned processing according to the program. The processor 1110 also secures a memory area in the main memory 1120 corresponding to the above-mentioned memory unit 39 according to the program. The program may be for realizing part of the function to be performed by the computer 1100. For example, the program may be for performing a function by combining it with other programs already stored in the storage 1130 or by combining it with other programs implemented in other devices. The computer 1100 may also be provided with a custom LSI (Large Scale Integrated Circuit) such as a PLD (Programmable Logic Device) in addition to or instead of the above configuration. Examples of PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Array). In this case, some or all of the functions realized by the processor 1110 may be realized by the integrated circuit. Examples of storage 1130 include a magnetic disk, a magneto-optical disk, and a semiconductor memory. Storage 1130 may be an internal medium directly connected to the bus of computer 1100, or an external medium connected to computer 1100 via interface 1140 or a communication line. In addition, when this program is distributed to computer 1100 via a communication line, computer 1100 that receives the program may expand the program in main memory 1120 and execute the above-mentioned processing. In the above embodiment, storage 1130 is a non-transitory tangible storage medium. In addition, the program may be for realizing some of the above-mentioned functions. Furthermore, the program may be a so-called differential file (differential program) that realizes the above-mentioned functions in combination with other programs already stored in storage 1130.
＜付記＞
　各実施形態に記載の組立システム、制御方法、およびプログラムは、例えば以下のように把握される。 <Additional Notes>
The assembly system, the control method, and the program described in each embodiment can be understood, for example, as follows.
（１）第１の態様に係る組立システム１００は、複数の関節部１１０を有するアーム１１と、前記アーム１１によって移動されるとともに、ナット２０２を保持して前記ナット２０２を軸線（回転軸線Ａｒまたは軸線Ａｒ１）回りに回転させるエンドエフェクタ１２と、各々の前記関節部１１０の関節角を取得する第１センサ１３（エンコーダ１３０）と、施工対象物２００に設けられたボルト２０１に前記ナット２０２が接触した場合に、前記エンドエフェクタ１２から伝達された接触力およびトルクを取得する第２センサ１４（力覚センサ１４０，電流センサ１４１）と、を有するマニピュレータ１０と、前記マニピュレータ１０を制御する制御装置３０と、を備え、前記制御装置３０は、各々の前記関節部１１０の関節角と、前記接触力およびトルクとを第１所定周期で取得する取得部３１と、前記取得された各々の前記関節部１１０の関節角に基づき、前記ナット２０２の位置座標および姿勢角を算出する算出部３２と、第１時刻における前記ナット２０２の位置座標および姿勢角と、前記第１時刻における前記接触力およびトルクとが入力されると、前記ナット２０２が前記ボルト２０１に締結された状態を示す前記ナット２０２の目標位置座標および目標姿勢角に近づいた、前記第１時刻よりも後の第２時刻における前記ナット２０２の位置座標および姿勢角を示す指令値を出力するように学習された第１学習済みモデル３９１を用いて、前記エンドエフェクタ１２を制御する制御部３６と、を備える。 (1) The assembly system 100 according to the first aspect includes a manipulator 10 having an arm 11 having a plurality of joints 110, an end effector 12 that is moved by the arm 11 and holds a nut 202 to rotate the nut 202 around an axis (rotation axis Ar or axis Ar1), a first sensor 13 (encoder 130) that acquires the joint angle of each of the joints 110, and a second sensor 14 (force sensor 140, current sensor 141) that acquires the contact force and torque transmitted from the end effector 12 when the nut 202 comes into contact with a bolt 201 provided on a construction object 200, and a control device 30 that controls the manipulator 10, and the control device 30 controls each of the related The system includes an acquisition unit 31 that acquires the joint angle of the joint portion 110 and the contact force and torque at a first predetermined period, a calculation unit 32 that calculates the position coordinates and attitude angle of the nut 202 based on the acquired joint angles of each of the joint portions 110, and a control unit 36 that controls the end effector 12 using a first trained model 391 that has been trained to output command values indicating the position coordinates and attitude angle of the nut 202 at a second time after the first time when the position coordinates and attitude angle of the nut 202 at a first time and the contact force and torque at the first time are close to the target position coordinates and target attitude angle of the nut 202 that indicate a state in which the nut 202 is fastened to the bolt 201.
　これにより、第１学習済みモデル３９１に対しての入力が行われるたびに、ナット２０２の状態を、ナット２０２がボルト２０１に締結された状態に向けて遷移させることができる。 This allows the state of the nut 202 to transition toward a state in which the nut 202 is fastened to the bolt 201 each time an input is made to the first trained model 391.
（２）第２の態様に係る組立システム１００は、（１）の組立システム１００であって、前記第１学習済みモデル３９１は、前記ナット２０２の前記目標位置座標および目標姿勢角から、ランダムに与えられた量外れた前記ナット２０２の既知位置座標および既知姿勢角が、前記目標位置座標および目標姿勢角になるように前記エンドエフェクタ１２を制御する教示動作中に取得された複数の前記ナット２０２の位置座標および姿勢角と、各々の前記ナット２０２の位置座標および姿勢角が取得された時刻に対応する前記接触力およびトルクと、を含む教師データセットが入力される学習ステップが繰り返し実行されたことで、前記入力に応じた前記指令値を出力するように学習されていてもよい。 (2) The assembly system 100 according to the second aspect is the assembly system 100 according to (1), and the first trained model 391 may be trained to output the command value according to the input by repeatedly executing a learning step in which a teacher data set including position coordinates and attitude angles of the nuts 202 acquired during a teaching operation for controlling the end effector 12 so that known position coordinates and known attitude angles of the nuts 202 that deviate from the target position coordinates and target attitude angles of the nuts 202 by a randomly given amount become the target position coordinates and target attitude angles, and the contact force and torque corresponding to the time when the position coordinates and attitude angles of each of the nuts 202 were acquired, are input.
　これにより、上記作用をより具体的な設定で実現することができる。また、例えば、実運用時にナット２０２およびボルト２０１の締結を行う際にナット２０２のボルト２０１に対する位置座標のずれや姿勢角のずれが生じても、実際に締結にかかった時間が、予測された締結にかかる時間から乖離することを抑制することができる。 This allows the above-mentioned action to be realized with more specific settings. Furthermore, for example, even if there is a deviation in the position coordinates or attitude angle of the nut 202 relative to the bolt 201 when tightening the nut 202 and bolt 201 during actual operation, it is possible to prevent the time actually required for tightening from deviating from the predicted time required for tightening.
（３）第３の態様に係る組立システム１００は、（１）の組立システム１００であって、前記エンドエフェクタ１２に設けられ、前記施工対象物２００を撮像した画像を生成する撮像装置２０を更に備え、前記取得部３１は、前記画像を第２所定周期で更に取得し、前記制御部３６は、前記ナット２０２が前記ボルト２０１に接触していない第１位置で前記取得された第３時刻における前記画像と、前記第３時刻における前記ナット２０２の位置座標および姿勢角とが入力されると、前記ナット２０２が前記ボルト２０１に接触した第２位置に前記ナット２０２が近づいた、前記第３時刻よりも後の第４時刻における前記ナット２０２の位置座標および姿勢角を示す指令値を出力するように学習された第２学習済みモデル３９２を用いて、前記エンドエフェクタ１２を制御してもよい。 (3) The assembly system 100 according to the third aspect is the assembly system 100 according to (1), further comprising an imaging device 20 provided on the end effector 12 and configured to generate an image of the construction target 200, the acquisition unit 31 further acquires the image at a second predetermined period, and the control unit 36 may control the end effector 12 using a second trained model 392 trained to output command values indicating the position coordinates and attitude angle of the nut 202 at a fourth time after the third time when the nut 202 approaches the second position where the nut 202 contacts the bolt 201 when the image acquired at a third time when the nut 202 is not in contact with the bolt 201 and the position coordinates and attitude angle of the nut 202 at the third time are input.
　これにより、第２学習済みモデル３９２に対しての入力が行われるたびに、第１位置に位置するナット２０２を、第２位置に向けて遷移させることができる。 This allows the nut 202, which is located in the first position, to transition toward the second position each time an input is made to the second trained model 392.
（４）第４の態様に係る組立システム１００は、（３）の組立システム１００であって、前記第２学習済みモデル３９２は、前記第２位置から前記第１位置に前記エンドエフェクタ１２を移動させるダイレクトティーチング動作中に取得された複数の前記画像と、各々の前記画像が取得された時刻に対応する前記ナット２０２の位置座標および姿勢角と、を含む教師データセットが入力される学習ステップが繰り返し実行されたことで、前記入力に応じた前記指令値を出力するように学習されていてもよい。 (4) The assembly system 100 according to the fourth aspect is the assembly system 100 according to (3), in which the second trained model 392 may be trained to output the command value according to the input by repeatedly executing a learning step in which a teacher data set including a plurality of images acquired during a direct teaching operation that moves the end effector 12 from the second position to the first position and the position coordinates and attitude angle of the nut 202 corresponding to the time when each of the images was acquired is input.
　これにより、ノイズなどの割合が少ないより高品質な教師データセットを取得することができる。 This allows us to obtain a higher quality training dataset with less noise.
（５）第５の態様に係る組立システム１００は、（３）または（４）の組立システム１００であって、前記制御部３６は、前記第２位置で前記取得された前記画像と、前記画像が取得された時刻に対応する前記ナット２０２の位置座標および姿勢角と、前記画像が取得された時刻に対応する前記接触力およびトルクとが入力されると、前記ナット２０２および前記ボルト２０１の螺合の成否を示す判定結果を出力するように学習された第３学習済みモデル３９３を用いて、前記エンドエフェクタ１２を制御してもよい。 (5) The assembly system 100 according to the fifth aspect is the assembly system 100 according to (3) or (4), and the control unit 36 may control the end effector 12 using a third trained model 393 trained to output a judgment result indicating whether the nut 202 and the bolt 201 are screwed together when the image acquired at the second position, the position coordinates and attitude angle of the nut 202 corresponding to the time when the image was acquired, and the contact force and torque corresponding to the time when the image was acquired are input.
　これにより、例えば、ナット２０２およびボルト２０１の螺合が誤った状態で締結されることを回避できる。 This makes it possible to prevent, for example, the nut 202 and the bolt 201 from being tightened in an incorrect state.
（６）第６の態様に係る組立システム１００は、（１）から（４）のうちいずれか１つの組立システム１００であって、前記ナット２０２の位置座標は、前記マニピュレータ１０が持つ原点に準拠した世界座標系で表現されており、前記制御装置３０は、前記算出された前記ナット２０２の位置座標を前記世界座標系からボルト中心座標系に変換する座標変換部３３を更に備え、前記制御部３６は、前記変換された前記ナット２０２の位置座標を前記第１学習済みモデル３９１に入力してもよい。 (6) The assembly system 100 according to the sixth aspect is any one of the assembly systems 100 of (1) to (4), in which the position coordinates of the nut 202 are expressed in a world coordinate system based on the origin of the manipulator 10, the control device 30 further includes a coordinate conversion unit 33 that converts the calculated position coordinates of the nut 202 from the world coordinate system to a bolt-centered coordinate system, and the control unit 36 may input the converted position coordinates of the nut 202 to the first trained model 391.
　これにより、世界座標系で表現されるナット２０２の位置座標がボルト２０１ごとに生じる固有の位置座標であるのに対して、ボルト２０１中心座標といった同一の尺度でナット２０２のボルト２０１に対する相対的な位置座標を表すことができる。 As a result, while the position coordinates of the nut 202 expressed in the world coordinate system are unique position coordinates that arise for each bolt 201, the relative position coordinates of the nut 202 with respect to the bolt 201 can be expressed on the same scale, such as the center coordinates of the bolt 201.
（７）第７の態様に係る組立システム１００は、（１）から（４）のうちいずれかの１つの組立システム１００であって、前記制御装置３０は、前記取得された各々の前記関節部１１０の関節角に基づく前記エンドエフェクタ１２の自重に関する補正値を用いて、前記取得された前記接触力を補正する補正部３４を更に備え、前記制御部３６は、前記補正された前記接触力を前記第１学習済みモデル３９１に入力してもよい。 (7) The assembly system 100 according to the seventh aspect is any one of the assembly systems 100 of (1) to (4), and the control device 30 further includes a correction unit 34 that corrects the acquired contact force using a correction value related to the weight of the end effector 12 based on the acquired joint angles of each of the joint units 110, and the control unit 36 may input the corrected contact force to the first trained model 391.
　これにより、エンドエフェクタ１２から伝達された接触力よりも実際に生じた接触力に近づいた接触力を第１学習済みモデル３９１の入力要素とすることができる。 This allows a contact force that is closer to the actual contact force than the contact force transmitted from the end effector 12 to be used as an input element for the first trained model 391.
（８）第８の態様に係る組立システム１００は、（１）から（４）のうちいずれかの１つの組立システム１００であって、前記第２センサ１４は、各々の前記関節部１１０が有するモータ１１７の電流値を取得可能な電流センサ１４１であってもよい。 (8) The assembly system 100 according to the eighth aspect is any one of the assembly systems 100 of (1) to (4), and the second sensor 14 may be a current sensor 141 capable of acquiring the current value of the motor 117 of each of the joints 110.
　これにより、例えば、第２センサ１４としての力覚センサ１４０などを用いる場合と比較して、マニピュレータ１０全体のサイズおよび重量を抑えることができる。 This allows the overall size and weight of the manipulator 10 to be reduced compared to when, for example, a force sensor 140 is used as the second sensor 14.
（９）第９の態様に係る組立システム１００は、（１）から（４）のうちいずれかの１つの組立システム１００であって、前記マニピュレータ１０が載置された台座部１５と、前記台座部１５を前記施工対象物２００の周囲で移動可能にさせる移動機構１６（車輪１６０、電動スライダ１６１）と、を更に備えてもよい。 (9) The assembly system 100 according to the ninth aspect may be any one of the assembly systems 100 of (1) to (4), and may further include a base 15 on which the manipulator 10 is placed, and a movement mechanism 16 (wheels 160, electric slider 161) that enables the base 15 to move around the construction target 200.
　これにより、より広い範囲でマニピュレータ１０が動作を行うことができる。 This allows the manipulator 10 to operate over a wider range.
（１０）第１０の態様に係る組立システム１００は、（３）または（４）の組立システム１００であって、前記エンドエフェクタ１２は、基部１２２と、前記基部１２２から延びるとともに前記軸線を中心とした筒状を成し、先端側に前記ナット２０２の少なくとも一部を収容可能な収容部１２３ｒが形成されたソケット部１２３とを有するナットランナ１２１と、前記ソケット部１２３の外周側に回動可能に固定されて前記軸線に交差する方向に延びる第１部分１２６と、前記第１部分１２６から前記先端側に延びる延在部１２７ａおよび前記延在部１２７ａの先端に設けられて前記軸線に近づくように前記延在部１２７ａから突出することで前記ナット２０２に当接する爪部１２７ｂを有する第２部分１２７とを有するナット保持部材１２４と、前記基部１２２と前記第１部分１２６との間に設けられ、前記第１部分１２６に対して前記先端側に向かう弾性力を付与する弾性部材１２８と、を有してもよい。 (10) The assembly system 100 according to the tenth aspect is the assembly system 100 according to (3) or (4), and the end effector 12 includes a nut runner 121 having a base 122, a socket portion 123 extending from the base 122 and having a cylindrical shape centered on the axis, the socket portion 123 having an accommodating portion 123r formed at the tip end capable of accommodating at least a portion of the nut 202, and a second nut runner 121 fixed to the outer periphery of the socket portion 123 and extending in a direction intersecting the axis. The nut holding member 124 may have a first part 126, an extension part 127a extending from the first part 126 toward the tip side, and a second part 127 having a claw part 127b provided at the tip of the extension part 127a and protruding from the extension part 127a so as to approach the axis and abutting against the nut 202, and an elastic member 128 provided between the base 122 and the first part 126 and applying an elastic force to the first part 126 toward the tip side.
　これにより、例えば、ナット２０２を吸引して保持する機構を備えたエンドエフェクタ１２などと比較して、より簡易な構成でエンドエフェクタ１２がナット２０２を保持することができる。 This allows the end effector 12 to hold the nut 202 with a simpler configuration than, for example, an end effector 12 equipped with a mechanism for suctioning and holding the nut 202.
（１１）第１１の態様に係る組立システム１００は、（１０）の組立システム１００であって、前記撮像装置２０は、前記収容部１２３ｒよりも前記基部１２２に近い側の前記ソケット部１２３内に配置され、前記撮像装置２０は、前記ナット２０２の孔（ナット孔２０２ｈ）を通じて、前記施工対象物２００を撮像した前記画像を生成する撮像部２１と、前記ナット２０２の孔（ナット孔２０２ｈ）を通じて、前記施工対象物２００に向けて光を照射する照明部２２と、を有してもよい。 (11) The assembly system 100 according to the eleventh aspect is the assembly system 100 according to (10), in which the imaging device 20 is disposed in the socket portion 123 closer to the base portion 122 than the storage portion 123r, and the imaging device 20 may have an imaging unit 21 that generates the image of the construction object 200 captured through a hole (nut hole 202h) of the nut 202, and an illumination unit 22 that irradiates light toward the construction object 200 through the hole (nut hole 202h) of the nut 202.
　これにより、撮像装置２０の撮像範囲を集中的に照らしながら画像を生成することができる。 This allows images to be generated while concentrating illumination on the imaging range of the imaging device 20.
（１２）第１２の態様に係る制御方法は、複数の関節部１１０を有するアーム１１と、前記アーム１１によって移動されるとともに、ナット２０２を保持して前記ナット２０２を軸線回りに回転させるエンドエフェクタ１２と、各々の前記関節部１１０の関節角を取得する第１センサ１３と、施工対象物２００に設けられたボルト２０１に前記ナット２０２が接触した場合に、前記エンドエフェクタ１２から伝達された接触力およびトルクを取得する第２センサ１４と、を有するマニピュレータ１０を制御する制御方法であって、各々の前記関節部１１０の関節角と、前記接触力およびトルクとを第１所定周期で取得するステップと、前記取得された各々の前記関節部１１０の関節角に基づき、前記ナット２０２の位置座標および姿勢角を算出するステップと、第１時刻における前記ナット２０２の位置座標および姿勢角と、前記第１時刻における前記接触力およびトルクとが入力されると、前記ナット２０２が前記ボルト２０１に締結された状態を示す前記ナット２０２の目標位置座標および目標姿勢角に近づいた、前記第１時刻よりも後の第２時刻における前記ナット２０２の位置座標および姿勢角を示す指令値を出力するように学習された第１学習済みモデル３９１を用いて、前記エンドエフェクタ１２を制御するステップと、を実行する。 (12) A control method according to a twelfth aspect is a control method for controlling a manipulator 10 having an arm 11 having a plurality of joints 110, an end effector 12 that is moved by the arm 11 and holds a nut 202 to rotate the nut 202 around an axis, a first sensor 13 that acquires joint angles of each of the joints 110, and a second sensor 14 that acquires a contact force and torque transmitted from the end effector 12 when the nut 202 contacts a bolt 201 provided on a construction target 200, the control method including: a switch that acquires the joint angle of each of the joints 110 and the contact force and torque at a first predetermined period; a step of calculating a position coordinate and an attitude angle of the nut 202 based on the acquired joint angles of each of the joint parts 110; and a step of controlling the end effector 12 using a first trained model 391 that has been trained to output command values indicating the position coordinate and attitude angle of the nut 202 at a second time after the first time when the position coordinate and attitude angle of the nut 202 approach the target position coordinate and target attitude angle of the nut 202 that indicate a state in which the nut 202 is fastened to the bolt 201 when the position coordinate and attitude angle of the nut 202 at a first time and the contact force and torque at the first time are input.
（１３）第１３の態様に係るプログラムは、複数の関節部１１０を有するアーム１１と、前記アーム１１によって移動されるとともに、ナット２０２を保持して前記ナット２０２を軸線回りに回転させるエンドエフェクタ１２と、各々の前記関節部１１０の関節角を取得する第１センサ１３と、施工対象物２００に設けられたボルト２０１に前記ナット２０２が接触した場合に、前記エンドエフェクタ１２から伝達された接触力およびトルクを取得する第２センサ１４と、を有するマニピュレータ１０を制御する制御装置３０のコンピュータ１１００に、各々の前記関節部１１０の関節角と、前記接触力およびトルクとを第１所定周期で取得するステップと、前記取得された各々の前記関節部１１０の関節角に基づき、前記ナット２０２の位置座標および姿勢角を算出するステップと、第１時刻における前記ナット２０２の位置座標および姿勢角と、前記第１時刻における前記接触力およびトルクとが入力されると、前記ナット２０２が前記ボルト２０１に締結された状態を示す前記ナット２０２の目標位置座標および目標姿勢角に近づいた、前記第１時刻よりも後の第２時刻における前記ナット２０２の位置座標および姿勢角を示す指令値を出力するように学習された第１学習済みモデル３９１を用いて、前記エンドエフェクタ１２を制御するステップと、を実行させる。 (13) The program relating to the thirteenth aspect includes an arm 11 having a plurality of joints 110, an end effector 12 that is moved by the arm 11 and holds a nut 202 to rotate the nut 202 around an axis, a first sensor 13 that acquires the joint angles of each of the joints 110, and a second sensor 14 that acquires the contact force and torque transmitted from the end effector 12 when the nut 202 contacts a bolt 201 provided on a construction object 200. The program instructs a computer 1100 of a control device 30 that controls a manipulator 10 having the arm 11 having a plurality of joints 110, a first sensor 13 that acquires the joint angles of each of the joints 110, and the contact force and torque transmitted from the end effector 12 when the nut 202 contacts a bolt 201 provided on a construction object 200 to transmit the joint angles of each of the joints 110 and the contact force and torque at a first predetermined period. The method executes the steps of: acquiring the position coordinates and the attitude angle of the nut 202 based on the acquired joint angles of each of the joint parts 110; calculating the position coordinates and the attitude angle of the nut 202 based on the acquired joint angles of each of the joint parts 110; and controlling the end effector 12 using a first trained model 391 that has been trained to output command values indicating the position coordinates and the attitude angle of the nut 202 at a second time after the first time when the position coordinates and the attitude angle of the nut 202 approach the target position coordinates and the target attitude angle of the nut 202 that indicate the state in which the nut 202 is fastened to the bolt 201 when the position coordinates and the attitude angle of the nut 202 at a first time and the contact force and the torque at the first time are input.
　本開示によれば、より円滑に締結することができる組立システム、制御方法、およびプログラムを提供することができる。 The present disclosure provides an assembly system, control method, and program that allows for smoother fastening.
１０…マニピュレータ
１１…アーム
１２…エンドエフェクタ
１３…第１センサ
１４…第２センサ
１５…台座部
１６…移動機構
１７…キャリブレーション機構
２０…撮像装置
２１…撮像部
２２…照明部
３０…制御装置
３１…取得部
３２…算出部
３３…座標変換部
３４…補正部
３５…判定部
３６…制御部
３７…学習部
３８…カウンタ部
３９…記憶部
１００…組立システム
１１０…関節部
１１１…第１関節部
１１２…第２関節部
１１３…第３関節部
１１４…第４関節部
１１５…第５関節部
１１６…第６関節部
１１７…モータ
１１７ａ…第１モータ
１１７ｂ…第２モータ
１１７ｃ…第３モータ
１１７ｄ…第４モータ
１１７ｅ…第５モータ
１１７ｆ…第６モータ
１１８…接続部
１１８ａ…第１接続部
１１８ｂ…第２接続部
１１８ｃ…第３接続部
１２０…本体部
１２０ａ…締結用モータ
１２１…ナットランナ
１２２…基部
１２２ａ…前面
１２３…ソケット部
１２３ａ…テーパ面
１２３ｂ…締結面
１２３ｃ…位置決め面
１２３ｅ…開口
１２３ｉ…内周面
１２３ｏ…外周面
１２３ｒ…収容部
１２４…ナット保持部材
１２５…回動軸
１２６…第１部分
１２７…第２部分
１２７ａ…延在部
１２７ｂ…爪部
１２８…弾性部材
１３０…エンコーダ
１３１…第１エンコーダ
１３２…第２エンコーダ
１３３…第３エンコーダ
１３４…第４エンコーダ
１３５…第５エンコーダ
１３６…第６エンコーダ
１４０…力覚センサ
１４１…電流センサ
１６０…車輪
１６１…電動スライダ
２００…施工対象物
２０１…ボルト
２０１ａ…雄ねじ部
２０１ｅ，２０２ｅ…端面
２０２…ナット
２０２ａ…ナット本体
２０２ｂ…カラー
２０２ｈ…ナット孔
２１０…第１部品
２２０…第２部品
３００…作業台
３５１…第１判定部
３５２…第２判定部
３５３…第３判定部
３５４…第４判定部
３５５…第５判定部
３５６…第６判定部
３６１…第１制御部
３６２…第２制御部
３６３…第３制御部
３９１…第１学習済みモデル
３９２…第２学習済みモデル
３９３…第３学習済みモデル
３９４…第４学習済みモデル
３９５…ＣＡＤデータ
３９６…第５学習済みモデル
３９７…ばらつき生成モデル
１１００…コンピュータ
１１１０…プロセッサ
１１２０…メインメモリ
１１３０…ストレージ
１１４０…インターフェース
Ａｒ…回転軸線
Ａｒ１…軸線
Ａｒ２…中心線
ＣＧ…エンドエフェクタの重心位置
Ｄａ…軸線方向
Ｄａ１…一方側
Ｄａ２…他方側
Ｄｖ…鉛直方向
Ｆｂ…接触力
Ｆｇ…自重
Ｆｇ１…補正値
Ｆｔ…トルク
Ｏ１…第１回動軸線
Ｏ２…第２回動軸線
Ｏ３…第３回動軸線
Ｏ４…第４回動軸線
Ｏ５…第５回動軸線
Ｏ６…第６回動軸線
Ｒ…収容空間
Ｓ…制御対象装置 10...manipulator 11...arm 12...end effector 13...first sensor 14...second sensor 15...base 16...movement mechanism 17...calibration mechanism 20...imaging device 21...imaging section 22...illumination section 30...control device 31...acquisition section 32...calculation section 33...coordinate conversion section 34...correction section 35...determination section 36...control section 37...learning section 38...counter section 39...storage section 100...assembly system 110...joint section 111...first joint section 112...second joint section 113...third joint section 114...fourth joint section 115...fifth joint section 116...sixth joint section 117...motor 117a...first motor 117b...second motor 117c...third motor 117d...fourth Motor 117e...fifth motor 117f...sixth motor 118...connection portion 118a...first connection portion 118b...second connection portion 118c...third connection portion 120...main body portion 120a...fastening motor 121...nut runner 122...base portion 122a...front surface 123...socket portion 123a...tapered surface 123b...fastening surface 123c...positioning surface 123e...opening 123i...inner circumferential surface 123o...outer circumferential surface 123r...accommodating portion 124...nut holding member 125...rotating shaft 126...first portion 127...second portion 127a...extending portion 127b...claw portion 128...elastic member 130...encoder 131...first encoder 132...second encoder 133...third encoder 13 4...fourth encoder 135...fifth encoder 136...sixth encoder 140...force sensor 141...current sensor 160...wheel 161...electric slider 200...work object 201...bolt 201a...male threaded portion 201e, 202e...end face 202...nut 202a...nut body 202b...collar 202h...nut hole 210...first part 220...second part 300...workbench 351...first judgment unit 352...second judgment unit 353...third judgment unit 354...fourth judgment unit 355...fifth judgment unit 356...sixth judgment unit 361...first control unit 362...second control unit 363...third control unit 391...first learned model 392...second learned model 393 ...Third trained model 394...Fourth trained model 395...CAD data 396...Fifth trained model 397...Variation generation model 1100...Computer 1110...Processor 1120...Main memory 1130...Storage 1140...Interface Ar...Rotation axis Ar1...Axis Ar2...Center line CG...Center of gravity position of end effector Da...Axial direction Da1...One side Da2...Other side Dv...Vertical direction Fb...Contact force Fg...Self-weight Fg1...Correction value Ft...Torque O1...First rotation axis O2...Second rotation axis O3...Third rotation axis O4...Fourth rotation axis O5...Fifth rotation axis O6...Sixth rotation axis R...Accommodation space S...Device to be controlled

Claims (13)

  1. 　複数の関節部を有するアームと、
    　前記アームによって移動されるとともに、ナットを保持して前記ナットを軸線回りに回転させるエンドエフェクタと、
    　各々の前記関節部の関節角を取得する第１センサと、
    　施工対象物に設けられたボルトに前記ナットが接触した場合に、前記エンドエフェクタから伝達された接触力およびトルクを取得する第２センサと、
    　を有するマニピュレータと、
    　前記マニピュレータを制御する制御装置と、
    　を備え、
    　前記制御装置は、
    　各々の前記関節部の関節角と、前記接触力およびトルクとを第１所定周期で取得する取得部と、
    　前記取得された各々の前記関節部の関節角に基づき、前記ナットの位置座標および姿勢角を算出する算出部と、
    　第１時刻における前記ナットの位置座標および姿勢角と、前記第１時刻における前記接触力およびトルクとが入力されると、前記ナットが前記ボルトに締結された状態を示す前記ナットの目標位置座標および目標姿勢角に近づいた、前記第１時刻よりも後の第２時刻における前記ナットの位置座標および姿勢角を示す指令値を出力するように学習された第１学習済みモデルを用いて、前記エンドエフェクタを制御する制御部と、
    　を備える組立システム。     An arm having a plurality of joints;
    an end effector that is moved by the arm and holds a nut to rotate the nut about an axis;
    A first sensor that acquires a joint angle of each of the joint parts;
    a second sensor that acquires a contact force and a torque transmitted from the end effector when the nut comes into contact with a bolt provided on a work object;
    A manipulator having
    A control device for controlling the manipulator;
    Equipped with
    The control device includes:
    an acquisition unit that acquires the joint angle of each of the joints, the contact force, and the torque at a first predetermined period;
    a calculation unit that calculates a position coordinate and an attitude angle of the nut based on the acquired joint angles of each of the joint parts;
    a control unit that controls the end effector using a first trained model that has been trained to output, when a position coordinate and an attitude angle of the nut at a first time and the contact force and torque at the first time are input, command values indicating a position coordinate and an attitude angle of the nut at a second time that is later than the first time and that approaches a target position coordinate and a target attitude angle of the nut that indicate a state in which the nut is fastened to the bolt;
    An assembly system comprising:
  2. 　前記第１学習済みモデルは、
    　前記ナットの前記目標位置座標および目標姿勢角から、ランダムに与えられた量外れた前記ナットの既知位置座標および既知姿勢角が、前記目標位置座標および目標姿勢角になるように前記エンドエフェクタを制御する教示動作中に取得された複数の前記ナットの位置座標および姿勢角と、
    　各々の前記ナットの位置座標および姿勢角が取得された時刻に対応する前記接触力およびトルクと、
    　を含む教師データセットが入力される学習ステップが繰り返し実行されたことで、前記入力に応じた前記指令値を出力するように学習されている
    　請求項１に記載の組立システム。     The first trained model is
    A plurality of position coordinates and attitude angles of the nut acquired during a teaching operation for controlling the end effector so that known position coordinates and known attitude angles of the nut, which are deviated from the target position coordinates and target attitude angle of the nut by a randomly given amount, become the target position coordinates and target attitude angles;
    The contact force and torque corresponding to the time when the position coordinates and the attitude angle of each nut are acquired;
    The assembly system according to claim 1 , wherein the assembly system is trained to output the command value in response to an input by repeatedly executing a learning step in which a teacher data set including:
  3. 　前記エンドエフェクタに設けられ、前記施工対象物を撮像した画像を生成する撮像装置を更に備え、
    　前記取得部は、前記画像を第２所定周期で更に取得し、
    　前記制御部は、
    　前記ナットが前記ボルトに接触していない第１位置で前記取得された第３時刻における前記画像と、前記第３時刻における前記ナットの位置座標および姿勢角とが入力されると、前記ナットが前記ボルトに接触した第２位置に前記ナットが近づいた、前記第３時刻よりも後の第４時刻における前記ナットの位置座標および姿勢角を示す指令値を出力するように学習された第２学習済みモデルを用いて、前記エンドエフェクタを制御する
    　請求項１に記載の組立システム。     The end effector further includes an imaging device that generates an image of the object to be processed,
    The acquisition unit further acquires the image at a second predetermined period,
    The control unit is
    2. The assembly system according to claim 1, wherein when the image acquired at a third time at a first position where the nut is not in contact with the bolt and the position coordinates and attitude angle of the nut at the third time are input, the end effector is controlled using a second trained model that is trained to output command values indicating the position coordinates and attitude angle of the nut at a fourth time after the third time at which the nut approaches a second position where the nut is in contact with the bolt.
  4. 　前記第２学習済みモデルは、
    　前記第２位置から前記第１位置に前記エンドエフェクタを移動させるダイレクトティーチング動作中に取得された複数の前記画像と、
    　各々の前記画像が取得された時刻に対応する前記ナットの位置座標および姿勢角と、
    　を含む教師データセットが入力される学習ステップが繰り返し実行されたことで、前記入力に応じた前記指令値を出力するように学習されている
    　請求項３に記載の組立システム。     The second trained model is
    A plurality of the images acquired during a direct teaching operation of moving the end effector from the second position to the first position;
    The position coordinates and the attitude angle of the nut corresponding to the time when each of the images was acquired;
    The assembly system according to claim 3 , wherein the system is trained to output the command value in response to an input by repeatedly executing a learning step in which a teacher data set including the input is input.
  5. 　前記制御部は、
    　前記第２位置で前記取得された前記画像と、前記画像が取得された時刻に対応する前記ナットの位置座標および姿勢角と、前記画像が取得された時刻に対応する前記接触力およびトルクとが入力されると、前記ナットおよび前記ボルトの螺合の成否を示す判定結果を出力するように学習された第３学習済みモデルを用いて、前記エンドエフェクタを制御する
    　請求項３または請求項４に記載の組立システム。     The control unit is
    5. The assembly system according to claim 3, wherein the end effector is controlled using a third trained model that is trained to output a judgment result indicating whether or not the nut and the bolt are screwed together when the image acquired at the second position, the position coordinates and attitude angle of the nut corresponding to the time when the image was acquired, and the contact force and torque corresponding to the time when the image was acquired are input.
  6. 　前記ナットの位置座標は、前記マニピュレータが持つ原点に準拠した世界座標系で表現されており、
    　前記制御装置は、前記算出された前記ナットの位置座標を前記世界座標系からボルト中心座標系に変換する座標変換部を更に備え、
    　前記制御部は、前記変換された前記ナットの位置座標を前記第１学習済みモデルに入力する
    　請求項１から請求項４のうちいずれか１項に記載の組立システム。     The position coordinates of the nut are expressed in a world coordinate system based on the origin of the manipulator,
    The control device further includes a coordinate conversion unit that converts the calculated position coordinates of the nut from the world coordinate system to a bolt center coordinate system,
    The assembly system according to claim 1 , wherein the control unit inputs the converted position coordinates of the nut to the first trained model.
  7. 　前記制御装置は、前記取得された各々の前記関節部の関節角に基づく前記エンドエフェクタの自重に関する補正値を用いて、前記取得された前記接触力を補正する補正部を更に備え、
    　前記制御部は、前記補正された前記接触力を前記第１学習済みモデルに入力する
    　請求項１から請求項４のうちいずれか１項に記載の組立システム。     the control device further includes a correction unit that corrects the acquired contact force by using a correction value related to a weight of the end effector based on the acquired joint angles of each of the joint parts;
    The assembly system according to claim 1 , wherein the control unit inputs the corrected contact force into the first trained model.
  8. 　前記第２センサは、各々の前記関節部が有するモータの電流値を取得可能な電流センサである
    　請求項１から請求項４のうちいずれか１項に記載の組立システム。     The assembly system according to claim 1 , wherein the second sensor is a current sensor capable of acquiring a current value of a motor included in each of the joints.
  9. 　前記マニピュレータが載置された台座部と、
    　前記台座部を前記施工対象物の周囲で移動可能にさせる移動機構と、
    　を更に備える
    　請求項１から請求項４のうちいずれか１項に記載の組立システム。     A base portion on which the manipulator is placed;
    A moving mechanism that enables the base portion to move around the object to be processed;
    The assembly system according to any one of claims 1 to 4, further comprising:
  10. 　前記エンドエフェクタは、
    　基部と、前記基部から延びるとともに前記軸線を中心とした筒状を成し、先端側に前記ナットの少なくとも一部を収容可能な収容部が形成されたソケット部とを有するナットランナと、
    　前記ソケット部の外周側に回動可能に固定されて前記軸線に交差する方向に延びる第１部分と、前記第１部分から前記先端側に延びる延在部および前記延在部の先端に設けられて前記軸線に近づくように前記延在部から突出することで前記ナットに当接する爪部を有する第２部分とを有するナット保持部材と、
    　前記基部と前記第１部分との間に設けられ、前記第１部分に対して前記先端側に向かう弾性力を付与する弾性部材と、
    　を有する
    　請求項３または請求項４に記載の組立システム。     The end effector includes:
    a nut runner having a base portion and a socket portion extending from the base portion and having a cylindrical shape centered on the axis line, the socket portion having a housing portion formed at a tip end thereof capable of housing at least a portion of the nut;
    a nut holding member having a first portion rotatably fixed to an outer periphery of the socket portion and extending in a direction intersecting the axis, an extension portion extending from the first portion toward the tip end, and a second portion having a claw portion provided at a tip end of the extension portion and protruding from the extension portion so as to approach the axis and come into contact with the nut;
    an elastic member provided between the base portion and the first portion and configured to apply an elastic force toward the tip side to the first portion;
    The assembly system according to claim 3 or claim 4, further comprising:
  11. 　前記撮像装置は、前記収容部よりも前記基部に近い側の前記ソケット部内に配置され、
    　前記撮像装置は、
    　前記ナットの孔を通じて、前記施工対象物を撮像した前記画像を生成する撮像部と、
    　前記ナットの孔を通じて、前記施工対象物に向けて光を照射する照明部と、
    　を有する
    　請求項１０に記載の組立システム。     the imaging device is disposed in the socket portion closer to the base portion than the housing portion;
    The imaging device includes:
    An imaging unit that generates the image of the construction object by capturing an image of the construction object through a hole of the nut;
    An illumination unit that irradiates light toward the object through a hole in the nut;
    The assembly system of claim 10, comprising:
  12. 　複数の関節部を有するアームと、
    　前記アームによって移動されるとともに、ナットを保持して前記ナットを軸線回りに回転させるエンドエフェクタと、
    　各々の前記関節部の関節角を取得する第１センサと、
    　施工対象物に設けられたボルトに前記ナットが接触した場合に、前記エンドエフェクタから伝達された接触力およびトルクを取得する第２センサと、
    　を有するマニピュレータを制御する制御方法であって、
    　各々の前記関節部の関節角と、前記接触力およびトルクとを第１所定周期で取得するステップと、
    　前記取得された各々の前記関節部の関節角に基づき、前記ナットの位置座標および姿勢角を算出するステップと、
    　第１時刻における前記ナットの位置座標および姿勢角と、前記第１時刻における前記接触力およびトルクとが入力されると、前記ナットが前記ボルトに締結された状態を示す前記ナットの目標位置座標および目標姿勢角に近づいた、前記第１時刻よりも後の第２時刻における前記ナットの位置座標および姿勢角を示す指令値を出力するように学習された第１学習済みモデルを用いて、前記エンドエフェクタを制御するステップと、
    　を実行する制御方法。     An arm having a plurality of joints;
    an end effector that is moved by the arm and holds a nut to rotate the nut about an axis;
    A first sensor that acquires a joint angle of each of the joint parts;
    a second sensor that acquires a contact force and a torque transmitted from the end effector when the nut comes into contact with a bolt provided on a work object;
    A method for controlling a manipulator having a
    acquiring the joint angle of each of the joints, the contact force, and the torque at a first predetermined period;
    calculating a position coordinate and an attitude angle of the nut based on the acquired joint angles of each of the joint parts;
    controlling the end effector using a first trained model that has been trained to output, when a position coordinate and an attitude angle of the nut at a first time and the contact force and torque at the first time are input, command values indicating a position coordinate and an attitude angle of the nut at a second time that is later than the first time and that approaches a target position coordinate and a target attitude angle of the nut that indicate a state in which the nut is fastened to the bolt;
    A control method for performing the above.
  13. 　複数の関節部を有するアームと、
    　前記アームによって移動されるとともに、ナットを保持して前記ナットを軸線回りに回転させるエンドエフェクタと、
    　各々の前記関節部の関節角を取得する第１センサと、
    　施工対象物に設けられたボルトに前記ナットが接触した場合に、前記エンドエフェクタから伝達された接触力およびトルクを取得する第２センサと、
    　を有するマニピュレータを制御する制御装置のコンピュータに、
    　各々の前記関節部の関節角と、前記接触力およびトルクとを第１所定周期で取得するステップと、
    　前記取得された各々の前記関節部の関節角に基づき、前記ナットの位置座標および姿勢角を算出するステップと、
    　第１時刻における前記ナットの位置座標および姿勢角と、前記第１時刻における前記接触力およびトルクとが入力されると、前記ナットが前記ボルトに締結された状態を示す前記ナットの目標位置座標および目標姿勢角に近づいた、前記第１時刻よりも後の第２時刻における前記ナットの位置座標および姿勢角を示す指令値を出力するように学習された第１学習済みモデルを用いて、前記エンドエフェクタを制御するステップと、
    　を実行させるプログラム。     An arm having a plurality of joints;
    an end effector that is moved by the arm and holds a nut to rotate the nut about an axis;
    A first sensor that acquires a joint angle of each of the joint parts;
    a second sensor that acquires a contact force and a torque transmitted from the end effector when the nut comes into contact with a bolt provided on a work object;
    A computer of a control device for controlling a manipulator having the
    acquiring the joint angle of each of the joints, and the contact force and torque at a first predetermined period;
    calculating a position coordinate and an attitude angle of the nut based on the acquired joint angles of each of the joint parts;
    controlling the end effector using a first trained model that has been trained to output, when a position coordinate and an attitude angle of the nut at a first time and the contact force and torque at the first time are input, command values indicating a position coordinate and an attitude angle of the nut at a second time that is later than the first time and that approaches a target position coordinate and a target attitude angle of the nut that indicate a state in which the nut is fastened to the bolt;
    A program that executes the following.
PCT/JP2023/043647 2022-12-14 2023-12-06 Assembly system, control method, and program WO2024128092A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2022-199698 2022-12-14
JP2022199698A JP2024085261A (en) 2022-12-14 2022-12-14 ASSEMBLY SYSTEM, CONTROL METHOD, AND PROGRAM

Publications (1)

Publication Number Publication Date
WO2024128092A1 true WO2024128092A1 (en) 2024-06-20

Family

ID=91484903

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/043647 WO2024128092A1 (en) 2022-12-14 2023-12-06 Assembly system, control method, and program

Country Status (2)

Country Link
JP (1) JP2024085261A (en)
WO (1) WO2024128092A1 (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2019093461A (en) * 2017-11-20 2019-06-20 株式会社安川電機 Holding system, learning device, holding method and model manufacturing method
JP2020157380A (en) * 2019-03-25 2020-10-01 Ntn株式会社 Fitting work device
WO2020218533A1 (en) * 2019-04-26 2020-10-29 株式会社エスイーフォー Method and device for assigning attribute information to object

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2019093461A (en) * 2017-11-20 2019-06-20 株式会社安川電機 Holding system, learning device, holding method and model manufacturing method
JP2020157380A (en) * 2019-03-25 2020-10-01 Ntn株式会社 Fitting work device
WO2020218533A1 (en) * 2019-04-26 2020-10-29 株式会社エスイーフォー Method and device for assigning attribute information to object

Also Published As

Publication number Publication date
JP2024085261A (en) 2024-06-26

Similar Documents

Publication Publication Date Title
JP6811635B2 (en) Robot system and its control method
JP5114019B2 (en) Method for controlling the trajectory of an effector
JP5606241B2 (en) Visual cognitive system and method for humanoid robot
US20160375532A1 (en) Fastening device, robot system, and fastening method for fastening plurality of fastening members
US11977365B2 (en) Skill transfer mechanical apparatus
EP3403774B1 (en) Robot hand, robot apparatus, and control method for robot hand
Ju et al. Kinematics modeling and experimental verification of baxter robot
US7446496B2 (en) Abnormality detection system of mobile robot
JP6870122B2 (en) Robot system and robot control method
JP5586163B2 (en) Robot and control method thereof
US7348746B2 (en) Abnormality detection system of mobile robot
JP2015042437A (en) Robot system and calibration method of robot system
US20220212340A1 (en) Control device, control system, mechanical apparatus system, and controlling method
JP2010058256A (en) Arm position regulating method and apparatus and robot system
EP0411139A1 (en) Positioning correction for robots
JP6023251B2 (en) Robot system
WO2024128092A1 (en) Assembly system, control method, and program
JPH05261682A (en) Industrial robot calibration method
TW202231424A (en) Robot system and workpiece supply method
JP6995602B2 (en) Robot hand, robot hand control method, robot device, article manufacturing method, control program and recording medium
US20230038804A1 (en) Master-slave system and controlling method
Mobedi et al. An adaptive control approach to robotic assembly with uncertainties in vision and dynamics
US20210053238A1 (en) Deflection amount estimating device, robot control device and method of estimating amount of deflection
JP7308324B1 (en) modular robot
JP6985838B2 (en) Screw tightening method using a robot and robot for screw tightening