WO2022215262A1 - ロボット管理装置、制御方法及び記憶媒体 - Google Patents
ロボット管理装置、制御方法及び記憶媒体 Download PDFInfo
- Publication number
- WO2022215262A1 WO2022215262A1 PCT/JP2021/015053 JP2021015053W WO2022215262A1 WO 2022215262 A1 WO2022215262 A1 WO 2022215262A1 JP 2021015053 W JP2021015053 W JP 2021015053W WO 2022215262 A1 WO2022215262 A1 WO 2022215262A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- information
- robot
- external input
- operating terminal
- task
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims description 27
- 238000004891 communication Methods 0.000 claims description 34
- 238000012545 processing Methods 0.000 claims description 27
- 230000008569 process Effects 0.000 claims description 16
- 230000033001 locomotion Effects 0.000 description 35
- 230000015654 memory Effects 0.000 description 30
- 230000009471 action Effects 0.000 description 15
- 230000006870 function Effects 0.000 description 10
- 230000005856 abnormality Effects 0.000 description 5
- 230000004044 response Effects 0.000 description 5
- 230000007704 transition Effects 0.000 description 5
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 230000002123 temporal effect Effects 0.000 description 4
- 238000011960 computer-aided design Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000012546 transfer Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 239000012636 effector Substances 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000036544 posture Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000002310 reflectometry Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1679—Programme controls characterised by the tasks executed
- B25J9/1689—Teleoperation
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J13/00—Controls for manipulators
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1628—Programme controls characterised by the control loop
- B25J9/1653—Programme controls characterised by the control loop parameters identification, estimation, stiffness, accuracy, error analysis
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1656—Programme controls characterised by programming, planning systems for manipulators
- B25J9/1669—Programme controls characterised by programming, planning systems for manipulators characterised by special application, e.g. multi-arm co-operation, assembly, grasping
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
- G06Q10/06398—Performance of employee with respect to a job function
Definitions
- the present disclosure relates to the technical field of controlling the motion of robots.
- Patent Literature 1 discloses a robot system that requests remote operation to an operation terminal operated by an operator when autonomous control alone is difficult.
- Patent Document 1 When operating a robot based on external input, the required operations differ depending on the work content.
- the robot system described in Patent Document 1 is an interactive robot, and does not consider the diversity of operation methods when selecting an operation terminal.
- One of the objects of the present disclosure is to provide a robot management device, a control method, and a storage medium that can suitably determine an operation terminal that generates an external input, in view of the above-described problems.
- One aspect of the robot management device includes: external input necessity determination means for determining necessity of control based on external input of a robot executing a task; When the control based on the external input is required, the external input is generated based on operating terminal information including information on the types of a plurality of operating terminals that are candidates for generating the external input, and information on the task.
- an operating terminal determining means for determining an operating terminal It is a robot management device having
- One aspect of the control method is the computer Judging whether or not control is necessary based on the external input of the robot executing the task, When the control based on the external input is required, the external input is generated based on operating terminal information including information on the types of a plurality of operating terminals that are candidates for generating the external input, and information on the task. determine the operating terminal, control method.
- One aspect of the storage medium is Judging whether or not control is necessary based on the external input of the robot executing the task, When the control based on the external input is required, the external input is generated based on operating terminal information including information on the types of a plurality of operating terminals that are candidates for generating the external input, and information on the task.
- a storage medium storing a program for causing a computer to execute processing for determining an operation terminal.
- FIG. 1 shows the configuration of a robot control system according to a first embodiment;
- A shows the hardware configuration of the robot controller;
- B shows the hardware configuration of the operation terminal;
- C shows the hardware configuration of the robot management device;
- An example of the data structure of application information is shown.
- A It is an example of the data structure of operating terminal information.
- B is an example of the data structure of operator information.
- It is an example of a functional block showing an outline of processing of a robot control system.
- It is an example of a functional block showing a functional configuration of an operation sequence generator.
- It is a first display example of the operation screen.
- It is a second display example of the operation screen.
- 4 is an example of a flowchart executed by the robot management device in the first embodiment;
- the schematic block diagram of the robot management apparatus in 2nd Embodiment is shown. It is an example of a flow chart showing processing of the robot management device in the second embodiment.
- FIG. 1 shows the configuration of a robot control system 100 according to the first embodiment.
- the robot control system 100 mainly includes operation terminals 2 (2A, 2B, . . . ), a robot management device 3, and a plurality of task execution systems 50 (50A, 50B, . . . ).
- the operation terminal 2 , the robot management device 3 and the task execution system 50 perform data communication via the communication network 6 .
- the operation terminal 2 is a terminal that receives support operations necessary for the robot 5 in the task execution system 50 to execute a task, and is used by operators (operators a to c, . . . ). Specifically, when there is a task execution system 50 requesting assistance, the operation terminal 2 establishes a communication connection with the task execution system 50 based on connection control by the robot management device 3, and the operator's operation ( The external input signal "Se" generated by the manual operation) is sent to the task execution system 50 of the request source.
- the external input signal Se is an operator's input signal representing a command that directly or indirectly defines the motion of the robot 5 that requires assistance.
- the operation terminals 2 include multiple types of terminals with different operation methods.
- the operation terminal 2 includes a tablet terminal, a stationary personal computer, a game controller, a virtual reality (VR) terminal, and the like. Then, as will be described later, when there is a task execution system 50 requesting support, an appropriate type of operation terminal 2 is selected as the operation terminal 2 that provides support according to the content of the task to be supported.
- VR virtual reality
- the operation terminal 2 and the operator are not necessarily in a one-to-one relationship. There may be a mode of use and the like.
- the operation terminal 2 may further receive an operator's input specifying a task to be executed in the task execution system 50 . In this case, the operation terminal 2 transmits the task designation information generated by the input to the target task execution system 50 .
- the robot management device 3 manages the connection between the task execution system 50 that requires support from the operation terminal 2 and the operation terminal 2 that provides support.
- the robot management device 3 selects the operation terminal 2 (and operator) suitable for supporting the requesting task execution system 50. Then, the selected operation terminal 2 (also referred to as "target operation terminal 2") and the task execution system 50 of the request source execute connection control for establishing a communication connection.
- the mode of communication between the task execution system 50 and the target operation terminal 2 may be a mode in which a VPN (Virtual Private Network) or the like is constructed and data communication is performed directly. Data communication may be performed indirectly by performing communication transfer processing.
- the robot management device 3 as connection control, for example, controls at least one of the task execution system 50 (specifically, the robot controller 1) and the target operation terminal 2 so that they can directly communicate with each other. process to send the communication address of In the latter mode, the robot management device 3 performs, as connection control, a process of establishing transfer-only communication connections with each of the task execution system 50 and the target operation terminal 2, for example.
- the task execution system 50 is a robot system that executes designated tasks and is provided in different environments.
- the task execution system 50 may be a system that performs picking work in a factory (for example, picking parts from a shelf and placing the picked parts on a tray) as a task, and any robot other than the factory. It may be a system. Examples of such robot systems are retail shelving (for example, arranging products in containers on store shelves), product checking (for example, removing expired products from shelves or discounting them). This includes robot systems that apply stickers).
- Each task execution system 50 includes a robot controller 1 (1A, 1B,%), a robot 5 (5A, 5B,%), and a sensor 7 (7A, 7B,).
- the robot controller 1 formulates a motion plan for the robot 5 and controls the robot 5 based on the motion plan. For example, the robot controller 1 converts a task represented by temporal logic into a sequence for each time step (time increments) of tasks that can be accepted by the robot 5, and controls the robot 5 based on the generated sequence.
- a task (command) obtained by decomposing a task into units acceptable to the robot 5 is also called a "subtask”
- a sequence of subtasks to be executed by the robot 5 in order to accomplish the task is called a "subtask sequence” or an "operation”. called sequence.
- the subtasks include tasks that require assistance (that is, manual control) from the operation terminal 2 .
- the robot controller 1 also has an application information storage unit 41 (41A, 41B, . Details of the application information will be described later with reference to FIG.
- the robot controller 1 performs data communication with the robot 5 and the sensor 7 belonging to the same task execution system 50 via a communication network or direct wireless or wired communication. For example, the robot controller 1 transmits control signals for controlling the robot 5 to the robot 5 . In another example, robot controller 1 receives a sensor signal generated by sensor 7 . Furthermore, the robot controller 1 performs data communication with the operation terminal 2 via the communication network 6 .
- One or a plurality of robots 5 exist for each task execution system 50 and perform task-related work based on control signals supplied from robot controllers 1 belonging to the same task execution system 50 .
- the robot 5 may be a vertical articulated robot, a horizontal articulated robot, or any other type of robot, and has control objects (manipulators, end effectors) such as robot arms that operate independently. may have multiple Also, the robot 5 may perform cooperative work with other robots, workers, or machine tools that operate within the workspace. Also, the robot controller 1 and the robot 5 may be configured integrally.
- the robot 5 may supply a status signal indicating the status of the robot 5 to the robot controller 1 belonging to the same task execution system 50 .
- This state signal may be an output signal of a sensor that detects the state (position, angle, etc.) of the entire robot 5 or a specific part such as a joint, and is a signal indicating the progress of the motion sequence supplied to the robot 5. There may be.
- the sensor 7 is one or more sensors such as a camera, range sensor, sonar, or a combination thereof that detect the state in the workspace where tasks are executed in each task execution system 50 .
- the sensor 7 supplies the generated signal (also called “sensor signal”) to the robot controller 1 belonging to the same task execution system 50 .
- the sensors 7 may be self-propelled or flying sensors (including drones) that move within the workspace.
- the sensors 7 may also include sensors provided on the robot 5, sensors provided on other objects in the work space, and the like. Sensors 7 may also include sensors that detect sounds within the workspace. In this way, the sensor 7 may include various sensors that detect conditions within the work space, and may include sensors provided at arbitrary locations.
- the configuration of the robot control system 100 shown in FIG. 1 is an example, and various modifications may be made to the configuration.
- the robot control functions of the robot controller 1 may be integrated into the robot management device 3 .
- the robot management device 3 generates an action sequence for the robot 5 in each task execution system 50 and performs control necessary for the robot 5 to execute the action sequence.
- the robot management device 3 may consist of multiple devices.
- the plurality of devices constituting the robot management device 3 exchange information necessary for executing previously assigned processing among the plurality of devices.
- the application information storage unit 41 may be composed of one or more external storage devices that perform data communication with the robot controller 1 .
- the external storage device may be one or a plurality of server devices that store the application information storage unit 41 commonly referred to by each task execution system 50 .
- FIG. 2A shows the hardware configuration of the robot controller 1 (1A, 1B, . . . ).
- the robot controller 1 includes a processor 11, a memory 12, and an interface 13 as hardware.
- Processor 11 , memory 12 and interface 13 are connected via data bus 10 .
- the processor 11 functions as a controller (arithmetic device) that performs overall control of the robot controller 1 by executing programs stored in the memory 12 .
- the processor 11 is, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit).
- Processor 11 may be composed of a plurality of processors.
- the memory 12 is composed of various volatile and nonvolatile memories such as RAM (Random Access Memory), ROM (Read Only Memory), and flash memory.
- the memory 12 also stores a program for executing the process executed by the robot controller 1 .
- the memory 12 also functions as an application information storage unit 41 . Part of the information stored in the memory 12 may be stored in one or a plurality of external storage devices that can communicate with the robot controller 1, or may be stored in a storage medium detachable from the robot controller 1. good.
- the interface 13 is an interface for electrically connecting the robot controller 1 and other devices. These interfaces may be wireless interfaces such as network adapters for wirelessly transmitting and receiving data to and from other devices, or hardware interfaces for connecting to other devices via cables or the like.
- wireless interfaces such as network adapters for wirelessly transmitting and receiving data to and from other devices, or hardware interfaces for connecting to other devices via cables or the like.
- the hardware configuration of the robot controller 1 is not limited to the configuration shown in FIG. 2(A).
- the robot controller 1 may be connected to or built in at least one of a display device, an input device, and a sound output device.
- FIG. 2(B) shows the hardware configuration of the operation terminal 2.
- the operation terminal 2 includes, as hardware, a processor 21, a memory 22, an interface 23, an input section 24a, a display section 24b, and a sound output section 24c.
- Processor 21 , memory 22 and interface 23 are connected via data bus 20 .
- the interface 23 is also connected to an input section 24a, a display section 24b, and a sound output section 24c.
- the processor 21 executes a predetermined process by executing a program stored in the memory 22.
- the processor 21 is a processor such as a CPU, GPU, or TPU.
- the processor 21 also controls at least one of the display unit 24b and the sound output unit 24c through the interface 23 based on information received from the task execution system 50 through the interface 23.
- FIG. Thereby, the processor 21 presents the operator with information that supports the operation that the operator should perform.
- the processor 21 transmits the signal generated by the input unit 24a via the interface 23 to the task execution system 50, which is the transmission source of the support request information Sr, as an external input signal Se.
- Processor 21 may be composed of a plurality of processors.
- Processor 21 is an example of a computer.
- the memory 22 is composed of various volatile and nonvolatile memories such as RAM, ROM, and flash memory. In addition, the memory 22 stores programs for executing processes executed by the operation terminal 2 .
- the interface 23 is an interface for electrically connecting the operation terminal 2 and other devices. These interfaces may be wireless interfaces such as network adapters for wirelessly transmitting and receiving data to and from other devices, or hardware interfaces for connecting to other devices via cables or the like. Further, the interface 23 performs interface operations of the input section 24a, the display section 24b, and the sound output section 24c.
- the input unit 24a is an interface that receives user input, and corresponds to, for example, a touch panel, buttons, keyboard, voice input device, and the like.
- the input unit 24 a may include an operation unit that receives user input representing a command that directly defines the motion of the robot 5 .
- This operation unit may be, for example, a robot controller (operating panel) operated by a user in controlling the robot 5 based on an external input, and is a robot controller that generates an operation command to the robot 5 according to the user's movement. It may be an input system or it may be a game controller.
- the above-described robot controller includes, for example, various buttons for designating the parts of the robot 5 to be moved and for designating movements, and an operation bar for designating the direction of movement.
- the robot input system described above includes, for example, various sensors (eg, cameras, wearable sensors, etc.) used in motion capture and the like.
- the display unit 24b is, for example, a display, a projector, etc., and performs display under the control of the processor 21. Also, the display unit 24b may be a combination of a combiner (a plate-shaped member having reflectivity and transparency) for realizing virtual reality and a light source that emits display light. Also, the sound output unit 24 c is, for example, a speaker, and outputs sound based on the control of the processor 21 .
- the hardware configuration of the operation terminal 2 is not limited to the configuration shown in FIG. 2(B).
- at least one of the input unit 24a, the display unit 24b, and the sound output unit 24c may be configured as a separate device electrically connected to the operation terminal 2.
- the operation terminal 2 may be connected to various devices such as a camera, or may incorporate them.
- FIG. 2(C) shows the hardware configuration of the robot management device 3.
- the robot management device 3 includes a processor 31, a memory 32, and an interface 33 as hardware.
- Processor 31 , memory 32 and interface 33 are connected via data bus 30 .
- the processor 31 functions as a controller (arithmetic device) that controls the entire robot management device 3 by executing programs stored in the memory 32 .
- the processor 31 is, for example, a processor such as a CPU, GPU, or TPU.
- the processor 31 may be composed of multiple processors.
- Processor 31 is an example of a computer.
- the memory 32 is composed of various volatile and nonvolatile memories such as RAM, ROM, and flash memory.
- the memory 32 also stores a program for executing the process executed by the robot controller 1 .
- the memory 32 stores operating terminal information 38, which is information about the operating terminal 2 capable of receiving support for the task execution system 50, and operator information 39, which is information about the operator who operates the operating terminal 2. . Details of the operating terminal information 38 and the operator information 39 will be described later.
- the operating terminal information 38 and the operator information 39 may be information generated based on an administrator's input by an input unit (not shown) connected via the interface 33 and stored in the memory 32. Information received from the operation terminal 2 or the like may be used.
- the interface 33 is an interface for electrically connecting the robot management device 3 and other devices. These interfaces may be wireless interfaces such as network adapters for wirelessly transmitting and receiving data to and from other devices, or hardware interfaces for connecting to other devices via cables or the like.
- the hardware configuration of the robot management device 3 is not limited to the configuration shown in FIG. 2(C).
- the robot management device 3 may be connected to or built in at least one of a display device, an input device, and a sound output device.
- FIG. 3 shows an example of the data structure of application information.
- the application information includes abstract state designation information I1, constraint information I2, motion limit information I3, subtask information I4, abstract model information I5, and object model information I6.
- the abstract state designation information I1 is information that designates an abstract state that needs to be defined when generating an operation sequence.
- This abstract state is an abstract state of an object in the work space, and is defined as a proposition to be used in a target logic formula to be described later.
- the abstract state designation information I1 designates an abstract state that needs to be defined for each task type.
- Constraint condition information I2 is information indicating the constraint conditions when executing a task. For example, if the task is a pick-and-place task, the constraint information I2 includes a constraint that the robot 5 (robot arm) must not come into contact with an obstacle, a constraint that the robots 5 (robot arms) must not come into contact with each other, and so on. Indicates conditions, etc. Note that the constraint condition information I2 may be information in which a constraint condition suitable for each type of task is recorded.
- the motion limit information I3 indicates information about the motion limits of the robot 5 controlled by the robot controller 1.
- the motion limit information I3 is, for example, information that defines the upper limits of the speed, acceleration, or angular velocity of the robot 5 .
- the motion limit information I3 may be information defining motion limits for each movable part or joint of the robot 5 .
- the subtask information I4 indicates information on subtasks that are components of the operation sequence.
- a “subtask” is a task obtained by decomposing a task into units that the robot 5 can accept, and refers to the subdivided motion of the robot 5 .
- the subtask information I4 defines reaching, which is movement of the robot arm of the robot 5, and grasping, which is grasping by the robot arm, as subtasks.
- the subtask information I4 may indicate information on subtasks that can be used for each type of task.
- the subtask information I4 includes information on subtasks that require operation commands by external input (also called “external input type subtasks").
- the subtask information I4 related to the external input type subtask includes, for example, identification information of the subtask, flag information for identifying the external input type subtask, and information on the operation content of the robot 5 in the external input type subtask. and are included.
- the subtask information I4 related to the external input type subtask may further include text information for requesting external input to the operation terminal 2, information related to expected work time length, and the like.
- the abstract model information I5 is information about an abstract model that abstracts the dynamics in the work space.
- the abstract model is represented by a model in which real dynamics are abstracted by a hybrid system, as will be described later.
- the abstract model information I5 includes information indicating conditions for switching dynamics in the hybrid system described above.
- the switching condition is, for example, in the case of pick-and-place, in which the robot 5 grabs an object to be worked on by the robot 5 (also called an “object”) and moves it to a predetermined position, the object must be gripped by the robot 5. This applies to conditions such as not being able to move
- the abstract model information I5 has information on an abstract model suitable for each type of task.
- the object model information I6 is information on the object model of each object in the work space to be recognized from the signal generated by the sensor 7.
- the objects described above correspond to, for example, the robot 5, obstacles, tools and other objects handled by the robot 5, working objects other than the robot 5, and the like.
- the object model information I6 includes, for example, information necessary for the robot controller 1 to recognize the types, positions, postures, motions currently being executed, etc. of each object described above, and information for recognizing the three-dimensional shape of each object. and three-dimensional shape information such as CAD (Computer Aided Design) data.
- the former information includes the parameters of the reasoner obtained by learning a learning model in machine learning such as a neural network. For example, when an image is input, this inference unit is trained in advance so as to output the type, position, orientation, etc. of an object that is a subject in the image.
- the application information storage unit 41 may store various information related to the operation sequence generation processing and the display processing required to receive the operation of generating the external input signal Se.
- FIG. 4A is an example of the data structure of the operating terminal information 38.
- the operating terminal information 38 exists for each operating terminal 2 and mainly includes a terminal ID 381, terminal type information 382, address information 383, and a corresponding operator ID 384. I'm in.
- the terminal ID 381 is the terminal ID of the corresponding operation terminal 2. Note that the terminal ID 381 may be arbitrary identification information with which the operation terminal 2 can be identified.
- the terminal type information 382 is information representing the terminal type of the corresponding operation terminal 2 . The types of operation terminals 2 are classified, for example, based on differences in accepted operation modes.
- the address information 383 is communication information necessary for communicating with the corresponding operation terminal 2, for example, information regarding communication addresses (including IP addresses, etc.) necessary for communication according to a predetermined communication protocol.
- the address information 383 is used, for example, in connection control for establishing communication connection between the target operation terminal 2 and the task execution system 50 .
- the corresponding operator ID 384 is identification information (operator ID) of an operator who operates the corresponding operation terminal 2 .
- the corresponding operator ID 384 may indicate operator IDs of a plurality of operators.
- FIG. 4(B) is an example of the data structure of the operator information 39.
- the operator information 39 exists for each operator registered as a person who supports the task execution system 50, and mainly includes an operator ID 391, skill information 392, Operation record information 393 , state management information 394 and operable terminal ID 395 are included.
- the operator ID 391 is the operator ID of the corresponding operator.
- the skill information 392 is information representing the skill (proficiency level) of the operation using the operation terminal 2 of the corresponding operator.
- the skill information 392 may indicate the skill level of the operation of the operator for each type of operation terminal 2 to be operated.
- the operation record information 393 is the operator's record information regarding the operation in response to the support request from the task execution system 50 .
- the operation record information 393 may indicate the operation record of the operator for each type of the operation terminal 2 operated.
- the operation record information 393 includes, for example, the number of operations in response to support requests from the task execution system 50, the registration period (years of experience) as an operator, and the number or ratio of successful and unsuccessful operations.
- “success and failure” are, for example, when the request for support is from the task execution system 50 based on the occurrence of an error, the external input signal Se from the operation terminal 2 is supplied to the requesting task execution system 50. A determination is made based on whether the error has been resolved. In cases other than the occurrence of an error, "success or failure” may be determined, for example, based on whether or not the task is normally completed after the external input signal Se is supplied from the operation terminal 2.
- the operation record information 393 may include an operation history generated for each operation in response to a support request from the task execution system 50.
- various information such as information on the task for which the task execution system 50 has requested support, information on the task execution system 50 that requested the support, information on the date and time when the operation was performed, operation time length, etc. are used as the operation history. It is recorded in the performance information 393 .
- the processor 31 updates the operation record information 393 based on the information received from the task execution system 50 that is the source of the request each time a support request is made from the task execution system 50 and support is provided in response to the request.
- the state management information 394 is information related to the state management of the corresponding operator. It may be information indicating whether or not the user is in a seated state.
- the processor 31 receives, for example, operator schedule information and presence/absence information from another system that manages the schedule of each operator, or receives manual input regarding the status of the operator. , the state management information 394 may be updated at a predetermined timing. Thus, each information of the operating terminal information 38 and the operator information 39 is updated at necessary timing.
- the operable terminal ID 395 is a terminal ID (terminal ID 381 in FIG. 4A) that can be operated by the corresponding operator. Note that the operable terminal ID 395 may be the terminal ID of one operation terminal 2 or the terminal IDs of a plurality of operation terminals 2 .
- the data structures of the operating terminal information 38 and the operator information 39 are not limited to the data structures shown in FIGS. 4(A) and 4(B).
- the operating terminal information 38 may further include information for managing the state of the operating terminal 2 , which corresponds to the state management information 394 included in the operator information 39 .
- the operating terminal information 38 and the operator information 39 may be integrated into one of them.
- FIG. 5 is an example of functional blocks outlining the processing of the robot control system 100 .
- the processor 11 of the robot controller 1 functionally includes an output control unit 15 , an operation sequence generation unit 16 , a robot control unit 17 and a switching determination unit 18 .
- the processor 21 of the operation terminal 2 functionally has an information presenting section 25 and an external control section 26 .
- the processor 31 of the robot management device 3 functionally includes an external input necessity determination unit 35 , an operation terminal determination unit 36 , and a connection control unit 37 .
- the blocks that exchange data are connected by solid lines, but the combinations of blocks that exchange data and the data that exchanges are not limited to those shown in FIG. The same applies to other functional block diagrams to be described later.
- FIG. 5 an example of an operation mode of the operator using the operation terminal 2 is illustrated within a balloon 60 .
- the robot controller 1 controls the robot 5 based on the generated operation sequence, and transmits support request information Sr to the robot management device 3 when determining that support from the operation terminal 2 is required. As a result, the robot controller 1 smoothly switches the control mode of the robot 5 to the control based on the external input signal Se (also referred to as "external input control") to accomplish the task even when automatic control alone cannot handle the task. .
- the external input signal Se also referred to as "external input control
- the output control unit 15 performs processing related to transmission of the support request information Sr via the interface 13 and reception of the external input signal Se. In this case, the output control unit 15 transmits support request information Sr requesting necessary external input to the operation terminal 2 when a switch command “Sw” to switch to external input control is supplied from the switching determination unit 18 . .
- the support request information Sr includes information on tasks (subtasks) that require support.
- the support request information Sr includes, for example, date and time information indicating the date and time when the request was required, type information of the robot 5 to be supported, task identification information, and subtask identification to be supported. information, expected working time length of the subtask, necessary operation contents of the robot 5, and error information regarding the error when an error occurs.
- the error information is an error code or the like indicating the type of error.
- the error information may include information (for example, video information) representing the situation at the time of error occurrence.
- the output control unit 15 controls the operation terminal 2 to display the operation screen of the operator. Information (also referred to as “operation screen information”) is transmitted to the operation terminal 2 . Further, when the output control unit 15 receives an external input signal Se from the operation terminal 2 , the output control unit 15 supplies the external input signal Se to the robot control unit 17 .
- the motion sequence generator 16 generates the motion sequence "Sv" of the robot 5 required to complete the specified task based on the signal output by the sensor 7 and the application information.
- the action sequence Sv corresponds to a sequence of subtasks (subtask sequence) that the robot 5 should perform in order to accomplish the task, and defines a series of actions of the robot 5 .
- the motion sequence generation unit 16 then supplies the generated motion sequence Sv to the robot control unit 17 and the switching determination unit 18 .
- the operation sequence Sv includes information indicating the execution order and execution timing of each subtask.
- the robot control unit 17 controls the operation of the robot 5 by supplying control signals to the robot 5 via the interface 13 . After receiving the motion sequence Sv from the motion sequence generator 16 , the robot controller 17 controls the robot 5 . In this case, the robot control unit 17 transmits a control signal to the robot 5 to execute position control or torque control of the joints of the robot 5 for realizing the motion sequence Sv. Then, the robot control unit 17 switches the control mode of the robot 5 to external input control based on the switching command Sw supplied from the switching determination unit 18 .
- the robot control unit 17 receives an external input signal Se generated by the operation terminal 2 via the interface 13 .
- This external input signal Se includes, for example, information that defines a specific motion of the robot 5 (for example, information corresponding to a control input that directly determines the motion of the robot 5).
- the robot control unit 17 controls the robot 5 by generating a control signal based on the received external input signal Se and transmitting the generated control signal to the robot 5 .
- the control signal generated by the robot control unit 17 in the external input control is, for example, a signal obtained by converting the external input signal Se into a data format that the robot 5 can accept.
- the robot control section 17 may supply the external input signal Se to the robot 5 as it is as a control signal.
- the external input signal Se may be information that assists the task execution system 50 in recognizing the work state, instead of being information that defines the specific motion of the robot 5 .
- the external input signal Se may be information indicating the position of the object.
- the operation terminal 2 receives an image of the working environment from the operation terminal 2, and accepts an operator's operation to designate a target object from the image. , to generate an external input signal Se specifying the region of the object.
- the robot control unit 17 recognizes the object based on the external input signal Se, and resumes the robot control based on the interrupted action sequence.
- the robot 5 may have a function corresponding to the robot control unit 17. In this case, the robot 5 performs an action based on the action sequence Sv generated by the action sequence generation section 16, the switching command Sw generated by the switching determination section 18, and the external input signal Se.
- the switching determination unit 18 determines whether it is necessary to switch the control mode to external input control based on the operation sequence Sv and the like. For example, the switching determination unit 18 determines that it is necessary to switch the control mode to the external input control when it is time to execute the external input type subtask incorporated in the operation sequence Sv. In another example, when the generated motion sequence Sv is not executed as planned, the switching determination unit 18 determines that some kind of abnormality has occurred and that control of the robot 5 needs to be switched to external input control. judge. In this case, for example, the switching determination unit 18 determines that some kind of abnormality has occurred when it detects that there is a temporal or/or spatial deviation from the plan based on the operation sequence Sv.
- the switching determination unit 18 detects the occurrence of an abnormality by receiving an error signal or the like from the robot 5 or by analyzing a sensor signal (such as an image of the working space) output by the sensor 7. good too. Then, when the switching determination unit 18 determines that it is necessary to switch the control mode to the external input control, the switching determination unit 18 issues a switching command Sw instructing switching of the control mode to the external input control to the output control unit 15 and the robot control unit 17 . supply to
- the information presentation unit 25 displays operation screen information and the like supplied from the task execution system 50 when communication connection is established between the task execution system 50 requesting support and the operation terminal 2 based on connection control by the robot management device 3. , the operation screen is displayed on the display unit 24b.
- the operation screen displays, for example, information about the action content of the robot 5 to be specified by external input.
- the information presenting unit 25 presents the information required for the operation to the operator.
- the information presentation unit 25 may output voice guidance necessary for the operation by controlling the sound output unit 24c.
- the external control unit 26 acquires the signal output by the input unit 24a in response to the operation by the operator who refers to the operation screen as the external input signal Se, and transmits the acquired external input signal Se to the task execution system 50 of the support request source. , via interface 23 .
- the external control unit 27 acquires the external input signal Se and transmits the external input signal Se in real time according to the operator's operation, for example.
- the external input necessity determination unit 35 determines whether support by external input is necessary. In the present embodiment, the external input necessity determining unit 35 determines that external input support is necessary when the support request information Sr is received from the task execution system 50 via the interface 33 . Then, the external input necessity determination unit 35 supplies the support request information Sr to the operating terminal determination unit 36 .
- the operating terminal determination unit 36 determines the operating terminal 2 and the operator that support the task execution system 50 that is the source of the support request information Sr. to decide. A specific example of this determination method will be described later.
- the connection control unit 37 performs connection control to establish a communication connection between the target operation terminal 2 determined by the operation terminal determination unit 36 and the task execution system 50 that requested assistance.
- the connection control unit 37 controls at least one of the target operating terminal 2 and the task execution system 50 so that the target operating terminal 2 and the task execution system 50 can directly establish a communication connection. Send communication address.
- the connection control unit 37 establishes a communication connection between the target operation terminal 2 and the task execution system 50 necessary for transferring data exchanged between the target operation terminal 2 and the task execution system 50 during external input control. Establish. In this case, during the external input control, the connection control unit 37 receives the external input signal Se generated by the operation terminal 2, etc., and sends the received external input signal Se, etc. to the task execution system 50 (specifically, the robot controller 1). A transfer process, and a process of receiving operation screen information and the like generated by the task execution system 50 and transferring the received operation screen information and the like to the operation terminal 2 are performed.
- each component of the external input necessity determination unit 35, the operating terminal determination unit 36, and the connection control unit 37 can be realized by the processor 31 executing a program, for example. Further, each component may be realized by recording necessary programs in an arbitrary nonvolatile storage medium and installing them as necessary. Note that at least part of each of these components may be realized by any combination of hardware, firmware, and software, without being limited to being implemented by program software. Also, at least part of each of these components may be implemented using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, this integrated circuit may be used to implement a program composed of the above components.
- FPGA Field-Programmable Gate Array
- each component may be composed of an ASSP (Application Specific Standard Produce), an ASIC (Application Specific Integrated Circuit), or a quantum computer control chip.
- ASSP Application Specific Standard Produce
- ASIC Application Specific Integrated Circuit
- quantum computer control chip a quantum computer control chip.
- each component may be realized by various hardware. The above also applies to other embodiments described later.
- each of these components may be implemented by cooperation of a plurality of computers using, for example, cloud computing technology. The same applies to the components of the robot controller 1 and the operation terminal 2 shown in FIG.
- the operating terminal determination unit 36 refers to the information about the task included in the support request information Sr and at least the terminal type information 382 of the operating terminal information 38 to determine the target operating terminal 2 . A specific example of this will be described below.
- the operating terminal determination unit 36 determines the operating terminal 2 that performs the assisting operation based on the type of the robot 5 of the task execution system 50 that issued the request and the terminal type information 382 of the operating terminal information 38. .
- the user interface that is easy to operate differs depending on the type of robot. Therefore, the operating terminal determination unit 36 according to the first example selects the operating terminal 2 having a user interface suitable for supporting operation of the robot 5 of the task execution system 50 that made the request. For example, when the target robot 5 is a robot arm, the operating terminal determination unit 36 selects the operating terminal 2 having a game controller as a user interface. In another example, when the target robot 5 is a humanoid robot, the operating terminal determination unit 36 selects the operating terminal 2 that can be operated by VR.
- the memory 32 stores information (also referred to as “robot/operating terminal correspondence information”) in which the type of operating terminal 2 suitable for assisting operation is associated with each type of robot 5 .
- the operating terminal determination unit 36 refers to the robot/operating terminal correspondence information, and recognizes the type of the operating terminal 2 suitable for the assisting operation from the type of the robot 5 indicated by the assistance request information Sr.
- the operating terminal determination unit 36 identifies the operating terminal 2 corresponding to the type based on the terminal type information 382 included in the operating terminal information 38 and determines the identified operating terminal 2 as the target operating terminal 2 .
- the operation terminal determination unit 36 selects the target operation terminal based on at least one of second to fourth examples described later. 2 may be determined, or the target operation terminal 2 may be determined by random sampling.
- the operating terminal determination unit 36 determines the error information included in the support request information Sr and the terminal type information in the operating terminal information 38. 382, the target operation terminal 2 is determined. As a result, the operating terminal determination unit 36 suitably determines the operating terminal 2 that can easily handle the error that has occurred as the target operating terminal 2 . For example, if the error information indicates that the pick-and-place operation failed, the operating terminal determination unit 36 selects the operating terminal 2 having a game controller as a user interface. In another example, the operating terminal determination unit 36 selects the operating terminal 2, which is a personal computer, when the error information indicates that acquisition of product information has failed.
- the memory 32 stores information (also referred to as “error/operating terminal correspondence information”) in which the type of operating terminal 2 suitable for assisting operation is associated with each type of error that may occur. Then, the operating terminal determining unit 36 refers to the error/operating terminal correspondence information, and recognizes the type of the operating terminal 2 suitable for the assist operation from the error type indicated by the assist request information Sr. Then, the operating terminal determination unit 36 identifies the operating terminal 2 corresponding to the type based on the terminal type information 382 included in the operating terminal information 38 and determines the identified operating terminal 2 as the target operating terminal 2 .
- information also referred to as “error/operating terminal correspondence information”
- the operating terminal determining unit 36 refers to the error/operating terminal correspondence information, and recognizes the type of the operating terminal 2 suitable for the assist operation from the error type indicated by the assist request information Sr. Then, the operating terminal determination unit 36 identifies the operating terminal 2 corresponding to the type based on the terminal type information 382 included in the operating terminal information 38 and determines the
- the operation terminal determination unit 36 selects at least one of the first example and third to fourth examples to be described later. Based on this, the target operation terminal 2 may be determined, or the target operation terminal 2 may be determined by random sampling.
- the operating terminal determination unit 36 further uses information included in the operating terminal information 38 or the operator information 39 other than the terminal type information 382 to determine the target operating terminal 2. you can go This specific example will be described as a third example and a fourth example. The following third and fourth examples are performed, for example, in combination with the first or second example described above.
- the operating terminal determination unit 36 determines the operator based on the type of error indicated by the support request information Sr.
- the memory 32 stores information (also referred to as "error/operator correspondence information") that defines conditions of the operator's track record and/or skill required for assisting operations for each type of error. It is Then, the operating terminal determining unit 36 refers to the error/operator correspondence information, and recognizes the operator's track record and/or skill conditions required for the support operation from the type of error indicated by the support request information Sr.
- the operating terminal determining unit 36 refers to the skill information 392 and/or the operation result information 393 included in the operator information 39 to specify an operator who satisfies the conditions of the performance and/or skill, and identifies the specified operation.
- the operating terminal 2 used by the user is determined as the target operating terminal 2 .
- the operating terminal determination unit 36 determines, as the target operating terminal 2, the operating terminal 2 used by the operator who is in a state where the support operation can be performed, based on the state management information 394.
- the operating terminal determining unit 36 refers to the state management information 394 to identify an operator who can respond at the present time, and determines the operating terminal 2 used by the identified operator as the target operating terminal 2 .
- the operating terminal determination unit 36 selects an operator who can handle existing operator) can be appropriately selected, and the operation terminal 2 used by the operator can be determined as the target operation terminal 2 .
- the operating terminal determination unit 36 transmits information to the task execution system 50 that requested assistance, prompting robot control without external input control such as autonomous recovery.
- the operating terminal determination unit 36 accumulates support request information Sr for which support has not yet been executed, and puts the task execution system 50 that has requested support into a standby state until the support becomes executable.
- the operating terminal determination unit 36 may process the support request information Sr in order according to the FIFO (First In, First Out) method, determine the priority of each accumulated support request information Sr, and determine the priority of each stored support request information Sr.
- the support request information Sr may be processed in order.
- the operating terminal determination unit 36 determines, for example, the priority of the task based on the information about the type of task or the priority of the task included in the support request information Sr, or/and the task execution system 50 of the support request source.
- the priority of each piece of support request information Sr is determined based on the degree of urgency of support identified from the work progress of .
- the operation sequence generation unit 16 functionally includes an abstract state setting unit 161, a target logical expression generation unit 162, a time step logical expression generation unit 163, an abstract model generation unit 164, a control input generation unit 165, and a subtask sequence generator 166 .
- the abstract state setting unit 161 sets the abstract state in the work space based on the sensor signal supplied from the sensor 7, the abstract state designation information I1, and the object model information I6. In this case, the abstract state setting unit 161 recognizes an object in the workspace that needs to be considered when executing the task, and generates a recognition result “Im” regarding the object. Then, based on the recognition result Im, the abstract state setting unit 161 defines a proposition to be represented by a logical formula for each abstract state that needs to be considered when executing the task. The abstract state setting unit 161 supplies information indicating the set abstract state (also referred to as “abstract state setting information IS”) to the target logical expression generating unit 162 .
- information indicating the set abstract state also referred to as “abstract state setting information IS”
- the target logical formula generation unit 162 converts the task into a temporal logic logical formula (also referred to as "target logical formula Ltag") representing the final achievement state based on the abstract state setting information IS.
- the target logical expression generation unit 162 refers to the constraint condition information I2 from the application information storage unit 41 to add the constraint conditions to be satisfied in executing the task to the target logical expression Ltag.
- the target logical expression generation unit 162 then supplies the generated target logical expression Ltag to the time step logical expression generation unit 163 .
- the time step logical expression generation unit 163 converts the target logical expression Ltag supplied from the target logical expression generation unit 162 into a logical expression representing the state at each time step (also called “time step logical expression Lts”). do.
- the time step logical expression generation unit 163 then supplies the generated time step logical expression Lts to the control input generation unit 165 .
- the abstract model generation unit 164 generates an abstract model " ⁇ ”.
- the abstract model ⁇ may be, for example, a hybrid system in which continuous dynamics and discrete dynamics are mixed in the target dynamics.
- the abstract model generator 164 supplies the generated abstract model ⁇ to the control input generator 165 .
- the control input generation unit 165 satisfies the time step logical expression Lts supplied from the time step logical expression generation unit 163 and the abstract model ⁇ supplied from the abstract model generation unit 164, and generates an evaluation function (for example, The control input to the robot 5 is determined for each time step to optimize the function representing the amount of energy to be applied).
- the control input generator 165 then supplies the subtask sequence generator 166 with information indicating the control input to the robot 5 at each time step (also referred to as “control input information Icn”).
- the subtask sequence generation unit 166 generates an operation sequence Sv, which is a sequence of subtasks, based on the control input information Icn supplied from the control input generation unit 165 and the subtask information I4 stored in the application information storage unit 41.
- the sequence Sv is supplied to the robot control section 17 and the switching determination section 18 .
- FIG. 7 is a first display example of the operation screen displayed by the operation terminal 2.
- the information presentation unit 25 of the operation terminal 2 receives the operation screen information from the robot controller 1 of the task execution system 50 that sent the support request information Sr, and controls to display the operation screen shown in FIG. .
- the robot controller 1 receives the external input signal Se necessary for executing the external input type subtask because the execution timing of the external input type subtask (that is, the work process requiring the external input) has come.
- the robot management device 3 establishes a communication connection with the operation terminal 2 based on connection control by the robot management device 3 .
- the operation screen shown in FIG. 7 mainly has a workspace display field 70 and an operation content display area 73 .
- the workspace display field 70 an image of the current workspace or a workspace image that is a CAD image that schematically represents the current workspace is displayed. Contents required to operate the robot 5 by external input are displayed.
- the target subtask is a subtask of moving an object adjacent to an obstacle, which the robot 5 cannot directly grip, to a grippable position and gripping it.
- the operation terminal 2 displays a guide sentence instructing the action content to be executed by the robot 5 (here, moving the object to a predetermined position and grasping it with the first arm) on the action content display area 73 .
- the operation terminal 2 includes a thick round frame 71 surrounding an object to be worked on, a broken-line round frame 72 indicating the destination of the object, and a robot. 5, name of each arm (first arm, second arm).
- the operation terminal 2 becomes a robot arm necessary for work and a work target for the operator who refers to the text sentence in the operation content display area 73. It is possible to favorably recognize the object and its destination.
- the action content of the robot 5 shown in the action content display area 73 satisfies the conditions (also called “sequence transition conditions") for transitioning to the next subtask of the target subtask.
- the sequence transition condition corresponds to a condition indicating the end state of the target subtask (or the start state of the next subtask) assumed in the generated operation sequence Sv.
- the sequence transition condition in the example of FIG. 7 indicates that the first arm is gripping the object at a predetermined position. In this way, the operation terminal 2 displays the guide text instructing the action required to satisfy the sequence transition condition in the action content display area 73, thereby allowing the external input necessary for smooth transition to the next subtask. can be suitably supported.
- FIG. 8 shows a second display example of the operation screen.
- the information presentation unit 25 of the operation terminal 2 receives the operation screen information from the robot controller 1 of the task execution system 50 that sent the support request information Sr, and controls to display the operation screen shown in FIG. .
- the operation screen shown in FIG. 8 mainly has a workspace display field 70 and an operation content display area 73 .
- the robot controller 1 detects that there is a temporal and/or spatial deviation from the plan based on the motion sequence Sv, and thus determines that continuation of autonomous robot control is inappropriate. Then, the support request information Sr is transmitted to the robot management device 3, and then the operation screen information is transmitted to the operation terminal 2 with which the communication connection has been established.
- the information presenting unit 25 notifies that an abnormality has occurred in the pick-and-place of the object and that an external input is required to move the object to the goal point. It is displayed on the display area 73 .
- the output control unit 15 displays on the image displayed in the work space display field 70 a thick round frame 71 that surrounds the object to be worked on, and the names of the arms of the robot 5 (first arm, second arm). ) and are displayed.
- the operation terminal 2 may output a guidance voice instructing an operation for generating the necessary external input signal Se together with the display of the operation screens of FIGS. 7 and 8 .
- FIG. 9 is an example of a flow chart showing an outline of processing executed by the robot management device 3 in the first embodiment.
- the external input necessity determination unit 35 of the robot management device 3 determines whether or not the support request information Sr has been received (step S11).
- the process proceeds to step S12.
- the operating terminal determination unit 36 of the robot management device 3 determines the target operating terminal 2 based on the support request information Sr, the operating terminal information 38, and the like (step S12).
- the operating terminal determination unit 36 may further refer to the operator information 39 in addition to the operating terminal information 38 to perform processing for determining an operator suitable for the support operation.
- connection control unit 37 of the robot management device 3 performs connection control to establish a communication connection between the target operation terminal 2 and the requesting task execution system 50 (step S13).
- the connection control unit 37 establishes a communication connection between the determined target operation terminal 2 and the task execution system 50 that is the source of the request.
- the determined target operation terminal 2 and the requesting task execution system 50 exchange operation screen information, an external input signal Se, and the like.
- step S14 determines whether or not the processing of the flowchart should be terminated. For example, the robot management device 3 determines that the processing of the flowchart should be terminated when the operation time of the robot control system 100 is exceeded or when other predetermined termination conditions are satisfied. If the robot management device 3 should end the processing of the flowchart (step S14; Yes), the robot management device 3 ends the processing of the flowchart. On the other hand, if the robot management device 3 should not end the process of the flowchart (step S14; No), the process returns to step S11.
- information on candidate motion sequences to be commanded to the robot 5 may be stored in the storage device 4 in advance, and the motion sequence generator 16 may execute the optimization process of the control input generator 165 based on the information. . Thereby, the motion sequence generator 16 selects the optimum candidate and determines the control input for the robot 5 .
- the operation sequence generation unit 16 does not have to have functions corresponding to the abstract state setting unit 161, the target logical expression generation unit 162, and the time step logical expression generation unit 163 in generating the operation sequence Sv.
- the application information storage unit 41 may store in advance information about the execution results of some of the functional blocks of the operation sequence generation unit 16 shown in FIG.
- the application information includes in advance design information such as a flow chart for designing the operation sequence Sv corresponding to the task, and the operation sequence generator 16 refers to the design information to A motion sequence Sv may be generated.
- design information such as a flow chart for designing the operation sequence Sv corresponding to the task
- the operation sequence generator 16 refers to the design information to A motion sequence Sv may be generated.
- FIG. 10 shows a schematic configuration diagram of the robot management device 3X in the second embodiment.
- the robot management device 3X mainly has an external input necessity determination means 35X and an operation terminal determination means 36X.
- the robot management device 3X may be composed of a plurality of devices.
- the robot management device 3X can be, for example, the robot management device 3 of the first embodiment (including the case where some functions of the robot controller 1 are incorporated).
- the external input necessity determination means 35X determines the necessity of control (external input control) based on the external input of the robot executing the task.
- the external input necessity determination unit 35X can be, for example, the external input necessity determination unit 35 in the first embodiment.
- the operating terminal determining means 36X selects an external input based on operating terminal information including information on the types of a plurality of operating terminals that are candidates for generating an external input and information on a task. Determine the operation terminal to be generated.
- "Task-related information” includes various types of information included in the support request information Sr in the first embodiment, such as information on the type of task, information on the robot that executes the task, and information on errors that have occurred in the task.
- the operating terminal determining means 36X can be, for example, the operating terminal determining section 36 in the first embodiment.
- FIG. 11 is an example of a flowchart in the second embodiment.
- the external input necessity determination means 35X determines necessity of control based on the external input of the robot executing the task (step S21). If the control based on the external input is required (step S22; Yes), the operating terminal determining means 36X determines the operating terminal information including information on the types of a plurality of operating terminals that are candidates for generating the external input, and information on the task. , the operating terminal that generates the external input is determined (step S23). On the other hand, if the control based on the external input is not required (step S22; No), the operating terminal determining means 36X ends the processing of the flowchart without performing the processing of step S23.
- the robot management device 3X can suitably determine the operation terminal that generates the external input when there is a robot that requires control based on the external input.
- Non-transitory computer readable media include various types of tangible storage media (Tangible Storage Medium).
- non-transitory computer-readable media examples include magnetic storage media (e.g., floppy disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical discs), CD-ROMs (Read Only Memory), CD-Rs, CD-R/W, semiconductor memory (eg mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)).
- the program may also be delivered to the computer on various types of transitory computer readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable media can deliver the program to the computer via wired channels, such as wires and optical fibers, or wireless channels.
- an operating terminal determining means for determining an operating terminal A robot management device having [Appendix 2] The robot management apparatus according to appendix 1, further comprising connection control means for establishing a communication connection between the operating terminal determined by the operating terminal determining means and the robot or a robot controller controlling the robot.
- the information about the task includes error information about an error that occurred in the task; 3.
- the robot management apparatus according to appendix 1 or 2, wherein the operating terminal determination means determines the operating terminal that generates the external input based on the operating terminal information and the error information.
- the information about the task includes the type information of the robot; 4.
- the robot management apparatus according to any one of appendices 1 to 3, wherein the operating terminal determination means determines the operating terminal that generates the external input based on the operating terminal information and the robot type information. .
- the operating terminal determining means determines the operating terminal that generates the external input based on operator information that is information about an operator of the operating terminal, the operating terminal information, and information about the task. 5.
- the robot management device according to any one of 1 to 4.
- the operator information includes information about the operator's skill or operation record, 6.
- the operator information includes state management information that is information related to state management of the operator, The operating terminal determination means determines, based on the state management information, the operating terminal used by the operator that is in a state capable of executing an operation related to the external input as the operating terminal that generates the external input. 7.
- the external input necessity determination means determines that control based on the external input is necessary when support request information including information about the task is received from the robot or a robot controller controlling the robot. 8. The robot management device according to any one of -7.
- the external input necessity determination means determines that control based on the external input is necessary when an error occurs in the execution of the task by the robot or when the work process requires the external input. 9. The robot management device according to any one of Appendices 1 to 8.
- a storage medium storing a program that causes a computer to execute processing for determining an operating terminal.
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Human Resources & Organizations (AREA)
- Robotics (AREA)
- Mechanical Engineering (AREA)
- Strategic Management (AREA)
- Development Economics (AREA)
- Economics (AREA)
- Entrepreneurship & Innovation (AREA)
- Educational Administration (AREA)
- Operations Research (AREA)
- Marketing (AREA)
- Game Theory and Decision Science (AREA)
- Quality & Reliability (AREA)
- Tourism & Hospitality (AREA)
- Physics & Mathematics (AREA)
- General Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Manipulator (AREA)
Abstract
Description
タスクを実行するロボットの外部入力に基づく制御の要否を判定する外部入力要否判定手段と、
前記外部入力に基づく制御が必要な場合、前記外部入力を生成する候補となる複数の操作端末の種類に関する情報を含む操作端末情報と、前記タスクに関する情報とに基づき、前記外部入力を生成する前記操作端末を決定する操作端末決定手段と、
を有するロボット管理装置である。
コンピュータが、
タスクを実行するロボットの外部入力に基づく制御の要否を判定し、
前記外部入力に基づく制御が必要な場合、前記外部入力を生成する候補となる複数の操作端末の種類に関する情報を含む操作端末情報と、前記タスクに関する情報とに基づき、前記外部入力を生成する前記操作端末を決定する、
制御方法である。
タスクを実行するロボットの外部入力に基づく制御の要否を判定し、
前記外部入力に基づく制御が必要な場合、前記外部入力を生成する候補となる複数の操作端末の種類に関する情報を含む操作端末情報と、前記タスクに関する情報とに基づき、前記外部入力を生成する前記操作端末を決定する処理を
コンピュータに実行させるプログラムが格納された記憶媒体である。
(1)システム構成
図1は、第1実施形態に係るロボット制御システム100の構成を示す。ロボット制御システム100は、主に、操作端末2(2A、2B、…)と、ロボット管理装置3と、複数のタスク実行システム50(50A、50B、…)とを有する。操作端末2とロボット管理装置3とタスク実行システム50とは、通信網6を介してデータ通信を行う。
図2(A)は、ロボットコントローラ1(1A、1B、…)のハードウェア構成を示す。ロボットコントローラ1は、ハードウェアとして、プロセッサ11と、メモリ12と、インターフェース13とを含む。プロセッサ11、メモリ12及びインターフェース13は、データバス10を介して接続されている。
まず、アプリケーション情報記憶部41が記憶するアプリケーション情報のデータ構造について説明する。
図5は、ロボット制御システム100の処理の概要を示す機能ブロックの一例である。ロボットコントローラ1のプロセッサ11は、機能的には、出力制御部15と、動作シーケンス生成部16と、ロボット制御部17と、切替判定部18とを有する。また、操作端末2のプロセッサ21は、機能的には、情報提示部25と、外部制御部26とを有する。また、ロボット管理装置3のプロセッサ31は、機能的には、外部入力要否判定部35と、操作端末決定部36と、接続制御部37とを有する。なお、図5に示す機能ブロックでは、データの授受が行われるブロック同士を実線により結んでいるが、データの授受が行われるブロックの組合せ及び授受が行われるデータは図5に限定されない。後述する他の機能ブロックの図においても同様である。また、図5では、操作端末2による操作者の操作態様の例を、吹き出し60内において例示している。
次に、操作端末決定部36による対象操作端末2の決定方法について説明する。概略的には、操作端末決定部36は、支援要求情報Srに含まれるタスクに関する情報と、少なくとも操作端末情報38の端末種類情報382とを参照し、対象操作端末2を決定する。以下では、この具体例について説明する。
図6は、動作シーケンス生成部16の機能的な構成を示す機能ブロックの一例である。動作シーケンス生成部16は、機能的には、抽象状態設定部161と、目標論理式生成部162と、タイムステップ論理式生成部163と、抽象モデル生成部164と、制御入力生成部165と、サブタスクシーケンス生成部166と、を有する。
図7は、操作端末2が表示する操作画面の第1表示例である。操作端末2の情報提示部25は、支援要求情報Srの送信元のタスク実行システム50のロボットコントローラ1から操作画面情報を受信することで、図7に示す操作画面を表示するよう制御している。ここでは、ロボットコントローラ1は、外部入力型サブタスクの実行タイミング(即ち、外部入力が必要な作業工程)となったことから、当該外部入力型サブタスクの実行に必要な外部入力信号Seを受信するために支援要求情報Srをロボット管理装置3に送信し、その後にロボット管理装置3による接続制御に基づき、操作端末2と通信接続を確立している。図7に示す操作画面は、主に、作業空間表示欄70と、動作内容表示領域73とを有している。
図9は、第1実施形態においてロボット管理装置3が実行する処理の概要を示すフローチャートの一例である。
図6に示す動作シーケンス生成部16のブロック構成は一例であり、種々の変更がなされてもよい。
図10は、第2実施形態におけるロボット管理装置3Xの概略構成図を示す。ロボット管理装置3Xは、主に、外部入力要否判定手段35Xと、操作端末決定手段36Xとを有する。なお、ロボット管理装置3Xは、複数の装置から構成されてもよい。ロボット管理装置3Xは、例えば、第1実施形態のロボット管理装置3(ロボットコントローラ1の一部の機能が組み込まれている場合も含む)とすることができる。
タスクを実行するロボットの外部入力に基づく制御の要否を判定する外部入力要否判定手段と、
前記外部入力に基づく制御が必要な場合、前記外部入力を生成する候補となる複数の操作端末の種類に関する情報を含む操作端末情報と、前記タスクに関する情報とに基づき、前記外部入力を生成する前記操作端末を決定する操作端末決定手段と、
を有するロボット管理装置。
[付記2]
前記操作端末決定手段が決定した前記操作端末と、前記ロボット又は前記ロボットを制御するロボットコントローラとの通信接続を確立する制御を行う接続制御手段をさらに有する、付記1に記載のロボット管理装置。
[付記3]
前記タスクに関する情報は、前記タスクにおいて生じたエラーに関するエラー情報を含み、
前記操作端末決定手段は、前記操作端末情報と、前記エラー情報とに基づき、前記外部入力を生成する前記操作端末を決定する、付記1または2に記載のロボット管理装置。
[付記4]
前記タスクに関する情報は、前記ロボットの種類情報を含み、
前記操作端末決定手段は、前記操作端末情報と、前記ロボットの種類情報とに基づき、前記外部入力を生成する前記操作端末を決定する、付記1~3のいずれか一項に記載のロボット管理装置。
[付記5]
前記操作端末決定手段は、前記操作端末の操作者に関する情報である操作者情報と、前記操作端末情報と、前記タスクに関する情報とに基づき、前記外部入力を生成する前記操作端末を決定する、付記1~4のいずれか一項に記載のロボット管理装置。
[付記6]
前記操作者情報は、前記操作者のスキル又は操作実績に関する情報を含み、
前記操作端末決定手段は、前記タスクに関する情報に基づき定まる必要なスキル又は操作実績を満たす操作者が使用する前記操作端末を、前記外部入力を生成する前記操作端末として決定する、付記5に記載のロボット管理装置。
[付記7]
前記操作者情報は、前記操作者の状態管理に関する情報である状態管理情報を含み、
前記操作端末決定手段は、前記状態管理情報に基づき、前記外部入力に関する操作を実行可能な状態である操作者が使用する前記操作端末を、前記外部入力を生成する前記操作端末として決定する、付記5または6に記載のロボット管理装置。
[付記8]
前記外部入力要否判定手段は、前記タスクに関する情報を含む支援要求情報を、前記ロボット又は前記ロボットを制御するロボットコントローラから受信した場合に、前記外部入力に基づく制御が必要と判定する、付記1~7のいずれか一項に記載のロボット管理装置。
[付記9]
前記外部入力要否判定手段は、前記ロボットによる前記タスクの実行においてエラーが発生した場合、又は、前記外部入力が必要な作業工程となった場合、前記外部入力に基づく制御が必要であると判定する、付記1~8のいずれか一項に記載のロボット管理装置。
[付記10]
コンピュータが、
タスクを実行するロボットの外部入力に基づく制御の要否を判定し、
前記外部入力に基づく制御が必要な場合、前記外部入力を生成する候補となる複数の操作端末の種類に関する情報を含む操作端末情報と、前記タスクに関する情報とに基づき、前記外部入力を生成する前記操作端末を決定する、
制御方法。
[付記11]
タスクを実行するロボットの外部入力に基づく制御の要否を判定し、
前記外部入力に基づく制御が必要な場合、前記外部入力を生成する候補となる複数の操作端末の種類に関する情報を含む操作端末情報と、前記タスクに関する情報とに基づき、前記外部入力を生成する前記操作端末を決定する処理を
コンピュータに実行させるプログラムが格納された記憶媒体。
2、2A、2B 操作端末
3、3X ロボット管理装置
5 ロボット
7 センサ
41 アプリケーション情報記憶部
100 ロボット制御システム
Claims (11)
- タスクを実行するロボットの外部入力に基づく制御の要否を判定する外部入力要否判定手段と、
前記外部入力に基づく制御が必要な場合、前記外部入力を生成する候補となる複数の操作端末の種類に関する情報を含む操作端末情報と、前記タスクに関する情報とに基づき、前記外部入力を生成する前記操作端末を決定する操作端末決定手段と、
を有するロボット管理装置。 - 前記操作端末決定手段が決定した前記操作端末と、前記ロボット又は前記ロボットを制御するロボットコントローラとの通信接続を確立する制御を行う接続制御手段をさらに有する、請求項1に記載のロボット管理装置。
- 前記タスクに関する情報は、前記タスクにおいて生じたエラーに関するエラー情報を含み、
前記操作端末決定手段は、前記操作端末情報と、前記エラー情報とに基づき、前記外部入力を生成する前記操作端末を決定する、請求項1または2に記載のロボット管理装置。 - 前記タスクに関する情報は、前記ロボットの種類情報を含み、
前記操作端末決定手段は、前記操作端末情報と、前記ロボットの種類情報とに基づき、前記外部入力を生成する前記操作端末を決定する、請求項1~3のいずれか一項に記載のロボット管理装置。 - 前記操作端末決定手段は、前記操作端末の操作者に関する情報である操作者情報と、前記操作端末情報と、前記タスクに関する情報とに基づき、前記外部入力を生成する前記操作端末を決定する、請求項1~4のいずれか一項に記載のロボット管理装置。
- 前記操作者情報は、前記操作者のスキル又は操作実績に関する情報を含み、
前記操作端末決定手段は、前記タスクに関する情報に基づき定まる必要なスキル又は操作実績を満たす操作者が使用する前記操作端末を、前記外部入力を生成する前記操作端末として決定する、請求項5に記載のロボット管理装置。 - 前記操作者情報は、前記操作者の状態管理に関する情報である状態管理情報を含み、
前記操作端末決定手段は、前記状態管理情報に基づき、前記外部入力に関する操作を実行可能な状態である操作者が使用する前記操作端末を、前記外部入力を生成する前記操作端末として決定する、請求項5または6に記載のロボット管理装置。 - 前記外部入力要否判定手段は、前記タスクに関する情報を含む支援要求情報を、前記ロボット又は前記ロボットを制御するロボットコントローラから受信した場合に、前記外部入力に基づく制御が必要と判定する、請求項1~7のいずれか一項に記載のロボット管理装置。
- 前記外部入力要否判定手段は、前記ロボットによる前記タスクの実行においてエラーが発生した場合、又は、前記外部入力が必要な作業工程となった場合、前記外部入力に基づく制御が必要であると判定する、請求項1~8のいずれか一項に記載のロボット管理装置。
- コンピュータが、
タスクを実行するロボットの外部入力に基づく制御の要否を判定し、
前記外部入力に基づく制御が必要な場合、前記外部入力を生成する候補となる複数の操作端末の種類に関する情報を含む操作端末情報と、前記タスクに関する情報とに基づき、前記外部入力を生成する前記操作端末を決定する、
制御方法。 - タスクを実行するロボットの外部入力に基づく制御の要否を判定し、
前記外部入力に基づく制御が必要な場合、前記外部入力を生成する候補となる複数の操作端末の種類に関する情報を含む操作端末情報と、前記タスクに関する情報とに基づき、前記外部入力を生成する前記操作端末を決定する処理を
コンピュータに実行させるプログラムが格納された記憶媒体。
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2023512634A JPWO2022215262A5 (ja) | 2021-04-09 | ロボット管理装置、制御方法及び、プログラム | |
PCT/JP2021/015053 WO2022215262A1 (ja) | 2021-04-09 | 2021-04-09 | ロボット管理装置、制御方法及び記憶媒体 |
US18/285,025 US20240165817A1 (en) | 2021-04-09 | 2021-04-09 | Robot management device, control method, and recording medium |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/JP2021/015053 WO2022215262A1 (ja) | 2021-04-09 | 2021-04-09 | ロボット管理装置、制御方法及び記憶媒体 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2022215262A1 true WO2022215262A1 (ja) | 2022-10-13 |
Family
ID=83545252
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2021/015053 WO2022215262A1 (ja) | 2021-04-09 | 2021-04-09 | ロボット管理装置、制御方法及び記憶媒体 |
Country Status (2)
Country | Link |
---|---|
US (1) | US20240165817A1 (ja) |
WO (1) | WO2022215262A1 (ja) |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2008140011A1 (ja) * | 2007-05-09 | 2008-11-20 | Nec Corporation | 遠隔操作システム、サーバ、被遠隔操作装置、遠隔操作サービス提供方法 |
JP2012230506A (ja) * | 2011-04-25 | 2012-11-22 | Sony Corp | 評価装置及び評価方法、サービス提供システム、並びにコンピューター・プログラム |
JP2016068161A (ja) * | 2014-09-26 | 2016-05-09 | トヨタ自動車株式会社 | ロボット制御方法 |
JP2020502636A (ja) * | 2016-11-28 | 2020-01-23 | ブレーン コーポレーションBrain Corporation | ロボットの遠隔操作および/または監視のためのシステムおよび方法 |
JP2020507164A (ja) * | 2017-02-02 | 2020-03-05 | ブレーン コーポレーションBrain Corporation | ロボット装置を支援するためのシステムおよび方法 |
-
2021
- 2021-04-09 WO PCT/JP2021/015053 patent/WO2022215262A1/ja active Application Filing
- 2021-04-09 US US18/285,025 patent/US20240165817A1/en active Pending
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2008140011A1 (ja) * | 2007-05-09 | 2008-11-20 | Nec Corporation | 遠隔操作システム、サーバ、被遠隔操作装置、遠隔操作サービス提供方法 |
JP2012230506A (ja) * | 2011-04-25 | 2012-11-22 | Sony Corp | 評価装置及び評価方法、サービス提供システム、並びにコンピューター・プログラム |
JP2016068161A (ja) * | 2014-09-26 | 2016-05-09 | トヨタ自動車株式会社 | ロボット制御方法 |
JP2020502636A (ja) * | 2016-11-28 | 2020-01-23 | ブレーン コーポレーションBrain Corporation | ロボットの遠隔操作および/または監視のためのシステムおよび方法 |
JP2020507164A (ja) * | 2017-02-02 | 2020-03-05 | ブレーン コーポレーションBrain Corporation | ロボット装置を支援するためのシステムおよび方法 |
Also Published As
Publication number | Publication date |
---|---|
US20240165817A1 (en) | 2024-05-23 |
JPWO2022215262A1 (ja) | 2022-10-13 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP7198831B2 (ja) | オンデマンド遠隔操作を備えた自律ロボット | |
JP2015520040A (ja) | 産業用ロボットを訓練および動作させること | |
JP6258043B2 (ja) | 産業用ロボットを制御する制御システム、制御装置、溶接物製造方法およびプログラム | |
CN112230649B (zh) | 机器学习方法及移动机器人 | |
JP7264253B2 (ja) | 情報処理装置、制御方法及びプログラム | |
WO2022215262A1 (ja) | ロボット管理装置、制御方法及び記憶媒体 | |
CN112894827B (zh) | 一种机械臂运动控制方法、***、装置及可读存储介质 | |
WO2021085429A1 (ja) | 被遠隔操作装置、遠隔操作システム、遠隔操作装置 | |
JP7452619B2 (ja) | 制御装置、制御方法及びプログラム | |
JP7448024B2 (ja) | 制御装置、制御方法及びプログラム | |
WO2021171357A1 (ja) | 制御装置、制御方法及び記録媒体 | |
JP7491400B2 (ja) | 支援制御装置、支援装置、ロボット制御システム、支援制御方法及びプログラム | |
WO2022074825A1 (ja) | 動作指令生成装置、動作指令生成方法及び記憶媒体 | |
WO2022049756A1 (ja) | 判定装置、判定方法及び記憶媒体 | |
WO2022224447A1 (ja) | 制御装置、制御方法及び記憶媒体 | |
WO2022224449A1 (ja) | 制御装置、制御方法及び記憶媒体 | |
WO2022244060A1 (ja) | 動作計画装置、動作計画方法及び記憶媒体 | |
JP7323045B2 (ja) | 制御装置、制御方法及びプログラム | |
JP7468694B2 (ja) | 情報収集装置、情報収集方法及びプログラム | |
JP2015116631A (ja) | 制御装置、ロボット、制御方法及びロボットシステム | |
JP7409474B2 (ja) | 制御装置、制御方法及びプログラム | |
JP7364032B2 (ja) | 制御装置、制御方法及びプログラム | |
JP7435814B2 (ja) | 時相論理式生成装置、時相論理式生成方法及びプログラム | |
US20240066694A1 (en) | Robot control system, robot control method, and robot control program | |
JP7276466B2 (ja) | 情報処理装置、制御方法及びプログラム |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21936065 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 18285025 Country of ref document: US |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2023512634 Country of ref document: JP |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 21936065 Country of ref document: EP Kind code of ref document: A1 |