WO2022234678A1 - 機械学習装置、分類装置、及び制御装置 - Google Patents
機械学習装置、分類装置、及び制御装置 Download PDFInfo
- Publication number
- WO2022234678A1 WO2022234678A1 PCT/JP2021/017578 JP2021017578W WO2022234678A1 WO 2022234678 A1 WO2022234678 A1 WO 2022234678A1 JP 2021017578 W JP2021017578 W JP 2021017578W WO 2022234678 A1 WO2022234678 A1 WO 2022234678A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- data
- video data
- unit
- feature
- machine learning
- Prior art date
Links
- 238000010801 machine learning Methods 0.000 title claims abstract description 66
- 238000000605 extraction Methods 0.000 claims abstract description 37
- 239000000284 extract Substances 0.000 claims abstract description 19
- 230000009471 action Effects 0.000 claims abstract description 16
- 238000012549 training Methods 0.000 claims abstract description 15
- 238000013075 data extraction Methods 0.000 claims abstract description 9
- 238000002372 labelling Methods 0.000 claims description 42
- 230000005856 abnormality Effects 0.000 claims description 11
- 238000007405 data analysis Methods 0.000 claims description 10
- 238000012937 correction Methods 0.000 claims description 9
- 238000003860 storage Methods 0.000 description 21
- 238000000034 method Methods 0.000 description 14
- 238000012545 processing Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 10
- 230000033001 locomotion Effects 0.000 description 8
- 230000008859 change Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 230000002159 abnormal effect Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 4
- 238000001514 detection method Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000003754 machining Methods 0.000 description 3
- 238000004519 manufacturing process Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 238000001746 injection moulding Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000005242 forging Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000003825 pressing Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/7715—Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0631—Resource planning, allocation, distributing or scheduling for enterprises or organisations
- G06Q10/06311—Scheduling, planning or task assignment for a person or group
- G06Q10/063114—Status monitoring or status determination for a person or group
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20076—Probabilistic image processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
Definitions
- the present invention relates to machine learning devices, classification devices, and control devices.
- One aspect of the machine learning device of the present disclosure is a video data feature quantity extraction unit that extracts a feature quantity indicating the action of the worker from video data including the worker captured by at least one camera; When a feature amount indicating a specific action registered in advance is extracted from the feature amounts extracted by the video data feature amount extraction unit, the work content indicated by the extracted specific action is transferred to the corresponding video data.
- a teacher data extracting unit for labeling and extracting teacher data of the labeled input data of the video data and the label data of the work content; and a feature quantity related to the teacher data extracted from the operation data of the industrial machine.
- a labeling reference creation unit that creates respective standards for labeling; and a labeling reference for labeling the video data and a labeling reference for the operation data.
- Machine learning is performed using a labeling unit for labeling and the teacher data including the video data labeled by the labeling unit, and learning to classify the work performed by the worker from the input video data. a learning unit that generates a trained model.
- One aspect of the classification device of the present disclosure is a trained model generated by the machine learning device of (1), an input unit for inputting video data including workers captured by at least one camera, and a work determination unit that determines the work of the worker from the video data input by the input unit, based on the learned model.
- control device of the present disclosure includes the classification device of (2).
- FIG. 1 is a functional block diagram showing a functional configuration example of a control system according to an embodiment
- FIG. It is a figure which shows an example of the image
- FIG. 10 is a diagram showing an example of a setup time width for product A;
- FIG. 4 is a diagram showing an example of teacher data for each task;
- FIG. 4 is a diagram showing an example of teacher data for each task;
- storage part. 4 is a flowchart for explaining classification processing of the classification device in an operation phase;
- FIG. 1 is a functional block diagram showing a functional configuration example of a control system according to one embodiment. As shown in FIG. 1, the control system 1 has a machine tool 10, a camera 20, a classification device 30, and a machine learning device 40.
- the machine tool 10, the camera 20 that captures images (moving images) at a predetermined frame rate, the classification device 30, and the machine learning device 40 may be directly connected to each other via a connection interface (not shown).
- the machine tool 10, the camera 20, the classification device 30, and the machine learning device 40 may be interconnected via a network (not shown) such as a LAN (Local Area Network) or the Internet.
- the machine tool 10, the camera 20, the classification device 30, and the machine learning device 40 are provided with communication units (not shown) for mutual communication through such connections.
- the control device 110 included in the machine tool 10 may include the classification device 30 and the machine learning device 40, as will be described later.
- the machine tool 10 is a machine tool known to those skilled in the art and includes a controller 110 .
- Machine tool 10 operates based on an operation command from control device 110 .
- the control device 110 is, for example, a numerical control device known to those skilled in the art, generates an operation command based on control information, and transmits the generated operation command to the machine tool 10 . Thereby, the control device 110 controls the operation of the machine tool 10 .
- the control device 110 is a device that causes the machine tool 10 to perform predetermined machining by controlling the machine tool 10 .
- a machining program describing the operation of the machine tool 10 is provided to the control device 110 . Based on a given machining program, the control device 110 creates an operation command including a movement command for each axis and a rotation command for the motor that drives the main axis, etc., and transmits the operation command to the machine tool 10. , controls the motors of the machine tool 10 .
- the control device 110 also transmits to the classification device 30 operation data of the machine tool 10 including operation commands, door opening/closing, motor torque values, and the like. Furthermore, based on a clock signal from a clock (not shown) included in the control device 110, the control device 110 stores time information indicating the time when the operation command, the opening and closing of the door, the torque value of the motor, etc. are measured as operation data. may be added to and output to the classification device 30 . In addition, the control device 110 may check the time of a clock (not shown) included in the camera 20 (to be described later) at predetermined time intervals and synchronize them with each other.
- control device 110 may be a robot control device or the like.
- devices controlled by the control device 110 are not limited to the machine tool 10 and robots, and can be widely applied to industrial machines in general.
- Industrial machinery includes various machines such as, for example, machine tools, industrial robots, service robots, forging machines and injection molding machines.
- a numerical control device is exemplified as the control device 110 .
- the camera 20 is, for example, a digital camera such as a surveillance camera, and is arranged in a factory where the machine tool 10 is arranged.
- the camera 20 outputs frame images captured at a predetermined frame rate to the classification device 30 as video.
- the camera 20 acquires the time when each frame image was captured based on a clock signal of a clock (not shown) included in the camera 20, adds time information indicating the acquired time to the video, output to FIG. 2 is a diagram showing an example of an image captured by the camera 20. As shown in FIG. FIG. 2 shows images of three machine tools 10 captured by the camera 20 . Although one camera 20 is arranged in FIG. 1 , two or more cameras 20 may be arranged to image the machine tool 10 .
- the classification device 30 may acquire an image captured by the camera 20 and added with time information. Further, the classification device 30 may acquire operation data of the machine tool 10 from the control device 110, for example. The classification device 30 inputs the acquired video to a trained model provided from a machine learning device 40, which will be described later, to determine ( classification). Further, the classification device 30 may transmit the acquired operation data of the machine tool 10 and the image data captured by the camera 20 to the machine learning device 40 described later.
- the classification device 30 receives a trained model generated by a machine learning device 40, which will be described later, and inputs video data captured by the camera 20 to the received trained model, so that The work content of the worker may be determined (classified), and the determination result may be displayed on a display unit (not shown) such as a liquid crystal display included in the control device 110 . Then, if there is an error in the determination result, the classification device 30 acquires the correct work content input in the control device 110, and machine-learns the input data of the determined video data and the label data of the acquired correct work content. By outputting to the device 40, the machine learning device 40 may be caused to update the learned model.
- the machine learning device 40 acquires, for example, video data captured by the camera 20, and extracts a feature quantity indicating the worker's motion from the acquired video data, as will be described later. As will be described later, the machine learning device 40 extracts, from among the extracted feature amounts, a feature amount of a hand gesture as a specific action of a pre-registered worker. is attached to the corresponding video data, and the teacher data of the input data of the labeled video data and the label data of the work contents are extracted. In addition, the machine learning device 40 acquires operation data of the machine tool 10, extracts feature amounts related to the above-described teacher data from the acquired operation data, and extracts the extracted teacher data and operation data, as will be described later.
- the machine learning device 40 labels at least unlabeled image data based on the criteria for assigning labels for image data and the criteria for assigning labels for operation data.
- the machine learning device 40 performs supervised learning using teacher data (training data) including newly labeled video data, and constructs a learned model described later. By doing so, the machine learning device 40 can construct a trained model with high judgment accuracy even with a small amount of teacher data, and can provide the constructed trained model to the classification device 30 .
- the machine learning device 40 will be specifically described.
- the machine learning device 40 includes a video data acquisition unit 401, a video data feature amount extraction unit 402, a teacher data extraction unit 403, an operation data acquisition unit 404, an operation data feature amount extraction unit 405, a labeling reference It has a creating unit 406 , a labeling unit 407 , a learning unit 408 and a storage unit 409 .
- the storage unit 409 is, for example, a ROM (Read Only Memory) or a HDD (Hard Disk Drive), and stores system programs and machine learning application programs executed by a processor (not shown) included in the machine learning device 40. . Further, the storage unit 409 stores a learned model generated by the learning unit 408 to be described later, and has a teacher data extraction signature storage unit 4091 .
- the teacher data extraction signature storage unit 4091 stores, for example, work content to be assigned as a label in association with a specific action (hand gesture, etc.) indicating each work content.
- FIG. 3 is a diagram showing an example of a time series of work. As shown in FIG. 3, the work contents include "standby", "preparation for product A", “cleaning", and "tool change".
- FIG. 4A and 4B are diagrams showing examples of hand gestures indicating the work content of tool exchange.
- FIG. 4A shows an example of a hand gesture indicating the beginning of a tool change
- FIG. 4B shows an example of a hand gesture indicating the end of a tool change.
- a worker faces the camera 20 and performs hand gestures (specific actions) that are pre-registered for each type of work, such as tool exchange, at the start and end of each work. to be specified.
- the video data can be labeled with the content of the work indicated by the hand gesture, and can be used as teacher data. By doing so, it is possible to easily collect teaching data for generating a trained model for determining work content from video data.
- the image data acquisition unit 401 acquires image data including the worker imaged by the camera 20 via the classification device 30 .
- the video data feature quantity extraction unit 402 extracts a feature quantity indicating the worker's motion from the acquired video data. Specifically, the video data feature amount extraction unit 402 uses a known method (for example, Kousuke Kanno, Kenta Oku, Kyoji Kawagoe, "Motion detection and classification method from multidimensional time series data", DEIM Forum 2016 G4- 5, or Shohei Uezono, Satoshi Ono, "Feature extraction of multimodal series data using LSTM Autoencoder", Japanese Society for Artificial Intelligence, SIG-KBS-B802-01, 2018). From the added image data, the feature values of time-series data of the coordinates and angles of the joints of the worker's body (fingers, arms, legs, etc.) are extracted.
- a known method for example, Kousuke Kanno, Kenta Oku, Kyoji Kawagoe, "Motion detection and classification method from multidimensional time series data", DEIM Forum 2016 G4- 5, or Shohei Uezono, Satoshi Ono,
- the image data feature amount extraction unit 402 acquires statistical feature amounts (average value, peak value, etc.) extracted from the time-series data of body joint coordinates and angles extracted from the image data.
- the video data feature amount extraction unit 402 determines detailed motions from the coordinates and angles of the joints of the body, and determines the feature amounts (numerical control (NC) operations, work in the machine tool 10, etc., which make up the setup). Whether it is an operation or not, and in the case of a setup, the time of the setup, etc.) are extracted.
- NC numerical control
- the image data feature quantity extraction unit 402 may extract the feature quantity indicating the worker's motion using the operation data of the machine tool 10 acquired by the operation data acquisition unit, which will be described later.
- the video data feature amount extraction unit 402 grasps the opening/closing timing of the door of the machine tool 10 from the operation data, and if there is a portion in the video data that changes at the same timing, determines that the portion is the door of the machine tool 10. You may As a result, the video data feature amount extraction unit 402 can extract the feature amount of "work in the machine tool" when the worker puts the upper half of the body in the location.
- the video data feature quantity extraction unit 402 extracts a feature quantity such as "Working with an automatic tool changer (ATF)" when the worker's upper body is facing the upper side of the machine tool 10, and When the upper half of the body is facing downward of the machine tool 10, a feature amount such as "working at a table” may be extracted.
- the video data feature extracting unit 402 may determine that the region is the NC region. As a result, the video data feature amount extraction unit 402 can extract the feature amount of "NC operation" when the hand is placed on the location.
- the teacher data extracting unit 403 extracts a feature amount indicating a hand gesture, which is a specific action registered in advance, from among the feature amounts extracted by the video data feature amount extracting unit 402, the extracted hand gesture indicates The video data corresponding to the work content is labeled, and a set of the input data of the labeled video data and the label data of the work content is extracted as teacher data.
- the operation data acquisition unit 404 acquires operation data of the machine tool 10 via the classification device 30.
- the operation data feature amount extraction unit 405 extracts feature amounts related to the teacher data extracted from the operation data of the machine tool 10 . Specifically, the operation data feature amount extraction unit 405 calculates, for example, the time width from the start and end times of the task of the teacher data extracted by the teacher data extraction unit 403, or calculates the time duration for each labeled task. Calculate the average width.
- FIG. 5 is a diagram showing an example of the setup time width for product A. As shown in FIG. As shown in FIG. 5, the operation data feature amount extraction unit 405 cuts the time with the time width of the setup for the product A calculated while overlapping, and classifies the operation data of the machine tool 10 for each cut time.
- Signal information of the machine tool 10 related to work (chuck open/close, tool abnormal value calculated from motor torque, etc.), information for determining whether the same machining is performed (machining program, serial number), and machine tool 10
- the operating state of the machine 10 and the transition of the operating state (for example, a change in work content such as a return to normal operation from a tool change alarm) are extracted as feature amounts of the operation data.
- the video data feature quantity extraction unit 402 cuts the video data by the time width of each task calculated while overlapping, and extracts the feature quantity of the video data in that time, as in the case of the operation data feature quantity extraction unit 405 . may be extracted.
- the labeling reference creation unit 406 creates a reference for giving a label for the video data and a reference for giving a label for the operation data, respectively.
- create. 6A and 6B are diagrams showing an example of teacher data for each task.
- FIG. 6A shows the teacher data in the case of setup work (the operating state is stopped), and
- FIG. 6B shows the teacher data in the case of tool replacement (the operating state is alarming).
- the chuck open/close signal in FIG. 6A changes from "closed” to "open” at the start of work inside the machine tool, and changes from “open” to "closed” at the end of work inside the machine tool.
- the chuck open/close signal of FIG. 6B remains "closed” during work in the machine tool.
- the operating status includes operating, (network) disconnection, emergency stop, temporary stop, manual operation, warm-up operation, and the like.
- the labeling reference creation unit 406 calculates, for example, the average of the feature amounts extracted by the video data feature amount extraction unit 402 .
- the labeling reference creation unit 406 calculates the distance, such as the Mahalanobis distance, between the calculated average of the feature amount and the feature amount of the unlabeled video data for each operation of the training data, as a reference for giving labels for the video data.
- the labeling reference creation unit 406 calculates the average of the feature amounts extracted by the operation data feature amount extraction unit 405 for each operation of the teacher data shown in FIGS. 6A and 6B.
- the label assignment reference creation unit 406 calculates the distance, such as the Mahalanobis distance, between the calculated average of the feature amount and the feature amount of the unlabeled operation data for each operation of the training data, as a reference for assigning labels for the operation data. Calculate as Although the labeling reference creation unit 406 has calculated the distance, it is not limited to this. It is also possible to calculate the probability of being classified into which work.
- the labeling unit 407 may also determine which label to give to unlabeled video data and operation data. By doing so, the labeling unit 407 enables labeling of complicated work with a small amount of teacher data, and even if there are some differences in the video data, the feature values of the operation data are almost the same. can be labeled.
- the labeling unit 407 performs weighting based on the probability (reference) of the video data and the operation data and (Formula 2). If the calculated probability is smaller than a certain probability, the unlabeled video data and operation data may be labeled.
- (Formula 2) Probability using two criteria Probability calculated from video data x 0.8 + Probability calculated from operation data x 0.2
- the criteria for labeling may be sequentially updated using the labeled video data and operation data.
- the labeling reference creation unit 406 recalculates the reference from the teacher data and the labeled data based on a known method (for example, co-training of the bootstrap method), and the labeling unit 407 calculates the reference Labeling may be performed based on
- the learning unit 408 performs machine learning (for example, gradient boosting, neural network, etc.) using the teacher data including the video data labeled by the labeling unit 407, and the input video data is used by the worker. Build a trained model 361 that classifies the work that is being done. The learning unit 408 then provides the constructed trained model 361 to the classification device 30 . Note that, after providing the trained model 361 to the classification device 30, the learning unit 408 acquires from the classification device 30 teacher data of a set of new video data input data and work content label data. Machine learning may be performed again using teacher data to update the learned model 361 .
- machine learning for example, gradient boosting, neural network, etc.
- the learning unit 408 may perform online learning, batch learning, or mini-batch learning.
- Online learning is a learning method in which supervised learning is performed immediately each time teacher data is acquired from the classification device 30 .
- batch learning means that a plurality of pieces of teacher data are collected according to the repetition while acquiring teacher data from the classification device 30 is repeated, and supervised learning is performed using all the collected teacher data. It is a learning method of doing.
- mini-batch learning is an intermediate learning method between online learning and batch learning, in which supervised learning is performed each time a certain amount of teacher data is accumulated. Machine learning for generating the trained model 361 included in the classification device 30 has been described above. Next, the classification device 30 in the operation phase will be described.
- the classification device 30 in the operation phase includes an input unit 301, a work determination unit 302, a determination result writing unit 303, a determination result correction reading unit 304, a determination data analysis unit 305, and a storage unit 306.
- the classification device 30 includes an arithmetic processing device (not shown) such as a CPU (Central Processing Unit) in order to realize the operation of the functional blocks shown in FIG.
- the classification device 30 also includes an auxiliary storage device (not shown) such as a ROM or HDD storing various control programs, and an auxiliary storage device (not shown) for storing data temporarily required when the arithmetic processing unit executes the program.
- a main memory such as a RAM (not shown) is provided.
- the arithmetic processing unit reads the OS and application software from the auxiliary storage device, develops the read OS and application software in the main storage device, and performs arithmetic processing based on the OS and application software. . Based on this calculation result, the classification device 30 controls each piece of hardware. Thereby, the processing by the functional blocks in FIG. 1 is realized. That is, the classification device 30 can be realized by cooperation of hardware and software.
- the storage unit 306 is a ROM, HDD, or the like, and may have a learned model 361 and a determination result storage unit 362 along with various control programs.
- the determination result storage unit 362 associates and stores the determination result for the video data determined by the work determination unit 302, which will be described later, and the feature amount of the determined video data.
- the input unit 301 inputs video data including the worker imaged by the camera 20 .
- the work determination unit 302 determines the worker's work from the video data input by the input unit 301.
- the judgment result writing unit 303 displays the judgment result of the work judgment unit 302 on the display unit (not shown) of the control device 110 . By doing so, the worker can judge whether the classification by the trained model 361 is good or bad.
- the determination result correction reading unit 304 acquires the correct work content input by the worker via an input unit (not shown) such as a keyboard or touch panel included in the control device 110, and performs determination.
- the input data of the obtained video data and the acquired label data of the correct work content are output to the machine learning device 40, and the machine learning device 40 is caused to update the learned model 361.
- FIG. It should be noted that the reading of results and the writing of corrections may use, for example, the storage unit 306 or the PMC area, which will be described later.
- the determination data analysis unit 305 detects whether there is an abnormality in the feature amount based on the determination result stored in the determination result storage unit 362 and the feature amount in the determined video data. Specifically, the determination data analysis unit 305 uses, for example, a known unsupervised anomaly detection algorithm such as the k-nearest neighbor method, and, as shown in FIG. Detects the presence or absence of abnormalities in quantity.
- a known unsupervised anomaly detection algorithm such as the k-nearest neighbor method
- the determination data analysis unit 305 may determine that one piece of video data that was not detected is abnormal.
- a piece of video data determined to be abnormal in this manner can also be determined to be abnormal by a known algorithm for unsupervised abnormality detection such as the k-neighborhood method described above.
- the determination data analysis unit 305 may display the detection result on the display unit (not shown) of the control device 110 when detecting an abnormality in the feature quantity such as the chuck opening/closing signal. In this manner, the determination data analysis unit 305 detects an error in the determination result by detecting an abnormality in the feature amount such as the chuck opening/closing signal, and can detect an abnormality in the work content.
- FIG. 8 is a flowchart for explaining the classification processing of the classification device 30 in the operation phase. The flow shown here is repeatedly executed while video data is input from the camera 20 .
- step S11 video data including the worker imaged by the camera 20 is input.
- step S12 the work determination unit 302 inputs the video data input in step S11 to the learned model 361, and determines the worker's work.
- step S ⁇ b>13 the determination result writing unit 303 displays the determination result of step S ⁇ b>12 on the display unit (not shown) of the control device 110 .
- step S14 the determination result correction reading unit 304 determines whether or not the correct work content input by the worker via the input unit (not shown) of the control device 110 has been acquired. If the entered correct work content is acquired, the process proceeds to step S15. On the other hand, if the input correct work content is not acquired, the process proceeds to step S16.
- step S15 the determination result correction reading unit 304 outputs the determined input data of the video data and the acquired label data of the correct work content to the machine learning device 40, and the machine learning device 40 receives the learned model 361. to update.
- step S16 the determination data analysis unit 305 detects whether there is an abnormality in the feature amount based on the determination result stored in the determination result storage unit 362 and the feature amount in the determined video data.
- the machine learning device 40 enables the worker to clearly indicate the start and end times of the work with hand gestures for each work toward the camera 20, and input data of the video data of the period.
- the operation data of the machine tool 10 collected at the same time as the teacher data is also used to create a reference for labeling the video data and the operation data.
- the machine learning device 40 labels unlabeled video data from the viewpoint of both video data and operation data by semi-supervised learning based on the created criteria.
- the machine learning device 40 can generate a learned model with high judgment accuracy even with a small amount of teacher data, does not require a new device, and easily generates a trained model 361 with less burden on workers. be able to.
- the classification device 30 can easily classify and recognize any complicated work performed by the worker from the camera 20 using the learned model 361 .
- the classification device 30 and the machine learning device 40 are not limited to the above-described embodiments, and include modifications, improvements, etc. within the scope that can achieve the purpose.
- the machine learning device 40 is illustrated as a device different from the control device 110 and the classification device 30. can be
- the classification device 30 is exemplified as a device different from the control device 110 , but the control device 110 may have some or all of the functions of the classification device 30 .
- the input unit 301, the work determination unit 302, the determination result writing unit 303, the determination result correction reading unit 304, the determination data analysis unit 305, and the storage unit 306 of the classification device 30 are You may prepare.
- each function of the classification device 30 may be implemented using a virtual server function or the like on the cloud.
- the classification device 30 may be a distributed processing system in which each function of the classification device 30 is appropriately distributed to a plurality of servers.
- the learning unit 408 performs machine learning using a set of teacher data consisting of input data of video data and label data of work content, and the operator performs machine learning based on the input video data.
- the trained model 361 that classifies the work that is performed is constructed, it is not limited to this.
- the learning unit 408 acquires the input data of the video data and the operation data and the teacher data of the label data of the work content. may be used to perform machine learning, and a learned model 361 that classifies the work performed by the worker from the input video data and operation data may be constructed.
- Each function included in the classification device 30 and the machine learning device 40 in one embodiment can be realized by hardware, software, or a combination thereof.
- “implemented by software” means implemented by a computer reading and executing a program.
- Each component included in the classification device 30 and the machine learning device 40 can be realized by hardware including electronic circuits, software, or a combination thereof.
- the programs that make up this software are installed on the computer. These programs may be recorded on removable media and distributed to users, or may be distributed by being downloaded to users' computers via a network.
- some or all of the functions of each component included in the above device are, for example, ASIC (Application Specific Integrated Circuit), gate array, FPGA (Field Programmable Gate Array), CPLD ( It can be composed of an integrated circuit (IC) such as a Complex Programmable Logic Device.
- Non-transitory computer-readable media include various types of tangible storage media.
- Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible discs, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical discs), CD-ROMs (Read Only Memory), CD- R, CD-R/W, semiconductor memory (eg, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM).
- the program may also be supplied to the computer on various types of transitory computer readable medium. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable media can deliver the program to the computer via wired communication channels, such as wires and optical fibers, or wireless communication channels.
- steps of writing a program recorded on a recording medium include not only processes that are executed chronologically in order, but also processes that are executed in parallel or individually, even if they are not necessarily processed chronologically. It also includes
- machine learning device classification device
- control device of the present disclosure can take various embodiments having the following configurations.
- the machine learning device 40 of the present disclosure includes a video data feature quantity extraction unit 402 that extracts a feature quantity indicating the action of the worker from video data including the worker captured by at least one camera 20, and a video
- a feature amount representing a pre-registered specific action is extracted from the feature amounts extracted by the data feature amount extraction unit 402
- the work content indicated by the extracted specific action is labeled to the corresponding video data.
- a teacher data extraction unit 403 for extracting teacher data of input data of labeled video data and label data of work contents; Based on the data feature quantity extraction unit 405 and the feature quantity of the video data of the extracted teacher data and the feature quantity of the operation data, the reference for giving the label for the video data and the reference for giving the label for the operation data are determined respectively.
- a labeling reference creation unit 406 that creates labels, and a labeling unit that labels unlabeled video data and operation data based on the reference for giving labels for video data and the reference for giving labels for operation data. 407, and machine learning using teacher data including video data labeled by the labeling unit 407, and learning to generate a trained model 361 that classifies the work performed by the worker from the input video data. a portion 408; According to this machine learning device 40, it is possible to generate a trained model with high judgment accuracy even with a small amount of teacher data.
- the image data feature amount extraction unit 402 extracts the following information in the machine tool 10 in the image data based on the timing of change in the machine tool 10 in the image data and the operation data.
- a feature quantity indicating the operation of the worker or a feature quantity indicating the operation of the machine tool 10 by the worker may be extracted.
- the machine learning device 40 can generate a learned model 361 that can determine the worker's work details in more detail.
- the labeling reference creation unit 406 calculates the average of the feature values extracted by the video data feature value extraction unit 402 for each training data task. Then, the distance between the calculated average of the feature amount and the feature amount of the unlabeled video data is used as a reference for assigning the label for the video data for each work of the teacher data, and extracted by the operation data feature amount extraction unit 405. Calculate the average of the feature values obtained for each operation of the training data, and for each operation of the training data, calculate the distance between the calculated average of the feature values and the feature value of the unlabeled operation data as the label for the operation data. It is good also as a reference
- the classification device 30 of the present disclosure combines the learned model 361 generated by the machine learning device 40 according to any one of (1) to (3) and the worker captured by at least one camera 20 an input unit 301 for inputting video data including a model 361; According to this classification device 30, it is possible to accurately determine the work of the worker from the video data.
- the determination result writing unit 303 that displays the determination result of the work determination unit 302 on the control device 110 that controls the machine tool 10, and if there is an error in the determination result,
- the control device 110 acquires the correct work content input, outputs the determined input data of the video data and the label data of the acquired correct work content to the machine learning device 40, and the machine learning device 40 is already learned.
- a determination result correction reading unit 304 that updates the model 361 .
- determination data for detecting the presence or absence of an abnormality in the feature amount based on the determination result of the work determination unit 302 and the feature amount in the determined video data.
- An analysis unit 305 may be provided. By doing so, the classification device 30 can detect an error in the determination result by detecting an abnormality, and can detect an abnormality in the work content.
- the machine learning device 40 according to any one of (1) to (3) may be provided. By doing so, the classification device 30 can achieve the same effect as any of (1) to (6) described above.
- the control device 110 of the present disclosure includes the classification device 30 according to any one of (4) to (7). According to this control device 110, the same effects as those of (1) to (7) can be obtained.
- control system 10 machine tool 110 control device 20 camera 30 classification device 301 input unit 302 work determination unit 303 determination result writing unit 304 determination result correction reading unit 305 determination data analysis unit 361 trained model 362 determination result storage unit 40 machine learning Apparatus 401 video data acquisition unit 402 video data feature amount extraction unit 403 teacher data extraction unit 404 operation data acquisition unit 405 operation data feature amount extraction unit 406 labeling reference creation unit 407 labeling unit 408 learning unit 4091 teacher data extraction sign storage unit
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Business, Economics & Management (AREA)
- Human Resources & Organizations (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Software Systems (AREA)
- Strategic Management (AREA)
- Entrepreneurship & Innovation (AREA)
- Economics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- Tourism & Hospitality (AREA)
- Marketing (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- Human Computer Interaction (AREA)
- Quality & Reliability (AREA)
- Educational Administration (AREA)
- Game Theory and Decision Science (AREA)
- Databases & Information Systems (AREA)
- General Business, Economics & Management (AREA)
- Operations Research (AREA)
- Development Economics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
Abstract
Description
例えば、センサやレーザを作業員に装着させるとともに、ビデオカメラで作業者を撮像し、センサ、レーザ及びビデオカメラそれぞれにより測定されたトラッキングデータからタクト単位作業及び最小単位作業を抽出し、抽出した最小単位作業から特徴ベクトルを算出し、算出した特徴ベクトルに基づいて当該作業員の作業動作の分析処理を行い、作業ミスの検出や、作業者の姿勢変化の検出、実質作業時間の検出等を行う技術が提案されている。例えば、特許文献1参照。
しかしながら、学習済みモデルや分類器を生成する深層学習を行うには大量の教師データが必要であり、教師データを収集するのに手間と時間がかかるという問題がある。
<一実施形態>
図1は、一実施形態に係る制御システムの機能的構成例を示す機能ブロック図である。図1に示すように、制御システム1は、工作機械10、カメラ20、分類装置30、及び機械学習装置40を有する。
具体的には、制御装置110は、工作機械10を制御することにより、工作機械10に所定の機械加工を行わせる装置である。制御装置110には、工作機械10の動作を記述した加工プログラムが与えられる。制御装置110は、与えられた加工プログラムに基づいて、各軸に対する移動指令、主軸を駆動するモータへの回転指令等を含む動作指令を作成し、この動作指令を工作機械10に送信することにより、工作機械10のモータを制御する。これにより、工作機械10による所定の機械加工が実行される。
また、制御装置110は、動作指令や扉の開閉、モータのトルク値等を含む工作機械10の稼働データを分類装置30に送信する。さらに、制御装置110は、制御装置110に含まれるクロック(図示しない)のクロック信号に基づいて、動作指令や扉の開閉や、モータのトルク値等が測定された時刻を示す時刻情報を稼働データに付加して分類装置30に出力してもよい。また、制御装置110は、例えば、所定の時間間隔で、後述するカメラ20に含まれるクロック(図示しない)の時刻をチェックし互いに同期させるようにしてもよい。
なお、工作機械10がロボット等の場合、制御装置110は、ロボット制御装置等でもよい。
また、制御装置110の制御対象の装置は工作機械10やロボットに限定されず、産業機械全般に広く適用することができる。産業機械とは、例えば、工作機械、産業用ロボット、サービス用ロボット、鍛圧機械及び射出成形機といった様々な機械を含む。
本実施形態では、制御装置110として、数値制御装置を例示する。
図2は、カメラ20により撮像された映像の一例を示す図である。図2では、カメラ20が3台の工作機械10を撮像した映像を示す。
なお、図1では、カメラ20は1台が配置されたが、2台以上のカメラ20が配置され工作機械10を撮像してもよい。
また、分類装置30は、取得した工作機械10の稼働データ、及びカメラ20により撮像された映像データを、後述する機械学習装置40に送信してもよい。
また、分類装置30は、運用フェーズにおいて、後述する機械学習装置40により生成された学習済みモデルを受信し、受信した学習済みモデルにカメラ20により撮像された映像データを入力することで、映像内の作業員の作業内容を判定(分類)し、当該判定結果を制御装置110に含まれる液晶ディスプレイ等の表示部(図示しない)に表示するようにしてもよい。そして、分類装置30は、判定結果に誤りがある場合、制御装置110において入力された正しい作業内容を取得し、判定された映像データの入力データと取得した正しい作業内容のラベルデータとを機械学習装置40に出力することで、機械学習装置40に対して学習済みモデルを更新させるようにしてもよい。
機械学習装置40は、例えば、カメラ20により撮像された映像データを取得し、取得した映像データから、後述するように、作業員の動作を示す特徴量を抽出する。
機械学習装置40は、抽出された特徴量のうち、後述するように、予め登録された作業員の特定の動作としてのハンドジェスチャの特徴量を抽出した場合、抽出されたハンドジェスチャが示す作業内容のラベルを対応する映像データにラベル付けし、ラベル付けした映像データの入力データと作業内容のラベルデータとの教師データを抽出する。
また、機械学習装置40は、工作機械10の稼働データを取得し、取得した稼働データから、後述するように、上述の教師データに関連する特徴量を抽出し、抽出された教師データと稼働データとの特徴量から、後述するように、映像データ用のラベルを付与する基準、及び稼働データ用のラベルを付与する基準をそれぞれ作成する。
機械学習装置40は、映像データ用のラベルを付与する基準と稼働データ用のラベルを付与する基準とに基づいて少なくともラベル付けされていない映像データにラベル付けする。
機械学習装置40は、新たにラベル付けされた映像データを含む教師データ(訓練データ)により教師あり学習を行い、後述する学習済みモデルを構築する。
そうすることで、機械学習装置40は、少ない教師データでも判定精度の高い学習済みモデルを構築することができ、構築した学習済みモデルを分類装置30に提供することができる。
機械学習装置40について、具体的に説明する。
教師データ抽出サイン記憶部4091は、例えば、ラベルとして付与する作業内容と、各作業内容を示す特定の動作(ハンドジェスチャ等)と、を対応付けて格納する。
図3は、作業の時系列の一例を示す図である。
図3に示すように、作業内容には、「待機」、「製品A用の段取り」、「掃除」、「工具交換」が含まれる。また、作業内容は、「製品B用の段取り」等、作業員等のユーザにより自由に定義されてもよい。
図4A及び図4Bは、工具交換の作業内容を示すハンドジェスチャの一例を示す図である。図4Aは、工具交換の開始を示すハンドジェスチャの一例を示し、図4Bは、工具交換の終了を示すハンドジェスチャの一例を示す。
すなわち、作業員がカメラ20に向かって工具交換等の作業内容毎に予め登録されたハンドジェスチャ(特定の動作)を各作業の開始時と終了時とに行うことにより、どの作業を行っているかを明示する。これにより、映像データには、ハンドジェスチャにより示された作業内容のラベルを付与することができ、教師データとすることができる。
そうすることで、映像データから作業内容を判定する学習済みモデルを生成するための教師データを容易に収集することができる。
具体的には、映像データ特徴量抽出部402は、公知の手法(例えば、菅野滉介、奥健太、川越恭二、「多次元時系列データからのモーション検出・分類手法」、DEIM Forum 2016 G4-5、又は、上園翔平、小野智司、「LSTM Autoencoderを用いたマルチモーダル系列データの特徴抽出」、人工知能学会研究会資料、SIG-KBS-B802-01、2018)を用いて、時刻情報が付加された映像データから作業員の体(指、腕、脚等)の関節の座標、角度の時系列データの特徴量を抽出する。また、映像データ特徴量抽出部402は、映像データから抽出した体の関節の座標、角度の時系列データから抽出される統計的な特徴量(平均値、ピーク値等)を取得する。また、映像データ特徴量抽出部402は、体の関節の座標と角度から、細かい動作を判断して特徴量(数値制御(NC)操作や工作機械10内での作業等、段取りを構成する細かい動作か否か、段取りの場合には当該段取りの時刻等)を抽出する。
例えば、映像データ特徴量抽出部402は、稼働データから工作機械10の扉の開閉タイミングを把握し、映像データ内で同じタイミングで変化する箇所がある場合、当該箇所を工作機械10の扉と判断してもよい。これにより、映像データ特徴量抽出部402は、当該箇所において作業員が上半身を入れている場合、「工作機械内での作業」という特徴量を抽出することができる。また、映像データ特徴量抽出部402は、作業員の上半身が工作機械10の上の方を向いている場合に「自動工具交換装置(ATF)で作業」等の特徴量を抽出し、作業員の上半身が工作機械10の下の方を向いている場合に「テーブルで作業」等の特徴量を抽出するようにしてもよい。
また、映像データ特徴量抽出部402は、稼働データから作業員がNC操作のタイミングにおいて、映像データにおいて作業員がボタンを押している箇所がある場合、当該箇所をNC領域と判断してもよい。これにより、映像データ特徴量抽出部402は、当該箇所に手を置いている場合、「NC操作」という特徴量を抽出することができる。
具体的には、稼働データ特徴量抽出部405は、例えば、教師データ抽出部403により抽出された教師データの作業の開始と終了との時刻から時間幅を算出、又はラベリングされた作業毎に時間幅の平均を算出する。
図5は、製品A用の段取り時間幅の一例を示す図である。
図5に示すように、稼働データ特徴量抽出部405は、オーバーラップさせながら算出した製品A用の段取りの時間幅で時間を切り取り、切り取った時間毎に工作機械10の稼働データから、分類する作業に関連する工作機械10の信号情報(チャックの開閉、モータのトルクから計算した工具の異常値等)と、同じ加工をしているのか判断する情報(加工プログラム、製造番号)と、及び工作機械10の稼働状態及び稼働状態の推移(例えば、工具交換のアラームから通常の稼働に復帰等の作業内容の変化)と、を稼働データの特徴量として抽出する。
なお、映像データ特徴量抽出部402は、稼働データ特徴量抽出部405の場合と同様に、オーバーラップしながら算出した各作業の時間幅で映像データを切り取り、当該時間での映像データの特徴量を抽出するようにしてもよい。
図6A及び図6Bは、作業毎の教師データの一例を示す図である。図6Aでは、段取り作業(稼働状態が停止中)の場合の教師データを示し、図6Bでは、工具交換(稼働状態がアラーム中)の場合の教師データを示す。なお、図6Aのチャック開閉信号は、工作機械内作業の開始時に「閉」から「開」に変化し、工作機械内作業の終了時に「開」から「閉」に変化している。一方、図6Bのチャック開閉信号は、工作機械内作業の間、「閉」のままである。また、稼働状態には、稼働中、(ネットワーク)切断中、非常停止中、一時停止中、手動運転中、暖機運転中等がある。
図6A及び図6Bに示す教師データの作業毎に、ラベル付与基準作成部406は、例えば、映像データ特徴量抽出部402により抽出された特徴量の平均を算出する。ラベル付与基準作成部406は、教師データの作業毎に、算出した特徴量の平均とラベル付けされていない映像データの特徴量とのマハラノビス距離等の距離を、映像データ用のラベルを付与する基準として計算する。
また、ラベル付与基準作成部406は、図6A及び図6Bに示す教師データの作業毎に、稼働データ特徴量抽出部405により抽出された特徴量の平均を算出する。ラベル付与基準作成部406は、教師データの作業毎に、算出した特徴量の平均とラベル付けされていない稼働データの特徴量とのマハラノビス距離等の距離を、稼働データ用のラベルを付与する基準として計算する。
なお、ラベル付与基準作成部406は、距離を算出したが、これに限定されず、例えば機械学習(例えば、CART(Classification And Regression Tree)等の決定木アルゴリズム)を行い、分類器を作成してどの作業に分類されるかの確率を算出するようにしてもよい。
具体的には、ラベル付与部407は、例えば、ラベル付与基準作成部406により算出された映像データ及び稼働データの距離(基準)と(式1)とに基づいて重み付けした距離(基準)を算出し、算出した距離が一定以上の距離より小さい場合、ラベル付けされていない映像データ及び稼働データに対してラベル付けを行う。なお、重み付け係数「0.8」、「0.2」は一例であり、任意の値を用いてもよい。
(式1)
2つの基準を使った距離 =
映像データから算出した距離×0.8+稼働データから算出した距離×0.2
そうすることで、ラベル付与部407は、複雑な作業のラベル付けを少ない教師データで可能にするとともに、映像データでは違う部分が多少あったとしても、稼働データの特徴量がほぼ同じであるためラベル付けすることができる。
(式2)
2つの基準を使った確率 =
映像データから算出した確率×0.8+稼働データから算出した確率×0.2
そして、学習部408は、構築した学習済みモデル361を分類装置30に対して提供する。
なお、学習部408は、学習済みモデル361を分類装置30に提供した後、分類装置30から新たな映像データの入力データと作業内容のラベルデータとの組の教師データを取得した場合、取得した教師データを用いて再度機械学習を行い、学習済みモデル361を更新するようにしてもよい。
オンライン学習とは、分類装置30から教師データを取得する都度、即座に教師あり学習を行うという学習方法である。また、バッチ学習とは、分類装置30から教師データを取得することが繰り返される間に、繰り返しに応じた複数の教師データが収集され、収集された全ての教師データを用いて、教師あり学習を行うという学習方法である。さらに、ミニバッチ学習とは、オンライン学習と、バッチ学習の中間的な、ある程度教師データが溜まるたびに教師あり学習を行うという学習方法である。
以上、分類装置30が備える学習済みモデル361を生成するための機械学習について説明した。
次に、運用フェーズにおける分類装置30について説明する。
図1に示すように、運用フェーズにおける分類装置30は、入力部301、作業判定部302、判定結果書込部303、判定結果修正読込部304、判定データ解析部305、及び記憶部306を含んで構成される。
なお、分類装置30は、図1の機能ブロックの動作を実現するために、CPU(Central Processing Unit)等の図示しない演算処理装置を備える。また、分類装置30は、各種の制御用プログラムを格納したROMやHDD等の図示しない補助記憶装置や、演算処理装置がプログラムを実行する上で一時的に必要とされるデータを格納するためのRAMといった図示しない主記憶装置を備える。
判定結果記憶部362は、後述する作業判定部302により判定された映像データに対する判定結果と判定された映像データにおける特徴量とを対応付けて記憶する。
そうすることで、作業員は、学習済みモデル361による分類の良し悪しを判定することができる。
なお、結果の読み込み、修正の書き込みは、例えば後述する記憶部306やPMC領域等を使用してもよい。
具体的には、判定データ解析部305は、例えば、k近傍法等の公知の教師無し異常検知のアルゴリズムを使用し、図7に示すように、判定結果記憶部362の同じ結果のデータから特徴量の異常の有無を検出する。
あるいは、例えば、段取り作業と判定された映像データが100個あり、99個の映像データにおいてチャック開閉信号のON(開)からOFF(閉)の特徴量が検知され、1個の映像データにおいてチャック開閉信号のON(開)からOFF(閉)の特徴量が検知されなかった場合、判定データ解析部305は、検出されなかった1個の映像データを異常と判定するようにしてもよい。なお、このように異常と判定される1個の映像データは、前述したk近傍法等の公知の教師無し異常検知のアルゴリズムによっても異常と判定することができる。
そして、判定データ解析部305は、チャック開閉信号等の特徴量の異常を検出した場合、検出結果を制御装置110の表示部(図示しない)に表示するようにしてもよい。
このように、判定データ解析部305は、チャック開閉信号等の特徴量の異常を検知することで、判定結果の誤りが分かり、作業内容の異常を検知することができる。
次に、本実施形態に係る分類装置30の分類処理に係る動作について説明する。
図8は、運用フェーズにおける分類装置30の分類処理について説明するフローチャートである。ここで示すフローは、カメラ20から映像データが入力される間繰り返し実行される。
また、分類装置30は、学習済みモデル361を用いて、作業員が行っている複雑な任意の作業を、カメラ20から容易に分類し認識することができる。
上述の実施形態では、機械学習装置40は、制御装置110及び分類装置30と異なる装置として例示したが、機械学習装置40の一部又は全部の機能を、制御装置110又は分類装置30が備えるようにしてもよい。
また例えば、上述の実施形態では、分類装置30は、制御装置110と異なる装置として例示したが、分類装置30の一部又は全部の機能を、制御装置110が備えるようにしてもよい。
あるいは、分類装置30の入力部301、作業判定部302、判定結果書込部303、判定結果修正読込部304、判定データ解析部305、及び記憶部306の一部又は全部を、例えば、サーバが備えるようにしてもよい。また、クラウド上で仮想サーバ機能等を利用して、分類装置30の各機能を実現してもよい。
さらに、分類装置30は、分類装置30の各機能を適宜複数のサーバに分散される、分散処理システムとしてもよい。
また例えば、上述の実施形態では、学習部408は、映像データの入力データと作業内容のラベルデータとの組の教師データを用いて機械学習を行い、入力された映像データから作業員が行っている作業を分類する学習済みモデル361を構築したが、これに限定されない。例えば、分類装置30の作業判定部302が映像データと稼働データの両方を取得することができる場合、学習部408は、映像データと稼働データとの入力データと作業内容のラベルデータとの教師データを用いて機械学習を行い、入力された映像データと稼働データとから作業員が行っている作業を分類する学習済みモデル361を構築してもよい。
この機械学習装置40によれば、少ない教師データでも判定精度の高い学習済みモデルを生成することができる。
そうすることで、機械学習装置40は、より詳細に作業員の作業内容を判定できる学習済みモデル361を生成することができる。
そうすることで、機械学習装置40は、複雑な作業のラベル付けを少ない教師データで可能にする。
この分類装置30によれば、映像データから作業員の作業を精度良く判定することができる。
そうすることで、作業員は、学習済みモデル361による分類の良し悪しを判定することができ、分類装置30は、作業員から正しい作業内容を受けることで学習済みモデル361を更新することができる。
そうすることで、分類装置30は、異常を検知することで、判定結果の誤りが分かり、作業内容の異常を検知することができる。
そうすることで、分類装置30は、上述の(1)から(6)のいずれかと同様の効果を奏することができる。
この制御装置110によれば、上述の(1)から(7)のいずれかと同様の効果を奏することができる。
10 工作機械
110 制御装置
20 カメラ
30 分類装置
301 入力部
302 作業判定部
303 判定結果書込部
304 判定結果修正読込部
305 判定データ解析部
361 学習済みモデル
362 判定結果記憶部
40 機械学習装置
401 映像データ取得部
402 映像データ特徴量抽出部
403 教師データ抽出部
404 稼働データ取得部
405 稼働データ特徴量抽出部
406 ラベル付与基準作成部
407 ラベル付与部
408 学習部
4091 教師データ抽出サイン記憶部
Claims (8)
- 少なくとも1つのカメラにより撮像された作業員を含む映像データから、前記作業員の動作を示す特徴量を抽出する映像データ特徴量抽出部と、
前記映像データ特徴量抽出部により抽出された前記特徴量のうち、予め登録された特定の動作を示す特徴量を抽出した場合、抽出された前記特定の動作が示す作業内容を対応する映像データにラベル付けし、ラベル付けした前記映像データの入力データと前記作業内容のラベルデータとの教師データを抽出する教師データ抽出部と、
産業機械の稼働データから、抽出した前記教師データに関連する特徴量を抽出する稼働データ特徴量抽出部と、
抽出された前記教師データの映像データの特徴量と前記稼働データの特徴量とから、映像データ用のラベルを付与する基準、及び稼働データ用のラベルを付与する基準をそれぞれ作成するラベル付与基準作成部と、
前記映像データ用のラベルを付与する基準と前記稼働データ用のラベルを付与する基準とに基づいてラベル付けされていない映像データ及び稼働データにラベル付けするラベル付与部と、
前記ラベル付与部でラベル付けされた前記映像データを含む前記教師データを用いて機械学習を行い、入力された映像データから作業員が行っている作業を分類する学習済みモデルを生成する学習部と、
を備える機械学習装置。 - 前記映像データ特徴量抽出部は、映像データと稼働データとにおける前記産業機械の変化のタイミングに基づいて、映像データにおいて前記産業機械内における前記作業員の動作を示す特徴量、又は前記作業員による前記産業機械に対する操作を示す特徴量を抽出する、請求項1に記載の機械学習装置。
- 前記ラベル付与基準作成部は、前記映像データ特徴量抽出部により抽出された特徴量の平均を前記教師データの作業毎に算出し、前記教師データの作業毎に、算出した前記特徴量の平均とラベル付けされていない映像データの特徴量との距離を映像データ用のラベルを付与する基準とするとともに、前記稼働データ特徴量抽出部により抽出された特徴量の平均を前記教師データの作業毎に算出し、前記教師データの作業毎に、算出した前記特徴量の平均とラベル付けされていない稼働データの特徴量との距離を稼働データ用のラベルを付与する基準とする、請求項1又は請求項2に記載の機械学習装置。
- 請求項1から請求項3のいずれか1項に記載の機械学習装置により生成された学習済みモデルと、
少なくとも1つのカメラにより撮像された作業員を含む映像データを入力する入力部と、
前記学習済みモデルに基づいて、前記入力部により入力された前記映像データから前記作業員の作業を判定する作業判定部と、
を備える分類装置。 - 前記作業判定部の判定結果を前記産業機械を制御する制御装置に表示させる判定結果書込部と、
前記判定結果に誤りがある場合、前記制御装置において入力された正しい作業内容を取得し、判定された前記映像データの入力データと取得した前記正しい作業内容のラベルデータとを前記機械学習装置に出力し、前記機械学習装置に対して前記学習済みモデルを更新させる判定結果修正読込部と、を備える、請求項4に記載の分類装置。 - 前記作業判定部の判定結果と判定された前記映像データにおける特徴量とに基づいて、前記特徴量の異常の有無を検出する判定データ解析部を備える、請求項4又は請求項5に記載の分類装置。
- 請求項1から請求項3のいずれか1項に記載の機械学習装置を備える、請求項4から請求項5のいずれか1項に記載の分類装置。
- 請求項4から請求項7のいずれか1項に記載の分類装置を備える、制御装置。
Priority Applications (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
DE112021007196.3T DE112021007196T5 (de) | 2021-05-07 | 2021-05-07 | Maschinenlernvorrichtung, klassifizierungsvorrichtung und steuervorrichtung |
PCT/JP2021/017578 WO2022234678A1 (ja) | 2021-05-07 | 2021-05-07 | 機械学習装置、分類装置、及び制御装置 |
CN202180097691.XA CN117296067A (zh) | 2021-05-07 | 2021-05-07 | 机器学习装置、分类装置以及控制装置 |
JP2023518606A JPWO2022234678A1 (ja) | 2021-05-07 | 2021-05-07 | |
US18/287,085 US20240193919A1 (en) | 2021-05-07 | 2021-05-07 | Machine learning device, classification device, and control device |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/JP2021/017578 WO2022234678A1 (ja) | 2021-05-07 | 2021-05-07 | 機械学習装置、分類装置、及び制御装置 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2022234678A1 true WO2022234678A1 (ja) | 2022-11-10 |
Family
ID=83932057
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2021/017578 WO2022234678A1 (ja) | 2021-05-07 | 2021-05-07 | 機械学習装置、分類装置、及び制御装置 |
Country Status (5)
Country | Link |
---|---|
US (1) | US20240193919A1 (ja) |
JP (1) | JPWO2022234678A1 (ja) |
CN (1) | CN117296067A (ja) |
DE (1) | DE112021007196T5 (ja) |
WO (1) | WO2022234678A1 (ja) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11887367B1 (en) | 2023-04-19 | 2024-01-30 | OpenAI Opco, LLC | Using machine learning to train and use a model to perform automatic interface actions based on video and input datasets |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2018198028A (ja) * | 2017-05-25 | 2018-12-13 | 日本電信電話株式会社 | 移動状況認識モデル学習装置、移動状況認識装置、方法、及びプログラム |
US20200082210A1 (en) * | 2018-09-07 | 2020-03-12 | International Business Machines Corporation | Generating and augmenting transfer learning datasets with pseudo-labeled images |
CN111563424A (zh) * | 2020-04-20 | 2020-08-21 | 清华大学 | 基于半监督学习的行人再识别方法及装置 |
JP2020144660A (ja) * | 2019-03-07 | 2020-09-10 | キヤノン株式会社 | 情報処理装置、情報処理方法及びプログラム |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2006209468A (ja) | 2005-01-28 | 2006-08-10 | Yakahi Kikuko | 作業動作解析装置、作業動作解析方法および作業動作解析プログラム |
-
2021
- 2021-05-07 DE DE112021007196.3T patent/DE112021007196T5/de active Pending
- 2021-05-07 WO PCT/JP2021/017578 patent/WO2022234678A1/ja active Application Filing
- 2021-05-07 US US18/287,085 patent/US20240193919A1/en active Pending
- 2021-05-07 CN CN202180097691.XA patent/CN117296067A/zh active Pending
- 2021-05-07 JP JP2023518606A patent/JPWO2022234678A1/ja active Pending
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2018198028A (ja) * | 2017-05-25 | 2018-12-13 | 日本電信電話株式会社 | 移動状況認識モデル学習装置、移動状況認識装置、方法、及びプログラム |
US20200082210A1 (en) * | 2018-09-07 | 2020-03-12 | International Business Machines Corporation | Generating and augmenting transfer learning datasets with pseudo-labeled images |
JP2020144660A (ja) * | 2019-03-07 | 2020-09-10 | キヤノン株式会社 | 情報処理装置、情報処理方法及びプログラム |
CN111563424A (zh) * | 2020-04-20 | 2020-08-21 | 清华大学 | 基于半监督学习的行人再识别方法及装置 |
Non-Patent Citations (1)
Title |
---|
KAWAMORI, DAIKI: "Semi-Supervised Temporal Segmentation of Industrial Operation Video based on Deep Metric Learning", IEICE TECHNICAL REPORT, IEICE, JP, vol. 119, no. 481, 9 March 2020 (2020-03-09), JP , pages 139 - 144 (PRMU2019-92), XP009540699, ISSN: 2432-6380 * |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11887367B1 (en) | 2023-04-19 | 2024-01-30 | OpenAI Opco, LLC | Using machine learning to train and use a model to perform automatic interface actions based on video and input datasets |
Also Published As
Publication number | Publication date |
---|---|
JPWO2022234678A1 (ja) | 2022-11-10 |
DE112021007196T5 (de) | 2024-01-18 |
CN117296067A (zh) | 2023-12-26 |
US20240193919A1 (en) | 2024-06-13 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11701772B2 (en) | Operation prediction system and operation prediction method | |
US11442429B2 (en) | Systems and methods for advance anomaly detection in a discrete manufacturing process with a task performed by a human-robot team | |
JP6897037B2 (ja) | 作業性評価装置 | |
US10740122B2 (en) | System module of simulating machine operation screen based on non-invasive data-extraction system | |
CN113412177B (zh) | 机器人控制装置、机器人***以及机器人控制方法 | |
CN111308925B (zh) | 信息处理装置、生产指示辅助方法 | |
JP7275062B2 (ja) | 時変システム動作における異常検出のためのシステム、方法およびコンピュータ読取可能記憶媒体 | |
US11269710B2 (en) | Diagnostic apparatus | |
WO2022234678A1 (ja) | 機械学習装置、分類装置、及び制御装置 | |
Wang et al. | A smart operator advice model by deep learning for motion recognition in human–robot coexisting assembly line | |
US20230068757A1 (en) | Work rate measurement device and work rate measurement method | |
EP4016376A1 (en) | Computer-implemented process monitoring method | |
JP6731603B1 (ja) | 検査システム | |
JP7170496B2 (ja) | 工作機械における加工異常診断モデル構築用の加工データへのラベル付与方法及び加工異常診断システム | |
WO2022234668A1 (ja) | 稼働管理装置 | |
WO2022097310A1 (ja) | 作業推定装置、作業推定装置の制御方法、情報処理プログラム、および記録媒体 | |
WO2022209638A1 (ja) | 作業指示システム | |
US20240160202A1 (en) | Information processing device, information processing method, and non-transitory storage medium encoded with computer-readable information processing program | |
WO2022157988A1 (ja) | 作業推定装置、作業推定装置の制御方法、情報処理プログラム、および記録媒体 | |
Jin et al. | Determination of defects for dynamic objects using instance segmentation | |
EP4187334A1 (en) | A method for generating presentations on a display screen | |
Xu et al. | Research on Intelligent Multimodal Gesture-Guided Control of Robotic Arms | |
JP2024101163A (ja) | 情報処理方法、情報処理装置、ロボットシステム、ロボットシステムの制御方法、物品の製造方法、プログラム及び記録媒体。 | |
Saboury et al. | Robotic Monitoring Enhancement with Deep Learning and Conformal Prediction for Indoor Anomaly Detection in Emergency Situations | |
Slama et al. | Advancing human-robot interaction in a manufacturing environment by incorporating hand movements for remote control |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21939868 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2023518606 Country of ref document: JP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 18287085 Country of ref document: US |
|
WWE | Wipo information: entry into national phase |
Ref document number: 202180097691.X Country of ref document: CN Ref document number: 112021007196 Country of ref document: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 21939868 Country of ref document: EP Kind code of ref document: A1 |