CN110181503B

CN110181503B - Anomaly detection method and device, intelligent equipment and storage medium

Info

Publication number: CN110181503B
Application number: CN201810154107.XA
Authority: CN
Inventors: 梁彬欣
Original assignee: Beijing Orion Star Technology Co Ltd
Current assignee: Beijing Orion Star Technology Co Ltd
Priority date: 2018-02-22
Filing date: 2018-02-22
Publication date: 2021-09-10
Anticipated expiration: 2038-02-22
Also published as: CN110181503A

Abstract

The invention discloses an anomaly detection method, an anomaly detection device, intelligent equipment and a storage medium. The method comprises the following steps: when the intelligent equipment executes a target task, acquiring a scene image of a scene to be detected, wherein the scene to be detected is a scene which is corresponding to a preset target task and needs to be subjected to abnormal detection; and according to the scene image of the scene to be detected, carrying out abnormity detection on the workflow of the target task executed by the intelligent equipment. The method can enable the intelligent equipment to have the function of workflow abnormity detection, ensures that the intelligent equipment can provide effective service, does not need artificial detection in the whole process, and greatly reduces the labor cost.

Description

Anomaly detection method and device, intelligent equipment and storage medium

Technical Field

The present invention relates to the field of intelligent device control, and in particular, to an anomaly detection method and apparatus, an intelligent device, and a computer-readable storage medium.

Background

With the continuous improvement of the living standard requirements of people, intelligent devices (such as anthropomorphic intelligent robots, mechanical arms and the like) are increasingly applied to the life and service of people so as to realize various interactions with human beings. At present, how to solve the interaction problem between the equipment and people is more considered in the design of the intelligent equipment, for example, in interaction modes such as a touch screen and voice, the intelligent equipment can obtain feedback information from a user so as to determine the workflow of the next step.

When the intelligent device provides service for the user, the intelligent device can interact with other things in the environment, the perception of interaction scene information is also important, and even the work effect of the intelligent device can be directly influenced. For example, when a robot arm is used to make coffee, the coffee making is automatically completed according to known action steps, and interaction of other things (such as people, equipment or tools) is used in the process. However, in such an interaction scenario, various situations may occur due to external factors, for example, after the mechanical arm presses a button of the milk cabinet, a milk tube on the milk cabinet does not output milk, and the intelligent device in the prior art usually implements abnormality detection of the mechanical arm through manual monitoring without a corresponding own abnormality detection strategy, which often wastes a large amount of labor cost, and weakens the intelligence of the mechanical arm.

Disclosure of Invention

The object of the present invention is to solve at least to some extent one of the above mentioned technical problems.

To this end, a first object of the invention is to propose an anomaly detection method. The method can enable the intelligent equipment to have the function of workflow abnormity detection, ensures that the intelligent equipment can provide effective service, does not need artificial detection in the whole process, and greatly reduces the labor cost.

A second object of the present invention is to provide an abnormality detection device.

A third object of the present invention is to provide a smart device.

A fourth object of the invention is to propose a computer-readable storage medium.

In order to achieve the above object, an abnormality detection method according to an embodiment of a first aspect of the present invention includes: when the intelligent equipment executes a target task, acquiring a scene image of a scene to be detected, wherein the scene to be detected is a preset scene which corresponds to the target task and needs to be subjected to abnormal detection; and according to the scene image of the scene to be detected, carrying out abnormity detection on the workflow of the target task executed by the intelligent equipment.

According to the anomaly detection method provided by the embodiment of the invention, when the intelligent equipment executes the target task, the scene image of the scene to be detected is obtained, and the anomaly detection is carried out on the workflow of the target task executed by the intelligent equipment according to the scene image of the scene to be detected. In other words, in the workflow of executing a certain task by the intelligent device, when information fed back by an interaction scene of an environment or other things (such as equipment or tools) needs to be confirmed, an image under the scene can be acquired, and whether the scene is abnormal or not is judged by analyzing the image, so that the information fed back by the interaction scene of the external environment or other things is sensed by using the visual sensor and is combined with the workflow set by the self program, the purpose of providing effective product service is achieved, the intelligent device has the function of detecting the abnormality of the workflow of the intelligent device, the whole process does not need artificial detection, the labor cost is greatly reduced, and the intelligent device is ensured to be capable of providing effective service.

According to one embodiment of the invention, each scene to be detected corresponds to an image class detection model, and the image class detection model is a pre-trained neural network model for image classification; the image categories include at least one normal category and at least one abnormal category; according to the scene image of the scene to be detected, abnormal detection is carried out on the workflow of the intelligent equipment executing the target task, and the method comprises the following steps:

acquiring an image category detection model corresponding to the scene to be detected; and judging whether the scene to be detected is abnormal or not based on the acquired image type detection model and the scene image of the scene to be detected.

According to an embodiment of the present invention, the image category detection model is generated specifically by the following method: acquiring scene images of various image categories as sample data for each scene to be detected; and training to obtain an image type detection model corresponding to each scene to be detected based on the sample data.

According to an embodiment of the present invention, the abnormal detection of the workflow of the intelligent device executing the target task according to the scene image of the scene to be detected includes: extracting a characteristic vector of a scene image of the scene to be detected; determining the image category corresponding to the extracted feature vector according to the extracted feature vector and the pre-established corresponding relation between the feature vector and the image category; the image categories in the corresponding relation comprise at least one normal category and at least one abnormal category; and judging whether the scene to be detected is abnormal or not according to the determined image category.

According to an embodiment of the present invention, extracting a feature vector of a scene image of the scene to be detected includes: acquiring a feature vector extraction model; the feature vector extraction model is a pre-trained neural network model used for extracting image feature vectors; and extracting the feature vector of the scene image of the scene to be detected based on the feature vector extraction model.

According to an embodiment of the present invention, determining an image category corresponding to an extracted feature vector according to the extracted feature vector and a pre-established correspondence between the feature vector and the image category specifically includes: and determining the image category corresponding to the extracted feature vector by adopting a clustering algorithm according to the extracted feature vector and the pre-established corresponding relation between the feature vector and the image category.

According to an embodiment of the present invention, determining an image category corresponding to an extracted feature vector by using a clustering algorithm according to the extracted feature vector and a pre-established correspondence between the feature vector and the image category specifically includes: determining the distance between the extracted feature vector and each feature vector in the pre-established corresponding relation between the feature vector and the image category; determining K feature vectors closest to the extracted feature vectors as neighbor feature vectors, wherein K is a positive integer greater than 1; determining the image category corresponding to each adjacent feature vector according to the corresponding relation; and determining the image category with the highest proportion in the determined image categories as the image category corresponding to the extracted feature vector.

According to an embodiment of the present invention, before determining the image class with the highest proportion in the determined image classes as the image class corresponding to the extracted feature vector, the method further includes: and determining that the proportion of the image category with the highest proportion reaches a preset proportion.

According to an embodiment of the present invention, the correspondence between the feature vector and the image category is specifically established as follows: acquiring scene images of various image categories as sample images in each scene to be detected; extracting a feature vector of each sample image; and establishing a corresponding relation between the feature vector of each sample image and the corresponding image category.

According to one embodiment of the invention, the target task comprises a plurality of action steps; the scene to be detected comprises: a scenario before the smart device performs the first specified action step, and/or a scenario after the smart device performs the second specified action step.

According to one embodiment of the invention, the smart device comprises a robotic arm.

In order to achieve the above object, an abnormality detection device according to an embodiment of a second aspect of the present invention includes: the image acquisition module is used for acquiring a scene image of a scene to be detected when the intelligent equipment executes a target task, wherein the scene to be detected is a preset scene which corresponds to the target task and needs to be subjected to abnormity detection; and the anomaly detection module is used for carrying out anomaly detection on the workflow of the target task executed by the intelligent equipment according to the scene image of the scene to be detected.

According to the anomaly detection device provided by the embodiment of the invention, the scene image of the scene to be detected can be acquired by the image acquisition module when the intelligent equipment executes the target task, and the anomaly detection module carries out anomaly detection on the workflow of the target task executed by the intelligent equipment according to the scene image of the scene to be detected. In other words, in the workflow of executing a certain task by the intelligent device, when information fed back by an interaction scene of an environment or other things (such as equipment or tools) needs to be confirmed, an image under the scene can be acquired, and whether the scene is abnormal or not is judged by analyzing the image, so that the information fed back by the interaction scene of the external environment or other things is sensed by using the visual sensor and is combined with the workflow set by the self program, the purpose of providing effective product service is achieved, the intelligent device has the function of detecting the abnormality of the workflow of the intelligent device, the whole process does not need artificial detection, the labor cost is greatly reduced, and the intelligent device is ensured to be capable of providing effective service.

According to one embodiment of the invention, each scene to be detected corresponds to an image class detection model, and the image class detection model is a pre-trained neural network model for image classification; the image categories include at least one normal category and at least one abnormal category; wherein the anomaly detection module comprises: the acquisition unit is used for acquiring an image type detection model corresponding to the scene to be detected; and the first anomaly detection unit is used for judging whether the scene to be detected is anomalous or not based on the acquired image type detection model and the scene image of the scene to be detected.

According to an embodiment of the invention, the apparatus further comprises: the model pre-generation module is used for pre-generating an image category detection model corresponding to each scene to be detected; wherein the model pre-generation module is specifically configured to: acquiring scene images of various image categories as sample data for each scene to be detected; and training to obtain an image type detection model corresponding to each scene to be detected based on the sample data.

According to one embodiment of the invention, the anomaly detection module comprises: the feature vector extraction unit is used for extracting feature vectors of the scene images of the scene to be detected; the image type determining unit is used for determining the image type corresponding to the extracted feature vector according to the extracted feature vector and the corresponding relation between the pre-established feature vector and the image type; the image categories in the corresponding relation comprise at least one normal category and at least one abnormal category; and the second abnormity detection unit is used for judging whether the scene to be detected is abnormal or not according to the determined image category.

According to an embodiment of the present invention, the feature vector extraction unit is specifically configured to: acquiring a feature vector extraction model; the feature vector extraction model is a pre-trained neural network model used for extracting image feature vectors; and extracting the feature vector of the scene image of the scene to be detected based on the feature vector extraction model.

According to an embodiment of the present invention, the image category determining unit is specifically configured to: and determining the image category corresponding to the extracted feature vector by adopting a clustering algorithm according to the extracted feature vector and the pre-established corresponding relation between the feature vector and the image category.

According to an embodiment of the present invention, the image category determining unit is specifically configured to: determining the distance between the extracted feature vector and each feature vector in the pre-established corresponding relation between the feature vector and the image category; determining K feature vectors closest to the extracted feature vectors as neighbor feature vectors, wherein K is a positive integer greater than 1; determining the image category corresponding to each adjacent feature vector according to the corresponding relation; and determining the image category with the highest proportion in the determined image categories as the image category corresponding to the extracted feature vector.

According to an embodiment of the invention, the image class determination unit is further configured to: and judging whether the proportion of the image category with the highest proportion in the determined image categories reaches a preset proportion, if so, determining the image category with the highest proportion in the determined image categories as the image category corresponding to the extracted feature vector.

According to an embodiment of the invention, the apparatus further comprises: a corresponding relation pre-establishing module for pre-establishing the corresponding relation between the characteristic vector and the image category; wherein the correspondence pre-establishing module is specifically configured to: acquiring scene images of various image categories as sample images in each scene to be detected; extracting a feature vector of each sample image; and establishing a corresponding relation between the feature vector of each sample image and the corresponding image category.

According to one embodiment of the invention, the target task comprises a plurality of action steps;

the scene to be detected comprises: a scenario before the smart device performs the first specified action step, and/or a scenario after the smart device performs the second specified action step.

In order to achieve the above object, an intelligent device according to an embodiment of the third aspect of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor executes the computer program to implement the abnormality detection method according to the embodiment of the first aspect of the present invention.

To achieve the above object, a non-transitory computer-readable storage medium according to a fourth embodiment of the present invention stores thereon a computer program, which when executed by a processor implements the abnormality detection method according to the first embodiment of the present invention.

Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.

Drawings

The above and/or additional aspects and advantages of the present invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which,

FIG. 1 is a flow diagram of an anomaly detection method according to one embodiment of the present invention;

FIG. 2 is a flow diagram of an anomaly detection method according to one embodiment of the present invention;

FIG. 3 is a flow diagram of generating an image class detection model according to an embodiment of the invention;

FIG. 4 is a flow diagram of an anomaly detection method according to another embodiment of the present invention;

FIG. 5 is a flow diagram of generating a sample feature vector set according to an embodiment of the invention;

fig. 6 is a schematic structural view of an abnormality detection apparatus according to an embodiment of the present invention;

FIG. 7 is a schematic diagram of an anomaly detection apparatus according to an embodiment of the present invention;

FIG. 8 is a schematic structural diagram of an abnormality detection apparatus according to another embodiment of the present invention;

FIG. 9 is a schematic structural diagram of an abnormality detection apparatus according to yet another embodiment of the present invention;

FIG. 10 is a schematic structural diagram of an abnormality detection apparatus according to still another embodiment of the present invention;

fig. 11 is a schematic structural diagram of a smart device according to an embodiment of the present invention.

Detailed Description

Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are illustrative and intended to be illustrative of the invention and are not to be construed as limiting the invention.

An abnormality detection method, apparatus, smart device, and computer-readable storage medium according to embodiments of the present invention are described below with reference to the accompanying drawings.

FIG. 1 is a flow diagram of an anomaly detection method according to one embodiment of the present invention. It should be noted that the abnormality detection method according to the embodiment of the present invention is applicable to the abnormality detection apparatus according to the embodiment of the present invention. Wherein, the abnormality detection device can be configured on the intelligent device. As an example, the smart device may include, but is not limited to, a robotic arm, such as also a smart robot in an anthropomorphic form.

As shown in fig. 1, the abnormality detection method may include:

s110, when the intelligent device executes the target task, a scene image of a scene to be detected is obtained, wherein the scene to be detected is a preset scene which corresponds to the target task and needs to be subjected to abnormal detection.

It will be appreciated that the smart device may provide services or tasks to the user, each of which will have a corresponding workflow. When the intelligent device provides a certain task, the certain task can be realized through a plurality of action steps. In this step, when the intelligent device executes a target task, it may be determined whether the current scene of the intelligent device is the scene to be detected first, in order to intelligently and automatically detect whether the current scene is abnormal or not when the intelligent device executes a certain task. Wherein, in an embodiment of the present invention, the target task may include a plurality of action steps. The scene to be detected may include a scene before the smart device performs the first specified action step and/or a scene after the smart device performs the second specified action step.

Optionally, in the process of executing the target task by the smart device, when it is determined that the smart device is about to execute a first specified action step, a scene image before the first specified action step is executed may be acquired; or when the intelligent device is determined to execute the second specified action step, acquiring a scene image after the second specified action step is executed; alternatively, during the process of executing the target task by the smart device, the scene image before the first specified action step is executed may be acquired, and the scene image after the second specified action step is executed may be acquired.

As an example, when it is determined that the current scene of the smart device is the scene to be detected, a scene image of the scene to be detected may be acquired through an image acquisition module in the smart device. For example, the scene to be detected can be shot by a camera in the intelligent device to obtain a scene image of the scene to be detected.

For example, when the intelligent device is used as a mechanical arm and the target task is coffee making, in the process of coffee making by the mechanical arm, and when the current scene is judged to be the scene to be detected, an image acquisition module (such as a camera) on the mechanical arm acquires an image of the scene to be detected, so as to obtain a scene image of the scene to be detected. For example, taking the example that the mechanical arm needs to cooperate with an automatic coffee machine and a milk cabinet to make espresso coffee, the mechanical arm for making the coffee comprises the following basic operation steps: grabbing a milk cylinder, placing the milk cylinder below a milk pipe of a milk cabinet, and pressing a button of the milk cabinet to discharge milk from the milk pipe; taking a coffee cup, and pressing a button of an automatic coffee machine to make espresso; pressing a button of the coffee maker to make milk foam; flower drawing; completing the coffee making the coffee cup is placed in a specific position. In the workflow of the mechanical arm to provide the espresso, assume that the scenario requiring anomaly detection is as follows: 1. whether a milk tank is arranged at the position to be grabbed or not; 2. whether the milk tube is expected to produce milk; 3. whether the coffee maker is working (e.g., can be judged according to the state of an indicator light) 4, and whether a cup is in a position to be grabbed. Thus, when the mechanical arm makes the Italian coffee, an image of a scene in front of the milk grabbing cylinder can be acquired; for another example, an image of a scene after the milk outlet button of the milk cabinet is operated can be obtained; as another example, an image of the scene after the coffee cup was grabbed may be obtained, and so on.

It should be noted that, in an embodiment of the present invention, the scene to be detected may also refer to a specific time. In the embodiment of the invention, when the intelligent device executes the target task, the scene image at the specified time can be acquired, wherein the specified time can be a preset scene corresponding to the target task and needing abnormality detection. That is, to achieve a certain target task, the action steps performed by the smart device in completing the target task are all defined with corresponding times. For example, a cup needs to be grabbed at the 20 th second, or water needs to be filled into the cup at the 30 th second, so that when the smart device executes the target task and determines that the current time is the designated time, the image of the scene corresponding to the designated time can be acquired.

And S120, according to the scene image of the scene to be detected, carrying out abnormity detection on the workflow of the target task executed by the intelligent equipment.

Optionally, when the scene image of the scene to be detected is obtained, the scene image can be predicted through a pre-established image type detection model so as to judge whether an abnormal condition exists in the scene to be detected; or, the scene situation type of the scene image can be judged through the combination of the general deep neural network and the clustering algorithm, and whether the execution scene has abnormal situations or not is judged according to the judgment result, so that the abnormal detection of the workflow of the target task executed by the intelligent equipment is realized. The specific implementation process can be referred to the description of the subsequent embodiments.

Optionally, in an embodiment of the present invention, when it is determined that the workflow of the intelligent device executing the target task has an abnormal condition, a prompt message may be generated, and the prompt message may be provided to the user. For example, the user may be alerted by flashing a light: the scene to be detected has abnormal conditions, or the prompt information can be provided for the user in a voice broadcasting mode so as to remind the user that the scene to be detected has abnormal conditions and the like. It is understood that while alerting the user, the next action step of the smart device may also be stopped, such as controlling the smart device to suspend execution of the action step to be executed. And when judging that the workflow of the intelligent device for executing the target task is not in an abnormal condition, controlling the intelligent device to continue executing the next action step.

FIG. 2 is a flow chart of an anomaly detection method according to one embodiment of the present invention.

In order to implement the function of detecting the abnormality of the workflow of the intelligent device, in the embodiment of the present invention, when the scene image of the scene to be detected is obtained, the scene image may be predicted through the image type detection model corresponding to the scene to be detected, so as to determine whether the scene to be detected has an abnormal condition. Specifically, as shown in fig. 2, the abnormality detection method may include:

s210, when the intelligent device executes the target task, a scene image of a scene to be detected is obtained, wherein the scene to be detected is a scene which is corresponding to the preset target task and needs to be subjected to abnormal detection.

Wherein, in an embodiment of the present invention, the target task may include a plurality of action steps. The scene to be detected may include a scene before the smart device performs the first specified action step and/or a scene after the smart device performs the second specified action step.

S220, obtaining an image type detection model corresponding to the scene to be detected.

It should be noted that, in the embodiment of the present invention, each scene to be detected may correspond to one image class detection model. Therefore, when the scene to be detected is determined and the scene image of the scene to be detected is obtained, the image type detection model corresponding to the scene to be detected can be obtained, so that the abnormal detection of the scene to be detected can be realized through the image type detection model in the following.

In the embodiment of the present invention, the image classification detection model may be a pre-trained neural network model for performing image classification; the image categories may include at least one normal category and at least one abnormal category. Optionally, which scenes the intelligent device has when executing the target task are used as the scenes to be detected may be obtained in advance, and which scene condition categories, such as abnormal conditions or normal conditions, each scene to be detected corresponds to are determined, so that, for each scene to be detected, scene images of various condition categories are used as sample data, and an image category detection model is trained independently for each scene to be detected, so that each scene to be detected corresponds to an image category detection model. As an example, as shown in fig. 3, the image category detection model may be generated specifically as follows:

s310, acquiring scene images of various image types as sample data for each scene to be detected;

and S320, training to obtain an image type detection model corresponding to each scene to be detected based on the sample data.

Optionally, after determining each scene to be detected, the scene images corresponding to various scene condition categories in each scene to be detected may be obtained, and the scene images are used as sample data, where the sample data may include the scene images corresponding to various image categories in each scene to be detected and the image categories (i.e., scene condition categories, such as normal conditions or abnormal conditions) corresponding to the scene images of each image category. And then, extracting the feature vector in the scene image corresponding to each image category, and training a classifier model based on the feature vector in the scene image corresponding to each image category and the image category corresponding to the scene image of each image category to obtain an image category detection model of each scene to be detected.

For example, taking the example that the mechanical arm needs to cooperate with an automatic coffee machine and a milk cabinet to make espresso coffee, the mechanical arm for making the coffee comprises the following basic operation steps: grabbing a milk cylinder, placing the milk cylinder below a milk pipe of a milk cabinet, and pressing a button of the milk cabinet to discharge milk from the milk pipe; taking a coffee cup, and pressing a button of an automatic coffee machine to make espresso; pressing a button of the coffee maker to make milk foam; flower drawing; completing the coffee making the coffee cup is placed in a specific position. In the workflow of the mechanical arm to provide the espresso, assume that the scenario requiring anomaly detection is as follows: 1. whether a milk tank is arranged at the position to be grabbed or not; 2. whether the milk tube is expected to produce milk; 3. whether the coffee maker is working (e.g., can be judged according to the state of an indicator light) 4, and whether a cup is in a position to be grabbed.

For the scenes needing to be subjected to the anomaly detection, different scene conditions may occur, and at this time, for each scene condition, a large number of sample images of each scene condition are respectively used for specially training the image type detection model of each scene needing to be subjected to the anomaly detection (namely, each scene to be detected). Therefore, in practical application, when the scene image of the scene to be detected is obtained, the image type detection model corresponding to the scene to be detected can be used for carrying out anomaly detection on the scene to be detected.

For example, as shown in table 1 below, three scenes that need to be subjected to anomaly detection (i.e., three scenes to be detected) are given, and each scene to be detected corresponds to a different scene condition, so that an image class detection model suitable for the scene that needs to be subjected to anomaly detection can be trained through sample image data of each scene condition, and thus three image class detection models can be trained.

Table 1

For example, taking the scene (a) to be detected in table 1 as an example, for the scene to be detected after the milk cabinet button is pressed, two different scene conditions can be provided under the scene to be detected, namely: the method comprises the steps that a milk tube milk A1 and a milk tube milk A2 are obtained, for the two scene conditions, a plurality of scene images corresponding to the scene condition of milk tube milk A1 can be obtained, a plurality of scene images corresponding to the scene condition of milk tube milk A2 can be obtained, feature vectors in the scene images under each scene condition are extracted, and a classifier model is trained on the basis of the feature vectors in the scene images under each scene condition and the corresponding scene condition classes thereof, so that an image class detection model corresponding to the scene to be detected after the milk cabinet button is pressed is obtained.

Therefore, for each scene to be detected, an image class detection model can be independently trained by using scene images of various conditions and classes as sample data, so that each scene to be detected corresponds to one image class detection model, and therefore, in subsequent practical application, the current scene to be detected can be subjected to anomaly detection through the image class detection model corresponding to the current scene to be detected.

And S230, judging whether the scene to be detected is abnormal or not based on the acquired image type detection model and the scene image of the scene to be detected.

Specifically, the obtained image category detection model can be used for performing classification prediction on the scene image of the scene to be detected so as to judge whether the scene to be detected is abnormal or not. Optionally, the feature vector of the scene image of the scene to be detected is extracted, the feature vector is classified and predicted through the obtained image class detection model, so as to obtain which image class the scene image belongs to, and whether the scene to be detected is abnormal or not is judged according to the image class.

For example, assuming that the scene to be detected is the scene (a) in table 1, the scene image of the scene to be detected can be classified and predicted by the image class detection model corresponding to the scene, and if it is predicted that the information fed back by the image is milk from the milk pipe, it can be determined that the scene to be detected is normal, and then the next action step can be performed; if the information fed back by the image is predicted to be that the milk pipe does not produce milk, the scene to be detected can be judged to be an abnormal situation, at the moment, the normal workflow of the intelligent equipment can be stopped, a corresponding abnormal processing flow is called, for example, the next action step is stopped, and abnormal alarm is carried out.

According to the anomaly detection method provided by the embodiment of the invention, when the scene image of the scene to be detected is obtained, the image type detection model corresponding to the scene to be detected can be obtained, and whether the scene to be detected is anomalous or not is judged based on the obtained image type detection model and the scene image of the scene to be detected, so that a model is trained for each scene condition, and in practical application, the anomaly detection is carried out on the current scene to be detected by obtaining the model corresponding to the current scene to be detected, thus the anomaly detection of the intelligent equipment workflow can be realized, the anomaly state can be found in time, and the accuracy of the workflow anomaly detection is greatly improved.

FIG. 4 is a flow chart of an anomaly detection method according to another embodiment of the present invention.

In order to realize the function of detecting the abnormality of the workflow of the intelligent device, in the embodiment of the invention, when the scene image of the scene to be detected is obtained, the combination of the general deep neural network and the clustering algorithm can be used for judging whether the scene to be detected is abnormal or not. Specifically, as shown in fig. 4, the abnormality detection method may include:

s410, when the intelligent device executes the target task, acquiring a scene image of a scene to be detected, wherein the scene to be detected is a scene which is corresponding to the preset target task and needs to be subjected to abnormal detection.

And S420, extracting the characteristic vector of the scene image of the scene to be detected.

Optionally, a feature vector extraction model may be obtained, and based on the feature vector extraction model, the feature vector of the scene image of the scene to be detected is extracted.

In the embodiment of the present invention, the feature vector extraction model may be a neural network model trained in advance for extracting the feature vector of the image. As an example, the feature vector extraction model may be embodied as a depth network classification model with the softmax output layer removed, which may be used for image classification, for example, the depth network classification model may be a convolutional neural network model. It can be understood that the convolutional neural network model can have good abstract extraction capability on image features, and can output corresponding feature vectors on an input image. For example, the method can be directly modified on the basis of the original convolutional neural network of the image classifier for multi-class classification, an output layer of the convolutional neural network is removed (label values without classification are not needed), and only the feature vectors of the model for performing multiple times of abstract extraction on various input samples are obtained to establish the feature vector extraction model. For another example, an image classifier for constructing a deep convolutional neural network model may use images with known mass labels (e.g., image data of mass cats, dogs, cars, etc.) on the existing network as training data, train a convergence model for classification, and then take off the last output layer, where the obtained model is the feature vector extraction model.

It should be noted that the obtaining method of the feature vector extraction model according to the embodiment of the present invention is not limited to the above obtaining based on the convolutional neural network model, and the feature vector extraction model according to the embodiment of the present invention may also be obtained by using other deep neural network models, for example, the feature vector extraction model is constructed by using an RNN model with an image classification function, a DNN model, and the like, and for example, the feature vector extraction model may be obtained by removing an output layer in the RNN model with the image classification function. The present invention does not specifically limit the manner of obtaining the feature vector extraction model.

S430, determining an image type corresponding to the extracted feature vector according to the extracted feature vector and a pre-established corresponding relation between the feature vector and the image type; wherein the image categories in the corresponding relationship include at least one normal category and at least one abnormal category.

Optionally, according to the extracted feature vector and a pre-established correspondence between the feature vector and the image category, determining the image category corresponding to the extracted feature vector by using a clustering algorithm. In the embodiment of the present invention, the clustering algorithm may be, but is not limited to, a K-nearest neighbor algorithm, a naive bayes algorithm, a decision tree algorithm, or an SVM algorithm, and the present invention is not limited in particular.

As an example, the image category corresponding to the extracted feature vector may be determined by using a K-nearest neighbor algorithm according to the extracted feature vector and a pre-established correspondence relationship between the feature vector and the image category. Specifically, the distance between the extracted feature vector and each feature vector in the pre-established correspondence relationship between the feature vector and the image category can be determined, and K feature vectors closest to the extracted feature vector are determined as neighbor feature vectors, wherein K is a positive integer greater than 1; then, according to the corresponding relationship, the image category corresponding to each neighboring feature vector can be determined, and the image category with the highest proportion in the determined image categories is determined as the image category corresponding to the extracted feature vector.

For example, it is assumed that the corresponding relationship includes six image categories, i.e., a1, a2, B1, B2, C1, and C2, where a1 and a2 are two different image categories under the scene to be detected (a) in table 1, B1 and B2 are two different image categories under the scene to be detected (B) in table 1, and C1 and C2 are two different image categories under the scene to be detected (C) in table 1, and the corresponding relationship includes a feature vector set corresponding to each image category, that is, each image category corresponds to one feature vector set including feature vectors that meet the current image category. In the process of executing a target task by the intelligent device, when it is necessary to judge whether an abnormality exists in a scene to be detected through image information, a scene image of the scene to be detected may be obtained first, the scene image is input into the feature vector extraction model, a corresponding feature vector S is obtained, distances between the feature vector S and each feature vector in the correspondence relationship are calculated, K feature vectors closest to the feature vector S are determined from feature vector sets corresponding to the six image categories of a1, a2, B1, B2, C1, and C2 according to the calculation results, and the K feature vectors closest to the feature vector S are used as neighboring feature vectors. Then, according to the corresponding relationship, the image category corresponding to each neighboring feature vector can be determined, and the image category with the highest proportion in the determined image categories is determined as the image category corresponding to the extracted feature vector. For example, a feature vector S of the scene image is obtained, 7 (i.e., K is 7) neighboring feature vectors are found from feature vector sets corresponding to two image categories of a1 and a2, and it is determined that 5 of the 7 neighboring feature vectors belong to an a1 set and 2 of the 7 neighboring feature vectors belong to an a2 set, that is, it is determined that 5 neighboring feature vectors correspond to the image category of a1 and 2 neighboring feature vectors correspond to the image category of a2, at this time, it is determined that the image category corresponding to the feature vector S belongs to an a1 category, that is, it is determined that the current scene to be detected belongs to the scene condition corresponding to the a1 category.

In order to improve the accuracy of determining the image categories, optionally, in an embodiment of the present invention, before determining the image category with the highest proportion in the determined image categories as the image category corresponding to the extracted feature vector, it may be further determined that the proportion of the image category with the highest proportion reaches a preset proportion. That is to say, after determining the image category corresponding to each neighboring feature vector according to the correspondence, it may be further determined whether the ratio of the image category with the highest ratio among the determined image categories reaches a preset ratio, and if so, the image category with the highest ratio among the determined image categories is determined as the image category corresponding to the extracted feature vector. As an example, the preset ratio may be 70%.

For example, taking the feature vector S of the scene image, finding 7 (i.e., K is 7) neighboring feature vectors from the feature vector sets corresponding to the two image categories of a1 and a2 as an example, determining that 5 of the 7 neighboring feature vectors belong to the a1 set and 2 of the 7 neighboring feature vectors belong to the a2 set, that is, determining that 5 neighboring feature vectors correspond to the image category of a1 and 2 neighboring feature vectors correspond to the image category of a2, at this time, it can be determined that the ratio of the image categories belonging to the a1 is the highest, and the ratio reaches 70%, at this time, it can be determined that the image category corresponding to the feature vector S is the a1 category, that is, it is determined that the current scene to be detected belongs to the scene situation corresponding to the a1 category.

It should be noted that the correspondence between the feature vectors and the image categories may be established by training a small number of sample images. Optionally, in an embodiment of the present invention, as shown in fig. 5, the correspondence between the feature vector and the image category may be pre-established specifically in the following manner:

s510, acquiring scene images of various image types as sample images in each scene to be detected;

s520, extracting a feature vector of each sample image;

s530, establishing a corresponding relation between the feature vector of each sample image and the corresponding image category.

Optionally, scene images corresponding to various scene condition categories (i.e., image categories) in each scene to be detected may be acquired, and these scene images are used as sample images, where each sample image corresponds to one image category (i.e., scene condition category, such as normal situation or abnormal situation). Then, the feature vector of each sample image can be extracted, and the corresponding relation between the feature vector of each sample image and the corresponding image category is established, so that the corresponding relation between the feature vector and the image category can be obtained.

For example, taking the example that the mechanical arm needs to cooperate with an automatic coffee machine and a milk cabinet to make espresso coffee, the mechanical arm for making the coffee comprises the following basic operation steps: grabbing a milk cylinder, placing the milk cylinder below a milk pipe of a milk cabinet, and pressing a button of the milk cabinet to discharge milk from the milk pipe; taking a coffee cup, and pressing a button of an automatic coffee machine to make espresso; pressing a button of the coffee maker to make milk foam; flower drawing; completing the coffee making the coffee cup is placed in a specific position. In the workflow of the mechanical arm to provide the espresso, assume that the scenario requiring anomaly detection is as follows: 1. whether a milk tank is arranged at the position to be grabbed or not; 2. whether the milk tube is expected to produce milk; 3. whether a cup is at the position to be grabbed.

For the scenes requiring anomaly detection, different scene conditions may occur, and it is assumed that each of the scenes requiring anomaly detection has two scene condition categories, at this time, n images of each scene condition category in the scene requiring anomaly detection are acquired respectively (where n may be set according to actual effects and empirical values, generally tens of images to tens of images, and the number is not too large), the images are input into the feature vector extraction model to obtain feature vectors of each scene condition category, in this way, the feature vectors of each scene condition category in each scene requiring anomaly detection can be trained, for example, the feature vectors corresponding to six image categories, i.e., a1, a2, B1, B2, C1, and C2, can be obtained as given in the example given in table 1 above, and establishing a corresponding relation between each feature vector and the corresponding image category, so as to obtain the corresponding relation between the feature vectors and the image categories.

Therefore, a small amount of images of respective scene conditions in each scene needing anomaly detection are respectively collected and input into the feature vector extraction model, feature vectors of the respective scene conditions are trained, and the corresponding relation between the feature vectors and the image types can be established based on the feature vectors of the respective scene conditions and the corresponding image types. Therefore, in practical application, based on the corresponding relationship between the feature vector and the image category, the feature vector of the scene image can be judged by using a clustering algorithm (such as a k-nearest neighbor algorithm) so as to realize the anomaly detection of the scene to be detected.

And S440, judging whether the scene to be detected is abnormal or not according to the determined image category.

Optionally, when the image category corresponding to the scene image is obtained, whether the scene to be detected is abnormal or not may be determined according to the image category. For example, taking the example given in table 1 above as an example, assuming that the image category corresponding to the scene image is the category a, it may be determined that the milk is normally output from the milk tube currently, that is, it is determined that the scene to be detected belongs to a normal scene condition, and at this time, the intelligent device may continue to make coffee according to the predetermined working setting.

According to the anomaly detection method of the embodiment of the invention, when a scene image of a scene to be detected is obtained, a feature vector of the scene image of the scene to be detected can be extracted, an image category corresponding to the extracted feature vector is determined according to the extracted feature vector and a pre-established corresponding relation between the feature vector and the image category, and whether the scene to be detected is abnormal or not is judged according to the determined image category, so that the feature vectors under different scene condition categories are trained by using a small amount of image data on the basis of a general deep neural network, the corresponding relation between the feature vector and the image category is established, in practical application, an unknown image is subjected to anomaly detection on the basis of a clustering algorithm and the pre-established corresponding relation between the feature vector and the image category, the anomaly detection of the workflow of an intelligent device can be realized, and the abnormal state can be found in time, the accuracy of workflow anomaly detection is greatly improved.

In accordance with the foregoing embodiments of the present invention, an embodiment of the present invention further provides an abnormality detection apparatus, and since the abnormality detection apparatus provided in the embodiment of the present invention corresponds to the foregoing embodiments of the present invention, the embodiments of the foregoing abnormality detection method are also applicable to the abnormality detection apparatus provided in the embodiment, and will not be described in detail in the embodiment. Fig. 6 is a schematic structural diagram of an abnormality detection apparatus according to an embodiment of the present invention. It should be noted that the abnormality detection apparatus according to the embodiment of the present invention may be configured on an intelligent device. As an example, the smart device may include, but is not limited to, a robotic arm, such as also a smart robot in an anthropomorphic form.

As shown in fig. 6, the abnormality detection apparatus 600 may include: an image acquisition module 610 and an anomaly detection module 620.

Specifically, the image obtaining module 610 is configured to obtain a scene image of a scene to be detected when the intelligent device executes a target task, where the scene to be detected is a scene that needs to be subjected to anomaly detection and corresponds to a preset target task.

As one example, the target task may include a plurality of action steps; the scene to be detected may include: a scenario before the smart device performs the first specified action step, and/or a scenario after the smart device performs the second specified action step.

The anomaly detection module 620 is configured to perform anomaly detection on the workflow of the target task executed by the intelligent device according to the scene image of the scene to be detected.

As an example implementation manner, each scene to be detected corresponds to an image class detection model, and the image class detection model is a pre-trained neural network model for image classification; the image categories include at least one normal category and at least one abnormal category. In this example, as shown in fig. 7, the anomaly detection module 620 may include: an acquisition unit 621 and a first abnormality detection unit 622. The obtaining unit 621 is configured to obtain an image category detection model corresponding to a scene to be detected; the first anomaly detection unit 622 is configured to determine whether the scene to be detected is anomalous based on the acquired image category detection model and the scene image of the scene to be detected.

Optionally, in an embodiment of the present invention, as shown in fig. 8, the anomaly detection apparatus 600 may further include a model pre-generation module 630, configured to pre-generate an image class detection model corresponding to each scene to be detected. In an embodiment of the present invention, the model pre-generation module 630 may be specifically configured to: acquiring scene images of various image categories as sample data for each scene to be detected; and training to obtain an image type detection model corresponding to each scene to be detected based on the sample data. Therefore, the abnormal detection is carried out on the current scene to be detected by acquiring the model corresponding to the current scene to be detected, the abnormal detection of the workflow of the intelligent equipment can be realized, the abnormal state can be found in time, and meanwhile, the accuracy of the abnormal detection of the workflow is greatly improved.

As another example implementation manner, as shown in fig. 9, the anomaly detection module 620 may include: a feature vector extraction unit 623, an image category determination unit 624, and a second abnormality detection unit 625. The feature vector extraction unit 623 is configured to extract a feature vector of a scene image of the scene to be detected; the image category determining unit 624 is configured to determine an image category corresponding to the extracted feature vector according to the extracted feature vector and a pre-established correspondence between the feature vector and the image category; the image categories in the corresponding relation comprise at least one normal category and at least one abnormal category; the second anomaly detection unit 625 is configured to determine whether the scene to be detected is anomalous according to the determined image category.

Alternatively, in this example, the feature vector extraction unit 623 may acquire a feature vector extraction model; the feature vector extraction model is a pre-trained neural network model used for extracting image feature vectors, and the feature vectors of the scene images of the scene to be detected are extracted based on the feature vector extraction model. In the embodiment of the present invention, the feature vector extraction model is specifically a deep network classification model with a softmax output layer removed, and the deep network classification model can be used for image classification.

As an example, the image category determining unit 624 is specifically configured to: and determining the image category corresponding to the extracted feature vector by adopting a clustering algorithm according to the extracted feature vector and the pre-established corresponding relation between the feature vector and the image category. Specifically, the image category determining unit 624 may determine a distance between the extracted feature vector and each feature vector in a correspondence relationship between the feature vectors and image categories established in advance, determine K feature vectors closest to the extracted feature vectors as neighboring feature vectors, where K is a positive integer greater than 1, then determine an image category corresponding to each neighboring feature vector according to the correspondence relationship, and determine an image category corresponding to the highest proportion of the determined image categories as an image category corresponding to the extracted feature vectors.

In order to improve the accuracy of determining the image category, optionally, in an embodiment of the present invention, the image category determining unit 624 is further configured to: and judging whether the proportion of the image category with the highest proportion in the determined image categories reaches a preset proportion, if so, determining the image category with the highest proportion in the determined image categories as the image category corresponding to the extracted feature vector.

Optionally, in an embodiment of the present invention, as shown in fig. 10, the abnormality detecting apparatus 600 may further include: a corresponding relationship pre-establishing module 640, configured to pre-establish a corresponding relationship between the feature vector and the image category. In the embodiment of the present invention, the correspondence relationship pre-establishing module 640640 may be specifically configured to: acquiring scene images of various image categories as sample images in each scene to be detected; extracting a feature vector of each sample image; and establishing a corresponding relation between the feature vector of each sample image and the corresponding image category. Therefore, in practical application, the abnormal detection is carried out on the unknown picture based on the clustering algorithm and the pre-established corresponding relation between the characteristic vector and the image category, the abnormal detection of the workflow of the intelligent equipment can be realized, the abnormal state can be found in time, and the accuracy of the abnormal detection of the workflow is greatly improved.

According to the anomaly detection device provided by the embodiment of the invention, the scene image of the scene to be detected can be acquired by the acquisition module when the intelligent equipment executes the target task, and the detection module carries out anomaly detection on the workflow of the target task executed by the intelligent equipment according to the scene image of the scene to be detected. In other words, in the workflow of executing a certain task by the intelligent device, when information fed back by an interaction scene of an environment or other things (such as equipment or tools) needs to be confirmed, an image under the scene can be acquired, and whether the scene is abnormal or not is judged by analyzing the image, so that the information fed back by the interaction scene of the external environment or other things is sensed by using the visual sensor and is combined with the workflow set by the self program, the purpose of providing effective product service is achieved, the intelligent device has the function of detecting the abnormality of the workflow of the intelligent device, the whole process does not need artificial detection, the labor cost is greatly reduced, and the intelligent device is ensured to be capable of providing effective service.

In order to realize the embodiment, the invention further provides the intelligent device.

Fig. 11 is a schematic structural diagram of a smart device according to an embodiment of the present invention. It should be noted that, in the embodiment of the present invention, the smart device may be a robot arm. As shown in fig. 11, the smart device 1100 may include: a memory 1110, a processor 1120, and a computer program 1130 stored in the memory 1110 and operable on the processor 1120, wherein the processor 1120 executes the program 1130 to implement the method for detecting an abnormality according to any one of the above embodiments of the present invention.

In order to implement the above embodiments, the present invention also proposes a non-transitory computer-readable storage medium on which a computer program is stored, the program, when executed by a processor, implementing the anomaly detection method according to any one of the above embodiments of the present invention.

In the description of the present invention, it is to be understood that the terms "first", "second" and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implying any number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.

In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.

Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and alternate implementations are included within the scope of the preferred embodiment of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present invention.

The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.

It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.

It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable storage medium, and when the program is executed, the program includes one or a combination of the steps of the method embodiments.

In addition, functional units in the embodiments of the present invention may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium.

The storage medium mentioned above may be a read-only memory, a magnetic or optical disk, etc. Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and that variations, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present invention.

Claims

1. An abnormality detection method characterized by comprising:

when a mechanical arm executes a target task, acquiring a scene image of a scene to be detected, wherein the target task comprises a plurality of action steps, and the scene to be detected comprises: a scenario related to the first specified action before the robotic arm performs the first specified action step, and/or a scenario related to the second specified action after the robotic arm performs the second specified action step;

according to the scene image of the scene to be detected, carrying out abnormity detection on the workflow of the mechanical arm executing the target task;

the abnormal detection of the workflow of the mechanical arm executing the target task according to the scene image of the scene to be detected comprises the following steps: extracting a characteristic vector of a scene image of the scene to be detected; determining the image category corresponding to the extracted feature vector according to the extracted feature vector and the pre-established corresponding relation between the feature vector and the image category; the image categories in the corresponding relation comprise at least one normal category and at least one abnormal category; judging whether the scene to be detected is abnormal or not according to the determined image category;

or each scene to be detected corresponds to an image category detection model, and the image category detection model is a pre-trained neural network model for image classification; the image categories include at least one normal category and at least one abnormal category; the abnormal detection of the workflow of the mechanical arm executing the target task according to the scene image of the scene to be detected comprises the following steps:

2. The method of claim 1, wherein the image class detection model is generated by:

acquiring scene images of various image categories as sample data for each scene to be detected;

and training to obtain an image type detection model corresponding to each scene to be detected based on the sample data.

3. The method of claim 1, wherein extracting feature vectors of a scene image of the scene to be detected comprises:

acquiring a feature vector extraction model; the feature vector extraction model is a pre-trained neural network model used for extracting image feature vectors;

and extracting the feature vector of the scene image of the scene to be detected based on the feature vector extraction model.

4. The method according to claim 1, wherein determining the image category corresponding to the extracted feature vector according to the extracted feature vector and a pre-established correspondence between the feature vector and the image category specifically comprises:

and determining the image category corresponding to the extracted feature vector by adopting a clustering algorithm according to the extracted feature vector and the pre-established corresponding relation between the feature vector and the image category.

5. The method according to claim 4, wherein determining the image category corresponding to the extracted feature vector by using a clustering algorithm according to the extracted feature vector and a pre-established correspondence between the feature vector and the image category specifically comprises:

determining the distance between the extracted feature vector and each feature vector in the pre-established corresponding relation between the feature vector and the image category;

determining K feature vectors closest to the extracted feature vectors as neighbor feature vectors, wherein K is a positive integer greater than 1;

determining the image category corresponding to each adjacent feature vector according to the corresponding relation;

and determining the image category with the highest proportion in the determined image categories as the image category corresponding to the extracted feature vector.

6. The method of claim 5, further comprising, before determining a highest proportion of the determined image classes as the image class to which the extracted feature vector corresponds:

and determining that the proportion of the image category with the highest proportion reaches a preset proportion.

7. The method of claim 1, wherein the correspondence between the feature vectors and the image categories is established by:

acquiring scene images of various image categories as sample images in each scene to be detected;

extracting a feature vector of each sample image;

and establishing a corresponding relation between the feature vector of each sample image and the corresponding image category.

8. An abnormality detection device characterized by comprising:

the image acquisition module is used for acquiring a scene image of a scene to be detected when the mechanical arm executes a target task, wherein the target task comprises a plurality of action steps, and the scene to be detected comprises: a scenario related to the first specified action before the robotic arm performs the first specified action step, and/or a scenario related to the second specified action after the robotic arm performs the second specified action step;

the anomaly detection module is used for carrying out anomaly detection on the workflow of the mechanical arm executing the target task according to the scene image of the scene to be detected;

wherein the anomaly detection module comprises: the feature vector extraction unit is used for extracting feature vectors of the scene images of the scene to be detected; the image type determining unit is used for determining the image type corresponding to the extracted feature vector according to the extracted feature vector and the corresponding relation between the pre-established feature vector and the image type; the image categories in the corresponding relation comprise at least one normal category and at least one abnormal category; the second anomaly detection unit is used for judging whether the scene to be detected is anomalous or not according to the determined image category;

or each scene to be detected corresponds to an image category detection model, and the image category detection model is a pre-trained neural network model for image classification; the image categories include at least one normal category and at least one abnormal category; wherein the anomaly detection module comprises:

the acquisition unit is used for acquiring an image type detection model corresponding to the scene to be detected; and the first anomaly detection unit is used for judging whether the scene to be detected is anomalous or not based on the acquired image type detection model and the scene image of the scene to be detected.

9. The apparatus of claim 8, wherein the apparatus further comprises:

the model pre-generation module is used for pre-generating an image category detection model corresponding to each scene to be detected;

wherein the model pre-generation module is specifically configured to:

10. The apparatus of claim 8, wherein the feature vector extraction unit is specifically configured to:

11. The apparatus of claim 8, wherein the image category determination unit is specifically configured to:

12. The apparatus according to claim 11, wherein the image class determination unit is specifically configured to:

13. The apparatus of claim 12, wherein the image class determination unit is further to: and judging whether the proportion of the image category with the highest proportion in the determined image categories reaches a preset proportion, if so, determining the image category with the highest proportion in the determined image categories as the image category corresponding to the extracted feature vector.

14. The apparatus of claim 8, wherein the apparatus further comprises:

a corresponding relation pre-establishing module for pre-establishing the corresponding relation between the characteristic vector and the image category;

wherein the correspondence pre-establishing module is specifically configured to:

extracting a feature vector of each sample image;

15. A robot comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of detecting an abnormality of any one of claims 1 to 7 when executing the program.

16. A non-transitory computer-readable storage medium on which a computer program is stored, the program, when executed by a processor, implementing the anomaly detection method according to any one of claims 1 to 7.