WO2021147325A1 - 一种物体检测方法、装置以及存储介质 - Google Patents

一种物体检测方法、装置以及存储介质 Download PDF

Info

Publication number
WO2021147325A1
WO2021147325A1 PCT/CN2020/112796 CN2020112796W WO2021147325A1 WO 2021147325 A1 WO2021147325 A1 WO 2021147325A1 CN 2020112796 W CN2020112796 W CN 2020112796W WO 2021147325 A1 WO2021147325 A1 WO 2021147325A1
Authority
WO
WIPO (PCT)
Prior art keywords
detected
image
different domains
relationship
training
Prior art date
Application number
PCT/CN2020/112796
Other languages
English (en)
French (fr)
Inventor
徐航
周峰暐
黎嘉伟
梁小丹
李震国
钱莉
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2021147325A1 publication Critical patent/WO2021147325A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/36—Creation of semantic tools, e.g. ontology or thesauri
    • G06F16/367—Ontology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
    • G06F18/24133—Distances to prototypes
    • G06F18/24137—Distances to cluster centroïds
    • G06F18/2414—Smoothing the distance, e.g. radial basis function networks [RBFN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/35—Categorising the entire scene, e.g. birthday party or wedding scene

Definitions

  • This application relates to the field of computer vision, and in particular to an object detection method, device and storage medium.
  • Artificial intelligence is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results.
  • artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a similar way to human intelligence.
  • Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.
  • Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and basic AI theories.
  • Object detection is a basic computer vision task that can identify the location and category of objects in an image.
  • researchers and engineers will create data sets for different specific problems according to the application scenarios and actual task requirements, and use them to train highly customized and unique automatic object detectors.
  • Object detection across data sets is an efficient method to achieve large-scale object detection.
  • the existing multi-task learning only handles multiple tasks at the same time by adding multiple branches to the model, and cannot realize the interaction between different data sets and different object categories, and cannot capture the internal relationship between the objects to be detected in different data sets. , So the effect is not good.
  • the present application provides an object detection method, device, and computer storage medium to improve the effect of object detection.
  • the first aspect of the present application provides an object detection method, which may include: acquiring an image to be detected. Determine the initial image characteristics of the object to be detected in the image to be detected. Determine the enhanced image feature of the object to be detected according to the cross-domain knowledge map information.
  • the cross-domain knowledge map information can include the association relationship between the object categories corresponding to the object to be detected in different domains, and the enhanced image feature indicates that the object in different domains is related to the object to be detected.
  • the semantic information of the object category corresponding to the other objects in the link According to the initial image feature of the object to be detected and the enhanced image feature of the object to be detected, the candidate frame and classification of the object to be detected are determined.
  • the above object detection method can be applied in different application scenarios.
  • the above object detection method can be applied in the scene of recognizing everything, and it can also be applied in the scene of street view recognition.
  • the above-mentioned image to be detected may be an image taken by the mobile terminal through a camera, or an image already stored in the mobile terminal's album.
  • the above-mentioned image to be detected may be a street view image taken by a camera on the roadside.
  • the object categories in the first domain or the first data set include men, women, boys, girls, roads, and streets.
  • Object categories in the second domain include people, handbags, school bags, cars, and trucks. It can be considered that the men, women, boys, and girls in the first domain have an association relationship with the people in the second domain. The women and girls in the first domain have an association with the handbags in the second domain. There is an association between roads and streets in the first domain and cars and trucks in the second domain.
  • Semantic information can refer to high-level information that can assist in image detection.
  • the above-mentioned semantic information can specifically be what the object is and what is around the object (semantic information is generally different from low-level information, such as image edges, pixels, brightness, etc.).
  • the object to be detected is a woman, and other objects associated with the bicycle in the image to be detected include a handbag, then the enhanced image feature of the object to be detected may indicate semantic information of the handbag.
  • the solution provided by this application can effectively use a large number of different data sets and different types of information to train the same network at the same time, which greatly improves the data utilization rate and makes the detection performance higher.
  • the cross-domain knowledge graph may include nodes and node edges, nodes corresponding to objects to be detected, and node edges corresponding to high-level semantic features of different objects to be detected
  • the method may also include: obtaining classification layer parameters corresponding to different domains. According to the classification weights of the initial image features in different domains on different object categories, the classification layer parameters corresponding to different domains are weighted and merged to obtain the high-level semantic features of the object to be detected. The weight of the relationship between the object categories corresponding to the object to be detected in different domains is projected onto the node connection edge of the object to be detected, and the weight of the node connection edge is obtained.
  • the method may further include: determining the relationship according to the distance relationship between the object categories corresponding to the objects to be detected in different domains Weights.
  • the distance relationship between object categories corresponding to the object to be detected may include one or more of the following information: Attribute relationships between object categories corresponding to objects to be detected in different domains. The positional relationship or active-object relationship between object categories corresponding to objects to be detected in different domains. The similarity of word embeddings between the object categories corresponding to the objects to be detected in different domains is constructed using linguistic knowledge. The distance relationship between the object categories corresponding to the objects to be detected in different domains is obtained by training the neural network model according to the training data.
  • the enhanced image feature of the object to be detected is determined according to the cross-domain knowledge map information, and Including: performing convolution processing on the high-level semantic features according to the weights of the edges of the nodes to obtain the enhanced image features of the object to be detected.
  • a second aspect of the present application provides an image detection device, which may include: an image acquisition module for acquiring an image to be detected.
  • the feature extraction module is used to determine the initial image feature of the object to be detected in the image to be detected.
  • the feature extraction module is also used to determine the enhanced image features of the object to be detected according to the cross-domain knowledge map information.
  • the cross-domain knowledge map information can include the association relationship between the object categories corresponding to the object to be detected in different domains, and the enhanced image feature indicates different Semantic information of object categories corresponding to other objects in the domain associated with the object to be detected.
  • the detection module is used to determine the candidate frame and classification of the object to be detected according to the initial image feature of the object to be detected and the enhanced image feature of the object to be detected.
  • the cross-domain knowledge graph may include nodes and node edges, nodes corresponding to objects to be detected, and node edges corresponding to high-level semantic features of different objects to be detected
  • the image detection device may also include a parameter acquisition module and a projection module.
  • the parameter acquisition module is used to acquire classification layer parameters corresponding to different domains.
  • the feature extraction module is specifically used to weight and fuse the classification layer parameters corresponding to different domains according to the classification weights of the initial image features in different domains on different object categories to obtain the high-level semantic features of the object to be detected.
  • the projection module is used to project the weights of the relationships between the object categories corresponding to the objects to be detected in different domains onto the edges of the nodes of the objects to be detected to obtain the weights of the edges of the nodes.
  • the second possible implementation may also include a relationship weight determination module, a relationship weight determination module, configured to correspond to objects to be detected in different domains. The distance relationship between the object categories determines the relationship weight.
  • the distance relationship between object categories corresponding to the object to be detected may include one or more of the following information: Attribute relationships between object categories corresponding to objects to be detected in different domains. The positional relationship or active-object relationship between object categories corresponding to objects to be detected in different domains. The similarity of word embeddings between the object categories corresponding to the objects to be detected in different domains is constructed using linguistic knowledge. The distance relationship between the object categories corresponding to the objects to be detected in different domains is obtained by training the neural network model according to the training data.
  • the feature extraction module is specifically used to compare the high-level semantics according to the weight of the node connection
  • the features are processed by convolution to obtain the enhanced image features of the object to be detected.
  • the third aspect of the present application provides a neural network training method.
  • the method includes: acquiring training data, the training data including training images and object detection and labeling results of the objects to be detected in the training images; extracting the training images from the neural network The initial image features of the object to be detected; the enhanced image features of the object to be detected in the training image are extracted according to the neural network and the cross-domain knowledge map information; the initial image features and the enhanced image features of the object to be detected are processed according to the neural network , Obtain the object detection result of the object to be detected; determine the model parameters of the neural network according to the object detection result of the object to be detected in the training image and the object detection label result of the object to be detected in the training image.
  • the cross-domain knowledge map information may include the association relationship between the object categories corresponding to the object to be detected in different domains, and the enhanced image feature indicates semantic information of the object category corresponding to other objects in the different domains associated with the object to be detected.
  • the object detection and labeling result of the object to be detected in the training image includes the labeling candidate frame and labeling classification result of the object to be detected in the training image.
  • a set of initial model parameters can be set for the neural network, and then based on the object detection result of the object to be detected in the training image and the object detection labeling result of the object to be detected in the training image.
  • the neural network obtained through training in the third aspect can be used to implement the method in the first aspect of the present application.
  • the cross-domain knowledge graph may include nodes and node edges, where nodes correspond to objects to be detected, and node edges correspond to the high-level semantics of different objects to be detected
  • the relationship between features According to the classification weights of the initial image features in different domains on different object categories, the classification layer parameters corresponding to different domains are weighted and merged to obtain the high-level semantic features of the object to be detected.
  • the classification layer parameters can be understood as maintaining a class of the category. center.
  • the weight of the relationship between the object categories corresponding to the object to be detected in different domains is projected onto the node connection edge of the object to be detected, and the weight of the node connection edge is obtained.
  • the second possible implementation manner may further include determining the relationship weight according to the distance relationship between the object categories corresponding to the objects to be detected in different domains.
  • the distance relationship between the object categories corresponding to the object to be detected may include one or more of the following information: Attribute relationships between object categories corresponding to objects to be detected in different domains. The positional relationship or active-object relationship between object categories corresponding to objects to be detected in different domains. The similarity of word embeddings between the object categories corresponding to the objects to be detected in different domains is constructed using linguistic knowledge. The distance relationship between the object categories corresponding to the objects to be detected in different domains is obtained by training the neural network model according to the training data.
  • the high-level semantic features are convolved according to the weights of the node edges to obtain Enhanced image characteristics of the object to be detected.
  • an object detection device in a fourth aspect, includes modules for executing the method in the first aspect.
  • a neural network training device in a fifth aspect, includes various modules for executing the method in the third aspect.
  • an object detection device in a sixth aspect, includes: a memory for storing a program; a processor for executing the program stored in the memory, and when the program stored in the memory is executed, the processing The device is used to perform the method in the first aspect described above.
  • a neural network training device in a seventh aspect, includes: a memory for storing a program; a processor for executing the program stored in the memory, and when the program stored in the memory is executed, the device The processor is used to execute the method in the third aspect described above.
  • an electronic device which includes the object detection device in the fourth aspect or the sixth aspect.
  • an electronic device in a ninth aspect, includes the object detection device in the fifth aspect or the seventh aspect.
  • the above-mentioned electronic device may specifically be a mobile terminal (for example, a smart phone), a tablet computer, a notebook computer, an augmented reality/virtual reality device, a vehicle-mounted terminal device, and so on.
  • a mobile terminal for example, a smart phone
  • a tablet computer for example, a tablet computer
  • a notebook computer for example, a tablet computer
  • an augmented reality/virtual reality device for example, a vehicle-mounted terminal device, and so on.
  • a computer storage medium stores program code, and the program code includes instructions for executing the steps in the method in the first aspect or the third aspect.
  • a computer program product containing instructions is provided, when the computer program product runs on a computer, the computer executes the method in the first aspect or the third aspect.
  • a chip in a twelfth aspect, includes a processor and a data interface.
  • the processor reads instructions stored in a memory through the data interface and executes the method in the first aspect or the third aspect.
  • the chip may further include a memory in which instructions are stored, and the processor is configured to execute instructions stored on the memory.
  • the processor is used to execute the method in the first aspect.
  • the above-mentioned chip may specifically be a field programmable gate array FPGA or an application-specific integrated circuit ASIC.
  • the above-mentioned method of the first aspect may specifically refer to the first aspect and a method in any one of the various implementation manners of the first aspect.
  • the foregoing method of the third aspect may specifically refer to the third aspect and a method in any one of the various implementation manners of the third aspect.
  • a cross-domain knowledge graph is constructed, which can capture the intrinsic relationship between different objects to be detected, and the enhanced image features include the semantics of object categories corresponding to other objects in different domains associated with the object to be detected Information, so this application can improve the effect of the object detection method.
  • FIG. 1 is a schematic structural diagram of a system architecture provided by an embodiment of the present application.
  • FIG. 2 is a schematic diagram of object detection using a convolutional neural network model provided by an embodiment of the present application
  • FIG. 3 is a schematic diagram of a chip hardware structure provided by an embodiment of the present application.
  • FIG. 4 is a schematic flowchart of an object detection method according to an embodiment of the present application.
  • FIG. 5 is a schematic diagram of the association relationship of an embodiment of the present application.
  • Fig. 6 is a flowchart of an object detection method according to an embodiment of the present application.
  • FIG. 7 is a flowchart of an object detection method according to an embodiment of the present application.
  • FIG. 8 is a schematic flowchart of a neural network training method according to an embodiment of the present application.
  • FIG. 9 is a schematic block diagram of an object detection device according to an embodiment of the present application.
  • FIG. 10 is a schematic block diagram of an object detection device according to an embodiment of the present application.
  • Fig. 11 is a schematic block diagram of a neural network training device according to an embodiment of the present application.
  • the embodiments of this application are mainly applied in scenes of large-scale object detection, such as mobile phone face recognition, mobile phone recognition of everything, the perception system of unmanned vehicles, security cameras, photo object recognition on social networking sites, smart robots, and so on.
  • object detection such as mobile phone face recognition, mobile phone recognition of everything, the perception system of unmanned vehicles, security cameras, photo object recognition on social networking sites, smart robots, and so on.
  • the object detection method of the embodiment of the application can be used to detect objects in the pictures taken by the mobile phone. Since the object detection method of the embodiment of the application combines the cross-domain knowledge graph when detecting objects, the method of the embodiment of the application is adopted The object detection method performs better object detection on the pictures taken by the mobile phone (for example, the position of the object and the classification of the object are more accurate).
  • Cameras deployed on the street can take pictures of passing vehicles and people. After the pictures are obtained, the pictures can be uploaded to the control center equipment, and the control center equipment can perform object detection on the pictures and obtain the object detection results. When abnormalities occur, the pictures can be uploaded to the control center equipment. The control center can send out an alarm when the object is missing.
  • the neural network training method provided in the embodiments of this application involves computer vision processing, and can be specifically applied to data processing methods such as data training, machine learning, and deep learning. Labeling results) Carry out symbolic and formal intelligent information modeling, extraction, preprocessing, training, etc., and finally get a trained neural network.
  • the object detection method provided by the embodiments of this application can use the above-mentioned trained neural network to input input data (such as the picture in this application) into the trained neural network to obtain output data (such as the picture in this application). Test results).
  • the neural network training method provided in the embodiments of this application and the object detection method in the embodiments of this application are inventions based on the same concept, and can also be understood as two parts in a system, or an overall process Two stages: such as model training stage and model application stage.
  • a neural network can be composed of neural units.
  • a neural unit can refer to an arithmetic unit that takes xs and intercept 1 as inputs.
  • the output of the arithmetic unit can be:
  • s 1, 2,...n, n is a natural number greater than 1
  • Ws is the weight of xs
  • b is the bias of the neural unit.
  • f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal.
  • the output signal of the activation function can be used as the input of the next convolutional layer.
  • the activation function can be a sigmoid function.
  • a neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit.
  • the input of each neural unit can be connected with the local receptive field of the previous layer to extract the characteristics of the local receptive field.
  • the local receptive field can be a region composed of several neural units.
  • Deep neural network can be understood as a neural network with many hidden layers. There is no special metric for "many” here. The essence of the multi-layer neural network and deep neural network we often say The above is the same thing. From the division of DNN according to the location of different layers, the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the number of layers in the middle are all hidden layers. The layers are fully connected, that is to say, any neuron in the i-th layer must be connected to any neuron in the i+1th layer. Although DNN looks very complicated, it is not complicated in terms of the work of each layer.
  • the linear coefficients from the fourth neuron in the second layer to the second neuron in the third layer are defined as
  • the superscript 3 represents the number of layers where the coefficient W is located, and the subscript corresponds to the output third-level index 2 and the input second-level index 4.
  • the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as Note that the input layer has no W parameter.
  • more hidden layers make the network more capable of portraying complex situations in the real world.
  • a model with more parameters is more complex and has a greater "capacity", which means that it can complete more complex learning tasks.
  • Convolutional neural network (convolutional neuron network, CNN) is a deep neural network with a convolutional structure.
  • the convolutional neural network contains a feature extractor composed of a convolutional layer and a sub-sampling layer.
  • the feature extractor can be regarded as a filter.
  • the convolutional layer refers to the neuron layer that performs convolution processing on the input signal in the convolutional neural network.
  • a neuron can be connected to only part of the neighboring neurons.
  • a convolutional layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. Neural units in the same feature plane share weights, and the shared weights here are the convolution kernels.
  • Sharing weight can be understood as the way of extracting image information has nothing to do with location.
  • the convolution kernel can be initialized in the form of a matrix of random size. In the training process of the convolutional neural network, the convolution kernel can obtain reasonable weights through learning. In addition, the direct benefit of sharing weights is to reduce the connections between the layers of the convolutional neural network, and at the same time reduce the risk of overfitting.
  • the classifier is generally composed of a fully connected layer and a softmax function, which can output probabilities of different categories according to the input.
  • FPN is based on the original detector to independently predict in different feature layers.
  • the original intention of migration learning is to deal with the problem of insufficient training samples, so that the model can use the existing source domain data (source domain data) to migrate to related but not identical target domain data (target domain data), thereby training suitable for the target
  • the model of the domain is an abstract concept that refers to tasks with similar properties. Specifically, a domain (or domain) can be a detection task on a specific data set, or it can refer to a detection task for a specific object (such as a human face), and so on. There are often obvious differences between different domains, which are difficult to deal with in a unified manner.
  • the global domain refers to the collective name of all domains including all potential tasks. It is the complete set of domains and is generally used for definitions and conceptual expressions.
  • the core algorithm of transfer learning is to extract domain-invariant information by maximizing a specific domain similarity measure, so that data in different domains can learn from each other to obtain a model suitable for the target domain.
  • a graph is a data format that can be used to represent social networks, communication networks, protein molecular networks, etc.
  • the nodes in the graph represent individuals in the network, and the lines represent the connections between individuals.
  • Many machine learning tasks such as community discovery, link prediction, etc. require graph structure data. Therefore, the emergence of graph convolutional neural networks (GCN) provides new ideas for solving these problems.
  • GCN can be used for deep learning of graph data.
  • GCN is a natural promotion of convolutional neural networks in the graph domain. It can perform end-to-end learning of node feature information and structural information at the same time, and is currently the best choice for graph data learning tasks.
  • the applicability of GCN is extremely wide, and it is suitable for nodes and graphs of any topology.
  • Fig. 1 is a schematic diagram of the system architecture of an embodiment of the present application.
  • the system architecture 100 includes an execution device 110, a training device 120, a database 130, a client device 140, a data storage system 150, and a data collection system 160.
  • the execution device 110 includes a calculation module 111, an I/O interface 112, a preprocessing module 113, and a preprocessing module 114.
  • the calculation module 111 may include the target model/rule 101, and the preprocessing module 113 and the preprocessing module 114 are optional.
  • the data collection device 160 is used to collect training data.
  • the training data may include training images of different domains or different data sets and the annotation results corresponding to the training images.
  • the labeling result of the training image may be the (manually) pre-labeled classification result of each object to be detected in the training image.
  • the data collection device 160 stores the training data in the database 130, and the training device 120 trains to obtain the target model/rule 101 based on the training data maintained in the database 130.
  • the training device 120 performs object detection on the input training image, and compares the output detection result with the object pre-labeled detection result, until the training device 120 outputs The difference between the detection result of the object and the pre-labeled detection result is less than a certain threshold, thereby completing the training of the target model/rule 101.
  • the above-mentioned target model/rule 101 can be used to implement the object detection method of the embodiment of the present application, that is, input the image to be detected (after relevant preprocessing) into the target model/rule 101 to obtain the detection result of the image to be detected.
  • the target model/rule 101 in the embodiment of the present application may specifically be a neural network.
  • the training data maintained in the database 130 may not all come from the collection of the data collection device 160, and may also be received from other devices.
  • the training device 120 does not necessarily perform the training of the target model/rule 101 completely based on the training data maintained by the database 130. It may also obtain training data from the cloud or other places for model training.
  • the above description should not be used as a reference to this application. Limitations of the embodiment.
  • the target model/rule 101 trained according to the training device 120 can be applied to different systems or devices, such as the execution device 110 shown in FIG. 1, which can be a terminal, such as a mobile phone terminal, a tablet computer, notebook computers, augmented reality (AR)/virtual reality (VR), vehicle-mounted terminals, etc., can also be servers or clouds.
  • the execution device 110 is configured with an input/output (input/output, I/O) interface 112 for data interaction with external devices.
  • the user can input data to the I/O interface 112 through the client device 140.
  • the input data in this embodiment of the present application may include: a to-be-processed image input by the client device.
  • the client device 140 here may specifically be a terminal device.
  • the preprocessing module 113 and the preprocessing module 114 are used to perform preprocessing according to the input data (such as the image to be processed) received by the I/O interface 112.
  • the preprocessing module 113 and the preprocessing module may not be provided.
  • 114 there may only be one preprocessing module, and the calculation module 111 is directly used to process the input data.
  • the execution device 110 may call data, codes, etc. in the data storage system 150 for corresponding processing .
  • the data, instructions, etc. obtained by corresponding processing may also be stored in the data storage system 150.
  • the I/O interface 112 presents the processing result, such as the detection result of the object obtained above, to the client device 140 to provide it to the user.
  • the training device 120 can generate corresponding target models/rules 101 based on different training data for different goals or tasks, and the corresponding target models/rules 101 can be used to achieve the above goals or complete The above tasks provide users with the desired results.
  • the user can manually set input data, and the manual setting can be operated through the interface provided by the I/O interface 112.
  • the client device 140 can automatically send input data to the I/O interface 112. If the client device 140 is required to automatically send the input data and the user's authorization is required, the user can set the corresponding authority in the client device 140.
  • the user can view the result output by the execution device 110 on the client device 140, and the specific presentation form may be a specific manner such as display, sound, and action.
  • the client device 140 can also be used as a data collection terminal to collect the input data of the input I/O interface 112 and the output result of the output I/O interface 112 as new sample data and store it in the database 130 as shown in the figure.
  • the I/O interface 112 directly uses the input data input to the I/O interface 112 and the output result of the output I/O interface 112 as a new sample as shown in the figure.
  • the data is stored in the database 130.
  • FIG. 1 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, devices, modules, etc. shown in the figure does not constitute any limitation.
  • the data The storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.
  • the target model/rule 101 obtained by training according to the training device 120 can be the neural network in this application in the embodiment of the application.
  • the neural network provided in the embodiment of the application can be CNN and deep convolution.
  • Neural networks deep convolutional neural networks, DCNN) and so on.
  • CNN is a very common neural network
  • the structure of CNN will be introduced in detail below in conjunction with Figure 2.
  • a convolutional neural network is a deep neural network with a convolutional structure. It is a deep learning architecture.
  • the deep learning architecture refers to the algorithm of machine learning. Multi-level learning is carried out on the abstract level of.
  • CNN is a feed-forward artificial neural network. Each neuron in the feed-forward artificial neural network can respond to the input image.
  • a convolutional neural network (CNN) 200 may include an input layer 210, a convolutional layer/pooling layer 220 (the pooling layer is optional), and a neural network layer 230.
  • CNN convolutional neural network
  • the convolutional layer/pooling layer 220 shown in FIG. 2 may include layers 221-226 as shown in the examples.
  • layer 221 is a convolutional layer
  • layer 222 is a pooling layer
  • layer 223 is a convolutional layer.
  • Layers, 224 is the pooling layer
  • 225 is the convolutional layer
  • 226 is the pooling layer; in another implementation, 221 and 222 are the convolutional layers, 223 is the pooling layer, and 224 and 225 are the convolutional layers.
  • Layer, 226 is the pooling layer. That is, the output of the convolutional layer can be used as the input of the subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.
  • the convolution layer 221 can include many convolution operators.
  • the convolution operator is also called a kernel. Its function in image processing is equivalent to a filter that extracts specific information from the input image matrix.
  • the convolution operator is essentially It can be a weight matrix. This weight matrix is usually pre-defined. In the process of convolution on the image, the weight matrix is usually one pixel after one pixel (or two pixels after two pixels) along the horizontal direction on the input image. ...It depends on the value of stride) to complete the work of extracting specific features from the image.
  • the size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix and the depth dimension of the input image are the same.
  • the weight matrix will extend to Enter the entire depth of the image. Therefore, convolution with a single weight matrix will produce a single depth dimension convolution output, but in most cases, a single weight matrix is not used, but multiple weight matrices of the same size (row ⁇ column) are applied. That is, multiple homogeneous matrices.
  • the output of each weight matrix is stacked to form the depth dimension of the convolutional image, where the dimension can be understood as determined by the "multiple" mentioned above.
  • Different weight matrices can be used to extract different features in the image. For example, one weight matrix is used to extract edge information of the image, another weight matrix is used to extract specific colors of the image, and another weight matrix is used to eliminate unwanted noise in the image.
  • the multiple weight matrices have the same size (row ⁇ column), the size of the convolution feature maps extracted by the multiple weight matrices of the same size are also the same, and then the multiple extracted convolution feature maps of the same size are combined to form The output of the convolution operation.
  • weight values in these weight matrices need to be obtained through a lot of training in practical applications.
  • Each weight matrix formed by the weight values obtained through training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions. .
  • the initial convolutional layer (such as 221) often extracts more general features, which can also be called low-level features; with the convolutional neural network
  • the features extracted by the subsequent convolutional layers (for example, 226) become more and more complex, such as features such as high-level semantics, and features with higher semantics are more suitable for the problem to be solved.
  • the pooling layer can be a convolutional layer followed by a layer.
  • the pooling layer can also be a multi-layer convolutional layer followed by one or more pooling layers.
  • the sole purpose of the pooling layer is to reduce the size of the image space.
  • the pooling layer may include an average pooling operator and/or a maximum pooling operator for sampling the input image to obtain an image with a smaller size.
  • the average pooling operator can calculate the pixel values in the image within a specific range to generate an average value as the result of the average pooling.
  • the maximum pooling operator can take the pixel with the largest value within a specific range as the result of the maximum pooling.
  • the operators in the pooling layer should also be related to the image size.
  • the size of the image output after processing by the pooling layer can be smaller than the size of the image of the input pooling layer, and each pixel in the image output by the pooling layer represents the average value or the maximum value of the corresponding sub-region of the image input to the pooling layer.
  • the convolutional neural network 200 After processing by the convolutional layer/pooling layer 220, the convolutional neural network 200 is not enough to output the required output information. Because as mentioned above, the convolutional layer/pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate final output information (required class information or other related information), the convolutional neural network 200 needs to use the neural network layer 230 to generate one or a group of required classes of output. Therefore, the neural network layer 230 can include multiple hidden layers (231, 232 to 23n as shown in FIG. 2) and an output layer 240. The parameters contained in the hidden layers can be based on specific task types. Relevant training data of, is obtained through pre-training. For example, the task type can include image recognition, image classification, image super-resolution reconstruction, and so on.
  • the output layer 240 After the multiple hidden layers in the neural network layer 230, that is, the final layer of the entire convolutional neural network 200 is the output layer 240.
  • the output layer 240 has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error.
  • the convolutional neural network 200 shown in FIG. 2 is only used as an example of a convolutional neural network. In specific applications, the convolutional neural network may also exist in the form of other network models.
  • CNN convolutional neural network
  • FIG. 2 may be used to execute the object detection method of the embodiment of the present application.
  • the image to be processed passes through the input layer 210 and the convolutional layer/pooling layer 220. After processing with the neural network layer 230, the detection result of the image can be obtained.
  • FIG. 3 is a hardware structure of a chip provided by an embodiment of the application, and the chip includes a neural network processor.
  • the chip may be set in the execution device 110 as shown in FIG. 1 to complete the calculation work of the calculation module 111.
  • the chip can also be set in the training device 120 as shown in FIG. 1 to complete the training work of the training device 120 and output the target model/rule 101.
  • the algorithms of each layer in the convolutional neural network as shown in Figure 2 can be implemented in the chip as shown in Figure 3.
  • the neural network processor NPU is mounted as a coprocessor to a main central processing unit (central processing unit, CPU) (host CPU), and the main CPU distributes tasks.
  • the core part of the NPU is the arithmetic circuit 303.
  • the controller 304 controls the arithmetic circuit 303 to extract data from the memory (weight memory or input memory) and perform calculations.
  • the arithmetic circuit 303 includes multiple processing units (process engines, PE). In some implementations, the arithmetic circuit 303 is a two-dimensional systolic array. The arithmetic circuit 303 may also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 303 is a general-purpose matrix processor.
  • the arithmetic circuit 303 fetches the data corresponding to the matrix B from the weight memory 302 and caches it on each PE in the arithmetic circuit 303.
  • the arithmetic circuit 303 fetches the matrix A data and matrix B from the input memory 301 to perform matrix operations, and the partial result or final result of the obtained matrix is stored in an accumulator 308.
  • the vector calculation unit 307 can perform further processing on the output of the arithmetic circuit 303, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, and so on.
  • the vector calculation unit 307 can be used for network calculations in the non-convolutional/non-FC layer of the neural network, such as pooling, batch normalization, local response normalization, etc. .
  • the vector calculation unit 307 can store the processed output vector to the unified buffer 306.
  • the vector calculation unit 307 may apply a nonlinear function to the output of the arithmetic circuit 303, such as a vector of accumulated values, to generate the activation value.
  • the vector calculation unit 307 generates a normalized value, a combined value, or both.
  • the processed output vector can be used as an activation input to the arithmetic circuit 303, for example for use in a subsequent layer in a neural network.
  • the unified memory 306 is used to store input data and output data.
  • the weight data directly transfers the input data in the external memory to the input memory 301 and/or the unified memory 306 through the storage unit access controller 305 (direct memory access controller, DMAC), stores the weight data in the external memory into the weight memory 302, and The data in the unified memory 306 is stored in the external memory.
  • DMAC direct memory access controller
  • the bus interface unit (BIU) 310 is used to implement interaction between the main CPU, the DMAC, and the instruction fetch memory 309 through the bus.
  • An instruction fetch buffer 309 connected to the controller 304 is used to store instructions used by the controller 304.
  • the controller 304 is used to call the instructions cached in the memory 309 to control the working process of the computing accelerator.
  • the unified memory 306, the input memory 301, the weight memory 302, and the instruction fetch memory 309 are all on-chip (On-Chip) memories.
  • the external memory is a memory external to the NPU.
  • the external memory can be a double data rate synchronous dynamic random access memory.
  • Memory double data rate synchronous dynamic random access memory, referred to as DDR SDRAM
  • HBM high bandwidth memory
  • each layer in the convolutional neural network shown in FIG. 2 can be executed by the arithmetic circuit or the vector calculation module 307.
  • the execution device 110 in FIG. 1 introduced above can execute each step of the object detection method in the embodiment of the present application.
  • the CNN model shown in FIG. 2 and the chip shown in FIG. 3 can also be used to execute the object in the embodiment of the present application.
  • the object detection method of the embodiment of the present application will be described in detail below with reference to the accompanying drawings.
  • the method shown in FIG. 4 can be applied in different scenarios. Specifically, the method shown in FIG. 4 can be applied in scenarios such as recognizing everything and street view recognition.
  • the image to be detected in step 401 may be an image taken by the mobile terminal through a camera, or an image already stored in the mobile terminal's album.
  • the image to be detected in step 401 may be a street view image taken by a camera on the roadside.
  • the method shown in FIG. 4 may be executed by a neural network (model). Specifically, the method shown in FIG. 4 may be executed by CNN or DNN.
  • the entire image of the image to be detected may be subjected to convolution processing or regularization processing, etc., to obtain the image characteristics of the entire image, and then the initial image characteristics corresponding to the object to be detected are obtained from the image characteristics of the entire image. .
  • performing convolution processing on the image to be detected to obtain the initial image feature of the object to be detected includes: performing convolution processing on the entire image of the image to be detected to obtain the complete image feature of the image to be detected; Among the complete image features of the image to be detected, the image feature corresponding to the object to be detected is determined as the initial image feature of the object to be detected.
  • performing convolution processing on the image to be detected to obtain the initial image feature of the object to be detected includes: separately acquiring the image feature corresponding to each object to be detected each time.
  • the cross-domain knowledge map information includes the association relationship between object categories corresponding to the objects to be detected in different domains, and the enhanced image features indicate semantic information of object categories corresponding to other objects in different domains that are associated with the objects to be detected.
  • the object categories in the first domain or the first data set include men, women, boys, girls, roads, and streets.
  • Object categories in the second domain include people, handbags, school bags, cars, and trucks. It can be considered that the men, women, boys, and girls in the first domain have an association relationship with the people in the second domain. The women and girls in the first domain have an association with the handbags in the second domain. The boys and girls in the first domain have an association with the school bags in the second domain. There is an association between roads and streets in the first domain and cars and trucks in the second domain.
  • Semantic information can refer to high-level information that can assist in image detection.
  • the above-mentioned semantic information can specifically be what the object is and what is around the object (semantic information is generally different from low-level information, such as image edges, pixels, brightness, etc.).
  • the object to be detected is a woman, and other objects associated with the woman in the image to be detected include a handbag, then the enhanced image feature of the object to be detected may indicate semantic information of the handbag.
  • the cross-domain knowledge graph may include nodes and node edges, where nodes correspond to objects to be detected, and node edges correspond to relationships between high-level semantic features of different objects to be detected.
  • the classification layer parameters corresponding to different domains are weighted and merged to obtain the high-level semantic features of the object to be detected.
  • the classification layer parameters can be understood as maintaining a class of the category. Center, class center refers to the high-level semantic features of the category.
  • the weight of the relationship between the object categories corresponding to the object to be detected in different domains is projected onto the node connection edge of the object to be detected, and the weight of the node connection edge is obtained.
  • the weight of the edge between the i-th node and the j-th node of the area graph in the S domain is Where Cf j is the feature of the i-th object to be detected in one domain and the feature of the j-th object to be detected in another domain.
  • G SP is the weight of the relationship between object categories corresponding to the objects to be detected in different domains, and G SP can be regarded as a matrix.
  • the weight of the relationship between the object categories corresponding to the object to be detected in different domains is projected onto the node connection edge of the object to be detected, and the weight of the node connection edge is obtained, which can be expressed by the following formula, where T represents the transposition of the matrix:
  • the process of projection can be regarded as the process of converting the weight of the relationship between the object categories into the weight of the relationship between the objects to be detected, and the weight of the relationship between the objects to be detected is the weight of the edges of the nodes.
  • the high-level semantic features are convolved according to the weights of the edges of the nodes, and the enhanced image features of the object to be detected can be obtained.
  • the relationship weight may be determined according to the distance relationship between the object categories corresponding to the objects to be detected in different domains.
  • the distance relationship includes one or more of the following information:
  • the color of an apple is red, and the color of a strawberry is also red. Then, apples and strawberries have the same color attributes (or, it can be said that apples and strawberries are relatively close in color attributes).
  • the similarity of word embedding constructed with linguistic knowledge can be understood as the degree of similarity between word vectors of different object categories.
  • the weight of the edge between the i-th node in one domain and the j-th node in the other domain is
  • f i and f j are the feature of the i-th object to be detected in one domain and the feature of the j-th object to be detected in the other domain (abbreviation of the initial image feature of the object to be detected).
  • the candidate frame and classification of the object to be detected determined in step 404 may be the final candidate frame and the final classification (result) of the object to be detected, respectively.
  • step 404 the initial image feature of the object to be detected and the enhanced image feature of the object to be detected can be combined to obtain the final image feature of the object to be detected, and then the candidate of the object to be detected can be determined according to the final image feature of the object to be detected. Box and classification.
  • the initial image feature of the object to be detected is a convolution feature map with a size of M1 ⁇ N1 ⁇ C1 (M1, N1, and C1 can represent width, height, and number of channels, respectively), and the enhanced image feature of the object to be detected is a size It is a convolution feature map of M1 ⁇ N1 ⁇ C2 (M1, N1, and C2 represent width, height, and number of channels, respectively). Then, by combining these two convolution feature maps, the final image features of the object to be detected can be obtained , The final image feature is a convolution feature map with a size of M1 ⁇ N1 ⁇ (C1+C2).
  • the description here is based on an example in which the convolution feature map of the initial image feature and the convolution feature map of the enhanced image feature have the same size (same width and height) but different channel numbers.
  • the initial image feature and the enhanced image feature can also be combined.
  • the size of the convolution feature map and the convolution feature map of the enhanced image feature are unified (the width and height are unified), and then the convolution feature map of the initial image feature and the convolution feature map of the enhanced image feature are combined to obtain the final image feature Convolution feature map.
  • the detection result of the object to be detected is comprehensively determined by the initial image feature of the object to be detected and the enhanced image feature, and the detection result is obtained by considering only the initial image feature of the object to be detected In comparison, better detection results can be obtained.
  • this application when determining the detection result of the object to be detected, this application not only considers the initial image characteristics reflecting the characteristics of the object to be detected, but also considers the semantic information of other objects in the image to be detected that are associated with the object to be detected.
  • a cross-domain knowledge graph also known as a transferable knowledge graph in multiple scenarios
  • the present invention can capture the internal relationship between different objects, and use graph convolutional networks to fuse a large number of different data sets and different types of information. The data utilization rate is greatly improved, the detection performance is higher, and the large-scale object detection is truly realized.
  • a model trained only through the second domain mentioned above may be determined to be a person and a handbag when the detection result is determined. If the solution provided in this application is passed, the second training through the first domain and the second domain The model test, after confirming that the test result may be a woman carrying a handbag. And finally improve the effect of object detection.
  • the method shown in FIG. 4 further includes: determining the initial candidate frame of the object to be detected according to the initial image feature of the object to be detected.
  • the entire image of the image to be detected is first subjected to convolution processing to obtain the convolution characteristics of the entire image of the image to be detected, and then according to the fixed size requirements, the image to be detected Divide into different boxes, score the features corresponding to the image in each box, and filter out the boxes with higher scores as the initial candidate boxes.
  • the image to be detected is the first image.
  • the entire image of the first image can be convolved to obtain the convolution characteristics of the entire image of the first image. , And then divide the first image into 3 ⁇ 3 boxes, and score the corresponding features of each box image. Finally, box A and box B with higher scores can be screened out as initial candidate boxes.
  • the process of determining the candidate frame and classification of the object to be detected may be to first combine the initial image feature and the enhanced image feature to obtain After the final image feature, the initial candidate frame is adjusted according to the final image feature to obtain the candidate frame, and the initial classification result is corrected according to the final image feature to obtain the classification result.
  • the foregoing adjustment of the initial candidate frame according to the final image feature may be adjusting the coordinates around the initial candidate frame according to the final image feature until the candidate frame is obtained, and the foregoing adjustment of the initial classification result according to the final image feature may be: Build a classifier to reclassify, and then get the classification result.
  • Fig. 6 is a schematic flowchart of an object detection method according to an embodiment of the present application.
  • the method shown in FIG. 6 may be executed by an object detection device, which may be an electronic device with an object detection function.
  • the form of the device specifically included in the electronic device can be as described above in the method shown in FIG. 4.
  • the method shown in FIG. 6 includes steps 601 to 609, and these steps are described in detail below.
  • step 602 and step 603 may be detailed implementations of step 402 (or referred to as specific implementations), and steps 604 to 608 can be detailed implementations of step 403 (or referred to as specific implementations).
  • Step 601 can be understood with reference to step 401 in the embodiment corresponding to FIG. 4, and details are not repeated here.
  • the image to be detected can be input into a traditional object detector for processing (such as Faster-RCNN) to obtain the initial candidate area. Since this application performs object detection for multiple different domains, each domain has its own corresponding initial candidate area.
  • a traditional object detector for processing such as Faster-RCNN
  • the image to be detected can be convolved first to obtain the convolution characteristics of the entire image of the image to be detected, and then the image to be detected is divided into different boxes according to certain size requirements, and then for different domains, Score the features corresponding to the image in each box, and filter out the boxes with higher scores as the initial candidate boxes, thereby obtaining the initial candidate boxes corresponding to different domains.
  • CNN can be used to extract the image features of the initial candidate region. For example, if the first image is the image to be detected, in order to obtain the initial candidate frame of the object to be detected in the first image, the first image can be convolved to obtain the convolution feature of the first image, and then Divide the first image into 4 ⁇ 4 boxes (can also be divided into other numbers of boxes), score the corresponding features of the image of each box, and score the higher box A and box B Filtered out as the initial candidate frame.
  • the image features of the entire image of the image to be detected (the image features of the entire image of the image to be detected can be obtained by convolution processing the entire image of the image to be detected) corresponding to the square
  • the initial image feature corresponding to box A and the initial image feature corresponding to box B are obtained.
  • the domain-related semantic pool records the high-level semantic features of each category.
  • the classification layer parameters corresponding to different categories in the classifier may continuously change. In this case, the semantic pool can be classified The corresponding classification layer parameters are updated.
  • the extracted classification layer parameters may be the classification layer parameters of all classifications in the classifiers corresponding to different domains in the object detector for object detection of the object to be detected.
  • the high-level semantic features in the semantic pool corresponding to the domain are mapped to the nodes of the area map in the domain to obtain the high-level of the object to be detected Semantic representation.
  • the weights on the edges of the area graph nodes in the domain are given.
  • the weight of the edge between the i-th node and the j-th node in the region graph is Where f i and f i are the feature of the i-th object to be detected in one domain and the feature of the j-th object to be detected in the other domain.
  • an intra-domain area map can be constructed separately according to the above method.
  • the high-level semantic features of the semantic pool are mapped to the nodes of the inter-domain map to obtain the high-level semantics of the object to be detected Express.
  • the weight of the relationship between the categories is given, and then projected to the edge of the inter-domain graph node to obtain the weight of the node's edge of the inter-domain graph.
  • the distance in step 606 can be understood with reference to the explanation of the distance in the embodiment corresponding to FIG. 4, and details are not repeated here.
  • the feature construction method on the nodes of the inter-domain graph is the same as that of the intra-domain graph.
  • the weight of the edge between the i-th node in one domain and the j-th node in the other domain of the inter-domain graph is Where f i and f i are the feature of the i-th object to be detected in one domain and the feature of the j-th object to be detected in the other domain.
  • the intra-domain graph convolutional network is used to spread the high-level semantic representations of different objects to be detected on the nodes, and the features that are combined with the high-level semantic representations of other objects to be detected after inference and inference are obtained.
  • the graph convolution of the spatial information mechanism can be selected.
  • the relative spatial information between the objects to be detected is used to learn K Gaussian kernels.
  • the specific formula is:
  • ⁇ (k) is the k-th Gaussian kernel
  • ⁇ k and ⁇ k are the learnable mean vector and covariance vector
  • g ij represents the relative spatial relationship between the i-th and j-th objects to be detected
  • f′ k (i) ⁇ j ⁇ adjacent node (i) ⁇ k (g ij )x j e ij .
  • the K features obtained by the intra-domain graph convolution on each node will be fused into the corresponding high-level semantic representation of the object to be detected.
  • the high-level semantic representation of the object to be detected in different domains on the node is propagated using the inter-domain graph convolutional network, and the features of the high-level semantic representation of the object to be detected in different domains are obtained after inference and inference.
  • Step 609 can be understood with reference to step 404 in the embodiment corresponding to FIG. 4, and details are not repeated here.
  • the object detection method of the embodiment of the present application is described in detail above in combination with the flowchart. In order to better understand the object detection method of the embodiment of the present application, the object detection method of the embodiment of the present application will be described in detail below in conjunction with a more specific flowchart. description of.
  • FIG. 7 is a schematic flowchart of an object detection method according to an embodiment of the present application.
  • the method shown in FIG. 7 may be executed by an object detection device, which may be an electronic device with an object detection function.
  • the form of the device specifically included in the electronic device can be as described above in the method shown in FIG. 4 introduced above.
  • Step 1 Input the picture and pass through a traditional object detector to obtain a preliminary candidate frame and the characteristics of the object to be detected.
  • Step 2 Use classifiers corresponding to different domains in the object detector to extract classification layer parameters, and construct a domain-related semantic pool for each domain to record the high-level semantic features of each category. This semantic pool will be continuously updated as the classifier is optimized during the training process.
  • Step 3 Construct a region map in the domain: According to the classification weights of the features of the object to be detected in different categories given by the detection network, map the high-level semantic features of the semantic pool to the nodes of the region map in the domain to obtain the high-level semantics of the object to be detected Express. According to the relationship between the features of different objects to be detected, the weights on the edges of the nodes in the area graph are given.
  • Step 4 Construct the inter-domain area map: According to the classification weights of the object features to be detected in the respective domains given by the detection network in different categories, map the high-level semantic features of the semantic pool to the nodes of the inter-domain area graph to obtain the High-level semantic representation of detected objects. According to the distance between the classification features of the detected objects in two different domains, the weight of the relationship between the categories is given, and then projected to the edge of the inter-domain graph node to obtain the weight of the node's edge of the inter-domain graph.
  • Step 5 Intra-domain graph convolution: Through the constructed intra-domain map, the intra-domain graph convolution network is used to spread the high-level semantic representations of different objects to be detected on the nodes, and the features that are combined with the high-level semantic representations of other objects to be detected after inference are obtained . By learning a sparse area map to fuse the high-level semantic representation of different objects to be detected, the feature expression ability of different objects to be detected is enhanced.
  • Step 6 Inter-domain graph convolution: Through the constructed inter-domain area graph, the inter-domain graph convolution network is used to propagate the high-level semantic representation of the objects to be detected in different domains on the node, and the inference and inference of the fusion of different domains are obtained. Detect features of high-level semantic representations of objects.
  • Step 7 Optimize and enhance the feature layer of the candidate region: project the features obtained after inference inference from the intra-domain graph convolution and the inter-domain graph convolution into the corresponding high-level semantic representation of the object to be detected, and perform classification and regression to achieve improvement The purpose of large-scale testing performance.
  • the first method shown in Table 1 is the FPN detection method
  • the second method is the multi-branch detection method (Multi Branches).
  • the data set used for training the model includes three data sets: MSCOCO data set, visual genome (VG) data set and ADE data set, that is, the three data sets are used to train the model together, and the test phase is performed for each data set separately test.
  • the MSCOCO data set has 80 general object detection annotations, containing about 110,000 training data sets and 5,000 test sets.
  • the VG data set has a total of 1,000 large-scale general object detection data sets, a training data set of 88,000 images, and a test set of 5,000.
  • the ADE dataset has 445 types of large-scale general object detection datasets, a training dataset of 20,000 images, and a test set of 1,000.
  • the average precision (AP) and average recall (AR) are mainly used for evaluation, and the accuracy under different thresholds is considered in the comparison.
  • the average precision and average recall of the object are mainly used for evaluation, and the accuracy under different thresholds is considered in the comparison.
  • the three data sets are used for model training.
  • the AP and AR of the method of this application are respectively greater than the first The AP and AR of one method and the second method, and the larger the value of AP and AR, the better the effect of object detection. It can be seen from Table 1 that the method of the present application has a significant improvement in effect compared with several existing object detection methods.
  • the training model in Table 1 uses three data sets for training.
  • this application provides The method is compared with the effects of several existing object detection methods.
  • several other object detection methods can also be included, such as the third method: fine-tuning, fourth One method: overlap label detection method (overlap labels), and the fifth method: pseudo label detection method (pseudo labels).
  • any two data sets of the three data sets are used for model training.
  • the AP and AR must be larger than AP and AR of the first object detection method to the sixth object detection method, and the larger the value of AP and AR, the better the effect of object detection. It can be seen from Table 2 that the method of the present application has a significant improvement in effect compared with several existing object detection methods.
  • the method provided in this application has a significant improvement in the detection effect in situations where there are serious object occlusions, blurred categories, and small-scale objects.
  • our method effectively captures the internal relationship between different objects by constructing a multi-domain transferable knowledge map, and uses graph convolutional networks to fuse a large number of different data sets and different categories Information greatly improves the data utilization rate, makes the detection performance higher, and truly realizes large-scale object detection.
  • Fig. 8 is a schematic flowchart of a neural network training method according to an embodiment of the present application.
  • the method shown in FIG. 8 can be executed by a device with strong computing capabilities such as a computer device, a server device, or a computing device.
  • a device with strong computing capabilities such as a computer device, a server device, or a computing device.
  • the training data includes training images in different domains and object detection and labeling results of the objects to be detected in the training images.
  • the cross-domain knowledge map information includes the association relationship between object categories corresponding to the objects to be detected in different domains, and the enhanced image features indicate semantic information of object categories corresponding to other objects in different domains that are associated with the objects to be detected.
  • the object detection and annotation result of the object to be detected in the training image includes the annotation candidate frame and the annotation classification result of the object to be detected in the training image.
  • a set of initial model parameters can be set for the neural network, and then based on the object detection result of the object to be detected in the training image and the object detection labeling result of the object to be detected in the training image.
  • Gradually adjust the model parameters of the neural network until the difference between the object detection structure of the object to be detected in the training image and the object detection and annotation results of the object to be detected in the training image is within a certain preset range, or when When the number of times of training reaches the preset number of times, the model parameters of the neural network at this time are determined as the final parameters of the neural network model, thus completing the training of the neural network.
  • neural network trained through the method shown in FIG. 8 can be used to implement the object detection method of the embodiment of the present application.
  • the training method of the present application extracts more features for object detection during the training process, and can train a neural network with better performance, so that the neural network for object detection can achieve better object detection results. .
  • the cross-domain knowledge graph may include nodes and node edges, where nodes correspond to objects to be detected, and node edges correspond to relationships between high-level semantic features of different objects to be detected.
  • the classification layer parameters corresponding to different domains are weighted and merged to obtain the high-level semantic features of the object to be detected.
  • the classification layer parameters can be understood as maintaining a class of the category. center. The weight of the relationship between the object categories corresponding to the object to be detected in different domains is projected onto the node connection edge of the object to be detected, and the weight of the node connection edge is obtained.
  • the high-level semantic features are convolved according to the weights of the edges of the nodes, and the enhanced image features of the object to be detected can be obtained.
  • the relationship weight may be determined according to the distance relationship between the object categories corresponding to the objects to be detected in different domains.
  • the distance relationship includes one or more of the following information:
  • the color of an apple is red, and the color of a strawberry is also red. Then, apples and strawberries have the same color attributes (or, it can be said that apples and strawberries are relatively close in color attributes).
  • the similarity of word embedding constructed with linguistic knowledge can be understood as the degree of similarity between word vectors of different object categories.
  • the weight of the edge between the i-th node in one domain and the j-th node in the other domain is
  • f i and f j are the feature of the i-th object to be detected in one domain and the feature of the j-th object to be detected in the other domain (abbreviation of the initial image feature of the object to be detected).
  • FIG. 9 and FIG. 10 can execute each step of the object detection method of the embodiment of the present application
  • the neural network training device shown in FIG. 11 can execute each of the neural network training method of the embodiment of the present application. Steps, the repeated description will be appropriately omitted when introducing the devices shown in FIG. 9 to FIG. 11 below.
  • Fig. 9 is a schematic block diagram of an object detection device according to an embodiment of the present application.
  • the object detection device 7000 shown in FIG. 9 includes:
  • the image acquisition module 901 is configured to perform step 401 in the embodiment corresponding to FIG. 4 and step 601 in the embodiment corresponding to FIG. 6.
  • the feature extraction module 902 is configured to perform step 402 in the embodiment corresponding to FIG. 4, step 602 in the embodiment corresponding to FIG. 6, step 603 in the embodiment corresponding to FIG. 6, and step in the embodiment corresponding to FIG. 6 607, step 608 in the embodiment corresponding to FIG. 6.
  • the detection module 903 is configured to execute step 404 in the embodiment corresponding to FIG. 4 and step 609 in the embodiment corresponding to FIG. 6.
  • the parameter extraction module 904 is configured to execute step 403 in the embodiment corresponding to FIG. 4 and step 604 in the embodiment corresponding to FIG. 6.
  • the projection module 905 is configured to perform step 605 in the embodiment corresponding to FIG. 6 and step 606 in the embodiment corresponding to FIG. 6.
  • the relationship weight determination module 906 is configured to perform step 605 in the embodiment corresponding to FIG. 6 and step 606 in the embodiment corresponding to FIG. 6.
  • the image acquisition module 901 in the above object detection device may be equivalent to the I/O interface 112 in the execution device 110, and the object detection device
  • the feature extraction module 902 and the detection module 903 are equivalent to the calculation module 111 in the execution device 110.
  • the image acquisition module 901 in the above object detection device may be equivalent to the bus interface unit 510 in the neural network processor, and the object detection device
  • the feature extraction module 902 and the detection module 903 in the execution device 110 are equivalent to the arithmetic circuit 503 in the execution device 110, or the feature extraction module 902 and the detection module 903 in the object detection device can also be equivalent to the arithmetic circuit 303+vector calculation in the execution device 110 Unit 307 + accumulator 308.
  • Fig. 10 is a schematic block diagram of an object detection device according to an embodiment of the present application.
  • the object detection device module shown in FIG. 10 includes a memory 1001, a processor 1002, a communication interface 1003, and a bus 1004.
  • the memory 1001, the processor 1002, and the communication interface 1003 implement communication connections between each other through the bus 1004.
  • the communication interface 1003 is equivalent to the image acquisition module 901 in the object detection device, and the processor 1002 is equivalent to the feature extraction module 902 and the detection module 903 in the object detection device.
  • the following is a detailed introduction to each module and modules in the object detection device module.
  • the memory 1001 may be a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM).
  • the memory 1001 may store a program.
  • the processor 1002 and the communication interface 1003 are used to execute each step of the object detection method in the embodiment of the present application.
  • the communication interface 1003 may obtain the image to be detected from a memory or other devices, and then the processor 1002 performs object detection on the image to be detected.
  • the processor 1002 may adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more
  • the integrated circuit is used to execute related programs to realize the functions required by the modules in the object detection device of the embodiment of the present application (for example, the processor 1002 can implement the feature extraction module 902 and the detection module 903 in the above-mentioned object detection device).
  • the function to be executed or execute the object detection method of the embodiment of the present application.
  • the processor 1002 may also be an integrated circuit chip with signal processing capability.
  • each step of the object detection method in the embodiment of the present application can be completed by an integrated logic circuit of hardware in the processor 1002 or instructions in the form of software.
  • the above-mentioned processor 1002 may also be a general-purpose processor, a digital signal processing (digital signal processing, DSP), an ASIC, an off-the-shelf programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices , Discrete hardware components.
  • the aforementioned general-purpose processor may be a microprocessor or the processor may also be any conventional processor.
  • the steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor.
  • the software module can be located in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers.
  • the storage medium is located in the memory 1001, and the processor 1002 reads the information in the memory 1001, and combines its hardware to complete the functions required by the modules included in the object detection apparatus of the embodiment of the present application, or perform the object detection of the method embodiment of the present application method.
  • the communication interface 1003 uses a transceiving device such as but not limited to a transceiver to implement communication between the device module and other devices or a communication network.
  • a transceiving device such as but not limited to a transceiver to implement communication between the device module and other devices or a communication network.
  • the image to be processed can be obtained through the communication interface 1003.
  • the bus 1004 may include a path for transferring information between various components of the device module (for example, the memory 1001, the processor 1002, and the communication interface 1003).
  • FIG. 11 is a schematic diagram of the hardware structure of a neural network training device according to an embodiment of the present application. Similar to the above device, the neural network training device shown in FIG. 11 includes a memory 1101, a processor 1102, a communication interface 1103, and a bus 1104. Among them, the memory 1101, the processor 1102, and the communication interface 1103 implement communication connections between each other through the bus 1104.
  • the memory 1101 may store a program.
  • the processor 1102 is configured to execute each step of the neural network training method of the embodiment of the present application.
  • the processor 1102 may adopt a general CPU, a microprocessor, an ASIC, a GPU, or one or more integrated circuits for executing related programs to implement the neural network training method of the embodiment of the present application.
  • the processor 1102 may also be an integrated circuit chip with signal processing capabilities.
  • each step of the neural network training method (the method shown in FIG. 8) of the embodiment of the present application can be completed by the integrated logic circuit of the hardware in the processor 1102 or the instructions in the form of software.
  • the neural network is trained by the neural network training device shown in FIG. 11, and the trained neural network can be used to execute the object detection method of the embodiment of the present application (the method shown in FIG. 8).
  • the device shown in FIG. 11 can obtain training data and the neural network to be trained from the outside through the communication interface 1103, and then the processor trains the neural network to be trained according to the training data.
  • the device modules and devices only show the memory, processor, and communication interface, in the specific implementation process, those skilled in the art should understand that the device modules and devices may also include other necessary for normal operation. Device. At the same time, according to specific needs, those skilled in the art should understand that the device modules and devices may also include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the device module and the device may also only include the necessary devices for implementing the embodiments of the present application, and not necessarily all the devices shown in FIG. 10 and FIG. 11.
  • the disclosed system, device, and method can be implemented in other ways.
  • the device embodiments described above are merely illustrative.
  • the division of the modules is only a logical function division, and there may be other divisions in actual implementation, for example, multiple modules or components may be combined or It can be integrated into another system, or some features can be ignored or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or modules, and may be in electrical, mechanical or other forms.
  • modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
  • each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist alone physically, or two or more modules may be integrated into one module.
  • the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer readable storage medium.
  • the technical solution of the present application essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disk and other media that can store program code .

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Multimedia (AREA)
  • Computational Linguistics (AREA)
  • Evolutionary Computation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Molecular Biology (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Software Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Mathematical Physics (AREA)
  • Computing Systems (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Animal Behavior & Ethology (AREA)
  • Databases & Information Systems (AREA)
  • Image Analysis (AREA)

Abstract

本申请实施例公开了一种物体检测方法和装置，涉及人工智能领域，具体涉及计算机视觉领域。该方法可以包括获取待检测图像。确定待检测图像中的待检测物体的初始图像特征。根据跨域知识图谱信息确定待检测物体的增强图像特征，跨域知识图谱信息包括不同域中待检测物体对应的物体类别之间的关联关系，增强图像特征指示不同域中与待检测物体相关联的其他物体对应的物体类别的语义信息。根据待检测物体的初始图像特征和待检测物体的增强图像特征，确定待检测物体的候选框和分类。通过本申请提供的技术方案，构建跨域知识图谱，可以捕捉到不同待检测物体间的内在关系，提高物体检测的效果。

Description

一种物体检测方法、装置以及存储介质
本申请要求于2020年1月21日提交中国专利局，申请号为202010072238.0、发明名称为“一种物体检测方法、装置以及存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及计算机视觉领域，尤其涉及一种物体检测方法、装置以及存储介质。
背景技术
人工智能(artificial intelligence,AI)是利用数字计算机或者数字计算机控制的机器模拟、延伸和扩展人的智能，感知环境、获取知识并使用知识获得最佳结果的理论、方法、技术及应用***。换句话说，人工智能是计算机科学的一个分支，它企图了解智能的实质，并生产出一种新的能以人类智能相似的方式作出反应的智能机器。人工智能也就是研究各种智能机器的设计原理与实现方法，使机器具有感知、推理与决策的功能。人工智能领域的研究包括机器人，自然语言处理，计算机视觉，决策与推理，人机交互，推荐与搜索，AI基础理论等。
物体检测是一项基本的计算机视觉任务，它可以识别图像中物体的位置和类别。在实际应用中，研究员和工程师们会根据应用场景与实际任务需求的不同创建针对不同具体问题的数据集，用于训练高度定制化和独特的自动物体检测器。
发明内容
跨数据集的物体检测是实现大规模物体检测的高效方法。但现有多任务学习仅通过在模型中加入多个分支来同时处理多个任务，无法实现不同数据集以及不同物体类别之间的交互，无法捕捉到不同数据集中待检测物体之间的内在关系，因此效果不佳。
本申请提供一种物体检测方法、装置和计算机存储介质，以提高物体检测的效果。
本申请第一方面提供一种物体检测方法，可以包括：获取待检测图像。确定待检测图像中的待检测物体的初始图像特征。根据跨域知识图谱信息确定待检测物体的增强图像特征，跨域知识图谱信息可以包括不同域中待检测物体对应的物体类别之间的关联关系，增强图像特征指示不同域中与待检测物体相关联的其他物体对应的物体类别的语义信息。根据待检测物体的初始图像特征和待检测物体的增强图像特征，确定待检测物体的候选框和分类。
上述物体检测方法可以应用在不同的应用场景中，例如，上述物体检测方法可以应用在识别万物的场景中，也可以应用在街景识别的场景中。
当上述方法应用在利用移动终端来识别万物的场景时，上述待检测图像可以是移动终端通过摄像头拍摄的图像，也可以是移动终端相册中已经存储的图像。
当上述方法应用在街景识别的场景时，上述待检测图像可以是路边的摄像头拍摄的街景图像。
两个类别在同一个图像中同时出现的概率越大，则认为两个类别之间有关联关系。举例说明，第一个域中或者说第一个数据集中的物体类别包括男人，女人，男孩，女孩，马路，街道。第二个域中的物体类别包括人，手提包，书包，汽车，卡车。可以认为第一个域中的男人，女人，男孩，女孩与第二个域中的人之间具有关联关系。第一个域中的女人 和女孩与第二个域中的手提包具有关联关系。第一个域中的马路和街道与第二个域中的汽车、卡车之间具有关联关系。
语义信息可以是指能够辅助进行图像检测的高级别的信息。例如，上述语义信息具体可以是物体是什么，物体的周围有什么(语义信息一般不同于低级别的信息，如图像的边，像素点和亮度等等)。例如，待检测物体为女人，待检测图像中与该自行车相关联的其他物体包括手提包，那么，上述待检测物体的增强图像特征指示的可以是手提包的语义信息。
由第一方面可知，本申请提供的方案可以同时有效利用大量不同数据集、不同类别的信息来训练同一个网络，极大提升了数据利用率，使得检测性能更高。
可选地，结合上述第一方面，在第一种可能的实现方式中，跨域知识图谱可以包括节点和节点连边，节点对应待检测物体，节点连边对应不同待检测物体的高级语义特征之间的关系，方法还可以包括：获取不同域对应的分类层参数。根据不同域中初始图像特征在不同物体类别上的分类权重，将不同域对应的分类层参数加权融合，得到待检测物体的高级语义特征。将不同域中待检测物体对应的物体类别之间的关系权重投影到待检测物体的节点连边上，得到节点连边的权重。
可选地，结合上述第一方面第一种可能的实现方式，在第二种可能的实现方式中，该方法还可以包括：根据不同域中待检测物体对应的物体类别间的距离关系确定关系权重。
可选地，结合上述第一方面第二种可能的实现方式，在第三种可能的实现方式中，待检测物体对应的物体类别间的距离关系可以包括以下信息中的一种或几种：不同域中待检测物体对应的物体类别间的属性关系。不同域中待检测物体对应的物体类别间的位置关系或者主动宾关系。不同域中待检测物体对应的物体类别间的利用语言学知识构建的词嵌入相似度。不同域中待检测物体对应的物体类别间的根据训练数据对神经网络模型进行训练得到的距离关系。
可选地，结合上述第一方面第一种至第一方面第三种可能的实现方式，在第四种可能的实现方式中，根据跨域知识图谱信息确定待检测物体的增强图像特征，可以包括：根据节点连边的权重对高级语义特征进行卷积处理，得到待检测物体的增强图像特征。由第一方面第四种可能的实现方式可知，通过图卷积可以在多个不同域中合并和传递相关语义信息，并能有效地捕捉不同数据集下不同物体之间的内在关系，使得不同域或者不同数据集的标注信息可以互补。
本申请第二方面提供一种图像检测装置，可以包括：图像获取模块，用于获取待检测图像。特征提取模块，用于确定待检测图像中的待检测物体的初始图像特征。特征提取模块，还用于根据跨域知识图谱信息确定待检测物体的增强图像特征，跨域知识图谱信息可以包括不同域中待检测物体对应的物体类别之间的关联关系，增强图像特征指示不同域中与待检测物体相关联的其他物体对应的物体类别的语义信息。检测模块，用于根据待检测物体的初始图像特征和待检测物体的增强图像特征，确定待检测物体的候选框和分类。
可选地，结合上述第二方面，在第一种可能的实现方式中，跨域知识图谱可以包括节点和节点连边，节点对应待检测物体，节点连边对应不同待检测物体的高级语义特征之间的关系，图像检测装置还可以包括参数获取模块以及投影模块，参数获取模块，用于获取不同域对应的分类层参数。特征提取模块，具体用于根据不同域中初始图像特征在不同物 体类别上的分类权重，将不同域对应的分类层参数加权融合，得到待检测物体的高级语义特征。投影模块，用于将不同域中待检测物体对应的物体类别之间的关系权重投影到待检测物体的节点连边上，得到节点连边的权重。
可选地，结合上述第二方面第一种可能的实现方式，在第二种可能的实现方式中，还可以包括关系权重确定模块，关系权重确定模块，用于根据不同域中待检测物体对应的物体类别间的距离关系确定关系权重。
可选地，结合上述第二方面第二种可能的实现方式，在第三种可能的实现方式中，待检测物体对应的物体类别间的距离关系可以包括以下信息中的一种或几种：不同域中待检测物体对应的物体类别间的属性关系。不同域中待检测物体对应的物体类别间的位置关系或者主动宾关系。不同域中待检测物体对应的物体类别间的利用语言学知识构建的词嵌入相似度。不同域中待检测物体对应的物体类别间的根据训练数据对神经网络模型进行训练得到的距离关系。
可选地，结合上述第二方面第一种至第二方面第三种可能的实现方式，在第四种可能的实现方式中，特征提取模块，具体用于根据节点连边的权重对高级语义特征进行卷积处理，得到待检测物体的增强图像特征。
本申请第三方面提供了一种神经网络的训练方法，该方法包括：获取训练数据，该训练数据包括训练图像以及训练图像中待检测物体的物体检测标注结果；根据神经网络提取该训练图像中的待检测物体的初始图像特征；根据该神经网络以及跨域知识图谱信息提取该训练图像中的待检测物体的增强图像特征；根据该神经网络对待检测物体的初始图像特征和增强图像特征进行处理，得到该待检测物体的物体检测结果；根据该训练图像的中的待检测物体的物体检测结果与该训练图像中的待检测物体的物体检测标注结果，确定该神经网络的模型参数。
其中，跨域知识图谱信息可以包括不同域中待检测物体对应的物体类别之间的关联关系，增强图像特征指示不同域中与待检测物体相关联的其他物体对应的物体类别的语义信息。
上述训练图像中的待检测物体的物体检测标注结果包括该训练图像中的待检测物体的标注候选框和标注分类结果。
在对上述神经网络进行训练的过程中，可以为神经网络设置一套初始的模型参数，然后根据训练图像中的待检测物体的物体检测结果与训练图像中的待检测物体的物体检测标注结果的差异来逐渐调整神经网络的模型参数，直到训练图像中的待检测物体的物体检测结构与训练图像中的待检测物体的物体检测标注结果之间的差异在一定的预设范围内，或者，当训练的次数达到预设次数时，将此时的神经网络的模型参数确定为该神经网络模型的最终的参数，这样就完成了对神经网络的训练了。应理解，通过第三方面训练得到的神经网络能够用于执行本申请第一方面中的方法。
可选地，结合上述第三方面，在第一种可能的实现方式中，跨域知识图谱可以包括节点和节点连边，其中节点对应待检测物体，节点连边对应不同待检测物体的高级语义特征之间的关系。根据不同域中初始图像特征在不同物体类别上的分类权重，将不同域对应的分类层参数加权融合，得到待检测物体的高级语义特征，分类层参数可以理解成是在维护 该类别的一个类中心。将不同域中待检测物体对应的物体类别之间的关系权重投影到待检测物体的节点连边上，得到节点连边的权重。
可选地，结合上述第三方面第一种可能的实现方式，在第二种可能的实现方式中，还可以包括，根据不同域中待检测物体对应的物体类别间的距离关系确定关系权重。
可选地，结合上述第三方面第二种可能的实现方式，在第三种可能的实现方式中，待检测物体对应的物体类别间的距离关系可以包括以下信息中的一种或几种：不同域中待检测物体对应的物体类别间的属性关系。不同域中待检测物体对应的物体类别间的位置关系或者主动宾关系。不同域中待检测物体对应的物体类别间的利用语言学知识构建的词嵌入相似度。不同域中待检测物体对应的物体类别间的根据训练数据对神经网络模型进行训练得到的距离关系。
可选地，结合上述第三方面第一种至第三方面第三种可能的实现方式，在第四种可能的实现方式中，根据节点连边的权重对高级语义特征进行卷积处理，得到待检测物体的增强图像特征。
第四方面，提供了一种物体检测装置，该物体检测装置包括用于执行上述第一方面中的方法中的各个模块。
第五方面，提供了一种神经网络的训练装置，该装置包括用于执行上述第三方面中的方法中的各个模块。
第六方面，提供了一种物体检测装置，该装置包括：存储器，用于存储程序；处理器，用于执行所述存储器存储的程序，当所述存储器存储的程序被执行时，所述处理器用于执行上述第一方面中的方法。
第七方面，提供了一种神经网络的训练装置，该装置包括：存储器，用于存储程序；处理器，用于执行所述存储器存储的程序，当所述存储器存储的程序被执行时，所述处理器用于执行上述第三方面中的方法。
第八方面，提供了一种电子设备，该电子设备包括上述第四方面或者第六方面中的物体检测装置。
第九方面，提供了一种电子设备，该电子设备包括上述第五方面或者第七方面中的物体检测装置。
上述电子设备具体可以是移动终端(例如，智能手机)，平板电脑，笔记本电脑，增强现实/虚拟现实设备以及车载终端设备等等。
第十方面，提供一种计算机存储介质，该计算机存储介质存储有程序代码，该程序代码包括用于执行第一方面或者第三方面中的方法中的步骤的指令。
第十一方面，提供一种包含指令的计算机程序产品，当该计算机程序产品在计算机上运行时，使得计算机执行上述第一方面或者第三方面中的方法。
第十二方面，提供一种芯片，所述芯片包括处理器与数据接口，所述处理器通过所述数据接口读取存储器上存储的指令，执行上述第一方面或者第三方面中的方法。
可选地，作为一种实现方式，所述芯片还可以包括存储器，所述存储器中存储有指令，所述处理器用于执行所述存储器上存储的指令，当所述指令被执行时，所述处理器用于执行第一方面中的方法。上述芯片具体可以是现场可编程门阵列FPGA或者专用集成电路 ASIC。
应理解，上述第一方面的方法具体可以是指第一方面以及第一方面中各种实现方式中的任意一种实现方式中的方法。上述第三方面的方法具体可以是指第三方面以及第三方面中各种实现方式中的任意一种实现方式中的方法。
通过本申请提供的技术方案，构建跨域知识图谱，可以捕捉到不同待检测物体间的内在关系，增强图像特征包括不同域中与所述待检测物体相关联的其他物体对应的物体类别的语义信息，因此本申请能够提高物体检测方法的效果。
附图说明
图1是本申请实施例提供的***架构的结构示意图；
图2是利用本申请实施例提供的卷积神经网络模型进行物体检测的示意图；
图3是本申请实施例提供的一种芯片硬件结构示意图；
图4是本申请实施例的物体检测方法的示意性流程图；
图5是本申请实施例的关联关系的示意图；
图6是本申请实施例的物体检测方法的流程图；
图7是本申请实施例的物体检测方法的流程图；
图8是本申请实施例的神经网络的训练方法的示意性流程图；
图9是本申请实施例的物体检测装置的示意性框图；
图10是本申请实施例的物体检测装置的示意性框图；
图11是本申请实施例的神经网络训练装置的示意性框图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。
本申请实施例主要应用在于大规模物体检测的场景中，比如手机人脸识别，手机识别万物，无人车的感知***，安防摄像头，社交网站照片物体识别，智能机器人等等。下面将对几个典型的应用场景进行简要的介绍：
手机识别万物：
利用手机上的摄像头，可以拍摄包含各种事物的图片。在获取图片之后，接下来通过对该图片进行物体检测，能够确定图片中的每个物体的位置和类别。
利用本申请实施例的物体检测方法能够对手机拍摄到的图片进行物体检测，由于本申请实施例的物体检测方法在对物体进行检测时结合了跨域知识图谱，因此，采用本申请实施例的物体检测方法对手机拍摄到的图片进行物体检测时的效果更好(例如，物体的位置以及物体的分类更加准确)。
街景识别：
通过部署在街边的摄像头可以对往来的车辆和人群进行拍照，在获取到图片之后，可以将图片上传到控制中心设备，由控制中心设备对图片进行物体检测，得到物体检测结果，当出现异常的物体时，控制中心可以发出报警。
下面从模型训练侧和模型应用侧对本申请提供的方法进行描述：
本申请实施例提供的神经网络的训练方法，涉及计算机视觉的处理，具体可以应用于 数据训练、机器学习、深度学习等数据处理方法，对训练数据(如本申请中的训练图片以及训练图片的标注结果)进行符号化和形式化的智能信息建模、抽取、预处理、训练等，最终得到训练好的神经网络。
本申请实施例提供的物体检测方法可以运用上述训练好的神经网络，将输入数据(如本申请中的图片)输入到所述训练好的神经网络中，得到输出数据(如本申请中的图片的检测结果)。需要说明的是，本申请实施例提供的神经网络的训练方法和本申请实施例的物体检测方法是基于同一个构思产生的发明，也可以理解为一个***中的两个部分，或一个整体流程的两个阶段：如模型训练阶段和模型应用阶段。
由于本申请实施例涉及到了大量神经网络的应用，为了便于理解，下面先对本申请实施例涉及的相关术语及神经网络等相关概念进行介绍。
(1)神经网络
神经网络可以是由神经单元组成的，神经单元可以是指以xs和截距1为输入的运算单元，该运算单元的输出可以为：
Figure PCTCN2020112796-appb-000001
其中，s＝1、2、……n，n为大于1的自然数，Ws为xs的权重，b为神经单元的偏置。f为神经单元的激活函数(activation functions)，用于将非线性特性引入神经网络中，来将神经单元中的输入信号转换为输出信号。该激活函数的输出信号可以作为下一层卷积层的输入。激活函数可以是sigmoid函数。神经网络是将许多个上述单一的神经单元联结在一起形成的网络，即一个神经单元的输出可以是另一个神经单元的输入。每个神经单元的输入可以与前一层的局部接受域相连，来提取局部接受域的特征，局部接受域可以是由若干个神经单元组成的区域。
(2)深度神经网络
深度神经网络(deep neural network，DNN)，可以理解为具有很多层隐含层的神经网络，这里的“很多”并没有特别的度量标准，我们常说的多层神经网络和深度神经网络其本质上是同一个东西。从DNN按不同层的位置划分，DNN内部的神经网络可以分为三类：输入层，隐含层，输出层。一般来说第一层是输入层，最后一层是输出层，中间的层数都是隐含层。层与层之间是全连接的，也就是说，第i层的任意一个神经元一定与第i+1层的任意一个神经元相连。虽然DNN看起来很复杂，但是就每一层的工作来说，其实并不复杂，简单来说就是如下线性关系表达式：y′＝α(Wx′+b)，其中，
Figure PCTCN2020112796-appb-000002
是输入向量，
Figure PCTCN2020112796-appb-000003
是输出向量，
Figure PCTCN2020112796-appb-000004
是偏移向量，w是权重矩阵(也称系数)，α()是激活函数。每一层仅仅是对输入向量
Figure PCTCN2020112796-appb-000005
经过如此简单的操作得到输出向量
Figure PCTCN2020112796-appb-000006
由于DNN层数多，则系数W和偏移向量
Figure PCTCN2020112796-appb-000007
的数量也就是很多了。那么，具体的参数在DNN是如何定义的呢，首先我们来看看系数W的定义。以一个三层的DNN为例，如：第二层的第4个神经元到第三层的第2个神经元的线性系数定义为
Figure PCTCN2020112796-appb-000008
上标3代表系数W所在的层数，而下标对应的是输出的第三层索引2和输入的第二层索引4。总结下，第L-1层的第k个神经元到第L层的第j个神经元的系数定义为
Figure PCTCN2020112796-appb-000009
注意，输入层是没有W参数的。在深度神经网络中，更多的隐含层让网络更能够刻画现实世界中的复杂情形。理论上而言，参数越多的模型复杂度越高，“容量”也就越大，也就意味着它能完成更复杂的学习任务。
(3)卷积神经网络
卷积神经网络(convolutional neuron network，CNN)是一种带有卷积结构的深度神经网络。卷积神经网络包含了一个由卷积层和子采样层构成的特征抽取器，该特征抽取器可以看作是滤波器。卷积层是指卷积神经网络中对输入信号进行卷积处理的神经元层。在卷积神经网络的卷积层中，一个神经元可以只与部分邻层神经元连接。一个卷积层中，通常包含若干个特征平面，每个特征平面可以由一些矩形排列的神经单元组成。同一特征平面的神经单元共享权重，这里共享的权重就是卷积核。共享权重可以理解为提取图像信息的方式与位置无关。卷积核可以以随机大小的矩阵的形式初始化，在卷积神经网络的训练过程中卷积核可以通过学习得到合理的权重。另外，共享权重带来的直接好处是减少卷积神经网络各层之间的连接，同时又降低了过拟合的风险。
(4)分类器
很多神经网络结构最后都有一个分类器，用于对图像中的物体进行分类。分类器一般由全连接层(fully connected layer)和softmax函数组成，能够根据输入而输出不同类别的概率。
(5)特征金字塔网络(feature pyramid networks，FPN)
原来多数的物体检测算法都是只采用顶层特征做预测，但我们知道低层的特征语义信息比较少，但是目标位置准确；高层的特征语义信息比较丰富，但是目标位置比较粗略。FPN的是在原来的检测器的基础上对在不同特征层独立进行预测。
(6)迁移学习(transfer learning)
迁移学习的初衷是处理训练样本不足的问题，让模型可以通过已有的源域数据(source domain data)向相关但不完全相同的目标域数据(target domain data)迁移，从而训练出适用于目标域的模型。其中的域是一个抽象概念，指代具有相似性质的任务。具体而言，一个域(或称领域)可以是在一个特定的数据集上的检测任务，也可以指针对特定物体(如人脸)的检测任务等等。不同的域之间往往有明显的差异，难以统一处理。全域是指包含所有潜在任务在内的所有的域的统称，是域的全集，一般用于定义与概念表述。迁移学习的核心算法是通过最大化特定的领域相似性度量，提取出具有领域不变性的信息，使得不同领域的数据能互相协同学习，得到适用于目标域的模型。
以上对神经网络的一些基本内容做了简单介绍，下面针对图像数据处理时可能用到的一些特定神经网络进行介绍。
(7)图卷积神经网络
图(graph)是一种数据格式，它可以用于表示社交网络、通信网络、蛋白分子网络等，图中的节点表示网络中的个体，连线表示个体之间的连接关系。许多机器学习任务例如社团发现、链路预测等都需要用到图结构数据，因此图卷积神经网络(graphconvolutional network，GCN)的出现为这些问题的解决提供了新的思路。利用GCN能够对图数据进行深度学习。
GCN是对卷积神经网络在图域(graph domain)上的自然推广。它能同时对节点特征信息与结构信息进行端对端学习，是目前对图数据学习任务的最佳选择。GCN的适用性极广，适用于任意拓扑结构的节点与图。
下面结合图1对本申请实施例的***架构进行详细的介绍。
图1是本申请实施例的***架构的示意图。如图1所示，***架构100包括执行设备110、训练设备120、数据库130、客户设备140、数据存储***150、以及数据采集***160。
另外，执行设备110包括计算模块111、I/O接口112、预处理模块113和预处理模块114。其中，计算模块111中可以包括目标模型/规则101，预处理模块113和预处理模块114是可选的。
数据采集设备160用于采集训练数据。针对本申请实施例的物体检测方法来说，训练数据可以包括不同域或者不同数据集的训练图像以及训练图像对应的标注结果。其中，训练图像的标注结果可以是(人工)预先标注的训练图像中的各个待检测物体的分类结果。在采集到训练数据之后，数据采集设备160将这些训练数据存入数据库130，训练设备120基于数据库130中维护的训练数据训练得到目标模型/规则101。
下面对训练设备120基于训练数据得到目标模型/规则101进行描述，训练设备120对输入的训练图像进行物体检测，将输出的检测结果与物体预先标注的检测结果进行对比，直到训练设备120输出的物体的检测结果与预先标注的检测结果的差异小于一定的阈值，从而完成目标模型/规则101的训练。
上述目标模型/规则101能够用于实现本申请实施例的物体检测方法，即，将待检测图像(通过相关预处理后)输入该目标模型/规则101，即可得到待检测图像的检测结果。本申请实施例中的目标模型/规则101具体可以为神经网络。需要说明的是，在实际的应用中，所述数据库130中维护的训练数据不一定都来自于数据采集设备160的采集，也有可能是从其他设备接收得到的。另外需要说明的是，训练设备120也不一定完全基于数据库130维护的训练数据进行目标模型/规则101的训练，也有可能从云端或其他地方获取训练数据进行模型训练，上述描述不应该作为对本申请实施例的限定。
根据训练设备120训练得到的目标模型/规则101可以应用于不同的***或设备中，如应用于图1所示的执行设备110，所述执行设备110可以是终端，如手机终端，平板电脑，笔记本电脑，增强现实(augmented reality，AR)/虚拟现实(virtual reality，VR)，车载终端等，还可以是服务器或者云端等。在图1中，执行设备110配置输入/输出(input/output，I/O)接口112，用于与外部设备进行数据交互，用户可以通过客户设备140向I/O接口112输入数据，所述输入数据在本申请实施例中可以包括：客户设备输入的待处理图像。这里的客户设备140具体可以是终端设备。
预处理模块113和预处理模块114用于根据I/O接口112接收到的输入数据(如待处理图像)进行预处理，在本申请实施例中，也可以没有预处理模块113和预处理模块114(也可以只有其中的一个预处理模块)，而直接采用计算模块111对输入数据进行处理。
在执行设备110对输入数据进行预处理，或者在执行设备110的计算模块111执行计算等相关的处理过程中，执行设备110可以调用数据存储***150中的数据、代码等以用于相应的处理，也可以将相应处理得到的数据、指令等存入数据存储***150中。
最后，I/O接口112将处理结果，如上述得到的物体的检测结果呈现给客户设备140，从而提供给用户。
值得说明的是，训练设备120可以针对不同的目标或称不同的任务，基于不同的训练数据生成相应的目标模型/规则101，该相应的目标模型/规则101即可以用于实现上述目标或完成上述任务，从而为用户提供所需的结果。
在图1中所示情况下，用户可以手动给定输入数据，该手动给定可以通过I/O接口112提供的界面进行操作。另一种情况下，客户设备140可以自动地向I/O接口112发送输入数据，如果要求客户设备140自动发送输入数据需要获得用户的授权，则用户可以在客户设备140中设置相应权限。用户可以在客户设备140查看执行设备110输出的结果，具体的呈现形式可以是显示、声音、动作等具体方式。客户设备140也可以作为数据采集端，采集如图所示输入I/O接口112的输入数据及输出I/O接口112的输出结果作为新的样本数据，并存入数据库130。当然，也可以不经过客户设备140进行采集，而是由I/O接口112直接将如图所示输入I/O接口112的输入数据及输出I/O接口112的输出结果，作为新的样本数据存入数据库130。
值得注意的是，图1仅是本申请实施例提供的一种***架构的示意图，图中所示设备、器件、模块等之间的位置关系不构成任何限制，例如，在图1中，数据存储***150相对执行设备110是外部存储器，在其它情况下，也可以将数据存储***150置于执行设备110中。
如图1所示，根据训练设备120训练得到目标模型/规则101，在本申请实施例中可以是本申请中的神经网络，具体的，本申请实施例提供的神经网络可以CNN以及深度卷积神经网络(deep convolutional neural networks,DCNN)等等。
由于CNN是一种非常常见的神经网络，下面结合图2重点对CNN的结构进行详细的介绍。如上文的基础概念介绍所述，卷积神经网络是一种带有卷积结构的深度神经网络，是一种深度学习(deep learning)架构，深度学习架构是指通过机器学习的算法，在不同的抽象层级上进行多个层次的学习。作为一种深度学习架构，CNN是一种前馈(feed-forward)人工神经网络，该前馈人工神经网络中的各个神经元可以对输入其中的图像作出响应。
如图2所示，卷积神经网络(CNN)200可以包括输入层210，卷积层/池化层220(其中池化层为可选的)，以及神经网络层230。下面对这些层的相关内容做详细介绍。
卷积层/池化层220：
卷积层：
如图2所示卷积层/池化层220可以包括如示例221-226层，举例来说：在一种实现中，221层为卷积层，222层为池化层，223层为卷积层，224层为池化层，225为卷积层，226为池化层；在另一种实现方式中，221、222为卷积层，223为池化层，224、225为卷积层，226为池化层。即卷积层的输出可以作为随后的池化层的输入，也可以作为另一个卷积层的输入以继续进行卷积操作。
下面将以卷积层221为例，介绍一层卷积层的内部工作原理。
卷积层221可以包括很多个卷积算子，卷积算子也称为核，其在图像处理中的作用相当于一个从输入图像矩阵中提取特定信息的过滤器，卷积算子本质上可以是一个权重矩阵，这个权重矩阵通常被预先定义，在对图像进行卷积操作的过程中，权重矩阵通常在输入图像上沿着水平方向一个像素接着一个像素(或两个像素接着两个像素……这取决于步长 stride的取值)的进行处理，从而完成从图像中提取特定特征的工作。该权重矩阵的大小应该与图像的大小相关，需要注意的是，权重矩阵的纵深维度(depth dimension)和输入图像的纵深维度是相同的，在进行卷积运算的过程中，权重矩阵会延伸到输入图像的整个深度。因此，和一个单一的权重矩阵进行卷积会产生一个单一纵深维度的卷积化输出，但是大多数情况下不使用单一权重矩阵，而是应用多个尺寸(行×列)相同的权重矩阵，即多个同型矩阵。每个权重矩阵的输出被堆叠起来形成卷积图像的纵深维度，这里的维度可以理解为由上面所述的“多个”来决定。不同的权重矩阵可以用来提取图像中不同的特征，例如一个权重矩阵用来提取图像边缘信息，另一个权重矩阵用来提取图像的特定颜色，又一个权重矩阵用来对图像中不需要的噪点进行模糊化等。该多个权重矩阵尺寸(行×列)相同，经过该多个尺寸相同的权重矩阵提取后的卷积特征图的尺寸也相同，再将提取到的多个尺寸相同的卷积特征图合并形成卷积运算的输出。
这些权重矩阵中的权重值在实际应用中需要经过大量的训练得到，通过训练得到的权重值形成的各个权重矩阵可以用来从输入图像中提取信息，从而使得卷积神经网络200进行正确的预测。
当卷积神经网络200有多个卷积层的时候，初始的卷积层(例如221)往往提取较多的一般特征，该一般特征也可以称之为低级别的特征；随着卷积神经网络200深度的加深，越往后的卷积层(例如226)提取到的特征越来越复杂，比如高级别的语义之类的特征，语义越高的特征越适用于待解决的问题。
池化层：
由于常常需要减少训练参数的数量，因此卷积层之后常常需要周期性的引入池化层，在如图2中220所示例的221-226各层，可以是一层卷积层后面跟一层池化层，也可以是多层卷积层后面接一层或多层池化层。在图像处理过程中，池化层的唯一目的就是减少图像的空间大小。池化层可以包括平均池化算子和/或最大池化算子，以用于对输入图像进行采样得到较小尺寸的图像。平均池化算子可以在特定范围内对图像中的像素值进行计算产生平均值作为平均池化的结果。最大池化算子可以在特定范围内取该范围内值最大的像素作为最大池化的结果。另外，就像卷积层中用权重矩阵的大小应该与图像尺寸相关一样，池化层中的运算符也应该与图像的大小相关。通过池化层处理后输出的图像尺寸可以小于输入池化层的图像的尺寸，池化层输出的图像中每个像素点表示输入池化层的图像的对应子区域的平均值或最大值。
神经网络层230：
在经过卷积层/池化层220的处理后，卷积神经网络200还不足以输出所需要的输出信息。因为如前所述，卷积层/池化层220只会提取特征，并减少输入图像带来的参数。然而为了生成最终的输出信息(所需要的类信息或其他相关信息)，卷积神经网络200需要利用神经网络层230来生成一个或者一组所需要的类的数量的输出。因此，在神经网络层230中可以包括多层隐含层(如图2所示的231、232至23n)以及输出层240，该多层隐含层中所包含的参数可以根据具体的任务类型的相关训练数据进行预先训练得到，例如该任务类型可以包括图像识别，图像分类，图像超分辨率重建等等。
在神经网络层230中的多层隐含层之后，也就是整个卷积神经网络200的最后层为输 出层240，该输出层240具有类似分类交叉熵的损失函数，具体用于计算预测误差，一旦整个卷积神经网络200的前向传播(如图2由210至240方向的传播为前向传播)完成，反向传播(如图2由240至210方向的传播为反向传播)就会开始更新前面提到的各层的权重值以及偏差，以减少卷积神经网络200的损失，及卷积神经网络200通过输出层输出的结果和理想结果之间的误差。
需要说明的是，如图2所示的卷积神经网络200仅作为一种卷积神经网络的示例，在具体的应用中，卷积神经网络还可以以其他网络模型的形式存在。
应理解，可以采用图2所示的卷积神经网络(CNN)200执行本申请实施例的物体检测方法，如图2所示，待处理图像经过输入层210、卷积层/池化层220和神经网络层230的处理之后可以得到图像的检测结果。
图3为本申请实施例提供的一种芯片硬件结构，该芯片包括神经网络处理器。该芯片可以被设置在如图1所示的执行设备110中，用以完成计算模块111的计算工作。该芯片也可以被设置在如图1所示的训练设备120中，用以完成训练设备120的训练工作并输出目标模型/规则101。如图2所示的卷积神经网络中各层的算法均可在如图3所示的芯片中得以实现。
神经网络处理器NPU作为协处理器挂载到主中央处理器(central processing unit，CPU)(host CPU)上，由主CPU分配任务。NPU的核心部分为运算电路303，控制器304控制运算电路303提取存储器(权重存储器或输入存储器)中的数据并进行运算。
在一些实现中，运算电路303内部包括多个处理单元(process engine,PE)。在一些实现中，运算电路303是二维脉动阵列。运算电路303还可以是一维脉动阵列或者能够执行例如乘法和加法这样的数学运算的其它电子线路。在一些实现中，运算电路303是通用的矩阵处理器。
举例来说，假设有输入矩阵A，权重矩阵B，输出矩阵C。运算电路303从权重存储器302中取矩阵B相应的数据，并缓存在运算电路303中每一个PE上。运算电路303从输入存储器301中取矩阵A数据与矩阵B进行矩阵运算，得到的矩阵的部分结果或最终结果，保存在累加器(accumulator)308中。
向量计算单元307可以对运算电路303的输出做进一步处理，如向量乘，向量加，指数运算，对数运算，大小比较等等。例如，向量计算单元307可以用于神经网络中非卷积/非FC层的网络计算，如池化(pooling)，批归一化(batch normalization)，局部响应归一化(local response normalization)等。
在一些实现中，向量计算单元能307将经处理的输出的向量存储到统一缓存器306。例如，向量计算单元307可以将非线性函数应用到运算电路303的输出，例如累加值的向量，用以生成激活值。在一些实现中，向量计算单元307生成归一化的值、合并值，或二者均有。在一些实现中，处理过的输出的向量能够用作到运算电路303的激活输入，例如用于在神经网络中的后续层中的使用。
统一存储器306用于存放输入数据以及输出数据。
权重数据直接通过存储单元访问控制器305(direct memory accesscontroller，DMAC)将外部存储器中的输入数据搬运到输入存储器301和/或统一存储器306、将外部存储器中 的权重数据存入权重存储器302，以及将统一存储器306中的数据存入外部存储器。
总线接口单元(bus interface unit，BIU)310，用于通过总线实现主CPU、DMAC和取指存储器309之间进行交互。
与控制器304连接的取指存储器(instruction fetch buffer)309，用于存储控制器304使用的指令。
控制器304，用于调用指存储器309中缓存的指令，实现控制该运算加速器的工作过程。
一般地，统一存储器306，输入存储器301，权重存储器302以及取指存储器309均为片上(On-Chip)存储器，外部存储器为该NPU外部的存储器，该外部存储器可以为双倍数据率同步动态随机存储器(double data rate synchronous dynamic random accessmemory，简称DDR SDRAM)、高带宽存储器(high bandwidth memory，HBM)或其他可读可写的存储器。
其中，图2所示的卷积神经网络中各层的运算可以由运算电路或向量计算模块307执行。
上文中介绍的图1中的执行设备110能够执行本申请实施例的物体检测方法的各个步骤，图2所示的CNN模型和图3所示的芯片也可以用于执行本申请实施例的物体检测方法的各个步骤。下面结合附图对本申请实施例的物体检测方法进行详细的介绍。
下面结合图4对本申请实施例的物体检测方法进行详细的介绍。
401、获取待检测图像。
图4所示的方法可以应用在不同的场景下，具体地，图4所示方法可以应用在识别万物以及街景识别等场景中。
当图4所示的方法应用在移动终端识别万物的场景时，步骤401中的待检测图像可以是移动终端通过摄像头拍摄的图像，也可以是移动终端相册中已经存储的图像。
当图4所示的方法应用在街景识别的场景时，步骤401中的待检测图像可以是路边的摄像头拍摄的街景图像。
图4所示的方法可以由神经网络(模型)来执行，具体地，图4所示的方法可以由CNN或者DNN来执行。
402、确定待检测图像中的待检测物体的初始图像特征。
在步骤402中，可以先对待检测图像的整个图像进行卷积处理或者正则化处理等，以得到整个图像的图像特征，然后再从该整个图像的图像特征中获取待检测物体对应的初始图像特征。
在一个具体的实施方式中，上述对待检测图像进行卷积处理，得到待检测物体的初始图像特征，包括：对待检测图像的整个图像进行卷积处理，得到待检测图像的完整图像特征；将该待检测图像的完整图像特征中与待检测物体对应的图像特征确定为待检测物体的初始图像特征。
在一个具体的实施方式中，上述对待检测图像进行卷积处理，得到待检测物体的初始图像特征，包括：每次单独获取各个待检测物体对应的图像特征。
403、根据跨域知识图谱信息确定待检测物体的增强图像特征。
跨域知识图谱信息包括不同域中待检测物体对应的物体类别之间的关联关系，增强图 像特征指示不同域中与待检测物体相关联的其他物体对应的物体类别的语义信息。
两个类别在同一个图像中同时出现的概率越大，则认为两个类别之间有关联关系。举例说明，如图5中所示，第一个域中或者说第一个数据集中的物体类别包括男人，女人，男孩，女孩，马路，街道。第二个域中的物体类别包括人，手提包，书包，汽车，卡车。可以认为第一个域中的男人，女人，男孩，女孩与第二个域中的人之间具有关联关系。第一个域中的女人和女孩与第二个域中的手提包具有关联关系。第一个域中的男孩和女孩与第二个域中的书包具有关联关系。第一个域中的马路和街道与第二个域中的汽车、卡车之间具有关联关系。
语义信息可以是指能够辅助进行图像检测的高级别的信息。例如，上述语义信息具体可以是物体是什么，物体的周围有什么(语义信息一般不同于低级别的信息，如图像的边，像素点和亮度等等)。
例如，待检测物体为女人，待检测图像中与女人相关联的其他物体包括手提包，那么，上述待检测物体的增强图像特征指示的可以是手提包的语义信息。
在一个具体的实施方式中，跨域知识图谱可以包括节点和节点连边，其中节点对应待检测物体，节点连边对应不同待检测物体的高级语义特征之间的关系。根据不同域中初始图像特征在不同物体类别上的分类权重，将不同域对应的分类层参数加权融合，得到待检测物体的高级语义特征，分类层参数可以理解成是在维护该类别的一个类中心，类中心是指类别的高级语义特征。将不同域中待检测物体对应的物体类别之间的关系权重投影到待检测物体的节点连边上，得到节点连边的权重。下面对这一过程进行解释：用ε S-P代表节点连边的权重，S和P分别代表两个域，cM S和M P代表各自域内的待检测物f i体在对应域的物体类别上的分类权重构成的矩阵,以M S为例进行解释，M S的第i行第j列的元素为
Figure PCTCN2020112796-appb-000010
s ij是第i个待检测物体的初始图像特征和分类器第j个分类类别对应的参数的内积。S域内区域图的第i个节点和第j个节点之间连边的权重为
Figure PCTCN2020112796-appb-000011
其中和Cf j为一个域中第i个待检测物体的特征和另一个域中第j个待检测物体的特征。G S-P是不同域中待检测物体对应的物体类别之间的关系权重，可以将G S-P看作一个矩阵。将不同域中待检测物体对应的物体类别之间的关系权重投影到待检测物体的节点连边上，得到节点连边的权重，可以用如下公式进行表示，其中T代表矩阵的转置：
Figure PCTCN2020112796-appb-000012
可以将投影的过程看作将物体类别之间的关系权重转换成待检测物体之间的关系权重的过程，待检测物体之间的关系权重即为节点连边的权重。
在一个具体的实施方式中，根据节点连边的权重对高级语义特征进行卷积处理，可以得到待检测物体的增强图像特征。
在一个具体的实施方式中，可以根据不同域中待检测物体对应的物体类别间的距离关系确定关系权重。距离关系包括以下信息中的一种或几种：
(1)不同域中不同物体类别的属性关系。
例如，苹果的颜色是红色，草莓的颜色也是红色，那么，苹果和草莓在颜色上具有相同的属性(或者，也可以说苹果和草莓在颜色属性上比较接近)。
(2)不同域中不同物体类别的位置关系或者主动宾关系。
例如，街道上的汽车，女人提着手提包，那么街道和汽车之间的位置接近，女人和手提包满足主动宾关系。
(3)不同域中不同物体类别的利用语言学知识构建的词嵌入相似度。
利用语言学知识构建的词嵌入相似度可以理解为不同物体类别的词向量之间的相似程度。
(4)根据训练数据对神经网络模型进行训练得到的不同域中不同待检测物体的特征之间的距离关系。
例如，对于两个不同的域，其中一个域中的第i个节点和另一个域中的第j个节点之间的连边的权重为
Figure PCTCN2020112796-appb-000013
其中f i和f j为一个域中第i个待检测物体的特征和另一个域中第j个待检测物体的特征(待检测物体的初始图像特征的简称)。需要说明的是，在这种情况下，即根据训练数据对神经网络模型进行训练得到的不同域中不同待检测物体的特征之间的距离关系时，由于已经获得了待检测物体之间的关系权重，所以在这种情况下，不再需要投影的过程。
404、根据待检测物体的初始图像特征和待检测物体的增强图像特征，确定待检测物体的候选框和分类。
步骤404确定得到的待检测物体的候选框和分类可以分别是待检测物体的最终候选框和最终分类(结果)。
在步骤404中，可以先将待检测物体的初始图像特征和待检测物体的增强图像特征组合起来得到待检测物体的最终图像特征，然后再根据待检测物体的最终图像特征确定待检测物体的候选框和分类。
例如，待检测物体的初始图像特征为一个大小为M1×N1×C1(M1、N1和C1可以分别表示宽、高以及通道数)的卷积特征图，待检测物体的增强图像特征是一个大小为M1×N1×C2(M1、N1和C2分别表示宽、高以及通道数)的卷积特征图，那么，通过对这两个卷积特征图的组合，可以得到待检测物体的最终图像特征，该最终图像特征是一个大小为M1×N1×(C1+C2)卷积特征图。
应理解，这里是以初始图像特征的卷积特征图与增强图像特征的卷积特征图的尺寸相同(宽和高相同)，但是通道数不同为例进行了说明。事实上，当初始图像特征的卷积特征图与增强图像特征的卷积特征图的尺寸不同时，也可以对初始图像特征和增强图像特征进行组合，此时，可以先将初始图像特征的卷积特征图与增强图像特征的卷积特征图的尺寸统一(将宽和高统一)，然后再将初始图像特征的卷积特征图与增强图像特征的卷积特征图进行组合，得到最终图像特征的卷积特征图。
本申请中，在对待检测图像进行物体检测时，通过待检测物体的初始图像特征和增强图像特征来综合确定待检测物体的检测结果，与仅考虑待检测物体的初始图像特征获取检测结果的方式相比，能够得到更好的检测结果。
具体地，本申请在确定待检测物体的检测结果时，不仅考虑到了反映待检测物体本身特性的初始图像特征，还考虑到了待检测图像中与待检测物体相关联的其他物体的语义信息。本发明通过构建跨域知识图谱(也可以称为多场景下可迁移知识图谱)，可以捕捉到 不同物体间的内在关系，利用图卷积网络可融合大量不同数据集、不同类别的信息，极大提升了数据利用率，使得检测性能更高，真正实现了大规模物体检测。
例如，只通过上述提到的第二个域训练的模型，在确定检测检测结果可能确定为人和手提包，假如通过本申请提供的方案，通过第一个域和第二个域第二个训练的模型检测，在确定检测结果可能为女人提着手提包。进而最终提高物体检测的效果。
在一个具体的实施方式中，在上述步骤401之后，图4所示的方法还包括：根据待检测物体的初始图像特征确定待检测物体的初始候选框。
在确定待检测物体的初始候选框过程中，一般是先对待检测图像的整个图像进行卷积处理，得到待检测图像的整个图像的卷积特征，然后再根据固定的尺寸要求，将待检测图像划分成不同的方框，对每个方框内的图像对应的特征进行打分，将打分较高的方框筛选出来作为初始候选框。
例如，待检测图像为第一图像，为了获得第一图像中的待检测物体的初始候选框，可以先对第一图像的整个图像进行卷积处理，得到第一图像的整个图像的卷积特征，然后将第一图像划分成3×3个方框，对每个方框的图像对应的特征进行打分。最后可以将打分较高的方框A和方框B筛选出来作为初始候选框。
在上述步骤404中根据待检测物体的初始图像特征和待检测物体的增强图像特征，确定待检测物体的候选框和分类的过程中，可以是先对初始图像特征和增强图像特征进行组合，得到最终图像特征，然后再根据该最终图像特征对初始候选框进行调整得到候选框，根据该最终图像特征对初始分类结果进行修正，得到分类结果。具体地，上述根据最终图像特征对初始候选框进行调整可以是根据该最终图像特征对初始候选框的四周的坐标进行调整，直到得到候选框，上述根据最终图像特征对初始分类结果进行调整可以是建立一个分类器进行重新分类，进而得到分类结果。
为了更好地理解本申请实施例的物体检测方法的完整流程，下面结合图6对本申请实施例的物体检测方法进行说明。
图6是本申请实施例的物体检测方法的示意性流程图。
图6所示的方法可以由物体检测装置执行，该物体检测装置可以是具有物体检测功能的电子设备。该电子设备具体包含的装置的形态可以如上文介绍图4所示的方法中的相关描述。
图6所示的方法包括步骤601至步骤609，下面对这些步骤进行详细的介绍。
其中，步骤602和步骤603可以是步骤402的细化实施方式(或者称为具体实施方式)，步骤604至608可以是步骤403的细化实施方式(或者称为具体实施方式)。
601、获取待检测图像。
步骤601可以参阅图4对应的实施例中的步骤401进行理解，此处不再重复赘述。
602、选定初始候选区域。
可以将待检测图像输入到一个传统的物体检测器中进行处理(如Faster-RCNN)，得到初始的候选区域。由于本申请针对多个不同的域进行物体检测，所以每一个域都有各自对应的初始的候选区域。
具体地，可以先对待检测图像进行卷积处理，得到待检测图像的全图的卷积特征，然 后再根据一定的尺寸要求，将待检测图像划分成不同的方框，然后针对不同的域，对每个方框内的图像对应的特征进行打分，将打分较高的方框筛选出来作为初始候选框，由此得到不同的域对应的初始候选框。
603、提取初始候选区域的初始图像特征。
可以通过CNN来提取初始候选区域的图像特征。例如，第一图像为待检测图像，那么，为了得到第一图像中的待检测物体的初始候选框，可以先对该第一图像进行卷积处理，得到该第一图像的卷积特征，然后将该第一图像划分成4×4个方框(也可以划分成其它数量的方框)，对每个方框的图像对应的特征进行打分，将打分较高的方框A和方框B筛选出来作为初始候选框。
进一步的，在获取了初始候选框之后，还可以待检测图像的整个图像的图像特征(可以通过对待检测图像的整个图像进行卷积处理来得到待检测图像的整个图像的图像特征)对应到方框A和方框B中，以获取方框A对应的初始图像特征和方框B对应的初始图像特征。
604、提取分类层参数。
可以使用物体检测器中的对应不同域的分类器提取分类层参数，比如对于每一个域可以采用物体检测器(如Faster-RCNN)中的分类器提取分类层参数，对每一个域分别构建一个与域相关的语义池记录每个类别的高级语义特征，在训练过程中，分类器中的不同分类对应的分类层参数可能会不断的发生变化，在这种情况下，可以对语义池中分类对应的分类层参数进行更新。提取的分类层参数可以是对待检测物体进行物体检测的物体检测器中的不同域对应的分类器中的所有分类的分类层参数。
605、构建域内区域图。
根据检测网络给出的待检测物体的初始图像特征在不同物体类别上的分类权重，将该域对应的语义池中的高级语义特征，映射到域内区域图的节点上，得到待检测物体的高级语义表示。根据域内不同待检测物体对应的物体类别之间的关系权重给出域内区域图节点连边上的权重。
具体的，语义池为
Figure PCTCN2020112796-appb-000014
即分类器的参数，其中C T为类别个数，D为每一个类别对应的分类器权重的维数，通过X＝M TP T映射到域内区域图的节点上，得到待检测物体的高级语义表示，其中M T的第i行第j列的元素为
Figure PCTCN2020112796-appb-000015
s ij是第i个待检测物体的初始图像特征和分类器第j个分类类别对应的参数的内积。域内区域图第i个节点和第j个节点之间连边的权重为
Figure PCTCN2020112796-appb-000016
其中f i和f i为一个域中第i个待检测物体的特征和另一个域中第j个待检测物体的特征。
对于每一个域都可以分别按照上述方法分别构建一个域内区域图。
606、构建域间区域图。
根据检测网络给出的各自域下的待检测物体的初始图像特征在不同类别上的分类权重，将语义池的高级语义特征，映射到域间区域图的节点上，得到待检测物体的高级语义表示。根据两个不同域下的检测物体分类类别特征之间的距离给出类别之间的关系权重，再投影到域间区域图节点连边，得到域间区域图节点连边的权重。步骤606中的距离可以 参照图4对应的实施例中的对距离的解释进行理解，此处不再重复赘述。
对于两个不用的域，域间区域图节点上的特征构造方式与域内区域图一样，域间区域图一个域中第i个节点和另一个域中第j个节点之间连边的权重为
Figure PCTCN2020112796-appb-000017
其中f i和f i为一个域中第i个待检测物体的特征和另一个域中第j个待检测物体的特征。
607、域内图卷积网络推理推断。
通过构建的域内区域图，利用域内图卷积网络传播节点上不同待检测物体的高级语义表示，得到经过推理推断后的融合了其它待检测物体高级语义表示的特征。
具体的，可以选择空间信息机制的图卷积。待检测物体之间的相对空间信息被用来学习K个高斯核，具体公式为：
Figure PCTCN2020112796-appb-000018
其中ω(k)是第k个高斯核，μ k和∑ k是可学习的均值向量和协方差向量，g ij表示第i个和第j个待检测物体之间的相对空间关系，具体公式为：
Figure PCTCN2020112796-appb-000019
其中x i和x j是X的第i行和第j行，w i，w j，h i和h j是第i个和第j个待检测物体候选框的宽度和高度。每个图卷积的输出为：
f′ k(i)＝∑ j∈邻接节点(i)ω k(g ij)x je ij。
域内图卷积在每个节点上得到的K个特征会被融合到对应的待检测物体的高级语义表示中。
608、域间图卷积网络推理推断。
通过构建的域间区域图，利用域间图卷积网络传播节点上不同域下待检测物体的高级语义表示，得到经过推理推断后的融合了不同域下待检测物体高级语义表示的特征。
609、根据待检测物体的初始图像特征和待检测物体的增强图像特征，确定待检测物体的候选框和分类。
将域内图卷积、域间图卷积经过推理推断后得到的特征投影到相应的待检测物体的高级语义表示中，并进行分类和回归。
步骤609可以参照图4对应的实施例中的步骤404进行理解，此处不再重复赘述。
上文结合流程图对本申请实施例的物体检测方法进行了详细的说明，为了更好地理解本申请实施例的物体检测方法，下面结合更具体的流程图对本申请实施例的物体检测方法进行详细的描述。
图7是本申请实施例的物体检测方法的示意性流程图。
图7所示的方法可以由物体检测装置执行，该物体检测装置可以是具有物体检测功能的电子设备。电子设备具体包含的装置的形态可以如上文介绍图4所示的方法中的相关描 述。
步骤1：输入图片，经过一个传统的物体检测器，得到一个初步的备选框以及待检测物体的特征。
步骤2：使用物体检测器中的对应不同域的分类器提取分类层参数，对每一个域分别构建一个域相关的语义池记录每个类别的高级语义特征。这个语义池会在训练过程中随着分类器的优化而不断更新。
步骤3：构建域内区域图：根据检测网络给出的待检测物体特征在不同类别上的分类权重，将语义池的高级语义特征，映射到域内区域图的节点上，得到待检测物体的高级语义表示。根据不同待检测物体特征之间的关系给出域内区域图节点连边上的权重。
步骤4构建域间区域图：根据检测网络给出的各自域下的待检测物体特征在不同类别上的分类权重，将语义池的高级语义特征，映射到域间区域图的节点上，得到待检测物体的高级语义表示。根据两个不同域下的检测物体分类类别特征之间的距离给出类别之间的关系权重，再投影到域间区域图节点连边，得到域间区域图节点连边的权重。
步骤5：域内图卷积：通过构建的域内区域图，利用域内图卷积网络传播节点上不同待检测物体的高级语义表示，得到经过推理推断后的融合了其它待检测物体高级语义表示的特征。通过学习一个稀疏的区域图以对不同待检测物体的高级语义表示进行融合，从而增强不同待检测物体的特征表达能力。
步骤6：域间图卷积：通过构建的域间区域图，利用域间图卷积网络传播节点上不同域下待检测物体的高级语义表示，得到经过推理推断后的融合了不同域下待检测物体高级语义表示的特征。
步骤7：优化增强候选区域特征层：将域内图卷积、域间图卷积经过推理推断后得到的特征投影到相应的待检测物体的高级语义表示中，并进行分类和回归，以达到提高大规模检测性能的目的。
上文结合附图对本申请实施例的物体检测方法进行了详细的介绍，为了更好的说明本申请实施例的物体检测方法的有益效果，下面结合表1和表2以具体的实例对本申请实施例的物体检测方法相对于现有的物体检测方法的效果进行详细的说明。
下面结合具体的实验数据，以表1为例，对现有的几种物体检测的方法与本申请提供的方法的物体检测的效果进行比较说明。表1中所示的第一种方法为FPN检测方法，第二种方法为多分支的检测方法(Multi Branches)。训练模型用的数据集包括MSCOCO数据集，视觉基因组(visual genome，VG)数据集和ADE数据集三种数据集，即采用三种数据集共同训练模型，测试阶段，对于每一个数据集分别进行测试。其中，MSCOCO数据集拥有80个通用物体的检测标注，含有约11万训练数据集以及5千张测试集。VG数据集共有1000的大规模通用物体检测数据集，8.8万张图片的训练数据集以及数量为5000的测试集。而ADE数据集拥有445类的大规模通用物体检测数据集，2万的张图片的训练数据集以及1千张的测试集。
在对物体检测效果进行评价时，主要采用了平均精确度(average precision，AP)和平均召回率(average recall，AR)进行评价，在对比时考虑了不同阈值下的精确度，与不同大小的物体的平均精确度和平均召回率。
如表1所示，通过MSCOCO数据集、VG数据集以及ADE数据集，三个数据集共同进行模型训练，在不同的数据集下进行测试时，本申请方法的AP和AR都要分别大于第一种方法以及第二种方法的AP和AR，而AP和AR的数值越大说明进行物体检测的效果越好。由表1可知，本申请方法相对于现有的几种物体检测的方法有比较明显的效果提升。
表1：
Figure PCTCN2020112796-appb-000020
表1中训练模型采用了三种数据集进行训练，为了更好的体现本申请提供的方法所带来的有益效果，下面结合表2对训练模型采用两个数据集进行训练时，本申请提供的方法与现有的几种物体检测方法的效果进行比较说明。除了上述提到的几种物体检测的方法，当采用两个数据集进行模型训练时，还可以包括其他几种物体检测方法，比如第三种方法：微调检测方法(fine-tuning)，第四种方法：重叠标签检测方法(overlap labels)，第五种方法：伪标签检测方法(pseudo labels)。
如表2所示，通过MSCOCO数据集、VG数据集以及ADE数据集，三个数据集中的任意两个数据集共同进行模型训练，在不同的数据集下进行测试时，本申请方法的AP和AR都要分别大于第一种物体检测方法至第六种物体检测方法的AP和AR，而AP和AR的数值越大说明进行物体检测的效果越好。由表2可知，本申请方法相对于现有的几种物体检测的方法有比较明显的效果提升。
表2：
Figure PCTCN2020112796-appb-000021
本申请提提供的方法在有严重物体遮挡，类别模糊不清，小尺度物体等情形下，检测效果有明显提升。相比于其它的域迁移物体检测方法，我们的方法通过构建多域下可迁移知识图谱，有效地捕捉到了不同物体间的内在关系，利用图卷积网络融合了大量不同数据集、不同类别的信息，极大提升了数据利用率，使得检测性能更高，真正实现了大规模物体检测。
图8是本申请实施例的神经网络的训练方法的示意性流程图。图8所示的方法可以由计算机设备、服务器设备或者运算设备等运算能力较强的设备来执行。下面进行详细的介绍。
801、获取训练数据。
训练数据包括不同域的训练图像以及训练图像中待检测物体的物体检测标注结果。
802、根据神经网络提取该训练图像中的待检测物体的初始图像特征。
803、根据该神经网络以及跨域知识图谱信息提取该训练图像中的待检测物体的增强图像特征。
跨域知识图谱信息包括不同域中待检测物体对应的物体类别之间的关联关系，增强图像特征指示不同域中与待检测物体相关联的其他物体对应的物体类别的语义信息。
804、根据该神经网络对待检测物体的初始图像特征和增强图像特征进行处理，得到该待检测物体的物体检测结果。
805、根据该训练图像的中的待检测物体的物体检测结果与该训练图像中的待检测物体的物体检测标注结果，确定该神经网络的模型参数。
可选地，上述训练图像中的待检测物体的物体检测标注结果包括该训练图像中的待检测物体的标注候选框和标注分类结果。
另外，在上述训练的过程中，可以采用多个不同的域或者不同的数据集，采用的训练图像一般是多个。
在对上述神经网络进行训练的过程中，可以为神经网络设置一套初始的模型参数，然后根据训练图像中的待检测物体的物体检测结果与训练图像中的待检测物体的物体检测标注结果的差异来逐渐调整神经网络的模型参数，直到训练图像中的待检测物体的物体检测结构与训练图像中的待检测物体的物体检测标注结果之间的差异在一定的预设范围内，或者，当训练的次数达到预设次数时，将此时的神经网络的模型参数确定为该神经网络模型的最终的参数，这样就完成了对神经网络的训练了。
应理解，通过图8所示的方法训练得到的神经网络能够用于执行本申请实施例的物体检测方法。
本申请中，在训练神经网络时，不仅提取了训练图像中的待检测物体的初始图像特征，还提取了训练图像中的待检测物体的增强图像特征，并综合根据待检测物体的初始图像特征和增强图像特征来确定待检测物体的物体检测结果。也就是说，本申请的训练方法在训练过程中提取了更多的特征来进行物体检测，可以训练得到性能更好的神经网络，使得利用该神经网络进行物体检测能够取得更好的物体检测效果。
在一个具体的实施方式中，跨域知识图谱可以包括节点和节点连边，其中节点对应待检测物体，节点连边对应不同待检测物体的高级语义特征之间的关系。根据不同域中初始图像特征在不同物体类别上的分类权重，将不同域对应的分类层参数加权融合，得到待检测物体的高级语义特征，分类层参数可以理解成是在维护该类别的一个类中心。将不同域中待检测物体对应的物体类别之间的关系权重投影到待检测物体的节点连边上，得到节点连边的权重。
在一个具体的实施方式中，根据节点连边的权重对高级语义特征进行卷积处理，可以得到待检测物体的增强图像特征
在一个具体的实施方式中，可以根据不同域中待检测物体对应的物体类别间的距离关系确定关系权重。距离关系包括以下信息中的一种或几种：
(1)不同域中不同物体类别的属性关系。
例如，苹果的颜色是红色，草莓的颜色也是红色，那么，苹果和草莓在颜色上具有相同的属性(或者，也可以说苹果和草莓在颜色属性上比较接近)。
(2)不同域中不同物体类别的位置关系或者主动宾关系。
例如，街道上的汽车，女人提着手提包，那么街道和汽车之间的位置接近，女人和手提包满足主动宾关系。
(3)不同域中不同物体类别的利用语言学知识构建的词嵌入相似度。
利用语言学知识构建的词嵌入相似度可以理解为不同物体类别的词向量之间的相似程度。
(4)不同域中不同物体类别的根据训练数据对神经网络模型进行训练得到的距离关系。
例如，对于两个不同的域，其中一个域中的第i个节点和另一个域中的第j个节点之间的连边的权重为
Figure PCTCN2020112796-appb-000022
其中f i和f j为一个域中第i个待检测物体的特征和另一个域中第j个待检测物体的特征(待检测物体的初始图像特征的简称)。
上文结合附图对本申请实施例的物体检测方法和神经网络训练方法进行了详细的描述，下面结合图9至图11对本申请实施例的相关装置进行详细的介绍。应理解，图9和图10所示的物体检测装置能够执行本申请实施例的物体检测方法的各个步骤，图11所示的神经网络训练装置能够执行本申请实施例的神经网络训练方法的各个步骤，下面在介绍图9至图11所示的装置时适当省略重复的描述。
图9是本申请实施例的物体检测装置的示意性框图。图9所示的物体检测装置7000包括：
图像获取模块901，用于执行图4对应的实施例中的步骤401，图6对应的实施例中的步骤601。
特征提取模块902，用于执行图4对应的实施例中的步骤402，图6对应的实施例中的步骤602，图6对应的实施例中的步骤603，图6对应的实施例中的步骤607，图6对应的实施例中的步骤608。
检测模块903，用于执行图4对应的实施例中的步骤404，图6对应的实施例中的步骤609。
参数提取模块904，用于执行图4对应的实施例中的步骤403，图6对应的实施例中的步骤604。
投影模块905，用于执行图6对应的实施例中的步骤605，图6对应的实施例中的步骤606。
关系权重确定模块906，用于执行图6对应的实施例中的步骤605，图6对应的实施例中的步骤606。
本申请中，可以同时有效利用大量不同数据集、不同类别的信息来训练同一个网络，极大提升了数据利用率，使得检测性能更高。通过图卷积可以在多个不同域中合并和传递相关语义信息，并能有效地捕捉不同数据集下不同物体之间的内在关系，使得不同域、不同数据集的标注信息可以互补。经过域内、域间图卷积增强过的待检测物体的高级语义信 息能同时用于多个不同域中，进行物体的识别和分类，极大提升了识别准确率。
当本申请实施例的物体检测方法由图1中的执行设备110执行时，上述物体检测装置中的图像获取模块901可以相当于执行设备110中的I/O接口112，而物体检测装置中的特征提取模块902和检测模块903相当于执行设备110中的计算模块111。
当本申请实施例的物体检测方法由图3中的神经网络处理器执行时，上述物体检测装置中的图像获取模块901可以相当于神经网络处理器中的总线接口单元510，而物体检测装置中的特征提取模块902和检测模块903相当于执行设备110中的运算电路503，或者，物体检测装置中的特征提取模块902和检测模块903还可以相当于执行设备110中的运算电路303+向量计算单元307+累加器308。
图10是本申请实施例的物体检测装置的示意性框图。图10所示的物体检测装置模块包括存储器1001、处理器1002、通信接口1003以及总线1004。其中，存储器1001、处理器1002、通信接口1003通过总线1004实现彼此之间的通信连接。
上述通信接口1003相当于物体检测装置中的图像获取模块901，上述处理器1002相当于物体检测装置中的特征提取模块902和检测模块903。下面对物体检测装置模块中的各个模块和模块进行详细的介绍。
存储器1001可以是只读存储器(read only memory，ROM)，静态存储设备，动态存储设备或者随机存取存储器(random access memory，RAM)。存储器1001可以存储程序，当存储器1001中存储的程序被处理器1002执行时，处理器1002和通信接口1003用于执行本申请实施例的物体检测方法的各个步骤。具体地，通信接口1003可以从存储器或者其他设备中获取待检测图像，然后由处理器1002对该待检测图像进行物体检测。
处理器1002可以采用通用的中央处理器(central processing unit，CPU)，微处理器，应用专用集成电路(application specific integrated circuit，ASIC)，图形处理器(graphics processing unit，GPU)或者一个或多个集成电路，用于执行相关程序，以实现本申请实施例的物体检测装置中的模块所需执行的功能(例如，处理器1002可以实现上述物体检测装置中的特征提取模块902和检测模块903所需执行的功能)，或者执行本申请实施例的物体检测方法。
处理器1002还可以是一种集成电路芯片，具有信号的处理能力。在实现过程中，本申请实施例的物体检测方法的各个步骤可以通过处理器1002中的硬件的集成逻辑电路或者软件形式的指令完成。
上述处理器1002还可以是通用处理器、数字信号处理器(digital signalprocessing，DSP)、ASIC、现成可编程门阵列(field programmable gate array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。上述通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本申请实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存储器，闪存、只读存储器，可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器1001，处理器1002读取存储器1001中的信息，结合其硬件完成本申请实施例的物体检测装置中包括的模块所需执行的功能，或者执行本申请方法实施例的物体检测方法。
通信接口1003使用例如但不限于收发器一类的收发装置，来实现装置模块与其他设备或通信网络之间的通信。例如，可以通过通信接口1003获取待处理图像。
总线1004可包括在装置模块各个部件(例如，存储器1001、处理器1002、通信接口1003)之间传送信息的通路。
图11是本申请实施例的神经网络训练装置的硬件结构示意图。与上述装置类似，图11所示的神经网络训练装置包括存储器1101、处理器1102、通信接口1103以及总线1104。其中，存储器1101、处理器1102、通信接口1103通过总线1104实现彼此之间的通信连接。
存储器1101可以存储程序，当存储器1101中存储的程序被处理器1102执行时，处理器1102用于执行本申请实施例的神经网络的训练方法的各个步骤。
处理器1102可以采用通用的CPU，微处理器，ASIC，GPU或者一个或多个集成电路，用于执行相关程序，以实现本申请实施例的神经网络的训练方法。
处理器1102还可以是一种集成电路芯片，具有信号的处理能力。在实现过程中，本申请实施例的神经网络的训练方法(如图8所示的方法)的各个步骤可以通过处理器1102中的硬件的集成逻辑电路或者软件形式的指令完成。
应理解，通过图11所示的神经网络训练装置对神经网络进行训练，训练得到的神经网络就可以用于执行本申请实施例的物体检测方法(如图8所示的方法)。
具体地，图11所示的装置可以通过通信接口1103从外界获取训练数据以及待训练的神经网络，然后由处理器根据训练数据对待训练的神经网络进行训练。
应注意，尽管上述装置模块和装置仅仅示出了存储器、处理器、通信接口，但是在具体实现过程中，本领域的技术人员应当理解，装置模块和装置还可以包括实现正常运行所必须的其他器件。同时，根据具体需要，本领域的技术人员应当理解，装置模块和装置还可包括实现其他附加功能的硬件器件。此外，本领域的技术人员应当理解，装置模块和装置也可仅仅包括实现本申请实施例所必须的器件，而不必包括图10和图11中所示的全部器件。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的模块及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的***、装置和模块的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的***、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述模块的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个模块或组件可以结合或者可以集成到另一个***，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或模块的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的模块可以是或者也可以不是物理上分开的，作为模块显示的 部件可以是或者也可以不是物理模块，即可以位于一个地方，或者也可以分布到多个网络模块上。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能模块可以集成在一个处理模块中，也可以是各个模块单独物理存在，也可以两个或两个以上模块集成在一个模块中。
所述功能如果以软件功能模块的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(read-only memory，ROM)、随机存取存储器(random access memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (18)

  1. 一种物体检测方法，其特征在于，包括：
    获取待检测图像；
    确定所述待检测图像中的待检测物体的初始图像特征；
    根据跨域知识图谱信息确定所述待检测物体的增强图像特征，所述跨域知识图谱信息包括不同域中待检测物体对应的物体类别之间的关联关系，所述增强图像特征指示所述不同域中与所述待检测物体相关联的其他物体对应的物体类别的语义信息；
    根据所述待检测物体的初始图像特征和所述待检测物体的增强图像特征，确定所述待检测物体的候选框和分类。
  2. 根据权利要求1所述的方法，其特征在于，所述跨域知识图谱包括节点和节点连边，所述节点对应所述待检测物体，所述节点连边对应不同所述待检测物体的高级语义特征之间的关系，所述方法还包括：
    获取所述不同域对应的分类层参数；
    根据所述不同域中所述初始图像特征在不同物体类别上的分类权重，将所述不同域对应的分类层参数加权融合，得到所述待检测物体的所述高级语义特征；
    将所述不同域中所述待检测物体对应的物体类别之间的关系权重投影到所述待检测物体的节点连边上，得到所述节点连边的权重。
  3. 根据权利要求2所述的方法，其特征在于，所述方法还包括：
    根据所述不同域中所述待检测物体对应的物体类别间的距离关系确定所述关系权重。
  4. 根据权利要求3所述的方法，其特征在于，所述待检测物体对应的物体类别间的距离关系包括以下信息中的一种或几种：
    不同域中所述待检测物体对应的物体类别间不同物体类别的属性关系；
    不同域中所述待检测物体对应的物体类别间的位置关系或者主动宾关系；
    不同域中所述待检测物体对应的物体类别间的利用语言学知识构建的词嵌入相似度；
    不同域中所述待检测物体对应的物体类别间的根据训练数据对神经网络模型进行训练得到的距离关系。
  5. 根据权利要求2至4任一项所述的方法，其特征在于，所述根据跨域知识图谱信息确定所述待检测物体的增强图像特征，包括：
    根据所述节点连边的权重对所述高级语义特征进行卷积处理，得到所述待检测物体的增强图像特征。
  6. 一种图像检测装置，其特征在于，包括：
    图像获取模块，用于获取待检测图像；
    特征提取模块，用于确定所述待检测图像中的待检测物体的初始图像特征；
    所述特征提取模块，还用于根据跨域知识图谱信息确定所述待检测物体的增强图像特征，所述跨域知识图谱信息包括不同域中待检测物体对应的物体类别之间的关联关系，所述增强图像特征指示所述不同域中与所述待检测物体相关联的其他物体对应的物体类别的语义信息；
    检测模块，用于根据所述待检测物体的初始图像特征和所述待检测物体的增强图像特 征，确定所述待检测物体的候选框和分类。
  7. 根据权利要求6所述的图像检测装置，其特征在于，所述跨域知识图谱包括节点和节点连边，所述节点对应所述待检测物体，所述节点连边对应不同所述待检测物体的高级语义特征之间的关系，所述图像检测装置还包括参数获取模块以及投影模块，
    所述参数获取模块，用于获取所述不同域对应的分类层参数；
    所述特征提取模块，具体用于根据所述不同域中所述初始图像特征在不同物体类别上的分类权重，将所述不同域对应的分类层参数加权融合，得到所述待检测物体的所述高级语义特征；
    所述投影模块，用于将所述不同域中所述待检测物体对应的物体类别之间的关系权重投影到所述待检测物体的节点连边上，得到所述节点连边的权重。
  8. 根据权利要求7所述的图像检测装置，其特征在于，还包括关系权重确定模块，
    所述关系权重确定模块，用于根据所述不同域中所述待检测物体对应的物体类别间的距离关系确定所述关系权重。
  9. 根据权利要求8所述的图像检测装置，其特征在于，所述待检测物体对应的物体类别间的距离关系包括以下信息中的一种或几种：
    不同域中所述待检测物体对应的物体类别间的属性关系；
    不同域中所述待检测物体对应的物体类别间的位置关系或者主动宾关系；
    不同域中所述待检测物体对应的物体类别间的利用语言学知识构建的词嵌入相似度；
    不同域中所述待检测物体对应的物体类别间的根据训练数据对神经网络模型进行训练得到的距离关系。
  10. 根据权利要求7至9中任一项所述的图像检测装置，其特征在于，
    所述特征提取模块，具体用于根据所述节点连边的权重对所述高级语义特征进行卷积处理，得到所述待检测物体的增强图像特征。
  11. 一种物体检测装置，其特征在于，包括：
    存储器，用于存储程序；
    处理器，用于执行所述存储器存储的程序，当所述存储器存储的程序被执行时，所述处理器用于执行如权利要求1-5中任一项所述的方法。
  12. 一种计算机存储介质，其特征在于，所述计算机存储介质存储有程序代码，所述程序代码包括用于执行如权利要求1-5中任一项所述的方法中的步骤的指令。
  13. 一种神经网络的训练方法，其特征在于，包括：
    获取训练数据，所述训练数据包括训练图像以及所述训练图像中待检测物体的物体检测标注结果；
    根据神经网络提取所述训练图像中的待检测物体的初始图像特征；
    根据所述神经网络以及跨域知识图谱信息提取所述训练图像中的待检测物体的增强图像特征；
    根据所述神经网络对所述待检测物体的初始图像特征和增强图像特征进行处理，得到所述待检测物体的物体检测结果；
    根据所述训练图像中的待检测物体的物体检测结果与所述训练图像中的待检测物体的 物体检测标注结果，确定所述神经网络的模型参数。
  14. 一种芯片***，其特征在于，包括：所述芯片***包括至少一个处理器，和接口电路，所述接口电路和所述至少一个处理器通过线路互联，所述处理器通过运行指令，以执行权利要求1至权利要求5中任一项所述的方法。
  15. 一种处理器，其特征在于，用于执行如权利要求1至5任一项所述的方法。
  16. 一种物体检测装置，用于执行权利要求1至5项任一项所述方法。
  17. 一种包含指令的计算机程序产品，当其在计算机上运行时，使得计算机执行上述权利要求1至5任一项所述的方法。
  18. 一种电子设备，所述电子设备包括图像检测装置，所述物体检测装置是权利要求6至10任一项所描述的图像检测装置。
PCT/CN2020/112796 2020-01-21 2020-09-01 一种物体检测方法、装置以及存储介质 WO2021147325A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010072238.0 2020-01-21
CN202010072238.0A CN111310604A (zh) 2020-01-21 2020-01-21 一种物体检测方法、装置以及存储介质

Publications (1)

Publication Number Publication Date
WO2021147325A1 true WO2021147325A1 (zh) 2021-07-29

Family

ID=71161604

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/112796 WO2021147325A1 (zh) 2020-01-21 2020-09-01 一种物体检测方法、装置以及存储介质

Country Status (2)

Country Link
CN (1) CN111310604A (zh)
WO (1) WO2021147325A1 (zh)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113807247A (zh) * 2021-09-16 2021-12-17 清华大学 基于图卷积网络的行人重识别高效标注方法及装置
CN114579981A (zh) * 2022-03-10 2022-06-03 北京国腾创新科技有限公司 一种跨域漏洞检测方法、***、存储介质和电子设备
CN114881329A (zh) * 2022-05-09 2022-08-09 山东大学 一种基于引导图卷积神经网络的轮胎质量预测方法及***
CN115830721A (zh) * 2022-11-02 2023-03-21 深圳市新良田科技股份有限公司 活体检测方法、装置、终端设备和可读存储介质
CN116244284A (zh) * 2022-12-30 2023-06-09 成都中轨轨道设备有限公司 一种基于立体内容的大数据处理方法
CN116665095A (zh) * 2023-05-18 2023-08-29 中国科学院空间应用工程与技术中心 一种运动舰船检测方法、***、存储介质和电子设备

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110378381B (zh) * 2019-06-17 2024-01-19 华为技术有限公司 物体检测方法、装置和计算机存储介质
CN111310604A (zh) * 2020-01-21 2020-06-19 华为技术有限公司 一种物体检测方法、装置以及存储介质
CN111783457B (zh) * 2020-07-28 2021-05-11 北京深睿博联科技有限责任公司 一种基于多模态图卷积网络的语义视觉定位方法及装置
CN112925920A (zh) * 2021-03-23 2021-06-08 西安电子科技大学昆山创新研究院 一种智慧社区大数据知识图谱网络社团检测方法
CN114627443B (zh) * 2022-03-14 2023-06-09 小米汽车科技有限公司 目标检测方法、装置、存储介质、电子设备及车辆

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7664339B2 (en) * 2004-05-03 2010-02-16 Jacek Turski Image processing method for object recognition and dynamic scene understanding
CN104573711A (zh) * 2014-12-22 2015-04-29 上海交通大学 基于文本-物体-场景关系的物体和场景的图像理解方法
CN110378381A (zh) * 2019-06-17 2019-10-25 华为技术有限公司 物体检测方法、装置和计算机存储介质
CN110704626A (zh) * 2019-09-30 2020-01-17 北京邮电大学 一种用于短文本的分类方法及装置
CN111310604A (zh) * 2020-01-21 2020-06-19 华为技术有限公司 一种物体检测方法、装置以及存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7664339B2 (en) * 2004-05-03 2010-02-16 Jacek Turski Image processing method for object recognition and dynamic scene understanding
CN104573711A (zh) * 2014-12-22 2015-04-29 上海交通大学 基于文本-物体-场景关系的物体和场景的图像理解方法
CN110378381A (zh) * 2019-06-17 2019-10-25 华为技术有限公司 物体检测方法、装置和计算机存储介质
CN110704626A (zh) * 2019-09-30 2020-01-17 北京邮电大学 一种用于短文本的分类方法及装置
CN111310604A (zh) * 2020-01-21 2020-06-19 华为技术有限公司 一种物体检测方法、装置以及存储介质

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113807247A (zh) * 2021-09-16 2021-12-17 清华大学 基于图卷积网络的行人重识别高效标注方法及装置
CN113807247B (zh) * 2021-09-16 2024-04-26 清华大学 基于图卷积网络的行人重识别高效标注方法及装置
CN114579981A (zh) * 2022-03-10 2022-06-03 北京国腾创新科技有限公司 一种跨域漏洞检测方法、***、存储介质和电子设备
CN114881329A (zh) * 2022-05-09 2022-08-09 山东大学 一种基于引导图卷积神经网络的轮胎质量预测方法及***
CN115830721A (zh) * 2022-11-02 2023-03-21 深圳市新良田科技股份有限公司 活体检测方法、装置、终端设备和可读存储介质
CN115830721B (zh) * 2022-11-02 2024-05-03 深圳市新良田科技股份有限公司 活体检测方法、装置、终端设备和可读存储介质
CN116244284A (zh) * 2022-12-30 2023-06-09 成都中轨轨道设备有限公司 一种基于立体内容的大数据处理方法
CN116244284B (zh) * 2022-12-30 2023-11-14 成都中轨轨道设备有限公司 一种基于立体内容的大数据处理方法
CN116665095A (zh) * 2023-05-18 2023-08-29 中国科学院空间应用工程与技术中心 一种运动舰船检测方法、***、存储介质和电子设备
CN116665095B (zh) * 2023-05-18 2023-12-22 中国科学院空间应用工程与技术中心 一种运动舰船检测方法、***、存储介质和电子设备

Also Published As

Publication number Publication date
CN111310604A (zh) 2020-06-19

Similar Documents

Publication Publication Date Title
WO2021147325A1 (zh) 一种物体检测方法、装置以及存储介质
WO2020253416A1 (zh) 物体检测方法、装置和计算机存储介质
US20220092351A1 (en) Image classification method, neural network training method, and apparatus
WO2021155792A1 (zh) 一种处理装置、方法及存储介质
Zhang et al. Artificial intelligence for remote sensing data analysis: A review of challenges and opportunities
WO2021042828A1 (zh) 神经网络模型压缩的方法、装置、存储介质和芯片
WO2020216227A9 (zh) 图像分类方法、数据处理方法和装置
WO2021043112A1 (zh) 图像分类方法以及装置
WO2020244653A1 (zh) 物体识别方法及装置
WO2021057056A1 (zh) 神经网络架构搜索方法、图像处理方法、装置和存储介质
CN112990211B (zh) 一种神经网络的训练方法、图像处理方法以及装置
WO2022001805A1 (zh) 一种神经网络蒸馏方法及装置
WO2021164750A1 (zh) 一种卷积层量化方法及其装置
WO2021218517A1 (zh) 获取神经网络模型的方法、图像处理方法及装置
US20220148291A1 (en) Image classification method and apparatus, and image classification model training method and apparatus
CN110222718B (zh) 图像处理的方法及装置
WO2021073311A1 (zh) 图像识别方法、装置、计算机可读存储介质及芯片
WO2022007867A1 (zh) 神经网络的构建方法和装置
WO2022217434A1 (zh) 感知网络、感知网络的训练方法、物体识别方法及装置
Grigorev et al. Depth estimation from single monocular images using deep hybrid network
WO2023125628A1 (zh) 神经网络模型优化方法、装置及计算设备
CN112464930A (zh) 目标检测网络构建方法、目标检测方法、装置和存储介质
CN115375781A (zh) 一种数据处理方法及其装置
WO2021136058A1 (zh) 一种处理视频的方法及装置
CN116258176A (zh) 一种数据处理方法及其装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20914791

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20914791

Country of ref document: EP

Kind code of ref document: A1