CN111496784A - Space environment identification method and system for robot intelligent service - Google Patents
Space environment identification method and system for robot intelligent service Download PDFInfo
- Publication number
- CN111496784A CN111496784A CN202010228789.1A CN202010228789A CN111496784A CN 111496784 A CN111496784 A CN 111496784A CN 202010228789 A CN202010228789 A CN 202010228789A CN 111496784 A CN111496784 A CN 111496784A
- Authority
- CN
- China
- Prior art keywords
- article
- relationship
- image
- visual
- acquiring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 230000000007 visual effect Effects 0.000 claims abstract description 75
- 238000012545 processing Methods 0.000 claims abstract description 11
- 238000010586 diagram Methods 0.000 claims description 15
- 239000013598 vector Substances 0.000 claims description 8
- 238000001514 detection method Methods 0.000 claims description 7
- 238000013528 artificial neural network Methods 0.000 claims description 5
- 230000002457 bidirectional effect Effects 0.000 claims description 5
- 238000001914 filtration Methods 0.000 claims description 5
- 238000013527 convolutional neural network Methods 0.000 claims description 4
- 238000002372 labelling Methods 0.000 claims description 4
- 230000008859 change Effects 0.000 claims description 2
- 125000004122 cyclic group Chemical group 0.000 claims 2
- 230000006870 function Effects 0.000 description 9
- 230000008569 process Effects 0.000 description 9
- 238000004590 computer program Methods 0.000 description 8
- 238000010276 construction Methods 0.000 description 8
- 238000003860 storage Methods 0.000 description 6
- 238000012549 training Methods 0.000 description 5
- 230000007613 environmental effect Effects 0.000 description 3
- 230000001149 cognitive effect Effects 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000000306 recurrent effect Effects 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000019771 cognition Effects 0.000 description 1
- 150000001875 compounds Chemical class 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1694—Programme controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
- B25J9/1697—Vision controlled systems
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J11/00—Manipulators not otherwise provided for
- B25J11/008—Manipulators for service tasks
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1602—Programme controls characterised by the control system, structure, architecture
Landscapes
- Engineering & Computer Science (AREA)
- Robotics (AREA)
- Mechanical Engineering (AREA)
- Automation & Control Theory (AREA)
- Image Analysis (AREA)
- Manipulator (AREA)
Abstract
The utility model provides a space environment recognition method and system for robot intelligent service, belonging to the robot technical field, which obtains at least one image of the space environment area to be recognized; acquiring a model by adopting a preset visual relationship according to the acquired image to obtain a visual relationship triple of the object in the image; obtaining a model by using a preset attribute according to the obtained article characteristics to obtain the attribute of each article; obtaining an article multiple relation graph in the current image according to the obtained visual relation triples of each article and the attributes of each article; processing each image of the environment area to be identified to obtain an article multiple relation graph, and constructing a robot semantic visual space according to the article multiple relation graph of each obtained image to realize identification of the space environment; according to the method and the device, the relation between the objects in the space and the relation between the objects and the attributes are obtained by constructing the relation triple, and the environment recognition capability of the space is greatly improved.
Description
Technical Field
The disclosure relates to the technical field of robots, in particular to a space environment identification method and system for robot intelligent service.
Background
The statements in this section merely provide background information related to the present disclosure and may not necessarily constitute prior art.
When the robot is in intelligent service, the environment needs to be understood at a semantic level, particularly, the environment can be mapped to a semantic space in the aspect of vision, the cognitive intelligence of the robot is enhanced, and tasks are better analyzed.
The inventor of the present disclosure finds that (1) most of the existing space identification methods simply identify images without considering the attributes of the articles in the images, which results in inaccurate space identification and can not define the relationship of each article in the space well; (2) the existing recognition algorithm mainly adopts a neural network algorithm, realizes the feature extraction of an image so as to realize the recognition of articles in the image, has less research on the relative position relationship in the specific image, and cannot effectively express the relative position relationship among the articles in the space.
Disclosure of Invention
In order to solve the defects of the prior art, the space environment identification method and system for robot intelligent service are provided by the disclosure, the relation among all articles in the space and the relation among the articles and the attributes are obtained by constructing the relation triple, and the environment identification capability of the space is greatly improved.
In order to achieve the purpose, the following technical scheme is adopted in the disclosure:
the first aspect of the disclosure provides a space environment identification method for robot intelligent service.
A space environment identification method for robot intelligent service comprises the following steps:
acquiring at least one image of a spatial environment region to be identified;
acquiring a model by adopting a preset visual relationship according to the acquired image, acquiring article characteristics in the image and relationship characteristics among articles, and further acquiring a visual relationship triple of the articles in the image;
obtaining a model by using a preset attribute according to the obtained article characteristics to obtain the attribute of each article;
obtaining an article multiple relation graph in the current image according to the obtained visual relation triples of each article and the attributes of each article;
and processing each acquired image of the environment area to be identified to obtain an article multiple relation graph, and constructing a robot semantic visual space according to the article multiple relation graph of each acquired image to realize identification of the space environment.
A second aspect of the present disclosure provides a spatial environment recognition system for a robot-intelligent service.
A spatial environment recognition system for robotic intelligence services, comprising:
a data acquisition module configured to: acquiring at least one image of a spatial environment region to be identified;
a visual relationship triplet acquisition module configured to: acquiring a model by adopting a preset visual relationship according to the acquired image, acquiring article characteristics in the image and relationship characteristics among articles, and further acquiring a visual relationship triple of the articles in the image;
an item attribute acquisition module configured to: obtaining a model by using a preset attribute according to the obtained article characteristics to obtain the attribute of each article;
an item multiple relationship diagram acquisition module configured to: obtaining an article multiple relation graph in the current image according to the obtained visual relation triples of each article and the attributes of each article;
an environment identification module configured to: and processing each image of the environment area to be identified to obtain an article multiple relation graph, and constructing a robot semantic visual space according to the obtained article multiple relation graph of each image to realize the identification of the environment.
A third aspect of the present disclosure provides a medium on which a program is stored, the program implementing, when executed by a processor, the steps in the spatial environment recognition method for robot intelligence service according to the first aspect of the present disclosure.
A fourth aspect of the present disclosure provides an electronic device, including a memory, a processor, and a program stored on the memory and executable on the processor, where the processor executes the program to implement the steps in the method for identifying a spatial environment for robot intelligent services according to the first aspect of the present disclosure.
Compared with the prior art, the beneficial effect of this disclosure is:
1. according to the method, the system, the medium and the electronic equipment, the preset visual relationship obtaining model is adopted, the object features in the image and the relationship features between the objects are obtained, the triple features of the objects in the image are obtained, the visual relationship triple of each object in the image is further obtained, the identification of the relative position relationship of each object in the image is achieved, and the identification capacity of the space is improved.
2. According to the method, the system, the medium and the electronic equipment, the model is obtained by using the preset attribute according to the obtained article characteristics, the attribute of each article is obtained, the article multi-relation graph in the current image is obtained according to the obtained visual relation triple of each article and the attribute of each article, the visual relation of each article and the attribute of each article are effectively combined, and the space environment is more efficiently and accurately identified.
Drawings
Fig. 1 is a schematic diagram of a visual relationship acquisition framework provided in embodiment 1 of the present disclosure.
Fig. 2 is a schematic diagram of a BRNN framework in the semantic visual space construction provided in embodiment 1 of the present disclosure.
Fig. 3 is a multiple relation diagram in a semantic visual space provided in embodiment 1 of the present disclosure.
Detailed Description
It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the disclosure. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments according to the present disclosure. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.
The embodiments and features of the embodiments in the present disclosure may be combined with each other without conflict.
Example 1:
as shown in fig. 1 to 3, embodiment 1 of the present disclosure provides a spatial environment recognition method for robot intelligent service, including the following steps:
acquiring at least one image of a spatial environment region to be identified;
according to the obtained image, a preset visual relationship obtaining model is adopted, the object features in the image and the relationship features between the objects are obtained, the triple features of the objects in the image are obtained, and then the visual relationship triple of the objects in the image is obtained;
obtaining a model by using a preset attribute according to the obtained article characteristics to obtain the attribute of each article;
obtaining an article multiple relation graph in the current image according to the obtained visual relation triples of each article and the attributes of each article;
and processing each acquired image of the environment area to be identified to obtain an article multiple relation graph, and constructing a robot semantic visual space according to the article multiple relation graph of each acquired image to realize identification of the space environment.
The visual space is constructed by the following steps:
firstly, the object relation in the environment is mapped to a semantic level, and the cognition of the robot to the service environment space relation is enhanced. Next, in order to further understand the intrinsic logical relationship of the item, a method of obtaining the item attribute from the network is given. And finally, storing the obtained environmental knowledge in a form of a multi-relation graph to complete the construction of the visual space.
The method specifically comprises the following steps:
(1) acquisition of visual relationships
In the semantic visual space construction process, firstly, the object visual relationships in the scene, such as "Computer on task", are obtained, and these relationships generally express < subject, predictor, object > in the form of triples. The visual relation realizes higher-level understanding of semantic level and is an important way for realizing the increase from perception intelligence to cognitive intelligence. In the construction work of the semantic visual space, the visual relation directly provides the leading knowledge in the environment. Fig. 1 provides a method for acquiring a visual relationship, which combines an image and semantic information to acquire a visual relationship.
In the method, firstly, a fast RCNN (Faster convolutional neural network based on regions) technology is used for target detection, and the semantics of a target object and the coordinate information of a detection frame of the target object are obtained. And performing feature coding on the semantics of the target object, and acquiring the relationship features between the objects according to the detection frame information to complete the construction of the triple features. Inputting the triple characteristics into a Bi-RNN (bidirectional recurrent neural network) network to construct a relation triple. In particular, considering that the existence of the dynamic object can influence the robustness of the constructed visual relationship, in the framework, the dynamic article filtering module is used at the same time, so that the influence of the dynamic object on the visual relationship library is reduced.
(1-1) triple feature construction
The triple features include word vectors for constructing entity targets, and entity relationship coding. A BERT (Bidirectional Encoder representation of transducers) model is adopted for constructing word vectors of an entity target, semantic labels in a data set are used as training data and input into the BERT model, fine adjustment of the model is carried out, parameters obtained by the model are used as characteristics, and the characteristics are the word vectors.
When obtaining the relational features, the spatial relationships between the targets are utilized to model the relational features. Relationship between objects uses [ x ]1,x2,x3]Denotes x1Represents object 1, x3Represents object 3, x2Representing a relationship. The spatial relationship s is represented as follows:
in the formula 1, the first and second groups of the compound,
a prediction block (i ═ 1,3) representing a target object;
refers to the center coordinates, w, of the prediction boxiAnd hiRepresenting the width and height of the prediction box, respectively. W and H represent the width and height of the intersection of the target 1 and target 3 prediction boxes;
after the spatial relation s is obtained, the s is input into a multilayer perceptron (M L P) to obtain a feature representation x with the same dimension as a word vector2And thus obtain the feature expression of the triplet.
(1-2) construction of visual relationship
And after the triple features are obtained, inputting the features into the Bi-RNN for visual relation prediction. The BRNN (bidirectional recurrent neural network) is an efficient natural language processing model, which is formed by superposing two RNN networks up and down, and the output is determined by the states of the two RNN (convolutional neural networks). In the visual relationship acquisition model, the output of the Bi-RNN is designed to be one rather than multiple outputs, and the input is a feature of a triplet, as shown in fig. 2.
In the framework shown in fig. 2, forward propagation is included
And backward propagation
The framework is mainly divided into an input layer, a hidden layer and an output layer. Wherein the formulas for the hidden layer and the output layer are defined as follows:
in the context of these formulas, the expression,
and
the parameter matrixes respectively represent an input layer-hidden layer and a hidden layer-hidden layer in the forward propagation process.
And
respectively, the hidden layer-output layer parameter matrix and the offset vector in the forward propagation process. The meaning of the parameters in the back propagation process can be obtained in the same way. x is the number oftRepresenting the input, y the output, f the activation function, byIs an offset value.
From the framework shown in FIG. 2 and equation (4), the predicted relationship output y can be obtained:
wherein h is1,h2,h3The values in the forward propagation and the backward propagation can be obtained by formula (2) and formula (3), and y represents the predicted relationship, such as "while", "on", etc. And after the word vectors and the relation characteristics are output through Bi-RNN training, the visual relation triple can be obtained.
(1-3) dynamic object Filtering
In the process of acquiring the visual relationship, the service robot pays attention to the relationship among the articles in the environment, and the relationship among the articles is always kept unchanged for a certain time. However, the existence of dynamic objects in the environment directly affects the timeliness of the visual relationship. For example, when detecting visual relationships, relationships between people and items are often extracted, and these relationships may change rapidly. If the relation is added into the relation library, inconsistency with reality is generated when the visual relation is searched in subsequent services, meanwhile, the redundancy degree of the relation library is increased, and the service efficiency of the robot is reduced. Therefore, the obtained triple relationship containing the dynamic object needs to be filtered.
The dynamic filtering module mainly provides a dynamic object list, and when detecting that the visual relation triplets contain dynamic objects, the triplets are filtered. The objects in the list may be added according to common sense, e.g. "people". The objects can also be added according to experience in actual operation, and through multiple operation observation, if the visual relationship of a certain type of objects changes rapidly, the objects are added. Thereby finally completing the acquisition of the visual relationship.
(2) Attribute knowledge acquisition
When the robot executes the intelligent service, not only the physical relationship between the objects needs to be understood, but also the deep-level attributes of the objects need to be further understood, so that the relationship of the objects on the logical level can be better understood. Therefore, when building the semantic visual space, the attributes of the item also need to be added. Due to the diversity of items and related attributes in the environment, it is desirable to obtain the related attributes of the items from the network.
In the process of acquiring the related attributes of the article, firstly, the article semantics are used as key words to search related information on the network, and after the search is carried out, a series of result character strings are obtained. Attributes of the item are extracted from the result strings.
In the method, text information is extracted by using a BERT model, characteristic coding is carried out, and the coding is input to a CRF layer for information decoding to obtain a labeling sequence, wherein the CRF refers to a conditional random field and is a discriminant probability undirected graph model used for labeling and dividing sequence data, the BERT-CRF is an improvement on the basis of Bi-L STM, the BERT model is used for replacing Bi-L STM, and the BERT model and the CRF have similar training principles.
Input sequence X ═ X1,X2,…,Xn) Obtaining a predicted tag sequence y (y) through output of a BERT-CRF model1,y2,..,yn)。
The scores defining the predicted sequences are as follows:
wherein
Indicating that the output result at the ith position is yiThe probability of (c).
Representing the transition probability from state i to i + 1. The scoring function takes the previous states into account and the obtained result is more consistent with the actual output result.
During training, for each sample X, all possible sequences y are scored as s (X, y), and the total score is normalized:
in equation 7, Y represents the correct annotation sequence and Y represents all possible annotation sequences. The loss function can be obtained by equation 7, as follows:
through continuous iterative training of the loss function in the model, the loss function is reduced to the minimum value, and therefore an ideal entity labeling effect is obtained. The attribute keywords in the result character string are extracted through the model, and the relationship is established with the objects in the environment, so that the object semantics in the environment are enriched, and the knowledge system in the semantic visual space is further improved.
(3) Storage of environmental knowledge
After the visual relationship and the attribute of the article in the environment are acquired, the semantic visual space is constructed to finish data preparation, and the connection form of the data knowledge is diversified, so that the diversified information needs to be efficiently organized and expressed in a multi-relationship graph form. The basic construction form of the multi-relationship graph is nodes and edges, wherein the nodes store entity semantic and item attribute information, and the edges represent the relationship between entities and the relationship between the entities and the attributes. Unlike a general graph, a multi-relationship graph contains multiple types of nodes and multiple types of edges. FIG. 3 is an example of a multiple relationship graph store.
In fig. 3, the gray boxes represent entity information in the environment and the white boxes represent attributes of the item. It can be seen that the visual relationship and the article attribute in the environment are combined in the form of a multiple relationship diagram, and through the combination, the robot can understand the article relationship in the environment not only at the semantic level, but also increase the knowledge of the article at the logical level, thereby forming a uniform and standard expression.
And storing the acquired environmental knowledge through a multi-relation graph to finally form a semantic visual space suitable for the intelligent service of the robot.
Example 2:
an embodiment 2 of the present disclosure provides a spatial environment recognition system for robot intelligent service, including:
a data acquisition module configured to: acquiring at least one image of a spatial environment region to be identified;
a visual relationship triplet acquisition module configured to: according to the obtained image, a preset visual relationship obtaining model is adopted, the object features in the image and the relationship features between the objects are obtained, the triple features of the objects in the image are obtained, and then the visual relationship triple of the objects in the image is obtained;
an item attribute acquisition module configured to: obtaining a model by using a preset attribute according to the obtained article characteristics to obtain the attribute of each article;
an item multiple relationship diagram acquisition module configured to: obtaining an article multiple relation graph in the current image according to the obtained visual relation triples of each article and the attributes of each article;
an environment identification module configured to: and processing each image of the environment area to be identified to obtain an article multiple relation graph, and constructing a robot semantic visual space according to the obtained article multiple relation graph of each image to realize the identification of the environment.
The working method of the system described in this embodiment is the same as the spatial environment recognition method for the robot intelligent service in embodiment 1, and details are not repeated here.
Example 3:
the embodiment 3 of the present disclosure provides a medium on which a program is stored, which when executed by a processor, implements the steps in the spatial environment recognition method for robot intelligent service according to the embodiment 1 of the present disclosure.
Example 4:
the embodiment 4 of the present disclosure provides an electronic device, which includes a memory, a processor, and a program stored in the memory and executable on the processor, and when the processor executes the program, the steps in the method for identifying a spatial environment for robot intelligent services according to embodiment 1 of the present disclosure are implemented.
As will be appreciated by one skilled in the art, embodiments of the present disclosure may be provided as a method, system, or computer program product. Accordingly, the present disclosure may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.
The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure, and various modifications and changes may be made to the present disclosure by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present disclosure should be included in the protection scope of the present disclosure.
Claims (10)
1. A space environment identification method for robot intelligent service is characterized by comprising the following steps:
acquiring at least one image of a spatial environment region to be identified;
acquiring a model by adopting a preset visual relationship according to the acquired image, acquiring article characteristics in the image and relationship characteristics among articles, and further acquiring a visual relationship triple of the articles in the image;
obtaining a model by using a preset attribute according to the obtained article characteristics to obtain the attribute of each article;
obtaining an article multiple relation graph in the current image according to the obtained visual relation triples of each article and the attributes of each article;
and processing each acquired image of the environment area to be identified to obtain an article multiple relation graph, and constructing a robot semantic visual space according to the article multiple relation graph of each acquired image to realize identification of the space environment.
2. The spatial environment recognition method for robot intelligent services according to claim 1, wherein the preset visual relationship acquisition model comprises a convolutional neural network and a bidirectional cyclic neural network for target detection, and specifically comprises:
performing target detection on the obtained image according to the convolutional neural network, acquiring the semantics of a target article and the coordinate information of a detection frame thereof, performing feature coding on the semantics of the target article and a non-target article, and acquiring the relationship features between the articles according to the coordinate information of the detection frame to obtain triple features;
and inputting the obtained triple features into a bidirectional cyclic neural network for visual relationship prediction to obtain the visual relationship triples of each article in the image.
3. The spatial environment recognition method for robot intelligent services according to claim 2, wherein the preset visual relationship obtaining model further includes a dynamic filtering module, and the dynamic filtering module filters out the visual relationship triples when detecting that the visual relationship triples contain dynamic objects through a preset dynamic object list.
4. The method of claim 3, wherein if the speed of change of the visual relationship of a certain type of objects exceeds a preset threshold, adding the objects into the dynamic object list to filter out the corresponding visual relationship triples.
5. The spatial environment recognition method for robot intelligent services according to claim 1, wherein the attributes of each item are obtained using a preset attribute acquisition model, specifically:
searching article correlation in a preset database by taking the semantics of the articles as keywords to obtain a plurality of result character strings;
according to the obtained result character string, identifying attribute keywords in the result by adopting a BERT-CRF model, which specifically comprises the following steps:
semantic text information of the articles is extracted by using a BERT model, characteristic coding is carried out, the codes are input into a CRF layer for information decoding, and the labeling sequences of the articles are obtained.
6. The spatial environment recognition method for robotic intelligence services as claimed in claim 1 wherein the item multiple relationship graph includes a plurality of nodes for storing entity semantics and item attribute information and a plurality of edges representing relationships between entities and attributes.
7. The spatial environment recognition method for robotic intelligence services as claimed in claim 1, wherein the relationship features between items are characterized by spatial relationships between objects, specifically:
the triple features comprise word vectors of objects in the image, and the relation between the targets uses [ x ]1，x2，x3]Denotes x1Represents object 1, x3Represents object 3, x2Representing the relationship, then the spatial relationship s is represented as follows:
wherein,
a prediction block representing a target object, wherein i-1 or 3,
to predict the center coordinates of the box, wiAnd hiRespectively representing the width and height of the prediction box, W and H representing the width and height of the intersection of the target 1 and target 3 prediction boxes,
is the encoding of the relevant spatial relationships;
after the spatial relation s is obtained, the s is input into a multilayer perceptron, and a feature representation x with the same dimension as the word vector is obtained2And thus obtain the feature expression of the triplet.
8. A spatial environment recognition system for robotic intelligence services, comprising:
a data acquisition module configured to: acquiring at least one image of a spatial environment region to be identified;
a visual relationship triplet acquisition module configured to: acquiring a model by adopting a preset visual relationship according to the acquired image, acquiring article characteristics in the image and relationship characteristics among articles, and further acquiring a visual relationship triple of the articles in the image;
an item attribute acquisition module configured to: obtaining a model by using a preset attribute according to the obtained article characteristics to obtain the attribute of each article;
an item multiple relationship diagram acquisition module configured to: obtaining an article multiple relation graph in the current image according to the obtained visual relation triples of each article and the attributes of each article;
an environment identification module configured to: and processing each image of the environment area to be identified to obtain an article multiple relation graph, and constructing a robot semantic visual space according to the obtained article multiple relation graph of each image to realize the identification of the environment.
9. A medium on which a program is stored, characterized in that the program, when being executed by a processor, realizes the steps in the spatial environment recognition method for robot intelligence services according to any one of claims 1 to 7.
10. An electronic device comprising a memory, a processor and a program stored on the memory and executable on the processor, wherein the processor implements the steps of the method for spatial environment recognition for robot intelligence services according to any of claims 1-7 when executing the program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010228789.1A CN111496784B (en) | 2020-03-27 | 2020-03-27 | Space environment identification method and system for robot intelligent service |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010228789.1A CN111496784B (en) | 2020-03-27 | 2020-03-27 | Space environment identification method and system for robot intelligent service |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111496784A true CN111496784A (en) | 2020-08-07 |
| CN111496784B CN111496784B (en) | 2021-05-07 |
Family
ID=71867152
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010228789.1A Active CN111496784B (en) | 2020-03-27 | 2020-03-27 | Space environment identification method and system for robot intelligent service |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111496784B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112765992A (en) * | 2021-01-14 | 2021-05-07 | 深圳市人马互动科技有限公司 | Training data construction method and device, computer equipment and storage medium |
| CN114630280A (en) * | 2020-12-14 | 2022-06-14 | 曲阜师范大学 | Indoor intelligent positioning system and control method thereof |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107516127A (en) * | 2017-08-21 | 2017-12-26 | 山东大学 | Service robot independently obtains people and wears the method and system for taking article ownership semanteme |
| CN107807986A (en) * | 2017-10-31 | 2018-03-16 | 中南大学 | A kind of method for the remote sensing image intelligent Understanding for describing atural object spatial relation semantics |
| CN109816686A (en) * | 2019-01-15 | 2019-05-28 | 山东大学 | Robot semanteme SLAM method, processor and robot based on object example match |
| CN110334219A (en) * | 2019-07-12 | 2019-10-15 | 电子科技大学 | The knowledge mapping for incorporating text semantic feature based on attention mechanism indicates learning method |
| CN110489395A (en) * | 2019-07-27 | 2019-11-22 | 西南电子技术研究所（中国电子科技集团公司第十研究所） | Automatically the method for multi-source heterogeneous data knowledge is obtained |
| CN110717052A (en) * | 2019-10-15 | 2020-01-21 | 山东大学 | Environment characterization method in service robot intelligent service |
| WO2020020085A1 (en) * | 2018-07-24 | 2020-01-30 | 华为技术有限公司 | Representation learning method and device |
| CN110781254A (en) * | 2020-01-02 | 2020-02-11 | 四川大学 | Automatic case knowledge graph construction method, system, equipment and medium |
| CN110825829A (en) * | 2019-10-16 | 2020-02-21 | 华南理工大学 | Method for realizing autonomous navigation of robot based on natural language and semantic map |
| CN110866190A (en) * | 2019-11-18 | 2020-03-06 | 支付宝(杭州)信息技术有限公司 | Method and device for training neural network model for representing knowledge graph |
-
2020
- 2020-03-27 CN CN202010228789.1A patent/CN111496784B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107516127A (en) * | 2017-08-21 | 2017-12-26 | 山东大学 | Service robot independently obtains people and wears the method and system for taking article ownership semanteme |
| CN107807986A (en) * | 2017-10-31 | 2018-03-16 | 中南大学 | A kind of method for the remote sensing image intelligent Understanding for describing atural object spatial relation semantics |
| WO2020020085A1 (en) * | 2018-07-24 | 2020-01-30 | 华为技术有限公司 | Representation learning method and device |
| CN109816686A (en) * | 2019-01-15 | 2019-05-28 | 山东大学 | Robot semanteme SLAM method, processor and robot based on object example match |
| CN110334219A (en) * | 2019-07-12 | 2019-10-15 | 电子科技大学 | The knowledge mapping for incorporating text semantic feature based on attention mechanism indicates learning method |
| CN110489395A (en) * | 2019-07-27 | 2019-11-22 | 西南电子技术研究所（中国电子科技集团公司第十研究所） | Automatically the method for multi-source heterogeneous data knowledge is obtained |
| CN110717052A (en) * | 2019-10-15 | 2020-01-21 | 山东大学 | Environment characterization method in service robot intelligent service |
| CN110825829A (en) * | 2019-10-16 | 2020-02-21 | 华南理工大学 | Method for realizing autonomous navigation of robot based on natural language and semantic map |
| CN110866190A (en) * | 2019-11-18 | 2020-03-06 | 支付宝(杭州)信息技术有限公司 | Method and device for training neural network model for representing knowledge graph |
| CN110781254A (en) * | 2020-01-02 | 2020-02-11 | 四川大学 | Automatic case knowledge graph construction method, system, equipment and medium |
Non-Patent Citations (2)
| Title |
|---|
| YI YANG: "Large-scale 3D Semantic Mapping Using Stereo Vision", 《INTERNATIONAL JOURNAL OF AUTOMATION AND COMPUTING》 * |
| 吴晓健: "基于知识图谱的服务机器人环境信息表征机制研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114630280A (en) * | 2020-12-14 | 2022-06-14 | 曲阜师范大学 | Indoor intelligent positioning system and control method thereof |
| CN114630280B (en) * | 2020-12-14 | 2024-06-04 | 曲阜师范大学 | Indoor intelligent positioning system and control method thereof |
| CN112765992A (en) * | 2021-01-14 | 2021-05-07 | 深圳市人马互动科技有限公司 | Training data construction method and device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111496784B (en) | 2021-05-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110276765B (en) | Image panorama segmentation method based on multitask learning deep neural network | |
| CN109344285B (en) | Monitoring-oriented video map construction and mining method and equipment | |
| CN110298404B (en) | Target tracking method based on triple twin Hash network learning | |
| CN107146237B (en) | Target tracking method based on online state learning and estimation | |
| CN109993102A (en) | Similar face retrieval method, apparatus and storage medium | |
| CN110457514A (en) | A kind of multi-tag image search method based on depth Hash | |
| Zhou et al. | Deep learning enabled cutting tool selection for special-shaped machining features of complex products | |
| CN111496784B (en) | Space environment identification method and system for robot intelligent service | |
| CN112884802B (en) | Attack resistance method based on generation | |
| Perveen et al. | Facial expression recognition using facial characteristic points and Gini index | |
| CN110147841A (en) | The fine grit classification method for being detected and being divided based on Weakly supervised and unsupervised component | |
| EP1801731B1 (en) | Adaptive scene dependent filters in online learning environments | |
| CN110826056A (en) | Recommendation system attack detection method based on attention convolution self-encoder | |
| CN111259264B (en) | Time sequence scoring prediction method based on generation countermeasure network | |
| Shukla et al. | A survey on image mining, its techniques and application | |
| CN111241326B (en) | Image visual relationship indication positioning method based on attention pyramid graph network | |
| CN117423032B (en) | Time sequence dividing method for human body action with space-time fine granularity, electronic equipment and computer readable storage medium | |
| AFFES et al. | Comparison of YOLOV5, YOLOV6, YOLOV7 and YOLOV8 for Intelligent Video Surveillance. | |
| CN116051924B (en) | Divide-and-conquer defense method for image countermeasure sample | |
| CN110111358B (en) | Target tracking method based on multilayer time sequence filtering | |
| CN111259176A (en) | Cross-modal Hash retrieval method based on matrix decomposition and integrated with supervision information | |
| Tao et al. | Semi-supervised online learning for efficient classification of objects in 3d data streams | |
| CN111090723B (en) | Knowledge graph-based recommendation method for safe production content of power grid | |
| CN111832815A (en) | Scientific research hotspot prediction method and system | |
| CN109934302B (en) | New category identification method and robot system based on fuzzy theory and deep learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |