Summary of the invention
The object of the present invention is to provide the method and system that elevator is taken by a kind of robot, can avoid waste passenger-in-elevator when
Between, improve the friendly interaction capabilities of robot.
Technical solution provided by the invention is as follows:
On the one hand, a kind of method that elevator is taken by robot is provided, comprising:
When elevator door is opened, the image of elevator interior is obtained;
Described image is inputted in trained neural network model in advance, object detection results and predeterminable area inspection are obtained
It surveys as a result, the predeterminable area is the region that robot enters that elevator needs to occupy;
According to the object detection results and predeterminable area testing result, judge whether that the elevator can be entered, if so,
Into the elevator, if it is not, then discharging the elevator, and elevator is called again.
It is further preferred that described according to the object detection results and predeterminable area testing result, judging whether can be into
Enter the elevator to specifically include:
According to the predeterminable area testing result, judge whether the predeterminable area is occupied, if it is not, then determining to enter
The elevator;
If judging that the predeterminable area is occupied, then according to the target detection according to the predeterminable area testing result
As a result, judgement occupies whether object is object;
If the object that occupies is object, judgement can not enter the elevator;
If the object that occupies is not object, evacuation information is exported, after waiting preset time, is obtained in the elevator again
The new images in portion input the new images in the neural network model, obtain object detection results and predeterminable area again
Testing result, and according to the object detection results and predeterminable area testing result obtained again, judge whether that the electricity can be entered
Ladder when repeating to obtain the number of image of the elevator interior greater than preset threshold then determines that the elevator can not be entered.
It is further preferred that described input described image in trained neural network model in advance, target inspection is obtained
It surveys result and predeterminable area testing result specifically includes:
Described image is inputted in trained neural network model in advance, the characteristic pattern of different layers is extracted;
Fusion Features and increase resolution are carried out to the characteristic pattern of different layers;
According to the characteristic pattern obtained after Fusion Features and increase resolution, object detection results and predeterminable area detection are exported
As a result.
It is further preferred that described input described image in trained neural network model in advance, difference is extracted
The characteristic pattern of layer specifically includes:
Described image is inputted in trained neural network model in advance, the fisrt feature figure of the last layer is extracted,
And the second feature figure of front layer is extracted, the resolution ratio of the second feature figure is the two of the resolution ratio of the fisrt feature figure
Times;
The characteristic pattern to different layers carries out Fusion Features and increase resolution specifically includes:
According to default step, the fisrt feature figure is handled, makes the increase resolution one of the fisrt feature figure
Times, obtain third feature figure;
It will be merged after the second feature figure 1 × 1 process of convolution of progress with the third feature figure, obtain the 4th spy
Sign figure;
According to the default step, the fourth feature figure is repeatedly handled, by the resolution of the fourth feature figure
Rate promotes more times, obtains fifth feature figure;
It is described according to the characteristic pattern obtained after Fusion Features and increase resolution, export object detection results and predeterminable area
Testing result specifically includes:
According to the fifth feature figure, object detection results and predeterminable area testing result are exported.
It is further preferred that it is described according to default step, the fisrt feature figure is handled, the fisrt feature is made
One times of the increase resolution of figure obtains third feature figure and specifically includes:
Deconvolution is carried out to the fisrt feature figure;
Channel separation is carried out to the fisrt feature figure after deconvolution;
Process of convolution is carried out to the fisrt feature figure after channel separation using two different convolution kernels;
Channel fusion is carried out to the characteristic pattern obtained after process of convolution, obtains third feature figure.
On the other hand, a kind of system that elevator is taken by robot is also provided, comprising:
Image collection module, for obtaining the image of elevator interior when elevator door is opened;
Detection module obtains target detection knot for inputting described image in trained neural network model in advance
Fruit and predeterminable area testing result, the predeterminable area are the region that robot enters that elevator needs to occupy;
Processing module, for judging whether that institute can be entered according to the object detection results and predeterminable area testing result
Elevator is stated, if so, if it is not, then discharging the elevator, and calling elevator again into the elevator.
It is further preferred that the processing module includes:
Processing unit, for judging whether the predeterminable area is occupied according to the predeterminable area testing result, if
It is no, then determine that the elevator can be entered;
The processing unit, if being also used to judge that the predeterminable area is occupied according to the predeterminable area testing result,
Then according to the object detection results, judgement occupies whether object is object;
The processing unit, if being also used to the object that occupies is object, judgement can not enter the elevator;
The processing unit exports evacuation information if being also used to the object that occupies is not object, waits preset time
Afterwards, the new images are inputted in the neural network model, obtain mesh again by the new images for obtaining the elevator interior again
Mark testing result and predeterminable area testing result, and according to the object detection results and predeterminable area testing result obtained again,
Judge whether that the elevator can be entered, when repeating to obtain the number of image of the elevator interior greater than preset threshold, then sentences
Surely it can not enter the elevator.
It is further preferred that the detection module includes:
Feature extraction unit extracts difference for inputting described image in trained neural network model in advance
The characteristic pattern of layer;
Increase resolution unit carries out Fusion Features and increase resolution for the characteristic pattern to different layers;
Task output unit, for exporting target detection according to the characteristic pattern obtained after Fusion Features and increase resolution
As a result with predeterminable area testing result.
It is further preferred that the feature extraction unit, is also used to inputting described image into trained nerve net in advance
In network model, the fisrt feature figure of the last layer is extracted, and extracts the second feature figure of front layer, the second feature figure
Resolution ratio be twice of resolution ratio of the fisrt feature figure;
The increase resolution unit includes:
Increase resolution subelement, for handling the fisrt feature figure, making described first according to default step
One times of the increase resolution of characteristic pattern, obtains third feature figure;
Fusion Features subelement, for by the second feature figure carry out 1 × 1 process of convolution after with the third feature figure
It is merged, obtains fourth feature figure;
The increase resolution subelement, for repeatedly being located to the fourth feature figure according to the default step
Reason, by more times of increase resolution of the fourth feature figure, obtains fifth feature figure;
The task output unit is also used to export object detection results and predeterminable area according to the fifth feature figure
Testing result.
It is further preferred that the increase resolution subelement, is also used to carry out deconvolution to the fisrt feature figure;It is right
The fisrt feature figure after deconvolution carries out channel separation;Using two different convolution kernels to described after channel separation
One characteristic pattern carries out process of convolution;Channel fusion is carried out to the characteristic pattern obtained after process of convolution, obtains third feature figure.
Compared with prior art, the method and system that elevator is taken by a kind of robot provided by the invention have beneficial below
Effect: the present invention detects pair whether predeterminable area in elevator is occupied and occupies by obtaining elevator interior image
As, to determine whether elevator can be entered, when elevator can not be entered, elevator is directly discharged, avoids the time for excessively occupying passenger-in-elevator,
To improve the friendly interaction capabilities of robot;In addition, the present invention by obtain the detection mode of image and neural network model come
Judge whether that elevator can be entered, avoiding can not examine using causing to be blocked because not knowing object category when depth transducer partially
It measures, so that the problem of space judges incorrectly, improves judging nicety rate.
Specific embodiment
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, Detailed description of the invention will be compareed below
A specific embodiment of the invention.It should be evident that drawings in the following description are only some embodiments of the invention, for
For those of ordinary skill in the art, without creative efforts, it can also be obtained according to these attached drawings other
Attached drawing, and obtain other embodiments.
It should be appreciated that when used in this manual, term " includes " indicates the Expressive Features, entirety, step, behaviour
Make, the presence of element and/or component, but one or more other features, entirety, step, operation, element, component are not precluded
And/or the presence or addition of set.
To make simplified form, part related to the present invention is only schematically shown in each figure, they are not represented
Its practical structures as product.In addition, there is identical structure or function in some figures so that simplified form is easy to understand
Component only symbolically depicts one of those, or has only marked one of those.Herein, "one" is not only indicated
" only this ", can also indicate the situation of " more than one ".
The first embodiment provided according to the present invention, as shown in Figure 1, a kind of method that elevator is taken by robot, comprising:
S100 obtains the image of elevator interior when elevator door is opened;
S200 inputs described image in trained neural network model in advance, obtains object detection results and preset areas
Domain testing result, the predeterminable area are the region that robot enters that elevator needs to occupy;
S300 according to the object detection results and predeterminable area testing result, judge whether can enter the elevator, if
It is then to enter the elevator, if it is not, then discharging the elevator, and call elevator again.
Specifically, when robot needs to take elevator, robot first runs to elevator position, then calling electricity
Ladder;When elevator door is opened, robot face elevator obtains the figure of elevator interior by the camera being mounted in robot
Picture.Camera in addition to mountable in robot, it is also mountable in elevator, when robot call elevator after, server scheduling
Destination where target elevator to robot, when the destination where target elevator arrival robot, server control
Camera in elevator processed obtains the image of elevator interior.
After getting the image of elevator interior, by preparatory trained neural network model, the image of acquisition is carried out
Identification obtains object detection results and predeterminable area testing result.Target refers to relatively common object or person, for example, curing
When in institute's scene, target refers to people or in position classifications objects such as the relatively common hospital bed of hospital, cart, wheelchairs.Predeterminable area
Refer to that robot is able to enter the position occupied required for elevator.Due to the size of robot be it is known, can be in elevator
Middle extract sets the predeterminable area that can accommodate robot, and the size of predeterminable area is greater than the size of robot.Robot exists
When into elevator, only moves forward or back, can not also be moved left and right, therefore, predeterminable area is a fixation in elevator
Region.
Neural network model is improved with Objects as Points foundation structure, is used in basic network part
The structure of mobilenetv3 extracts feature, and joined intermediate module, intermediate module be used for from
The characteristic pattern extracted in mobilenetv3 carries out increase resolution, and the characteristic pattern after increase resolution is then inputted more
Output module be engaged in output test result, i.e., is added to the branch of region decision on the basis of Objects asPoints.
After neural network model is built, the image of a large amount of elevator interior is obtained, then by way of manually marking,
Target is marked out in the picture and state that predeterminable area is occupied;Collection and test set is respectively trained in the image marked, is led to
It crosses in the neural network model that training set builds input and is trained, then by test set to trained neural network
Model is tested, and according to test result, carries out data enhancing to the specific training sample in training set, then will be enhanced
Data set is further divided into training set and test set, and trained neural network model is trained and is tested again, repeated data
The step of collection enhancing, training and test, until obtaining a limit detection effect, obtain trained neural network model.
After obtaining object detection results and predeterminable area testing result, it can judge whether enter according to the result of acquisition
Elevator.For example, if detecting, predeterminable area is not occupied, illustrates that robot can be directly entered elevator, if detecting preset areas
Domain is occupied by hospital bed, then illustrates that robot can not enter elevator, need to discharge the elevator, call other elevators again.
In the present invention, by obtaining elevator interior image, and detect whether the predeterminable area in elevator is occupied and accounts for
According to object, to determine whether can enter elevator, when elevator can not be entered, directly release elevator, avoid excessively occupying passenger-in-elevator
Time, to improve the friendly interaction capabilities of robot;In addition, detection of the present invention by acquisition image and neural network model
Mode is to determine whether can enter elevator, avoiding leads to the part that is blocked because not knowing object category using when depth transducer
It can not detect, so that the problem of space judges incorrectly, improves judging nicety rate.
The second embodiment provided according to the present invention, as shown in Fig. 2, a kind of method that elevator is taken by robot, above-mentioned
On the basis of first embodiment, according to the object detection results and predeterminable area testing result, judge whether to enter described
Elevator specifically includes:
According to the predeterminable area testing result, judge whether the predeterminable area is occupied, if it is not, then determining to enter
The elevator;
If judging that the predeterminable area is occupied, then according to the target detection according to the predeterminable area testing result
As a result, judgement occupies whether object is object;
If the object that occupies is object, judgement can not enter the elevator;
If the object that occupies is not object, evacuation information is exported, after waiting preset time, is obtained in the elevator again
The new images in portion input the new images in the neural network model, obtain object detection results and predeterminable area again
Testing result, and according to the object detection results and predeterminable area testing result obtained again, judge whether that the electricity can be entered
Ladder when repeating to obtain the number of image of the elevator interior greater than preset threshold then determines that the elevator can not be entered.
Specifically, after obtaining object detection results and predeterminable area testing result, first according to predeterminable area testing result,
Judge the state of predeterminable area, if predeterminable area is not occupied, can directly judge that robot can enter elevator.
If at least 10% (can be set according to the actual situation in discovery predeterminable area according to predeterminable area testing result
Set) region when being classified as occupy state, the object of predeterminable area is occupied according to object detection results analysis.If occupying default
Region be the non-pedestrian such as hospital bed when, then determine that elevator, directly release elevator can not be entered, and call other elevators again, into
Enter new detection and judges process.
If that occupy predeterminable area is pedestrian, robot carries out voice broadcast, prompts pedestrian's evacuation;Then it waits default
After time (as waited 5 seconds), then the new images for obtaining the elevator interior are re-shoot, and by neural network model to new images
It is detected, new object detection results and predeterminable area testing result is obtained, to judge whether pedestrian avoids.If row
People has avoided, then judges that robot can enter elevator, robot drives directly into elevator.
If pedestrian goes back and avoids, voice broadcast is carried out again, prompts pedestrian's evacuation, after waiting preset time, then
Secondary progress entirely judges process, that is, re-executes above-mentioned steps S100, S200 and S300;Process is entirely judged when repeating
When number is more than preset threshold (such as 2 times, 3 inferior), then illustrate the pedestrian in elevator can not avoid or elevator in pedestrian
It is unwilling to be avoided, at this point, directly discharging elevator, abandons this boarding, and call elevator again.
In the present solution, by repeated detection and judgement, whether distinguishable pedestrian out is friendly when predeterminable area is occupied by pedestrian
Good interaction directly discharges elevator when pedestrian is unfriendly interactive, avoids the time of waste pedestrian and robot itself.
The 3rd embodiment provided according to the present invention, as shown in figure 3, the method that elevator is taken by a robot, above-mentioned
On the basis of the first embodiment or the second embodiment, described image is inputted trained neural network model in advance by step S200
In, it obtains object detection results and predeterminable area testing result specifically includes:
S210 inputs described image in trained neural network model in advance, extracts the characteristic pattern of different layers;
S220 carries out Fusion Features and increase resolution to the characteristic pattern of different layers;
S230 exports object detection results and predeterminable area according to the characteristic pattern obtained after Fusion Features and increase resolution
Testing result.
Specifically, after the image for the elevator interior that will acquire inputs in trained neural network model, pass through
Mobilenetv3 extracts the characteristic pattern of different layers.In convolutional neural networks, high-level characteristic pattern has stronger semanteme
Property, lower resolution ratio maps out global and profile feature.The characteristic pattern of low level has weaker Semantic, higher
Resolution ratio maps out part and minutia.
Fusion Features are carried out to the characteristic patterns of different layers, feature in characteristic pattern to combine low level and high-level
Feature in characteristic pattern, and then Mutually fusion is carried out to feature, resolution ratio then is carried out to the characteristic pattern obtained after Fusion Features
It is promoted, further according to the characteristic pattern obtained after increase resolution, obtains object detection results and predeterminable area testing result.By right
Feature carries out fusion and increase resolution, can take into account well and carry out detection to the target in image and to the region in image
It is detected.
Illustratively, the acquisition process of predeterminable area testing result are as follows: assuming that carrying out the characteristic pattern after increase resolution
Resolution ratio is the 1/n of the image of the elevator interior (original image) obtained, that is, each pair of point in characteristic pattern finally obtained answers original image
In a n × n region, in the characteristic pattern finally obtained each point carry out classification and Detection, i.e., to each n in original image
The region of × n carries out classification and Detection, judges that whether each region is occupied in original image, then therefrom extracts in predeterminable area
Whether each region n × n is occupied, and predeterminable area testing result can be obtained.
In the present solution, being carried out by the feature in the characteristic pattern to low level and the feature in high-level characteristic pattern mutual
Fusion is mended, the high efficiency and accuracy rate to target detection can be improved.
The fourth embodiment provided according to the present invention, as shown in figure 4, the method that elevator is taken by a robot, comprising:
S100 obtains the image of elevator interior when elevator door is opened;
S211 inputs described image in trained neural network model in advance, extracts the fisrt feature of the last layer
Figure, and the second feature figure of front layer is extracted, the resolution ratio of the second feature figure is the resolution ratio of the fisrt feature figure
Twice;
S221 is handled the fisrt feature figure according to default step, proposes the resolution ratio of the fisrt feature figure
One times is risen, third feature figure is obtained;
S222 will the second feature figure carry out 1 × 1 process of convolution after merge with the third feature figure, obtain the
Four characteristic patterns;
S223 is repeatedly handled the fourth feature figure according to the default step, by the fourth feature figure
More times of increase resolution, obtain fifth feature figure;
S231 exports object detection results and predeterminable area testing result, the preset areas according to the fifth feature figure
Domain is the region that robot enters that elevator needs to occupy;
S300 according to the object detection results and predeterminable area testing result, judge whether can enter the elevator, if
It is then to enter the elevator, if it is not, then discharging the elevator, and call elevator again.
Specifically, in the network architecture, usually as depth increase can reduce resolution ratio and improve port number.Such as Fig. 5 institute
Show, it is assumed that the resolution ratio of each characteristic pattern of the last layer extracted from mobilenetv3 is the 1/32 of original image, by last
Each characteristic pattern of layer is defined as fisrt feature figure;Then each characteristic pattern of 1/16 that layer that resolution ratio is original image is chosen again,
Each characteristic pattern of this layer is defined as second feature figure.
Then according to default step, fisrt feature figure is handled, makes one times of increase resolution of fisrt feature figure, obtains
To third feature figure, the resolution ratio of third feature figure is the 1/16 of original image.
Second feature figure is subjected to 1 × 1 convolution transform again, keeps second feature figure identical as the port number of third feature figure,
Then by after convolution transform second feature figure and third feature figure merge, obtain fourth feature figure.
2 Sub-reso promotions processing is carried out to fourth feature figure according still further to above-mentioned default step, obtains fifth feature figure,
The resolution ratio of the obtained fifth feature figure is the 1/4 of original image.Pass through Objects as Points after obtaining fifth feature figure
Then target in foundation structure detection image carries out territorial classification detection by the region decision branch of addition.
It is to each point minute in the fifth feature figure when carrying out territorial classification detection by fifth feature figure
Class, judges whether each point is occupied, and in fifth feature figure a point corresponds to one 4 × 4 region in original image, is equivalent to
Original image is divided into one by one 4 × 4 region, classifies to each point in fifth feature figure, is equivalent to in original image
Each 4 × 4 region carries out classification and Detection, and then show that the region in original image is occupied situation, finally from the detection knot of original image
The case where predeterminable area is occupied is obtained in fruit, and then obtains target area testing result.
Preferably, S221 is handled the fisrt feature figure according to default step, makes point of the fisrt feature figure
Resolution promotes one times, obtains third feature figure and specifically includes:
Deconvolution is carried out to the fisrt feature figure;
Channel separation is carried out to the fisrt feature figure after deconvolution;
Process of convolution is carried out to the fisrt feature figure after channel separation using two different convolution kernels;
Channel fusion is carried out to the characteristic pattern obtained after process of convolution, obtains third feature figure.
Specifically, default step includes deconvolution, channel separation, process of convolution and channel fusion.Its specific steps such as Fig. 5
It is shown.Deconvolution first is carried out to the 1/32 fisrt feature figure that resolution ratio is original image, then carries out channel separation, then carry out respectively
3 × 3 convolution sum, 5 × 5 convolution, finally carries out channel fusion again, obtains third feature figure, the resolution ratio of obtained third feature figure
One times is promoted, the resolution ratio of third feature figure is the 1/16 of original image.
It will be merged after second feature figure 1 × 1 process of convolution of progress with third feature figure, obtained fourth feature figure
Resolution ratio is also the 1/16 of original image.Then the resolution ratio carried out twice according still further to above-mentioned default step to fourth feature figure mentions
It rises, the resolution ratio of obtained fifth feature figure is the 1/4 of original image.By by the 1/ of the increase resolution of fifth feature figure to original image
4, keep the region segmentation carried out to original image more reasonable, avoid because region segmentation is excessive or it is too small due to it is influence area classification and Detection
Accuracy rate, while detection of the algorithm Objects as Points to target can also be taken into account.
It should be understood that in the above embodiments, the size of each step number is not meant that the order of the execution order, it is each to walk
Rapid execution sequence should determine that the implementation process without coping with the embodiment of the present invention constitutes any limit with function and internal logic
It is fixed.
The 5th embodiment provided according to the present invention, as shown in fig. 6, the system that elevator is taken by a kind of robot, comprising:
Image collection module 10, for obtaining the image of elevator interior when elevator door is opened;
Detection module 20 obtains target detection for inputting described image in trained neural network model in advance
As a result with predeterminable area testing result, the predeterminable area is the region that robot enters that elevator needs to occupy;
Processing module 30, for judging whether to enter according to the object detection results and predeterminable area testing result
The elevator, if so, if it is not, then discharging the elevator, and calling elevator again into the elevator.
Specifically, the present embodiment is the corresponding Installation practice of above method embodiment, specific effect is referring to above-mentioned implementation
Example, this is no longer going to repeat them.
The sixth embodiment provided according to the present invention, a kind of system that elevator is taken by robot, in above-mentioned 5th embodiment
On the basis of, processing module 30 includes:
Processing unit, for judging whether the predeterminable area is occupied according to the predeterminable area testing result, if
It is no, then determine that the elevator can be entered;
Processing unit, if being also used to judge that the predeterminable area is occupied, then root according to the predeterminable area testing result
According to the object detection results, judgement occupies whether object is object;
Processing unit, if being also used to the object that occupies is object, judgement can not enter the elevator;
Processing unit exports evacuation information if being also used to the object that occupies is not object, after waiting preset time, then
The secondary new images for obtaining the elevator interior, the new images are inputted in the neural network model, obtain target inspection again
Result and predeterminable area testing result are surveyed, and according to the object detection results and predeterminable area testing result obtained again, judgement
Whether can enter the elevator, when repeating to obtain the number of image of the elevator interior greater than preset threshold, then determine not
The elevator can be entered.
Specifically, the present embodiment is the corresponding Installation practice of above method embodiment, specific effect is referring to above-mentioned implementation
Example, this is no longer going to repeat them.
The 7th embodiment provided according to the present invention, a kind of system that elevator is taken by robot, in above-mentioned 5th embodiment
Or on the basis of sixth embodiment, detection module 20 includes:
Feature extraction unit extracts difference for inputting described image in trained neural network model in advance
The characteristic pattern of layer;
Increase resolution unit carries out Fusion Features and increase resolution for the characteristic pattern to different layers;
Task output unit, for exporting target detection according to the characteristic pattern obtained after Fusion Features and increase resolution
As a result with predeterminable area testing result.
Specifically, the present embodiment is the corresponding Installation practice of above method embodiment, specific effect is referring to above-mentioned implementation
Example, this is no longer going to repeat them.
The 8th embodiment provided according to the present invention, a kind of system that elevator is taken by robot, in above-mentioned 7th embodiment
On the basis of,
Feature extraction unit is also used to input described image in trained neural network model in advance, extract most
The fisrt feature figure of later layer, and extract the second feature figure of front layer, the resolution ratio of the second feature figure are described the
Twice of the resolution ratio of one characteristic pattern;
Increase resolution unit includes:
Increase resolution subelement, for handling the fisrt feature figure, making described first according to default step
One times of the increase resolution of characteristic pattern, obtains third feature figure;
Fusion Features subelement, for by the second feature figure carry out 1 × 1 process of convolution after with the third feature figure
It is merged, obtains fourth feature figure;
Increase resolution subelement will for repeatedly being handled the fourth feature figure according to the default step
More times of the increase resolution of the fourth feature figure, obtains fifth feature figure;
Task output unit is also used to export object detection results and predeterminable area detection according to the fifth feature figure
As a result.
Preferably, increase resolution subelement is also used to carry out deconvolution to the fisrt feature figure;After deconvolution
The fisrt feature figure carries out channel separation;Using two different convolution kernels to the fisrt feature figure after channel separation into
Row process of convolution;Channel fusion is carried out to the characteristic pattern obtained after process of convolution, obtains third feature figure.
Specifically, the present embodiment is the corresponding Installation practice of above method embodiment, specific effect is referring to above-mentioned implementation
Example, this is no longer going to repeat them.
It should be noted that above-described embodiment can be freely combined as needed.The above is only of the invention preferred
Embodiment, it is noted that for those skilled in the art, in the premise for not departing from the principle of the invention
Under, several improvements and modifications can also be made, these modifications and embellishments should also be considered as the scope of protection of the present invention.