The content of the invention
The object of the present invention is to provide a kind of scene cut and the method and system of detection of obstacles, this method and system are borrowed
Help feature to share, and merged scene separation and detection of obstacles, so as to drastically reduce the memory money that algorithm is consumed
Source improves the speed of algorithm, and scene cut and obstacle detection is made to have more real-time.
To achieve these goals, a kind of method that the present invention provides scene cut and detection of obstacles, which is characterized in that
This method includes:The feature f of scene image is extracted, can be realized using convolution and pondization operation;The feature f is divided
Class obtains and carries out sorted feature f to each scene characteristic in the scene images;By the feature f and the feature
fsFusion is characterized fd;And it by the area-of-interest in feature fd described in Region Feature Extraction, obtains to the region of interest
The target classification result in domain.
Wherein, it is described to classify to the feature f, it obtains after classifying to each scene characteristic in scene image
Feature fsIncluding:Scene cut is carried out according to the feature f, and output token is per the scene cut result of a kind of scene
fScene cut;And to the scene cut result fScene cutFurther judge the concrete kind per a kind of scene by pixel classifications
Not, the feature f is obtaineds。
Wherein, it is described to the scene cut result fScene cutFurther judged by pixel classifications described per a kind of scene
Specific category, obtain the feature fsIncluding:Using sorting algorithm to the scene cut result fScene cutIt carries out at classification
Reason;And by top sampling method by the size restoration of sorted scene cut result to original size, so as to obtain with it is former
The feature f of the corresponding classification results of beginning sizes。
Wherein, feature f described in Region Feature Extraction is passed throughdIn area-of-interest, obtain to the area-of-interest
Target classification result includes:Obtain the feature fdIn area-of-interest coordinate;According to the feature fdWith it is described interested
The coordinate in region obtains the feature f of area-of-interest by Region Feature Extractionr;To the feature frIn area-of-interest into
Row classification obtains the target classification result of area-of-interest.
Wherein, the method further includes:According to the feature frBy convolution operation, the position for obtaining area-of-interest is sat
Target offset;And the coordinate of the area-of-interest is adjusted according to the offset.
According to another aspect of the present invention, the system for also providing a kind of scene cut and detection of obstacles, the system include:
Shared network, for extracting the feature f of scene image;Scene cut network, for classifying to the feature f, acquisition pair
Each scene characteristic in the scene image carries out sorted feature fs;Merge network, for by the feature f and described
Feature fsFusion is characterized fd;And target detection network, for passing through the region of interest in feature fd described in Region Feature Extraction
Domain obtains the target classification result to the area-of-interest.
Wherein, the scene cut network includes:Scene cut module, for carrying out scene cut according to the feature f,
And output token is per the scene cut result f of a kind of sceneScene cut;And sort module, for the scene cut result
fScene cutFurther judge the specific category per a kind of scene by pixel classifications, obtain the feature fs。
Wherein, the sort module includes:Classification processing module, for utilizing sorting algorithm to the scene cut result
fScene cutCarry out classification processing;And up-sampling module, for passing through top sampling method by the ruler of sorted scene cut result
It is very little to be restored to original size, so as to obtain the feature f of classification results corresponding with original sizes。
Wherein, the target detection network includes:RPN (zone scheme network, Region Proposal Network) net
Network, for obtaining the coordinate of the area-of-interest in the feature fd;Region Feature Extraction network, for according to the feature fd
With the coordinate of the area-of-interest, pass through the feature f of Region Feature Extraction acquisition area-of-interestr;And target classification net
Network for classifying to the area-of-interest in the feature fr, obtains the target classification result of area-of-interest.
Wherein, which further includes position Recurrent networks, for according to the feature frBy convolution operation, it is emerging to obtain sense
The offset of the position coordinates in interesting region, and according to the coordinate of the offset adjustment area-of-interest.
Through the above technical solutions, the scene cut and the method and system of detection of obstacles, are shared using feature, it will
The feature extracted from original scene image shares to scene cut and target detection (or being detection of obstacles), and target
Detection improves scene cut and target inspection further using scene cut as a result, so as to save substantial amounts of algorithm resource
The speed of survey, and this method and system combine scene cut and target detection, can export simultaneously scene segmentation result and
Object detection results (or detection of obstacles result).
Other features and advantages of the present invention will be described in detail in subsequent specific embodiment part.
Specific embodiment
The specific embodiment of the present invention is described in detail below in conjunction with attached drawing.It should be appreciated that this place is retouched
The specific embodiment stated is merely to illustrate and explain the present invention, and is not intended to limit the invention.
Fig. 1 is the flow chart of the method for one scene cut and detection of obstacles according to an embodiment of the invention.Such as Fig. 1 institutes
Show, this method may comprise steps of:
In the step s 100, the feature f of scene image is extracted.By operations such as multiple convolution and ponds, it is to size
Wi×HiPending original scene image carry out preliminary feature extraction, and the feature f that final output dimension is W × H × N
The feature representation of (feature map) as the image, wherein, Wi、HiThe width and height of respectively original scene image,
W, H, N are respectively width, height and the port number of this feature expression.
In step s 200, classify to the feature f, obtain and each scene characteristic in scene image is divided
Feature f after classs.F is exported for example, image convolution can be passed through and finally obtain the scene cut that dimension size is W × H × ns,
Middle n is scene type number to be split.This step can export scene cut as a result, the result of the scene cut can be with
Such as it is used to identify road or positioning etc. in Vehicular automatic driving.
In step S300, by the feature f and the feature fsFusion is characterized fd.It for example, can be by feature f and field
The feature f of scape segmentation outputsIt is input to concat network layers, concat network layers are by feature f and feature fsMerge, so as to obtain
Dimension is the fusion feature f of W × H × (N+n)d。
In step S400, pass through feature f described in Region Feature ExtractiondIn area-of-interest, obtain to region of interest
The target classification result in domain.By the step, each target is identification as a result, simultaneously in the final exportable image to original scene
It is marked, for example, when applied to automatic Pilot, can identify and mark the obstacles such as automobile, the pedestrian in driving front
Object.
Fig. 2 is the flow chart of the method for two scene cut and detection of obstacles according to an embodiment of the invention.Such as Fig. 2 institutes
Show, on the basis of embodiment one, the step S200 may comprise steps of:
In step S210, scene cut is carried out by convolution operation according to the feature f, and output token is per a kind of field
The scene cut result f of scapeScene cut.Each target in scene image can roughly be divided by difference by convolution operation
Classification, and each of which class scene can be marked.For precise classification, which can include multilayer convolution and grasp
Make.
In S220, to the scene cut result fScene cutFurther judged by pixel classifications described per a kind of scene
Specific category, obtain the feature fs。
Wherein, as shown in figure 3, step S220 can preferably include following steps:
In step S221, using sorting algorithm to the scene cut result fScene cutCarry out classification processing.It for example, can
To utilize the most described scene cut result f of Softmax classificationScene cutIn marked each classification its tool that may belong to
Body classification is matched, for example, it is assumed that fScene cutValue at (p, q) isWhereinRepresent fs
Value at the kth passage of (p, q) point, the then probability that (p, q) belongs to kth class scene are
Therefore, the scene type belonging to (p, q) isWhen the probability closer to 1 when, can
By fScene cutValue at (p, q) can be classified into kth class scene.The kth class scene can be road, pedestrian, building
Object, vehicle etc..The scene cut result that size is W × H can be obtained by the step.
In step S222 and by top sampling method by the size restoration of sorted scene cut result to original
Size, so as to obtain the feature f of classification results corresponding with original sizes.In the extraction feature of original scene image and right
The feature of extraction further during classification, in order to reduce operand, make use of pondization to operate, the image after pondization operation
Size ratio original size it is small, therefore for the size for reducing original image, make scene cut result and object detection results
It is more true, the top sampling method using quadratic interpolattion may be employed, by the scene cut result be reduced to size for Wi ×
The segmentation result of Hi, wherein WiAnd HiIt is the size of original scene image respectively.
As shown in figure 3, in embodiment two, the step S400 may comprise steps of:
In step S410, the feature f is obtaineddIn area-of-interest coordinate.The area-of-interest can be
Automobile, pedestrian, road in scene image etc..The step can for example be completed by convolution operation.
In the step s 420, according to the feature fdWith the coordinate of the area-of-interest, obtained by Region Feature Extraction
Take the feature f of area-of-interestr。
In step S430 and to the feature frIn area-of-interest classify, obtain area-of-interest
Target classification result.The classification for example may be employed Softmax sorting algorithms mentioned above and be realized based on territorial classification.
As shown in figure 3, in embodiment two, the method for the scene cut and detection of obstacles can further include
Following steps:
In step S510, according to the feature frBy convolution operation, obtain area-of-interest position coordinates it is inclined
Shifting amount.
The coordinate of the area-of-interest is adjusted in step S520 and according to the offset, so that target is examined
The mark result of survey is more accurate.
Fig. 4 is the structure chart of the system of three scene cut and detection of obstacles according to an embodiment of the invention.Such as Fig. 4 institutes
Show, which includes:Shared network 100, for extracting the feature f of scene image;Scene cut network 200, for described
Feature f classifies, and obtains and carries out sorted feature f to each scene characteristic in scene images;Merge network 300, use
In by the feature f and the feature fsFusion is characterized fd;And target detection network 400, for passing through Region Feature Extraction
The feature fdIn area-of-interest, obtain the target classification result to area-of-interest.Wherein, shared network can be
VGG models, LEnet models, Alexnet models, ResNet models etc. can need to select according to specific exploitation.
By means of sharing network 100, the feature of original image is extracted, this feature can be applied to scene cut network simultaneously
200 and target detection network 400, while the output result of scene cut network 200 can also be by 400 profit of target detection network
With, thus in combination with two kinds of functions of scene cut and target detection, and save substantial amounts of algorithm resource.
Fig. 5 is the structure chart of the system of four scene cut and detection of obstacles according to an embodiment of the invention.Such as Fig. 5 institutes
Show, in example IV, the scene cut network 200 can include:Scene cut module 210, for according to the feature f
Scene cut is carried out, and output token is per the scene cut result f of a kind of sceneScene cut;And sort module 220, for institute
State scene cut result fScene cutFurther judge the specific category per a kind of scene by pixel classifications, obtain the spy
Levy fs。
Further, as shown in fig. 6, the sort module can also preferably include:Classification processing module 221, is used for
Using sorting algorithm to the scene cut result fScene cutCarry out classification processing;And up-sampling module 222, for passing through
(Upsampling) method of sampling by the size restoration of sorted scene cut result to original size, so as to obtain with it is original
The feature f of the corresponding classification results of sizes.Processing module 221 of classifying is to scene cut result fScene cutThe mistake specifically classified
Journey may be employed Softmax algorithms mentioned above and realize.
As shown in figure 5, in example IV, the target detection network can preferably include:RPN networks 410, are used for
Root obtains the feature fdIn area-of-interest coordinate;Region Feature Extraction network 420, for according to the feature fdWith
The coordinate of the area-of-interest obtains the feature f of area-of-interest by Region Feature Extractionr;And target classification network
430, for the feature frIn area-of-interest classify, obtain the target classification result of area-of-interest.
Wherein, in example IV, the system of the scene cut and detection of obstacles further includes position Recurrent networks
500, for obtaining the offset of the position coordinates of area-of-interest, and the area-of-interest is adjusted according to the offset
Coordinate.
Fig. 7 is the signal of the realization principle of the system of four scene cut and detection of obstacles according to an embodiment of the invention
Figure.With reference to figure 7, original scene image is input to shared network, and the feature W × H being extracted is exported after shared network processes
× N, this feature output can export after scene cut network processes carries out original image specific sorted segmentation result W
× H × n, and scene segmentation figure picture can be exported simultaneously.The feature and scene that target detection network is extracted according to shared network are divided
Feature W × H × (N+n) of the scene cut result of network output after merging network processes is cut, further detects original graph
Each target (or being barrier) as in, and final output classification results.
Fig. 8 and Fig. 9 is the detection knot when system of the scene cut and detection of obstacles is applied to automatic driving
Fruit example.
As shown in figure 8, be the example of the result of detection of obstacles, it in fig. 8, can be accurate by the system of the present invention
The barriers such as vehicle, the pedestrian of identifying rows front side so as to the car deceleration or avoidance in control process automatic Pilot, are kept away
Exempt from traffic accident.Using the position Recurrent networks 500, for example, the box deviation lower section of the vehicle when mark driving front, does not have
Have that when including being marked as the region where the automobile of barrier completely, position Recurrent networks are by identifying mark position and automobile
The deviation of the coordinate of actual position, and further using the deviation adjusting mark position makes to the mark of barrier more
Accurately.
As shown in figure 9, it is a kind of example of the scene cut result of scene cut network output, in fig.9, original image
Middle different scene type, such as automobile, road, building etc. are mutually split, and are usually marked with different phases.Example
Such as in automatic Pilot, the road in scene cut result identification front can be utilized, so as to assist positioning traffic route.
In vehicle traveling process, some barriers (or being target), such as building, road etc., due to the sky occupied by its volume
Between it is larger, it is not easy to marked by target detection, can identify that these are not easy by the image of scene cut result at this time
The barrier of mark.
It will be appreciated by those skilled in the art that it is that can pass through to implement the method for the above embodiments
Program instructs relevant hardware to complete, which uses including some instructions so that one in a storage medium
A (can be microcontroller, chip etc.) or processor (processor) perform the whole of each embodiment the method for the application
Or part steps.And foregoing storage medium includes:USB flash disk, mobile hard disk, read-only memory (ROM, Read-Only
Memory), random access memory (RAM, Random Access Memory), magnetic disc or CD etc. are various can store journey
The medium of sequence code.
The preferred embodiment of the present invention is described in detail above in association with attached drawing, still, the present invention is not limited to above-mentioned realities
The detail in mode is applied, within the scope of the technical concept of the present invention, a variety of letters can be carried out to technical scheme
Monotropic type, these simple variants all belong to the scope of protection of the present invention.
It is further to note that the specific technical features described in the above specific embodiments, in not lance
In the case of shield, can be combined by any suitable means, in order to avoid unnecessary repetition, the present invention to it is various can
The combination of energy no longer separately illustrates.
In addition, various embodiments of the present invention can be combined randomly, as long as it is without prejudice to originally
The thought of invention, it should also be regarded as the disclosure of the present invention.