CN108245385B - A kind of device helping visually impaired people's trip - Google Patents
A kind of device helping visually impaired people's trip Download PDFInfo
- Publication number
- CN108245385B CN108245385B CN201810037687.4A CN201810037687A CN108245385B CN 108245385 B CN108245385 B CN 108245385B CN 201810037687 A CN201810037687 A CN 201810037687A CN 108245385 B CN108245385 B CN 108245385B
- Authority
- CN
- China
- Prior art keywords
- image
- unit
- visually impaired
- cloud
- impaired people
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61H—PHYSICAL THERAPY APPARATUS, e.g. DEVICES FOR LOCATING OR STIMULATING REFLEX POINTS IN THE BODY; ARTIFICIAL RESPIRATION; MASSAGE; BATHING DEVICES FOR SPECIAL THERAPEUTIC OR HYGIENIC PURPOSES OR SPECIFIC PARTS OF THE BODY
- A61H3/00—Appliances for aiding patients or disabled persons to walk about
- A61H3/06—Walking aids for blind persons
- A61H3/061—Walking aids for blind persons with electronic detecting or guiding means
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61H—PHYSICAL THERAPY APPARATUS, e.g. DEVICES FOR LOCATING OR STIMULATING REFLEX POINTS IN THE BODY; ARTIFICIAL RESPIRATION; MASSAGE; BATHING DEVICES FOR SPECIAL THERAPEUTIC OR HYGIENIC PURPOSES OR SPECIFIC PARTS OF THE BODY
- A61H2201/00—Characteristics of apparatus not provided for in the preceding codes
- A61H2201/16—Physical interface with patient
- A61H2201/1602—Physical interface with patient kind of interface, e.g. head rest, knee support or lumbar support
- A61H2201/165—Wearable interfaces
Landscapes
- Health & Medical Sciences (AREA)
- Epidemiology (AREA)
- Pain & Pain Management (AREA)
- Physical Education & Sports Medicine (AREA)
- Rehabilitation Therapy (AREA)
- Life Sciences & Earth Sciences (AREA)
- Animal Behavior & Ethology (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- Traffic Control Systems (AREA)
- Image Analysis (AREA)
Abstract
The present invention provides a kind of device of help visually impaired people trip, comprising: depth image computing unit puts cloud construction unit, ground detection unit, object segmentation unit, for removing ground and below cloud, and the method based on cluster carries out point cloud segmentation, obtains each object；Subject tracking unit calculates it in the size of three bit spaces, the direction of motion, motion profile and movement velocity for tracking to each object；Object identification unit, for for each object, projection to obtain image-region, extraction RGB image, based on RGB image progress Object identifying；Speech synthesis and output unit, for by the type of object, position, the direction of motion and velocity composite voice, output to synthesize voice to notify visually impaired people.It provides comprehensive information for making decisions on one's own for visually impaired people, effective to help visually impaired people's avoiding obstacles, improves the safety of visually impaired people.
Description
Technical field
Present invention relates in general to using vision and voice technology to carry out the device of detection of obstacles and tracking, more specifically
Ground is related to cromogram using binocular camera or stereoscopic camera and depth map to detect barrier and carry out the dress of voice reminder
It sets.
Background technique
The perception information of the mankind 70% comes from vision；Visually impaired people makes up and enhances with greater need for more serious hope than normal person.
Visually impaired people thirsts for freely going on a journey.Outdoor trip, static and mobile barrier are likely encountered at any time, they from it is more
Far, orientation how, speed how them how are found in real time and notify the key point for being to aid in visually impaired people's trip.
Currently, academia and industry attempt to solve the problems, such as this attempting a variety of methods.
Oxonian Shi Difen Hicks research in 2014 adds the method for OLED vision enhancement using depth transducer, will
Depth image is converted into sound field.Image information is switched to the dot matrix of electrod-array by BrainPort project research in 2014, by tongue
Head feels to attempt to make brain to form visual perception.When both methods is required to visually impaired people and carries out considerably long one section using equipment
Between training, to attempt to make visually impaired people oneself to form a kind of new cognitive method to space.It sees at present, enforcement difficulty is big,
Also it is still necessary to verify for effect.
The Dulight of 2015 Nian9Yue Baidu rely on the image recognition of " Baidu's brain ", recognition of face, speech recognition and
The related core technologyes such as deep learning make the daily life Personal Assistant of blind person.Its combine Baidu's big data analysis ability and
Natural human-computer interaction technology helps blind person " to see very clearly " real world, object that camera is seen etc. can be informed blind person, but nothing
Method knows state (distance, motion state, size etc.) of the object relative to blind person, helps outdoor trip limited.
The Seeing AI application publication of Microsoft in 2017, uses monocular glasses+depth learning technology (mobile phone application)
Mode assists mainly in blind person to understand scene, interact with voice mode and blind person.It is aligned at one's side by camera
People or object can be the explanation of blind person's language, serve as the eyes of blind person, can carry out recognition of face, and can say this person by voice
Gender, approximate age and present emotional status, it is outgoing doorplate to be eaten also by " Seeing AI " when doing shopping
Product information even currency is identified to be quoted by voice again.The birth of " Seeing AI " can allow more amblyopia groups most
Possible " independence ", narrows the gap with normal life.Its existing main problem is also that object can not be known relative to blind person's
State (distance, motion state, size etc.) helps outdoor trip limited.
The collaborative project NavCog research of IBM and Ka Neijimeilong in 2015 uses the side of bluetooth beacon combination cell phone application
Formula is improved Voice Navigation for blind person.This App can be blind man navigation by voice and vibration two ways, while can also be
The processing of ambient enviroment locating for blind person is 3d space model, is passed in their ears by ultrasonic wave, by earphone voice or is led to
It crosses and creates subtle smart phone vibration, more accurate positioning and navigation are provided.In order to allow blind person to know whom has at one's side, it
Human face scanning function is also provided heart to heart, and is able to tell that the mood of the people of blind person at one's side.Its main problem is to need to rely on
Signal is provided in bluetooth beacon, is a kind of eaily signal transmitting carrier.But bluetooth beacon is rare in China, greatly limits
The feasibility of its application is made.
In addition, to help outdoor trip, glasses for guiding blind product also has research and development, is made of electronic box and glasses two parts.
Two ultrasonic transducers and an insert earphone are housed, energy converter can emit forwards and receive reflected ultrasound on glasses
Impulse wave, blind person perceive the barrier in front by the sound variation that earphone issues.Specifically, glasses for guiding blind is by integrated electricity
Road device is mounted on spectacle frame and spectacle lens, and Vltrasonic device constantly emits forward ultrasonic wave, is reflected when encountering barrier
Back, after the ultrasonic probe, ultrasonic receiver on glasses receives, by the processing of electronic circuit, become audible sound, the sound
The tone and repetition rate of sound are changed as the distance of barrier changes, and blind person can be according to sound, tone and repetition frequency
Rate judges that there are clear and obstacle distance in front, so as to avoid collision, so that blind person is walked convenient, reduces the pain of collision
Hardship is the advanced tools that blind person's guide is led the way.Glasses for guiding blind is front to the investigative range of small barrier, left and right is 10 degree each,
5 degree of top, 35 degree of lower section can detect each 0.4 meter around, 0.2 meter of top, lower section 1.4 in 2 meters from glasses remote places
Barrier in the spatial dimension of rice；For big barrier, investigative range will also be more greatly.Detection range is divided to 1.5 meters and 4 meter two
Shelves are selected by the toggle switch on electronic box.When encountering barrier in certain distance, earphone just issues the sound, barrier
Closer, sound repeats more rapid, and tone is also higher and higher.Power supply uses 9 volts of 6F22 laminated cells of a section, and a batteries can make
With 60 hours.In use, first wearing glasses, choose apart from shelves: the place more than the people should select 1.5 meters of distance shelves, few in people
Optional 4 meters of the distance shelves in place.The problem of product maximum is that the field range of work is small, operating distance is close, can not know
Type, the size motion state of barrier.
Summary of the invention
In view of the foregoing, the present invention provides a kind of devices of help visually impaired people trip, for overcoming the prior art
Existing various problems.
It is as follows that the present invention solves the technical solution that above-mentioned technical problem is taken:
Depth image computing unit obtains left and right color image for connecting binocular camera, and based on binocular camera, calculates
Obtain depth image, synchronism output color image and depth image；
Point cloud construction unit, is based on color image and depth image, constructs point cloud (x, y, z, r, g, b) information, i.e., each
Point includes 3 dimension space coordinate informations and 3 dimension colour informations；
Ground detection unit, the 3 dimension space coordinate informations (x, y, z) based on cloud carry out plane monitoring-network, and output position is most
It is low and include the most plane of points as reference ground；
Object segmentation unit removes ground and below cloud, and the method based on cluster carries out point cloud segmentation, obtains each
Object；
Subject tracking unit tracks each object, calculates it in the size, the direction of motion, movement of three bit spaces
Track and movement velocity；
Object identification unit, for each object, projection obtains image-region, extracts RGB image, based on RGB image come
Carry out Object identifying；
Speech synthesis and output unit, by the type of object, position, the direction of motion and velocity composite voice, output synthesis
Voice notifies visually impaired people.
Compared with the method for relevant help blind person trip, difference is analyzed as follows the present invention:
The present invention has the advantages that field angle is big, visual range is wide using binocular equipment, depth image is obtained by calculation
Being converted to 6D point cloud can accurate description outdoor space scene.It is real based on 3D coordinate and colour information using embedding assembly equipment
When ground is calculated, Real-time segmentation obtains the types of objects on ground, and carries out the real-time tracking of object, Object identifying, so
Synthesis voice informs visually impaired people with a kind of natural way afterwards.The device, which uses, is similar to the think of that the mankind carry out 3D visual perception
Road understands outdoor environment, has a certain robustness to illumination variation, at the same the size of its perceptive object, track, motion state and
Classification information provides comprehensive information for making decisions on one's own for visually impaired people.
Detailed description of the invention
Fig. 1 shows a kind of system diagram for helping visually impaired people's trip device according to the present invention；
Fig. 2 shows a kind of functional block diagrams for helping visually impaired people's trip device according to an embodiment of the present invention；
Fig. 3, which is shown, according to an embodiment of the present invention illustrates the mankind to 3d space and directional perception schematic diagram；
Fig. 4 shows depth image computing unit output RGB figure and depth map schematic diagram according to an embodiment of the present invention；
It is different object 3D point cloud schematic diagrames that Fig. 5, which shows point cloud segmentation according to an embodiment of the present invention,；
Fig. 6 shows the result schematic diagram of Object identifying according to an embodiment of the present invention.
Specific embodiment
In order to make those skilled in the art more fully understand the present invention, with reference to the accompanying drawings and detailed description to this hair
It is bright to be described in further detail.
Firstly, we analyze the concrete behavior mode that the mankind carry out outdoor trip.What outdoor trip faced is one true
3 real dimension spaces.3 dimension space of human perception is eyes by the way of.Distance perception and spatial impression are generated by eyes.Such as Fig. 3
Illustrated, the mankind have spatial impression to can determine the 3 dimension coordinates in space, there is sense of direction.Then, the presence of perceiving ground, ground
The region that can usually walk.Whether see on ground has the object for stopping to walk.If so, the size of judgment object, object
The type of the direction of motion, track and speed and object.It is judged according to these information brains on ground, before we are
Into, stay in original place, retreat still dodge.
Secondly, we analyze visually impaired people's state in which, i.e., where their ability and missing.Ignorantly scarce is view
Feel perception part, i.e., it is to its unfamiliar outdoor environment, does not know in 3 dimension spaces how is walkable region, there are those right
Its walking is influenced as existing.Its hearing ability is usually powerful, can quickly receive a large amount of voice messagings.Meanwhile having complete
High-speed decision and relevant action ability.It concludes, what visually impaired people uniquely lacked is outdoor 3D sensing capability, that is, lacks perception
Object wherein, object basic exercise state, the ability of object type.
Based on above-mentioned analysis, core of the invention thinking is to be obtained using the equipment for being similar to mankind's eyes to 3d space
The perception of distance carries out Object Segmentation and Obj State by ground sension unit, object segmentation unit, tracking and recognition unit
The Obj State information of acquisition is generated voice by speech synthesis unit by perception.Voice fast notification visually impaired people, by visually impaired
Personage independently completes decision, supports its outdoor trip.
Wherein, Fig. 1 is shown and is mainly given capital equipment of the invention and its defeated using system schematic of the invention
Enter and exports.
Wherein, acquisition equipment be a microminiature binocular camera, support USB power supply, can synchronous acquisition two-way colour
Image, as the binocular camera passes through the color image of camera acquisition two sides or so.
Another processing equipment is embedded processing equipment, and input is the two-way color image of the above synchronization, by equipment
In depth image computing unit, point cloud construction unit, ground detection unit, object segmentation unit, subject tracking unit and language
After the processing of sound synthesis unit, verbal announcement information is exported to visually impaired people.Inform in front space there is which kind of type on road surface
Object and object movement state information (size, the direction of motion, track and speed).Wherein, embedded processing equipment can be place
The strong cell phone apparatus of reason ability is also possible to other and contains embeded processor and have setting for deep learning processing capacity
Standby, which can use fexible unit on the market.
Fig. 2 shows a kind of functional block diagrams for helping visually impaired people's trip device according to an embodiment of the invention.
The functional module of device specifically includes that depth image computing unit 11, for connecting above-mentioned binocular camera, is obtained based on binocular camera
Left and right color image is taken, to carry out that depth image, and synchronism output color image and depth image is calculated；
Wherein, depth image (depth image) is also referred to as range image (range image), and referring to will be from image
Image of the distance (depth) of collector each point into scene as pixel value, it directly reflects the geometry of scenery visible surface
Shape.Depth image may be calculated point cloud data by coordinate conversion, and regular and necessary information point cloud data can also be with
Inverse is depth image data.
In picture frame provided by depth data stream, each pixel represent be in the visual field of depth inductor,
At specific (x, the y) coordinate with a distance from object to the object to the plane nearest from camera shooting head plane (in millimeters).
In this patent, further includes: point cloud construction unit 12 is based primarily upon color image and depth image, building point cloud
(x, y, z, r, g, b) information, i.e., each point include 3 dimension space coordinate informations and 3 dimension colour informations；
Ground detection unit 13, the 3 dimension space coordinate informations (x, y, z) based on cloud carry out plane monitoring-network, output position
It is minimum and include the most plane of points as reference ground；
Object segmentation unit 14 removes ground and below cloud, and the method based on cluster carries out point cloud segmentation, obtains each
A object；
Subject tracking unit 15, tracks each object, calculates it in the size, the direction of motion, fortune of three bit spaces
Dynamic rail mark and movement velocity；
Object identification unit 16, for each object, projection obtains image-region, extracts RGB image, is based on RGB image
To carry out Object identifying；
Speech synthesis and output unit 17, by the type of object, position, the direction of motion and velocity composite voice, output is closed
Visually impaired people is notified at voice.
Wherein, the concrete operating principle of each unit is described in detail below.
Depth image computing unit 11 obtains synchronous left and right color image from binocular camera, calculates depth image.
Binocular camera, can take at present common binocular solid camera on the market, and function is mainly used for rapid build
Three-dimensional video-frequency and stereo reconstruction.It can use binocular solid matching primitives, can obtain depth information of scene and three-dimensional mould in real time
Type.
In the present embodiment, binocular camera is mainly used for shooting the spatial image information (such as left images) of human peripheral.Its
Before use, being demarcated firstly the need of by collected left images.The generation of demarcating file, can be using based on chessboard
The method of lattice obtains the distortion matrix, camera internal reference and outer ginseng of left and right camera, and specific implementation can refer to OpenCV stereoscopic vision
Part is not described in detail herein.
For this purpose, the depth image computing unit 11, for inputting the demarcating file of left and right camera calibration, based on what is obtained
The distortion factor and internal reference of each road camera itself and outer ginseng complete correction and row pair to two-way camera acquired image frame
Together.
Wherein, specific steps include:
Using the left and right figure after correction, depth image is calculated based on matching technique.
Wherein, specific matching technique can cut method, half global registration using traditional Block- matching, dynamic programming, figure.
A kind of optimized calculation method of improved depth can also be used.
The key step of this method, specifically includes: firstly, obtaining initial parallax in conventional parallax calculation method；Then, right
All pixels construct a graph model in image, and the node of figure is the parallax value of pixel, similarity of the side of figure between pixel
Amount；The parallax value of pixel is propagated to reach the optimal of the overall situation by the successive ignition in graph model.According to the outer ginseng of camera and interior
Parallax information is converted to depth information by ginseng.
Fig. 4 gives the signal of depth image computing unit output RGB figure and depth map, and left figure is the cromogram of output
Picture, right figure be the depth image that is calculated wherein different colours represent the different pixels to image center distance value.
Described cloud construction unit 12, obtains color image and depth image first, and the projection inverse transformation based on camera is public
Formula obtains the 3D point cloud (x, y, z) of 3 dimension spaces；Each point respectively corresponds a pixel (p_ in color image in 3D point cloud
X, p_y), rgb data (r, g, b) of the pixel in color image containing triple channel.Each cloud (x, y, z) takes its colour
6D point cloud information (x, y, z, r, g, b) can be obtained in information (r, g, b).
The ground detection unit 13, for obtaining and obtaining above-mentioned 3D point cloud (x, y, z)；Plane is carried out in 3D point cloud
Fitting obtains multiple planes；Count the following parameter information of each plane: the height of number point_number, the 3D point of 3D point
The x coordinate of the average value height_average of value, minimax: x_max and x_min, z coordinate: the z_max of minimax and
z_min；Determine maximum planes based on following rule: height_average value is small, (x_max-x_min) * (z_max_z_
Min) greatly, point_number is big；The parameter of output maximum planes is the parameter on ground.
Object segmentation unit 14 connects the above ground detection unit 13, and first according to obtained ground, removably bread contains
Point, remove ground point below, specifically according to the y-coordinate of cloud with the y-coordinate on ground carry out traversal comparison can remove
The point of below ground.
For remaining cloud (x, y, z, r, g, b), different point cloud objects is divided into using the method for cluster, it can
Object Segmentation is directly carried out to a cloud (x, y, z, r, g, b) using KNN method；Density estimation method K mean value and LBP can also be used
Density Estimator is split.KNN, K mean value and LBP Density Estimator method are disclosed dividing methods, realize details herein
It is not repeating.
After Fig. 5 gives progress point cloud segmentation, the 3D point cloud (x, y, z) of obtained different objects is illustrated.
The subject tracking unit 15, carrying out tracking to each object includes:
Obtain the point cloud information (x, y, z, r, g, b) of each object；
Core correlation filtering KCF is based on to each object to track, it can also be using the more simple Kalman's filter of algorithm
Wave or particle filter carry out, and KCF, Kalman filtering and particle filter are well-known technique, realize that details is not repeating herein；
For each object traced into, really it is sized by point cloud that each object includes first, it is specific to calculate
Coordinate the maximum and minimum value x_max, x_min, y_max, y_min, z_max, z_min of each dimension of each object can be true
The size of fixed each object；
The point cloud for including according to each object calculates its barycentric coodinates (xc, yc, zc)；
Wherein, the barycentric coodinates of different moments connect the motion profile that can obtain the object；The tangent line of track
Direction is the direction of motion of object；Movement velocity is calculated by the barycentric coodinates and time difference of different moments.
Object identification unit 16, for being determined corresponding for each cloud object by the projective transformation matrix of camera
Picture point calculates the circumscribed rectangular region of these picture points；
Extract the RGB image in region；
Object is identified based on learning method, provides the classification of cutting object, such as automobile, pedestrian, bicycle, rubbish
Rubbish bucket, trees, fence etc..
Wherein, object is identified based on learning method, provide the main algorithm of the classification of cutting object and realizes step
It suddenly include: off-line step and on-line steps.Off-line step includes: using disclosed object identification data set as training sample
Geometry, such as imageNet data acquisition system；Model training uses the deep learning of CNN, specifically carries out mould using ResNet50
Type training；The output parameter file of model is obtained after training.On-line steps: load ResNet50 model and Parameter File, to defeated
The RGB image of the cutting object entered carries out size change over, carries out inference based on ResNet50 model and completes Object identifying, output pair
The classification of elephant.
Fig. 6 gives the result signal of Object identifying.
Speech synthesis and output unit 17, by the synthetic technology from text to voice, by the type of object, position, fortune
Dynamic direction and velocity composite voice；Output synthesizes voice to notify visually impaired people.
Wherein, of the invention, have the advantages that field angle is big, visual range is wide using binocular equipment, depth is obtained by calculation
Degree image is converted to 6D point cloud can accurate description outdoor space scene.
Using embedding assembly equipment, ground is calculated based on 3D coordinate and colour information in real time, Real-time segmentation obtains
Types of objects on ground, and the real-time tracking of object, Object identifying are carried out, it then synthesizes voice and is informed with a kind of natural way
Visually impaired people.
The device, which is used, carries out the thinking of 3D visual perception similar to the mankind to understand outdoor environment, has to illumination variation
Certain robustness, while the size of its perceptive object, track, motion state and classification information, for making decisions on one's own for visually impaired people
Provide comprehensive information.
Basic principle of the invention is described in conjunction with specific embodiments above, however, it is desirable to, it is noted that this field
For those of ordinary skill, it is to be understood that the whole or any steps or component of methods and apparatus of the present invention, Ke Yi
Any computing device (including processor, storage medium etc.) perhaps in the network of computing device with hardware, firmware, software or
Their combination is realized that this is that those of ordinary skill in the art use them in the case where having read explanation of the invention
Basic programming skill can be achieved with.
Therefore, the purpose of the present invention can also by run on any computing device a program or batch processing come
It realizes.The computing device can be well known fexible unit.Therefore, the purpose of the present invention can also include only by offer
The program product of the program code of the method or device is realized to realize.That is, such program product is also constituted
The present invention, and the storage medium for being stored with such program product also constitutes the present invention.Obviously, the storage medium can be
Any well known storage medium or any storage medium that developed in the future.
It may also be noted that in the apparatus and method of the present invention, it is clear that each component or each step are can to decompose
And/or reconfigure.These, which decompose and/or reconfigure, should be regarded as equivalent scheme of the invention.Also, execute above-mentioned series
The step of processing, can execute according to the sequence of explanation in chronological order naturally, but not need centainly sequentially in time
It executes.Certain steps can execute parallel or independently of one another.
Above-mentioned specific embodiment, does not constitute a limitation on the scope of protection of the present invention.Those skilled in the art should be bright
It is white, design requirement and other factors are depended on, various modifications, combination, sub-portfolio and substitution can occur.It is any
Made modifications, equivalent substitutions and improvements etc. within the spirit and principles in the present invention, should be included in the scope of the present invention
Within.
Claims (6)
1. a kind of device for helping visually impaired people's trip, comprising:
Depth image computing unit for connecting binocular camera, and is based on the collected left and right color image of binocular camera, calculates
Obtain depth image, synchronism output color image and depth image；
Each point cloud construction unit constructs point cloud (x, y, z, r, g, b) information, i.e., for being based on color image and depth image
Point includes 3 dimension space coordinate informations and 3 dimension colour informations；
Ground detection unit, 3 dimension space coordinate informations (x, y, z) based on cloud carry out plane monitoring-network, output position it is minimum and
Comprising most plane of counting as reference ground；
Object segmentation unit, the method for removing ground and below cloud, and based on cluster carry out point cloud segmentation, obtain each
A object；
Subject tracking unit calculates it in the size, the direction of motion, movement of three bit spaces for tracking to each object
Track and movement velocity；
Object identification unit extracts RGB image for for each object, projection to obtain image-region, based on RGB image come
Carry out Object identifying；
Speech synthesis and output unit, for by the type of object, position, the direction of motion and velocity composite voice, output to be synthesized
Voice notifies visually impaired people；
Described cloud construction unit, for being based on color image and depth image, building point cloud (x, y, z, r, g, b) information, packet
It includes:
It obtains color image and depth image, the projection inverse transformation based on camera obtains the 3D point cloud (x, y, z) of 3 dimension spaces；
(r, g, the b) information for obtaining each cloud respective pixel in color image, obtains 6D point cloud information (x, y, z, r, g, b)
；
The ground detection unit, the 3 dimension space coordinate informations (x, y, z) based on cloud carry out plane monitoring-network, and output position is most
It is low and include the most plane of points as reference ground, comprising:
3D point cloud (x, y, z) is obtained, plane monitoring-network is carried out in 3D point cloud and obtains multiple planes；
Count the average value height_ of the height value of number point_number, the 3D point for the 3D point that each plane includes
Average, the x coordinate of minimax: the z coordinate of x_max and x_min, minimax: z_max and z_min；It is rule-based:
Point_number is maximized, and height_average is minimized, and (x_max-x_min) * (z_max_z_min) takes
Maximum value, height_average are minimized, and obtain the plane that area is maximum and position is minimum, using the plane as with reference to ground
Face.
2. the device according to claim 1 for helping visually impaired people's trip, which is characterized in that the depth calculation unit is based on
The collected left and right color image of binocular camera, depth image, which is calculated, includes:
Input the demarcating file of left and right camera calibration comprising the respective internal reference of two cameras and distortion parameter and camera
Spatial relation；
The distortion correction and alignment of left and right camera image are carried out based on demarcating file；
Using the left and right figure after correction, depth image is calculated based on matching technique.
3. the device according to claim 1 for helping visually impaired people's trip, which is characterized in that the object segmentation unit is used for
Ground and below cloud are removed, and the method based on cluster carries out point cloud segmentation, obtains each object, comprising:
Remove ground and its below cloud；
For remaining cloud (x, y, z, r, g, b), it is divided into different point cloud objects using the method for cluster, can be used
KNN method directly carries out Object Segmentation to a cloud (x, y, z, r, g, b)；Density estimation method K mean value can also be used and LBP core is close
Degree estimation is split.
4. the device according to claim 1 for helping visually impaired people's trip, which is characterized in that the subject tracking unit is used for
Each object is tracked, calculates it in the size of three bit spaces, the direction of motion, motion profile and movement velocity, comprising:
Each object is tracked, and obtains the point cloud information (x, y, z, r, g, b) of each object；
Core correlation filtering KCF is based on to each object to track, and may be based on the simpler Kalman filtering of algorithm or grain
Son filtering is carried out to promote tracking velocity；
Calculate size, the direction of motion, motion profile and the movement velocity of each object.
5. the device according to claim 1 for helping visually impaired people's trip, which is characterized in that the object identification unit is used for
For each object, projection obtains image-region, extracts RGB image, carries out Object identifying based on RGB image, comprising:
For each object, pass through the region in projective transformation to image；
Calculate circumscribed rectangular region；
Extract the RGB image in region；
Off-line training good ResNet50 model and Parameter File are loaded, size change over is carried out to region RGB image is extracted, is based on
ResNet50 model makes inferences, and exports the classification of object.
6. the device according to claim 5 for helping visually impaired people's trip, which is characterized in that the classification of the cutting object
Including automobile, pedestrian, bicycle, dustbin, trees or fence.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810037687.4A CN108245385B (en) | 2018-01-16 | 2018-01-16 | A kind of device helping visually impaired people's trip |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810037687.4A CN108245385B (en) | 2018-01-16 | 2018-01-16 | A kind of device helping visually impaired people's trip |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108245385A CN108245385A (en) | 2018-07-06 |
| CN108245385B true CN108245385B (en) | 2019-10-29 |
Family
ID=62726791
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810037687.4A Expired - Fee Related CN108245385B (en) | 2018-01-16 | 2018-01-16 | A kind of device helping visually impaired people's trip |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108245385B (en) |
Families Citing this family (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111368607B (en) * | 2018-12-26 | 2023-06-02 | 北京欣奕华科技有限公司 | Robot, obstacle detection method and detection device |
| CN110378246A (en) * | 2019-06-26 | 2019-10-25 | 深圳前海达闼云端智能科技有限公司 | Ground detection method, apparatus, computer readable storage medium and electronic equipment |
| CN110322515A (en) * | 2019-07-02 | 2019-10-11 | 工极智能科技（苏州）有限公司 | Workpiece identification and crawl point extracting method based on binocular stereo vision |
| CN112402194A (en) * | 2019-08-23 | 2021-02-26 | 北京超维度计算科技有限公司 | Auxiliary terminal system for visually impaired people |
| CN110470307A (en) * | 2019-08-28 | 2019-11-19 | 中国科学院长春光学精密机械与物理研究所 | A kind of visually impaired patient navigation system and method |
| CN111031468B (en) * | 2019-11-13 | 2021-02-05 | 电子科技大学 | Visual auxiliary method and device based on individualized HRTF stereo |
| CN110879991B (en) * | 2019-11-26 | 2022-05-17 | 浙江光珀智能科技有限公司 | Obstacle identification method and system |
| CN111932866A (en) * | 2020-08-11 | 2020-11-13 | 中国科学技术大学先进技术研究院 | Wearable blind person outdoor traffic information sensing equipment |
| US11734850B2 (en) * | 2021-04-26 | 2023-08-22 | Ubtech North America Research And Development Center Corp | On-floor obstacle detection method and mobile machine using the same |
| CN113459102B (en) * | 2021-07-09 | 2022-07-05 | 郑州大学 | Human upper limb intention identification method based on projection reconstruction |
| CN114419697B (en) * | 2021-12-23 | 2022-12-02 | 北京深睿博联科技有限责任公司 | Vision-impaired person prompting method and device based on mechanical vibration |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103955920B (en) * | 2014-04-14 | 2017-04-12 | 桂林电子科技大学 | Binocular vision obstacle detection method based on three-dimensional point cloud segmentation |
| CN104055657B (en) * | 2014-06-18 | 2016-05-25 | 浙江师范大学 | A kind of blind-guide brick and its implementation based on Kinect |
| ES2597155B1 (en) * | 2015-06-12 | 2017-09-18 | Eyesynth, S.L. | PORTABLE SOUND OR TOUCH INTERPRETATION SYSTEM OF THE ENVIRONMENT FOR AN INVIDENT |
| CN105686936B (en) * | 2016-01-12 | 2017-12-29 | 浙江大学 | A kind of acoustic coding interactive system based on RGB-IR cameras |
| CN106214437B (en) * | 2016-07-22 | 2018-05-29 | 杭州视氪科技有限公司 | A kind of intelligent blind auxiliary eyeglasses |
| CN107341789B (en) * | 2016-11-23 | 2019-12-17 | 杭州视氪科技有限公司 | System and method for predicting pathway of visually impaired people based on RGB-D camera and stereo |
-
2018
- 2018-01-16 CN CN201810037687.4A patent/CN108245385B/en not_active Expired - Fee Related
Also Published As
| Publication number | Publication date |
|---|---|
| CN108245385A (en) | 2018-07-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108245385B (en) | A kind of device helping visually impaired people's trip | |
| Bai et al. | Smart guiding glasses for visually impaired people in indoor environment | |
| EP3742332B1 (en) | Method and apparatus for training model for recognizing key points of hand, and method and apparatus for recognizing key points of hand | |
| CN104036488B (en) | Binocular vision-based human body posture and action research method | |
| US10528815B2 (en) | Method and device for visually impaired assistance | |
| CN104036546B (en) | Method for carrying out face three-dimensional reconstruction at any viewing angle on basis of self-adaptive deformable model | |
| KR102441171B1 (en) | Apparatus and Method for Monitoring User based on Multi-View Face Image | |
| JP6771548B2 (en) | A portable system that allows the blind or visually impaired to interpret the surrounding environment by voice or touch. | |
| CN106843491A (en) | Smart machine and electronic equipment with augmented reality | |
| WO2003107039A2 (en) | Method and apparatus for a multisensor imaging and scene interpretation system to aid the visually impaired | |
| CN105686936B (en) | A kind of acoustic coding interactive system based on RGB-IR cameras | |
| Schwarze et al. | A camera-based mobility aid for visually impaired people | |
| CN105761235A (en) | Vision auxiliary method converting vision information to auditory information | |
| CN107689060A (en) | Visual processing method, device and the equipment of view-based access control model processing of destination object | |
| Mattoccia et al. | 3D glasses as mobility aid for visually impaired people | |
| CN105701811B (en) | A kind of acoustic coding exchange method based on RGB-IR cameras | |
| KR20210025835A (en) | Wearable device using stereo camera and infrared sensor for the visually impaired | |
| KR20140074201A (en) | Tracking device | |
| Wang et al. | An environmental perception and navigational assistance system for visually impaired persons based on semantic stixels and sound interaction | |
| Söveny et al. | Blind guide-A virtual eye for guiding indoor and outdoor movement | |
| Qureshi et al. | AI based app for blind people | |
| Pei et al. | Census-based vision for auditory depth images and speech navigation of visually impaired users | |
| Vorapatratorn et al. | Fast obstacle detection system for the blind using depth image and machine learning. | |
| CN110675413A (en) | Three-dimensional face model construction method and device, computer equipment and storage medium | |
| EP3438939A1 (en) | Information processing device, information processing method, and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20191029 Termination date: 20220116 |