CN107341789A - One kind is based on RGB D cameras and stereosonic visually impaired people's path precognition system and method - Google Patents
One kind is based on RGB D cameras and stereosonic visually impaired people's path precognition system and method Download PDFInfo
- Publication number
- CN107341789A CN107341789A CN201611048370.8A CN201611048370A CN107341789A CN 107341789 A CN107341789 A CN 107341789A CN 201611048370 A CN201611048370 A CN 201611048370A CN 107341789 A CN107341789 A CN 107341789A
- Authority
- CN
- China
- Prior art keywords
- mtd
- mrow
- infrared
- color
- mtr
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61H—PHYSICAL THERAPY APPARATUS, e.g. DEVICES FOR LOCATING OR STIMULATING REFLEX POINTS IN THE BODY; ARTIFICIAL RESPIRATION; MASSAGE; BATHING DEVICES FOR SPECIAL THERAPEUTIC OR HYGIENIC PURPOSES OR SPECIFIC PARTS OF THE BODY
- A61H3/00—Appliances for aiding patients or disabled persons to walk about
- A61H3/06—Walking aids for blind persons
- A61H3/061—Walking aids for blind persons with electronic detecting or guiding means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10048—Infrared image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30244—Camera pose
Landscapes
- Health & Medical Sciences (AREA)
- Epidemiology (AREA)
- Pain & Pain Management (AREA)
- Physical Education & Sports Medicine (AREA)
- Rehabilitation Therapy (AREA)
- Life Sciences & Earth Sciences (AREA)
- Animal Behavior & Ethology (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- Measurement Of Optical Distance (AREA)
Abstract
The invention discloses one kind based on RGB D cameras and stereosonic visually impaired people's path precognition system and method.This method projects sightless near-infrared static speckles using infrared projector, and using two infrared cameras and a RGB color camera collection image, compact processor handles the image of collection, calculates and obtain depth image.This method obtains the attitude angle information of camera using attitude angle transducer.Compact processor utilizes depth information and attitude angle information, calculate and obtain height image, and piecemeal is carried out to height image, the depth information that piecemeal is obtained, it is converted into stereophonic signal, finally visually impaired people is passed to using bone conduction earphone to be aided in, can meet the requirement of visually impaired people's path precognition well.
Description
Technical field
The invention belongs to visually impaired people's ancillary technique, binocular vision technology, three-dimensional environment cognition technology, stereo interactive skill
Art field.The present invention is based on RGB-D cameras and stereosonic visually impaired people's path prediction method comprising one kind, is related to a kind of utilization
Sightless near-infrared static speckles are projected using infrared projector, utilize two infrared cameras and a RGB camera collection figure
Picture, compact processor are handled the image of collection, are calculated and are obtained depth image.Compact processor utilizes attitude angle transducer
Obtain the attitude angle information of camera.Compact processor utilizes depth information and attitude angle information, calculates and obtains height image, and right
Height image carries out piecemeal, the depth information that will obtain piecemeal, is converted into stereophonic signal, is finally passed using bone conduction earphone
Pass the path prediction method that visually impaired people is aided in.
Background technology
Counted according to world health organization, the whole world there are 2.85 hundred million dysopia personages.Visually impaired people have lost normally
Vision, the understanding to color, shape, distance, motion is all highly difficult, and their live in daily life, trip etc. all receives
Tremendous influence.
Traditional visually impaired people's aid such as bland person walking stick, visually impaired people need to move repeatedly walking stick, can just known
Situation in front, it is both time-consuming or laborious.The detection range of bland person walking stick is limited, can only detect the barrier on pin side, also can not be anti-
Reflect distant place and aerial situation.Seeing-eye dog can provide help for visually impaired people, but the training of seeing-eye dog and maintenance costs are
High, average family is difficult to undertake.Some occasions, seeing-eye dog can not accompany blind person's entrance, such as bus and railway station, because
The auxiliary of this seeing-eye dog is limited.Bionic eye can help visually impaired people's recovered part vision, but the implantation of bionic eye
Need to perform the operation, high cost.Bionic eye, which is only applicable to retinal pigment degeneration or senile macular degeneration, causes the blind person of blindness.
The visually impaired people of optic nerve damage can not be by being implanted into bionic eye come recovered part vision.
The visually impaired aid of electronic type is mainly with ultrasonic technology, laser ranging technique, binocular vision technology, laser
Speckle encoding technology, laser radar technique, millimetre-wave radar technology, thermal imaging, global positioning system (GPS).Based on super
The finding range of technology of acoustic wave and laser ranging technique is limited, can only realize single-point ranging, the information content of acquisition very little, and power consumption
More, equipment is heavy, warning function can only be realized, easily by environmental disturbances.Auxiliary based on binocular vision technology depends on environment
The abundant degree of middle characteristic point and texture, failed for the single scene of some textures, such as indoor white wall, smooth earth etc..
Binocular vision technology can be cheated by special cases such as mirror-reflections, so as to cause to fail to judge or judge by accident.Compiled based on laser speckle
The auxiliary of code technology is in outdoor failure, because actively the structure light of projection is flooded by sunlight, so as to the speckle of None- identified coding.
Due to by power limit, maximum distance be present, the object more than maximum distance can not be to its ranging in laser speckle coding techniques.
Supplementary costs based on laser radar technique is high, and usual sample rate is low, sensitive to dust, haze, rainwater, and can not obtain face
Color and texture information.Auxiliary resolution ratio based on millimetre-wave radar is low, and signal processing is difficult.Auxiliary based on thermal imaging
Resolution ratio is low, and calibration process is complicated, and can only detect the thermal objects such as humans and animals.Auxiliary precision based on GPS is low, has letter
Number loss, it is impossible to use indoors, and the obstacle information of local dynamic station can not be obtained.
The interactive mode of traditional visually impaired people's auxiliary mainly has voice message, tactile vibrations.Barrier is generally reported in semanteme prompting
Hinder thing distance and direction, it is necessary to the regular hour plays, cause delay and accident risk, and transferable information content is few.
Tactile vibrations are hardware by shaking waistband or vibrations vest, and to shake to prompt the orientation of barrier, shaking device can be with
Solve the problems, such as delay, but burden is brought to visually impaired people, the wearing feeling of different people is by difference.
The content of the invention
A kind of based on RGB-D cameras and stereosonic regarded in view of the above-mentioned deficiencies in the prior art, it is an object of the present invention to provide
Hinder personage's path precognition system and method.
The purpose of the present invention is achieved through the following technical solutions：One kind is based on RGB-D cameras and stereosonic visually impaired
Personage's path predicts system, and the system includes an infrared projector, two identical infrared cameras, a color camera,
One attitude angle transducer, a usb hub, a compact processor, a bone conduction earphone module, two osteoacusis shakes
Dynamic model block, a battery module.Infrared projector, two infrared cameras, color camera, attitude angle transducer pass through USB set line
Device is connected with compact processor, and battery module is connected with compact processor.Color camera and infrared projector are positioned at two infrared
Between camera.Two infrared cameras, the optical axis of color camera are parallel to each other.The attitude angle of three cameras is consistent, and passes through posture
Angle transducer obtains in real time.Three-dimensional scenic projection is sightless static near red forwards for compact processor control infrared projector
Outer speckle, two infrared cameras gather two width infrared images of the three-dimensional scenic after projection in real time.Color camera is in real time
Gather the coloured image of three-dimensional scenic.Two width infrared images, a width coloured image, attitude angle information are transmitted to small by usb hub
Type processor.Compact processor is handled two width infrared images, the width coloured image of acquisition, obtains the depth of three-dimensional scenic
Spend image.Compact processor is handled depth information and attitude angle information, obtains the height image of three-dimensional scenic.Small-sized place
Manage device and piecemeal is carried out to height image, and the depth information after piecemeal is converted into stereophonic signal, and be transmitted to bone conduction earphone
Module.Stereophonic signal is switched to osteoacusis vibration signal by bone conduction earphone module, is transmitted to two osteoacusis shock modules.Two
Osteoacusis shock module, transmit osteoacusis vibration signal and give visually impaired user.
The path prediction method of said system is as follows:
(1) two infrared cameras are carried out with a binocular camera demarcation, obtains the focal length f of two infrared camerasIR, it is left infrared
Principle point location (the c of cameraIR-x,cIR-y), the parallax range B of two infrared camerasIR-IR。
(2) camera calibration is carried out to color camera, obtains the focal length f of color cameracolor, principle point location (cCOLOR-x,
cCOLOR-y)。
(3) a binocular camera demarcation is carried out to color camera and the infrared camera in left side, obtains left infrared camera and coloured silk
The parallax range B of form and aspect machineIR-COLOR。
(4) infrared projector projects sightless static near-infrared speckle into three-dimensional scenic in real time.
Two infrared image IR of (5) two infrared camera collection three-dimensional scenicsleftAnd IRright。
(6) the coloured image Color of color camera three-dimensional scenic.
(7) attitude angle transducer gathers the X, the corner Angle of the direction of principal axis of Y, Z tri- of three camerasX, AngleY, AngleZ。
(8) usb hub is by two infrared image IRleftAnd IRright, the direction of principal axis of coloured image Color, X, Y, Z tri-
Corner AngleX, AngleY, AngleZIt is transmitted to compact processor.
(9) compact processor is to two infrared image IRleftAnd IRrightSobel edges are extracted, obtain two Sobel sides
Edge image SobelleftAnd Sobelright。
(10) with left Sobel edge images SobelleftOn the basis of, to two Sobel edge images SobelleftWith
SobelrightThe images match based on image block is carried out, obtains a series of available point E={ e matched1,e2,e3,...,eM}。
In left Sobel edge images SobelleftIn, each available point is e=(u, v, d)T, u is abscissa pixel value, and v sits to be vertical
Pixel value is marked, d is parallax value.
(11) on the basis of the available point E matched, every three available points may make up a disparity plane, i-th of parallax
The equation of plane is d=aiu+biv+ci, wherein ai, bi, ciFor the coefficient of i-th of disparity plane.
(12) on the basis of these disparity planes, by the pixel not matched (u', v', d')TBe converted to the effective of matching
Point (u, v, d)T；Specially：The pixel (u', v', d')TDistance to the i-th disparity plane is
If energy function isWherein ε, σ are constant.To the pixel, disparity search is traveled through
All parallax value d'={ d' of scopemin,...,d'max, the parallax value for make it that energy function Energy (d') is minimum is obtained, when
Into the parallax value d of the pixel.In addition, u=u', v=v'.
(13) all pixels not matched are traveled through, obtain the parallax value of each pixel not matched, are obtained with left red
Anaglyph Disparity on the basis of outer cameraleft。
(14) according to the focal length f of two infrared camerasIRWith parallax range BIR-IR, travel through anaglyph in every bit (u,
V, d), its depth value isTherefore depth image DepthleftMiddle every bit corresponds to (u, v, depth), from
And obtain the depth image Depth on the basis of left infrared cameraleft。
(15) depth image Depth is utilizedleftWith coloured image Color, the focal length f of two infrared camerasIR, left infrared phase
Principle point location (the c of machineIR-x,cIR-y), the focal length f of color cameracolor, principle point location (cCOLOR-x,cCOLOR-y), and it is left infrared
The parallax range B of camera and color cameraIR-COLOR, depth image and coloured image can be alignd, obtain color camera and regard
The depth image Depth of fieldcolor。
(16) according to depth image Depthcolor, the focal length f of color cameracolorWith the principle point location of color camera
(cCOLOR-x,cCOLOR-y), three-dimensional coordinate (X, Y, Z) of the every bit under color camera coordinate system can be calculated.Depth image
DepthcolorIn a point coordinates be (u, v), its depth value is depth, then three-dimensional coordinate (X, Y, Z) can be calculated by formula (1)：
Z=depth
(17) three-dimensional coordinate (X, Y, Z) according to every bit in depth image under camera coordinates system, and attitude angle pass
The corner of the direction of principal axis of sensor three is respectively AngleX=α, AngleY=β, AngleZ=γ, then it can be calculated by formula (2) each
Coordinate (X of the point under world coordinate systemw,Yw,Zw)：
(18) the coordinate Y according to every bit under world coordinate systemw, i.e. every bit to the vertical of color camera wearing position
Highly, height image Height can be obtained.
(19) height image Height is from left to right divided into K, calculates each tile height image HeightKAverage height
heightK.(K value is typically between 2 to 10)
(20) K tile height images Height is represented with the (instrumental) ensemble of the musical instrument of K kind difference tone colorsK:Each piece respectively with difference
The sounding of the musical instrument of tone color represents.The height of given visually impaired user is H, the average height of the height image of different masses
heightKAnd H difference and musical instrument loudness Volume are inversely proportional, i.e.,：Average height heightKIt is closer with H, illustrate the block image
Interior material object is nearer from the ground, and road conditions are more suitable current, and loudness Volume is bigger；Average height heightKWith H further away from saying
Material object in the bright block image is more remote from the ground, and road conditions are more unsuitable current, and loudness Volume is smaller.The musical instrument sound in each direction
All to be stereo.The optional piano of musical instrument, violin, gong, trumpet, xylophone etc. have special tone color and melodious.
(21) stereophonic signal is transmitted to bone conduction earphone module by compact processor.
(22) stereophonic signal is switched to osteoacusis vibration signal by bone conduction earphone module.
(23) osteoacusis signal is passed to visually impaired user by osteoacusis shock module.
This method essentially consists in compared to the advantage of conventional visually impaired people's householder method：
1st, ambient adaptability.Due to use infrared projector and two infrared cameras, this method indoors, outdoor environment all
Compatible it can use.When indoors, the static near-infrared hot spot of infrared projector projection adds texture to three-dimensional scenic, favorably
In the depth image that acquisition is dense.At outdoor, the near-infrared part of sunlight and three-dimensional scenic combine, and are advantageous to obtain dense
Depth image.Dense depth image can ensure the accuracy of piecemeal height and the experience effect of auxiliary interaction.
2nd, applicability day and night.Due to use infrared projector and two infrared cameras, this method daytime, night all
Compatible it can use.During the day, the near-infrared composition in static the near-infrared hot spot and daylight of infrared projector projection all may be used
To increase texture to three-dimensional scenic, be advantageous to dense depth image.At night, the static near-infrared of infrared projector projection
Hot spot gives nearby three-dimensional scenic increase texture, can also obtain the depth image of nearby three-dimensional scenic.This method is in day and night
Reliable depth image can be obtained, so as to ensure the experience effect of the accuracy of piecemeal height and auxiliary interaction.
3rd, the road conditions such as stair, slope can be differentiated, due to using stereo interaction, stereophonic signal illustrates front all directions
Height value, therefore represent that stair, the sound of the road conditions such as slope and expression can pass through the sound of smooth-riding surface, sound and differ
Sample, voice signal not only anticipated that can traffic areas, the road conditions such as stair, slope can also be predicted.
4th, situations such as can differentiating road hole, due to using stereo interaction, stereophonic signal illustrates the height of front all directions
The sound for situations such as angle value, the height and normal road condition cheated due to road are different, therefore expression road is cheated and expression can pass through smooth
The sound on road surface, sound different, voice signal not only anticipated that can traffic areas, can also predict road hole etc. feelings
Condition.
5th, ears are not take up.This method delivers a signal to visually impaired user using bone conduction earphone, is listened to without prejudice to user outer
The sound on boundary.Visually impaired people is dependent on extraneous sound to carry out some interpretations, as judged road according to wagon flow sound mostly
Towards etc..
6th, both hands are not take up.The servicing unit of this method is wearable, and compact processor is portable, can be placed on pocket or small
Bao Li, very big burden will not be both brought to visually impaired, aid is taken with hand without visually impaired people.
7th, user will not be bothered.The stereo interactive mode of this method, using melodious instruments sound, it will not be used to visually impaired
Family is caused to bother, and allowing visually impaired user to listen melodious music when in use, just energy avoidance is passed through.
8th, sufficient information content is fed back.Compared to the voice broadcast of semantic formula, stereosonic interaction feedback utilizes different loudness,
The musical instrument of different tone colors represents the trafficability of road conditions, can simultaneously, fully transmit the road conditions of front different directions, precognition
Can traffic areas orientation.
9th, learnability and Comprehensible.Compared to the acoustic coding of complex form, the interaction of neutral body sound of the present invention is based on height
The piecemeal of degree, the elevation information after piecemeal processing will not be very lengthy and jumbled, and visually impaired user can rapidly learn and understand stereo letter
Number implication, and according to stereophonic signal select direction of travel.
10th, timely feed back.Compared to the voice broadcast of semantic formula, stereosonic interaction feedback is timely, not postpone.
So as to visually impaired people can timely select correctly can pass, ensure that the security of this method.
Brief description of the drawings
Fig. 1 is the module connection diagram that visually impaired people's path predicts system；
Fig. 2 is the structural representation that visually impaired people's path predicts glasses；
Fig. 3 is two infrared image IRleftAnd IRright；
Fig. 4 is the depth image Depth after gray processing processingleft(in former depth image, represented with pseudo-colours, it is more red more
Closely, more Lan Yueyuan).
Fig. 5 is the coloured image after gray processing processing, (be passage region with Green Marker in original color image, i.e. Horizon
Vertical height of the face at RGB-D wearings is close with the height of visually impaired user).
Fig. 6 is the stereo schematic diagram for representing path of musical instrument.
Embodiment
It is as shown in figure 1, a kind of based on RGB-D cameras and stereosonic visually impaired people's path precognition system, the system bag
Containing an infrared projector, two identical infrared cameras, a color camera, an attitude angle transducer, a USB set line
Device, a compact processor, a bone conduction earphone module, two osteoacusis shock modules, a battery module.Infrared projection
Device, two infrared cameras, color camera, attitude angle transducer are connected by usb hub with compact processor, battery module with
Compact processor is connected.Color camera and infrared projector are between two infrared cameras.Two infrared cameras, color cameras
Optical axis it is parallel to each other.The attitude angle of three cameras is consistent, and is obtained in real time by attitude angle transducer.Compact processor control
Three-dimensional scenic projects sightless static near-infrared speckle to infrared projector processed forwards, and two infrared cameras gather warp in real time
Two width infrared images of the three-dimensional scenic after projection.Color camera gathers the coloured image of three-dimensional scenic in real time.Usb hub
Two width infrared images, a width coloured image, attitude angle information are transmitted to compact processor.Compact processor is red to two of acquisition
Outer image, a width coloured image are handled, and obtain the depth image of three-dimensional scenic.Compact processor is to depth information and posture
Angle information is handled, and obtains the height image of three-dimensional scenic.Compact processor carries out piecemeal to height image, and by after piecemeal
Depth information be converted into stereophonic signal, and be transmitted to bone conduction earphone module.Bone conduction earphone module turns stereophonic signal
For osteoacusis vibration signal, two osteoacusis shock modules are transmitted to.Two osteoacusis shock modules, transmit osteoacusis vibration signal
Give visually impaired user.The system can be designed analogous to the glasses described in Fig. 2, to reach aesthetic.
The path prediction method of the system comprises the following steps:
(1) two infrared cameras are carried out with a binocular camera demarcation, obtains the focal length f of two infrared camerasIR, it is left infrared
Principle point location (the c of cameraIR-x,cIR-y), the parallax range B of two infrared camerasIR-IR。
(2) camera calibration is carried out to color camera, obtains the focal length f of color cameracolor, principle point location (cCOLOR-x,
cCOLOR-y)。
(3) a binocular camera demarcation is carried out to color camera and the infrared camera in left side, obtains left infrared camera and coloured silk
The parallax range B of form and aspect machineIR-COLOR。
(4) infrared projector projects sightless static near-infrared speckle into three-dimensional scenic in real time.
Two infrared image IR of (5) two infrared camera collection three-dimensional scenicsleftAnd IRright。
(6) the coloured image Color of color camera three-dimensional scenic.
(7) attitude angle transducer gathers the X, the corner Angle of the direction of principal axis of Y, Z tri- of three camerasX, AngleY, AngleZ。
(8) usb hub is by two infrared image IRleftAnd IRright, the direction of principal axis of coloured image Color, X, Y, Z tri-
Corner AngleX, AngleY, AngleZIt is transmitted to compact processor.
(9) compact processor is to two infrared image IRleftAnd IRrightSobel edges are extracted, obtain two Sobel sides
Edge image SobelleftAnd Sobelright。
(10) with left Sobel edge images SobelleftOn the basis of, to two Sobel edge images SobelleftWith
SobelrightThe images match based on image block is carried out, obtains a series of available point E={ e matched1,e2,e3,...,eM}。
In left Sobel edge images SobelleftIn, each available point is e=(u, v, d)T, u is abscissa pixel value, and v sits to be vertical
Pixel value is marked, d is parallax value.
(11) on the basis of the available point E matched, every three available points may make up a disparity plane, i-th of parallax
The equation of plane is d=aiu+biv+ci, wherein ai, bi, ciFor the coefficient of i-th of disparity plane.
(12) on the basis of these disparity planes, by the pixel not matched (u', v', d')TBe converted to the effective of matching
Point (u, v, d)T；Specially：The pixel (u', v', d')TDistance to the i-th disparity plane is
If energy function isWherein ε, σ are constant.To the pixel, disparity search is traveled through
All parallax value d'={ d' of scopemin,...,d'max, the parallax value for make it that energy function Energy (d') is minimum is obtained, when
Into the parallax value d of the pixel.In addition, u=u', v=v'.
(13) all pixels not matched are traveled through, obtain the parallax value of each pixel not matched, are obtained with left red
Anaglyph Disparity on the basis of outer cameraleft。
(14) according to the focal length f of two infrared camerasIRWith parallax range BIR-IR, travel through anaglyph in every bit (u,
V, d), its depth value isTherefore depth image DepthleftMiddle every bit corresponds to (u, v, depth), from
And obtain the depth image Depth on the basis of left infrared cameraleft。
(15) depth image Depth is utilizedleftWith coloured image Color, the focal length f of two infrared camerasIR, left infrared phase
Principle point location (the c of machineIR-x,cIR-y), the focal length f of color cameracolor, principle point location (cCOLOR-x,cCOLOR-y), and it is left infrared
The parallax range B of camera and color cameraIR-COLOR, depth image and coloured image can be alignd, obtain color camera and regard
The depth image Depth of fieldcolor。
(16) according to depth image Depthcolor, the focal length f of color cameracolorWith the principle point location of color camera
(cCOLOR-x,cCOLOR-y), three-dimensional coordinate (X, Y, Z) of the every bit under color camera coordinate system can be calculated.Depth image
DepthcolorIn a point coordinates be (u, v), its depth value is depth, then three-dimensional coordinate (X, Y, Z) can be calculated by formula (1)：
Z=depth
(17) three-dimensional coordinate (X, Y, Z) according to every bit in depth image under camera coordinates system, and attitude angle pass
The corner of the direction of principal axis of sensor three is respectively AngleX=α, AngleY=β, AngleZ=γ, then it can be calculated by formula (2) each
Coordinate (X of the point under world coordinate systemw,Yw,Zw)：
(18) the coordinate Y according to every bit under world coordinate systemw, i.e. every bit to the vertical of color camera wearing position
Highly, height image Height can be obtained.
(19) height image Height is from left to right divided into K, calculates each tile height image HeightKAverage height
heightK.(K value is typically between 2 to 10)
(20) K tile height images Height is represented with the (instrumental) ensemble of the musical instrument of K kind difference tone colorsK:Each piece respectively with difference
The sounding of the musical instrument of tone color represents.The height of given visually impaired user is H, the average height of the height image of different masses
heightKAnd H difference and musical instrument loudness Volume are inversely proportional, i.e.,：Average height heightKIt is closer with H, illustrate the block image
Interior material object is nearer from the ground, and road conditions are more suitable current, and loudness Volume is bigger；Average height heightKWith H further away from saying
Material object in the bright block image is more remote from the ground, and road conditions are more unsuitable current, and loudness Volume is smaller.The musical instrument sound in each direction
All to be stereo.The optional piano of musical instrument, violin, gong, trumpet, xylophone etc. have special tone color and melodious.
(21) stereophonic signal is transmitted to bone conduction earphone module by compact processor.
(22) stereophonic signal is switched to osteoacusis vibration signal by bone conduction earphone module.
(23) osteoacusis signal is passed to visually impaired user by osteoacusis shock module.
Claims (2)
1. one kind includes an infrared throwing based on RGB-D cameras and stereosonic visually impaired people's path precognition system, the system
Emitter, two identical infrared cameras, a color camera, an attitude angle transducer, a usb hub, one small-sized
Processor, a bone conduction earphone module, two osteoacusis shock modules, a battery module.It is infrared projector, two infrared
Camera, color camera, attitude angle transducer are connected by usb hub with compact processor, battery module and compact processor
It is connected.Color camera and infrared projector are between two infrared cameras.Two infrared cameras, color camera optical axis each other
It is parallel.The attitude angle of three cameras is consistent, and is obtained in real time by attitude angle transducer.Compact processor controls infrared projection
Three-dimensional scenic projects sightless static near-infrared speckle to device forwards, and two infrared cameras gather three after projection in real time
Tie up two width infrared images of scene.Color camera gathers the coloured image of three-dimensional scenic in real time.Usb hub is infrared by two
Image, a width coloured image, attitude angle information are transmitted to compact processor.Compact processor is to two width infrared images of acquisition, one
Width coloured image is handled, and obtains the depth image of three-dimensional scenic.Compact processor is entered to depth information and attitude angle information
Row processing, obtain the height image of three-dimensional scenic.Compact processor carries out piecemeal to height image, and the depth after piecemeal is believed
Breath is converted into stereophonic signal, and is transmitted to bone conduction earphone module.Stereophonic signal is switched to osteoacusis by bone conduction earphone module
Vibration signal, it is transmitted to two osteoacusis shock modules.Two osteoacusis shock modules, transmit osteoacusis vibration signal and used to visually impaired
Family.
2. the path prediction method of system described in a kind of claim 1, it is characterised in that comprise the following steps:
(1) two infrared cameras are carried out with a binocular camera demarcation, obtains the focal length f of two infrared camerasIR, left infrared camera
Principle point location (cIR-x,cIR-y), the parallax range B of two infrared camerasIR-IR。
(2) camera calibration is carried out to color camera, obtains the focal length f of color cameracolor, principle point location (cCOLOR-x,
cCOLOR-y)。
(3) a binocular camera demarcation is carried out to color camera and the infrared camera in left side, obtains left infrared camera and colored phase
The parallax range B of machineIR-COLOR。
(4) infrared projector projects sightless static near-infrared speckle into three-dimensional scenic in real time.
Two infrared image IR of (5) two infrared camera collection three-dimensional scenicsleftAnd IRright。
(6) the coloured image Color of color camera three-dimensional scenic.
(7) attitude angle transducer gathers the X, the corner Angle of the direction of principal axis of Y, Z tri- of three camerasX, AngleY, AngleZ。
(8) usb hub is by two infrared image IRleftAnd IRright, the corner of the direction of principal axis of coloured image Color, X, Y, Z tri-
AngleX, AngleY, AngleZIt is transmitted to compact processor.
(9) compact processor is to two infrared image IRleftAnd IRrightSobel edges are extracted, obtain two Sobel edge graphs
As SobelleftAnd Sobelright。
(10) with left Sobel edge images SobelleftOn the basis of, to two Sobel edge images SobelleftAnd Sobelright
The images match based on image block is carried out, obtains a series of available point E={ e matched1,e2,e3,...,eM}.On a left side
Sobel edge images SobelleftIn, each available point is e=(u, v, d)T, u is abscissa pixel value, and v is ordinate picture
Element value, d is parallax value.
(11) on the basis of the available point E matched, every three available points may make up a disparity plane, i-th of disparity plane
Equation be d=aiu+biv+ci, wherein ai, bi, ciFor the coefficient of i-th of disparity plane.
(12) on the basis of these disparity planes, by the pixel not matched (u', v', d')TBe converted to the available point of matching
(u,v,d)T；Specially：The pixel (u', v', d')TDistance to the i-th disparity plane is
If energy function isWherein ε, σ are constant.To the pixel, disparity search is traveled through
All parallax value d'={ d' of scopemin,...,d'max, the parallax value for make it that energy function Energy (d') is minimum is obtained, when
Into the parallax value d of the pixel.In addition, u=u', v=v'.
(13) all pixels not matched are traveled through, obtain the parallax value of each pixel not matched, are obtained with left infrared phase
Anaglyph Disparity on the basis of machineleft。
(14) according to the focal length f of two infrared camerasIRWith parallax range BIR-IR, the every bit (u, v, d) in anaglyph is traveled through,
Its depth value isTherefore depth image DepthleftMiddle every bit corresponds to (u, v, depth), so as to obtain
Obtain the depth image Depth on the basis of left infrared cameraleft。
(15) depth image Depth is utilizedleftWith coloured image Color, the focal length f of two infrared camerasIR, left infrared camera
Principle point location (cIR-x,cIR-y), the focal length f of color cameracolor, principle point location (cCOLOR-x,cCOLOR-y), and left infrared camera
With the parallax range B of color cameraIR-COLOR, depth image and coloured image can be alignd, obtain color camera visual field
Depth image Depthcolor。
(16) according to depth image Depthcolor, the focal length f of color cameracolorWith the principle point location (c of color cameraCOLOR-x,
cCOLOR-y), three-dimensional coordinate (X, Y, Z) of the every bit under color camera coordinate system can be calculated.Depth image DepthcolorIn
One point coordinates is (u, v), and its depth value is depth, then three-dimensional coordinate (X, Y, Z) can be calculated by formula (1)：
Z=depth
<mrow>
<mi>X</mi>
<mo>=</mo>
<mi>Z</mi>
<mo>&times;</mo>
<mfrac>
<mrow>
<mi>u</mi>
<mo>-</mo>
<msub>
<mi>c</mi>
<mrow>
<mi>C</mi>
<mi>O</mi>
<mi>L</mi>
<mi>O</mi>
<mi>R</mi>
<mo>-</mo>
<mi>x</mi>
</mrow>
</msub>
</mrow>
<msub>
<mi>f</mi>
<mrow>
<mi>C</mi>
<mi>O</mi>
<mi>L</mi>
<mi>O</mi>
<mi>R</mi>
</mrow>
</msub>
</mfrac>
<mo>-</mo>
<mo>-</mo>
<mo>-</mo>
<mrow>
<mo>(</mo>
<mn>1</mn>
<mo>)</mo>
</mrow>
</mrow>
<mrow>
<mi>Y</mi>
<mo>=</mo>
<mi>Z</mi>
<mo>&times;</mo>
<mfrac>
<mrow>
<mi>v</mi>
<mo>-</mo>
<msub>
<mi>c</mi>
<mrow>
<mi>C</mi>
<mi>O</mi>
<mi>L</mi>
<mi>O</mi>
<mi>R</mi>
<mo>-</mo>
<mi>y</mi>
</mrow>
</msub>
</mrow>
<msub>
<mi>f</mi>
<mrow>
<mi>C</mi>
<mi>O</mi>
<mi>L</mi>
<mi>O</mi>
<mi>R</mi>
</mrow>
</msub>
</mfrac>
</mrow>
(17) three-dimensional coordinate (X, Y, Z) according to every bit in depth image under camera coordinates system, and attitude angle transducer
The corner of three direction of principal axis is respectively AngleX=α, AngleY=β, AngleZ=γ, then every bit can be calculated by formula (2) and existed
Coordinate (X under world coordinate systemw,Yw,Zw)：
<mrow>
<mfenced open = "[" close = "]">
<mtable>
<mtr>
<mtd>
<msub>
<mi>X</mi>
<mi>w</mi>
</msub>
</mtd>
</mtr>
<mtr>
<mtd>
<msub>
<mi>Y</mi>
<mi>w</mi>
</msub>
</mtd>
</mtr>
<mtr>
<mtd>
<msub>
<mi>Z</mi>
<mi>z</mi>
</msub>
</mtd>
</mtr>
</mtable>
</mfenced>
<mo>=</mo>
<mfenced open = "[" close = "]">
<mtable>
<mtr>
<mtd>
<mrow>
<mi>cos</mi>
<mi>&gamma;</mi>
</mrow>
</mtd>
<mtd>
<mrow>
<mo>-</mo>
<mi>sin</mi>
<mi>&gamma;</mi>
</mrow>
</mtd>
<mtd>
<mn>0</mn>
</mtd>
</mtr>
<mtr>
<mtd>
<mrow>
<mi>sin</mi>
<mi>&gamma;</mi>
</mrow>
</mtd>
<mtd>
<mrow>
<mi>cos</mi>
<mi>&gamma;</mi>
</mrow>
</mtd>
<mtd>
<mn>0</mn>
</mtd>
</mtr>
<mtr>
<mtd>
<mn>0</mn>
</mtd>
<mtd>
<mn>0</mn>
</mtd>
<mtd>
<mn>1</mn>
</mtd>
</mtr>
</mtable>
</mfenced>
<mfenced open = "[" close = "]">
<mtable>
<mtr>
<mtd>
<mrow>
<mi>cos</mi>
<mi>&beta;</mi>
</mrow>
</mtd>
<mtd>
<mn>0</mn>
</mtd>
<mtd>
<mrow>
<mi>sin</mi>
<mi>&beta;</mi>
</mrow>
</mtd>
</mtr>
<mtr>
<mtd>
<mn>0</mn>
</mtd>
<mtd>
<mn>1</mn>
</mtd>
<mtd>
<mn>0</mn>
</mtd>
</mtr>
<mtr>
<mtd>
<mrow>
<mo>-</mo>
<mi>sin</mi>
<mi>&beta;</mi>
</mrow>
</mtd>
<mtd>
<mn>0</mn>
</mtd>
<mtd>
<mrow>
<mi>cos</mi>
<mi>&beta;</mi>
</mrow>
</mtd>
</mtr>
</mtable>
</mfenced>
<mfenced open = "[" close = "]">
<mtable>
<mtr>
<mtd>
<mn>1</mn>
</mtd>
<mtd>
<mn>0</mn>
</mtd>
<mtd>
<mn>0</mn>
</mtd>
</mtr>
<mtr>
<mtd>
<mn>0</mn>
</mtd>
<mtd>
<mrow>
<mi>cos</mi>
<mi>&alpha;</mi>
</mrow>
</mtd>
<mtd>
<mrow>
<mo>-</mo>
<mi>sin</mi>
<mi>&alpha;</mi>
</mrow>
</mtd>
</mtr>
<mtr>
<mtd>
<mn>0</mn>
</mtd>
<mtd>
<mrow>
<mi>sin</mi>
<mi>&alpha;</mi>
</mrow>
</mtd>
<mtd>
<mrow>
<mi>cos</mi>
<mi>&alpha;</mi>
</mrow>
</mtd>
</mtr>
</mtable>
</mfenced>
<mfenced open = "[" close = "]">
<mtable>
<mtr>
<mtd>
<mi>X</mi>
</mtd>
</mtr>
<mtr>
<mtd>
<mi>Y</mi>
</mtd>
</mtr>
<mtr>
<mtd>
<mi>Z</mi>
</mtd>
</mtr>
</mtable>
</mfenced>
<mo>-</mo>
<mo>-</mo>
<mo>-</mo>
<mrow>
<mo>(</mo>
<mn>2</mn>
<mo>)</mo>
</mrow>
</mrow>
(18) the coordinate Y according to every bit under world coordinate systemw, i.e. the vertical height of every bit to color camera wearing position,
Height image Height can be obtained.
(19) height image Height is from left to right divided into K, calculates each tile height image HeightKAverage height
heightK.(K value is typically between 2 to 10)
(20) K tile height images Height is represented with the (instrumental) ensemble of the musical instrument of K kind difference tone colorsK:Each piece respectively with different tone colors
The sounding of musical instrument represents.The height of given visually impaired user is H, the average height height of the height image of different massesKWith H it
Difference and musical instrument loudness Volume are inversely proportional, i.e.,：Average height heightKIt is closer with H, illustrate in kind liftoff in the block image
Face is nearer, and road conditions are more suitable current, and loudness Volume is bigger；Average height heightKWith H further away from illustrating in the block image
Material object it is more remote from the ground, road conditions are more unsuitable current, and loudness Volume is smaller.The musical instrument sound in each direction is stereo.
(21) stereophonic signal is transmitted to bone conduction earphone module by compact processor.
(22) stereophonic signal is switched to osteoacusis vibration signal by bone conduction earphone module.
(23) osteoacusis signal is passed to visually impaired user by osteoacusis shock module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611048370.8A CN107341789B (en) | 2016-11-23 | 2016-11-23 | System and method for predicting pathway of visually impaired people based on RGB-D camera and stereo |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611048370.8A CN107341789B (en) | 2016-11-23 | 2016-11-23 | System and method for predicting pathway of visually impaired people based on RGB-D camera and stereo |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107341789A true CN107341789A (en) | 2017-11-10 |
| CN107341789B CN107341789B (en) | 2019-12-17 |
Family
ID=60222763
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611048370.8A Active CN107341789B (en) | 2016-11-23 | 2016-11-23 | System and method for predicting pathway of visually impaired people based on RGB-D camera and stereo |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107341789B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108235774A (en) * | 2018-01-10 | 2018-06-29 | 深圳前海达闼云端智能科技有限公司 | Information processing method, device, cloud processing equipment and computer program product |
| CN108245385A (en) * | 2018-01-16 | 2018-07-06 | 曹醒龙 | A kind of device for helping visually impaired people's trip |
| CN108876798A (en) * | 2018-06-12 | 2018-11-23 | 杭州视氪科技有限公司 | A kind of stair detection system and method |
| CN109084700A (en) * | 2018-06-29 | 2018-12-25 | 上海摩软通讯技术有限公司 | The three dimensional local information acquisition methods and system of article |
| CN110399807A (en) * | 2019-07-04 | 2019-11-01 | 深圳前海达闼云端智能科技有限公司 | Detect method, apparatus, readable storage medium storing program for executing and the electronic equipment of ground obstacle |
| CN111932866A (en) * | 2020-08-11 | 2020-11-13 | 中国科学技术大学先进技术研究院 | Wearable blind person outdoor traffic information sensing equipment |
| CN112700484A (en) * | 2020-12-31 | 2021-04-23 | 南京理工大学智能计算成像研究院有限公司 | Depth map colorization method based on monocular depth camera |
| CN114724053A (en) * | 2022-04-11 | 2022-07-08 | 合肥工业大学 | Outdoor visual impairment assisting method based on deep intelligent interaction |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104021388A (en) * | 2014-05-14 | 2014-09-03 | 西安理工大学 | Reversing obstacle automatic detection and early warning method based on binocular vision |
| CN204766392U (en) * | 2015-05-14 | 2015-11-18 | 广州龙天软件科技有限公司 | Lead blind information processing apparatus |
| CN105701811A (en) * | 2016-01-12 | 2016-06-22 | 浙江大学 | Sound coding interaction method based on RGB-IR camera |
-
2016
- 2016-11-23 CN CN201611048370.8A patent/CN107341789B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104021388A (en) * | 2014-05-14 | 2014-09-03 | 西安理工大学 | Reversing obstacle automatic detection and early warning method based on binocular vision |
| CN204766392U (en) * | 2015-05-14 | 2015-11-18 | 广州龙天软件科技有限公司 | Lead blind information processing apparatus |
| CN105701811A (en) * | 2016-01-12 | 2016-06-22 | 浙江大学 | Sound coding interaction method based on RGB-IR camera |
Non-Patent Citations (1)
| Title |
|---|
| RUIQI CHENG 等: "A Ground and Obstacle Detection Algorithm for the Visually Impaired", 《ICBISP 2015》 * |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108235774B (en) * | 2018-01-10 | 2020-07-14 | 深圳前海达闼云端智能科技有限公司 | Information processing method and device, cloud processing equipment and computer program product |
| US11379963B2 (en) | 2018-01-10 | 2022-07-05 | Cloudminds Robotics Co., Ltd. | Information processing method and device, cloud-based processing device, and computer program product |
| WO2019136641A1 (en) * | 2018-01-10 | 2019-07-18 | 深圳前海达闼云端智能科技有限公司 | Information processing method and apparatus, cloud processing device and computer program product |
| CN108235774A (en) * | 2018-01-10 | 2018-06-29 | 深圳前海达闼云端智能科技有限公司 | Information processing method, device, cloud processing equipment and computer program product |
| CN108245385A (en) * | 2018-01-16 | 2018-07-06 | 曹醒龙 | A kind of device for helping visually impaired people's trip |
| CN108876798A (en) * | 2018-06-12 | 2018-11-23 | 杭州视氪科技有限公司 | A kind of stair detection system and method |
| CN108876798B (en) * | 2018-06-12 | 2022-03-18 | 杭州视氪科技有限公司 | Stair detection system and method |
| CN109084700B (en) * | 2018-06-29 | 2020-06-05 | 上海摩软通讯技术有限公司 | Method and system for acquiring three-dimensional position information of article |
| CN109084700A (en) * | 2018-06-29 | 2018-12-25 | 上海摩软通讯技术有限公司 | The three dimensional local information acquisition methods and system of article |
| CN110399807B (en) * | 2019-07-04 | 2021-07-16 | 达闼机器人有限公司 | Method and device for detecting ground obstacle, readable storage medium and electronic equipment |
| CN110399807A (en) * | 2019-07-04 | 2019-11-01 | 深圳前海达闼云端智能科技有限公司 | Detect method, apparatus, readable storage medium storing program for executing and the electronic equipment of ground obstacle |
| CN111932866A (en) * | 2020-08-11 | 2020-11-13 | 中国科学技术大学先进技术研究院 | Wearable blind person outdoor traffic information sensing equipment |
| CN112700484A (en) * | 2020-12-31 | 2021-04-23 | 南京理工大学智能计算成像研究院有限公司 | Depth map colorization method based on monocular depth camera |
| CN114724053A (en) * | 2022-04-11 | 2022-07-08 | 合肥工业大学 | Outdoor visual impairment assisting method based on deep intelligent interaction |
| CN114724053B (en) * | 2022-04-11 | 2024-02-20 | 合肥工业大学 | Outdoor visual impairment assisting method based on deep intelligent interaction |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107341789B (en) | 2019-12-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107341789A (en) | One kind is based on RGB D cameras and stereosonic visually impaired people's path precognition system and method | |
| CN106597690B (en) | One kind predicting glasses based on RGB-D camera and stereosonic visually impaired people's access | |
| CN106203390B (en) | A kind of intelligent blind auxiliary system | |
| CN106846350B (en) | One kind is based on RGB-D camera and stereosonic visually impaired people's barrier early warning system and method | |
| CN106817577B (en) | One kind is based on RGB-D cameras and stereosonic visually impaired people's barrier early warning glasses | |
| CN106214437A (en) | A kind of intelligent blind auxiliary eyeglasses | |
| EP3321888B1 (en) | Projected image generation method and device, and method for mapping image pixels and depth values | |
| US9370459B2 (en) | System and method for alerting visually impaired users of nearby objects | |
| US7598976B2 (en) | Method and apparatus for a multisensor imaging and scene interpretation system to aid the visually impaired | |
| CN104036488B (en) | Binocular vision-based human body posture and action research method | |
| JP2018060512A (en) | Image generating system, program and method, and simulation system, program and method | |
| CN106821692B (en) | One kind is based on RGB-D camera and stereosonic visually impaired people's stair detection system and method | |
| CN106651873B (en) | One kind detecting glasses based on RGB-D camera and stereosonic visually impaired people's zebra stripes | |
| CN105686936A (en) | Sound coding interaction system based on RGB-IR camera | |
| CN106843491A (en) | Smart machine and electronic equipment with augmented reality | |
| WO2018066352A1 (en) | Image generation system, program and method, and simulation system, program and method | |
| CN109106563A (en) | A kind of automation blind-guide device based on deep learning algorithm | |
| CN105701811A (en) | Sound coding interaction method based on RGB-IR camera | |
| CN106920260A (en) | Three-dimensional inertia blind-guiding method and device and system | |
| CN107610152A (en) | A kind of path detection method for being used to hide the water surface and barrier | |
| CN107049717B (en) | One kind is based on RGB-D camera and stereosonic visually impaired people's zebra stripes detection system and method | |
| CN108876798A (en) | A kind of stair detection system and method | |
| CN117323185A (en) | Blind person indoor navigation system and method based on computer vision and training method | |
| Ramer et al. | Sensor-guided jogging for visually impaired | |
| CN106821693B (en) | One kind detecting glasses based on RGB-D camera and stereosonic visually impaired people's stair |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |