CN107122770A - Many mesh camera systems, intelligent driving system, automobile, method and storage medium - Google Patents

Many mesh camera systems, intelligent driving system, automobile, method and storage medium Download PDF

Info

Publication number
CN107122770A
CN107122770A CN201710443135.9A CN201710443135A CN107122770A CN 107122770 A CN107122770 A CN 107122770A CN 201710443135 A CN201710443135 A CN 201710443135A CN 107122770 A CN107122770 A CN 107122770A
Authority
CN
China
Prior art keywords
camera
wide angle
image
stereoscopic
stereoscopic camera
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201710443135.9A
Other languages
Chinese (zh)
Other versions
CN107122770B (en
Inventor
张峰
张鹏
任杰
游宇
李斌
赵勇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Uisee Shanghai Automotive Technologies Ltd
Original Assignee
Uisee Shanghai Automotive Technologies Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Uisee Shanghai Automotive Technologies Ltd filed Critical Uisee Shanghai Automotive Technologies Ltd
Priority to CN201710443135.9A priority Critical patent/CN107122770B/en
Publication of CN107122770A publication Critical patent/CN107122770A/en
Application granted granted Critical
Publication of CN107122770B publication Critical patent/CN107122770B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00Scenes; Scene-specific elements
    • G06V20/50Context or environment of the image
    • G06V20/56Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • GPHYSICS
    • G01MEASURING; TESTING
    • G01CMEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C3/00Measuring distances in line of sight; Optical rangefinders
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/60Analysis of geometric attributes
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/80Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/30Subject of image; Context of image processing
    • G06T2207/30248Vehicle exterior or interior
    • G06T2207/30252Vehicle exterior; Vicinity of vehicle

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Remote Sensing (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Electromagnetism (AREA)
  • Geometry (AREA)
  • Multimedia (AREA)
  • Studio Devices (AREA)
  • Closed-Circuit Television Systems (AREA)
  • Image Analysis (AREA)

Abstract

The application provides a kind of many mesh camera systems, and it includes:Wide angle camera;Stereoscopic camera, wherein the field range of the stereoscopic camera exist with the field range of the wide angle camera it is overlapping;And image analysis processor, the image that its image and the stereoscopic camera that can receive the wide angle camera capture are captured, and following operate is performed to the image of capture:Target detection is carried out to the image that wide angle camera is captured;Enter row distance detection to the image that stereoscopic camera is captured, obtain the distance of each pixel;The image captured with the image for capturing the wide angle camera and the stereoscopic camera carries out time synchronized and matching is calibrated, so as to obtain the target range information in wide angle camera target identification information within the vision and the stereoscopic camera visual field.Many mesh camera systems of the application have the wide advantage in the wide angle camera visual field and many accurate advantages of mesh camera ranging concurrently, be particularly suitable for intelligent driving system, optimal auxiliary information is provided for intelligent driving system.

Description

Many mesh camera systems, intelligent driving system, automobile, method and storage medium
Technical field
The invention belongs to camera technique field, more particularly to a kind of big visual angle, many mesh camera systems of small blind area, intelligence are driven System, automobile, method and storage medium are sailed, many mesh camera systems are particularly suitable for intelligent driving system.
Background technology
Stereoscopic camera, generally using the less long focal length lens optical system that distorts, causes to stand for accurate measurement distance The visual angle of body phase machine covering is smaller, and blind area is larger.And wide angle camera generally distort it is larger, it is difficult to accurate measurement distance.
In the prior art also without a kind of big small blind area in visual angle and can precisely ranging camera.
The content of the invention
In view of the shortcomings of the prior art, the application provides the camera of the big small blind area in visual angle.
According to an aspect of the invention, there is provided a kind of many mesh camera systems, can include:Wide angle camera;Cubic phase Machine, wherein the field range of the stereoscopic camera exist with the field range of the wide angle camera it is overlapping, the wide angle camera Field range is more than the field range of stereoscopic camera;And image analysis processor, it can receive the wide angle camera capture Image and the image of stereoscopic camera capture, and following operate is performed to the image of capture:The wide angle camera is captured Image carry out target detection;Enter row distance detection to the image that the stereoscopic camera is captured;Captured with by the wide angle camera Image and the stereoscopic camera capture image carry out time synchronized and matching calibrate, so as to obtain wide angle camera field range Target range information in interior target identification information and the stereoscopic camera visual field.
Further, according to above-mentioned many mesh camera systems, the field range of the stereoscopic camera is completely by wide angle camera Field range is covered.
Further, according to above-mentioned many mesh camera systems, described image analysis processor further performs following operate:It is right In the region in the wide angle camera visual field but outside stereoscopic camera field range, carried out by the image captured to wide angle camera Processing, catches and tracks target;And when the target enters the field range of stereoscopic camera, the stereoscopic camera is captured Image in target enter row distance detection, so as to obtain wide angle camera target identification information within the vision and stereoscopic camera Target range information in the visual field.
Further, according to above-mentioned many mesh camera systems, described image analysis processor is also operable to:According to distance inspection Detected distance is surveyed, at least one of following control is carried out:Brake;Lane-change.
Further, above-mentioned many mesh camera systems also include:Wide angle camera and stereoscopic camera are demarcated, obtain three-dimensional The internal reference of camera, the internal reference of wide angle camera and outer ginseng, the outer position relationship and stereoscopic camera that include stereoscopic camera and extensively of joining Position relationship between the camera of angle.
Further, according to above-mentioned many mesh camera systems, in addition to:When predetermined situations occur, demarcation is re-started.
Further, according to above-mentioned many mesh camera systems, the predetermined situations include at least one of following item or It is combined:The scheduled time is gone over away from last time demarcation;The orientation of any of stereoscopic camera, wide angle camera changes;Week The change for exceeding predetermined threshold occurs for the scene for enclosing things.
Further, according to above-mentioned many mesh camera systems, including multiple stereoscopic cameras, the field range of each stereoscopic camera It is overlapping with the different zones within the vision of the wide angle camera.
Further, according to above-mentioned many mesh camera systems, in addition to output equipment, it exports the pattern analysis processor Target range information in the wide angle camera target identification information within the vision of acquisition and the stereoscopic camera visual field.
Further, according to above-mentioned many mesh camera systems, wherein many mesh camera systems are rotatable and/or liftable.
Further, according to above-mentioned many mesh camera systems, wherein the stereoscopic camera is arranged on circulator, it can be only Stand on the deflection angle that wide angle camera produces 0-45 °.
Further, according to above-mentioned many mesh camera systems, wherein the stereoscopic camera includes left mesh camera module and right mesh Camera module, the stereoscopic camera produces binocular image.
Further, according to above-mentioned many mesh camera systems, wherein the wide angle camera and stereoscopic camera are integral structures, The wide angle camera is arranged between the left mesh camera module and the right mesh camera module.
Further, according to above-mentioned many mesh camera systems, wherein carrying out target in the image captured to the wide angle camera Before detection, in addition to:Distortion is carried out to the image that wide angle camera is captured with the internal reference and outer ginseng of demarcation;And described Before " entering row distance detection to the image that the stereoscopic camera is captured ", in addition to:With the internal reference and outer ginseng of demarcation to cubic phase The image of machine capture carries out going distortion.
Further, according to above-mentioned many mesh camera systems, wherein described " enter line-spacing to the image that the stereoscopic camera is captured From detection " include:The image of capture is matched with Stereo Matching Algorithm, the range difference of respective pixel in the picture is obtained; The distance of each pixel in the image of at least one capture is calculated by the internal reference and outer ginseng of stereoscopic camera.
Further, according to above-mentioned many mesh camera systems, described time synchronized and matching calibration further comprises following One or more in step:The timestamp of the image captured to the wide angle camera and the image of stereoscopic camera capture It is compared, it is determined that being the image captured in synchronization;According to the outer ginseng of wide angle camera and stereoscopic camera come by image registration To under a coordinate system;The object pixel coordinate recognized in the image captured from the wide angle camera is mapped to from the solid In the distance obtained in the image of camera capture, range-to-go is obtained.
According to a further aspect of the invention, a kind of intelligent driving system is additionally provided, it includes above-mentioned many mesh camera systems.
According to a further aspect of the invention, a kind of intelligent driving automobile is additionally provided, it uses above-mentioned intelligent driving system.
According to a further aspect of the invention, a kind of method that intelligent driving automobile obtains target range is additionally provided, including:By Vehicle-mounted wide angle camera capture images;By vehicle-mounted stereoscopic camera capture images, wherein the field range of the stereoscopic camera with it is described There is overlapping, field range of the field range more than stereoscopic camera of the wide angle camera in the field range of wide angle camera;To institute The image for stating wide angle camera capture carries out target detection;Enter row distance detection to the image that the stereoscopic camera is captured;With by institute State the image of wide angle camera capture and the image of stereoscopic camera capture carries out time synchronized and matching is calibrated, so as to obtain wide The target range information in target identification information and the stereoscopic camera visual field in the range of the camera fields of view of angle.
Further, according to above-mentioned many mesh camera systems, the field range of the stereoscopic camera is completely by wide angle camera Field range is covered, and methods described also includes:For the area in the wide angle camera visual field but outside stereoscopic camera field range Domain, by handling the image that wide angle camera is captured, catches and tracks target, and when the target enters stereoscopic camera Field range when, to the stereoscopic camera capture image in target enter row distance detection, regarded so as to obtain wide angle camera The target range information in target identification information and the stereoscopic camera visual field in wild scope.
According to a further aspect of the invention, a kind of computer-readable storage medium is additionally provided, be stored with computer executable instructions, The instruction when being executed by a processor, is operable to perform following methods:Obtain the image captured by vehicle-mounted wide angle camera;Obtain The image captured by vehicle-mounted stereoscopic camera is obtained, wherein the field range of the stereoscopic camera and the field range of the wide angle camera There is overlapping, field range of the field range more than stereoscopic camera of the wide angle camera;The figure captured to the wide angle camera As carrying out target detection;Enter row distance detection to the image that the stereoscopic camera is captured;With the figure for capturing the wide angle camera Picture and the image of stereoscopic camera capture carry out time synchronized and matching is calibrated, so that it is within the vision to obtain wide angle camera Target range information in target identification information and the stereoscopic camera visual field.Many mesh camera systems of the application have wide angle camera concurrently and regarded Wild wide advantage and many accurate advantages of mesh camera ranging, are particularly suitable for intelligent driving system, are that intelligent driving system is carried For optimal auxiliary information.
Brief description of the drawings
In order to illustrate more clearly of the technical scheme of the embodiment of the present disclosure, the accompanying drawing of embodiment will be simply situated between below Continue, it should be apparent that, drawings in the following description merely relate to some embodiments of the present disclosure, rather than limitation of this disclosure.
Fig. 1 shows the structural representation of many mesh cameras of one embodiment of the invention.
Fig. 2 shows the example functional structural frames of image analysis processor 30 according to an embodiment of the invention Figure.
Fig. 3 shows the image analysis processing method 200 that image analysis processor 30 according to embodiments of the present invention is performed Overview flow chart.
Fig. 4 shows the flow chart of mono- specific implementation method of step S230 shown in Fig. 3.
Fig. 5 shows the flow chart of the image processing method 300 with anticipation function according to embodiments of the present invention.
Embodiment
To make the purpose, technical scheme and advantage of the embodiment of the present disclosure clearer, below in conjunction with the embodiment of the present disclosure Accompanying drawing, the technical scheme to the embodiment of the present disclosure is clearly and completely described.Obviously, described embodiment is this public affairs A part of embodiment opened, rather than whole embodiments.Based on described embodiment of the disclosure, ordinary skill The every other embodiment that personnel are obtained on the premise of without creative work, belongs to the scope of disclosure protection.
Term in the application is explicitly defined unless had in this application, is otherwise generally managed with those skilled in the art The implication of solution.
Stereo Matching Algorithm is a kind of common technology in computer vision, and its target is from two dimensional image or image sequence It is middle to obtain being appreciated and understood by for external world, i.e., recover the geological information of object in three-dimensional environment using two dimensional image, such as Shape, position, posture, motion etc., and can describe, recognize with being connected.Available for the Stereo Matching Algorithm in the application without spy Do not limit, as long as the need for disclosure satisfy that specific occasion.There are a variety of Stereo Matching Algorithms in the prior art, such as locally Bounding algorithm (it includes Feature Correspondence Algorithm, phase matched algorithm etc.) and global restriction algorithm (it include dynamic programming algorithm, Figure cuts algorithm, intelligent algorithm etc.) etc..
, it is necessary to carry out the demarcation of camera before target identification is carried out using camera.What camera calibration was mainly solved be from The world sits on camera coordinates, then to the projection matrix relation between imager coordinate., it is necessary to use during to camera calibration To " internal reference (i.e. the intrinsic parameter) " and " outer ginseng (outer parameter) " of camera.The meaning of intrinsic parameter includes image from camera coordinates system to picture The distortion factor when transformation parameter and shooting picture of plane coordinate system, in the case that outer parameter is embodied in many camera lenses, each mirror Position relationship between head.Camera internal reference is related to the distortion parameter of camera lens, including radial distortion and tangent line distortion are even There is prismatic distortion.Camera extrinsic can include movement and the three-phase rotation parameter of three-dimensional position.
Telephoto lens:Long focal length lens refers to the phtographic lens longer than the focal length of standard lens.In general, focal length is more than 60mm camera lens is that can be described as telephoto lens.Long focal length lens is divided into common long shot and the class of super long shot two.It is common remote The focal length for taking the photograph camera lens is near the mark camera lens, and the focal length of super long shot is far longer than standard lens.With 135 cameras Exemplified by, its lens focus is common long shot from 85 millimeters -300 millimeters of phtographic lens, and more than 300 millimeters are super remote Take the photograph camera lens.
Wide-angle lens:Wide-angle lens is that a kind of focal length is shorter than standard lens, visual angle and is more than standard lens, focal length and is longer than flake Camera lens, visual angle are less than fish-eye photographic mirror wide-angle lens.In general, visual angle is that can be described as extensively for more than 60 ° of camera lens Angle mirror head.
In this application, term " object " and " target " are when for describing the entity in image, with identical implication, Can be with used interchangeably.
Intelligent driving system, intelligent driving system herein mainly include sensor (broad sense, including vision sensor, Radar, ultrasonic sensor, infrared ray sensor) and control system, control system receives the input of sensor, through undue Analysis, processing, make a policy, and send control instruction, and control system is for example including processor (such as CPU) and memory.
The present invention is further detailed explanation with reference to the accompanying drawings and detailed description:
Fig. 1 is the structural representation of many mesh camera systems of one embodiment of the invention.
As shown in figure 1, many mesh camera apparatus include:Wide angle camera 10, stereoscopic camera 20 and image analysis processor 30.According to the preferred application of the present invention, many mesh camera systems are equipped on autonomous driving vehicle, specifically, the He of wide angle camera 10 Stereoscopic camera 20 is equipped on the front upper place of automobile, to shoot the front situation of automobile.
According to embodiments of the present invention, the field range of wide angle camera 10 is more than the field range of stereoscopic camera 20, described vertical The field range of body phase machine exists overlapping with the field range of the wide angle camera.Preferably, the field range of stereoscopic camera is complete Fall into entirely within the field range of wide angle camera.
Wide angle camera 10 has broader visual angle, and more scenery scope can be accommodated in limited distance, can also be entered The rough object positioning of row.The field range of stereoscopic camera 20 is narrower, but for its object within the vision, can carry out phase To accurately positioning.
As the example of stereoscopic camera, there are binocular camera, TOF camera, structure light camera etc..
Binocular camera is shown in Fig. 1 as the example of stereoscopic camera, binocular camera 20 includes the left He of mesh camera module 202 Right mesh camera module 203, is arranged on stereoscopic camera support 201, and capture respectively obtains left image and right image, left image and Right image can be sent to image analysis processor 30, and distortion correction, range information calculating etc. are carried out by image analysis processor Operation.
Binocular camera is most common a kind of form in stereoscopic camera, also more cost-effective, in many mesh camera systems The use of the left mesh camera module and right mesh camera module of binocular camera is than advantageous.
Stereoscopic camera is preferred to use the less long focal length lens optical system of distortion.
So that many mesh camera systems are equipped on autonomous driving vehicle in Examples below, and binocular camera is used as solid The example of camera is described.
Wide angle camera 10, stereoscopic camera 20 transmit image information to image analysis processor 30, image analysis processor 30 images for transmitting wide angle camera 10 carry out distortion correction, and within the vision to wide angle camera using algorithm of target detection Scene carries out target detection, while image analysis processor 30 enters row distance inspection to the object within the vision of stereoscopic camera 20 Survey, and the image of image and the stereoscopic camera capture captured to wide angle camera carries out time synchronized and matching calibration process.
Image analysis processor 30 can be such as dedicated computing device or general-purpose calculating appts, can have center processing Be stored with computer executable instructions in unit and memory, memory, and computer executable instructions can be loaded into internal memory in Central processor is performed, and image analysis processor can also be the device that is cooperated by distributed component, and such as section components can be Be desirably integrated on high in the clouds, or even partial function component physical in stereoscopic camera 20 and/or wide angle camera 10, for example go distortion, Distance calculating, camera calibration etc..
It is exemplary, stereoscopic camera and wide angle camera independently of one another between pattern analysis processor by wireless connection or Person is transmitted by wired connection.Connected mode between stereoscopic camera or wide angle camera and pattern analysis processor is this area skill Art personnel can carry out selection as needed, and respective advantage and disadvantage are that those skilled in the art can weigh as needed.
In the example illustrated in figure 1, wide angle camera and stereoscopic camera are integral structures, and wherein stereoscopic camera includes left mesh phase Machine module and right mesh camera module, wide angle camera are arranged between the left mesh camera module and right mesh camera module.But, replace Dai Di, wide angle camera and stereoscopic camera can be split-type structurals.So-called wide angle camera and stereoscopic camera are integral structures Represent that many mesh cameras are integral, can not be split;Contrastingly, wide angle camera and stereoscopic camera are that split-type structural represents both It can split.
In some instances, many mesh camera systems are fixed.In other example, many mesh camera systems are overall It is rotatable, for example rotated when turning round to the direction for needing to turn round.Many mesh camera systems integrally it is rotatable for example It can be realized by the way that many mesh camera systems are arranged on circulator.In other examples, many mesh cameras be it is liftable, For example by the way that many mesh camera systems are installed and can realized in Sheng Jia mechanisms.By causing many mesh camera systems rotatable and/or Liftable so that the field range of many mesh camera systems can optionally be adjusted flexibly.
In the example illustrated in figure 1, many mesh camera systems include a stereoscopic camera.But this is merely illustrative, other In example, many mesh camera systems can include multiple stereoscopic cameras, the field range of each stereoscopic camera and regarding for wide angle camera Different zones in wild scope are overlapping.
In the example depicted in fig. 1, many mesh camera systems include a wide angle camera.But this is merely illustrative, other In example, many mesh camera systems can include multiple wide angle cameras, it is preferable that different wide angle cameras cover different visual field models Enclose, can have overlapping between different wide angle camera field ranges.
In some instances, many mesh camera systems can also include output equipment, and its output pattern analysis processor is obtained Wide angle camera target identification information within the vision and the stereoscopic camera visual field in target range information.When many mesh camera systems System is installed on intelligent driving system for when carrying out target acquisition, these information are for holding for intelligent driving system The reference information of row automatic Pilot.Now, the input equipment connection of its output equipment and intelligent driving system, intelligent in other words to drive Sail system and obtain information from the output equipment.
In some instances, stereoscopic camera may be mounted on circulator, and it can produce 0-45 ° independently of wide angle camera Deflection angle so that the visual field of stereoscopic camera can change.Such many mesh camera systems are more suitable for being used for Intelligent driving system.In the case of automobile turning, if stereoscopic camera can deflect, then can detect better Target conditions on bend, so as to advantageously make correct Driving Decision-making in intelligent driving system.
Fig. 2 shows the example functional structural frames of image analysis processor 30 according to an embodiment of the invention Figure.
As shown in Fig. 2 image analysis processor 30 includes:Camera calibration device 31, wide angle camera image processor 32, solid Camera image processor 33 and image registration device 34.
31 pairs of camera calibration device many mesh cameras are demarcated, and obtain the internal reference and outer ginseng of many mesh cameras.Wherein institute State the internal reference of internal reference including stereoscopic camera and the internal reference of wide angle camera, outer ginseng include stereoscopic camera outer ginseng and stereoscopic camera and Outer ginseng between wide angle camera.
The image that wide angle camera image processor 32 is captured according to the internal reference of wide angle camera to wide angle camera carries out going distortion Processing, and carry out target detection.
The image progress that stereoscopic camera image device 33 is captured according to the internal reference and outer ginseng of stereoscopic camera to stereoscopic camera is gone abnormal Change is handled, and performs distance detection algorithm, obtains the range information of image pixel.
The image and the image of stereoscopic camera capture that image registration device 34 captures wide angle camera carry out time synchronized Calibrated with matching, so that based on wide angle camera target identification information within the vision, to obtain the range information of target.
Below with reference to the accompanying drawings the processing example that image analysis processor 30 is performed is specifically described.
Fig. 3 shows the image analysis processing method 200 that image analysis processor 30 according to embodiments of the present invention is performed Overview flow chart.
Firstly the need of explanation, before the work of many mesh camera systems, some initialization operations are carried out, including to wide Angle camera and stereoscopic camera are demarcated, and obtain the internal reference, the internal reference of wide angle camera and outer ginseng of stereoscopic camera.In cubic phase In the case of machine is binocular camera, the internal reference of stereoscopic camera includes each interior of left mesh camera and right mesh camera in binocular camera Ginseng;Outer ginseng includes the related outer ginseng of position relationship between left mesh camera and right mesh camera, and with stereoscopic camera and wide-angle The outer ginseng of position relationship correlation between camera.Internal reference for image go distortion processing needed for, outer ginseng is the registration between image It is required.The demarcation of the embodiment of the present invention includes the demarcation of stereoscopic camera itself, the demarcation of wide angle camera itself, and stereoscopic camera The demarcation of relation between wide angle camera.The demarcation of stereoscopic camera include internal reference (lens distortion) and it is outer join (two receive mesh it Between position relationship or send mesh and receive the position relationship between mesh), and the demarcation of stereoscopic camera and wide angle camera is related to two Position relationship between person.
In addition to being demarcated when initialization, it is preferable that when predetermined situations occur, camera system is carried out again Demarcation.Such as predetermined situations include at least one of following item or its combination:The scheduled time is gone over away from last time demarcation; The orientation of any of stereoscopic camera, wide angle camera changes;Change beyond predetermined threshold occurs for the scene of surrounding.
Fig. 3 flow chart assumes that relevant demarcation has been completed, and image analysis processor has been obtained for wide angle camera The image of image and the stereoscopic camera capture of capture.
In step S210, target detection is carried out to the image that the wide angle camera is captured, the pixel coordinate of target is obtained.
For example, via the pedestrian in image procossing detection image and/or vehicle.Relevant detection algorithm can use conventional Target Recognition Algorithms, such as neutral net, SVM, this non-invention main points omits detailed description thereof here.
In step S220, row distance detection is entered to the image that the stereoscopic camera is captured, the distance of each pixel is obtained.
Specifically, as implementation example, row distance detection is entered to the image that the stereoscopic camera is captured, each pixel is obtained Distance can include:The image of capture is matched with Stereo Matching Algorithm, the distance of respective pixel in the picture is obtained Difference;The distance of each pixel in the image of at least one capture is calculated by the internal reference and outer ginseng of stereoscopic camera.
For example, when stereoscopic camera is binocular camera, with Stereo Matching Algorithm for left mesh image and right mesh figure As being matched, the range difference of respective pixel in the picture is obtained;It is then based on the internal reference of left mesh camera and right mesh camera and outer Ginseng, at least one in left mesh image and right mesh image is obtained to calculate, such as using left mesh image as reference picture in the case of, it is left The distance of each pixel in mesh image, this distance characterizes corresponding range information of the physical points away from binocular camera of pixel.
Technology customary in the art can be used about Stereo Matching Algorithm and distance calculating method, this non-invention will Point, is repeated no more here.
It is further to note that in step S210 to wide angle camera capture image processing and step S220 in The processing of the image captured to stereoscopic camera, can independently be carried out between the two, not specially required for order each other, For example, can serially, parallel or any combination order carries out.
The image for capturing the wide angle camera in step S230 and the image of stereoscopic camera capture carry out the time Synchronous and matching calibration, so as to obtain the target in wide angle camera target identification information within the vision and the stereoscopic camera visual field Range information.
Fig. 4 shows the flow chart of a specific implementation method of the step S230 shown in Fig. 3.
The timestamp of the image captured in step S231 to the wide angle camera and the image of stereoscopic camera capture It is compared, it is determined that being the image captured in synchronization.
In this application, the image of synchronization capture refers to that the capture time difference of two images is less than certain threshold value (such as 50ms).If the such as capture time of stereoscopic camera image is T1, the capture time of wide angle camera image is T2, then only Need | T1-T2 |<50ms is to think that two images are in synchronization capture, you can carry out follow-up operation.
In step S232, image and the stereoscopic camera capture for being captured wide angle camera according to the outer ginseng of many mesh cameras Image registration is under a coordinate system.
In step S233, the object pixel coordinate recognized in the image captured from the wide angle camera is mapped to from institute In the distance obtained in the image for stating stereoscopic camera capture, range-to-go is obtained.
Fig. 3 is returned to, after step S230 processing is completed, the range information of target is just obtained.Such target away from The intelligent driving system to autonomous driving vehicle can be exported from information, intelligent driving system is determined according to the range information Plan, and send instruction for action actuating mechanism, such as instruction are braked, lane-change or provide alarm etc..
According to another embodiment of the present invention, stereoscopic camera field range, Yi Jichong is completely covered in wide angle camera field range Divide the advantage being had a wide sphere of vision using wide angle camera, predict the movement tendency of target, enter the moment in the stereoscopic camera visual field in target, By stereoscopic camera image and wide angle camera image registration, the distance of target is determined rapidly, and the distance of determination is supplied to Intelligent driving system, to carry out decision-making.
Image processing method 300 with anticipation function according to embodiments of the present invention is described with reference to Fig. 5.
As shown in figure 5, in step S310, in the wide angle camera visual field but outside stereoscopic camera field range Region, by handling the image that wide angle camera is captured, catches and tracks object, and predict the movement tendency of the object.
According to wide angle camera and the internal reference and outer ginseng of stereoscopic camera, which area in the image of wide angle camera capture can determine Domain is outside the field range of stereoscopic camera, for these regions, can catch and track object, and predict the motion of the object Trend.
Can use the Moving Objects of image processing field recognize and tracking technique come realize seizure to object, tracking, Movement tendency is predicted.
In step s 320, row distance detection is entered to the image that the stereoscopic camera is captured, the distance of each pixel is obtained. Step S320 realization is similar with the step S220 shown in Fig. 3, repeats no more here.
In step S330, judgment object is just entering the field range of stereoscopic camera.If it is, step S330 is proceeded to, Otherwise return in step S310.
The image for capturing the wide angle camera in step S340 and the image of stereoscopic camera capture carry out the time Synchronous and matching calibration, so as to obtain the target in wide angle camera target identification information within the vision and the stereoscopic camera visual field Range information.
More specifically, as an example, the direction of motion and movement velocity based on the object estimated by tracking object technology, energy Enough determine object probably when enter stereoscopic camera field range, then can the entrance stereoscopic camera of estimation field range At the moment (or somewhat shifting to an earlier date a little time), the part that stereoscopic camera field range is entered to wide angle camera objects in images is carried out It is determined that, then wide angle camera image and stereoscopic camera image are synchronized and registration, next enter cubic phase for object The part of machine field range determines range information.
Thereby, it is possible to the very first time know one tracking object accurate range information, now the tracking object for example before Square people or vehicle have just entered stereoscopic camera field range and (have been likely to have been enter into or will enter this vehicle for carrying stereoscopic camera Track in), know immediately that its range information, can judge accordingly between this car and target whether in safe distance, whether It is necessary the measure of taking action.
According to a further aspect of the invention, a kind of intelligent driving system is additionally provided, it includes above-mentioned many mesh camera systems, intelligence Can control loop as intelligent driving automobile brain, using many mesh camera systems provide target identification information and apart from letter Breath, to make intelligent driving decision-making, and sends control instruction to execution unit is driven.
According to a further aspect of the invention, a kind of intelligent driving automobile is additionally provided, it uses above-mentioned intelligent driving system.
According to a further aspect of the invention, a kind of method that intelligent driving automobile obtains target range is additionally provided, including:By Vehicle-mounted wide angle camera capture images;By vehicle-mounted stereoscopic camera shooting image;The image captured by the wide angle camera carries out mesh Mark detection, obtains the pixel coordinate of target;The image captured by the stereoscopic camera enters row distance detection, obtains each pixel Distance;The image captured with the image for capturing the wide angle camera and the stereoscopic camera carries out time synchronized and matching school Standard, so as to obtain the target range information in wide angle camera target identification information within the vision and the stereoscopic camera visual field.
According to a further aspect of the invention, a kind of computer-readable storage medium is additionally provided, be stored with computer executable instructions, The instruction when being executed by a processor, is operable to perform following methods:Obtain the image captured by vehicle-mounted wide angle camera;Obtain Obtain the image captured by vehicle-mounted stereoscopic camera;The image captured by the wide angle camera carries out target detection, obtains target Pixel coordinate;The image captured by the stereoscopic camera enters row distance detection, obtains the distance of each pixel;With will be described wide The image of angle camera capture and the image of stereoscopic camera capture carry out time synchronized and matched to calibrate, so as to obtain wide-angle phase Target range information in machine target identification information within the vision and the stereoscopic camera visual field.
Many mesh camera systems of the disclosure combine the essence of big visual angle, the advantage of small blind area and the stereoscopic camera of wide angle camera The advantage of quasi- ranging, can track the motion of target, predict the movement tendency of target, and fast provide may influence this car to drive Target range information, be particularly suitable for Driving Decision-making and the control of intelligent driving system.
Many mesh camera systems described herein are particularly suitable in intelligent driving system being used for the identification of target, and it can Target in enough identification is a wide range of, can recognize the target range in the range of emphasis again, can be provided for intelligent driving system compared with Good input data.Therefore, the application also provides a kind of intelligent driving system, and it includes the above-described many mesh cameras of any one System.
It is described above various embodiments of the present invention, described above is exemplary, and non-exclusive, and It is not limited to disclosed each embodiment.In the case of without departing from the scope and spirit of illustrated each embodiment, for this skill Many modifications and changes will be apparent from for the those of ordinary skill in art field.Therefore, protection scope of the present invention should It should be defined by scope of the claims.

Claims (10)

1. a kind of many mesh camera systems, it includes:
Wide angle camera;
Stereoscopic camera, wherein the field range of the stereoscopic camera exist with the field range of the wide angle camera it is overlapping, it is described The field range of wide angle camera is more than the field range of stereoscopic camera;With
The image of image analysis processor, its image that can receive the wide angle camera capture and stereoscopic camera capture, And following operate is performed to the image of capture:
Target detection is carried out to the image that the wide angle camera is captured;
Enter row distance detection to the image that the stereoscopic camera is captured;With
The image that the wide angle camera is captured and the image of stereoscopic camera capture carry out time synchronized and matching is calibrated, from And obtain the target range information in wide angle camera target identification information within the vision and the stereoscopic camera visual field.
2. many mesh camera systems according to claim 1, the field range of the stereoscopic camera is completely by wide angle camera Field range is covered.
3. many mesh camera systems according to claim 2, described image analysis processor further performs following operate:
For the region in the wide angle camera visual field but outside stereoscopic camera field range, pass through the figure captured to wide angle camera As being handled, catch and track target;And
When the target enters the field range of stereoscopic camera, the target in the image captured to the stereoscopic camera enters line-spacing From detection, so as to obtain the target range letter in wide angle camera target identification information within the vision and the stereoscopic camera visual field Breath.
4. many mesh camera systems according to claim 3, described image analysis processor is also operable to:
Distance according to detected by distance detection, carries out at least one of following control:
Brake;
Lane-change.
5. many mesh camera systems according to claim 1, in addition to:
Wide angle camera and stereoscopic camera are demarcated, the internal reference, the internal reference of wide angle camera and outer ginseng of stereoscopic camera is obtained, The outer position relationship joined between the position relationship and stereoscopic camera and wide angle camera that include stereoscopic camera.
6. many mesh camera systems according to claim 5, in addition to:
When predetermined situations occur, demarcation is re-started.
7. a kind of intelligent driving system, it includes many mesh camera systems any one of claim 1 to 6.
8. a kind of intelligent driving automobile, it uses intelligent driving system according to claim 7.
9. a kind of method that intelligent driving automobile obtains target range, including:
By vehicle-mounted wide angle camera capture images;
By vehicle-mounted stereoscopic camera capture images, wherein the field range of the stereoscopic camera and the field range of the wide angle camera There is overlapping, field range of the field range more than stereoscopic camera of the wide angle camera;
Target detection is carried out to the image that the wide angle camera is captured;
Enter row distance detection to the image that the stereoscopic camera is captured;With
The image that the wide angle camera is captured and the image of stereoscopic camera capture carry out time synchronized and matching is calibrated, from And obtain the target range information in wide angle camera target identification information within the vision and the stereoscopic camera visual field.
10. a kind of computer-readable storage medium, be stored with computer executable instructions, and the instruction when being executed by a processor, can Operate to perform following methods:
Obtain the image captured by vehicle-mounted wide angle camera;
The image that is captured by vehicle-mounted stereoscopic camera is obtained, wherein the field range of the stereoscopic camera and the wide angle camera are regarded There is overlapping, field range of the field range more than stereoscopic camera of the wide angle camera in wild scope;
Target detection is carried out to the image that the wide angle camera is captured;
Enter row distance detection to the image that the stereoscopic camera is captured;With
The image that the wide angle camera is captured and the image of stereoscopic camera capture carry out time synchronized and matching is calibrated, from And obtain the target range information in wide angle camera target identification information within the vision and the stereoscopic camera visual field.
CN201710443135.9A 2017-06-13 2017-06-13 Multi-camera system, intelligent driving system, automobile, method and storage medium Active CN107122770B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201710443135.9A CN107122770B (en) 2017-06-13 2017-06-13 Multi-camera system, intelligent driving system, automobile, method and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710443135.9A CN107122770B (en) 2017-06-13 2017-06-13 Multi-camera system, intelligent driving system, automobile, method and storage medium

Publications (2)

Publication Number Publication Date
CN107122770A true CN107122770A (en) 2017-09-01
CN107122770B CN107122770B (en) 2023-06-27

Family

ID=59730426

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710443135.9A Active CN107122770B (en) 2017-06-13 2017-06-13 Multi-camera system, intelligent driving system, automobile, method and storage medium

Country Status (1)

Country Link
CN (1) CN107122770B (en)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107862719A (en) * 2017-11-10 2018-03-30 未来机器人(深圳)有限公司 Scaling method, device, computer equipment and the storage medium of Camera extrinsic
CN108010055A (en) * 2017-11-23 2018-05-08 塔普翊海(上海)智能科技有限公司 The tracking system and its tracking of three-dimensional body
CN108805910A (en) * 2018-06-01 2018-11-13 海信集团有限公司 More mesh Train-borne recorders, object detection method, intelligent driving system and automobile
WO2020042000A1 (en) * 2018-08-29 2020-03-05 深圳市大疆创新科技有限公司 Camera device and focusing method
CN112602091A (en) * 2018-07-30 2021-04-02 优创半导体科技有限公司 Object detection using multiple neural networks trained for different image fields
CN112684424A (en) * 2020-12-30 2021-04-20 同济大学 Automatic calibration method for millimeter wave radar and camera
CN113306497A (en) * 2021-05-27 2021-08-27 新石器慧通(北京)科技有限公司 Camera fixed bolster and unmanned car
CN113379796A (en) * 2021-05-25 2021-09-10 成都飞机工业(集团)有限责任公司 Large-range multi-camera tracking control method and system
CN114677841A (en) * 2022-02-10 2022-06-28 浙江大华技术股份有限公司 Vehicle lane change detection method and vehicle lane change detection system

Citations (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH08329222A (en) * 1995-06-02 1996-12-13 Takenaka Komuten Co Ltd Three-dimensional position recognition device
CN101271187A (en) * 2008-03-20 2008-09-24 汤一平 Non-dead angle binocular solid all-directional vision sensing equipment
CN101308018A (en) * 2008-05-30 2008-11-19 汤一平 Stereo vision measuring apparatus based on binocular omnidirectional visual sense sensor
CN101373261A (en) * 2007-08-22 2009-02-25 鸿富锦精密工业(深圳)有限公司 Wide-angle lens and vehicle apparatus using the same
CN101543409A (en) * 2008-10-24 2009-09-30 南京大学 Long-distance iris identification device
CN101577002A (en) * 2009-06-16 2009-11-11 天津理工大学 Calibration method of fish-eye lens imaging system applied to target detection
JP2011120076A (en) * 2009-12-04 2011-06-16 Panasonic Corp Image detection apparatus and image detection method
CN102132556A (en) * 2008-08-20 2011-07-20 国立大学法人东京工业大学 Long-distance target detection camera system
CN102175221A (en) * 2011-01-20 2011-09-07 上海杰图软件技术有限公司 Vehicle-mounted mobile photographic surveying system based on fisheye lens
JP2011206425A (en) * 2010-03-30 2011-10-20 Fujifilm Corp Image processor, image processing method, image processing program, and stereoscopic endoscope
JP2012226645A (en) * 2011-04-21 2012-11-15 Sony Corp Image processing apparatus, image processing method, recording medium, and program
CN102822621A (en) * 2010-04-06 2012-12-12 富士胶片株式会社 Calibration data selection device, method of selection, selection program, and three dimensional position measuring device
CN103051887A (en) * 2013-01-23 2013-04-17 河海大学常州校区 Eagle eye-imitated intelligent visual sensing node and work method thereof
JP2014042160A (en) * 2012-08-22 2014-03-06 Ntt Docomo Inc Display terminal, setting method of target area of moving body detection and program
CN103770708A (en) * 2012-10-22 2014-05-07 通用汽车环球科技运作有限责任公司 Dynamic rearview mirror adaptive dimming overlay through scene brightness estimation
CN104270570A (en) * 2014-10-17 2015-01-07 北京英泰智软件技术发展有限公司 Binocular video camera and image processing method thereof
CN105550670A (en) * 2016-01-27 2016-05-04 兰州理工大学 Target object dynamic tracking and measurement positioning method
CN105678787A (en) * 2016-02-03 2016-06-15 西南交通大学 Heavy-duty lorry driving barrier detection and tracking method based on binocular fisheye camera
CN105893510A (en) * 2016-03-30 2016-08-24 北京格灵深瞳信息技术有限公司 Video structurization system and target search method thereof
CN105946853A (en) * 2016-04-28 2016-09-21 中山大学 Long-distance automatic parking system and method based on multi-sensor fusion
CN105989354A (en) * 2015-05-25 2016-10-05 乐视移动智能信息技术(北京)有限公司 Positioning method and system

Patent Citations (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH08329222A (en) * 1995-06-02 1996-12-13 Takenaka Komuten Co Ltd Three-dimensional position recognition device
CN101373261A (en) * 2007-08-22 2009-02-25 鸿富锦精密工业(深圳)有限公司 Wide-angle lens and vehicle apparatus using the same
CN101271187A (en) * 2008-03-20 2008-09-24 汤一平 Non-dead angle binocular solid all-directional vision sensing equipment
CN101308018A (en) * 2008-05-30 2008-11-19 汤一平 Stereo vision measuring apparatus based on binocular omnidirectional visual sense sensor
CN102132556A (en) * 2008-08-20 2011-07-20 国立大学法人东京工业大学 Long-distance target detection camera system
CN101543409A (en) * 2008-10-24 2009-09-30 南京大学 Long-distance iris identification device
CN101577002A (en) * 2009-06-16 2009-11-11 天津理工大学 Calibration method of fish-eye lens imaging system applied to target detection
JP2011120076A (en) * 2009-12-04 2011-06-16 Panasonic Corp Image detection apparatus and image detection method
JP2011206425A (en) * 2010-03-30 2011-10-20 Fujifilm Corp Image processor, image processing method, image processing program, and stereoscopic endoscope
CN102822621A (en) * 2010-04-06 2012-12-12 富士胶片株式会社 Calibration data selection device, method of selection, selection program, and three dimensional position measuring device
CN102175221A (en) * 2011-01-20 2011-09-07 上海杰图软件技术有限公司 Vehicle-mounted mobile photographic surveying system based on fisheye lens
JP2012226645A (en) * 2011-04-21 2012-11-15 Sony Corp Image processing apparatus, image processing method, recording medium, and program
JP2014042160A (en) * 2012-08-22 2014-03-06 Ntt Docomo Inc Display terminal, setting method of target area of moving body detection and program
CN103770708A (en) * 2012-10-22 2014-05-07 通用汽车环球科技运作有限责任公司 Dynamic rearview mirror adaptive dimming overlay through scene brightness estimation
CN103051887A (en) * 2013-01-23 2013-04-17 河海大学常州校区 Eagle eye-imitated intelligent visual sensing node and work method thereof
CN104270570A (en) * 2014-10-17 2015-01-07 北京英泰智软件技术发展有限公司 Binocular video camera and image processing method thereof
CN105989354A (en) * 2015-05-25 2016-10-05 乐视移动智能信息技术(北京)有限公司 Positioning method and system
CN105550670A (en) * 2016-01-27 2016-05-04 兰州理工大学 Target object dynamic tracking and measurement positioning method
CN105678787A (en) * 2016-02-03 2016-06-15 西南交通大学 Heavy-duty lorry driving barrier detection and tracking method based on binocular fisheye camera
CN105893510A (en) * 2016-03-30 2016-08-24 北京格灵深瞳信息技术有限公司 Video structurization system and target search method thereof
CN105946853A (en) * 2016-04-28 2016-09-21 中山大学 Long-distance automatic parking system and method based on multi-sensor fusion

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
单体岗: "履带式机器人的电子定向和双目视觉测距", 《CNKI》 *

Cited By (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107862719A (en) * 2017-11-10 2018-03-30 未来机器人(深圳)有限公司 Scaling method, device, computer equipment and the storage medium of Camera extrinsic
CN107862719B (en) * 2017-11-10 2020-10-27 未来机器人(深圳)有限公司 Method and device for calibrating external parameters of camera, computer equipment and storage medium
CN108010055A (en) * 2017-11-23 2018-05-08 塔普翊海(上海)智能科技有限公司 The tracking system and its tracking of three-dimensional body
CN108010055B (en) * 2017-11-23 2022-07-12 塔普翊海(上海)智能科技有限公司 Tracking system and tracking method for three-dimensional object
CN108805910A (en) * 2018-06-01 2018-11-13 海信集团有限公司 More mesh Train-borne recorders, object detection method, intelligent driving system and automobile
CN112602091A (en) * 2018-07-30 2021-04-02 优创半导体科技有限公司 Object detection using multiple neural networks trained for different image fields
WO2020042000A1 (en) * 2018-08-29 2020-03-05 深圳市大疆创新科技有限公司 Camera device and focusing method
CN111345025A (en) * 2018-08-29 2020-06-26 深圳市大疆创新科技有限公司 Camera device and focusing method
CN112684424A (en) * 2020-12-30 2021-04-20 同济大学 Automatic calibration method for millimeter wave radar and camera
CN112684424B (en) * 2020-12-30 2022-08-16 同济大学 Automatic calibration method for millimeter wave radar and camera
CN113379796A (en) * 2021-05-25 2021-09-10 成都飞机工业(集团)有限责任公司 Large-range multi-camera tracking control method and system
CN113306497A (en) * 2021-05-27 2021-08-27 新石器慧通(北京)科技有限公司 Camera fixed bolster and unmanned car
CN114677841A (en) * 2022-02-10 2022-06-28 浙江大华技术股份有限公司 Vehicle lane change detection method and vehicle lane change detection system
CN114677841B (en) * 2022-02-10 2023-12-29 浙江大华技术股份有限公司 Vehicle lane change detection method and vehicle lane change detection system

Also Published As

Publication number Publication date
CN107122770B (en) 2023-06-27

Similar Documents

Publication Publication Date Title
CN107122770A (en) Many mesh camera systems, intelligent driving system, automobile, method and storage medium
JP7345504B2 (en) Association of LIDAR data and image data
US7321386B2 (en) Robust stereo-driven video-based surveillance
US11019322B2 (en) Estimation system and automobile
KR102516326B1 (en) Camera extrinsic parameters estimation from image lines
US20180108150A1 (en) Systems and methods for in-field stereocamera calibration
EP3089449B1 (en) Method for obtaining light-field data using a non-light-field imaging device, corresponding device, computer program product and non-transitory computer-readable carrier medium
EP3028252B1 (en) Rolling sequential bundle adjustment
CN102313536B (en) Method for barrier perception based on airborne binocular vision
US20180137629A1 (en) Processing apparatus, imaging apparatus and automatic control system
JP2018522348A (en) Method and system for estimating the three-dimensional posture of a sensor
KR102151815B1 (en) Method and Apparatus for Vehicle Detection Using Lidar Sensor and Camera Convergence
KR20160062880A (en) road traffic information management system for g using camera and radar
JP2006252473A (en) Obstacle detector, calibration device, calibration method and calibration program
US20170116488A1 (en) Method for identifying an incoming vehicle and corresponding system
US20180276844A1 (en) Position or orientation estimation apparatus, position or orientation estimation method, and driving assist device
US20200358962A1 (en) Image Processing Device
Bazin et al. UAV attitude estimation by vanishing points in catadioptric images
US20220065976A1 (en) Mobile body detection device, mobile body detection method, and mobile body detection program
TWI499999B (en) The 3D ring car image system based on probability calculation and its obtaining method
CN109587304B (en) Electronic equipment and mobile platform
CN111144415A (en) Method for detecting micro pedestrian target
CN115797405A (en) Multi-lens self-adaptive tracking method based on vehicle wheel base
JP6886136B2 (en) Alignment device, alignment method and computer program for alignment
JP2007233487A (en) Pedestrian detection method, device, and program

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant