WO2017154389A1 - 画像処理装置、撮像装置、移動体機器制御システム、画像処理方法、及びプログラム - Google Patents
画像処理装置、撮像装置、移動体機器制御システム、画像処理方法、及びプログラム Download PDFInfo
- Publication number
- WO2017154389A1 WO2017154389A1 PCT/JP2017/002393 JP2017002393W WO2017154389A1 WO 2017154389 A1 WO2017154389 A1 WO 2017154389A1 JP 2017002393 W JP2017002393 W JP 2017002393W WO 2017154389 A1 WO2017154389 A1 WO 2017154389A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- road surface
- image
- distance
- vertical direction
- unit
- Prior art date
Links
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R21/00—Arrangements or fittings on vehicles for protecting or preventing injuries to occupants or pedestrians in case of accidents or other traffic risks
- B60R21/02—Occupant safety arrangements or fittings, e.g. crash pads
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R21/00—Arrangements or fittings on vehicles for protecting or preventing injuries to occupants or pedestrians in case of accidents or other traffic risks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/60—Memory management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/16—Anti-collision systems
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R21/00—Arrangements or fittings on vehicles for protecting or preventing injuries to occupants or pedestrians in case of accidents or other traffic risks
- B60R2021/003—Arrangements or fittings on vehicles for protecting or preventing injuries to occupants or pedestrians in case of accidents or other traffic risks characterised by occupant or pedestian
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R21/00—Arrangements or fittings on vehicles for protecting or preventing injuries to occupants or pedestrians in case of accidents or other traffic risks
- B60R21/01—Electrical circuits for triggering passive safety arrangements, e.g. airbags, safety belt tighteners, in case of vehicle accidents or impending vehicle accidents
- B60R2021/01204—Actuation parameters of safety arrangents
- B60R2021/01252—Devices other than bags
- B60R2021/01259—Brakes
Definitions
- the present invention relates to an image processing apparatus, an imaging apparatus, a mobile device control system, an image processing method, and a program.
- a V-Disparity image in which the coordinate in the vertical direction of an image is one axis, the parallax of the image is the other axis, and the frequency of parallax is a pixel value from a plurality of images captured by a stereo camera.
- the road surface is detected from the generated V-Disparity image.
- a U-Disparity image is generated with the horizontal coordinate of the image as the vertical axis, the parallax of the image as the horizontal axis, and the parallax frequency as the pixel value. Then, based on the generated U-Disparity image, an object such as a person or another vehicle is recognized.
- an image processing apparatus from a distance image having a distance value corresponding to a distance of a road surface in a plurality of photographed images respectively photographed by a plurality of imaging units, a vertical direction showing distribution of frequency of distance values in the vertical direction of the distance image.
- a generation unit for generating distribution data an extraction unit for setting a search range corresponding to a predetermined reference point in the vertical direction distribution data, and extracting a plurality of pixels from the search range, and the plurality of extracted pixels
- a detection unit that detects a road surface.
- FIG. 1 is a view showing the configuration of an in-vehicle device control system as a mobile device control system according to an embodiment of the present invention.
- the on-vehicle device control system 1 is mounted on a host vehicle 100 such as a car which is a moving body, and includes an imaging unit 500, an image analysis unit 600, a display monitor 103, and a vehicle travel control unit 104. Then, relative height information (relatively) of the road surface (moving surface) ahead of the host vehicle is obtained from the captured image data of the area ahead of the host vehicle traveling direction (imaging area) captured in front of the moving object by the imaging unit 500. Information indicating the inclination state is detected, and from the detection result, the three-dimensional shape of the traveling road surface ahead of the host vehicle is detected, and the moving object and various in-vehicle devices are controlled using the detection result.
- the control of the moving body includes, for example, notification of a warning, control of the steering wheel of the own vehicle 100 (self-moving body), or a brake of the own vehicle 100 (self-moving body).
- the imaging unit 500 is installed, for example, in the vicinity of a rearview mirror (not shown) of the windshield 105 of the vehicle 100.
- Various data such as captured image data obtained by imaging of the imaging unit 500 is input to an image analysis unit 600 as an image processing unit.
- the image analysis unit 600 analyzes the data transmitted from the imaging unit 500, and on the traveling road surface ahead of the vehicle with respect to the road surface portion on which the vehicle 100 is traveling (a road surface portion located directly below the vehicle). The relative height (position information) at each point is detected, and the three-dimensional shape of the traveling road surface ahead of the host vehicle is grasped. In addition, other objects ahead of the host vehicle, pedestrians, and various obstacles and other objects to be recognized are recognized.
- the analysis result of the image analysis unit 600 is sent to the display monitor 103 and the vehicle travel control unit 104.
- the display monitor 103 displays the captured image data and the analysis result obtained by the imaging unit 500.
- the vehicle travel control unit 104 notifies, for example, a warning to the driver of the vehicle 100 based on the recognition result of the recognition target such as another vehicle ahead of the vehicle by the image analysis unit 600, a pedestrian, and various obstacles. And driving assistance control such as controlling the steering wheel and brake of the own vehicle.
- FIG. 2 is a diagram showing the configuration of the imaging unit 500 and the image analysis unit 600.
- the imaging unit 500 is configured of a stereo camera provided with two imaging units 510a and 510b as imaging means, and the two imaging units 510a and 510b are the same.
- the respective imaging units 510a and 510b respectively include imaging lenses 511a and 511b, sensor substrates 514a and 514b including image sensors 513a and 513b on which light receiving elements are two-dimensionally arranged, and analog signals output from the sensor substrates 514a and 514b.
- Electric signal (electrical signal corresponding to the amount of light received by each light receiving element on the image sensor 513a, 513b) is converted into a digital electric signal, and it is composed of signal processing units 515a, 515b for generating and outputting ing.
- the imaging unit 500 outputs luminance image data and parallax image data.
- the imaging unit 500 further includes a processing hardware unit 510 including an FPGA (Field-Programmable Gate Array) or the like.
- the processing hardware unit 510 calculates the parallax values of the corresponding image portions between the images captured by the imaging units 510a and 510b.
- a parallax calculator 511 is provided as parallax image information generation means for calculating.
- the parallax value referred to here is a comparison image with respect to an image portion on the reference image corresponding to the same point in the imaging region, with one of the images taken by each of the imaging units 510a and 510b as the reference image and the other as the comparison image.
- the positional displacement amount of the upper image portion is calculated as the parallax value of the image portion.
- the image analysis unit 600 includes an image processing board or the like, and stores the luminance image data and the parallax image data output from the imaging unit 500 as storage means 601 including RAM, ROM, etc.
- a central processing unit (CPU) 602 that executes a computer program for performing parallax calculation control and the like, a data I / F (interface) 603, and a serial I / F 604 are provided.
- the FPGA that configures the processing hardware unit 510 performs processing that requires real-time processing on image data, such as gamma correction, distortion correction (parallelization of right and left captured images), and parallax calculation using block matching to generate a parallax image. And the like, and performs processing such as writing out in the RAM of the image analysis unit 600.
- the CPU 602 of the image analysis unit 600 is responsible for control of the image sensor controller of each of the imaging units 510a and 510b and overall control of the image processing board, detection processing of the three-dimensional shape of the road surface, guardrails, and other various objects (identification target Load a program to execute object detection processing etc.
- the processing result is data I / F 603 or serial I / F Output from F604 to the outside.
- vehicle operation information such as the vehicle speed, acceleration (acceleration mainly generated in the front and rear direction of the vehicle) of the vehicle 100, steering angle, yaw rate etc. It can also be used as a processing parameter.
- the data output to the outside is used as input data for performing control (brake control, vehicle speed control, warning control, etc.) of various devices of the host vehicle 100.
- the imaging unit 500 and the image analysis unit 600 may be configured as an imaging device 2 that is an integral device.
- FIG. 3 is a functional block diagram of the in-vehicle device control system 1 realized by the processing hardware unit 510, the image analysis unit 600, and the vehicle travel control unit 104 in FIG.
- the functional unit realized by the image analysis unit 600 is realized by processing that one or more programs installed in the image analysis unit 600 causes the CPU 602 of the image analysis unit 600 to execute.
- the parallax image generation unit 11 performs parallax image generation processing for generating parallax image data (parallax image information).
- the parallax image generation unit 11 is configured of, for example, a parallax calculation unit 511 (FIG. 2).
- luminance image data of one of the two imaging units 510a and 510b is used as reference image data, and luminance image data of the other imaging unit 510b is used as comparison image data.
- the disparity between the two is calculated using this to generate and output disparity image data.
- the parallax image data represents a parallax image in which pixel values corresponding to the parallax value d calculated for each image portion on the reference image data are represented as pixel values of the respective image portions.
- the parallax image generation unit 11 defines a block including a plurality of pixels (for example, 16 pixels ⁇ 1 pixel) centering on one target pixel for a row of reference image data.
- a block of the same size as the block of the defined reference image data is shifted by one pixel in the horizontal line direction (x direction) to indicate the feature of the pixel value of the block defined in the reference image data.
- a correlation value indicating a correlation between the feature amount and the feature amount indicating the feature of the pixel value of each block in the comparison image data is calculated.
- a matching process is performed to select a block of comparison image data that is most correlated with the block of reference image data among the blocks of comparison image data. Thereafter, the amount of positional deviation between the target pixel of the block of the reference image data and the corresponding pixel of the block of the comparison image data selected in the matching process is calculated as the parallax value d. It is possible to obtain parallax image data by performing such a process of calculating the parallax value d for the entire area of the reference image data or a specific area.
- the value (brightness value) of each pixel in the block can be used as the feature amount of the block used for the matching process, and the correlation value can be, for example, the value of each pixel in the block of the reference image data (brightness A sum of absolute values of differences between the values) and the values (brightness values) of the respective pixels in the block of the comparison image data respectively corresponding to these pixels can be used. In this case, it can be said that the block with the smallest total sum is most correlated.
- the matching processing in the parallax image generation unit 11 is realized by hardware processing, for example, SSD (Sum of Squared Difference), ZSSD (Zero-mean Sum of Squared Difference), SAD (Sum of Absolute Difference), ZSAD (Sum of Absolute Difference).
- SSD Sud of Squared Difference
- ZSSD Zero-mean Sum of Squared Difference
- SAD Sum of Absolute Difference
- ZSAD Quantum of Absolute Difference
- a method such as Zero-mean Sum of Absolute Difference, NCC (Normalized cross correlation), or the like can be used.
- NCC Normalized cross correlation
- the estimation method for example, an equiangular linear method, a quadratic curve method or the like can be used.
- Road surface estimation unit 12 estimates (detects) the road surface based on the parallax image generated by the parallax image generation unit 11.
- FIG. 4 is a functional block diagram of the road surface estimation unit 12.
- the road surface estimation unit 12 includes a V map generation unit 121, a sample point extraction unit 122, an outlier removal unit 123, a road surface shape detection unit 124, a road surface supplementation unit 125, a smoothing processing unit 126, and a road surface height table calculation unit 127. .
- V map generation process executes V map generation processing for generating a V map (V-Disparity Map, an example of “vertical direction distribution data”) based on parallax pixel data.
- V-Disparity Map an example of “vertical direction distribution data”
- Each piece of parallax pixel data included in parallax image data is indicated by a set (x, y, d) of an x-direction position, a y-direction position, and a parallax value d.
- This is converted into three-dimensional coordinate information (d, y, f) with d set on the X axis, y on the Y axis, and frequency f on the Z axis, or this three-dimensional coordinate information (d, y, f) , And three-dimensional coordinate information (d, y, f) limited to information exceeding a predetermined frequency threshold is generated as disparity histogram information.
- the parallax histogram information of the present embodiment is composed of three-dimensional coordinate information (d, y, f), and a distribution of this three-dimensional histogram information in an XY two-dimensional coordinate system is called a V map.
- the V map generation unit 121 calculates a disparity value frequency distribution for each row region obtained by dividing the disparity image in the vertical direction.
- Information indicating this disparity value frequency distribution is disparity histogram information.
- FIG. 5A and 5B are diagrams for explaining parallax image data and a V map generated from the parallax image data.
- FIG. 5A is a view showing an example of the parallax value distribution of the parallax image
- FIG. 5B is a view showing a V map showing the parallax value frequency distribution for each row of the parallax image of FIG. 5A.
- the V map generation unit 121 calculates a parallax value frequency distribution that is a distribution of the number of data of each parallax value for each row, This is output as disparity histogram information.
- a V map as shown in FIG. 5B can be obtained.
- This V map can also be expressed as an image in which pixels having pixel values according to the frequency f are distributed on the two-dimensional orthogonal coordinate system.
- FIGS. 6A and 6B are diagrams showing an image example of a captured image as a reference image captured by one imaging unit and a V map corresponding to the captured image.
- FIG. 6A is a captured image
- FIG. 6B is a V map. That is, the V map shown in FIG. 6B is generated from the photographed image as shown in FIG. 6A.
- the V map since parallax is not detected in the area below the road surface, parallax is not counted in the hatched area A.
- working the preceding vehicle 402 which exists in front of the own vehicle, and the telephone pole 403 which exists out of the road are shown.
- the V map shown in FIG. 6B there are a road surface 501, a leading vehicle 502, and a telephone pole 503 corresponding to the image example.
- the front road surface of the host vehicle is a relatively flat road surface, that is, the front road surface of the host vehicle is obtained by extending the plane parallel to the road surface portion directly below the host vehicle to the front of the host vehicle. It is a case where it corresponds to a reference road surface (virtual reference movement surface).
- the high frequency points are distributed in a substantially straight line with an inclination such that the parallax value d becomes smaller toward the upper side of the image. Pixels showing such a distribution are present at approximately the same distance in each row on the parallax image and have the highest occupancy rate, and pixels that project an object of identification whose distance increases continuously as they move upward in the image It can be said that
- the parallax value d of the road surface decreases toward the upper side of the image as shown in FIG. 6A.
- the high-frequency points distributed in a substantially straight line on the V map correspond to the features possessed by the pixels projecting the road surface (moving surface). Therefore, it is possible to estimate that the pixels of the point distributed on the approximate straight line obtained by straight line approximation of the high frequency points on the V map or in the vicinity thereof are the pixels showing the road surface with high accuracy. Further, the distance to the road surface portion shown in each pixel can be determined with high accuracy from the parallax value d of the corresponding point on the approximate straight line.
- the V map generation unit 121 may generate a V map using only pixels of a predetermined area (for example, an area where a road surface can be displayed) in a parallax image, or may use all pixels in the parallax image to generate a V map. May be generated.
- FIGS. 7A and 7B are diagrams showing an image example of a captured image as a reference image captured by one of the imaging units when noise is generated, and a V map corresponding to the captured image.
- FIGS. 7A and 7B a reflection 404 due to a puddle or the like is shown in FIG. 7A in comparison with the examples of FIGS. 6A and 6B, respectively. Further, in the V map shown in FIG. 7B, there is a reflection 504 due to a puddle or the like. As shown in FIGS. 7A and 7B, when reflection occurs, even if the parallax of the same road surface is incorrect, an incorrect matching or the like occurs when the parallax is calculated, and a value different from the original parallax is calculated. In this case, the parallax of the road surface is dispersed in the lateral direction as indicated by 504 in FIG. 7B.
- Sample point extraction processing The sample point extraction unit 122 extracts sample points used for estimation of the road surface from the V map generated by the V map generation unit 121.
- the V map is divided into a plurality of segments according to a parallax value (distance value from the host vehicle) and sample point extraction processing, road surface shape detection processing, etc. described later are performed.
- the sample point extraction process or the road surface shape detection process may be performed without dividing the V map.
- FIG. 8 is a flowchart showing an example of sample point extraction processing.
- the sample point extraction unit 122 divides the V map into a plurality of segments according to the value of disparity that is the horizontal axis of the V map (step S1).
- the sample point extraction unit 122 sets a range (search range) for searching for sample points to be extracted in the segment (first segment) having the largest parallax value (the shortest distance from the host vehicle) Step S2).
- FIG. 9 is a diagram for explaining an example of processing for extracting sample points from the first segment.
- the sample point extraction unit 122 sets a start point position 552 of a straight line corresponding to the default road surface 551 (“preset road surface”) as a reference point, and sets a predetermined range corresponding to the reference point. It is set as a search area 553 of sample points.
- the data of a default road surface are preset, for example, when a stereo camera is attached to the own vehicle. For example, flat road surface data may be set.
- the sample point extraction unit 122 searches two straight lines 554 and 555 starting from the reference point, and searches between the two straight lines 554 and 555 extending to the adjacent segment at a predetermined angle. It may be In this case, the sample point extraction unit 122 may determine the two straight lines 554 and 555 in accordance with the angle at which the road surface can tilt. For example, the lower straight line 554 is determined according to the inclination of the descending road surface which can be photographed by a stereo camera, and the upper straight line 555 is a preset upper limit of the ascending road inclination restricted by law. It may be determined correspondingly. Alternatively, the sample point extraction unit 122 may set the search area 553 to, for example, a rectangular area.
- the sample point extraction unit 122 extracts sample points from the pixels included in the search range (step S3).
- the sample point extraction unit 122 may extract, for example, one or more sample points for each coordinate position of each parallax value d from the pixels included in the search range.
- the sample point extraction unit 122 may extract, as a sample point, the most frequent point with the highest frequency at the coordinate position of each disparity value d from the pixels included in the search range.
- the sample point extraction unit 122 may set a plurality of coordinate positions including each parallax value d (for example, coordinate positions of each parallax value d and the left and right of the respective parallax values d) from the pixels included in the search range.
- the most frequent mode point in at least one of one or more coordinate positions may be extracted as a sampling point.
- step S4 the outlier removal unit 123, the road surface shape detection unit 124, and the road surface supplementation unit 125 detect the road surface in the segment including the sample points based on the sample points extracted in step S3 (step S4).
- the sample point extraction unit 122 determines whether there is a next segment (for example, a segment having a next smallest parallax value) (step S5), and if there is no next segment (NO in step S5), the process ends. .
- step S5 If there is the next segment (YES in step S5), the sample point extraction unit 122 acquires the end point position of the road surface detected in step S4 (step S6).
- the sample point extraction unit 122 sets a search range based on the end point position of the road surface acquired in step S5 (step S7), and the process proceeds to step S3.
- FIG. 10A and FIG. 10B are diagrams for explaining an example of processing for extracting a sample point from a second segment other than the first segment.
- the sample point extraction unit 122 for example, in the second segment, the end point position (the road surface 561 of the previous segment) of the road surface 561 of the previous segment (the segment adjacent to the second segment having already detected the road surface) A position in contact with the segment 2) is taken as a reference point 562. Then, as in the case of step S2 shown in FIG. 9, the sample point extraction unit 122 sets a predetermined range corresponding to the reference point 562 as a search area 563 for sample points.
- the sample point extraction unit 122 searches the area between the two straight lines 564 and 565 starting from the reference point and between the two straight lines 564 and 565 extending to the adjacent segment at a predetermined angle. It may be In this case, the sample point extraction unit 122 may determine the two straight lines 564 and 565 in accordance with the angle at which the road surface can tilt. Alternatively, the sample point extraction unit 122 may set the search area 563 to a rectangular area having a height corresponding to the y coordinate of the reference point 562, as shown in FIG. 10B, for example.
- FIG. 11 is a diagram for explaining another example of the process of extracting sample points from the second segment.
- the sample point extraction unit 122 sets, for example, the end point position of the road surface (history road surface) 561a of the segment adjacent to the second segment detected in the frame of the previous parallax image in the second segment as a reference point 562a. . Then, the sample point extraction unit 122 sets a predetermined range corresponding to the reference point 562a as a search area 563a of sample points.
- the difference between the default road surface and the correct (actual) road surface 566 is large, for example, in the case of a downhill road surface, it is detected in the first segment as in FIGS. 10A and 10B.
- a more appropriate search range can be set as compared with the case where the end point position of the road surface is used as the reference point.
- the history road surface may use the road surface detected in the previous one frame, or may use the average of the road surfaces detected in the plurality of previous frames.
- the outlier removal unit 123 excludes points not suitable for linear approximation.
- FIG. 12 is a flowchart illustrating an example of the outlier removal process.
- FIG. 13 is a diagram for explaining outlier removal processing.
- the outlier removal unit 123 calculates an approximate straight line from the sample points of each segment extracted by the sample point extraction unit 122 or the sample points of all the segments (step S20).
- the outlier removal unit 123 calculates an approximate straight line using, for example, the least squares method.
- the approximate straight line 541 is calculated in step S20 of FIG.
- the outlier removal unit 123 calculates a threshold according to the value of the X coordinate (step S21).
- a value D of a predetermined X coordinate for example, a parallax value corresponding to a distance of 50 m from the host vehicle
- D further from the host distance than the host vehicle
- the above is taken as ⁇ . This is to loose the threshold for removal on a road surface at a position where the parallax value is small, that is, the distance from the host vehicle is long, because the error when measuring with the stereo camera is large.
- the outlier removal unit 123 removes a sample point at which the Euclidean distance is greater than or equal to the threshold value calculated in step S21 with respect to the calculated approximate straight line (step S22).
- the sample points 542 which are D or more and which are separated by a predetermined threshold value ⁇ or more are removed.
- sample points 542 which are equal to or more than D and separated by a predetermined threshold value ⁇ or more are removed, but the present invention is not limited thereto.
- ⁇ may be set at two or more locations, or ⁇ may be calculated and set as a function of the parallax d.
- the road surface shape detection unit 124 extracts the shape of the road surface based on the sample points extracted by the sample point extraction unit 122 from the segments of the V map generated by the V map generation unit 121 and not removed by the outlier removal unit 123. Detect (Position, Height).
- the road surface shape detection unit 124 calculates an approximate straight line from sample points of each segment by, for example, the least squares method, and detects (estimates) the calculated approximate straight line as a road surface.
- the road surface supplement unit 125 determines whether the road surface detected (estimated) by the road surface shape detection unit 124 is inappropriate or not, and supplements the road surface if it is determined to be inappropriate. It should be noted that the determination as to whether it is inappropriate or not depends on the determination based on the angle of the road surface, for example, when the number of points used for estimation is smaller than a predetermined threshold or the correlation coefficient at the time of approximation by the least square method is a predetermined value It may be determined as inappropriate if it is smaller than that (if the variation of the point group is large).
- the road surface supplement unit 125 determines whether an inappropriate road surface that can not be photographed by the stereo camera has been estimated by the noise. Then, when it is determined that the inappropriate road surface has been estimated, the road surface supplementing unit 125 supplements (interpolates) the inappropriate road surface based on the data of the default road surface or the road surface estimated in the previous frame.
- the road surface supplement unit 125 detects the slope as the distance from the host vehicle increases. It is determined that the road surface is a steep downhill road. Then, the road surface supplementing part 125 removes the data of the road surface and supplements it with data of the default road surface etc. instead.
- the smoothing processing unit 126 corrects each road surface estimated in each segment so that each road surface is continuous.
- the smoothing processing unit 126 sets the inclination and intercept of each road surface so that the end point (end point) of one road surface and the start point (end point) of the other road surface of the road surfaces respectively estimated in two adjacent segments coincide with each other.
- Road surface height table calculation process The road surface height table calculation unit 127 calculates the road surface height (relative height with respect to the road surface portion immediately below the host vehicle) based on the road surface in each segment corrected by the smoothing processing unit 126, and generates a table. Road surface height table calculation processing is performed.
- the road surface height table calculation unit 127 calculates the distance to each road surface portion shown in each row area (each position in the vertical direction of the image) on the captured image from the information of the road surface in each segment. In addition, each surface portion in the traveling direction of the subject vehicle of the virtual plane extending forward to the traveling direction of the subject vehicle so that the road surface portion located directly below the subject vehicle is parallel to the plane is shown in any row region in the captured image.
- the virtual plane reference road surface
- the road surface height table calculation unit 127 can obtain the height of each road surface portion ahead of the host vehicle by comparing the road surface in each segment with the reference straight line.
- the road surface height table calculation unit 127 tabulates the height of each road surface portion obtained from the approximate straight line with respect to a necessary parallax range.
- the height from the road surface of the object shown in the captured image portion corresponding to the point where the Y-axis position is y ′ at a certain parallax value d is the Y-axis position on the road surface at the parallax value d as y0. It can be calculated from (y'-y0).
- the height H from the road surface for the object corresponding to the coordinates (d, y ′) on the V map can be calculated by the following equation.
- “f” is the focal length of the camera (y′ ⁇ y0) It is a value converted to the same unit as the unit.
- BF is a value obtained by multiplying the base length of the stereo camera and the focal length
- “offset” is a parallax value when an object at infinity is photographed.
- the clustering unit 13 sets a pair (x, y, d) of an x-direction position, a y-direction position, and a parallax value d in each parallax pixel data included in parallax image data to the x axis along the x axis and d and z along the y axis.
- the frequency is set on the axis, and XY two-dimensional histogram information (frequency U map) is created.
- the clustering unit 13 generates parallax images for which the height H from the road surface is in a predetermined height range (for example, 20 cm to 3 m) based on the heights of the road surface portions tabulated by the road surface height table calculation unit 127. Create a frequency U map only for points (x, y, d). In this case, an object present in the predetermined height range can be appropriately extracted from the road surface.
- a predetermined height range for example, 20 cm to 3 m
- the clustering unit 13 detects, in the frequency U map, an area where the frequency is higher than a predetermined value and in which the parallax is dense as the area of the object, and based on the coordinates on the parallax image and the actual size (size) of the object. It provides individual information such as the predicted type of object (person or pedestrian).
- the rejection unit 14 rejects the information of the object that is not the recognition target based on the parallax image, the frequency U map, and the individual information of the object.
- the tracking unit 15 determines whether or not the object to be tracked is a tracking object when the detected object appears continuously in frames of a plurality of parallax images.
- Driving support control The control unit 139 performs driving support control such as, for example, notifying a driver of the host vehicle 100 of a warning or controlling a steering wheel or a brake of the host vehicle based on the detection result of the object by the clustering unit 13. Do.
- a search range corresponding to a predetermined reference point is set, a plurality of pixels are extracted from the search range, and the road surface is detected based on the plurality of extracted pixels. .
- a distance image may be generated by integrating distance information generated using a detection device such as a millimeter wave radar or laser radar with respect to parallax images generated using a stereo camera.
- the detection accuracy may be further enhanced by using a stereo camera and a detection device such as a millimeter wave radar or a laser radar in combination and combining it with the above-described detection result of an object by the stereo camera.
- the functional units may not be provided.
- each functional unit of the processing hardware unit 510, the image analysis unit 600, and the vehicle travel control unit 104 may be realized by hardware, or the CPU executes a program stored in the storage device.
- the configuration may be realized by This program may be recorded in a computer readable recording medium and distributed by a file in an installable format or an executable format.
- a CD-R Compact Disc Recordable
- DVD Digital Versatile Disk
- Blu-ray Disc etc.
- recording media such as CD-ROM in which each program is stored, and HD 504 in which these programs are stored can be provided domestically or abroad as a program product (Program Product).
- In-vehicle device control system (an example of "device control system") 100 self-vehicle 101 imaging unit 103 display monitor 106 vehicle travel control unit (an example of “control unit”) 11 Parallax image generator (an example of “distance image generator”) 12 Road surface estimation unit 121 V map generation unit (an example of “generation unit”) 122 sample point extraction unit (an example of “extraction unit”) 123 Outlier removal part 124 Road surface shape detection part (an example of "detection part”) 125 Road surface supplementing part 126 Smoothing processing part 127 Road surface height table calculation part 13 Clustering part (an example of "object detection part”) 14 rejection unit 15 tracking unit 16 control unit 2 imaging device 510a, 510b imaging unit 510 processing hardware unit 600 image analysis unit (an example of "image processing apparatus”)
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Mechanical Engineering (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
- Traffic Control Systems (AREA)
- Image Processing (AREA)
Abstract
Description
図1は、本発明の実施形態に係る移動体機器制御システムとしての車載機器制御システムの構成を示す図である。
図2は、撮像ユニット500及び画像解析ユニット600の構成を示す図である。
視差画像生成部11は、視差画像データ(視差画像情報)を生成する視差画像生成処理を行う。なお、視差画像生成部11は、例えば視差演算部511(図2)によって構成される。
路面推定部12は、視差画像生成部11により生成された視差画像に基づき、路面を推定(検出)する。
Vマップ生成部121は、視差画素データに基づき、Vマップ(V-Disparity Map、「垂直方向分布データ」の一例)を生成するVマップ生成処理を実行する。視差画像データに含まれる各視差画素データは、x方向位置とy方向位置と視差値dとの組(x,y,d)で示される。これを、X軸にd、Y軸にy、Z軸に頻度fを設定した三次元座標情報(d,y,f)に変換したもの、又はこの三次元座標情報(d,y,f)から所定の頻度閾値を超える情報に限定した三次元座標情報(d,y,f)を、視差ヒストグラム情報として生成する。本実施形態の視差ヒストグラム情報は、三次元座標情報(d,y,f)からなり、この三次元ヒストグラム情報をX-Yの2次元座標系に分布させたものを、Vマップと呼ぶ。
標本点抽出部122は、Vマップ生成部121により生成されたVマップから、路面の推定に用いる標本点を抽出する。
図11は、第2のセグメントから、標本点を抽出する処理の他の例を説明する図である。
外れ点除去部123は、標本点抽出部122により抽出された標本点のうち、直線近似に適さない点を除外する。
路面形状検出部124は、Vマップ生成部121により生成されたVマップの各セグメントから、標本点抽出部122により抽出され、外れ点除去部123により除去されていない標本点に基づき、路面の形状(位置、高さ)を検出する。
路面補足部125は、路面形状検出部124により検出(推定)された路面が不適切か否かを判定し、不適切と判定した場合は、路面を補足する。なお、不適切か否かの判定は、路面の角度での判定によるほか、例えば、推定に使用した点数が所定の閾値より少ない場合や、最小二乗法で近似する際の相関係数が所定値よりも小さい場合(点群のばらつきが大きい場合)に不適切と判定してもよい。
スムージング処理部126は、各セグメントで推定された各路面を、当該各路面が連続するように修正する。スムージング処理部126は、隣り合う2つのセグメントにおいてそれぞれ推定された各路面のうち、一方の路面の終点(端点)と、他方の路面の始点(端点)が一致するよう、各路面の傾きと切片を変更する。
路面高さテーブル算出部127は、スムージング処理部126にて修正された各セグメントにおける路面に基づいて、路面高さ(自車両の真下の路面部分に対する相対的な高さ)を算出してテーブル化する路面高さテーブル算出処理を行う。
〈クラスタリング、棄却、トラッキング〉
クラスタリング部13は、視差画像データに含まれる各視差画素データにおけるx方向位置とy方向位置と視差値dとの組(x,y,d)を、X軸にx、Y軸にd、Z軸に頻度を設定し、X-Yの2次元ヒストグラム情報(頻度Uマップ)を作成する。
制御部139は、クラスタリング部13による、物体の検出結果に基づいて、例えば、自車両100の運転者へ警告を報知したり、自車両のハンドルやブレーキを制御したりするなどの走行支援制御を行う。
雨天時の路面の照り返し等により、視差にノイズが発生すると、推定される路面が正解に対して低すぎたり、逆に高すぎたりする問題が発生する。推定される路面が低すぎる場合、路面の一部を障害物と誤認識する場合がある。推定される路面が高すぎる場合、推定される路面よりも低い障害物等を検出できない場合がある。
100 自車両
101 撮像ユニット
103 表示モニタ
106 車両走行制御ユニット(「制御部」の一例)
11 視差画像生成部(「距離画像生成部」の一例)
12 路面推定部
121 Vマップ生成部(「生成部」の一例)
122 標本点抽出部(「抽出部」の一例)
123 外れ点除去部
124 路面形状検出部(「検出部」の一例)
125 路面補足部
126 スムージング処理部
127 路面高さテーブル算出部
13 クラスタリング部(「物体検出部」の一例)
14 棄却部
15 トラッキング部
16 制御部
2 撮像装置
510a,510b 撮像部
510 処理ハードウェア部
600 画像解析ユニット(「画像処理装置」の一例)
Claims (8)
- 複数の撮像部で各々撮影された複数の撮影画像における路面の距離に応じた距離値を有する距離画像から、前記距離画像の垂直方向に対する距離値の頻度の分布を示す垂直方向分布データを生成する生成部と、
前記垂直方向分布データにおいて、所定の基準点に応じた探索範囲を設定し、当該探索範囲から複数の画素を抽出する抽出部と、
前記抽出された複数の画素に基づいて、路面を検出する検出部と、
を備えることを特徴とする画像処理装置。 - 前記抽出部は、前記所定の基準点から、路面が傾き得る角度に応じた前記探索範囲を設定する、
ことを特徴とする請求項1記載の画像処理装置。 - 前記抽出部は、前記所定の基準点を、以前に撮影された前記複数の撮影画像に基づいて前記検出部により検出された路面、または予め設定されている路面に応じて決定する、
ことを特徴とする請求項1または2記載の画像処理装置。 - 前記抽出部は、前記垂直方向分布データを、距離値に応じて第1のセグメント、及び第2のセグメントに分割し、前記第2のセグメントにおける前記所定の基準点を、前記第1のセグメントにおいて検出された路面に応じて決定する、
ことを特徴とする請求項1乃至3のいずれか一項に記載の画像処理装置。 - 複数の撮像部と、
前記複数の撮像部で各々撮影された複数の撮影画像から、前記複数の撮影画像における路面の視差に応じた距離値を有する距離画像を生成する距離画像生成部と、
前記距離画像の垂直方向に対する距離値の頻度の分布を示す垂直方向分布データを生成する生成部と、
前記垂直方向分布データにおいて、所定の基準点に応じた探索範囲を設定し、当該探索範囲から複数の画素を抽出する抽出部と、
前記抽出された複数の画素に基づいて、前記路面を検出する検出部と、
を備える撮像装置。 - 移動体に搭載され、前記移動体の前方を撮像する複数の撮像部と、
前記複数の撮像部で各々撮影された複数の撮影画像から、前記複数の撮影画像における路面の視差に応じた距離値を有する距離画像を生成する距離画像生成部と、
前記距離画像の垂直方向に対する距離値の頻度の分布を示す垂直方向分布データを生成する生成部と、
前記垂直方向分布データにおいて、所定の基準点に応じた探索範囲を設定し、当該探索範囲から複数の画素を抽出する抽出部と、
前記抽出された複数の画素に基づいて、路面を検出する検出部と、
前記検出部により検出された路面、及び前記距離画像に基づいて、前記複数の撮影画像における物体を検出する物体検出部と、
前記物体検出部により検出された物体のデータに基づいて、前記移動体の制御を行う制御部と、
を備える移動体機器制御システム。 - コンピュータが、
複数の撮像部で各々撮影された複数の撮影画像における路面の距離に応じた距離値を有する距離画像から、前記距離画像の垂直方向に対する距離値の頻度の分布を示す垂直方向分布データを生成するステップと、
前記垂直方向分布データにおいて、所定の基準点に応じた探索範囲を設定し、当該探索範囲から複数の画素を抽出するステップと、
前記抽出された複数の画素に基づいて、前記路面を検出するステップと、
を実行する、画像処理方法。 - コンピュータに、
複数の撮像部で各々撮影された複数の撮影画像における路面の距離に応じた距離値を有する距離画像から、前記距離画像の垂直方向に対する距離値の頻度の分布を示す垂直方向分布データを生成するステップと、
前記垂直方向分布データにおいて、所定の基準点に応じた探索範囲を設定し、当該探索範囲から複数の画素を抽出するステップと、
前記抽出された複数の画素に基づいて、前記路面を検出するステップと、
を実行させるプログラム。
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2018504047A JP6769477B2 (ja) | 2016-03-10 | 2017-01-24 | 画像処理装置、撮像装置、移動体機器制御システム、画像処理方法、及びプログラム |
EP17762735.3A EP3428902A4 (en) | 2016-03-10 | 2017-01-24 | IMAGE PROCESSING DEVICE, IMAGING DEVICE, MOBILE DEVICE CONTROL SYSTEM, PICTURE PROCESSING PROCESS AND PROGRAM |
US16/122,445 US11691585B2 (en) | 2016-03-10 | 2018-09-05 | Image processing apparatus, imaging device, moving body device control system, image processing method, and program product |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2016047574 | 2016-03-10 | ||
JP2016-047574 | 2016-03-10 |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/122,445 Continuation US11691585B2 (en) | 2016-03-10 | 2018-09-05 | Image processing apparatus, imaging device, moving body device control system, image processing method, and program product |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2017154389A1 true WO2017154389A1 (ja) | 2017-09-14 |
Family
ID=59790244
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2017/002393 WO2017154389A1 (ja) | 2016-03-10 | 2017-01-24 | 画像処理装置、撮像装置、移動体機器制御システム、画像処理方法、及びプログラム |
Country Status (4)
Country | Link |
---|---|
US (1) | US11691585B2 (ja) |
EP (1) | EP3428902A4 (ja) |
JP (1) | JP6769477B2 (ja) |
WO (1) | WO2017154389A1 (ja) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2019114150A (ja) * | 2017-12-25 | 2019-07-11 | 株式会社Subaru | 車外環境認識装置 |
JP2021534481A (ja) * | 2018-08-15 | 2021-12-09 | 広州極飛科技股▲ふん▼有限公司Guangzhou Xaircraft Technology Co., Ltd. | 障害物又は地面の認識及び飛行制御方法、装置、機器及び記憶媒体 |
Families Citing this family (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP6406289B2 (ja) * | 2016-03-14 | 2018-10-17 | オムロン株式会社 | 路面形状測定装置、測定方法、及び、プログラム |
JP6743882B2 (ja) * | 2016-03-14 | 2020-08-19 | 株式会社リコー | 画像処理装置、機器制御システム、撮像装置、画像処理方法及びプログラム |
JP6960827B2 (ja) * | 2017-11-08 | 2021-11-05 | フォルシアクラリオン・エレクトロニクス株式会社 | 路面領域検出装置 |
CN109191512B (zh) * | 2018-07-27 | 2020-10-30 | 深圳市商汤科技有限公司 | 双目图像的深度估计方法及装置、设备、程序及介质 |
JP2022040651A (ja) * | 2020-08-31 | 2022-03-11 | 株式会社Subaru | 車外環境認識装置 |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2015057690A (ja) * | 2013-08-15 | 2015-03-26 | 株式会社リコー | 画像処理装置、認識対象物検出方法、認識対象物検出プログラム、および、移動体制御システム |
JP2015195018A (ja) * | 2014-03-18 | 2015-11-05 | 株式会社リコー | 画像処理装置、画像処理方法、運転支援システム、プログラム |
Family Cites Families (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP4719543B2 (ja) | 2005-09-26 | 2011-07-06 | 株式会社リコー | ワークフローシステム、サーバ装置、ワークフローシステムの処理方法及びワークフロープログラム |
JP2007102545A (ja) | 2005-10-05 | 2007-04-19 | Ricoh Co Ltd | 電子文書作成装置、電子文書作成方法及び電子文書作成プログラム |
JP4647515B2 (ja) | 2006-02-20 | 2011-03-09 | 株式会社リコー | 座標検出装置、筆記具および座標入力システム |
US7873215B2 (en) * | 2007-06-27 | 2011-01-18 | Seiko Epson Corporation | Precise identification of text pixels from scanned document images |
CN103177236B (zh) * | 2011-12-22 | 2016-06-01 | 株式会社理光 | 道路区域检测方法和装置、分道线检测方法和装置 |
CN103854008B (zh) * | 2012-12-04 | 2019-10-18 | 株式会社理光 | 路面检测方法和装置 |
JP6274557B2 (ja) * | 2013-02-18 | 2018-02-07 | 株式会社リコー | 移動面情報検出装置、及びこれを用いた移動体機器制御システム並びに移動面情報検出用プログラム |
JP6151150B2 (ja) | 2013-10-07 | 2017-06-21 | 日立オートモティブシステムズ株式会社 | 物体検出装置及びそれを用いた車両 |
JP5874756B2 (ja) * | 2014-02-07 | 2016-03-02 | トヨタ自動車株式会社 | 区画線検出システム及び区画線検出方法 |
JP6519262B2 (ja) * | 2014-04-10 | 2019-05-29 | 株式会社リコー | 立体物検出装置、立体物検出方法、立体物検出プログラム、及び移動体機器制御システム |
JP6550881B2 (ja) | 2014-07-14 | 2019-07-31 | 株式会社リコー | 立体物検出装置、立体物検出方法、立体物検出プログラム、及び移動体機器制御システム |
US20160019429A1 (en) | 2014-07-17 | 2016-01-21 | Tomoko Ishigaki | Image processing apparatus, solid object detection method, solid object detection program, and moving object control system |
JP6440411B2 (ja) * | 2014-08-26 | 2018-12-19 | 日立オートモティブシステムズ株式会社 | 物体検出装置 |
JP6805534B2 (ja) * | 2015-07-02 | 2020-12-23 | 株式会社リコー | 視差画像生成装置、視差画像生成方法及び視差画像生成プログラム、物体認識装置、機器制御システム |
WO2017090326A1 (ja) | 2015-11-27 | 2017-06-01 | 株式会社リコー | 画像処理装置、撮像装置、機器制御システム、分布データ生成方法、及びプログラム |
JP6597792B2 (ja) * | 2015-11-30 | 2019-10-30 | 株式会社リコー | 画像処理装置、物体認識装置、機器制御システム、画像処理方法およびプログラム |
EP3176013B1 (en) * | 2015-12-01 | 2019-07-17 | Honda Research Institute Europe GmbH | Predictive suspension control for a vehicle using a stereo camera sensor |
-
2017
- 2017-01-24 WO PCT/JP2017/002393 patent/WO2017154389A1/ja active Application Filing
- 2017-01-24 JP JP2018504047A patent/JP6769477B2/ja active Active
- 2017-01-24 EP EP17762735.3A patent/EP3428902A4/en active Pending
-
2018
- 2018-09-05 US US16/122,445 patent/US11691585B2/en active Active
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2015057690A (ja) * | 2013-08-15 | 2015-03-26 | 株式会社リコー | 画像処理装置、認識対象物検出方法、認識対象物検出プログラム、および、移動体制御システム |
JP2015195018A (ja) * | 2014-03-18 | 2015-11-05 | 株式会社リコー | 画像処理装置、画像処理方法、運転支援システム、プログラム |
Non-Patent Citations (1)
Title |
---|
See also references of EP3428902A4 * |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2019114150A (ja) * | 2017-12-25 | 2019-07-11 | 株式会社Subaru | 車外環境認識装置 |
US10810757B2 (en) | 2017-12-25 | 2020-10-20 | Subaru Corporation | Vehicle exterior environment recognition apparatus |
JP2021534481A (ja) * | 2018-08-15 | 2021-12-09 | 広州極飛科技股▲ふん▼有限公司Guangzhou Xaircraft Technology Co., Ltd. | 障害物又は地面の認識及び飛行制御方法、装置、機器及び記憶媒体 |
Also Published As
Publication number | Publication date |
---|---|
EP3428902A4 (en) | 2019-08-14 |
US11691585B2 (en) | 2023-07-04 |
US20190001910A1 (en) | 2019-01-03 |
EP3428902A1 (en) | 2019-01-16 |
JPWO2017154389A1 (ja) | 2019-02-28 |
JP6769477B2 (ja) | 2020-10-14 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10580155B2 (en) | Image processing apparatus, imaging device, device control system, frequency distribution image generation method, and recording medium | |
JP6733225B2 (ja) | 画像処理装置、撮像装置、移動体機器制御システム、画像処理方法、及びプログラム | |
US10755116B2 (en) | Image processing apparatus, imaging apparatus, and device control system | |
WO2017154389A1 (ja) | 画像処理装置、撮像装置、移動体機器制御システム、画像処理方法、及びプログラム | |
JP6705496B2 (ja) | 画像処理装置、撮像装置、移動体機器制御システム、移動体、画像処理方法、及びプログラム | |
JP6816401B2 (ja) | 画像処理装置、撮像装置、移動体機器制御システム、画像処理方法、及びプログラム | |
JP7206583B2 (ja) | 情報処理装置、撮像装置、機器制御システム、移動体、情報処理方法およびプログラム | |
JP6583527B2 (ja) | 画像処理装置、撮像装置、移動体機器制御システム、画像処理方法、及びプログラム | |
JP6711395B2 (ja) | 画像処理装置、撮像装置、移動体機器制御システム、移動体、画像処理方法、及びプログラム | |
WO2017145634A1 (ja) | 画像処理装置、撮像装置、移動体機器制御システム、画像処理方法、及びプログラム | |
US10628960B2 (en) | Information processing apparatus, imaging apparatus, device control system, moving object, information processing method, and recording medium | |
JP6733302B2 (ja) | 画像処理装置、撮像装置、移動体機器制御システム、画像処理方法、及びプログラム | |
JP6763198B2 (ja) | 画像処理装置、撮像装置、移動体機器制御システム、画像処理方法、及びプログラム | |
JP6711128B2 (ja) | 画像処理装置、撮像装置、移動体機器制御システム、画像処理方法、及びプログラム | |
JP2018088234A (ja) | 情報処理装置、撮像装置、機器制御システム、移動体、情報処理方法、及びプログラム | |
JP6812701B2 (ja) | 画像処理装置、移動体機器制御システム、画像処理方法、及びプログラム | |
US20240212194A1 (en) | Vehicle and control method thereof |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 2018504047 Country of ref document: JP |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2017762735 Country of ref document: EP |
|
ENP | Entry into the national phase |
Ref document number: 2017762735 Country of ref document: EP Effective date: 20181010 |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 17762735 Country of ref document: EP Kind code of ref document: A1 |