WO2020017334A1 - Vehicle-mounted environment recognition device - Google Patents

Vehicle-mounted environment recognition device Download PDF

Info

Publication number
WO2020017334A1
WO2020017334A1 PCT/JP2019/026561 JP2019026561W WO2020017334A1 WO 2020017334 A1 WO2020017334 A1 WO 2020017334A1 JP 2019026561 W JP2019026561 W JP 2019026561W WO 2020017334 A1 WO2020017334 A1 WO 2020017334A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
camera
viewpoint
conversion
viewpoint conversion
Prior art date
Application number
PCT/JP2019/026561
Other languages
French (fr)
Japanese (ja)
Inventor
雅幸 竹村
彰二 村松
Original Assignee
日立オートモティブシステムズ株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 日立オートモティブシステムズ株式会社 filed Critical 日立オートモティブシステムズ株式会社
Priority to CN201980046818.8A priority Critical patent/CN112424565B/en
Publication of WO2020017334A1 publication Critical patent/WO2020017334A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C3/00—Measuring distances in line of sight; Optical rangefinders
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C3/00—Measuring distances in line of sight; Optical rangefinders
    • G01C3/02—Details
    • G01C3/06—Use of electric means to obtain final indication
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras

Definitions

  • the present invention relates to an in-vehicle environment recognition device that recognizes a surrounding environment of a vehicle by a camera installed in the vehicle.
  • a method of estimating the position and orientation between two cameras installed in a vehicle and performing geometric calibration so that the images of the two cameras have a parallel positional relationship is generally used as information for obtaining the geometric relationship between the positions and postures of the two cameras.
  • the method of acquiring the corresponding point is to extract a unique point such as a corner (corner) on the image which is called a feature point from the left and right images, calculate a feature amount from a luminance change around the feature point, and calculate the feature amount.
  • Feature points having similar amounts are searched for on the left and right images, and feature points found are set as corresponding points, and geometric calibration is performed based on the coordinates of the corresponding points in the left and right images.
  • Patent Literature 1 calculates a distance to an object and estimates a three-dimensional position of the object based on outputs of left and right cameras arranged such that their fields of view overlap.
  • a calibration device for an in-vehicle stereo camera including an object recognition and feature point collection unit for recognizing and acquiring feature point coordinates of an object, and an inter-camera parameter estimation unit for obtaining an inter-camera parameter based on the feature point coordinates is disclosed. Have been.
  • the scenery or object between the two cameras is close to the camera or when the two cameras are far apart, the same object is observed from a greatly different viewpoint, and the two images are used. Of the same object may be greatly different.
  • the feature amount calculated based on the luminance change around the point between the two cameras Likely to be different. That is, since the feature values of the two feature points are calculated as different values, no corresponding point is found, an incorrect corresponding point (erroneous corresponding point) is found, or even if a corresponding point is found, the number is very small. And the like. The smaller the number of corresponding points or the more erroneous corresponding points, the lower the accuracy of the geometric calibration of the camera. Further, there may be a case where the position and orientation between the two cameras cannot be estimated by the convergence calculation.
  • An object of the present invention is to provide an in-vehicle environment recognizing device that can easily perform a geometric calibration of a camera even when an image captured by a plurality of cameras includes the same object having a significantly different appearance.
  • the present application includes a plurality of means for solving the above-mentioned problems.
  • a first camera and a second camera a first image captured by the first camera, and an image captured by the second camera are provided.
  • viewpoint conversion for converting the first image and the second image to an image from a common viewpoint by deforming at least one of the second images
  • a plurality of corresponding points are extracted, and the plurality of corresponding points are extracted before the viewpoint conversion.
  • dense corresponding points are extracted on an image by performing viewpoint conversion in a common visual field region of a plurality of cameras installed in a vehicle, and geometric calibration is performed based on the corresponding points.
  • viewpoint conversion in a common visual field region of a plurality of cameras installed in a vehicle
  • geometric calibration is performed based on the corresponding points.
  • the position and orientation between the cameras can be estimated with high accuracy, and highly accurate parallelization of the two cameras can be realized.
  • stereo matching in a state where high-precision parallelization is performed, generation of a high-density parallax image is realized, and high-precision distance restoration is enabled from the parallax.
  • FIG. 1 is a configuration diagram of an in-vehicle environment recognition device according to a first embodiment.
  • FIG. 3 is a functional block diagram of a viewpoint conversion unit.
  • FIG. 3 is a functional block diagram of a conversion parameter generation unit.
  • FIG. 3 is a functional block diagram of a corresponding point search unit.
  • FIG. 3 is a functional block diagram of a camera geometric calibration unit.
  • FIG. 3 is a functional block diagram of a parallax image generation unit (first embodiment).
  • FIG. 9 is a functional block diagram of a parallax image generation unit (second embodiment). Description of the process of calibrating camera geometry from corresponding points. Explanation of the problem of acquiring corresponding points. Explanation of solution by viewpoint conversion. An example of viewpoint conversion of upper and lower area division.
  • viewpoint conversion by shearing is an example of viewpoint conversion in six regions.
  • 3 is a processing flowchart of a control device according to the first embodiment.
  • 4 is an example of viewpoint conversion for free area division.
  • 9 is a flowchart of a parallax image generation process performed by the control device according to the second embodiment.
  • FIG. 1 shows a configuration diagram of an in-vehicle environment recognition device 1 according to the present embodiment.
  • the in-vehicle environment recognizing device 1 includes a left camera (first camera) 100 and a right camera (second camera) 110 arranged at intervals in the left and right directions in the horizontal direction, and imaging output from the two cameras 100 and 110.
  • a process of performing geometric calibration of the two cameras 100 and 110 based on an image (an image obtained by the left camera 100 may be referred to as a first image and an image obtained by the right camera 110 may be referred to as a second image);
  • a control device (computer) 10 that executes processing for creating a parallax image by performing stereo matching on two images captured by the two cameras 100 and 110 at the same timing is provided.
  • the control device (computer) 10 includes an arithmetic processing device (eg, CPU) not shown, a storage device (eg, memory, hard disk, flash memory) for storing a program executed by the arithmetic processing device, and internal devices. And a communication device for performing communication with external devices.
  • the control device 10 functions as the viewpoint conversion unit 200, the corresponding point search unit 300, the camera geometric calibration unit 400, and the parallax image generation unit 500 by executing a program stored in the storage device. Other functions can be implemented by adding a program.
  • a stereo camera consisting of two cameras on the left and right recognizes the environment around the vehicle using a common viewing area. After attaching the left and right cameras to the predetermined position of the support with high accuracy, the camera factory estimates the position and orientation of the left and right cameras while capturing the calibration chart with the left and right cameras, and uses the results to The parameters are corrected so that the captured images are parallel to each other. If stereo matching is performed in this state, it is possible to obtain a high-density parallax and a parallax image, and it is possible to measure a distance with high accuracy from the parallax image.
  • high-precision camera manufacturing and parts and structures that suppress deformation of the camera due to temperature change, shock, vibration, aging, and the like are expensive.
  • the present inventors can easily correct the deformation of the camera due to the positional shift of the camera, the temperature, the aging, etc. in order to suppress the cost of this kind, and realize the high-precision calibration during the traveling.
  • the in-vehicle environment recognition device 1 of the present embodiment has been invented.
  • feature points (unique points such as corners (corners) of an object on the image) are extracted from the images of the left and right cameras 100 and 110, and the feature points are extracted.
  • a feature amount is calculated from a change in surrounding brightness, a feature point having a similar feature amount is searched for on the left and right images, and a set of feature points having similar feature amounts on the left and right images is set as a set of corresponding points.
  • the image of FIG. 8 is an image in which a plurality of corresponding points on the searched left and right images are connected to each other with a line, and geometric calibration of the left and right cameras 100 and 110 is performed based on the plurality of corresponding points.
  • the corresponding point search unit 300 extracts the above-described feature point from the images of the left and right cameras 100 and 110, calculates a feature amount from a luminance change around the feature point, and calculates the feature amount on the left and right images. A feature point (corresponding point) having a similar feature amount is searched.
  • the camera geometric calibration unit 400 calculates geometric calibration parameters for making the left and right images parallel based on the plurality of corresponding points obtained by the corresponding point search unit 300.
  • the image obtained by geometrically calibrating the images of the left and right cameras 100 and 110 is an image having a horizontal positional relationship and no lens distortion at all. This makes it possible to prepare left and right images that are easily geometrically matched.
  • the parallax image generation unit 500 performs stereo matching on two images (parallelized images) captured at the same timing and corrected by the geometric calibration parameters of the camera geometric calibration unit 400, and performs two-dimensional matching on the two images.
  • the distance information (parallax information) indicating the deviation of the position where the same object (the same pattern) is photographed is calculated by a known method to generate a parallax image (distance image).
  • the parallax image generation unit 500 uses left and right images captured by the left camera 100 and the right camera 110. In the present embodiment, since the stereo matching is performed based on the right image (second image) from the right camera 110, the sensitivity, geometry, and the like are basically matched to the right reference.
  • the parallax image generation unit 500 receives the images of the left and right cameras to which the geometric and sensitivity corrections have been applied, performs stereo matching, generates a parallax image, and finally performs noise removal to remove the noise-reduced parallax. An image is obtained.
  • the processing performed by the viewpoint conversion unit 200 is the most characteristic part.
  • the viewpoint conversion unit 200 deforms at least one of the left image (first image) captured by the left camera 100 and the right image (second image) captured by the right camera 110, thereby forming the left and right images (first image and first image).
  • Viewpoint conversion for converting the second image) into an image from a common viewpoint This makes it easier for the corresponding point search unit 300 to find corresponding points from the left and right images.
  • the details of the image deformation method (viewpoint conversion method) will be described later.
  • the image (the left image, the first image) captured by the left camera 100 is made to approach the appearance from the viewpoint of the right camera 110.
  • the whole or a part of the region on the image has a larger number than before the viewpoint transformation.
  • the search result of the dense corresponding point can be obtained by the corresponding point search unit 300.
  • high-precision geometric calibration is realized in the processing of the camera geometric calibration unit 400 performed in the subsequent stage.
  • the parallax image generation unit 500 can generate a high-density and high-precision parallax image.
  • the viewpoint (position on the image) of four points on the road surface in the left image is matched with or similar to the right image. After conversion, search for corresponding points.
  • the corresponding point could hardly be searched for from the road surface, but the corresponding points were densely obtained from the road surface after the deformation as shown in the lowermost part of FIG. Can be confirmed.
  • the viewpoint conversion unit 200 includes an area setting unit 210 that sets an area where at least one of the left and right images is to be subjected to viewpoint conversion and an area that is not to be subjected to viewpoint conversion.
  • a transformation parameter generation unit 220 that generates a matrix or a parameter of a function necessary for transforming the left and right images into an image from a common viewpoint by transforming, and a parameter (conversion parameter) generated by the conversion parameter generation unit 220
  • a viewpoint conversion image generation unit 240 that generates an image (viewpoint conversion image) in which at least one of the left and right images is viewpoint-converted by using a matrix or a function included therein
  • An inverse transformation parameter calculation unit 230 that generates parameters (inverse transformation parameters) of matrices and functions is provided.
  • the viewpoint converted image generated by the viewpoint converted image generation unit 240 is used in the corresponding point search in the corresponding point search unit 300. Further, the inverse transformation parameter calculated by the inverse transformation parameter calculation unit 230 is used by the corresponding point inverse transformation correction unit 410 of the camera geometric calibration unit 400.
  • the region setting unit 210 may set all the regions in the camera image as regions where viewpoint conversion is performed or regions where viewpoint conversion is not performed.
  • a common viewpoint to which the left and right cameras 100 and 110 are converted there is a viewpoint of either one of the left and right cameras 100 and 110 and an arbitrary viewpoint different from any of the left and right cameras 100 and 110.
  • a predetermined viewpoint for example, the middle point between the left and right cameras 100 and 110 located between the left camera 100 and the right camera 110 can be set as a common viewpoint to perform viewpoint conversion.
  • viewpoint conversions a part (overlap area) where the viewpoint conversion unit 200 of the control device 10 is overlapped and captured in the left and right images (the first image and the second image). Is assumed to be a plane, the positions and orientations of the left camera 100 and the right camera 110 with respect to the plane are geometrically calculated based on the left and right images, and the left camera 100 and the right with respect to the calculated plane are calculated. Some cameras convert a left image (first image) into an image from the viewpoint of the right camera 110 based on the position and orientation of the camera 110.
  • this viewpoint conversion deforms the entire left image by focusing only on the road surface which is only a part of the left image
  • the viewpoint conversion is performed by focusing on the shape of a three-dimensional object that does not exist on the road surface (particularly, the building shown in FIG. 10). It can be confirmed that the image is greatly inclined after the conversion.
  • the appearance (shape) of the building before and after the deformation is more similar on the left and right images, and the original image (the original image) is denser at a distance from the cameras 100 and 110. It can be understood that a corresponding point can be obtained.
  • the left image is divided into two types of regions, a region where viewpoint conversion is performed and a region where viewpoint conversion is not performed.
  • the search is performed for the corresponding points by division.
  • the former area in which the viewpoint conversion is performed can be said to be an area in which the number of corresponding points increases when the viewpoint conversion is performed.
  • the latter area in which viewpoint conversion is not performed can be said to be an area in which dense corresponding points can be obtained even in the original image, and corresponds to, for example, an area far from the camera.
  • the left image is vertically divided into two by a horizontal boundary line based on a vanishing point VP in the image, and the upper region (upper region) of the two regions is divided.
  • the region (region) 111 is set as a region in which viewpoint conversion is not performed
  • the lower region (lower region) 112 is set as a region in which viewpoint conversion is performed.
  • the viewpoint conversion unit 200 of the control device 10 includes an upper region (distant view) 111 including a vanishing point VP or having a boundary in contact with the vanishing point VP, and a lower region (below the upper region 111).
  • the left image (first image) is divided vertically into two areas of two road areas 112, and it is assumed that at least a part of the lower area 112 is a plane, and the positions and postures of the left and right cameras 100 and 110 with respect to the plane. Is calculated, and the lower area 112 is converted into an image from the viewpoint of the right camera 110 based on the calculated position and orientation of the left and right cameras 100 and 110 with respect to the plane. (No viewpoint conversion) A combination of 111 and the left image (first image) after viewpoint conversion.
  • This viewpoint conversion is one of the simplest and most effective ones in camera geometric calibration of an in-vehicle camera (in-vehicle environment recognition device).
  • this method uses an image after viewpoint conversion (viewpoint conversion image) with hardware such as an FPGA (Field-Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), and a GPU (Graphics Processing Unit). Assuming, conversion is simplified with the restriction that the image is divided only into rectangles. A distant road surface can also be deformed on the assumption that it is flat, but if it is far away, the amount of deformation of the road surface due to viewpoint conversion is small.
  • FPGA Field-Programmable Gate Array
  • ASIC Application Specific Integrated Circuit
  • GPU Graphics Processing Unit
  • the upper region 111 including the vanishing point VP is set in the region setting unit 210 as a region where the viewpoint is not changed.
  • the lower region 112 is a region where the road surface is imaged, the lower region 112 is set by the region setting unit 210 as a region to be changed in viewpoint.
  • the viewpoint conversion unit 200 performs the viewpoint conversion of the lower region 112 using the conversion parameters generated by the conversion parameter generation unit 220.
  • a simple viewpoint conversion method there is a method in which conversion parameters are generated by the conversion parameter generation unit 220 using design values of the positions and postures of the cameras 100 and 110 mounted on the vehicle.
  • the corrected values may be used. If the posture of the road surface is estimated at the same time as the position and posture of the camera based on the distance information of at least three points on the road surface, the posture information of the road surface may be used.
  • the reason that the design values can be used as the position and orientation of the left and right cameras is that it is important that the left and right images are more similar than before the viewpoint conversion, and the two do not need to be mathematically completely identical. Because. This is because the purpose of the viewpoint conversion is to obtain corresponding points densely in the image after the viewpoint conversion, and it suffices to generate an image similar to the degree that the corresponding points can be obtained. Further, the coordinates of the corresponding point after the viewpoint conversion by the conversion parameter are returned to the coordinates before the viewpoint conversion by the inverse conversion parameter, and are used for the geometric calibration of the camera. Therefore, a high-precision camera position / posture is not always necessary.
  • the viewpoint conversion may be mathematically performed based on the relative positions and postures of the cameras 100 and 110 and the road surface.
  • the main purpose of the viewpoint conversion is not accurate conversion. It is to obtain corresponding points of the left and right images densely. For this reason, even if the conversion is not mathematically accurate, it can be replaced by a method that has a certain effect.
  • the conversion based on the mathematical operation in hardware may be omitted.
  • the conversion parameter generation unit 220 sets the center of the vanishing point VR as the center.
  • the viewpoint transformation may be performed by transforming the image by affine transformation including the rotation of the image and the shear deformation of the road surface.
  • the viewpoint conversion unit 200 of the control device 10 changes the parameters (matrix parameters for affine transformation) to the left image (first image) from the left camera 100 while changing the affine. Conversion is performed to generate a plurality of converted images, and corresponding points of each of the plurality of converted images and the right image (second image) are extracted. There is a process of using a large number of converted images that are equal to or more than a predetermined threshold (reference value) in a final viewpoint converted image to be used in subsequent processing (that is, an image obtained by converting a left image into an image from the viewpoint of the right camera 110).
  • a predetermined threshold reference value
  • affine transformation is performed for all combinations including at least one of scaling, rotation, translation, and shearing of the image while gradually changing parameters.
  • converted images of substantially all patterns and a conversion matrix used for the converted images can be generated.
  • the number of converted images can be adjusted at intervals at which parameters are changed.
  • a search for corresponding points with the right image is performed for all of the generated converted images, and the number of corresponding points obtained is compared, thereby transforming the left image into a shape most similar to the right image. You can get the image and the transformation matrix.
  • the parameters of the transformation matrix thus obtained can be used as transformation parameters.
  • a method of viewpoint transformation using shear deformation as affine transformation will be described with reference to FIG.
  • a corresponding point of each converted image with the right image is searched. It is determined that the shear amount of the converted image in which the number of corresponding points is the largest and which is larger than the reference (predetermined threshold) is used as the conversion parameter.
  • the viewpoint conversion image generation unit 240 When the viewpoint conversion of the lower region 112 is completed, the viewpoint conversion image generation unit 240 combines the lower region 112 after the viewpoint conversion and the upper region 111 that has not been subjected to the viewpoint conversion into a viewpoint conversion image (the left side after the viewpoint conversion). Image).
  • the viewpoint conversion image generated by the viewpoint conversion unit 200 is compared with the right image by the corresponding point search unit 300 to extract a corresponding point.
  • the coordinates of the extracted corresponding points are returned to the coordinates in the original image by the inverse transformation by the camera geometric calibration unit 400 as shown in FIG. 14, and are used for camera geometric calibration.
  • the viewpoint conversion unit 200 performs processing from generation of a viewpoint conversion parameter to generation of a viewpoint conversion image and calculation of an inverse conversion parameter.
  • the inverse transformation parameter is used in the camera geometric calibration unit 400 that integrates and uses feature points.
  • the conversion parameter generation unit 220 includes a parallax analysis unit 221, an attribute determination unit 222, and a conversion parameter calculation unit 223.
  • the parallax analysis unit 221 acquires the parallax of the region set as the region for performing the viewpoint conversion by the region setting unit 210 from the parallax image of the previous frame (for example, one frame before) generated by the parallax image generation unit 500, By analyzing the parallax, it is determined whether or not a portion that can be approximated to a plane exists in the region.
  • the attribute determining unit 222 determines the plane attribute of the area based on the analysis result of the parallax analyzing unit 221.
  • the plane attributes include “plane” indicating that there is a portion that can be approximated to a plane in the area, and “non-plane” indicating that there is no part that can be approximated to the plane in the area.
  • the former “plane” attribute further includes “road surface (ground)” and “wall surface (wall)” as attributes indicating the type of the area.
  • the latter attribute of “non-planar” includes “infinity” as an attribute indicating the type of the area.
  • each region may be given a plane attribute in advance.
  • the attribute determination unit 222 determines whether the plane attribute given in advance is appropriate based on the analysis result of the parallax analysis unit 221, and the distance from the left and right cameras 100 and 110 in the area determined to be appropriate is determined.
  • the plane attribute of the area smaller than the predetermined threshold is determined to be “plane”, and the area is determined as the target area of the viewpoint conversion.
  • the conversion parameter calculation unit 223 calculates a conversion parameter for performing viewpoint conversion on an area whose plane attribute is determined to be a plane by the attribute determination unit 222.
  • the conversion parameter can be calculated by a known method. Here, an example will be described.
  • the image coordinates can be known because the region to be transformed is set by itself. In this way, if the image coordinates of the four corners of the plane in the area to be transformed are input to equation (1), the three-dimensional world coordinates of the four corners can be calculated. Next, the three-dimensional coordinates of the four corners calculated by calculation are sequentially set to Xworld, Yworld, and Zworld at the right end of the right side. When the position and orientation of the right camera as viewed from the origin of the world coordinates are set in the matrix of external parameters, the positions of four points in the world coordinate system can be converted into image coordinates as viewed by the right camera. In this way, the image coordinates of the four corners of the plane in the region to be subjected to the viewpoint conversion are obtained to obtain the quadrangle conversion parameters. If the four corners can be calculated, all the coordinates inside the quadrangle can be calculated by interpolation.
  • the parallax analyzer 221 estimates that the lower region 112 where the road surface is often imaged is a portion that can be approximated to a plane. Whether or not each area includes a portion that can be approximated to a plane can be analyzed using parallax obtained from a stereo image. Assuming that the three-dimensional survey is performed with the left camera 100 as the center, the position of the road surface viewed from the left camera 100 can be analyzed using the parallax.
  • a plane attribute “road surface” is assigned to the lower region 112 in advance, and when the attribute is determined to be valid from the analysis result, the lower region 112 is determined to be a viewpoint conversion target. Conversely, if it is determined that the attribute is not valid, the lower area 112 is excluded from viewpoint conversion. That is, erroneous viewpoint conversion is suppressed by searching for a corresponding point without performing viewpoint conversion.
  • the conversion parameter calculation unit 223 calculates the conversion parameter of the area.
  • the corresponding point search unit 300 searches for a corresponding point between the left image (viewpoint converted image) subjected to viewpoint conversion by the viewpoint conversion unit 200 and the original image of the right image. In addition, when the viewpoint conversion is performed on the left and right images by the viewpoint conversion unit 200, a corresponding point is searched from the left and right images after the viewpoint conversion. As shown in FIG. 4, the corresponding point search unit 300 can function as a feature point extraction unit 310, a feature amount description unit 320, a maximum error setting unit 330, a corresponding point search unit 340, and a reliability calculation unit 350. .
  • the feature point extracting unit 310 extracts a feature point from the left image.
  • a feature point is a unique point such as a corner of an object on an image. Note that the feature point may be extracted from one of the left and right images, and the feature point may be extracted from the right image.
  • the feature value description unit 320 describes a feature value obtained by quantifying a change in the surrounding luminance of the feature point extracted by the feature point extraction unit 310.
  • the search range of the right image may be set to an area expanded by parallax.
  • the maximum error setting unit 330 sets a search range (vertical search range) in the vertical direction of the image in consideration of the maximum vertical error occurrence range of the left and right cameras 100 and 110 before performing the corresponding point search. For example, when obtaining feature points from the left image, the vertical search range is set to the right image. If the parallelization of the left and right images is perfect, the corresponding points of the feature points of the left image may be searched in the horizontal row of the same height in the right image. However, the range of the vertical error differs depending on the temperature characteristics and the assembling accuracy of the components to be used.
  • the vertical maximum error is defined, it is sufficient to perform a corresponding point search for the corresponding point of the feature point extracted from the left image within the range of ⁇ vertical maximum error from the same coordinate in the right image.
  • the vertical search range is reduced by the maximum error setting unit 330, and the horizontal direction is set by the maximum parallax value of the divided region of the left image. Thereby, the feature point candidates to be searched by the corresponding point search unit 340 can be reduced.
  • the corresponding point search unit 340 performs a corresponding point search on the right image while comparing the similarities of the feature amounts calculated for the feature points. Normally, a plurality of candidate corresponding points are found for one feature point. Among them, a candidate point having the highest similarity and having a similarity equal to or more than a predetermined threshold value is set as the corresponding point.
  • the left and right images are divided into an upper region and a lower region, and corresponding points are extracted from the image without viewpoint conversion in the upper region (infinity), and the viewpoint is extracted in the lower region (road surface).
  • the corresponding points are extracted from the converted image. Thereby, compared with the conventional case in which the viewpoint conversion is not performed, it is possible to obtain a large number of dense corresponding points from the upper and lower regions.
  • the reliability calculation unit 350 determines whether or not the area is an area that can be used for the camera geometry at the subsequent stage based on the similarity of the corresponding points obtained by the corresponding point search unit 340 and the number of corresponding points. Calculate the reliability, which is an index value. If the reliability is low, it is determined that the area cannot be used in the camera geometry. The reliability is calculated for all the regions to determine whether the region can be used for camera geometry. If it is determined that the area cannot be used in a number of areas equal to or greater than a certain threshold value, it is determined that calibration (geometric calibration) cannot be performed on the image acquired this time.
  • the camera geometric calibration unit 400 performs geometric calibration of the left and right cameras 100 and 110 based on the plurality of corresponding points obtained by the corresponding point search unit 300 so that the left and right images are parallel.
  • the camera geometric calibration unit 400 includes a corresponding point inverse transformation correction unit 410, a corresponding point aggregation unit 420, a noise corresponding point deletion unit 430, a geometric calibration parameter estimation unit 440, and an availability determination unit. 450, and may function as the geometric calibration reflection unit 460.
  • the corresponding point inverse transformation correction unit 410 performs a calculation to return the coordinates of the corresponding point obtained by using the viewpoint change or the image deformation to the coordinate system on the original image.
  • an inverse transformation parameter is used to return to the coordinate system of the original image.
  • the parameters of the inverse transformation have already been obtained by the inverse transformation parameter calculation unit 230, and the coordinates of the corresponding points of the viewpoint-transformed image (left image) of the left and right images are converted to the coordinates of the original image using the inverse transformation parameters.
  • Perform inverse conversion FIG. 14 shows an inverse conversion method when the viewpoint conversion is performed on the lower area of the left image.
  • the viewpoint is changed by modifying only the lower region of the left image as shown in the interruption image among the three stages.
  • a corresponding point search is performed in both the upper region and the lower region.
  • inverse transformation coordinate transformation
  • inverse transformation is performed to return the coordinates of the corresponding point (feature point) to the coordinates before the viewpoint transformation, and the position of the corresponding point in the image before transformation is calculated.
  • a number of corresponding points found based on the image after the viewpoint conversion are inversely transformed into coordinates on the original image and then used for geometric calibration.
  • the corresponding point inverse transformation correction unit 410 inversely transforms the corresponding point coordinates of the area subjected to the viewpoint transformation (deformation) and moves the coordinate on the coordinate system of the original image. Are aggregated as corresponding points in the coordinate system of the original image.
  • a certain evaluation scale is determined from among the fundamental matrices other than the outliers obtained in this way, and the fundamental matrix having the highest evaluation value is used as an initial value.
  • the evaluation scale is, for example, a random selection of a pair of reliable corresponding points excluding the eight corresponding points not used for the generation of the basic matrix, and how much error the pair of corresponding points causes in the basic matrix. Is used as an evaluation scale.
  • the basic matrix obtained in this manner is first set as an initial value of the basic matrix indicating the correspondence between the left and right cameras 100 and 110, and further, highly accurate geometric calibration is performed using the corresponding points determined to be reliable. Perform parameter optimization.
  • the geometric calibration parameter estimating unit 440 solves the problem. This makes it possible to estimate a basic matrix (geometric calibration parameter) with higher accuracy than the 8-point method.
  • the availability determining unit 450 first determines the number of corresponding points obtained from the corresponding point aggregating unit 420 (whether the number of corresponding points exceeds a predetermined number) and the outliers other than the outliers obtained from the noise corresponding point deleting unit 430. External information such as the number of pairs of corresponding points (whether the number of pairs exceeds a predetermined number) and the magnitude of the minimized distance error obtained from the geometric calibration parameter estimating unit 440 (whether the magnitude of the distance error is less than a predetermined value) Using the information, it is determined whether or not the result of the camera geometric calibration by the geometric calibration parameter estimation unit 440 can be used.
  • the left and right camera images are parallelized using the obtained geometric calibration parameters, there is a corresponding point pair in which a vertical error does not occur in the left and right image coordinates after the parallelization among the determined corresponding point pairs.
  • the availability is determined based on whether the ratio exceeds a predetermined ratio (for example, 95%).
  • the geometric calibration reflection unit 460 when the availability is determined by the availability determination unit 450, parallelization is performed using the parameter base matrix indicating the geometry of the left and right cameras 100 and 110 estimated by the geometric calibration parameter estimation unit 440. Update the affine table of the image transformation for generating the image.
  • the parallax image generation unit 500 generates a parallax image based on the left and right images captured by the right and left cameras 100 and 110 and the latest affine table updated in real time by the camera geometry correction unit 400.
  • the parallax image generation unit 500 according to the present embodiment functions as a parallelized image generation unit 510, a stereo matching unit 520, and a distance calculation unit 530, as shown in FIG.
  • the parallelized image generation unit 510 generates a left-right parallelized image using the affine table for generating a parallelized image updated by the camera geometric calibration unit 400.
  • the stereo matching unit 520 performs stereo matching on the parallelized left and right images to generate a parallax image.
  • the distance calculation unit 530 performs a three-dimensional distance conversion from the parallax image using the base line length of the left and right cameras 100 and 110 and the internal parameters (focal length and cell size) of the camera, thereby obtaining an arbitrary value on the left and right images. Calculate the distance to the object.
  • FIG. 15 Processing flowchart of control device 10>
  • a processing flow executed by the control device 10 when the left image is vertically divided into two as described above will be described.
  • the control device 10 repeats a series of processes shown in FIG. 15 at a predetermined cycle.
  • Step S01 first, the control device 10 (viewpoint conversion unit 200) inputs the left and right images captured by the left and right cameras (stereo cameras) 100 and 110.
  • step S02 the control device 10 (viewpoint conversion unit 200) determines to divide the left image input in step S04 into upper and lower parts. Note that the right image is not divided.
  • step S03 the control device 10 (viewpoint conversion unit 200) divides the left image into two parts vertically and sets an area including the vanishing point VP as the upper area 111.
  • the upper region 111 is a region including the vanishing point VP and in which a distant scene tends to be imaged, and is used for the corresponding point search without performing the viewpoint conversion.
  • step S04 the control device 10 (viewpoint conversion unit 200) sets a region obtained by removing the upper region 111 from the left image (a region located below the upper region 111) as a lower region 112, and proceeds to step S05. Transition.
  • the lower area 112 is an area in which the road surface on which the vehicle travels occupies the majority of the imaged object, and on the road surface relatively close to the left and right cameras 100 and 110, the change in the appearance of the left and right cameras 100 and 110 is particularly large. Split to perform conversion.
  • step S05 the control device 10 (viewpoint conversion unit 200) generates a conversion parameter for performing the viewpoint conversion of the lower region 112, and converts the corresponding point on the lower region 112 after the viewpoint conversion by the inverse conversion before the viewpoint conversion. Generate an inverse transformation parameter for returning to the coordinates of.
  • the viewpoint conversion based on the conversion parameters generated here assumes that at least a part of the lower region 112 is a plane, estimates the positions and orientations of the left and right cameras 100 and 110 with respect to the plane, and estimates the estimated left and right cameras 100 and 110. , 110 is converted into an image from the viewpoint of the right camera 110 based on the position and orientation of the right camera 110.
  • step S06 the control device 10 (viewpoint conversion unit 200) converts the viewpoint of the lower area 112 of the left image using the conversion parameters generated in step S05.
  • step S07 the control device 10 (viewpoint conversion unit 200) generates a viewpoint conversion image in which the upper region 111 divided in step S03 and the lower region 112 converted in step S06 are combined. Then, the control device 10 (corresponding point search unit 300) performs a corresponding point search on the viewpoint converted image and the right image input in step S01. That is, a process of searching for a corresponding point in the right image based on the feature points and feature amounts on the image of the upper region 111 that has not been transformed and the lower region 112 that has been transformed is executed.
  • the first correspondence that is a set of a plurality of corresponding points based on the feature points and the feature amounts is obtained from the lower area 112 of the left image after the viewpoint conversion and the lower area 112 of the right image that has not been subjected to the viewpoint conversion.
  • a point group is extracted, and from the upper region 111 of the left image before the viewpoint conversion and the upper region 111 of the right image without the viewpoint conversion, a second set of a plurality of corresponding points based on the feature points and the feature amounts is obtained.
  • a corresponding point group is extracted.
  • step S08 the control device 10 (camera geometric calibration unit 400) straddles the upper and lower regions 111 and 112 of the plurality of corresponding points (first corresponding point group) found in the lower region 112 in step S07.
  • the remaining corresponding points are excluded and the inverse transformation is performed using the inverse transformation parameter generated in step S05, and the coordinate values of the remaining corresponding points are corresponded to the original image (the left image input in step S01). Returns to the coordinate value when the point was taken.
  • step S09 the control device 10 (camera geometric calibration unit 400) inversely transforms the coordinate values of the plurality of corresponding points (second corresponding point group) on the upper region 111 found in step S07 in step S08.
  • the coordinate values of a plurality of corresponding points (first corresponding point group) on the lower area 112 are collected.
  • the coordinates of the corresponding point on the left image in the first corresponding point group are the coordinates obtained by inversely converting the coordinates before the viewpoint conversion
  • the coordinates of the corresponding point on the right image in the first corresponding point group are the coordinates of the unconverted viewpoint.
  • the coordinates of the second corresponding point group are the coordinates before the viewpoint conversion for both the left and right images.
  • step S10 the control device 10 (camera geometric calibration unit 400) performs noise removal. From the corresponding points aggregated in step S09, eight corresponding point pairs are selected at random so as to form a coordinate system scattered on the image, and a so-called eight corresponding point pair (input corresponding point) is selected based on the selected corresponding point pair (input corresponding point). Calculate the values of the fundamental matrix by the point method. Then, the input corresponding points of the basic matrix that did not become outliers based on the values of the basic matrix are distinguished from the input corresponding points that became outliers by setting a flag so that they can be used in subsequent processing.
  • step S11 the control device 10 (camera geometric calibration unit 400) estimates the parameters of the geometric calibration using the coordinates of the corresponding point not determined as noise in step S10.
  • the basic matrix obtained by the above method as an initial value, an optimization problem that minimizes the distance error between the corresponding point on the image and the estimated point calculated using the basic matrix as a cost function solve.
  • a geometric calibration parameter with higher accuracy than the 8-point method.
  • step S12 the control device 10 (camera geometric calibration unit 400) uses information such as whether the magnitude of the distance error calculated in step S11 is less than a predetermined value or whether the number of corresponding points is equal to or more than a predetermined value. It is determined whether the geometric calibration parameters calculated in step S11 can be used. If it is determined that the geometric calibration parameters can be used, the process proceeds to step S13. On the other hand, if it is determined that the affine table cannot be used, the process proceeds to step S14 without updating the affine table.
  • step S13 the control device 10 (camera geometric calibration unit 400) updates the affine table for parallelizing the left and right images used in the previous frame based on the geometric calibration parameters calculated in step S11.
  • step S14 the control device 10 (the parallax image generation unit 500) generates a parallelized image of the left and right images using the stored affine table, performs stereo matching using the generated parallelized image, and generates a parallax image. I do.
  • the left and right cameras 100 and 110 perform viewpoint conversion on a region (lower region 112) that is greatly different in appearance, thereby finding from the left and right images.
  • the use of the close correspondence points enables highly accurate geometric calibration in the processing by the camera geometric calibration unit 400.
  • the parallax image generation unit 500 can generate a high-density and high-precision parallax image.
  • the left and right images are divided into an upper region and a lower region.
  • the viewpoint conversion is performed on the left image, and then the corresponding point is searched.
  • the viewpoint conversion is performed on both the left and right images.
  • the configuration for dividing the left and right images into a plurality of regions is not essential.
  • a plurality of corresponding points are extracted from the left image after the viewpoint conversion and the right image without the viewpoint conversion, and a plurality of correspondence points are extracted from the left image before the viewpoint conversion and the right image without the viewpoint conversion.
  • the points (second corresponding point group) may be extracted and geometric calibration of the left and right cameras 100 and 110 may be performed.
  • viewpoint conversion unit 200 Another example of viewpoint conversion by the viewpoint conversion unit 200 will be described.
  • the method of dividing the camera image into two vertically has been described.
  • a method of dividing the camera image into six or a method of dividing the camera image into free areas can be used.
  • the image is divided into six regions, and a method of searching for corresponding points on the left and right images for the six regions is selected.
  • the control device 10 divides the left image into six rectangular areas, and the six rectangular areas are given a plane attribute that is predicted to appear in each rectangular area while the host vehicle is traveling. It is determined based on the parallax image of the previous frame whether or not the plane attributes given to the six rectangular areas are valid.
  • the left and right cameras 100 , 110 are determined as conversion target regions, the positions and orientations of the left and right cameras 100 and 110 with respect to the conversion target region are estimated, and the estimated positions and orientations of the left and right cameras 100 and 110 are determined.
  • the conversion target area is converted into an image from the viewpoint of the right camera based on the left image, and a left image after the viewpoint conversion is obtained by combining the remaining area excluding the conversion target area from the six areas and the conversion target area.
  • the six rectangular regions are obtained by dividing the left image into two vertically and three horizontally, and each rectangular region is arranged in two rows and three columns.
  • the image is divided into an upper stage and a lower stage, which are referred to as a first region, a second region, and a third region from the left side of the upper stage, and are referred to as a fourth region, a fifth region, and a sixth region from the left side of the lower stage.
  • the plane attribute of the lower three rectangular regions (the fourth to sixth regions) in the six rectangular regions is “road surface”, and two rectangular regions located on the left and right of the upper three rectangular regions in the six rectangular regions.
  • the plane attribute of the (first and third regions) is “wall”, and the plane attribute of the center rectangular region (second region) of the upper three rectangular regions in the six rectangular regions is “infinity”. It is.
  • This method is effective in extracting feature points, reducing the processing time of description, and narrowing down corresponding point search candidates.
  • the plane attribute predicted to appear in each area while the own vehicle is running is determined, it is easy to select the deformation amount for each of the six areas.
  • the lower three regions perform the viewpoint conversion based on the viewpoint conversion assumed to be the road surface (ground) as before. Since the middle area (second area) near the upper infinity (vanishing point) contains only a distant view or the sky, no deformation occurs.
  • the upper left and right areas depend on the scenery of the running path, but when traveling in the city, buildings, trees, and the like are "walled" against the running path. There are many things that exist on the left and right. In such a case, conversion is performed assuming walls existing on the left and right of the travel path.
  • the road surface or the wall may be converted assuming a certain fixed value, or only the lateral position and the rotation of the wall are estimated from the parallax image of the previous frame as shown in the middle diagram of FIG. A method that utilizes this may be used. Convert the disparity values in the divided area into distances and perform plane estimation, and determine the amount of outliers and how many percent of disparity points occupy within a certain distance from the final estimated plane.
  • Judgment as to whether or not it may be approximated to a plane may be performed. If it is determined that the plane can be approximated, it is also calculated whether this plane is close to the two camera viewpoints and the difference in appearance due to the change of the viewpoint is large. If the difference in appearance is small, the need for viewpoint conversion is low in the first place. If viewpoint conversion is performed when the difference in appearance is large, the search performance of the corresponding point is greatly improved, so even if there is some error in the viewpoint conversion, it is better than performing the corresponding point search on the original image before conversion. Significantly closer correspondence points can be obtained.
  • a plane attribute predicted to appear in each rectangular area while the vehicle is traveling is given to each area in advance.
  • the upper left and right regions are “walls”
  • the lower three regions are “road surfaces”
  • the upper central region is “infinity”.
  • the plane attribute “far” is attached, and it is determined whether or not the plane estimated from the parallax value of the previous frame is similar to the plane defined by these attributes. If the plane attribute is different from the pre-assigned plane attribute, it is assumed that the plane cannot be approximated well, and the viewpoint conversion is not performed. This makes it possible to avoid erroneous viewpoint conversion.
  • the plane attribute is determined in advance, it is relatively easy to remove an outlier that is an unstable element, and the parameters to be estimated are narrowed, so that the stability is enhanced.
  • the two areas (first and third areas) located on the left and right of the upper row are equivalent to walls when there are buildings and trees along the traveling path. It is possible to use viewpoint conversion assuming that there is a plane (a region whose plane attribute is “wall”). However, when there are few three-dimensional objects around the road on a rural road, it is better to assign a non-planar attribute, such as "infinity", to the two regions without assigning the attribute of "plane”. The number of corresponding points may increase. For this reason, it is possible to understand the tendency of the three-dimensional structure of the scenery appearing in each region from the parallax image of the previous frame, and then determine whether or not to perform the viewpoint conversion and perform the corresponding point search.
  • a camera image may be pasted on a three-dimensional plane and deformed. To split the camera image. Therefore, as shown in FIG. 16, a region division is performed using a diagonal line as a boundary line of each region. Based on information obtained from the parallax image of the previous frame (for example, a road surface area estimation result and a road edge area estimation result), it is determined whether or not a portion that can be approximated to a plane exists in the image. Then, a region including a portion determined to be a portion that can be approximated to a plane is estimated as a road surface plane region.
  • the distance from the area to the left and right cameras 100 and 110 is less than a threshold, and the area where the distance is less than the threshold may be set as a road surface area, that is, an area for performing viewpoint conversion. Similarly, it may be determined whether or not the same object is in a region where the left and right cameras 100 and 110 look greatly different.
  • area division based on the assumption that there is a wall along the traveling direction of the traveling path is performed by using both the image color and the three-dimensional area. Division may be performed. For each of the divided areas, whether or not it can be approximated to a three-dimensional plane is estimated from the parallax image of the previous frame, as in the case of the six areas. If the plane can be approximated, the viewpoint conversion is performed according to the plane.
  • the entire screen can be used relatively effectively.
  • the background of the camera image is complicated, it is difficult to divide the area, and it is better to use a premise that knows what plane is to some extent, such as six areas, for stable determination. High stability.
  • this method has an advantage in a case where the environment recognition apparatus includes three or more cameras and performs triangulation from a pair of multiple cameras, and it is not clear which camera mainly performs three-dimensional reconstruction. It is easy to use for three-dimensional surveying with a camera. As described above, the present invention does not need to be a three-dimensional restoration mainly performed by the right camera as shown in FIG.
  • the control device 10 of the present embodiment includes a parallax image generation unit 500A.
  • the other parts are the same as in the first embodiment, and a description thereof will be omitted.
  • the parallax image generation unit 500A illustrated in FIG. 7 includes a region-based viewpoint conversion parallelized image generation unit 550, a region-based matching unit 560, a result integration unit 570, and a distance calculation unit 580.
  • the right and left cameras 100 and 110 may differ in the appearance of a short-distance subject even in stereo matching at the time of generating a parallax image as in the case of a corresponding point search. For this reason, parallax matching may be difficult even in stereo matching.
  • viewpoint conversion is performed also during stereo matching as in the corresponding point search of the first embodiment, parallax of a plane such as a road surface can be obtained densely. Further, as with the corresponding points, the corresponding points can be obtained as appropriate for the distant scenery without deformation.
  • the corresponding point method is used for algorithms that analyze the road surface shape, analyze small irregularities, or mainly observe the road surface such as where the vehicle can travel.
  • the region-specific viewpoint-converted parallelized image generation unit 550 divides the left image into two regions (upper region and lower region) in the far and near regions as in the first embodiment.
  • the viewpoint conversion is not performed, and in the lower region, the viewpoint conversion to the right camera viewpoint is performed simultaneously with the parallelization by the affine table.
  • viewpoint conversion may be performed by dividing an image after parallelization.
  • the conversion parameters at the time of viewpoint conversion and the affine table at the time of parallelization those calculated by the viewpoint conversion unit 200 and the camera geometric calibration unit 400 are used as in the first embodiment.
  • the region-specific matching unit 560 calculates parallax values individually for the two regions generated by the image generation unit 550, and generates parallax images individually.
  • the parallax values (parallax images) belonging to the lower region among the parallax values calculated by the region-specific matching unit 560 are corrected according to the viewpoint conversion, so that the upper region and the lower region are corrected. Correction is performed to match the meaning of the parallax values, and then the parallax values (parallax images) of the upper region and the lower region are integrated.
  • the distance calculation unit 580 calculates the distance from the parallax images integrated by the result integration unit 570, using the information on the base line length of the left and right cameras 100 and 110 and the information on the internal parameters of the left and right cameras 100 and 110. As a result, a parallax image based on a greater number of densely corresponding points can be obtained in a lower region (a road surface region at a short distance from the camera), which can greatly differ in the appearance of the left and right images, so that the accuracy of the parallax image is improved. improves.
  • the left and right images that is, the pair of the right image and the left image before the viewpoint conversion (first pair)
  • the viewpoint conversion after the region division and parallelization Both the result and the result of matching the left and right images (that is, the pair of the right image and the left image after the viewpoint conversion (second pair)) with the viewpoint conversion after the region division and parallelization are generated, and two matching results are generated.
  • the parallax value and the parallax image may be generated by using the one with the higher matching score indicating how similar the left and right images are.
  • the matching result with the viewpoint conversion is returned to the inversely converted state, and that the matching score indicating how similar the left and right images are similar can be referred to from each parallax value in both cases.
  • a three-dimensional object such as a pedestrian is present in the lower region, and the stereo matching using the image after the viewpoint conversion can prevent the matching score from being reduced. Can be improved.
  • FIG. 18 Flowchart of parallax image generation processing by control device 10>
  • a processing flow executed by the control device 10 (the parallax image generation unit 500A) when the left image is vertically divided into two when generating the parallax image as described above will be described.
  • the control device 10 repeats a series of processes illustrated in FIG. 18 based on the input of the parallax image request command.
  • the process of searching for the corresponding points for calibration and estimating / updating the geometric calibration parameters performed in steps DS02 to DS04 in the figure may use any method, and the first method shown in FIG.
  • the method is not limited to the method of the embodiment, and a known method may be used.
  • an example of a method of applying the viewpoint transformation of the first embodiment to the generation of a parallax image assuming that the calibration for parallelization has already been performed on the left and right images, will be described.
  • step DS01 first, the control device 10 (the parallax image generation unit 500A) inputs the left and right images captured by the left and right cameras (stereo cameras) 100 and 110.
  • Step DS02 the control device 10 (corresponding point search unit 300) searches for corresponding points in the left and right images.
  • step DS03 the control device 10 (camera geometric calibration unit 400) estimates geometric calibration parameters for performing parallelization of the left and right images.
  • step DS04 the control device 10 (camera geometric calibration unit 400) updates the geometric calibration parameters used when creating the parallelized image of the left and right images with the geometric calibration parameters calculated in step D02. .
  • the estimated values of the relative positions and postures of the left and right cameras 100 and 110 and the parameters of the position and posture of the road surface and the stereo camera may be updated.
  • step DS05 the control device 10 (the parallax image generation unit 500A) determines to divide the left image input in step DS01 into upper and lower parts. Note that the right image is not divided.
  • step DS06 the control device 10 (the parallax image generation unit 500A) divides the left image into upper and lower parts, and sets the area including the vanishing point VP as the upper area 111.
  • the upper region 111 is a region including the vanishing point VP and in which a distant scene tends to be captured, and is used for stereo matching (step DS07) without performing viewpoint conversion.
  • stereo matching with the image may be performed (step DS07).
  • step DS08 the control device 10 (the parallax image generation unit 500A) sets a region obtained by removing the upper region 111 from the left image (a region located below the upper region 111) as the lower region 112.
  • the lower area 112 is an area in which the road surface on which the vehicle travels occupies the majority of the imaged object, and on the road surface relatively close to the left and right cameras 100 and 110, the change in the appearance of the left and right cameras 100 and 110 is particularly large. Split to perform conversion.
  • the control device 10 (the parallax image generation unit 500A) generates a conversion parameter for performing the viewpoint conversion of the lower region 112 and performs the inverse conversion of the corresponding point on the lower region 112 after the viewpoint conversion before the viewpoint conversion. Generate an inverse transformation parameter for returning to coordinates.
  • the viewpoint conversion based on the conversion parameters generated here assumes that at least a part of the lower region 112 is a plane, estimates the positions and orientations of the left and right cameras 100 and 110 with respect to the plane, and estimates the estimated left and right cameras 100 and 110. , 110 is converted into an image from the viewpoint of the right camera 110 based on the position and orientation of the right camera 110.
  • control device 10 (the parallax image generation unit 500A) performs viewpoint conversion of the lower region 112 of the left image using the generated conversion parameters.
  • the viewpoint conversion image generated in step DS08 may be a viewpoint conversion image in which the viewpoint of the left image is converted to the viewpoint of the right camera 110. Assuming that the cameras are located at the center positions of the left and right cameras 100 and 110, image conversion may be performed as if the images of the left and right cameras were brought to the position of the center of gravity of the stereo camera.
  • step DS07 the control device 10 (the parallax image generation unit 500A) performs parallax calculation by performing stereo matching on the upper region 111 in step DS06 and the corresponding right image region to generate a parallax image.
  • step DS09 the control device 10 (disparity image generation unit 500A) calculates the disparity value by performing stereo matching on the lower region 112 converted in viewpoint in step DS08 and the corresponding right image region. Generate an image.
  • step DS10 the control device 10 (the parallax image generation unit 500A) inversely transforms the parallax value based on the viewpoint conversion image generated in step DS10 by using an inverse conversion parameter, thereby obtaining the parallax value of the parallax value. Perform conversion.
  • step DS11 the control device 10 (the parallax image generation unit 500A) determines that if two images (left and right images) on the parallelized image coordinates before the viewpoint conversion have a superimposed region, the lower region before the viewpoint conversion is performed.
  • the matching score between the corresponding portion of the right image and the matching score of the corresponding portion of the right image is compared with the matching score of the lower region 112 and the corresponding portion of the right image after the viewpoint conversion, and the disparity value with the higher matching score is used as the disparity value of the lower region. select.
  • the comparison of score matching acts to preferentially use the disparity when the viewpoint is changed on the road surface and preferentially use the disparity when the viewpoint is not changed when a three-dimensional object is present. Image accuracy is improved.
  • step DS12 the control device 10 (the parallax image generation unit 500A) combines the parallax image of the upper region 111 generated in step DS07 and the parallax image of the lower region 112 selected through the comparison in step DS11 into one sheet.
  • the control device 10 the parallax image generation unit 500A
  • the left image is vertically divided.
  • the left image and the right image are regions that are significantly different in appearance and include a portion that can be approximated to a plane, the disparity value obtained by the viewpoint conversion is used.
  • stereo matching may be performed by performing viewpoint conversion on an area different from the lower area described above, and this type of area includes an area on which viewpoint conversion is performed in the first embodiment.
  • steps S10 and S12 can be omitted.
  • the present invention is not limited to the above embodiments, and includes various modifications without departing from the gist of the present invention.
  • the present invention is not limited to those having all the configurations described in the above embodiments, but also includes those in which some of the configurations are deleted. Further, a part of the configuration according to one embodiment can be added to or replaced by the configuration according to another embodiment.
  • the configuration of the control device 10 may be a program (software) that is read and executed by an arithmetic processing device (for example, a CPU) to realize each function of the configuration of the device.
  • Information related to the program can be stored in, for example, a semiconductor memory (flash memory, SSD, etc.), a magnetic storage device (hard disk drive, etc.), a recording medium (magnetic disk, optical disk, etc.), and the like.
  • control lines and the information lines that are understood to be necessary for the description of the embodiment are shown, but all the control lines and the information lines related to the product are not necessarily used. It is not necessarily shown. In fact, it can be considered that almost all components are interconnected.
  • viewpoint conversion unit 100 left camera, 110 right camera, 111 upper region, 112 lower region, 200 viewpoint conversion unit, 210 region setting unit, 220 conversion parameter generation unit, 221 parallax analysis unit, 222 viewpoint conversion Attribute determination unit, 223: conversion parameter calculation unit, 230: inverse conversion parameter calculation unit, 240: viewpoint conversion image generation unit, 300: corresponding point search unit, 310: feature point extraction unit, 320: feature amount description unit, 330 ...
  • Maximum error setting unit 340: Corresponding point searching unit, 350: Reliability calculating unit, 400: Camera geometric calibration unit, 410: Corresponding point inverse transformation correcting unit, 420: Corresponding point aggregating unit, 430: Noise corresponding point deleting unit, 440: geometric calibration parameter estimating unit, 450: availability determining unit, 460: geometric calibration reflecting unit, 500: parallax image generating unit, 510: parallelized image generating unit, 520: stereo Matching unit, 530 ... distance calculator

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Electromagnetism (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Theoretical Computer Science (AREA)
  • Image Analysis (AREA)
  • Closed-Circuit Television Systems (AREA)
  • Image Processing (AREA)
  • Measurement Of Optical Distance (AREA)

Abstract

The present invention provides a vehicle-mounted environment recognition device that makes geometric camera calibration easy even if a single object shown in images photographed using a plurality of cameras has significantly different appearances in the images. The present invention comprises a first camera 100, a second camera 110, and a control device 10 for extracting a plurality of corresponding points after viewpoint conversion in which a first image imaged by the first camera and a second image imaged by the second camera are converted into images from a common viewpoint through the transformation of at least one from among the first image and second image and using the coordinates of the plurality of corresponding points in the first image and second image before viewpoint conversion to geometrically calibrate the first camera and second camera.

Description

車載環境認識装置In-vehicle environment recognition device
　本発明は車両に設置されたカメラにより車両の周囲環境を認識する車載環境認識装置に関する。 The present invention relates to an in-vehicle environment recognition device that recognizes a surrounding environment of a vehicle by a camera installed in the vehicle.
　車両に設置された２つのカメラにより車両の周囲環境を認識する車載カメラに関する技術がある。予防安全技術の製品化が普及期に入りつつあり、より多機能化かつ広視野なセンシングが廉価に求められるようになっている。 技術 There is a technology related to an in-vehicle camera that recognizes a surrounding environment of a vehicle by two cameras installed in the vehicle. The commercialization of preventive safety technology is entering the spread period, and multifunctional and wide-field sensing is required at low cost.
　車両に設置された２つのカメラ間の位置、姿勢を推定し、２つのカメラの画像が平行な位置関係となるように、幾何校正（キャリブレーション）する方法がある。この際に、２つのカメラの位置、姿勢の幾何関係を求めるための情報として、２つのカメラの画像上から得られる対応点を利用する手法が一般的である。この対応点を取得する手法は、左右画像上から特徴点といわれる画像上の角（コーナー）などのユニークな点を抽出し、その特徴点の周囲における輝度変化から特徴量を演算し、この特徴量が類似する特徴点を左右画像上で探索して発見された特徴点を対応点とし、左右画像における対応点の座標を基に幾何校正を実施する。 が あ る There is a method of estimating the position and orientation between two cameras installed in a vehicle and performing geometric calibration so that the images of the two cameras have a parallel positional relationship. At this time, a method of using corresponding points obtained from images of the two cameras is generally used as information for obtaining the geometric relationship between the positions and postures of the two cameras. The method of acquiring the corresponding point is to extract a unique point such as a corner (corner) on the image which is called a feature point from the left and right images, calculate a feature amount from a luminance change around the feature point, and calculate the feature amount. Feature points having similar amounts are searched for on the left and right images, and feature points found are set as corresponding points, and geometric calibration is performed based on the coordinates of the corresponding points in the left and right images.
　例えば、特開２０１４－７４６３２号公報（特許文献１）には、互いの視野が重なるように配置された左右のカメラの出力に基づき物体までの距離を算出するとともに物体の３次元位置を推定する３次元座標推定部と、３次元座標推定部における推定処理に用いられるカメラ間パラメータを記憶するカメラ間パラメータ記憶部と、左右のカメラの出力に基づき予め大きさが分かっている平面状の対象物を認識するとともに、対象物の特徴点座標を取得する対象物認識及び特徴点収集部と、特徴点座標に基づきカメラ間パラメータを求めるカメラ間パラメータ推定部とを備える車載ステレオカメラの校正装置が開示されている。 For example, Japanese Patent Application Laid-Open No. 2014-74632 (Patent Literature 1) calculates a distance to an object and estimates a three-dimensional position of the object based on outputs of left and right cameras arranged such that their fields of view overlap. A three-dimensional coordinate estimating unit, an inter-camera parameter storage unit for storing inter-camera parameters used for estimation processing in the three-dimensional coordinate estimating unit, and a planar object whose size is known in advance based on the outputs of the left and right cameras A calibration device for an in-vehicle stereo camera including an object recognition and feature point collection unit for recognizing and acquiring feature point coordinates of an object, and an inter-camera parameter estimation unit for obtaining an inter-camera parameter based on the feature point coordinates is disclosed. Have been.
特開２０１４－７４６３２号公報JP 2014-74632 A
　しかしながら、２つのカメラ間に映る風景や物体がカメラから近距離の場合や、２つのカメラの距離が離れている場合には、大きく異なる視点から同じ物体を観測することになり、２つの画像での同一物体の見え方（写り方）が大きく異なってしまう場合がある。このように同一物体の見え方が大きく異なると、２つの画像から対応を取りたい特徴点の抽出がうまくいっても、その点の周囲の輝度変化を基に演算した特徴量が２つのカメラ間で異なる可能性が高い。すなわち２つの特徴点の特徴量が異なる値となって計算されるため、対応点が見つからない、間違った対応点（誤対応点）が見つかる、又は対応点が見つかってもその個数が非常に少ない、といった課題が生じる。対応点が少ない又は誤対応点が多ければ、それだけカメラの幾何校正の精度が低下する要因となる。また、収束演算で２つのカメラ間の位置、姿勢が推定できないような場合も発生する。 However, when the scenery or object between the two cameras is close to the camera or when the two cameras are far apart, the same object is observed from a greatly different viewpoint, and the two images are used. Of the same object may be greatly different. When the appearance of the same object is greatly different in this way, even if the feature point to be correlated with the two images is successfully extracted, the feature amount calculated based on the luminance change around the point between the two cameras Likely to be different. That is, since the feature values of the two feature points are calculated as different values, no corresponding point is found, an incorrect corresponding point (erroneous corresponding point) is found, or even if a corresponding point is found, the number is very small. And the like. The smaller the number of corresponding points or the more erroneous corresponding points, the lower the accuracy of the geometric calibration of the camera. Further, there may be a case where the position and orientation between the two cameras cannot be estimated by the convergence calculation.
　本発明の目的は、複数のカメラで撮影した画像に見え方の大きく異なる同一物体が写っている場合でも容易にカメラの幾何校正ができる車載環境認識装置を提供することにある。 An object of the present invention is to provide an in-vehicle environment recognizing device that can easily perform a geometric calibration of a camera even when an image captured by a plurality of cameras includes the same object having a significantly different appearance.
　本願は上記課題を解決する手段を複数含んでいるが、その一例を挙げるならば、第１カメラ及び第２カメラと、前記第１カメラによって撮像された第１画像と前記第２カメラによって撮像された第２画像の少なくとも一方を変形することで前記第１画像及び前記第２画像を共通の視点からの画像に変換する視点変換を行った後に複数の対応点を抽出し、前記視点変換前の前記第１画像及び前記第２画像における前記複数の対応点の座標を利用して前記第１カメラ及び前記第２カメラの幾何校正を行う制御装置とを備えることを特徴とする。 The present application includes a plurality of means for solving the above-mentioned problems. For example, a first camera and a second camera, a first image captured by the first camera, and an image captured by the second camera are provided. After performing viewpoint conversion for converting the first image and the second image to an image from a common viewpoint by deforming at least one of the second images, a plurality of corresponding points are extracted, and the plurality of corresponding points are extracted before the viewpoint conversion. A control device for performing geometric calibration of the first camera and the second camera using coordinates of the plurality of corresponding points in the first image and the second image.
　本発明によれば、車両に設置された複数台のカメラの共通視野領域において、視点変換を実施することで、画像上で密な対応点を抽出し、この対応点に基づいて幾何校正することで、高精度にカメラ間の位置、姿勢を推定、２つのカメラの高精度な平行化を実現する。高精度な平行化を実施した状態でステレオマッチングすることで、高密度な視差画像の生成を実現、更にこの視差から高精度な距離復元を可能とする。 According to the present invention, dense corresponding points are extracted on an image by performing viewpoint conversion in a common visual field region of a plurality of cameras installed in a vehicle, and geometric calibration is performed based on the corresponding points. Thus, the position and orientation between the cameras can be estimated with high accuracy, and highly accurate parallelization of the two cameras can be realized. By performing stereo matching in a state where high-precision parallelization is performed, generation of a high-density parallax image is realized, and high-precision distance restoration is enabled from the parallax.
第１実施形態に係る車載環境認識装置の構成図。FIG. 1 is a configuration diagram of an in-vehicle environment recognition device according to a first embodiment. 視点変換部の機能ブロック図。FIG. 3 is a functional block diagram of a viewpoint conversion unit. 変換パラメータ生成部の機能ブロック図。FIG. 3 is a functional block diagram of a conversion parameter generation unit. 対応点探索部の機能ブロック図。FIG. 3 is a functional block diagram of a corresponding point search unit. カメラ幾何校正部の機能ブロック図。FIG. 3 is a functional block diagram of a camera geometric calibration unit. 視差画像生成部の機能ブロック図（第１実施形態）。FIG. 3 is a functional block diagram of a parallax image generation unit (first embodiment). 視差画像生成部の機能ブロック図（第２実施形態）。FIG. 9 is a functional block diagram of a parallax image generation unit (second embodiment). 対応点からカメラ幾何校正をするプロセスの説明。Description of the process of calibrating camera geometry from corresponding points. 対応点取得の課題の説明。Explanation of the problem of acquiring corresponding points. 視点変換による解決方法の説明。Explanation of solution by viewpoint conversion. 上下領域分割の視点変換の一例。An example of viewpoint conversion of upper and lower area division. せん断（近似変形）による視点変換の一例。An example of viewpoint conversion by shearing (approximate deformation). ６領域分割の視点変換の一例。6 is an example of viewpoint conversion in six regions. 逆変換に関する説明図。Explanatory drawing about an inverse transformation. 第１実施形態の制御装置の処理フローチャート。3 is a processing flowchart of a control device according to the first embodiment. 自由領域分割の視点変換の一例。4 is an example of viewpoint conversion for free area division. 左右カメラに対する視点変換の一例。An example of viewpoint conversion for left and right cameras. 第２実施形態の制御装置による視差画像生成処理のフローチャート。9 is a flowchart of a parallax image generation process performed by the control device according to the second embodiment.
　以下、本発明の実施形態について図面を用いて説明する。 Hereinafter, embodiments of the present invention will be described with reference to the drawings.
　・第１実施形態　
＜図１　車載環境認識装置＞
　図１に本実施形態に係る車載環境認識装置１の構成図を示す。車載環境認識装置１は、水平方向の左右に間隔を介して配置された左カメラ（第１カメラ）１００と右カメラ（第２カメラ）１１０と、２台のカメラ１００，１１０から出力される撮像画像（左カメラ１００による画像を第１画像、右カメラ１１０による画像を第２画像と称することがある）を基に、２台のカメラ１００，１１０の幾何校正（キャリブレーション）を行う処理や、２台のカメラ１００，１１０によって同一タイミングで撮像された２枚の画像をステレオマッチングして視差画像を作成する処理などを実行する制御装置（コンピュータ）１０を備えている。 -First embodiment
<Figure 1 In-vehicle environment recognition device>
FIG. 1 shows a configuration diagram of an in-vehicle environment recognition device 1 according to the present embodiment. The in-vehicle environment recognizing device 1 includes a left camera (first camera) 100 and a right camera (second camera) 110 arranged at intervals in the left and right directions in the horizontal direction, and imaging output from the two cameras 100 and 110. A process of performing geometric calibration of the two cameras 100 and 110 based on an image (an image obtained by the left camera 100 may be referred to as a first image and an image obtained by the right camera 110 may be referred to as a second image); A control device (computer) 10 that executes processing for creating a parallax image by performing stereo matching on two images captured by the two cameras 100 and 110 at the same timing is provided.
　制御装置（コンピュータ）１０は、図示しない演算処理装置（例えばＣＰＵ）と、その演算処理装置が実行するプログラム等が記憶される記憶装置（例えば、メモリ、ハードディスク、フラッシュメモリ）と、内部の機器同士や外部との機器との通信を行うための通信装置等を備えている。制御装置１０は、記憶装置に記憶されたプログラムを実行することで、視点変換部２００、対応点探索部３００、カメラ幾何校正部４００及び視差画像生成部５００として機能する。なお、プログラムの追加によりその他の機能の実装も可能である。 The control device (computer) 10 includes an arithmetic processing device (eg, CPU) not shown, a storage device (eg, memory, hard disk, flash memory) for storing a program executed by the arithmetic processing device, and internal devices. And a communication device for performing communication with external devices. The control device 10 functions as the viewpoint conversion unit 200, the corresponding point search unit 300, the camera geometric calibration unit 400, and the parallax image generation unit 500 by executing a program stored in the storage device. Other functions can be implemented by adding a program.
　一般に左右２台のカメラから成るステレオカメラは共通視野領域を利用して車載周囲環境を認識する。左右カメラを支持体の所定の位置に高精度に取り付けた後、カメラ工場において左右カメラでキャリブレーションチャートを撮像しながら左右カメラの位置、姿勢を推定し、その結果を利用して、左右カメラから撮像される画像がお互い平行になるようにパラメータを補正する。この状態でステレオマッチングすれば、高密度な視差と視差画像の取得が可能となり、さらにはその視差画像から高精度な距離が計測可能となる。しかし、高精度なカメラ製造や、温度変化、衝撃、振動及び経年変化等によるカメラの変形を抑制する部品や構造は高価である。 Generally, a stereo camera consisting of two cameras on the left and right recognizes the environment around the vehicle using a common viewing area. After attaching the left and right cameras to the predetermined position of the support with high accuracy, the camera factory estimates the position and orientation of the left and right cameras while capturing the calibration chart with the left and right cameras, and uses the results to The parameters are corrected so that the captured images are parallel to each other. If stereo matching is performed in this state, it is possible to obtain a high-density parallax and a parallax image, and it is possible to measure a distance with high accuracy from the parallax image. However, high-precision camera manufacturing and parts and structures that suppress deformation of the camera due to temperature change, shock, vibration, aging, and the like are expensive.
　本願発明者らは、この種のコスト抑制のために、カメラの位置ズレや温度・経年変化等によるカメラの変形を事後的に容易に補正可能で、走行中の高精度なキャリブレーションを実現可能なものとして本実施形態の車載環境認識装置１を発明した。 The present inventors can easily correct the deformation of the camera due to the positional shift of the camera, the temperature, the aging, etc. in order to suppress the cost of this kind, and realize the high-precision calibration during the traveling. As such, the in-vehicle environment recognition device 1 of the present embodiment has been invented.
　まず、本実施形態では、公知の技術と同様に、左右カメラ１００，１１０の画像上から特徴点（例えば画像上の物体の角（コーナー）等のユニークな点）を抽出し、その特徴点の周囲の輝度変化から特徴量を計算し、左右画像上で類似する特徴量を保有する特徴点を探索し、左右画像上で特徴量が類似する１組の特徴点を１組の対応点として設定する。図８の画像は探索された左右画像上の複数の対応点同士を線でつないで表示した画像であり、この複数の対応点を基に左右カメラ１００，１１０の幾何校正を実施する。 First, in the present embodiment, similarly to the known technique, feature points (unique points such as corners (corners) of an object on the image) are extracted from the images of the left and right cameras 100 and 110, and the feature points are extracted. A feature amount is calculated from a change in surrounding brightness, a feature point having a similar feature amount is searched for on the left and right images, and a set of feature points having similar feature amounts on the left and right images is set as a set of corresponding points. I do. The image of FIG. 8 is an image in which a plurality of corresponding points on the searched left and right images are connected to each other with a line, and geometric calibration of the left and right cameras 100 and 110 is performed based on the plurality of corresponding points.
　図１において、対応点探索部３００は、左右カメラ１００，１１０の画像上から先ほど説明したような特徴点を抽出し、その特徴点の周囲の輝度変化から特徴量を計算し、左右画像上で類似する特徴量を保有する特徴点（対応点）を探索する。 In FIG. 1, the corresponding point search unit 300 extracts the above-described feature point from the images of the left and right cameras 100 and 110, calculates a feature amount from a luminance change around the feature point, and calculates the feature amount on the left and right images. A feature point (corresponding point) having a similar feature amount is searched.
　カメラ幾何校正部４００は、対応点探索部３００で得られた複数の対応点を基に左右画像を平行にする幾何校正パラメータを演算する。左右カメラ１００，１１０の画像を幾何校正した画像は、左右平行な位置関係であると共に、レンズ歪みの全くない画像となる。
  これにより幾何的にマッチングし易い、左右画像が準備可能となる。 The camera geometric calibration unit 400 calculates geometric calibration parameters for making the left and right images parallel based on the plurality of corresponding points obtained by the corresponding point search unit 300. The image obtained by geometrically calibrating the images of the left and right cameras 100 and 110 is an image having a horizontal positional relationship and no lens distortion at all.
This makes it possible to prepare left and right images that are easily geometrically matched.
　視差画像生成部５００は、同一タイミングで撮像されカメラ幾何校正部４００の幾何校正パラメータで補正された２枚の画像（平行化画像）に対してステレオマッチングを実施して、２枚の画像上で同じ物（同じパターン）が撮影された位置のズレを示す距離情報（視差情報）を公知の方法で計算して視差画像（距離画像）を生成する。視差画像生成部５００では、左カメラ１００と右カメラ１１０から撮像された左右の画像を利用する。本実施形態では右カメラ１１０による右画像（第２画像）をベースとしてステレオマッチングを実施するために、基本的には右基準に感度、幾何などを合わせるものとする。視差画像生成部５００は、幾何と感度の補正が適応された左右カメラの画像を入力してステレオマッチングを実施し、視差画像を生成し、最後にノイズ除去を実施することでノイズ除去された視差画像が得られる。 The parallax image generation unit 500 performs stereo matching on two images (parallelized images) captured at the same timing and corrected by the geometric calibration parameters of the camera geometric calibration unit 400, and performs two-dimensional matching on the two images. The distance information (parallax information) indicating the deviation of the position where the same object (the same pattern) is photographed is calculated by a known method to generate a parallax image (distance image). The parallax image generation unit 500 uses left and right images captured by the left camera 100 and the right camera 110. In the present embodiment, since the stereo matching is performed based on the right image (second image) from the right camera 110, the sensitivity, geometry, and the like are basically matched to the right reference. The parallax image generation unit 500 receives the images of the left and right cameras to which the geometric and sensitivity corrections have been applied, performs stereo matching, generates a parallax image, and finally performs noise removal to remove the noise-reduced parallax. An image is obtained.
　本実施形態では視点変換部２００で行われる処理が最も特徴的な部分となる。視点変換部２００は、左カメラ１００によって撮像された左画像（第１画像）と右カメラ１１０によって撮像された右画像（第２画像）の少なくとも一方を変形することで左右画像（第１画像及び前記第２画像）を共通の視点からの画像に変換する視点変換を行う。これにより対応点探索部３００において左右画像から対応点が見つかりやすくなる。画像の変形方法（視点変換の方法）の詳細については後述するが、例えば、左カメラ１００により撮像された画像（左画像,第１画像）を、右カメラ１１０の視点からの見え方に近づくように変形する（例えば、画像の拡大縮小、回転、平行移動、せん断を含むアフィン変換により画像を変形する）ことで、画像上の領域の全体又はその一部において、視点変換前よりも数の多い密な対応点の探索結果を対応点探索部３００で得ることができる。この密な対応点を利用することで、後段で行われるカメラ幾何校正部４００の処理では、高精度な幾何校正を実現する。幾何校正により正確に平行化された左右のペア画像を利用することで、視差画像生成部５００における高密度かつ高精度な視差画像の生成が可能となる。 で は In the present embodiment, the processing performed by the viewpoint conversion unit 200 is the most characteristic part. The viewpoint conversion unit 200 deforms at least one of the left image (first image) captured by the left camera 100 and the right image (second image) captured by the right camera 110, thereby forming the left and right images (first image and first image). Viewpoint conversion for converting the second image) into an image from a common viewpoint. This makes it easier for the corresponding point search unit 300 to find corresponding points from the left and right images. The details of the image deformation method (viewpoint conversion method) will be described later. For example, the image (the left image, the first image) captured by the left camera 100 is made to approach the appearance from the viewpoint of the right camera 110. (For example, by deforming the image by affine transformation including scaling, rotation, translation, and shearing of the image), the whole or a part of the region on the image has a larger number than before the viewpoint transformation. The search result of the dense corresponding point can be obtained by the corresponding point search unit 300. By utilizing these dense corresponding points, high-precision geometric calibration is realized in the processing of the camera geometric calibration unit 400 performed in the subsequent stage. By using the paired right and left images accurately parallelized by the geometric calibration, the parallax image generation unit 500 can generate a high-density and high-precision parallax image.
　２つのカメラ間の距離が比較的大きい長基線長のステレオカメラの場合や、近距離対象物が撮像された場合、同じ物体を全く異なる角度から撮像することになり、左右画像に撮像される当該物体の見え方は大きく異なるため、左右画像上で類似する特徴量を探索する一般的な対応点抽出手法では、対応点が取得しづらい。例えば、図９の上側に示す左右画像では、遠距離は左右画像での変形が相対的に少ないため対応点の取得が容易であるが、路面上に置いた物体９１をはじめとした近距離では変形が相対的に大きく対応点を密に得ることが難しい。対応点探索方法として、図１０の（２）、（３）に示すように、左画像での路面上の４点の見え方（画像上の位置）が右画像と一致又は類似するように視点変換してから対応点探索する。このようにすると、変形前の図９下側の画像では対応点が路面からほとんど探索できなかったが、図１０の最下段に示すように変形後は路面から対応点が密に得られていることが確認できる。 In the case of a stereo camera with a long base line in which the distance between the two cameras is relatively large, or when a short-distance object is imaged, the same object is imaged from completely different angles, and the right and left images are imaged. Since the appearance of the object is greatly different, it is difficult to obtain a corresponding point by a general corresponding point extraction method for searching for similar feature amounts on the left and right images. For example, in the left and right images shown in the upper part of FIG. 9, at a long distance, the deformation in the left and right images is relatively small, so that it is easy to obtain a corresponding point. However, at a short distance such as an object 91 placed on a road surface, The deformation is relatively large and it is difficult to obtain corresponding points densely. As a corresponding point search method, as shown in (2) and (3) of FIG. 10, the viewpoint (position on the image) of four points on the road surface in the left image is matched with or similar to the right image. After conversion, search for corresponding points. In this manner, in the image on the lower side of FIG. 9 before the deformation, the corresponding point could hardly be searched for from the road surface, but the corresponding points were densely obtained from the road surface after the deformation as shown in the lowermost part of FIG. Can be confirmed.
　このように画像の変形、例えば視点変換による変形などを活用して、密な対応点を得ることで、高精度に幾何校正を実施し、平行化してからステレオマッチングすることで、高密度、高精度な視差画像の生成と測距を実現する。 In this way, by utilizing image deformation, for example, deformation by viewpoint transformation, dense correspondence points are obtained, high-precision geometric calibration is performed, parallelization is performed, and then stereo matching is performed. Achieve accurate parallax image generation and ranging.
　＜図２　視点変換部２００＞
　次に視点変換部２００で実施可能な視点変換（画像変形）の例について説明する。図２に示すように視点変換部２００は、左右画像の少なくとも一方について視点変換を行う領域と行わない領域を設定する領域設定部２１０と、領域設定部２１０で設定された視点変換を行う領域を変形することで左右画像を共通の視点からの画像に変換するために必要な行列や関数のパラメータを生成する変換パラメータ生成部２２０と、変換パラメータ生成部２２０によって生成されたパラメータ（変換パラメータ）を有する行列や関数を利用することで左右画像の少なくとも一方を視点変換した画像（視点変換画像）を生成する視点変換画像生成部２４０と、視点変換後の画像を元に戻す逆変換に必要な逆行列や関数のパラメータ（逆変換パラメータ）を生成する逆変換パラメータ計算部２３０とを備えている。 <FIG. 2 Viewpoint conversion unit 200>
Next, an example of viewpoint transformation (image deformation) that can be performed by the viewpoint transformation unit 200 will be described. As illustrated in FIG. 2, the viewpoint conversion unit 200 includes an area setting unit 210 that sets an area where at least one of the left and right images is to be subjected to viewpoint conversion and an area that is not to be subjected to viewpoint conversion. A transformation parameter generation unit 220 that generates a matrix or a parameter of a function necessary for transforming the left and right images into an image from a common viewpoint by transforming, and a parameter (conversion parameter) generated by the conversion parameter generation unit 220 A viewpoint conversion image generation unit 240 that generates an image (viewpoint conversion image) in which at least one of the left and right images is viewpoint-converted by using a matrix or a function included therein, An inverse transformation parameter calculation unit 230 that generates parameters (inverse transformation parameters) of matrices and functions is provided.
　視点変換画像生成部２４０により生成された視点変換画像は対応点探索部３００における対応点探索で利用される。また、逆変換パラメータ計算部２３０で計算された逆変換パラメータはカメラ幾何校正部４００の対応点逆変換補正部４１０で利用される。 視点 The viewpoint converted image generated by the viewpoint converted image generation unit 240 is used in the corresponding point search in the corresponding point search unit 300. Further, the inverse transformation parameter calculated by the inverse transformation parameter calculation unit 230 is used by the corresponding point inverse transformation correction unit 410 of the camera geometric calibration unit 400.
　なお、領域設定部２１０ではカメラ画像内の全ての領域を視点変換を行う領域や視点変換を行わない領域として設定しても良い。左右カメラ１００，１１０の視点変換先である共通の視点としては、左右カメラ１００，１１０のいずれか一方の視点と、左右カメラ１００，１１０のいずれとも異なる任意の視点がある。後者の場合には、例えば、左カメラ１００と右カメラ１１０の間に位置する所定の視点（例えば左右カメラ１００，１１０の中点）を共通の視点に設定して視点変換することができる。 Note that the region setting unit 210 may set all the regions in the camera image as regions where viewpoint conversion is performed or regions where viewpoint conversion is not performed. As a common viewpoint to which the left and right cameras 100 and 110 are converted, there is a viewpoint of either one of the left and right cameras 100 and 110 and an arbitrary viewpoint different from any of the left and right cameras 100 and 110. In the latter case, for example, a predetermined viewpoint (for example, the middle point between the left and right cameras 100 and 110) located between the left camera 100 and the right camera 110 can be set as a common viewpoint to perform viewpoint conversion.
　（１）簡易的な視点変換
　視点変換の１つとしては、制御装置１０の視点変換部２００が、左右画像（第１画像及び前記第２画像）で重複して撮像された部分（重複領域）の少なくとも一部が平面であると仮定し、当該平面に対する左カメラ１００及び右カメラ１１０の位置及び姿勢を左右画像に基づいてカメラ幾何的に演算し、その演算した当該平面に対する左カメラ１００及び右カメラ１１０の位置及び姿勢に基づいて左画像（第１画像）を右カメラ１１０の視点からの画像に変換するものがある。 (1) Simple viewpoint conversion As one of viewpoint conversions, a part (overlap area) where the viewpoint conversion unit 200 of the control device 10 is overlapped and captured in the left and right images (the first image and the second image). Is assumed to be a plane, the positions and orientations of the left camera 100 and the right camera 110 with respect to the plane are geometrically calculated based on the left and right images, and the left camera 100 and the right with respect to the calculated plane are calculated. Some cameras convert a left image (first image) into an image from the viewpoint of the right camera 110 based on the position and orientation of the camera 110.
　この方法に関して図１０の例では、左カメラ１００の画像に撮像された一部が路面（平面）であると仮定して、左右カメラ１００,１１０と当該路面（平面）上の任意の点との相対的な位置・姿勢をカメラ幾何的に演算し、その位置・姿勢を基に左カメラ１００の画像を右カメラ１１０の位置・姿勢に視点変換した画像を生成することで密な対応点を取得している。 In the example of FIG. 10 regarding this method, assuming that a part captured in the image of the left camera 100 is a road surface (plane), the left and right cameras 100 and 110 and an arbitrary point on the road surface (plane) are assumed. Obtain dense corresponding points by calculating the relative position / posture camera geometrically and generating an image obtained by transforming the image of the left camera 100 into the position / posture of the right camera 110 based on the position / posture. are doing.
　ただし、ここで変形前の図９の下に示す画像（元画像）では遠方の立体物上で多くの密な対応点が抽出されており、反対に、変形後の図１０の下に示す画像（視点変換画像）では、遠方の立体物上で対応点があまり抽出されていない代わりに路面上から多くの密な対応点が得られている。図１０では、左カメラ１００の元画像中の左下にある路面が右カメラ１１０の画像中の当該路面と同じ見え方になるように左右方向に平行な水平せん断変形を左画像に施した。これにより左右画像で当該路面の形状が類似した結果、当該路面上の対応点が密に得られた。ただし、この視点変換は、左画像の一部に過ぎない路面のみに着目して左画像全体を変形したため、路面上に存在しない立体物（特に図１０に示す建物など）の形状に着目すると視点変換後に大きく斜めになっていることが確認できる。このような変形を施すと、当該建物については、むしろ変形前の方が左右画像上で見え方（形状）が類似しており、元々の画像（原画像）でもカメラ１００,１１０から遠方では密な対応点が得られることが理解できる。 However, here, in the image (original image) shown in the lower part of FIG. 9 before the deformation, many dense corresponding points are extracted on a distant three-dimensional object, and conversely, the image shown in the lower part of FIG. In the (viewpoint conversion image), many dense corresponding points are obtained from the road surface instead of extracting much corresponding points on a distant three-dimensional object. In FIG. 10, the left image is subjected to horizontal shear deformation parallel to the left and right directions so that the road surface at the lower left in the original image of the left camera 100 has the same appearance as the road surface in the image of the right camera 110. Thereby, as a result of the shape of the road surface being similar in the left and right images, corresponding points on the road surface were densely obtained. However, since this viewpoint conversion deforms the entire left image by focusing only on the road surface which is only a part of the left image, the viewpoint conversion is performed by focusing on the shape of a three-dimensional object that does not exist on the road surface (particularly, the building shown in FIG. 10). It can be confirmed that the image is greatly inclined after the conversion. When such a deformation is applied, the appearance (shape) of the building before and after the deformation is more similar on the left and right images, and the original image (the original image) is denser at a distance from the cameras 100 and 110. It can be understood that a corresponding point can be obtained.
　（２）上下２分割による視点変換
　そこで、本実施形態では、左右画像からより多くの対応点を得るために、視点変換を行う領域と視点変換を行わない領域の２種類の領域に左画像を分割して対応点の探索を行うこととする。前者の視点変換を行う領域は、視点変換を行うと対応点が増加する領域と換言でき、例えばカメラから近距離の領域が該当する。後者の視点変換を行わない領域は、原画像でも密な対応点が取得できる領域と換言でき、例えばカメラから遠距離の領域が該当する。 (2) Perspective conversion by upper and lower two divisions In this embodiment, in order to obtain more corresponding points from the left and right images, the left image is divided into two types of regions, a region where viewpoint conversion is performed and a region where viewpoint conversion is not performed. The search is performed for the corresponding points by division. The former area in which the viewpoint conversion is performed can be said to be an area in which the number of corresponding points increases when the viewpoint conversion is performed. The latter area in which viewpoint conversion is not performed can be said to be an area in which dense corresponding points can be obtained even in the original image, and corresponds to, for example, an area far from the camera.
　この種の視点変換としては、図１１に示すように、画像中の消失点ＶＰを基準として、左画像を水平方向の境界線で上下に２分割し、２つの領域のうち上側の領域（上側領域）１１１については視点変換を行わない領域として設定し、下側の領域（下側領域）１１２については視点変換を行う領域として設定するものがある。具体的には、制御装置１０の視点変換部２００が、消失点ＶＰを含む又は消失点ＶＰに接する境界を有する上側領域（遠景）１１１と、その上側領域１１１の下方に位置する下側領域（路面）１１２の２つの領域に左画像（第１画像）を上下に２分割し、下側領域１１２の少なくとも一部が平面であると仮定し、その平面に対する左右カメラ１００，１１０の位置及び姿勢を演算し、その演算した当該平面に対する左右カメラ１００，１１０の位置及び姿勢に基づいて下側領域１１２を右カメラ１１０の視点からの画像に変換し、その変換後の下側領域１１２と上側領域（視点変換無し）１１１を合わせたものを視点変換後の左画像（第１画像）とする。 As this kind of viewpoint conversion, as shown in FIG. 11, the left image is vertically divided into two by a horizontal boundary line based on a vanishing point VP in the image, and the upper region (upper region) of the two regions is divided. In some cases, the region (region) 111 is set as a region in which viewpoint conversion is not performed, and the lower region (lower region) 112 is set as a region in which viewpoint conversion is performed. Specifically, the viewpoint conversion unit 200 of the control device 10 includes an upper region (distant view) 111 including a vanishing point VP or having a boundary in contact with the vanishing point VP, and a lower region (below the upper region 111). The left image (first image) is divided vertically into two areas of two road areas 112, and it is assumed that at least a part of the lower area 112 is a plane, and the positions and postures of the left and right cameras 100 and 110 with respect to the plane. Is calculated, and the lower area 112 is converted into an image from the viewpoint of the right camera 110 based on the calculated position and orientation of the left and right cameras 100 and 110 with respect to the plane. (No viewpoint conversion) A combination of 111 and the left image (first image) after viewpoint conversion.
　この視点変換は車載カメラ（車載環境認識装置）のカメラ幾何校正において最も簡単かつ有効なものの１つである。また、この方法では、視点変換後の画像（視点変換画像）をＦＰＧＡ（Field-Programmable Gate Array），ＡＳＩＣ（Application Specific Integrated Circuit），ＧＰＵ（Graphics Processing Unit）等のハードウェアで利用すること等を想定し、長方形にしか分割しないという制約をつけて簡易的に変換している。遠方の路面も平面と仮定して変形可能であるが、そもそも遠方であれば視点変換による路面の変形量が小さい。また遠景の立体物なども視点変換による変形が小さいため、消失点ＶＰを含む上側の領域１１１は視点変換しない領域として領域設定部２１０において設定する。
  車載カメラの場合には、画面の下側の領域１１２には基本的に自車両が走行する道路が撮像されるため、多くの領域が平面である路面が撮像される可能性が非常に高い。そこで、下側領域１１２は、路面が撮像される領域である仮定して、視点変換する領域として領域設定部２１０において設定する。 This viewpoint conversion is one of the simplest and most effective ones in camera geometric calibration of an in-vehicle camera (in-vehicle environment recognition device). In addition, this method uses an image after viewpoint conversion (viewpoint conversion image) with hardware such as an FPGA (Field-Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), and a GPU (Graphics Processing Unit). Assuming, conversion is simplified with the restriction that the image is divided only into rectangles. A distant road surface can also be deformed on the assumption that it is flat, but if it is far away, the amount of deformation of the road surface due to viewpoint conversion is small. Further, since a three-dimensional object in a distant view is not significantly deformed by the viewpoint conversion, the upper region 111 including the vanishing point VP is set in the region setting unit 210 as a region where the viewpoint is not changed.
In the case of an in-vehicle camera, since the road on which the host vehicle travels is basically captured in the area 112 on the lower side of the screen, it is very likely that a road surface in which many areas are flat is captured. Therefore, assuming that the lower region 112 is a region where the road surface is imaged, the lower region 112 is set by the region setting unit 210 as a region to be changed in viewpoint.
　視点変換部２００は、変換パラメータ生成部２２０で生成された変換パラメータを利用して下側領域１１２の視点変換を行う。簡単な視点変換方法としては、車両に取り付けられたカメラ１００，１１０の位置・姿勢の設計値を利用して変換パラメータ生成部２２０で変換パラメータを生成する方法がある。車載環境認識装置１の画像処理によりカメラ１００，１１０の位置・姿勢を設計値から補正している場合には補正後の値を使っても良い。また、路面上の少なくとも３点の距離情報に基づいて当該路面の姿勢をカメラの位置・姿勢と同時に推定している場合には路面の姿勢情報を利用してよい。なお、ここで左右カメラの位置・姿勢として設計値が利用可能としているのは、左右画像が視点変換前より類似することが重要であり、両者が数学的に完全に一致する必要はないという理由からである。なぜならば、視点変換の目的は視点変換後の画像で対応点を密に得ることであり、対応点が得られる程度に類似した画像を生成できれば足りるからである。また、変換パラメータによる視点変換後の対応点の座標は、もう一度、逆変換パラメータによって視点変換前の座標に戻した上でカメラの幾何校正に利用される。そのため必ずしも高精度なカメラの位置・姿勢が必要なわけではない。 The viewpoint conversion unit 200 performs the viewpoint conversion of the lower region 112 using the conversion parameters generated by the conversion parameter generation unit 220. As a simple viewpoint conversion method, there is a method in which conversion parameters are generated by the conversion parameter generation unit 220 using design values of the positions and postures of the cameras 100 and 110 mounted on the vehicle. When the positions and orientations of the cameras 100 and 110 are corrected from the design values by the image processing of the in-vehicle environment recognition device 1, the corrected values may be used. If the posture of the road surface is estimated at the same time as the position and posture of the camera based on the distance information of at least three points on the road surface, the posture information of the road surface may be used. Here, the reason that the design values can be used as the position and orientation of the left and right cameras is that it is important that the left and right images are more similar than before the viewpoint conversion, and the two do not need to be mathematically completely identical. Because. This is because the purpose of the viewpoint conversion is to obtain corresponding points densely in the image after the viewpoint conversion, and it suffices to generate an image similar to the degree that the corresponding points can be obtained. Further, the coordinates of the corresponding point after the viewpoint conversion by the conversion parameter are returned to the coordinates before the viewpoint conversion by the inverse conversion parameter, and are used for the geometric calibration of the camera. Therefore, a high-precision camera position / posture is not always necessary.
　（２－１）上下２分割による視点変換（画像の拡大縮小、回転、平行移動、せん断を含むアフィン変換）
　上記のように、数学的にカメラ１００，１１０と路面の相対位置・姿勢から数学的に視点変換の演算を実施しても良いが、視点変換の主目的は正確な変換にあるのではなく、左右画像の対応点を密に得ることである。このため、数学的に正確な変換でなかったとしても、ある程度の効果が認められる手法で代替も可能である。ハードウェアでの数学演算に基づく変換を省略しても良い。 (2-1) Viewpoint transformation by upper and lower two divisions (affine transformation including image scaling, rotation, translation, and shearing)
As described above, the viewpoint conversion may be mathematically performed based on the relative positions and postures of the cameras 100 and 110 and the road surface. However, the main purpose of the viewpoint conversion is not accurate conversion. It is to obtain corresponding points of the left and right images densely. For this reason, even if the conversion is not mathematically accurate, it can be replaced by a method that has a certain effect. The conversion based on the mathematical operation in hardware may be omitted.
　例えば、左右カメラ１００，１１０の位置・姿勢が未知（未演算及び演算不可の場合を含む）で画像の変形量が決まっていない場合には、変換パラメータ生成部２２０において、消失点ＶＲを中心と考えた画像の回転や路面のせん断変形等を含むアフィン変換で画像を変形して視点変換を行っても良い。 For example, when the positions and orientations of the left and right cameras 100 and 110 are unknown (including the case where the calculation is not possible and the calculation is not possible) and the amount of deformation of the image is not determined, the conversion parameter generation unit 220 sets the center of the vanishing point VR as the center. The viewpoint transformation may be performed by transforming the image by affine transformation including the rotation of the image and the shear deformation of the road surface.
　この場合の視点変換の具体的方法としては、制御装置１０の視点変換部２００が、左カメラ１００による左画像（第１画像）にパラメータ（アフィン変換のための行列のパラメータ）を変化させながらアフィン変換を施して複数の変換画像を生成し、その複数の変換画像のそれぞれと右画像（第２画像）の対応点の抽出を実施し、その複数の変換画像の中で対応点の数が最も多くかつ所定の閾値（基準値）以上の変換画像を後続の処理で利用する最終的な視点変換画像（すなわち左画像を右カメラ１１０の視点からの画像に変換した画像）とする処理がある。この方法で複数の変換画像を生成する際には、例えば、画像の拡大縮小、回転、平行移動及びせん断の少なくとも１つを含む全ての組合せについて、パラメータを少しずつ変化させながらアフィン変換を施す。これにより実質的に全てのパターンの変換画像とそれに利用した変換行列が生成できる。その際、変換画像の数はパラメータを変化させる間隔で調整できる。そして、生成された全ての変換画像について右画像との対応点の探索を実行し、得られた対応点の数を比較することで、右画像に最も類似する形状に左画像を変形した視点変換画像と変換行列を取得できる。このように取得された変換行列のパラメータは変換パラメータとして利用できる。 As a specific method of viewpoint conversion in this case, the viewpoint conversion unit 200 of the control device 10 changes the parameters (matrix parameters for affine transformation) to the left image (first image) from the left camera 100 while changing the affine. Conversion is performed to generate a plurality of converted images, and corresponding points of each of the plurality of converted images and the right image (second image) are extracted. There is a process of using a large number of converted images that are equal to or more than a predetermined threshold (reference value) in a final viewpoint converted image to be used in subsequent processing (that is, an image obtained by converting a left image into an image from the viewpoint of the right camera 110). When generating a plurality of transformed images by this method, for example, affine transformation is performed for all combinations including at least one of scaling, rotation, translation, and shearing of the image while gradually changing parameters. As a result, converted images of substantially all patterns and a conversion matrix used for the converted images can be generated. At this time, the number of converted images can be adjusted at intervals at which parameters are changed. Then, a search for corresponding points with the right image is performed for all of the generated converted images, and the number of corresponding points obtained is compared, thereby transforming the left image into a shape most similar to the right image. You can get the image and the transformation matrix. The parameters of the transformation matrix thus obtained can be used as transformation parameters.
　図１２を用いてアフィン変換としてせん断変形を利用した視点変換の方法について説明する。図１２の例では、左画像の路面領域のせん断の変形量（せん断量）を所定の間隔で増やし続けて複数の変換画像を生成しつつ、各変換画像について右画像との対応点を探索し、その対応点の数が最も多くかつ基準（所定の閾値）より多い変換画像のせん断量を変換パラメータとして採用することを決定している。 A method of viewpoint transformation using shear deformation as affine transformation will be described with reference to FIG. In the example of FIG. 12, while generating a plurality of converted images by continuously increasing the amount of shear deformation (shear amount) of the road surface region of the left image at predetermined intervals, a corresponding point of each converted image with the right image is searched. It is determined that the shear amount of the converted image in which the number of corresponding points is the largest and which is larger than the reference (predetermined threshold) is used as the conversion parameter.
　せん断量をＡｐｉｘｅｌ、Ｂｐｉｘｅｌ、Ｃｐｉｘｅｌ（但し、Ａ＞Ｂ＞Ｃ）と増やすごとに左画像の路面領域（下側領域）の変形を増していき、その都度、左画像の路面と右画像の対応点数を記録する。その後、変形をＣｐｉｘｅｌまで続けていくと、Ｂｐｉｘｅｌ付近に対応点数のピークがあることがわかった。このような場合には、せん断量がＢｐｉｘｅｌの場合に右画像に最も類似した変形ができたことを示す。 Each time the shear amount is increased to Apixel, Bpixel, Cpixel (where A> B> C), the deformation of the road surface area (lower area) of the left image is increased, and each time, the correspondence between the road surface of the left image and the right image is increased. Record the score. Thereafter, when the deformation was continued up to Cpixel, it was found that there was a peak of the number of corresponding points near Bpixel. In such a case, it is indicated that when the shear amount is Bpixel, the deformation most similar to the right image has been performed.
　このため、変換パラメータの算出に際しては、本来であれば乗員の乗車や走行中のピッチングなどを考慮して路面とカメラの位置・姿勢の関係を補正する必要があるが、これらを無視した上記の簡易な方法でも密な対応点を取得することができる。パラメータが毎回固定となるだけでなく、ありふれた変形であるため、比較的スペックの低いハードウェアでの利用や共通化なども容易となる。なお、ここではせん断を利用する場合について説明したが、回転、拡大縮小、平行移動などのアフィン変換を利用する場合にも上記と同様のアプローチで視点変換が可能である。 For this reason, when calculating the conversion parameters, it is necessary to correct the relationship between the road surface and the position and orientation of the camera in consideration of the occupant's riding and pitching while traveling. Dense corresponding points can be obtained by a simple method. Not only are the parameters fixed each time, but also a common deformation, it is easy to use and share hardware with relatively low-spec hardware. Although the case where shearing is used has been described here, the viewpoint conversion can be performed using the same approach as described above when using affine transformation such as rotation, enlargement / reduction, or translation.
　下側領域１１２の視点変換が終わったら、視点変換画像生成部２４０は、視点変換後の下側領域１１２と視点変換を行わなかった上側領域１１１とを合わせて視点変換画像（視点変換後の左画像）を生成する。視点変換部２００で生成された視点変換画像は、対応点探索部３００で右画像と対比されて対応点が抽出される。抽出された対応点の座標は、図１４に示すようにカメラ幾何校正部４００による逆変換で原画像における座標に戻されてカメラ幾何校正に利用される。 When the viewpoint conversion of the lower region 112 is completed, the viewpoint conversion image generation unit 240 combines the lower region 112 after the viewpoint conversion and the upper region 111 that has not been subjected to the viewpoint conversion into a viewpoint conversion image (the left side after the viewpoint conversion). Image). The viewpoint conversion image generated by the viewpoint conversion unit 200 is compared with the right image by the corresponding point search unit 300 to extract a corresponding point. The coordinates of the extracted corresponding points are returned to the coordinates in the original image by the inverse transformation by the camera geometric calibration unit 400 as shown in FIG. 14, and are used for camera geometric calibration.
　このように、画像を遠景や立体物を含んだ上側領域１１１と、路面が大半を占める下側領域１１２に分割して対応点を得ることで、上側領域１１１では視点変換無しの画像から対応点を抽出し、下側領域１１２では視点変換有りの画像から対応点を抽出することとなり、上下から密な対応点を得ることができる。なお、視点変換部２００においては、視点変換のパラメータ生成から視点変換画像生成と、逆変換パラメータの計算までを実施する。逆変換パラメータは特徴点を統合して利用するカメラ幾何校正部４００で利用される。 As described above, by dividing the image into the upper region 111 including a distant view or a three-dimensional object and the lower region 112 occupying the majority of the road surface, corresponding points are obtained. Is extracted in the lower area 112, and corresponding points are extracted from the image with the viewpoint conversion, and dense corresponding points can be obtained from above and below. The viewpoint conversion unit 200 performs processing from generation of a viewpoint conversion parameter to generation of a viewpoint conversion image and calculation of an inverse conversion parameter. The inverse transformation parameter is used in the camera geometric calibration unit 400 that integrates and uses feature points.
　＜図３　変換パラメータ生成部＞
　図３に示すように変換パラメータ生成部２２０は、視差解析部２２１と、属性決定部２２２と、変換パラメータ演算部２２３を備えている。 <Fig. 3 Conversion parameter generator>
As illustrated in FIG. 3, the conversion parameter generation unit 220 includes a parallax analysis unit 221, an attribute determination unit 222, and a conversion parameter calculation unit 223.
　視差解析部２２１は、領域設定部２１０によって視点変換を行う領域として設定された領域の視差を、視差画像生成部５００で生成された前フレーム（例えば１フレーム前）の視差画像から取得し、その視差を解析することで当該領域内に平面に近似可能な部分が存在するか否かを判断する。 The parallax analysis unit 221 acquires the parallax of the region set as the region for performing the viewpoint conversion by the region setting unit 210 from the parallax image of the previous frame (for example, one frame before) generated by the parallax image generation unit 500, By analyzing the parallax, it is determined whether or not a portion that can be approximated to a plane exists in the region.
　属性決定部２２２は、視差解析部２２１の解析結果に基づいて当該領域の平面属性を決定する。平面属性としては、まず、当該領域に平面に近似可能な部分が存在することを示す「平面」と、当該領域には平面に近似可能な部分が存在しないことを示す「非平面」がある。前者の「平面」の属性には、さらに、当該領域の種別を示す属性として「路面（地面）」や「壁面（壁）」がある。後者の「非平面」の属性には当該領域の種別を示す属性として「無限遠」がある。また、各領域には予め平面属性が付与されている場合がある。
  この場合、属性決定部２２２は、予め付与された平面属性が妥当か否かを視差解析部２２１の解析結果に基づいて判断し、妥当と判断した領域のうち左右カメラ１００，１１０からの距離が所定の閾値未満の領域の平面属性を「平面」と決定し、その領域を視点変換の対象領域として決定する。距離が所定の閾値未満という条件を付することで、必要な場合にだけ、すなわち左右画像で同じ物体の見え方が大きく異なる場合にだけ視点変換を行うことができる。 The attribute determining unit 222 determines the plane attribute of the area based on the analysis result of the parallax analyzing unit 221. The plane attributes include “plane” indicating that there is a portion that can be approximated to a plane in the area, and “non-plane” indicating that there is no part that can be approximated to the plane in the area. The former “plane” attribute further includes “road surface (ground)” and “wall surface (wall)” as attributes indicating the type of the area. The latter attribute of “non-planar” includes “infinity” as an attribute indicating the type of the area. In addition, each region may be given a plane attribute in advance.
In this case, the attribute determination unit 222 determines whether the plane attribute given in advance is appropriate based on the analysis result of the parallax analysis unit 221, and the distance from the left and right cameras 100 and 110 in the area determined to be appropriate is determined. The plane attribute of the area smaller than the predetermined threshold is determined to be “plane”, and the area is determined as the target area of the viewpoint conversion. By applying the condition that the distance is less than the predetermined threshold, viewpoint conversion can be performed only when necessary, that is, only when the appearance of the same object is significantly different between the left and right images.
　変換パラメータ演算部２２３は、属性決定部２２２で平面属性が平面であると決定された領域を視点変換するための変換パラメータを演算する。変換パラメータは公知の方法で算出できる。ここではその一例を説明する。 The conversion parameter calculation unit 223 calculates a conversion parameter for performing viewpoint conversion on an area whose plane attribute is determined to be a plane by the attribute determination unit 222. The conversion parameter can be calculated by a known method. Here, an example will be described.
　＜視点変換計算＞
　まず、左カメラ１００から見た変換対象領域内の平面４隅の３次元座標を計算する。簡単な手法としては、路面平面が高さ０ｃｍにあると仮定すると共に、カメラ内部パラメータ（ｆ，ｋｎ，Ｃｎ）、外部パラメータ（ｒｎｎ，ｔｎ）、世界座標原点に対する左カメラ１００の設置位置・姿勢が既知とすると、下記式（１）の右辺における第１及び第２の行列が既知となる。更に同式右辺の右端に位置する４要素を持つベクトルのＹworldの世界座標が０であることから、未知の数字が世界座標のＸworldとＺworldのみで表現できる。画像座標は変換する領域を自ら設定しているため既知とできる。このようにして、変換したい領域内の平面４隅の画像座標を式（１）に入力すれば、その４隅の３次元の世界座標を演算できる。次に右辺右端のＸworld，Ｙworld，Ｚworldに演算で求めた４隅の３次元座標を順に設定する。そして外部パラメータの行列に世界座標原点から見た右カメラの位置・姿勢を設定すると、世界座標系の４点の位置を右カメラで見た場合の画像座標へ変換できる。このようにして視点変換したい領域内の平面の４隅の画像座標を求めて４角形の変換パラメータを得る。４隅が計算できれば、その４角形内部の座標は全て内挿で算出できる。 <Viewpoint conversion calculation>
First, the three-dimensional coordinates of the four corners of the plane in the conversion target area viewed from the left camera 100 are calculated. As a simple method, it is assumed that the road surface plane is at a height of 0 cm, the camera internal parameters (f, kn, Cn), the external parameters (rnn, tn), and the installation position / posture of the left camera 100 with respect to the world coordinate origin. Is known, the first and second matrices on the right side of the following equation (1) are known. Further, since the world coordinate of Yworld of a vector having four elements located at the right end of the right side of the equation is 0, unknown numbers can be represented only by Xworld and Zworld of world coordinates. The image coordinates can be known because the region to be transformed is set by itself. In this way, if the image coordinates of the four corners of the plane in the area to be transformed are input to equation (1), the three-dimensional world coordinates of the four corners can be calculated. Next, the three-dimensional coordinates of the four corners calculated by calculation are sequentially set to Xworld, Yworld, and Zworld at the right end of the right side. When the position and orientation of the right camera as viewed from the origin of the world coordinates are set in the matrix of external parameters, the positions of four points in the world coordinate system can be converted into image coordinates as viewed by the right camera. In this way, the image coordinates of the four corners of the plane in the region to be subjected to the viewpoint conversion are obtained to obtain the quadrangle conversion parameters. If the four corners can be calculated, all the coordinates inside the quadrangle can be calculated by interpolation.
Figure JPOXMLDOC01-appb-M000001
Figure JPOXMLDOC01-appb-M000001
　変換パラメータの生成に際しては、まず、左画像（視点変換対象の画像）内で視点変換に利用する平面を決定する必要がある。例えば、上記のように左画像を上下２分割した場合であれば、視差解析部２２１において、路面が撮像されることが多い下側領域１１２は平面に近似可能な部分と推定される。各領域に平面に近似可能な部分が含まれているか否かはステレオ画像から得られる視差を利用して解析可能である。左カメラ１００を中心に考えて３次元測量したとすると、左カメラ１００から見た路面平面の位置を視差を利用して解析することができる。下側領域１１２に予め「路面」という平面属性が付与されており、解析結果からその属性が妥当と判断された場合には、下側領域１１２は視点変換の対象と決定される。これとは反対に、その属性が妥当でないと判断された場合には、下側領域１１２は視点変換の対象から除外される。すなわち視点変換を実施しないで対応点探索されることで誤った視点変換を抑制する。領域の平面属性として「路面」や「壁」といった属性が決定された場合には、変換パラメータ演算部２２３は当該領域の変換パラメータを演算する。 When generating the conversion parameters, first, it is necessary to determine a plane used for viewpoint conversion in the left image (image to be converted). For example, if the left image is divided into upper and lower parts as described above, the parallax analyzer 221 estimates that the lower region 112 where the road surface is often imaged is a portion that can be approximated to a plane. Whether or not each area includes a portion that can be approximated to a plane can be analyzed using parallax obtained from a stereo image. Assuming that the three-dimensional survey is performed with the left camera 100 as the center, the position of the road surface viewed from the left camera 100 can be analyzed using the parallax. A plane attribute “road surface” is assigned to the lower region 112 in advance, and when the attribute is determined to be valid from the analysis result, the lower region 112 is determined to be a viewpoint conversion target. Conversely, if it is determined that the attribute is not valid, the lower area 112 is excluded from viewpoint conversion. That is, erroneous viewpoint conversion is suppressed by searching for a corresponding point without performing viewpoint conversion. When an attribute such as “road surface” or “wall” is determined as the plane attribute of the area, the conversion parameter calculation unit 223 calculates the conversion parameter of the area.
　＜図４　対応点探索部３００＞
　対応点探索部３００は、視点変換部２００で視点変換された左画像（視点変換画像）と、右画像の原画像とを対比して両者の対応点を探索する。なお、視点変換部２００で左右画像に対して視点変換がされた場合には視点変換後の左右画像から対応点を探索する。図４に示すように対応点探索部３００は、特徴点抽出部３１０と、特徴量記述部３２０と、最大誤差設定部３３０と、対応点探索部３４０と、信頼度計算部３５０として機能し得る。 <FIG. 4 Corresponding point search unit 300>
The corresponding point search unit 300 searches for a corresponding point between the left image (viewpoint converted image) subjected to viewpoint conversion by the viewpoint conversion unit 200 and the original image of the right image. In addition, when the viewpoint conversion is performed on the left and right images by the viewpoint conversion unit 200, a corresponding point is searched from the left and right images after the viewpoint conversion. As shown in FIG. 4, the corresponding point search unit 300 can function as a feature point extraction unit 310, a feature amount description unit 320, a maximum error setting unit 330, a corresponding point search unit 340, and a reliability calculation unit 350. .
　特徴点抽出部３１０は、左画像において特徴点を抽出する。特徴点は、例えば画像上の物体の角（コーナー）等のユニークな点である。なお、左右画像のいずれか一方で特徴点を抽出すればよく、右画像で特徴点を抽出しても良い。 The feature point extracting unit 310 extracts a feature point from the left image. A feature point is a unique point such as a corner of an object on an image. Note that the feature point may be extracted from one of the left and right images, and the feature point may be extracted from the right image.
　特徴量記述部３２０は、特徴点抽出部３１０で抽出した特徴点の周囲輝度変化を数値化した特徴量の記述を実施する。左画像上の特徴点・特徴量を基準にして右画像の対応点を探索する場合には、右画像の探索範囲を視差分だけ広げた領域に設定してもよい。 The feature value description unit 320 describes a feature value obtained by quantifying a change in the surrounding luminance of the feature point extracted by the feature point extraction unit 310. When searching for the corresponding point of the right image based on the feature points / features on the left image, the search range of the right image may be set to an area expanded by parallax.
　最大誤差設定部３３０は、対応点探索を実施する前に、左右カメラ１００，１１０の最大縦誤差発生範囲を考慮して、画像の縦方向における探索範囲（縦探索範囲）を設定する。例えば左画像から特徴点を得る場合には、縦探索範囲は右画像に設定される。左右画像の平行化が完璧であれば、左画像の特徴点の対応点は、右画像における同一高さの横一列を探索すればよい。しかしながら、利用する部品の温度特性や組み付け精度によって縦誤差の範囲が異なる。このような場合に縦最大誤差を定義すると、左画像から抽出された特徴点の対応点は、右画像における同一座標から±縦最大誤差の範囲内で対応点探索を実施すれば足りる。最大誤差設定部３３０において、このように縦探索範囲を限定し、また領域も限定することによって、対応点探索の候補数を大幅に減らすことができる。縦の探索範囲を最大誤差設定部３３０において削減し、横方向は左画像の分割領域の視差最大値で設定する。これにより対応点探索部３４０が探索すべき特徴点の候補を削減できる。 The maximum error setting unit 330 sets a search range (vertical search range) in the vertical direction of the image in consideration of the maximum vertical error occurrence range of the left and right cameras 100 and 110 before performing the corresponding point search. For example, when obtaining feature points from the left image, the vertical search range is set to the right image. If the parallelization of the left and right images is perfect, the corresponding points of the feature points of the left image may be searched in the horizontal row of the same height in the right image. However, the range of the vertical error differs depending on the temperature characteristics and the assembling accuracy of the components to be used. In such a case, if the vertical maximum error is defined, it is sufficient to perform a corresponding point search for the corresponding point of the feature point extracted from the left image within the range of ± vertical maximum error from the same coordinate in the right image. By limiting the vertical search range and the area in the maximum error setting unit 330 in this way, the number of candidates for corresponding point search can be significantly reduced. The vertical search range is reduced by the maximum error setting unit 330, and the horizontal direction is set by the maximum parallax value of the divided region of the left image. Thereby, the feature point candidates to be searched by the corresponding point search unit 340 can be reduced.
　対応点探索部３４０は、特徴点について計算された特徴量の類似性を比較しながら右画像で対応点探索を実施する。通常は１つの特徴点について対応点の候補が複数発見されるが、その中でも最も類似性が高くかつ所定の閾値以上の類似性のある候補点を対応点とする。図１４の例では、左右画像を上側領域と下側領域の２つの分割したうえで、上側領域（無限遠）では視点変換無しの画像から対応点を抽出し、下側領域（路面）では視点変換有りの画像から対応点を抽出する。これにより、視点変換を実施しない従前の場合と比較して、上下領域から多数の密な対応点を得ることができる。 The corresponding point search unit 340 performs a corresponding point search on the right image while comparing the similarities of the feature amounts calculated for the feature points. Normally, a plurality of candidate corresponding points are found for one feature point. Among them, a candidate point having the highest similarity and having a similarity equal to or more than a predetermined threshold value is set as the corresponding point. In the example of FIG. 14, the left and right images are divided into an upper region and a lower region, and corresponding points are extracted from the image without viewpoint conversion in the upper region (infinity), and the viewpoint is extracted in the lower region (road surface). The corresponding points are extracted from the converted image. Thereby, compared with the conventional case in which the viewpoint conversion is not performed, it is possible to obtain a large number of dense corresponding points from the upper and lower regions.
　信頼度計算部３５０は、対応点探索部３４０で得られた対応点の類似性の高さや対応点個数などから、当該領域が後段のカメラ幾何に利用してよい領域かどうかを判断するための指標値である信頼度を計算する。信頼度が低い場合には、その領域にカメラ幾何での利用不可の判定を下す。全ての領域において信頼度を計算してカメラ幾何への利用可否の判定を実施する。或る閾値以上の数の領域で利用不可とされた場合には、今回取得した画像ではキャリブレーション（幾何校正）ができないと判断する。 The reliability calculation unit 350 determines whether or not the area is an area that can be used for the camera geometry at the subsequent stage based on the similarity of the corresponding points obtained by the corresponding point search unit 340 and the number of corresponding points. Calculate the reliability, which is an index value. If the reliability is low, it is determined that the area cannot be used in the camera geometry. The reliability is calculated for all the regions to determine whether the region can be used for camera geometry. If it is determined that the area cannot be used in a number of areas equal to or greater than a certain threshold value, it is determined that calibration (geometric calibration) cannot be performed on the image acquired this time.
　＜図５　カメラ幾何校正部＞
　カメラ幾何校正部４００は、対応点探索部３００で得られた複数の対応点を基に左右画像が平行になるように左右カメラ１００，１１０の幾何校正を実施する。図５に示すように、カメラ幾何校正部４００は、対応点逆変換補正部４１０と、対応点集約部４２０と、ノイズ対応点削除部４３０と、幾何校正パラメータ推定部４４０と、利用可否判定部４５０と、幾何校正反映部４６０として機能し得る。 <Figure 5 Camera geometry calibration unit>
The camera geometric calibration unit 400 performs geometric calibration of the left and right cameras 100 and 110 based on the plurality of corresponding points obtained by the corresponding point search unit 300 so that the left and right images are parallel. As shown in FIG. 5, the camera geometric calibration unit 400 includes a corresponding point inverse transformation correction unit 410, a corresponding point aggregation unit 420, a noise corresponding point deletion unit 430, a geometric calibration parameter estimation unit 440, and an availability determination unit. 450, and may function as the geometric calibration reflection unit 460.
　まず、対応点逆変換補正部４１０においては、視点変化や、画像変形を利用して得られた対応点の座標を、原画像上の座標系に戻す計算を実施する。本実施形態では原画像の座標系に戻すために逆変換パラメータを利用する。逆変換のパラメータは、既に逆変換パラメータ計算部２３０で求められており、左右画像のうち視点変換された画像（左画像）の対応点の座標を逆変換パラメータを利用して原画像の座標に逆変換する。図１４に左画像の下側領域に視点変換を実施した場合の逆変換の方法について示す。この図の例では３段のうち中断の画像図に示すように左画像の下側領域だけに変形を加えて視点変換している。この状態で上側領域と下側領域の双方で対応点探索を実施する。対応点探索の後、下側領域では対応点（特徴点）の座標を視点変形前の座標に基に戻す逆変換（座標変換）を実施し、変形前の画像における対応点の位置を算出する。すなわち視点変換後の画像に基づいて発見した多数の対応点を原画像上の座標に逆変換した上で幾何校正に利用する。 First, the corresponding point inverse transformation correction unit 410 performs a calculation to return the coordinates of the corresponding point obtained by using the viewpoint change or the image deformation to the coordinate system on the original image. In the present embodiment, an inverse transformation parameter is used to return to the coordinate system of the original image. The parameters of the inverse transformation have already been obtained by the inverse transformation parameter calculation unit 230, and the coordinates of the corresponding points of the viewpoint-transformed image (left image) of the left and right images are converted to the coordinates of the original image using the inverse transformation parameters. Perform inverse conversion. FIG. 14 shows an inverse conversion method when the viewpoint conversion is performed on the lower area of the left image. In the example of this figure, the viewpoint is changed by modifying only the lower region of the left image as shown in the interruption image among the three stages. In this state, a corresponding point search is performed in both the upper region and the lower region. After the corresponding point search, in the lower area, inverse transformation (coordinate transformation) is performed to return the coordinates of the corresponding point (feature point) to the coordinates before the viewpoint transformation, and the position of the corresponding point in the image before transformation is calculated. . In other words, a number of corresponding points found based on the image after the viewpoint conversion are inversely transformed into coordinates on the original image and then used for geometric calibration.
　このように対応点逆変換補正部４１０において視点変換（変形）した領域の対応点座標を逆変換して原画像の座標系上に移動させ、対応点集約部４２０は左右画像の全ての対応点を原画像の座標系における対応点として集約する。 In this way, the corresponding point inverse transformation correction unit 410 inversely transforms the corresponding point coordinates of the area subjected to the viewpoint transformation (deformation) and moves the coordinate on the coordinate system of the original image. Are aggregated as corresponding points in the coordinate system of the original image.
　次に、対応点集約部４２０が集めた対応点を利用してカメラ幾何校正を実施したいが、集約した段階の対応点には誤対応点も含まれる。そこで対応点のノイズ成分除去を実施する。まず、得られる対応点が画面全体に散らばるような所定組数（ここでは８組とする）の対応点をランダムに抽出し、その８組の対応点を基に左右画像の対応関係を数学的に示す基礎行列を計算する。これを基礎行列の基となる対応点を変えて複数回繰り返す。その結果、多くの基礎行列を得ることができるが、８組の対応点に誤対応点が混入している場合には、算出された基礎行列が真の値からはずれることになる。反対に、８組の対応点が全て正しい場合には、基礎行列が類似した値に集約される。このため類似した基礎行列を出力した対応点は信頼できる対応点とし、類似しないはずれ値の基礎行列を出力した対応点は８組中どの対応点が誤対応点なのかは不明のためその後の処理では利用しない。ただし、別の８組として選ばれた際に、基礎行列が類似した値に集約され、はずれ値でないことがわかった際に利用するものとする。 (4) Next, it is desired to perform camera geometric calibration using the corresponding points collected by the corresponding point aggregating unit 420. However, the corresponding points at the stage of aggregation include erroneous corresponding points. Therefore, the noise component of the corresponding point is removed. First, a predetermined number of sets of corresponding points (here, eight sets) corresponding to the obtained corresponding points are scattered throughout the screen, and the correspondence between the left and right images is mathematically determined based on the eight sets of corresponding points. Calculate the fundamental matrix shown in This is repeated a plurality of times while changing the corresponding points serving as the base matrix. As a result, many basic matrices can be obtained, but when the erroneous corresponding points are mixed in the eight corresponding points, the calculated basic matrices deviate from the true values. Conversely, if all eight sets of corresponding points are correct, the underlying matrices are aggregated into similar values. For this reason, the corresponding points that output similar basic matrices are regarded as reliable corresponding points, and the corresponding points that output fundamental matrices with dissimilar outliers are unclear which of the eight corresponding points are erroneous corresponding points. Do not use. However, when it is selected as another eight sets, the base matrix is aggregated into similar values, and is used when it is found that it is not an outlier.
　このようにして得られたはずれ値以外の基礎行列の中から、ある評価尺度を決定し、最も評価値が高かった基礎行列を初期値に利用する。評価尺度は例えば、基礎行列生成に利用しなかった８組の対応点を除く信頼のおける対応点のペアを更にランダムで選択し、その対応点のペアが基礎行列にどの程度の誤差を生じるかなどを評価尺度とする。このようにして得られた基礎行列を左右カメラ１００，１１０の対応関係を示す基礎行列の初期値としてまずは設定し、更には、信頼できると判断された対応点を利用して高精度な幾何校正パラメータの最適化を実施する。 (4) A certain evaluation scale is determined from among the fundamental matrices other than the outliers obtained in this way, and the fundamental matrix having the highest evaluation value is used as an initial value. The evaluation scale is, for example, a random selection of a pair of reliable corresponding points excluding the eight corresponding points not used for the generation of the basic matrix, and how much error the pair of corresponding points causes in the basic matrix. Is used as an evaluation scale. The basic matrix obtained in this manner is first set as an initial value of the basic matrix indicating the correspondence between the left and right cameras 100 and 110, and further, highly accurate geometric calibration is performed using the corresponding points determined to be reliable. Perform parameter optimization.
　上記の手法で得られた基礎行列を初期値として利用して、画像上の対応点と、基礎行列を利用して算出された推定点との距離誤差をコスト関数として最小化する最適化問題を幾何校正パラメータ推定部４４０にて解く。これにより８点法と比較して高精度な基礎行列（幾何校正パラメータ）を推定できる。 Using the basic matrix obtained by the above method as an initial value, an optimization problem that minimizes the distance error between the corresponding point on the image and the estimated point calculated using the basic matrix as a cost function The geometric calibration parameter estimating unit 440 solves the problem. This makes it possible to estimate a basic matrix (geometric calibration parameter) with higher accuracy than the 8-point method.
　次に、利用可否判定部４５０は、まず、対応点集約部４２０から得られる対応点の数（対応点数は所定数を超えているか）と、ノイズ対応点削除部４３０から得られるはずれ値以外の対応点のペア数（ペア数は所定数を超えているか）、幾何校正パラメータ推定部４４０から得られる最小化された距離誤差の大きさ（距離誤差の大きさは所定値未満か）等の外部情報を利用して、幾何校正パラメータ推定部４４０によるカメラ幾何校正の結果を利用して良いか否かを判定する。更に、求められた幾何校正パラメータを利用して左右カメラ画像を平行化した場合、求められていた対応点ペアのうち平行化後の左右画像座標において縦誤差が発生していない対応点ペアの存在比率が所定の割合（例えば９５％）を超えているかどうかで利用可否を判定する。 Next, the availability determining unit 450 first determines the number of corresponding points obtained from the corresponding point aggregating unit 420 (whether the number of corresponding points exceeds a predetermined number) and the outliers other than the outliers obtained from the noise corresponding point deleting unit 430. External information such as the number of pairs of corresponding points (whether the number of pairs exceeds a predetermined number) and the magnitude of the minimized distance error obtained from the geometric calibration parameter estimating unit 440 (whether the magnitude of the distance error is less than a predetermined value) Using the information, it is determined whether or not the result of the camera geometric calibration by the geometric calibration parameter estimation unit 440 can be used. Furthermore, when the left and right camera images are parallelized using the obtained geometric calibration parameters, there is a corresponding point pair in which a vertical error does not occur in the left and right image coordinates after the parallelization among the determined corresponding point pairs. The availability is determined based on whether the ratio exceeds a predetermined ratio (for example, 95%).
　幾何校正反映部４６０では、利用可否判定部４５０において利用可能と判定された場合において、幾何校正パラメータ推定部４４０で推定された左右カメラ１００，１１０の幾何を示すパラメータ基礎行列を利用して平行化画像を生成するための画像変形のアフィンテーブルの更新を実行する。 In the geometric calibration reflection unit 460, when the availability is determined by the availability determination unit 450, parallelization is performed using the parameter base matrix indicating the geometry of the left and right cameras 100 and 110 estimated by the geometric calibration parameter estimation unit 440. Update the affine table of the image transformation for generating the image.
　＜図６　視差画像生成部５００＞
　視差画像生成部５００は、左右カメラ１００，１１０で撮像された左右画像と、カメラ幾何校正部４００でリアルタイムに更新される最新のアフィンテーブルに基づいて視差画像を生成する。本実施形態の視差画像生成部５００は、図６に示すように平行化画像生成部５１０と、ステレオマッチング部５２０と、距離演算部５３０として機能する。 <FIG. 6 Parallax image generation unit 500>
The parallax image generation unit 500 generates a parallax image based on the left and right images captured by the right and left cameras 100 and 110 and the latest affine table updated in real time by the camera geometry correction unit 400. The parallax image generation unit 500 according to the present embodiment functions as a parallelized image generation unit 510, a stereo matching unit 520, and a distance calculation unit 530, as shown in FIG.
　平行化画像生成部５１０は、カメラ幾何校正部４００にて更新された平行化画像生成のためのアフィンテーブルを利用して左右平行化画像を生成する。ステレオマッチング部５２０は、平行化された左右画像に対してステレオマッチングを実施して視差画像を生成する。距離演算部５３０は、左右カメラ１００，１１０の基線長や、カメラの内部パラメータ（焦点距離やセルサイズ）を利用して視差画像から３次元の距離変換を実施することで左右画像上の任意の物体までの距離を演算する。 The parallelized image generation unit 510 generates a left-right parallelized image using the affine table for generating a parallelized image updated by the camera geometric calibration unit 400. The stereo matching unit 520 performs stereo matching on the parallelized left and right images to generate a parallax image. The distance calculation unit 530 performs a three-dimensional distance conversion from the parallax image using the base line length of the left and right cameras 100 and 110 and the internal parameters (focal length and cell size) of the camera, thereby obtaining an arbitrary value on the left and right images. Calculate the distance to the object.
　＜図１５　制御装置１０の処理フローチャート＞
　ここで上記のように左画像を上下に２分割した場合に制御装置１０が実行する処理フローについて説明する。制御装置１０は所定の周期で図１５に示す一連の処理を繰り返している。 <FIG. 15 Processing flowchart of control device 10>
Here, a processing flow executed by the control device 10 when the left image is vertically divided into two as described above will be described. The control device 10 repeats a series of processes shown in FIG. 15 at a predetermined cycle.
　ステップＳ０１では、まず、制御装置１０（視点変換部２００）は左右カメラ（ステレオカメラ）１００，１１０で撮像された左右画像を入力する。 In Step S01, first, the control device 10 (viewpoint conversion unit 200) inputs the left and right images captured by the left and right cameras (stereo cameras) 100 and 110.
　ステップＳ０２では、制御装置１０（視点変換部２００）は、ステップＳ０４で入力した左画像を上下２分割することを決定する。なお、右画像については分割を行わない。 In step S02, the control device 10 (viewpoint conversion unit 200) determines to divide the left image input in step S04 into upper and lower parts. Note that the right image is not divided.
　ステップＳ０３では、制御装置１０（視点変換部２００）は、左画像を上下に２分割し、そのうち消失点ＶＰを含む領域を上側領域１１１と設定する。上側領域１１１は消失点ＶＰを含み遠方の景色が撮像される傾向のある領域であり、視点変換を施されることなくそのまま対応点探索に利用される。 In step S03, the control device 10 (viewpoint conversion unit 200) divides the left image into two parts vertically and sets an area including the vanishing point VP as the upper area 111. The upper region 111 is a region including the vanishing point VP and in which a distant scene tends to be imaged, and is used for the corresponding point search without performing the viewpoint conversion.
　ステップＳ０４では、制御装置１０（視点変換部２００）は、左画像から上側領域１１１を除いた領域（上側領域１１１の下方に位置する領域）を下側領域１１２と設定し、ステップＳ０５に処理を移行する。下側領域１１２は、自車が走行する路面が撮像物の大半を占める領域であり、左右カメラ１００，１１０から比較的近傍の路面では左右カメラ１００，１１０で見え方の変化が特に大きいので視点変換の実施のために分割する。 In step S04, the control device 10 (viewpoint conversion unit 200) sets a region obtained by removing the upper region 111 from the left image (a region located below the upper region 111) as a lower region 112, and proceeds to step S05. Transition. The lower area 112 is an area in which the road surface on which the vehicle travels occupies the majority of the imaged object, and on the road surface relatively close to the left and right cameras 100 and 110, the change in the appearance of the left and right cameras 100 and 110 is particularly large. Split to perform conversion.
　ステップＳ０５では、制御装置１０（視点変換部２００）は、下側領域１１２を視点変換するための変換パラメータの生成と、視点変換後の下側領域１１２上の対応点を逆変換により視点変換前の座標上に戻すための逆変換パラメータを生成する。ここで生成される変換パラメータによる視点変換は、下側領域１１２の少なくとも一部が平面であると仮定し、その平面に対する左右カメラ１００，１１０の位置及び姿勢を推定し、その推定した左右カメラ１００，１１０の位置及び姿勢に基づいて下側領域１１２を右カメラ１１０の視点からの画像に変換するものである。 In step S05, the control device 10 (viewpoint conversion unit 200) generates a conversion parameter for performing the viewpoint conversion of the lower region 112, and converts the corresponding point on the lower region 112 after the viewpoint conversion by the inverse conversion before the viewpoint conversion. Generate an inverse transformation parameter for returning to the coordinates of. The viewpoint conversion based on the conversion parameters generated here assumes that at least a part of the lower region 112 is a plane, estimates the positions and orientations of the left and right cameras 100 and 110 with respect to the plane, and estimates the estimated left and right cameras 100 and 110. , 110 is converted into an image from the viewpoint of the right camera 110 based on the position and orientation of the right camera 110.
　ステップＳ０６では、制御装置１０（視点変換部２００）は、ステップＳ０５で生成した変換パラメータを利用して左画像の下側領域１１２を視点変換する。 In step S06, the control device 10 (viewpoint conversion unit 200) converts the viewpoint of the lower area 112 of the left image using the conversion parameters generated in step S05.
　ステップＳ０７では、制御装置１０（視点変換部２００）は、ステップＳ０３で分割した上側領域１１１と、ステップＳ０６で視点変換した下側領域１１２を合わせた視点変換画像を生成する。そして、制御装置１０（対応点探索部３００）は、その視点変換画像とステップＳ０１で入力した右画像に対して対応点探索を実施する。すなわち、変形を実施していない上側領域１１１と変形を実施した下側領域１１２の画像上の特徴点及び特徴量を基準として右画像の対応点を探索する処理が実行される。これにより、視点変換後の左画像の下側領域１１２と、視点変換未実施の右画像の下側領域１１２とから、特徴点及び特徴量を基準として複数の対応点の集合である第１対応点群が抽出され、視点変換前の左画像の上側領域１１１と、視点変換未実施の右画像の上側領域１１１とから、特徴点及び特徴量を基準として複数の対応点の集合である第２対応点群が抽出される。 In step S07, the control device 10 (viewpoint conversion unit 200) generates a viewpoint conversion image in which the upper region 111 divided in step S03 and the lower region 112 converted in step S06 are combined. Then, the control device 10 (corresponding point search unit 300) performs a corresponding point search on the viewpoint converted image and the right image input in step S01. That is, a process of searching for a corresponding point in the right image based on the feature points and feature amounts on the image of the upper region 111 that has not been transformed and the lower region 112 that has been transformed is executed. As a result, the first correspondence that is a set of a plurality of corresponding points based on the feature points and the feature amounts is obtained from the lower area 112 of the left image after the viewpoint conversion and the lower area 112 of the right image that has not been subjected to the viewpoint conversion. A point group is extracted, and from the upper region 111 of the left image before the viewpoint conversion and the upper region 111 of the right image without the viewpoint conversion, a second set of a plurality of corresponding points based on the feature points and the feature amounts is obtained. A corresponding point group is extracted.
　ステップＳ０８では、制御装置１０（カメラ幾何校正部４００）は、ステップＳ０７で下側領域１１２から発見された複数の対応点（第１対応点群）のうち上下の領域１１１，１１２に跨がったものは排除し、残りの対応点についてはステップＳ０５で生成した逆変換パラメータを利用して逆変換し、当該残りの対応点の座標値を原画像（ステップＳ０１で入力した左画像）で対応点をとった場合の座標値に戻す。 In step S08, the control device 10 (camera geometric calibration unit 400) straddles the upper and lower regions 111 and 112 of the plurality of corresponding points (first corresponding point group) found in the lower region 112 in step S07. The remaining corresponding points are excluded and the inverse transformation is performed using the inverse transformation parameter generated in step S05, and the coordinate values of the remaining corresponding points are corresponded to the original image (the left image input in step S01). Returns to the coordinate value when the point was taken.
　ステップＳ０９では、制御装置１０（カメラ幾何校正部４００）は、ステップＳ０７で発見された上側領域１１１上の複数の対応点（第２対応点群）の座標値と、ステップＳ０８で逆変換された下側領域１１２上の複数の対応点（第１対応点群）の座標値を集約する。このとき、第１対応点群のうち左画像上の対応点の座標は視点変換前の座標に逆変換した座標となり、第１対応点群のうち右画像上の対応点の座標は視点変換未実施の元々の座標となる。また、第２対応点群の座標は左右画像ともに視点変換前の座標となる。これにより上下側領域１１１，１１２すべての対応点の座標系が、左右の原画像での座標系で統一的に扱うことが可能になる。 In step S09, the control device 10 (camera geometric calibration unit 400) inversely transforms the coordinate values of the plurality of corresponding points (second corresponding point group) on the upper region 111 found in step S07 in step S08. The coordinate values of a plurality of corresponding points (first corresponding point group) on the lower area 112 are collected. At this time, the coordinates of the corresponding point on the left image in the first corresponding point group are the coordinates obtained by inversely converting the coordinates before the viewpoint conversion, and the coordinates of the corresponding point on the right image in the first corresponding point group are the coordinates of the unconverted viewpoint. The original coordinates of the implementation. Further, the coordinates of the second corresponding point group are the coordinates before the viewpoint conversion for both the left and right images. As a result, the coordinate system of all the corresponding points in the upper and lower areas 111 and 112 can be treated uniformly with the coordinate system of the left and right original images.
　ステップＳ１０では、制御装置１０（カメラ幾何校正部４００）は、ノイズ除去を実施する。ステップＳ０９で集約された対応点のなかから、画像上で散らばる座標系になるようにかつランダムに８組の対応点ペアを選択し、選択した対応点ペア（入力対応点）を基にいわゆる８点法により基礎行列の値を計算する。そして、その基礎行列の値をベースにはずれ値とならなかった基礎行列の入力対応点を、その後の処理で利用できるようにフラグを立ててはずれ値となった入力対応点と区別する。 In step S10, the control device 10 (camera geometric calibration unit 400) performs noise removal. From the corresponding points aggregated in step S09, eight corresponding point pairs are selected at random so as to form a coordinate system scattered on the image, and a so-called eight corresponding point pair (input corresponding point) is selected based on the selected corresponding point pair (input corresponding point). Calculate the values of the fundamental matrix by the point method. Then, the input corresponding points of the basic matrix that did not become outliers based on the values of the basic matrix are distinguished from the input corresponding points that became outliers by setting a flag so that they can be used in subsequent processing.
　ステップＳ１１では、制御装置１０（カメラ幾何校正部４００）は、ステップＳ１０でノイズと判定されなかった対応点の座標を利用し、幾何校正のパラメータ推定を実施する。上記の手法で得られた基礎行列を初期値として利用して、画像上の対応点と、基礎行列を利用して算出された推定点との距離誤差をコスト関数として最小化する最適化問題を解く。これにより８点法と比較して高精度な幾何校正パラメータを演算できる。 In step S11, the control device 10 (camera geometric calibration unit 400) estimates the parameters of the geometric calibration using the coordinates of the corresponding point not determined as noise in step S10. Using the basic matrix obtained by the above method as an initial value, an optimization problem that minimizes the distance error between the corresponding point on the image and the estimated point calculated using the basic matrix as a cost function solve. Thereby, it is possible to calculate a geometric calibration parameter with higher accuracy than the 8-point method.
　ステップＳ１２では、制御装置１０（カメラ幾何校正部４００）は、ステップＳ１１で演算した距離誤差の大きさが所定値未満であるか、対応点の数が所定値以上か等の情報を利用して、ステップＳ１１で演算した幾何校正パラメータを利用して良いか判定する。ここで幾何校正パラメータが利用可能と判断された場合にはステップＳ１３に処理を進める。一方、利用不可と判断された場合には、アフィンテーブルを更新せずにステップＳ１４に処理を進める。 In step S12, the control device 10 (camera geometric calibration unit 400) uses information such as whether the magnitude of the distance error calculated in step S11 is less than a predetermined value or whether the number of corresponding points is equal to or more than a predetermined value. It is determined whether the geometric calibration parameters calculated in step S11 can be used. If it is determined that the geometric calibration parameters can be used, the process proceeds to step S13. On the other hand, if it is determined that the affine table cannot be used, the process proceeds to step S14 without updating the affine table.
　ステップＳ１３では、制御装置１０（カメラ幾何校正部４００）は、前フレームに利用していた左右画像の平行化のためのアフィンテーブルをステップＳ１１で演算した幾何校正パラメータに基づいて更新する。 In step S13, the control device 10 (camera geometric calibration unit 400) updates the affine table for parallelizing the left and right images used in the previous frame based on the geometric calibration parameters calculated in step S11.
　ステップＳ１４では、制御装置１０（視差画像生成部５００）は、記憶されているアフィンテーブルを利用して左右画像の平行化画像を生成し、これを利用してステレオマッチングを実施し視差画像を生成する。 In step S14, the control device 10 (the parallax image generation unit 500) generates a parallelized image of the left and right images using the stored affine table, performs stereo matching using the generated parallelized image, and generates a parallax image. I do.
　上記のように較正した本実施形態の車載環境認識装置では、左右カメラ１００，１１０（左右画像）で見え方の大きく異なる領域（下側領域１１２）に視点変換を施すことで、左右画像から発見される対応点の数を大幅に増加した。すなわち、カメラ１００により撮像された左画像の下側領域１１２を、右カメラ１１０の視点からの見え方に近づくように変形することで、視点変換前よりも数の多い密な対応点の探索結果を対応点探索部３００で得ることができる。この密な対応点を利用することで、カメラ幾何校正部４００による処理では高精度な幾何校正が可能となる。幾何校正により正確に平行化された左右のペア画像を利用することで、視差画像生成部５００における高密度かつ高精度な視差画像の生成が可能となる。 In the in-vehicle environment recognizing device of the present embodiment calibrated as described above, the left and right cameras 100 and 110 (left and right images) perform viewpoint conversion on a region (lower region 112) that is greatly different in appearance, thereby finding from the left and right images. The number of corresponding points to be significantly increased. That is, by deforming the lower area 112 of the left image captured by the camera 100 so as to approach the view from the viewpoint of the right camera 110, a search result of a denser corresponding point having a larger number than before the viewpoint conversion is performed. Can be obtained by the corresponding point search unit 300. The use of the close correspondence points enables highly accurate geometric calibration in the processing by the camera geometric calibration unit 400. By using the paired right and left images accurately parallelized by the geometric calibration, the parallax image generation unit 500 can generate a high-density and high-precision parallax image.
　なお、上記では左右画像を上側領域と下側領域に分割し、下側領域については左画像に視点変換を実施してから対応点を探索し、上側領域については左右画像ともに視点変換を実施することなく対応点を探索したが、左右画像を複数の領域に分割する構成は必須ではない。例えば、視点変換後の左画像と視点変換未実施の右画像から複数の対応点（第１対応点群）を抽出し、視点変換前の左画像と視点変換未実施の右画像から複数の対応点（第２対応点群）を抽出して左右カメラ１００，１１０の幾何校正を行っても良い。 In the above description, the left and right images are divided into an upper region and a lower region. For the lower region, the viewpoint conversion is performed on the left image, and then the corresponding point is searched. For the upper region, the viewpoint conversion is performed on both the left and right images. Although the corresponding points were searched without any problem, the configuration for dividing the left and right images into a plurality of regions is not essential. For example, a plurality of corresponding points (first corresponding point group) are extracted from the left image after the viewpoint conversion and the right image without the viewpoint conversion, and a plurality of correspondence points are extracted from the left image before the viewpoint conversion and the right image without the viewpoint conversion. The points (second corresponding point group) may be extracted and geometric calibration of the left and right cameras 100 and 110 may be performed.
　＜視点変換の変形例＞
　視点変換部２００による視点変換の他の例について説明する。上記ではカメラ画像を上下に２分割する方式について説明したが、６分割にする方式や、自由な領域に分割する方式が利用可能である。 <Modification of viewpoint conversion>
Another example of viewpoint conversion by the viewpoint conversion unit 200 will be described. In the above description, the method of dividing the camera image into two vertically has been described. However, a method of dividing the camera image into six or a method of dividing the camera image into free areas can be used.
　（１）６分割による視点変換
　カメラの幾何校正は、画像上の全体から対応点が取れたほうが精度が向上する。ある固定箇所から密に対応点が得られたような場合に、そのまま対応点を利用すると対応点が密に得られた部分だけの幾何校正が小さくなるように計算されるため、画像全体から満遍なく対応点が得られたほうが本来の幾何校正すべき値に近い値が得られることが多い。しかし、画像全体において特徴点の抽出、特徴量の記述、対応点の探索などを実行すると処理負荷が重くなる。 (1) Viewpoint conversion by six divisions In the geometric calibration of the camera, the accuracy is improved when the corresponding points are obtained from the entire image. When corresponding points are obtained densely from a certain fixed point, if the corresponding points are used as they are, the geometric calibration of only the parts where the corresponding points are densely obtained is calculated so as to be small, so the entire image is evenly distributed. When the corresponding point is obtained, a value closer to the original value to be geometrically calibrated is often obtained. However, when a feature point is extracted, a feature amount is described, and a corresponding point is searched for in the entire image, the processing load becomes heavy.
　そこで図１３に示すように画像を６領域に分割し、この６領域に関して左右画像上で対応点を探す方法を選定する。この方法では、制御装置１０は、左画像を６つの矩形領域に分割し、その６つの矩形領域には、自車走行中に各矩形領域に出現すると予測される平面属性が付与されており、その６つの矩形領域に付与された平面属性が妥当か否かを前フレームの視差画像を基に判断し、その６つの矩形領域において平面属性が妥当と判断された矩形領域のうち、左右カメラ１００，１１０の距離が所定の閾値未満の矩形領域を変換対象領域に決定し、その変換対象領域に対する左右カメラ１００，１１０の位置及び姿勢を推定し、その推定した左右カメラ１００，１１０の位置及び姿勢に基づいて変換対象領域を右カメラの視点からの画像に変換し、その６つの領域から変換対象領域を除いた残りの領域と変換対象領域を合わせたものを視点変換後の左画像とする。６つの矩形領域は、左画像を縦に２つ、横に３つに分割して得られ、２行３列で各矩形領域が配置されている。
  画像を上段と下段の２段に分け、上段の左側から第１領域、第２領域、第３領域と称し、下段の左側から第４領域、第５領域、第６領域と称する。６つの矩形領域における下側の３つの矩形領域（第４－６領域）の平面属性は「路面」であり、６つの矩形領域における上側の３つの矩形領域のうち左右に位置する２つの矩形領域（第１、第３領域）の平面属性は「壁」であり、６つの矩形領域における上側の３つの矩形領域のうち中央に位置する矩形領域（第２領域）の平面属性は「無限遠」である。 Therefore, as shown in FIG. 13, the image is divided into six regions, and a method of searching for corresponding points on the left and right images for the six regions is selected. In this method, the control device 10 divides the left image into six rectangular areas, and the six rectangular areas are given a plane attribute that is predicted to appear in each rectangular area while the host vehicle is traveling. It is determined based on the parallax image of the previous frame whether or not the plane attributes given to the six rectangular areas are valid. Of the six rectangular areas whose plane attributes have been determined to be valid, the left and right cameras 100 , 110 are determined as conversion target regions, the positions and orientations of the left and right cameras 100 and 110 with respect to the conversion target region are estimated, and the estimated positions and orientations of the left and right cameras 100 and 110 are determined. The conversion target area is converted into an image from the viewpoint of the right camera based on the left image, and a left image after the viewpoint conversion is obtained by combining the remaining area excluding the conversion target area from the six areas and the conversion target area. To. The six rectangular regions are obtained by dividing the left image into two vertically and three horizontally, and each rectangular region is arranged in two rows and three columns.
The image is divided into an upper stage and a lower stage, which are referred to as a first region, a second region, and a third region from the left side of the upper stage, and are referred to as a fourth region, a fifth region, and a sixth region from the left side of the lower stage. The plane attribute of the lower three rectangular regions (the fourth to sixth regions) in the six rectangular regions is “road surface”, and two rectangular regions located on the left and right of the upper three rectangular regions in the six rectangular regions. The plane attribute of the (first and third regions) is “wall”, and the plane attribute of the center rectangular region (second region) of the upper three rectangular regions in the six rectangular regions is “infinity”. It is.
　この方法は、特徴点抽出、記述の処理時間の削減、対応点探索候補の絞込みに効果がある。
  この方法による視点変換に際して、自車走行中に各領域に出現すると予測される平面属性を決定しておくと、６領域毎に変形量を選択しやすい。例えば、図１３に示す一例のように、基本を考えると、下段の３つの領域（第４－６領域）は従来どおり路面（地面）と仮定した視点変換による視点変換を実施する。上側の無限遠（消失点）辺りにある真ん中の領域（第２領域）は遠景や空しか入っていないために、変形無しとする。上側の左右の領域（第１，第３領域）であるが、これは走行路の風景にもよるが、街中を走行中などは、建築物や樹木などが走行路に対して「壁」のように左右に存在することが多々ある。このような場合には、走行路左右に存在する壁を想定した変換を実施する。路面や壁などを或る固定値を想定して変換してもよいし、図１３の真ん中の図のように、壁の横位置と回転だけを、前フレームの視差画像から推定しておいてそれを利用するような手法でも良い。
  分割した領域内の視差値を距離に変換して平面推定を実施し、はずれ値の多さや、最終推定平面からある距離の範囲内にどれだけの何％の視差点群が占有するかなどを利用して平面に近似してよいかの判断を実施しても良い。これにより平面近似が可能と判別された場合には、この平面が２つのカメラ視点から見て近距離かつ視点を変えたことによる見え方の違いが大きいかどうかも計算する。見え方の違いが小さい場合にはそもそも視点変換の必要性が低い。見え方の違いが大きい場合に視点変換を行うと、対応点の探索性能が大幅に上がるため、多少の誤差が視点変換にあったとしても変換前の原画像で対応点探索を実施するよりも大幅に密な対応点の取得が可能となる。 This method is effective in extracting feature points, reducing the processing time of description, and narrowing down corresponding point search candidates.
At the time of viewpoint conversion by this method, if the plane attribute predicted to appear in each area while the own vehicle is running is determined, it is easy to select the deformation amount for each of the six areas. For example, as in an example shown in FIG. 13, considering the basics, the lower three regions (the fourth to sixth regions) perform the viewpoint conversion based on the viewpoint conversion assumed to be the road surface (ground) as before. Since the middle area (second area) near the upper infinity (vanishing point) contains only a distant view or the sky, no deformation occurs. The upper left and right areas (the first and third areas) depend on the scenery of the running path, but when traveling in the city, buildings, trees, and the like are "walled" against the running path. There are many things that exist on the left and right. In such a case, conversion is performed assuming walls existing on the left and right of the travel path. The road surface or the wall may be converted assuming a certain fixed value, or only the lateral position and the rotation of the wall are estimated from the parallax image of the previous frame as shown in the middle diagram of FIG. A method that utilizes this may be used.
Convert the disparity values in the divided area into distances and perform plane estimation, and determine the amount of outliers and how many percent of disparity points occupy within a certain distance from the final estimated plane. Judgment as to whether or not it may be approximated to a plane may be performed. If it is determined that the plane can be approximated, it is also calculated whether this plane is close to the two camera viewpoints and the difference in appearance due to the change of the viewpoint is large. If the difference in appearance is small, the need for viewpoint conversion is low in the first place. If viewpoint conversion is performed when the difference in appearance is large, the search performance of the corresponding point is greatly improved, so even if there is some error in the viewpoint conversion, it is better than performing the corresponding point search on the original image before conversion. Significantly closer correspondence points can be obtained.
　図１３の各矩形領域内に示したように、自車走行中に各矩形領域に出現すると予測される平面属性が予め各領域に付されている。例えば、上段の左右の領域（第１，第３領域）は「壁」、下段の３つの領域（第４－６領域）は「路面」、上段の中央の領域（第２領域）は「無限遠」という平面属性が付されており、前フレームの視差値から推定される平面がこれらの属性が規定する平面と類似しているか否かを判定する。予め付された平面属性と異なる場合には、うまく平面近似できなかったと仮定し、視点変換を実施しない。これにより間違えた視点変換を避けることが可能となる。また、予め平面属性を決めているため、不安定要素となるはずれ値除去も比較的実施しやすく、推定するパラメータが絞り込まれるので安定性が強化される。 平面 As shown in each rectangular area in FIG. 13, a plane attribute predicted to appear in each rectangular area while the vehicle is traveling is given to each area in advance. For example, the upper left and right regions (first and third regions) are “walls”, the lower three regions (regions 4 to 6) are “road surfaces”, and the upper central region (second region) is “infinity”. The plane attribute “far” is attached, and it is determined whether or not the plane estimated from the parallax value of the previous frame is similar to the plane defined by these attributes. If the plane attribute is different from the pre-assigned plane attribute, it is assumed that the plane cannot be approximated well, and the viewpoint conversion is not performed. This makes it possible to avoid erroneous viewpoint conversion. In addition, since the plane attribute is determined in advance, it is relatively easy to remove an outlier that is an unstable element, and the parameters to be estimated are narrowed, so that the stability is enhanced.
　なお、上記のように６つの矩形領域に分割した場合、上段の左右に位置する２つの領域（第１，第３領域）は、走行路に沿った建物や樹木などがある場合に壁相当の平面（平面属性が「壁」の領域）があることを想定した視点変換を利用可能である。しかし、田舎道で走行路の周囲の立体物が少ない場合等にはその２つの領域に「平面」の属性を付与することなく、例えば「無限遠」等と非平面の属性を付与した方が対応点が多くなる場合もある。このため前フレームの視差画像から各領域に現れる風景の３次元構造の傾向を理解し、その後に視点変換の実施の有無を決定して対応点探索を実施しても良い。 When divided into six rectangular areas as described above, the two areas (first and third areas) located on the left and right of the upper row are equivalent to walls when there are buildings and trees along the traveling path. It is possible to use viewpoint conversion assuming that there is a plane (a region whose plane attribute is “wall”). However, when there are few three-dimensional objects around the road on a rural road, it is better to assign a non-planar attribute, such as "infinity", to the two regions without assigning the attribute of "plane". The number of corresponding points may increase. For this reason, it is possible to understand the tendency of the three-dimensional structure of the scenery appearing in each region from the parallax image of the previous frame, and then determine whether or not to perform the viewpoint conversion and perform the corresponding point search.
　（２）自由領域分割による視点変換
　また、ＧＰＵなどを活用した視点変換を実施する場合には、３次元の平面にカメラ画像を貼って変形すればよいので、矩形領域にこだわらずに自由な領域でカメラ画像を分割できる。そこで図１６に示すように各領域の境界線に斜め線も利用した領域分割を実施する。前フレームの視差画像から得られる情報（例えば、路面領域推定結果と路端領域推定結果）に基づいて画像内に平面に近似可能な部分が存在するか否かを判断する。そして、平面に近似可能な部分と判断された部分を含む領域を路面平面領域と推定する。このとき当該領域から左右カメラ１００，１１０までの距離が閾値未満か否かを判定し、距離が閾値未満の領域を路面平面領域、すなわち視点変換を行う領域として設定しても良い。また同様に、同一物体について左右カメラ１００，１１０で見え方が大きく異なる領域であるか否かを判定しても良い。 (2) Viewpoint conversion by free region division When performing viewpoint conversion utilizing a GPU or the like, a camera image may be pasted on a three-dimensional plane and deformed. To split the camera image. Therefore, as shown in FIG. 16, a region division is performed using a diagonal line as a boundary line of each region. Based on information obtained from the parallax image of the previous frame (for example, a road surface area estimation result and a road edge area estimation result), it is determined whether or not a portion that can be approximated to a plane exists in the image. Then, a region including a portion determined to be a portion that can be approximated to a plane is estimated as a road surface plane region. At this time, it may be determined whether or not the distance from the area to the left and right cameras 100 and 110 is less than a threshold, and the area where the distance is less than the threshold may be set as a road surface area, that is, an area for performing viewpoint conversion. Similarly, it may be determined whether or not the same object is in a region where the left and right cameras 100 and 110 look greatly different.
　また、路端領域の近辺には建築物や樹木がたっていることから、走行路の進行方向に沿って壁があると仮定した領域分割を画像の色合いと３次元の両方を活用して、領域分割を実施しても良い。分割した領域のそれぞれについて、３次元の平面に近似可能か否かを上記６領域の場合と同様に前フレームの視差画像から推定する。平面に近似可能な場合にはその平面に応じた視点変換を実施する。 In addition, since buildings and trees are standing near the roadside area, area division based on the assumption that there is a wall along the traveling direction of the traveling path is performed by using both the image color and the three-dimensional area. Division may be performed. For each of the divided areas, whether or not it can be approximated to a three-dimensional plane is estimated from the parallax image of the previous frame, as in the case of the six areas. If the plane can be approximated, the viewpoint conversion is performed according to the plane.
　先述の矩形領域よりも適切で柔軟な領域で画像を分割しているため、画面全体を比較的有効に活用することが可能となる。ただし、カメラ画像の背景が複雑な場合には、領域の分割が実施しづらく、安定的な判定は６領域のようにある程度、どのような平面になるかわかっている前提を利用するものの方が安定性が高い。 画像 Since the image is divided by a more appropriate and flexible area than the rectangular area described above, the entire screen can be used relatively effectively. However, if the background of the camera image is complicated, it is difficult to divide the area, and it is better to use a premise that knows what plane is to some extent, such as six areas, for stable determination. High stability.
　（３）左右カメラ双方の視点変換
　上記の例ではステレオカメラの一方のカメラ（左カメラ１００）の画像を他方のカメラ（右カメラ１１０）の視点からの画像に変換する場合について説明したが、ステレオカメラの両方のカメラ（左右カメラ１００，１１０）の画像を他の視点（例えば、左右カメラ１００，１１０間に位置する所定の視点）からの画像に変換しても良い。図１７の例では、左右画像を上下２分割にしている。そして、それぞれの下側領域を、左右カメラ１００，１１０の光軸の真ん中（すなわちステレオカメラの場合のベースラインの中央）に光軸を有する他のカメラの視点からの画像に視点変換している。 (3) Viewpoint Conversion of Both Left and Right Cameras In the above example, a case where an image of one camera (left camera 100) of a stereo camera is converted into an image from the viewpoint of the other camera (right camera 110) has been described. Images from both cameras (left and right cameras 100 and 110) may be converted to images from another viewpoint (for example, a predetermined viewpoint located between the left and right cameras 100 and 110). In the example of FIG. 17, the left and right images are divided into upper and lower parts. Then, each lower area is viewpoint-converted into an image from the viewpoint of another camera having an optical axis in the middle of the optical axes of the left and right cameras 100 and 110 (that is, the center of the baseline in the case of a stereo camera). .
　このように左右画像を視点変換すると、左右画像のそれぞれに視点変換による画質劣化が生じるため、同等の画質劣化状態でマッチングを実施でき、マッチングスコアの向上が見込める。更に、この方法の利点は環境認識装置が３つ以上のカメラを備え、複数カメラのペアから三角測量を実施するような場合であって、どのカメラ主体で３次元復元すべきかが明確でない多視点カメラによる３次元測量時に利用しやすい。このように本発明には図１４に示すような右カメラ主体の３次元復元である必要性はない。 視点 When the left and right images are viewpoint-converted in this way, image quality degradation occurs due to viewpoint conversion in each of the left and right images, so that matching can be performed in the same image quality degradation state, and an improvement in the matching score can be expected. Furthermore, this method has an advantage in a case where the environment recognition apparatus includes three or more cameras and performs triangulation from a pair of multiple cameras, and it is not clear which camera mainly performs three-dimensional reconstruction. It is easy to use for three-dimensional surveying with a camera. As described above, the present invention does not need to be a three-dimensional restoration mainly performed by the right camera as shown in FIG.
　・第２実施形態
　上記の実施形態では視差画像生成部５００における視差画像の生成に視点変換画像を利用しなかったが、視差画像の生成に視点変換画像を利用しても良い。本実施形態の制御装置１０は視差画像生成部５００Ａを備えている。他の部分については第１実施形態と同じであるとし、説明は省略する。 -2nd Embodiment Although the viewpoint conversion image was not used for the generation of the parallax image in the parallax image generation unit 500 in the above embodiment, the viewpoint conversion image may be used for the generation of the parallax image. The control device 10 of the present embodiment includes a parallax image generation unit 500A. The other parts are the same as in the first embodiment, and a description thereof will be omitted.
　＜図７　視差画像生成部＞
　図７に示す視差画像生成部５００Ａは、領域別視点変換平行化画像生成部５５０と、領域別マッチング部５６０と、結果統合部５７０と、距離演算部５８０を備えている。 <FIG. 7 Parallax image generation unit>
The parallax image generation unit 500A illustrated in FIG. 7 includes a region-based viewpoint conversion parallelized image generation unit 550, a region-based matching unit 560, a result integration unit 570, and a distance calculation unit 580.
　長基線長のステレオカメラでは、対応点探索と同様に視差画像生成時のステレオマッチングにおいても、近距離の被写体の見え方が左右カメラ１００，１１０で大きく異なり得る。このためステレオマッチングにおいても視差のマッチングが困難になるおそれがある。しかし、第１実施形態の対応点探索のようにステレオマッチング時にも視点変換を行えば、路面など平面で構成されるシーンについては、その平面の視差を密に得ることができる。また、対応点同様に遠方の風景に関しては変形しなくとも対応点がそれなりに得ることができる。路面形状を解析したり、小さな凹凸を解析したりする目的や、自車両がどこを走行してよいかなど路面平面を主に観測するような目的のアルゴリズムに対しては、対応点の手法と同様に左画像を視点変換し、左右画像のステレオマッチングを行うことが好ましい。 In a stereo camera with a long base line length, the right and left cameras 100 and 110 may differ in the appearance of a short-distance subject even in stereo matching at the time of generating a parallax image as in the case of a corresponding point search. For this reason, parallax matching may be difficult even in stereo matching. However, if viewpoint conversion is performed also during stereo matching as in the corresponding point search of the first embodiment, parallax of a plane such as a road surface can be obtained densely. Further, as with the corresponding points, the corresponding points can be obtained as appropriate for the distant scenery without deformation. For algorithms that analyze the road surface shape, analyze small irregularities, or mainly observe the road surface such as where the vehicle can travel, the corresponding point method is used. Similarly, it is preferable to convert the viewpoint of the left image and perform stereo matching of the left and right images.
　そこで、本実施形態では、まず、領域別視点変換平行化画像生成部５５０において、第１実施形態と同様に遠方と近傍で２つの領域（上側領域、下側領域）に左画像を分割する。そして、上側領域は視点変換を実施せず、下側領域は右カメラ視点への視点変換をアフィンテーブルによる平行化と同時に実施する。なお、平行化後に画像分割して視点変換を行っても良い。視点変換時の変換パラメータや平行化時（幾何校正時）のアフィンテーブルは第１実施形態と同様に視点変換部２００とカメラ幾何校正部４００で演算されたものを利用するものとする。次に、領域別マッチング部５６０において、画像生成部５５０で生成した２つの領域について個別に視差値を演算して個別に視差画像を生成する。結果統合部５７０において、領域別マッチング部５６０で演算された視差値のうち、下側領域に属する視差値（視差画像）には視点変換に応じた補正を施すことで上側領域と下側領域で視差値の意味合いを合わせる補正を実施し、その後に上側領域と下側領域の視差値（視差画像）の統合を実施する。更に、距離演算部５８０においては、左右カメラ１００，ｌ１０の基線長の情報と左右カメラ１００，１１０の内部パラメータの情報を利用して、結果統合部５７０で統合した視差画像から距離を演算する。これにより左右画像で見え方の大きく異なり得る下側領域（カメラから近距離の路面領域）については従前のステレオマッチングよりも多数の密な対応点に基づく視差画像が得られるので視差画像の精度が向上する。 Therefore, in the present embodiment, first, the region-specific viewpoint-converted parallelized image generation unit 550 divides the left image into two regions (upper region and lower region) in the far and near regions as in the first embodiment. In the upper region, the viewpoint conversion is not performed, and in the lower region, the viewpoint conversion to the right camera viewpoint is performed simultaneously with the parallelization by the affine table. Note that viewpoint conversion may be performed by dividing an image after parallelization. As the conversion parameters at the time of viewpoint conversion and the affine table at the time of parallelization (at the time of geometric calibration), those calculated by the viewpoint conversion unit 200 and the camera geometric calibration unit 400 are used as in the first embodiment. Next, the region-specific matching unit 560 calculates parallax values individually for the two regions generated by the image generation unit 550, and generates parallax images individually. In the result integrating unit 570, the parallax values (parallax images) belonging to the lower region among the parallax values calculated by the region-specific matching unit 560 are corrected according to the viewpoint conversion, so that the upper region and the lower region are corrected. Correction is performed to match the meaning of the parallax values, and then the parallax values (parallax images) of the upper region and the lower region are integrated. Further, the distance calculation unit 580 calculates the distance from the parallax images integrated by the result integration unit 570, using the information on the base line length of the left and right cameras 100 and 110 and the information on the internal parameters of the left and right cameras 100 and 110. As a result, a parallax image based on a greater number of densely corresponding points can be obtained in a lower region (a road surface region at a short distance from the camera), which can greatly differ in the appearance of the left and right images, so that the accuracy of the parallax image is improved. improves.
　ただし、大半が路面であっても下側領域の端に歩行者がいる場合などは、視点変換画像を利用したステレオマッチングでは、ほぼ路面付近に存在する物体は、密な視差画像を得ることができるが、反対に歩行者部分は視点変換しない方が視差画像を綺麗に得られることがある。そこで、別の実施形態として、下側領域については、領域分割及び平行化の後に視点変換無しで左右画像（つまり、右画像と視点変換前の左画像のペア（第１ペア））をマッチングした結果と、領域分割及び平行化の後に視点変換有りで左右画像（つまり、右画像と視点変換後の左画像のペア（第２ペア））をマッチングした結果の双方を生成し、２つのマッチング結果のうち、どれだけ左右画像が類似していたかを示すマッチングスコアが高い方を利用して視差値及び視差画像を生成しても良い。この際、視点変換有りのマッチング結果を逆変換した状態に戻し、更に、どれだけ左右画像が類似していたかを示すマッチングスコアも両方の場合の各視差値から参照可能な状態としておくと良い。この手法であれば、例えば下側領域内に歩行者などの立体物が存在しており視点変換後の画像によるステレオマッチングではマッチングスコアが低下することが回避されるので視差画像の精度の平均を向上できる。 However, if there is a pedestrian at the edge of the lower area even if the majority is on the road surface, dense parallax images can be obtained for objects almost near the road surface by stereo matching using the viewpoint conversion image. On the other hand, if the viewpoint is not changed in the pedestrian part, the parallax image may be obtained clearly. Therefore, as another embodiment, for the lower region, the left and right images (that is, the pair of the right image and the left image before the viewpoint conversion (first pair)) are matched without the viewpoint conversion after the region division and the parallelization. Both the result and the result of matching the left and right images (that is, the pair of the right image and the left image after the viewpoint conversion (second pair)) with the viewpoint conversion after the region division and parallelization are generated, and two matching results are generated. Among them, the parallax value and the parallax image may be generated by using the one with the higher matching score indicating how similar the left and right images are. At this time, it is preferable that the matching result with the viewpoint conversion is returned to the inversely converted state, and that the matching score indicating how similar the left and right images are similar can be referred to from each parallax value in both cases. According to this method, for example, a three-dimensional object such as a pedestrian is present in the lower region, and the stereo matching using the image after the viewpoint conversion can prevent the matching score from being reduced. Can be improved.
　＜図１８　制御装置１０による視差画像生成処理のフローチャート＞
　ここで上記のように視差画像生成時に左画像を上下に２分割した場合に制御装置１０（視差画像生成部５００Ａ）が実行する処理フローについて説明する。制御装置１０は視差画像の要求指令の入力に基づいて図１８に示す一連の処理を繰り返している。なお、図中のステップＤＳ０２－ＤＳ０４で実行されるキャリブレーション用の対応点探索や幾何校正パラメータの推定・更新の処理は、どのような手法を利用してもよく、図１５に示した第１実施形態の方法に限られず、公知の方法を利用しても構わない。また、ここでは、既に平行化のためのキャリブレーションは左右画像に実施済みとして、第１実施形態の視点変換を視差画像生成に応用する方法の一例を示す。 <FIG. 18 Flowchart of parallax image generation processing by control device 10>
Here, a processing flow executed by the control device 10 (the parallax image generation unit 500A) when the left image is vertically divided into two when generating the parallax image as described above will be described. The control device 10 repeats a series of processes illustrated in FIG. 18 based on the input of the parallax image request command. The process of searching for the corresponding points for calibration and estimating / updating the geometric calibration parameters performed in steps DS02 to DS04 in the figure may use any method, and the first method shown in FIG. The method is not limited to the method of the embodiment, and a known method may be used. Here, an example of a method of applying the viewpoint transformation of the first embodiment to the generation of a parallax image, assuming that the calibration for parallelization has already been performed on the left and right images, will be described.
　ステップＤＳ０１では、まず、制御装置１０（視差画像生成部５００Ａ）は左右カメラ（ステレオカメラ）１００，１１０で撮像された左右画像を入力する。 In step DS01, first, the control device 10 (the parallax image generation unit 500A) inputs the left and right images captured by the left and right cameras (stereo cameras) 100 and 110.
　ステップＤＳ０２では、制御装置１０（対応点探索部３００）は、左右画像の対応点を探索する。 In Step DS02, the control device 10 (corresponding point search unit 300) searches for corresponding points in the left and right images.
　ステップＤＳ０３では、制御装置１０（カメラ幾何校正部４００）は、左右画像の平行化を実施するための幾何校正パラメータを推定する。 In step DS03, the control device 10 (camera geometric calibration unit 400) estimates geometric calibration parameters for performing parallelization of the left and right images.
　ステップＤＳ０４では、制御装置１０（カメラ幾何校正部４００）は、ステップＤ０２で演算された幾何校正パラメータでもって、左右画像の平行化画像を作成する際に利用される幾何校正用のパラメータを更新する。このとき、左右カメラ１００，１１０の相対位置や姿勢の推定値や、路面とステレオカメラの位置姿勢のパラメータ更新も実施しても良い。 In step DS04, the control device 10 (camera geometric calibration unit 400) updates the geometric calibration parameters used when creating the parallelized image of the left and right images with the geometric calibration parameters calculated in step D02. . At this time, the estimated values of the relative positions and postures of the left and right cameras 100 and 110 and the parameters of the position and posture of the road surface and the stereo camera may be updated.
　ステップＤＳ０５では、制御装置１０（視差画像生成部５００Ａ）は、ステップＤＳ０１で入力した左画像を上下２分割することを決定する。なお、右画像については分割を行わない。 In step DS05, the control device 10 (the parallax image generation unit 500A) determines to divide the left image input in step DS01 into upper and lower parts. Note that the right image is not divided.
　ステップＤＳ０６では、制御装置１０（視差画像生成部５００Ａ）は、左画像を上下に２分割し、そのうち消失点ＶＰを含む領域を上側領域１１１と設定する。上側領域１１１は消失点ＶＰを含み遠方の景色が撮像される傾向のある領域であり、視点変換を施されることなくそのままステレオマッチング（ステップＤＳ０７）に利用される。ただし、特に下側領域１１２において路面以外の物体の視差を重要視するような場合や処理に余裕がある場合には、左画像の全てを上側領域に設定し、視点変換を実施することなく右画像とステレオマッチングを実施しても良い（ステップＤＳ０７）。 In step DS06, the control device 10 (the parallax image generation unit 500A) divides the left image into upper and lower parts, and sets the area including the vanishing point VP as the upper area 111. The upper region 111 is a region including the vanishing point VP and in which a distant scene tends to be captured, and is used for stereo matching (step DS07) without performing viewpoint conversion. However, especially when the parallax of an object other than the road surface is regarded as important in the lower region 112 or when there is room for processing, the entire left image is set to the upper region, and the right image is set without performing viewpoint conversion. Stereo matching with the image may be performed (step DS07).
　ステップＤＳ０８では、制御装置１０（視差画像生成部５００Ａ）は、左画像から上側領域１１１を除いた領域（上側領域１１１の下方に位置する領域）を下側領域１１２と設定する。下側領域１１２は、自車が走行する路面が撮像物の大半を占める領域であり、左右カメラ１００，１１０から比較的近傍の路面では左右カメラ１００，１１０で見え方の変化が特に大きいので視点変換の実施のために分割する。 In step DS08, the control device 10 (the parallax image generation unit 500A) sets a region obtained by removing the upper region 111 from the left image (a region located below the upper region 111) as the lower region 112. The lower area 112 is an area in which the road surface on which the vehicle travels occupies the majority of the imaged object, and on the road surface relatively close to the left and right cameras 100 and 110, the change in the appearance of the left and right cameras 100 and 110 is particularly large. Split to perform conversion.
　そして、制御装置１０（視差画像生成部５００Ａ）は、下側領域１１２を視点変換するための変換パラメータの生成と、視点変換後の下側領域１１２上の対応点を逆変換により視点変換前の座標上に戻すための逆変換パラメータを生成する。ここで生成される変換パラメータによる視点変換は、下側領域１１２の少なくとも一部が平面であると仮定し、その平面に対する左右カメラ１００，１１０の位置及び姿勢を推定し、その推定した左右カメラ１００，１１０の位置及び姿勢に基づいて下側領域１１２を右カメラ１１０の視点からの画像に変換するものである。 Then, the control device 10 (the parallax image generation unit 500A) generates a conversion parameter for performing the viewpoint conversion of the lower region 112 and performs the inverse conversion of the corresponding point on the lower region 112 after the viewpoint conversion before the viewpoint conversion. Generate an inverse transformation parameter for returning to coordinates. The viewpoint conversion based on the conversion parameters generated here assumes that at least a part of the lower region 112 is a plane, estimates the positions and orientations of the left and right cameras 100 and 110 with respect to the plane, and estimates the estimated left and right cameras 100 and 110. , 110 is converted into an image from the viewpoint of the right camera 110 based on the position and orientation of the right camera 110.
　さらに、制御装置１０（視差画像生成部５００Ａ）は、生成した変換パラメータを利用して左画像の下側領域１１２を視点変換する。 Furthermore, the control device 10 (the parallax image generation unit 500A) performs viewpoint conversion of the lower region 112 of the left image using the generated conversion parameters.
　なお、このステップＤＳ０８で生成する視点変換画像は、左画像の視点を右カメラ１１０の視点に変換したような視点変換画像を生成しても良いし。左右カメラ１００，１１０の中心位置にカメラがあると仮定したように左右カメラの画像を、ステレオカメラの重心位置に持っていったような画像変換をしても良い。 The viewpoint conversion image generated in step DS08 may be a viewpoint conversion image in which the viewpoint of the left image is converted to the viewpoint of the right camera 110. Assuming that the cameras are located at the center positions of the left and right cameras 100 and 110, image conversion may be performed as if the images of the left and right cameras were brought to the position of the center of gravity of the stereo camera.
　ステップＤＳ０７では、制御装置１０（視差画像生成部５００Ａ）は、ステップＤＳ０６の上側領域１１１とこれに対応する右画像の領域をステレオマッチングして視差値を演算と視差画像の生成を行う。これと同時に、ステップＤＳ０９では、制御装置１０（視差画像生成部５００Ａ）は、ステップＤＳ０８で視点変換した下側領域１１２とこれに対応する右画像の領域をステレオマッチングして視差値を演算と視差画像の生成を行う。 In step DS07, the control device 10 (the parallax image generation unit 500A) performs parallax calculation by performing stereo matching on the upper region 111 in step DS06 and the corresponding right image region to generate a parallax image. At the same time, in step DS09, the control device 10 (disparity image generation unit 500A) calculates the disparity value by performing stereo matching on the lower region 112 converted in viewpoint in step DS08 and the corresponding right image region. Generate an image.
　ステップＤＳ１０では、制御装置１０（視差画像生成部５００Ａ）は、ステップＤＳ１０で生成された視点変換画像ベースの視差値を、逆変換パラメータを利用して逆変換することで視差値の視点変換分の換算を実施する。 In step DS10, the control device 10 (the parallax image generation unit 500A) inversely transforms the parallax value based on the viewpoint conversion image generated in step DS10 by using an inverse conversion parameter, thereby obtaining the parallax value of the parallax value. Perform conversion.
　ステップＤＳ１１では、制御装置１０（視差画像生成部５００Ａ）は、視点変換前の平行化画像座標上における２枚の画像（左右画像）に重畳領域がある場合には、視点変換前の下側領域１１２と右画像の対応部分のマッチングスコアと、視点変換後の下側領域１１２と右画像の対応部分のマッチングスコアとを比較し、マッチングスコアが高い方の視差値を下側領域の視差値として選択する。これにより路面においては視点変換した場合の視差を優先的に利用し、立体物が存在する場合等には視点変換しない場合の視差を優先的に利用するようにスコアマッチングの比較が作用するので視差画像の精度が向上する。 In step DS11, the control device 10 (the parallax image generation unit 500A) determines that if two images (left and right images) on the parallelized image coordinates before the viewpoint conversion have a superimposed region, the lower region before the viewpoint conversion is performed. The matching score between the corresponding portion of the right image and the matching score of the corresponding portion of the right image is compared with the matching score of the lower region 112 and the corresponding portion of the right image after the viewpoint conversion, and the disparity value with the higher matching score is used as the disparity value of the lower region. select. As a result, the comparison of score matching acts to preferentially use the disparity when the viewpoint is changed on the road surface and preferentially use the disparity when the viewpoint is not changed when a three-dimensional object is present. Image accuracy is improved.
　ステップＤＳ１２では、制御装置１０（視差画像生成部５００Ａ）は、ステップＤＳ０７で生成した上側領域１１１の視差画像と、ステップＤＳ１１の比較を介して選択された下側領域１１２の視差画像を一枚の視差画像（合成視差画像）として合成することで、従来よりも密でノイズの少ない視差画像生成が可能となる。 In step DS12, the control device 10 (the parallax image generation unit 500A) combines the parallax image of the upper region 111 generated in step DS07 and the parallax image of the lower region 112 selected through the comparison in step DS11 into one sheet. By synthesizing a parallax image (combined parallax image), it is possible to generate a parallax image that is denser and has less noise than before.
　なお、本実施形態では、左画像を上下に分割する場合について説明したが、左右画像で見え方の大きく異なる領域であって平面に近似できる部分が含まれる領域であれば、視点変換による視差値の精度向上効果が望める。すなわち、上記で説明した下側領域とは異なる領域に視点変換を実施してステレオマッチングを行っても良く、この種の領域としては第１実施形態で視点変換を行う領域が含まれる。 Note that, in the present embodiment, the case where the left image is vertically divided is described. However, if the left image and the right image are regions that are significantly different in appearance and include a portion that can be approximated to a plane, the disparity value obtained by the viewpoint conversion is used. Can be expected to improve the accuracy of That is, stereo matching may be performed by performing viewpoint conversion on an area different from the lower area described above, and this type of area includes an area on which viewpoint conversion is performed in the first embodiment.
　＜その他＞
　図１５のフローチャートにおいてステップＳ１０，Ｓ１２は省略可能である。 <Others>
In the flowchart of FIG. 15, steps S10 and S12 can be omitted.
　本発明は，上記の各実施形態に限定されるものではなく，その要旨を逸脱しない範囲内の様々な変形例が含まれる。例えば，本発明は，上記の各実施形態で説明した全ての構成を備えるものに限定されず，その構成の一部を削除したものも含まれる。また，ある実施形態に係る構成の一部を，他の実施形態に係る構成に追加又は置換することが可能である。 The present invention is not limited to the above embodiments, and includes various modifications without departing from the gist of the present invention. For example, the present invention is not limited to those having all the configurations described in the above embodiments, but also includes those in which some of the configurations are deleted. Further, a part of the configuration according to one embodiment can be added to or replaced by the configuration according to another embodiment.
　また，上記の制御装置１０に係る各構成や当該各構成の機能及び実行処理等は，それらの一部又は全部をハードウェア（例えば各機能を実行するロジックを集積回路で設計する等）で実現しても良い。また，上記の制御装置１０に係る構成は，演算処理装置（例えばＣＰＵ）によって読み出し・実行されることで当該装置の構成に係る各機能が実現されるプログラム（ソフトウェア）としてもよい。当該プログラムに係る情報は，例えば，半導体メモリ（フラッシュメモリ，ＳＳＤ等），磁気記憶装置（ハードディスクドライブ等）及び記録媒体（磁気ディスク，光ディスク等）等に記憶することができる。 In addition, the components of the control device 10 and the functions and execution processes of the components are partially or wholly realized by hardware (for example, a logic for executing each function is designed by an integrated circuit). You may. The configuration of the control device 10 may be a program (software) that is read and executed by an arithmetic processing device (for example, a CPU) to realize each function of the configuration of the device. Information related to the program can be stored in, for example, a semiconductor memory (flash memory, SSD, etc.), a magnetic storage device (hard disk drive, etc.), a recording medium (magnetic disk, optical disk, etc.), and the like.
　また，上記の各実施形態の説明では，制御線や情報線は，当該実施の形態の説明に必要であると解されるものを示したが，必ずしも製品に係る全ての制御線や情報線を示しているとは限らない。実際には殆ど全ての構成が相互に接続されていると考えて良い。 Further, in the description of each of the above embodiments, the control lines and the information lines that are understood to be necessary for the description of the embodiment are shown, but all the control lines and the information lines related to the product are not necessarily used. It is not necessarily shown. In fact, it can be considered that almost all components are interconnected.
　１００…左カメラ，１１０…右カメラ，１１１…上側領域，１１２…下側領域，２００…視点変換部，２１０…領域設定部，２２０…変換パラメータ生成部，２２１…視差解析部，２２２…視点変換属性決定部，２２３…変換パラメータ演算部，２３０…逆変換パラメータ計算部，２４０…視点変換画像生成部，３００…対応点探索部，３１０…特徴点抽出部，３２０…特徴量記述部，３３０…最大誤差設定部，３４０…対応点探索部，３５０…信頼度計算部，４００…カメラ幾何校正部，４１０…対応点逆変換補正部，４２０…対応点集約部，４３０…ノイズ対応点削除部，４４０…幾何校正パラメータ推定部，４５０…利用可否判定部，４６０…幾何校正反映部，５００…視差画像生成部，５１０…平行化画像生成部，５２０…ステレオマッチング部，５３０…距離演算部 100 left camera, 110 right camera, 111 upper region, 112 lower region, 200 viewpoint conversion unit, 210 region setting unit, 220 conversion parameter generation unit, 221 parallax analysis unit, 222 viewpoint conversion Attribute determination unit, 223: conversion parameter calculation unit, 230: inverse conversion parameter calculation unit, 240: viewpoint conversion image generation unit, 300: corresponding point search unit, 310: feature point extraction unit, 320: feature amount description unit, 330 ... Maximum error setting unit, 340: Corresponding point searching unit, 350: Reliability calculating unit, 400: Camera geometric calibration unit, 410: Corresponding point inverse transformation correcting unit, 420: Corresponding point aggregating unit, 430: Noise corresponding point deleting unit, 440: geometric calibration parameter estimating unit, 450: availability determining unit, 460: geometric calibration reflecting unit, 500: parallax image generating unit, 510: parallelized image generating unit, 520: stereo Matching unit, 530 ... distance calculator

Claims (12)

  1. 　第１カメラ及び第２カメラと、
    　前記第１カメラによって撮像された第１画像と前記第２カメラによって撮像された第２画像の少なくとも一方を変形することで前記第１画像及び前記第２画像を共通の視点からの画像に変換する視点変換を行った後に複数の対応点を抽出し、前記視点変換前の前記第１画像及び前記第２画像における前記複数の対応点の座標を利用して前記第１カメラ及び前記第２カメラの幾何校正を行う制御装置と
    　を備えることを特徴とする車載環境認識装置。     A first camera and a second camera,
    The first image and the second image are converted into an image from a common viewpoint by deforming at least one of a first image captured by the first camera and a second image captured by the second camera. After performing the viewpoint conversion, a plurality of corresponding points are extracted, and the coordinates of the plurality of corresponding points in the first image and the second image before the viewpoint conversion are used for the first camera and the second camera. A vehicle environment recognition device comprising: a control device for performing geometric calibration.
  2. 　請求項１の車載環境認識装置において、
    　前記視点変換は、前記制御装置が、前記第１画像及び前記第２画像で重複して撮像された部分の少なくとも一部が平面であると仮定し、前記平面に対する前記第１カメラ及び前記第２カメラの位置及び姿勢を演算し、その演算した前記平面に対する前記第１カメラ及び前記第２カメラの位置及び姿勢に基づいて前記第１画像を前記第２カメラの視点からの画像に変換する視点変換であることを特徴とする車載環境認識装置。     The in-vehicle environment recognition device according to claim 1,
    In the viewpoint conversion, the control device may assume that at least a part of a portion captured in the first image and the second image in a redundant manner is a plane, and the first camera and the second camera with respect to the plane may be used. Viewpoint conversion for calculating the position and orientation of a camera and converting the first image into an image from the viewpoint of the second camera based on the calculated positions and orientations of the first camera and the second camera with respect to the plane; An in-vehicle environment recognition device, characterized in that:
  3. 　請求項２の車載環境認識装置において、
    　前記制御装置は、前記視点変換後の前記第１画像と前記視点変換が未実施の前記第２画像とから複数の対応点の集合である第１対応点群を抽出し、前記視点変換前の前記第１画像と前記視点変換が未実施の前記第２画像とから複数の対応点の集合である第２対応点群を抽出し、前記第１対応点群の座標を前記視点変換前の座標に変換した座標と、前記第２対応点群の座標とを利用して前記第１カメラ及び前記第２カメラの幾何校正を行うことを特徴とする車載環境認識装置。     The in-vehicle environment recognition device according to claim 2,
    The control device extracts a first corresponding point group, which is a set of a plurality of corresponding points, from the first image after the viewpoint conversion and the second image on which the viewpoint conversion has not been performed. A second corresponding point group, which is a set of a plurality of corresponding points, is extracted from the first image and the second image on which the viewpoint conversion has not been performed, and coordinates of the first corresponding point group are coordinates before the viewpoint conversion. The geometric calibration of the first camera and the second camera is performed using the coordinates converted into the coordinates and the coordinates of the second corresponding point group.
  4. 　請求項１の車載環境認識装置において、
    　前記視点変化は、前記制御装置が、前記第１画像にパラメータを変化させながらアフィン変換を施して複数の変換画像を生成し、前記複数の変換画像のそれぞれと前記第２画像の対応点の抽出を実施し、前記複数の変換画像の中で前記対応点の数が最も多くかつ所定の閾値以上の変換画像を、前記第１画像を前記第２カメラの視点からの画像に変換した画像とする視点変換であることを特徴とする車載環境認識装置。     The in-vehicle environment recognition device according to claim 1,
    In the viewpoint change, the control device performs affine transformation while changing parameters on the first image to generate a plurality of transformed images, and extracts corresponding points between each of the plurality of transformed images and the second image. And the converted image having the largest number of the corresponding points among the plurality of converted images and equal to or greater than a predetermined threshold value is an image obtained by converting the first image into an image from the viewpoint of the second camera. An in-vehicle environment recognition device, which is a viewpoint conversion.
  5. 　請求項１の車載環境認識装置において、
    　前記視点変換は、
    　前記制御装置が、
    　　前記第１画像を消失点を含む又は消失点と接する上側領域と前記上側領域の下方に位置する下側領域の２つの領域に上下に２分割し、
    　　前記下側領域の少なくとも一部が平面であると仮定し、
    　　前記平面に対する前記第１カメラ及び前記第２カメラの位置及び姿勢を推定し、
    　　その推定した前記平面に対する前記第１カメラ及び前記第２カメラの位置及び姿勢に基づいて前記下側領域を前記第２カメラの視点からの画像に変換し、
    　　その変換後の前記下側領域と前記上側領域を合わせたものを視点変換後の第１画像とする視点変換であることを特徴とする車載環境認識装置。     The in-vehicle environment recognition device according to claim 1,
    The viewpoint conversion includes:
    The control device,
    The first image is vertically divided into two regions, an upper region including a vanishing point or in contact with the vanishing point, and a lower region located below the upper region,
    Assuming that at least a portion of the lower region is planar,
    Estimating the position and orientation of the first camera and the second camera with respect to the plane,
    Converting the lower area to an image from the viewpoint of the second camera based on the position and orientation of the first camera and the second camera with respect to the estimated plane,
    An in-vehicle environment recognizing device, wherein viewpoint conversion is performed by combining the lower region and the upper region after the conversion with the first image after the viewpoint conversion.
  6. 　請求項１の車載環境認識装置において、
    　前記視点変換は、
    　前記制御装置が、
    　　前記第１画像を複数の領域に分割し、
    　　前記複数の領域内に平面に近似可能な部分が存在するか否かを前フレームの視差画像を基に判断し、
    　　前記複数の領域のうち前記平面に近似可能な部分が存在する領域のうち前記第１カメラ及び前記第２カメラからの距離が所定の閾値未満の領域を変換対象領域に決定し、
    　　前記変換対象領域に対する前記第１カメラ及び前記第２カメラの位置及び姿勢を推定し、
    　　その推定した前記第１カメラ及び前記第２カメラの位置及び姿勢に基づいて前記変換対象領域を前記第２カメラの視点からの画像に変換し、
    　　前記複数の領域から前記変換対象領域を除いた残りの領域と前記変換対象領域を合わせたものを視点変換後の第１画像とする視点変換であることを特徴とする車載環境認識装置。     The in-vehicle environment recognition device according to claim 1,
    The viewpoint conversion includes:
    The control device,
    Dividing the first image into a plurality of regions;
    Determine whether there is a portion that can be approximated to a plane in the plurality of regions based on the parallax image of the previous frame,
    Among the plurality of regions, a region having a portion that can be approximated to the plane exists, and a region whose distance from the first camera and the second camera is smaller than a predetermined threshold is determined as a conversion target region.
    Estimating the position and orientation of the first camera and the second camera with respect to the conversion target area,
    Converting the conversion target area into an image from the viewpoint of the second camera based on the estimated position and orientation of the first camera and the second camera;
    An in-vehicle environment recognizing device, wherein the viewpoint conversion is a first image after a viewpoint conversion is performed by combining a remaining region excluding the conversion target region from the plurality of regions and the conversion target region.
  7. 　請求項１の車載環境認識装置において、
    　前記視点変換は、
    　前記制御装置が、
    　　前記第１画像を複数の矩形領域に分割し、
    　　前記複数の矩形領域には、自車走行中に各矩形領域に出現すると予測される平面の属性が付与されており、
    　　前記複数の矩形領域に付与された平面の属性が妥当か否かを前フレームの視差画像を基に判断し、
    　　前記複数の矩形領域において前記平面の属性が妥当と判断された矩形領域のうち、前記第１カメラ及び前記第２カメラからの距離が所定の閾値未満の矩形領域を変換対象領域に決定し、
    　　前記変換対象領域に対する前記第１カメラ及び前記第２カメラの位置及び姿勢を推定し、
    　　その推定した前記第１カメラ及び前記第２カメラの位置及び姿勢に基づいて前記変換対象領域を前記第２カメラの視点からの画像に変換し、
    　　前記複数の領域から前記変換対象領域を除いた残りの領域と前記変換対象領域を合わせたものを視点変換後の第１画像とする視点変換であることを特徴とする車載環境認識装置。     The in-vehicle environment recognition device according to claim 1,
    The viewpoint conversion includes:
    The control device,
    Dividing the first image into a plurality of rectangular areas;
    The plurality of rectangular areas are provided with an attribute of a plane that is predicted to appear in each rectangular area during traveling of the own vehicle,
    Determine whether the attribute of the plane given to the plurality of rectangular areas is appropriate based on the parallax image of the previous frame,
    In the plurality of rectangular regions, among the rectangular regions in which the attribute of the plane is determined to be appropriate, a rectangular region whose distance from the first camera and the second camera is smaller than a predetermined threshold is determined as a conversion target region,
    Estimating the position and orientation of the first camera and the second camera with respect to the conversion target area,
    Converting the conversion target area into an image from the viewpoint of the second camera based on the estimated position and orientation of the first camera and the second camera;
    An in-vehicle environment recognizing device, wherein the viewpoint conversion is a first image after a viewpoint conversion is performed by combining a remaining region excluding the conversion target region from the plurality of regions and the conversion target region.
  8. 　請求項７の車載環境認識装置において、
    　前記複数の矩形領域は、前記第１画像を縦に２つ、横に３つに分割して得られる６つの矩形領域であり、
    　前記６つの矩形領域における下側の３つの矩形領域の平面の属性は路面であり、
    　前記６つの矩形領域における上側の３つの矩形領域のうち左右の２つの矩形領域の平面の属性は壁であり、
    　前記６つの矩形領域における上側の３つの矩形領域のうち中央の矩形領域の平面の属性は無限遠であることを特徴とする車載環境認識装置。     The in-vehicle environment recognition device according to claim 7,
    The plurality of rectangular areas are six rectangular areas obtained by dividing the first image into two vertically and three horizontally.
    The attribute of the plane of the lower three rectangular areas in the six rectangular areas is a road surface,
    The plane attribute of the left and right two rectangular areas of the upper three rectangular areas in the six rectangular areas is a wall,
    An on-vehicle environment recognizing device, wherein the attribute of the plane of the central rectangular area among the upper three rectangular areas in the six rectangular areas is infinity.
  9. 　請求項１の車載環境認識装置において、
    　前記制御装置は、前記第１画像及び前記第２画像を共通の視点からの画像に変換する視点変換を行った後にステレオマッチングして視差値を取得し、前記視差値に対して前記視点変換に応じた補正を施して視差画像を生成することを特徴とする車載環境認識装置。     The in-vehicle environment recognition device according to claim 1,
    The control device performs a viewpoint conversion for converting the first image and the second image into an image from a common viewpoint, obtains a parallax value by performing stereo matching, and performs the viewpoint conversion on the parallax value. An in-vehicle environment recognizing device that generates a parallax image by performing a corresponding correction.
  10. 　請求項５の車載環境認識装置において、
    　前記制御装置は、前記第２画像と前記視点変換後の第１画像をステレオマッチングして視差値を取得し、前記視差値のうち前記下側領域に属するものは前記視点変換に応じた補正を施して視差画像を生成することを特徴とする車載環境認識装置。     The in-vehicle environment recognition device according to claim 5,
    The control device obtains a parallax value by performing stereo matching on the second image and the first image after the viewpoint conversion, and among the parallax values, those belonging to the lower region perform correction according to the viewpoint conversion. An in-vehicle environment recognizing device that generates a parallax image by performing a parallax image processing.
  11. 　請求項１０の車載環境認識装置において、
    　前記制御装置は、前記第２画像と前記視点変換前の第１画像からなる第１ペアと、前記第２画像と前記視点変換後の第１画像からなる第２ペアの２つのペアでステレオマッチングを実施し、前記２つのペアのうちステレオマッチングのマッチングスコアが高い方を利用して視差画像を生成することを特徴とする車載環境認識装置。     The in-vehicle environment recognition device according to claim 10,
    The control device performs stereo matching on two pairs of a second pair including the second image and the first image before the viewpoint conversion and a second pair including the second image and the first image after the viewpoint conversion. Wherein the parallax image is generated using the higher matching score of the stereo matching among the two pairs.
  12. 　請求項１の車載環境認識装置において、
    　前記視点変換は、
    　前記制御装置が、前記第１画像を変形して前記第１カメラと前記第２カメラの間に位置する所定の視点からの画像に変換する視点変換と、前記第２画像を変形して前記所定の視点からの画像に変換する視点変換とを行う視点変換であることを特徴とする車載環境認識装置。     The in-vehicle environment recognition device according to claim 1,
    The viewpoint conversion includes:
    The control device transforms the first image to convert the image to an image from a predetermined viewpoint located between the first camera and the second camera, and transforms the second image to the predetermined image. And a viewpoint conversion for converting a viewpoint into an image from the viewpoint.
PCT/JP2019/026561 2018-07-18 2019-07-04 Vehicle-mounted environment recognition device WO2020017334A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201980046818.8A CN112424565B (en) 2018-07-18 2019-07-04 Vehicle-mounted environment recognition device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2018135236A JP7219561B2 (en) 2018-07-18 2018-07-18 In-vehicle environment recognition device
JP2018-135236 2018-07-18

Publications (1)

Publication Number Publication Date
WO2020017334A1 true WO2020017334A1 (en) 2020-01-23

Family

ID=69164301

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2019/026561 WO2020017334A1 (en) 2018-07-18 2019-07-04 Vehicle-mounted environment recognition device

Country Status (3)

Country Link
JP (1) JP7219561B2 (en)
CN (1) CN112424565B (en)
WO (1) WO2020017334A1 (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113810567A (en) * 2020-05-28 2021-12-17 三星电子株式会社 Electronic device for performing object detection and method of operating the same
WO2024004325A1 (en) * 2022-06-27 2024-01-04 日立Astemo株式会社 External world recognition device and external world recognition method

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7465671B2 (en) 2020-02-20 2024-04-11 株式会社Subaru Image processing device and image processing method
WO2024009386A1 (en) * 2022-07-05 2024-01-11 日立Astemo株式会社 Image processing device and image processing method

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001273488A (en) * 2000-03-27 2001-10-05 Toshiba Corp Obstacle detector
JP2003256875A (en) * 2002-02-27 2003-09-12 Fuji Heavy Ind Ltd Positional deviation adjusting device amd method for stereoscopic image, and stereoscopic type monitoring device
JP2005251123A (en) * 2004-03-08 2005-09-15 National Institute Of Advanced Industrial & Technology Image processor and image processing method
JP2009041972A (en) * 2007-08-07 2009-02-26 Toshiba Corp Image processing device and method therefor
JP2009276233A (en) * 2008-05-15 2009-11-26 Toyota Central R&D Labs Inc Parameter calculating apparatus, parameter calculating system and program
WO2016189878A1 (en) * 2015-05-27 2016-12-01 京セラ株式会社 Arithmetic logic device, camera device, vehicle and calibration method

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001243456A (en) * 2000-02-29 2001-09-07 Toshiba Corp Device and method for detecting obstacle
JP3897984B2 (en) * 2001-02-02 2007-03-28 三菱電機株式会社 Surveillance camera calibration method and road surveillance camera
JP2003209858A (en) * 2002-01-17 2003-07-25 Canon Inc Stereoscopic image generating method and recording medium
JP4414661B2 (en) * 2003-02-25 2010-02-10 オリンパス株式会社 Stereo adapter and range image input device using the same
JP2006054504A (en) * 2004-08-09 2006-02-23 Olympus Corp Image generating method and apparatus
JP2008187566A (en) * 2007-01-31 2008-08-14 Sanyo Electric Co Ltd Camera calibration apparatus and method and vehicle
JP5588812B2 (en) * 2010-09-30 2014-09-10 日立オートモティブシステムズ株式会社 Image processing apparatus and imaging apparatus using the same
JP5820787B2 (en) * 2012-08-30 2015-11-24 株式会社デンソー Image processing apparatus and program
JP2015119372A (en) * 2013-12-19 2015-06-25 株式会社日立製作所 Multi-camera photographing system and method of combining multi-camera photographic images
GB201409625D0 (en) * 2014-05-30 2014-07-16 Isis Innovation Vehicle localisation
CN107194972B (en) * 2017-05-16 2021-04-02 成都通甲优博科技有限责任公司 Camera calibration method and system

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001273488A (en) * 2000-03-27 2001-10-05 Toshiba Corp Obstacle detector
JP2003256875A (en) * 2002-02-27 2003-09-12 Fuji Heavy Ind Ltd Positional deviation adjusting device amd method for stereoscopic image, and stereoscopic type monitoring device
JP2005251123A (en) * 2004-03-08 2005-09-15 National Institute Of Advanced Industrial & Technology Image processor and image processing method
JP2009041972A (en) * 2007-08-07 2009-02-26 Toshiba Corp Image processing device and method therefor
JP2009276233A (en) * 2008-05-15 2009-11-26 Toyota Central R&D Labs Inc Parameter calculating apparatus, parameter calculating system and program
WO2016189878A1 (en) * 2015-05-27 2016-12-01 京セラ株式会社 Arithmetic logic device, camera device, vehicle and calibration method

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113810567A (en) * 2020-05-28 2021-12-17 三星电子株式会社 Electronic device for performing object detection and method of operating the same
WO2024004325A1 (en) * 2022-06-27 2024-01-04 日立Astemo株式会社 External world recognition device and external world recognition method

Also Published As

Publication number Publication date
JP7219561B2 (en) 2023-02-08
CN112424565B (en) 2023-01-03
JP2020012735A (en) 2020-01-23
CN112424565A (en) 2021-02-26

Similar Documents

Publication Publication Date Title
WO2020017334A1 (en) Vehicle-mounted environment recognition device
JP5870273B2 (en) Object detection apparatus, object detection method, and program
CN110348454B (en) Matching local image feature descriptors
US9153066B2 (en) Image processing device, imaging device, and image processing method
WO2019160032A1 (en) Three-dimensional measuring system and three-dimensional measuring method
CN110992356A (en) Target object detection method and device and computer equipment
US11184604B2 (en) Passive stereo depth sensing
JP2017526082A (en) Non-transitory computer-readable medium encoded with computer program code for causing a motion estimation method, a moving body, and a processor to execute the motion estimation method
JP5250847B2 (en) Image processing apparatus, information processing system, image processing method, and program
JP4701848B2 (en) Image matching apparatus, image matching method, and image matching program
CN111142514B (en) Robot and obstacle avoidance method and device thereof
US20170069072A1 (en) Image processing apparatus, image processing system, and image processing method
US10949988B2 (en) Information processing apparatus, information processing method, and program
JP2009186287A (en) Plane parameter estimating device, plane parameter estimating method, and plane parameter estimating program
CN114764823A (en) Self-correcting depth calculation method, system, medium, and depth image processing apparatus
JP2019120590A (en) Parallax value calculation device, parallax value calculation method and program
KR20220026423A (en) Method and apparatus for three dimesiontal reconstruction of planes perpendicular to ground
JP7195785B2 (en) Apparatus, method and program for generating 3D shape data
JP2017045124A (en) Parallax detection device
JP2001109879A (en) Device and method for image processing, and medium
US10417783B2 (en) Image processing apparatus, image processing method, and storage medium
US10453206B2 (en) Method, apparatus for shape estimation, and non-transitory computer-readable storage medium
KR20200057929A (en) Method for rectification of stereo images captured by calibrated cameras and computer program
JP2020173584A (en) Object detection device
JP2017084259A (en) Corresponding point search method and distance measuring device

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19838048

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19838048

Country of ref document: EP

Kind code of ref document: A1