WO2023139978A1 - Vehicle-mounted camera device, vehicle-mounted camera system, and image storage method - Google Patents
Vehicle-mounted camera device, vehicle-mounted camera system, and image storage method Download PDFInfo
- Publication number
- WO2023139978A1 WO2023139978A1 PCT/JP2022/045873 JP2022045873W WO2023139978A1 WO 2023139978 A1 WO2023139978 A1 WO 2023139978A1 JP 2022045873 W JP2022045873 W JP 2022045873W WO 2023139978 A1 WO2023139978 A1 WO 2023139978A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- vehicle
- image
- erroneous
- camera device
- vehicle camera
- Prior art date
Links
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
- B60W50/02—Ensuring safety in case of control system failures, e.g. by diagnosing, circumventing or fixing failures
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W60/00—Drive control systems specially adapted for autonomous road vehicles
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
Definitions
- the present invention relates to an in-vehicle camera device, an in-vehicle camera system, and an image storage method that generate learning data for machine learning from images captured while driving.
- AD autonomous driving
- ADAS advanced driver-assistance systems
- Machine learning uses data under various circumstances for learning.
- learning data By increasing the diversity of data used for machine learning (hereinafter referred to as "learning data"), it is possible to improve the recognition accuracy of an AI network that has machine-learned them.
- a learning data generating device disclosed in Patent Literature 1, which acquires learning data from output data of an in-vehicle sensor based on a driver's manual driving operation.
- the learning data generation device includes a storage unit that stores various data, a display unit that can display images, a driving environment acquisition unit that acquires information related to the vehicle's driving environment, an operation acquisition unit that acquires input related to vehicle operation, and learning data that associates information related to the driving environment and input related to operation. and a control unit for generating, the control unit causes the display unit to display information about the driving environment acquired by the driving environment acquisition unit, and generates learning data in which the input related to the operation acquired by the operation acquisition unit is associated with the displayed information related to the driving environment.”
- the learning data generation device of Patent Document 1 has the ultimate goal of realizing highly accurate automated driving, as stated in the subject column of the abstract, "Generate learning data for realizing highly accurate driving operations during automated driving.”
- the learning data generation device of Patent Document 1 is a system in which "the simulator system (learning data generation device) 1 is a system in which a vehicle travels in a virtually constructed driving environment and acquires the driving operations of the driver while driving.” There was a possibility that the accuracy of automated driving could not be sufficiently ensured in unlearned situations.
- an object of the present invention to provide an in-vehicle camera device, an in-vehicle camera system, and an image storage method that automatically extract and store images before and after the occurrence of misrecognition of the external environment from various images captured in an actual driving environment, contributing to the improvement of problematic AI networks.
- the in-vehicle camera device of the present invention is provided with a control feedback unit that receives feedback on the automatic control of the own vehicle, an erroneous control determination unit that determines erroneous automatic control based on the driver's driving operation information, and an image storage unit that saves an image, and saves the image when it is determined that an erroneous automatic control has occurred.
- FIG. 2 is a bird's-eye view for explaining the field-of-view angle of view of an in-vehicle camera device mounted on the own vehicle.
- 1 is a functional block diagram of an in-vehicle camera system according to an embodiment
- FIG. 4 is a bird's-eye view for explaining the behavior of the own vehicle when misidentification occurs.
- 5 is a flowchart for explaining a process of saving images before and after misidentification.
- FIG. 3 is a bird's-eye view for explaining the behavior of a vehicle when mistracking occurs.
- Fig. 6A legend. 4 is a flowchart for explaining a process of saving images before and after mistracking;
- FIG. 4 is a bird's-eye view for explaining the behavior of the own vehicle when misidentification occurs.
- 5 is a flowchart for explaining a process of saving images before and after misidentification.
- the in-vehicle camera device 100 is a device that identifies and saves images that caused AD control or ADAS control against the driver's will based on the details of the driver's manual driving operation, and additionally uses the saved images to make the AI network for external environment recognition learn.
- the vehicle-mounted camera system is a system that includes the vehicle-mounted camera device 100 and a vehicle control system 200 that controls the own vehicle based on the output of the device. Details of each will be described below.
- FIG. 1 is a plan view for explaining the field of view angle of view of an in-vehicle camera device 100 mounted on a vehicle.
- the in-vehicle camera device 100 is attached to the inner surface of the windshield of the vehicle or the like facing forward, and incorporates a left imaging unit 1L and a right imaging unit 1R.
- the left imaging unit 1L is an imaging unit arranged on the left side of the on-vehicle camera device 100 for capturing the left image PL .
- the right imaging unit 1R is arranged on the right side of the on - vehicle camera device 100 to capture the right image PR .
- the right visual field V R and the left visual field V L overlap in front of the host vehicle.
- this overlapping visual field area will be referred to as "stereo visual field area R S "
- the visual field area obtained by removing the stereo visual field area R S from the right visual field VR will be referred to as the "right monocular visual field area R R "
- the visual field area obtained by removing the stereo visual field area R S from the left visual field V L will be referred to as the "left monocular visual field area R L ".
- FIG. 2 is a diagram showing the correlation of each visual field area described above.
- the upper stage is a left image P L captured by the left imaging section 1L
- the middle stage is a right image P R captured by the right imaging section 1R
- the lower stage is a composite image P C obtained by synthesizing the left image PL and the right image PR .
- the composite image PC is divided into a right monocular viewing region RR , a stereo viewing region RS , and a left monocular viewing region RL as shown.
- FIG. 3 is a functional block diagram of the in-vehicle camera system according to this embodiment.
- the in-vehicle camera system of this embodiment is a system that includes an in-vehicle camera device 100 and a vehicle control system 200, and receives output data from each sensor, which will be described later.
- the vehicle-mounted camera device 100 includes a stereo matching unit 2, a monocular detection unit 3, a monocular distance measurement unit 4, a template creation unit 5, an image storage unit 6, a similar part search unit 7, an angle of view identification unit 8, a stereo detection unit 9, a speed calculation unit 10, a vehicle information input unit 11, a stereo distance measurement unit 12, a type identification unit 13, a driver operation input unit 14, and an identification history storage unit.
- the vehicle-mounted camera device 100 is connected to a vehicle speed sensor 31, a vehicle steering angle sensor 32, a yaw rate sensor 33, a steering sensor 34, an accelerator pedal sensor 35, and a brake pedal sensor 36 mounted on the own vehicle, and receives output data of each sensor.
- the configuration other than the left imaging unit 1L and the right imaging unit 1R is specifically a computer equipped with hardware such as an arithmetic device, a storage device, and a communication device.
- arithmetic units such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit), and CPLD (Complex Programmable Logic Device) execute a predetermined program obtained from a program recording medium or a distribution server outside the vehicle, realizing each processing unit such as the stereo matching unit 2.
- a storage device such as a semiconductor memory identifies Each storage unit such as the history storage unit 15 is implemented, but hereinafter, such well-known technology in the computer field will be omitted as appropriate.
- the vehicle control system 200 is connected to the in-vehicle camera device 100 and each of the sensors described above, and is a system that automatically controls the warning, braking, and steering of the own vehicle for the purposes of collision avoidance and reduction, following the preceding vehicle, and maintaining the own lane, according to the recognition results of the in-vehicle camera device 100.
- the left imaging unit 1L and the right imaging unit 1R are monocular cameras equipped with imaging sensors (CMOS (Complementary Metal Oxide Semiconductor), etc.) that convert light into electric signals.
- CMOS Complementary Metal Oxide Semiconductor
- the information converted into an electric signal by each imaging sensor is further converted into image data representing a captured image within each imaging unit.
- the images captured by the left imaging unit 1L and the right imaging unit 1R are sent as the left image PL and the right image PR to the stereo matching unit 2, the monocular detection unit 3, the monocular distance measurement unit 4, the template creation unit 5, the image storage unit 6, the similar part search unit 7, and the angle of view identification unit 8 at predetermined intervals (for example, every 17 ms).
- the stereo matching unit 2 receives data including image data from the left imaging unit 1L and the right imaging unit 1R, and processes the data to calculate parallax.
- Parallax is a difference in image coordinates in which the same object is captured, which is caused by differences in the positions of a plurality of imaging units.
- the parallax is large at short distances and small at long distances, and the distance can be calculated from the parallax.
- the stereo matching unit 2 also corrects the distortion of the image data of the left image PL and the right image PR . For example, the distortion of the image data is corrected so that objects having the same height and the same depth distance, which are called a central projection model or a perspective projection model, are arranged horizontally in the image coordinates.
- left imaging section 1L and the right imaging section 1R are arranged side by side in the horizontal direction is that the correction is made so that they are aligned in the horizontal direction.
- the corrected left image PL and right image PR one of these is used as reference image data as a reference, and the other is used as comparison image data for comparison to obtain parallax.
- the monocular detection unit 3 detects a specific three-dimensional object appearing in the left image PL or the right image PR .
- a specific three-dimensional object is an object that needs to be detected in order to realize appropriate AD control or ADAS control, and specifically includes pedestrians, other vehicles, and bicycles around the vehicle.
- Detection by the monocular detection unit 3 is made to detect three-dimensional objects within a certain range from the vehicle-mounted camera.
- the detection result by the monocular detection unit 3 includes information on the image coordinates of the left image PL or the right image PR in which the object to be detected is captured. For example, the detection result is held as the vertical and horizontal image coordinates of the upper left and lower left of a rectangular frame (hereinafter referred to as "monocular detection frame") surrounding the detection object.
- the stereo detection unit 9 detects locations within a certain size range at the same distance as three-dimensional objects.
- the detection result by the stereo detector 9 includes parallax image coordinate information of the detected object.
- the detection result is held as the vertical and horizontal image coordinates of the upper left and lower left of a rectangular frame (hereinafter referred to as "stereo detection frame") surrounding the detected object.
- the type identification unit 13 identifies the type of the object detected by the monocular detection unit 3 or the stereo detection unit 9 using an identification network such as a template image, pattern, or machine learning dictionary.
- an identification network such as a template image, pattern, or machine learning dictionary.
- the types of the objects identified by the type identification unit 13 are four-wheeled vehicles, two-wheeled vehicles, and pedestrians, the directions of the front surfaces of the objects, which are moving directions thereof, are also identified, and it is also identified whether they are moving bodies (three-dimensional objects moving against the background) that are assumed to cross in front of the own vehicle.
- the orientation of the front surface of these mobile objects is the orientation of the front surface of the object with respect to the in-vehicle camera device 100 . In this embodiment, the operation is limited to the directions of movement that are nearly orthogonal.
- Whether the direction of movement is nearly orthogonal or not is determined based on the angle of view and the orientation of the front surface of the object with respect to the vehicle-mounted camera device 100 . For example, in the case of an object captured on the right side of the in-vehicle camera device 100, the surface and side of the object in the traveling direction are captured. In the case of an angle of 45°, an object with a direction of motion close to orthogonal can be identified when the front and side faces are viewed at a 45° orientation.
- the type identification unit 13 transmits to the speed calculation unit 10 the determination result as to whether or not the moving direction of the object is nearly orthogonal.
- the monocular distance measurement unit 4 identifies the position of a specific object detected by the monocular detection unit 3, and obtains the distance and direction from the left imaging unit 1L or the right imaging unit 1R.
- the specified distance and direction are represented by a coordinate system that can specify the position on the plane of the depth distance in front of the own vehicle and the lateral distance in the lateral direction.
- information expressed in a polar coordinate system expressed by a Eugrid distance, which is a distance from the camera, and a direction may be held, and a trigonometric function may be used for interconversion between the depth and two horizontal axes.
- the monocular distance measurement unit 4 uses an overhead image obtained by projecting the left image P L , the right image P R , and the composite image P C onto the road surface, and specifies the position from the vertical and horizontal coordinates of the overhead image and the vertical and horizontal scales of the overhead image with respect to the actual road surface.
- it is not essential to use a bird's-eye view image to specify the position. It may be specified by performing geometric calculations using extrinsic parameters of the position and orientation of the camera, and information on the focal length, the pixel pitch of the imaging device, and the distortion of the optical system.
- the stereo ranging unit 12 identifies the position of the object detected by the stereo detection unit 9 and identifies the distance and direction.
- the specified distance and direction are represented by a coordinate system that can specify the position on the plane of the depth distance in front of the own vehicle and the lateral distance in the lateral direction.
- information expressed in a polar coordinate system expressed by a Eugrid distance, which is a distance from the camera, and a direction may be held, and a trigonometric function may be used for interconversion between the depth and two horizontal axes.
- the stereo ranging unit 12 calculates the distance in the depth direction from the parallax of the object.
- the stereo ranging unit 12 uses an average or a mode when there is variation in the calculated parallax of the object. When the disparity of parallax is large, a method of taking a specific outlier may be used.
- the horizontal distance is obtained from the horizontal angle of view of the detection frame of the stereo detection unit 9 and the depth distance using a trigonometric function.
- the template creation unit 5 selects one of the captured images, cuts out a specific region from this captured image (hereinafter referred to as "detected image”), and uses it as a template image. Specifically, the template creation unit 5 creates a template image for searching for similar portions from pixel information inside and around the monocular detection frame of the object detected by the monocular detection unit 3 .
- This template image is scaled to a predetermined image size. When enlarging or reducing the template image, maintain the aspect ratio or do not change it significantly.
- the brightness values of the pixels themselves or the reduced image is divided into a plurality of small regions (hereinafter also referred to as "kernels"), and the relationship between the brightness values in the kernels is stored as a feature of the image.
- Image features are extracted in various forms, and there are various feature amounts such as left-right and top-bottom average brightness differences in the kernel, average brightness differences between the periphery and the center, and brightness averages and variances.
- the background In the camera image, the background is reflected around the target, and when the image is cut out, the background is mixed. However, the background changes depending on the time, even if the target is the same, as the target and the camera move.
- machine learning is performed using camera images for the shape and texture of the type of target to be tracked among the feature quantities, and which feature quantities should be held with what weights to search for similar locations are stored as a tracking network.
- the image storage unit 6 holds the left image PL and the right image PR for a certain period of time, or until a certain number of left images PL and right images PR at different times are accumulated.
- a temporary storage device such as a DRAM (Dynamic Random Access Memory) is used for this holding.
- a plurality of addresses and ranges in the storage device to be held may be determined in advance, and the transfer destination address may be changed for each captured image, and after one cycle, the area where the old image is stored may be overwritten. Since the address is determined in advance, there is no need to notify the address when reading out the held image.
- the similar part searching unit 7 searches the images stored in the image storage unit 6 for parts similar to the template image created by the template creating unit 5 .
- an image for which the similarity search section 7 searches for a similarity to the template image will be referred to as a "search target image”.
- the search target image is an image different from the image at the time detected by the monocular detection unit 3 .
- the selection of an image at which time is based on selecting an image captured at a time before or after the image at the detected time. Similar locations are likely to exist near the coordinates of the monocular detection frame at the time of detection, and since it is expected that there will be little change in brightness due to changes in the orientation of the detection object and changes in exposure with respect to the template image, high-precision search is possible.
- the detection result of the monocular detection unit 3, the detection result of the stereo detection unit 9, the distance measurement result of the monocular distance measurement unit 4, and the distance measurement result of the stereo distance measurement unit 12, the position of the same target at each time is tracked.
- the process of creating a template image from the image at a certain time and searching for and tracking similar portions in the image at a different time is called image tracking processing by a tracking network.
- the imaging interval is short, an image from two or more previous times may be selected as the search target image.
- the image to be selected may be changed according to the vehicle speed. For example, if the vehicle speed is equal to or greater than a threshold, an older image close to the imaging time of the detected image is selected, and if the vehicle speed is slower than the threshold, an older image than the close old image may be selected.
- the vehicle speed is high, if the image is taken at a time close to the detected time, the appearance does not change greatly from the template image, and it is easy to ensure search accuracy.
- the vehicle speed is slow, the positional change of the detected object in the image for searching for a similar part increases with respect to the detected time, and the accuracy of the calculated speed can be improved.
- the angle-of-view specifying unit 8 specifies the horizontal angle of view of the camera for a specific object appearing in the similar location specified by the similar location searching unit 7 . However, if the height is specified along with the position, then the vertical image is also specified. Also, even if there is a possibility that the camera rolls, the speed can be calculated with high accuracy by specifying both the horizontal and vertical angles of view.
- the horizontal angle of view can be obtained by a trigonometric function from the ratio of the depth distance and the horizontal distance.
- the speed calculation unit 10 receives the distance measurement result from the monocular distance measurement unit 4 or the stereo distance measurement unit 12, receives the angle of view of the similar location from the angle of view identification unit 8, receives vehicle behavior information from the vehicle information input unit 11, and calculates the speed of the detected specific object. However, instead of receiving the vehicle speed from the vehicle information input unit 11, the speed calculation unit 10 may use the calculated differential value of the relative depth distance as the vehicle speed. This is useful when the depth distance can be measured with high accuracy and the horizontal distance is measured with low accuracy. Also, if the depth velocity is used instead of the vehicle velocity, it is possible to accurately predict a collision with a crossing vehicle that is not orthogonal.
- the vehicle information input unit 11 receives the vehicle speed information from the vehicle speed sensor 31 that measures the speed of the vehicle, the steering angle information from the vehicle steering angle sensor 32 that measures the steering angle of the steered wheels, and the turning speed of the vehicle from the yaw rate sensor 33 that measures the turning speed of the vehicle.
- the vehicle information input unit 11 may be realized by a communication module compatible with a communication port (for example, IEEE802.3), or may be realized by an AD converter capable of reading voltage and current.
- the driver operation input unit 14 is connected to a steering sensor 34 that acquires information on the angle of turning the steering wheel, an accelerator pedal sensor 35 that acquires information on the amount of depression of the access pedal, and a brake pedal sensor 36 that acquires information on the amount of depression of the brake pedal.
- the identification history storage unit 15 stores the three-dimensional object (hereinafter referred to as "target") identified by the type identification unit 13 together with the target identifier and the time of the target.
- the target identifier is a unique code assigned to each target. For example, if one vehicle and three pedestrians are identified as targets, identifiers such as vehicle A, pedestrian A, pedestrian B, and pedestrian C are assigned to each target.
- the tracking history storage unit 16 stores the success or failure of the template image creation by the template creation unit 5 and the similar location search unit 7, the template image when searching for similar locations at each time, the success or failure of the similar location search results, and the image detected as the similar location, together with the identifier of the target and the time.
- the position history storage unit 17 stores the location of the same target at each time, along with the target identifier and time, based on the search result, the detection result of the monocular detection unit 3, the detection result of the stereo detection unit 9, the distance measurement result of the monocular distance measurement unit 4, and the distance measurement result of the stereo distance measurement unit 12.
- the speed history storage unit 18 stores the speed calculated by the speed calculation unit 10 together with the identifier of the target and the time.
- the control feedback unit 23 is connected to the vehicle control system 200, and inputs control feedback information such as vehicle control content and time controlled by the vehicle control system 200, target target identifier, control decision reason, control cancellation content, and control cancellation reason.
- the erroneous control determination unit 21 receives the control feedback information of the own vehicle from the control feedback unit 23 and receives the driver's operation information from the driver's operation input unit 14 . Further, the vehicle control system 200 performs vehicle control based on the target object information output from the in-vehicle camera device 100, and after starting the alarm and automatic braking, it is determined whether or not the control is canceled by the driver's operation. Then, when there is such a driving operation, the erroneous control determination unit 21 determines that erroneous control by the vehicle control system 200 has occurred. The details of the erroneous control determination unit 21 will be described later.
- the erroneous identification identification unit 19 identifies cases of erroneous identification among the identification results of the targets that were the target of erroneous control after the erroneous control determination unit 21 determines that an erroneous control has occurred.
- the details of the erroneous identification specifying unit 19 will be described later.
- the erroneous tracking identification unit 20 identifies a case of erroneous tracking among the tracking results of the target that was the target of erroneous control after the erroneous control determination unit 21 determined that the target was erroneously controlled. The details of the mistracking identification unit 20 will be described later.
- the erroneous recognition image storage unit 22 extracts and stores images before and after the erroneous identification and erroneous tracking specified by the erroneous identification specifying unit 19 and the erroneous tracking specifying unit 20 .
- the additional learning unit 24 additionally learns the AI network for recognizing the external environment using the images before and after the erroneous identification and erroneous tracking stored in the erroneous recognition image storage unit 22 .
- FIG. 4 is a bird's-eye view showing an example of a situation in which erroneous identification occurs as a result of recognizing the external environment using an identification network or the like in the type identification unit 13 of the in-vehicle camera device 100 .
- the sampling period of the external environment is set to 200 ms here, the sampling period is not limited to this example.
- FIG. 4(a) is a bird's-eye view illustrating the external environment of the own vehicle at time T - 1 , which is 200 ms before time T0, which will be described later.
- the vehicle is traveling at a constant speed on a straight road, and the in-vehicle camera device 100 attached to the vehicle has not detected a three-dimensional object in front of the vehicle.
- FIG. 4B is a bird's-eye view illustrating the external environment of the host vehicle at time T0 when automatic control (automatic deceleration) is started.
- the in-vehicle camera device 100 erroneously detects a non-existent pedestrian as a result of processing the captured image using a slightly problematic identification network or the like. Therefore, the vehicle control system 200 that receives the detection result of the in-vehicle camera device 100 controls the braking system of the own vehicle and decelerates the own vehicle rapidly in order to prevent contact with a nonexistent pedestrian.
- a situation in which a non-existing pedestrian is erroneously detected is, for example, a situation in which a lump of exhaust gas is erroneously identified as a pedestrian.
- FIG. 4(c) is a bird's-eye view illustrating the external environment of the host vehicle at time T1 , 200 ms after time T0 .
- the driver notices that an abnormality (erroneous identification) has occurred in the in-vehicle camera device 100 because automatic deceleration has started without any reason to brake the vehicle.
- FIG. 4D is a bird's-eye view illustrating the external environment of the host vehicle at time T2 , 400 ms after time T0 .
- the driver who has confirmed the safety in front depresses the accelerator pedal in order to restore the decelerated speed to the pre-deceleration speed to accelerate the own vehicle.
- FIG. 4(e) is a bird's-eye view illustrating the external environment of the host vehicle at time T3 , 600 ms after time T0 .
- the vehicle has entered the area of the pedestrian that was misidentified at time T0 , but since there are no pedestrians there, the vehicle can safely pass through the area.
- the vehicle control system 200 starts automatic control based on the identification result of the on-vehicle camera device 100, if the driver performs a manual driving operation contrary to the automatic control and there is no contact with the identified three-dimensional object, it can be determined that an erroneous identification has occurred in the on-vehicle camera device 100.
- the apparatus of the present invention memorizes and additionally learns images of pedestrian targets misidentified at time T0 , which are determined to be misidentified.
- FIG. 5 is a flowchart to which the mechanism described in FIG. 4 is applied, and shows a method of saving an image at the time when an erroneous identification occurs in the on-vehicle camera device 100, and additionally learning an identification network or the like based on the saved image. Each step will be described below.
- step S1 the erroneous control determination unit 21 determines whether or not the automatic control for collision prevention has operated based on the data from the vehicle control system 200 that the control feedback unit 23 has obtained.
- the automatic control for collision prevention includes, for example, automatic braking for collision avoidance and mitigation, control for sounding a collision alarm, steering control for avoiding obstacles, and the like. Then, if the automatic control based on the output of the in-vehicle camera device 100 operates, the process proceeds to step S2, and if it does not operate, step S1 is executed again after a certain period of time.
- step S2 the erroneous control determination unit 21 determines whether or not a manual driving operation contrary to automatic control has been performed, based on data from the steering sensor 34 and the accelerator pedal 35 obtained by the driver operation input unit 14.
- the driver performs a manual operation contrary to the automatic control after the automatic control for preventing contact with the detected three-dimensional object is activated
- the operation is accepted as the operation for canceling the stop control, for the following reasons. That is, when the driver strongly depresses the accelerator pedal exceeding the threshold value Th2, there is a possibility that the driver misunderstands the access pedal as the brake pedal and strongly depresses it, and it is considered inappropriate to accept this as an operation to cancel the stop control. In addition, if the driver continues to depress the accelerator pedal weakly below the threshold Th1, the driver may not be aware of the possibility of a collision, and it is considered inappropriate to accept this as an operation to cancel the stop control. Therefore, only when the depression amount is between the threshold Th1 and the threshold Th2, the automatic brake is released and treated as erroneous control.
- ⁇ Second example of erroneous control>> A situation is assumed in which the vehicle control system 200 determines that the vehicle has collided with a target and starts automatic braking with the goal of stopping the vehicle before the collision point. In this case, after the vehicle is stopped by automatic braking, the driver depresses the accelerator pedal to restart the vehicle within a predetermined time, and the vehicle passes before the collision point.
- the time is, for example, 0.5 seconds, which is sufficiently shorter than the time required for the obstacle to retreat when the obstacle is present at the collision point.
- the time for the obstacle to retreat is the time for moving at the crossing speed by the width of the vehicle if the collision point is in the left-right center of the own vehicle.
- step S3 If the target is a non-vehicle such as a pedestrian, there is a possibility that the vehicle will be surprised and stop in a situation where there is a possibility of collision with the own vehicle.
- the vehicle control system 200 After the vehicle control system 200 has stopped before the collision point, if the vehicle restarts and passes through the collision point in a short period of time after the driver operates the accelerator pedal, it is determined that a driving operation contrary to the automatic control has been performed, and the process proceeds to step S3.
- the vehicle control system 200 determines a collision with a target and performs automatic steering operation to turn the vehicle to the right or left of the target in order to avoid the target, thereby controlling the collision avoidance operation. In this case, even if the driver performs a steering operation that interferes with the automatic steering operation, it is determined that a driving operation contrary to the automatic control has been performed, and the process proceeds to step S3.
- the above steering operation is, for example, when the vehicle control system 200 controls the automatic steering operation to turn the steering wheel clockwise by 45°, but the driver applies a force in the counterclockwise direction to make the steering wheel turn to a state sufficiently smaller than 45°, for example, to only about 22°.
- step S3 the erroneous control determination unit 21 calculates the travel trajectory after automatic control (the trajectory of the vehicle's position) based on the outputs of the vehicle speed sensor 31, the vehicle steering angle sensor 32, and the yaw rate sensor 33, and stores it together with the driving operation. Specifically, information on accelerator pedal operation, brake pedal operation, steering operation, vehicle speed, vehicle steering angle, and vehicle yaw rate is obtained from various sensors as a result of the driver's driving operation, and the travel trajectory necessary for the subsequent determination in step S4 is calculated and estimated.
- step S4 the erroneous identification identifying unit 19 identifies the occurrence of erroneous identification by determining whether the vehicle has traveled in manual operation on a travel trajectory that can avoid collision with the identified target, assuming that the identification result at the start of automatic control was correct. Specifically, the trajectory of the own vehicle estimated in step S3 is compared with the identification history and position history stored in the identification history storage unit 15 and the position history storage unit 17, and the collision avoidance of pedestrians, bicycles, vehicles, etc. is determined in the identification history for all historical positions of the identification result set by the vehicle control system 200, and whether or not the collision could not be avoided is determined.
- step S5 If there is even one historical position judged to be a travel locus where collision avoidance cannot be avoided, it is determined that the driver has selected the travel locus after determining that the driver can pass through that position safely (that is, it is determined that there was an erroneous identification that caused the vehicle control system 200 to activate an essentially unnecessary automatic control), and the process proceeds to step S5. On the other hand, if not, it is determined that the automatic control by the vehicle control system 200 was necessary (that is, the identification that caused the vehicle control system 200 to activate the automatic control was appropriate), and the process returns to step S1.
- step S ⁇ b>5 the erroneous identification specifying unit 19 reads the image of the erroneous identification, which is temporarily stored in the image storage unit 6 , and saves it in the erroneous recognition image storage unit 22 .
- the stored position and time in the history and the identification image corresponding to the identifier of the target object are stored in the memory.
- the number of images to be stored in the erroneously recognized image storage unit 22 can be reduced to, for example, at most several images (a data amount of about several hundred KB) for which erroneous identification occurs, so that the storage capacity of the erroneously recognized image storage unit 22 can be significantly reduced compared to the case where all the images captured while driving are stored as learning data.
- step S6 the additional learning unit 24 performs additional machine learning of an identification network, etc., using the erroneously identified images stored in the erroneously recognized image storage unit 22 in step S5 as learning data.
- the identification image stored in the misrecognized image storage unit 22 may be used as learning data for additional learning, or learning data obtained by processing the stored identification image (for example, learning data obtained by enlarging or reducing the identification image, or learning data obtained by rotating the identification image) may be used as learning data for additional learning.
- learning data obtained by processing the stored identification image for example, learning data obtained by enlarging or reducing the identification image, or learning data obtained by rotating the identification image
- step S5 it is desirable to store a label specifying the mode of misidentification together with the identification image. Since the vehicle control system 200 of this embodiment is designed to change the determination of whether or not to execute vehicle control according to the type of detected target (another vehicle, pedestrian, bicycle, etc.), an incorrect determination of the type of target is considered as a possible cause of erroneous control. Therefore, an identifier for identifying an erroneously identified identification network and a type of target erroneously identified by the identification network are recorded in the label attached to the identification image.
- step S5 In order to make the erroneous identification network learn that the identification was erroneous, along with the learning data (images), it is necessary to have type class information on how the target was erroneously identified. Therefore, by storing the type class information in the label in step S5, in step S6, using the image at the time of misidentification occurrence, it is possible to pinpoint additional machine learning for the problem identification network or the like specified by the label, and to improve the problem identification network or the like. In addition, because it is possible to exclude problem-free identification networks, etc., from the targets of additional learning, it is possible to avoid adverse effects such as the occurrence of erroneous learning of normal identification networks, etc., and the increase in computational load due to unnecessary additional learning of normal identification networks, etc.
- FIG. 6A is a bird's-eye view showing an example of a situation in which mistracking occurs as a result of recognizing the external environment using a template image with some problem, a tracking network, or the like in the similar location search unit 7 of the vehicle-mounted camera device 100
- FIG. 6B is a legend for FIG. 6A.
- the bird's-eye view shown in the figure is created, for example, by taking into account changes over time in projected images obtained by affine transforming the picked-up left image P L and right image PR using a predetermined affine table, and changes over time in the position of the vehicle. Since the method for creating the bird's-eye view of the surroundings of the vehicle using such a method is a well-known technique, the details thereof will not be described.
- FIGS. 6A (a) to (c) are overhead views illustrating the external environment of the host vehicle at times T ⁇ 3 , T ⁇ 2 , and T ⁇ 1 600 ms, 400 ms, and 200 ms before time T 0 described later, respectively.
- the vehicle is traveling on a straight road at a constant speed, and the in-vehicle camera device 100 tracks the movement of the pedestrian walking on the right sidewalk using a tracking network or the like. Since the vehicle control system 200 determines that the own vehicle will not come into contact with the pedestrian during the period from time T -3 to T -1 , automatic control for avoiding contact with the pedestrian is not performed at these times.
- FIG. 6A(d) is a bird's-eye view illustrating the external environment of the host vehicle at time T0 when automatic control (automatic deceleration) is started.
- the in-vehicle camera device 100 erroneously tracks the pedestrian who is actually on the sidewalk as having jumped out onto the roadway as a result of processing the picked-up image using a slightly problematic template image, tracking network, or the like. Therefore, the vehicle control system 200 that has received the tracking result of the in-vehicle camera device 100 controls the braking system of the own vehicle and decelerates the own vehicle rapidly in order to prevent contact with a non-existent pedestrian on the road.
- a situation in which a non-existing pedestrian on the road is erroneously detected is, for example, a situation in which a rising flag or plants existing in the direction of the pedestrian's path fluctuate due to a gust of wind, etc., and the fluctuation is misidentified as a pedestrian jumping out onto the road.
- OB -3 , OB -2 , OB -1 and OB 0 indicate the positions of targets (pedestrians) detected at times T -3 , T -2 , T -1 and T 0 respectively. Note that the position OB 0 is erroneously detected.
- v ⁇ 1 , v 0 , and v 1 are movement vectors of the target (pedestrian) during the periods of time T ⁇ 3 to T ⁇ 2 , time T ⁇ 2 to T ⁇ 1 , and time T ⁇ 1 to T 0 , respectively, and are obtained by converting the velocity vector of the target (pedestrian) calculated by the in-vehicle camera device 100 into the amount of movement in each period.
- E L and E R are the left and right ends of the travel range of the own vehicle, and indicate the left and right ends of the range in which the own vehicle is predicted to travel based on the steering angle, yaw rate, and speed of the own vehicle.
- the vehicle control system 200 of this embodiment determines that there is a possibility of contact with the target when a moving target enters a predetermined contact determination range including the travel range of the vehicle, and activates automatic control such as braking and steering to avoid contact with the target. Therefore, in FIG. 6B, at time T0 when the target position OB0 within the contact determination range is detected, the vehicle control system 200, for example, activates automatic control to rapidly decelerate the own vehicle.
- the length of the vehicle travel range is limited to a range in which the vehicle is expected to travel within a predetermined time period (for example, within 5 seconds), and even if a target exists, it is not necessary to include it in the travel range of the vehicle at that point in time where it is not necessary to immediately implement contact avoidance control.
- the width of the contact determination range may be changed according to the type information of the road being traveled obtained from GNSS (Global Navigation Satellite System) or the like. For example, it may be narrow on highways where the possibility of approaching targets (pedestrians, other vehicles, etc.) from the side is low, and may be widened on general roads where there is a high possibility of targets approaching from the side.
- GNSS Global Navigation Satellite System
- FIG. 6A(e) is a bird's-eye view illustrating the external environment of the host vehicle at time T1 , 200 ms after time T0 .
- the driver notices that some abnormality (erroneous tracking) has occurred in the in-vehicle camera device 100 because automatic deceleration has started without any reason to brake the vehicle. Therefore, if the safety ahead (absence of a vehicle in front, absence of pedestrians, etc.) can be confirmed, the driver depresses the accelerator pedal in order to restore the decelerated speed to the pre-deceleration speed, thereby accelerating the own vehicle.
- the vehicle approaches the area of the erroneously detected pedestrian, but since there are no pedestrians in the area (the actual pedestrian is on the right sidewalk), the vehicle can safely pass through the area.
- the vehicle control system 200 starts automatic control based on the tracking result of the vehicle-mounted camera device 100, if the driver performs a manual driving operation contrary to the automatic control and does not come into contact with the tracked three-dimensional object, it can be determined that the vehicle-mounted camera device 100 has mistracked.
- FIG. 7 is a flowchart to which the mechanism described in FIG. 6A is applied, and shows a method for storing images before and after the time when mistracking occurred in the in-vehicle camera device 100, and additionally learning a template image, a tracking network, etc. based on the stored images.
- Each step will be described below. 5 and 7 can be performed in parallel, and redundant description of points common to the flowchart of FIG. 5 will be omitted as necessary.
- step S1 the erroneous control determination unit 21 determines whether or not the automatic control for collision prevention has operated based on the data from the vehicle control system 200 that the control feedback unit 23 has obtained.
- step S2 the erroneous control determination unit 21 determines whether or not a manual driving operation contrary to automatic control has been performed, based on data from the steering sensor 34 and the accelerator pedal 35 obtained by the driver operation input unit 14. As described with reference to FIG. 6A , if the driver performs a manual operation contrary to the automatic control after the automatic control to prevent contact with the target being tracked has been activated, it is considered that the driver is performing the operation after confirming safety.
- step S3 the erroneous control determination unit 21 calculates the travel trajectory after automatic control (the trajectory of the vehicle's position) based on the outputs of the vehicle speed sensor 31, the vehicle steering angle sensor 32, and the yaw rate sensor 33, and stores it together with the driving operation.
- step S4a the erroneous tracking identification unit 20 identifies the occurrence of erroneous tracking by determining whether or not the vehicle has traveled manually along a trajectory that can avoid contact with the tracked target, assuming that the tracking result at the start of automatic control was correct.
- step S5a the mistracking identification unit 20 reads images before and after the occurrence of mistracking (for example, images for one second before and after the automatic control start time) from the image storage unit 6 and stores them in the misrecognition image storage unit 22.
- mistracking for example, images for one second before and after the automatic control start time
- step S6a the additional learning unit 24 performs additional machine learning of template images, tracking networks, etc., using the images before and after the occurrence of mistracking, which were stored in the misrecognized image storage unit 22 in step S5a, as learning data.
- the additional machine learning unit 24 performs additional machine learning of template images, tracking networks, etc., using the images before and after the occurrence of mistracking, which were stored in the misrecognized image storage unit 22 in step S5a, as learning data.
- step S5a of FIG. 7 it is desirable to follow step S5 of FIG. 5 and store a label identifying the mode of mistracking together with the identification image.
- the additional learning in step S6a can be executed only for template images, tracking networks, etc. that need to be improved by the same action as described in FIG. 5, so various effects similar to those in FIG. 5 can be obtained.
- step S4a there is a possibility that the same object as the template or the same object that appears in the image before the mistracked portion exists near the location identified as the occurrence of mistracking in step S4a. Therefore, by storing the images before and after the mistracked portion by enlarging the area to which the detection frame is attached by tracking, an image including the location where the same object appears can be stored, and the location that should be tracked correctly can be appropriately additionally learned during additional learning.
- FIG. 8 is a bird's-eye view for explaining the behavior of the own vehicle at the time of misidentification and the method of determining miscontrol/misidentification, taking into account the movement of pedestrians, with respect to the target identified at time T0 and the control of the subject vehicle being activated.
- the pedestrian prediction range shown in FIG. 8(c) is obtained by multiplying the elapsed time after the pedestrian is identified in FIG.
- the moving speed of the pedestrian is a speed that is preset as a pedestrian-like speed, and may be assumed to be an average walking speed, for example, set to 5 km/h.
- the pedestrian existence radius in the legend of FIG. 8 indicates the radius of the pedestrian prediction range, and is the length L obtained by multiplying the elapsed time T after the pedestrian is identified and the movement speed VP of the pedestrian.
- the target identified at time T0 is at time T3 and the pedestrian presence prediction range is covered by the own vehicle, so it is determined that the target identified at time T0 was erroneously identified.
- a target at a time when an erroneous identification is made is specified based on the overlapping degree of the pedestrian prediction range and the own vehicle range with respect to the detection/identification target at each time when the collision is determined when the automatic control is performed.
- FIG. 9 is a flow chart for explaining a process of storing images before and after misidentification, taking into account the movement of pedestrians. Note that redundant description of the points in common with FIG. 5 is omitted.
- Step S7 in FIG. 9 is a process performed after step S3, and is a process of estimating the predicted range of the pedestrian after automatic control.
- the pedestrian presence radius is obtained from the elapsed time T after the pedestrian is identified and the pedestrian movement speed VP to specify the pedestrian presence range.
- the subsequent step S4b is a process for determining whether the manual operation to avoid the identification target is performed, and it is determined whether the manual operation to avoid the identification target has been performed based on the travel trajectory verified in step S3 and the pedestrian prediction range estimated in step S7. When the vehicle covers the pedestrian prediction range on the travel path of the own vehicle, it is determined that the manual operation to avoid the identification target of the pedestrian was not performed.
- ⁇ Effect of this embodiment> it is possible to automatically extract and save images before and after the occurrence of misrecognition of the external environment, which contributes to the improvement of problematic AI networks, from among various images captured in an actual driving environment.
- the in-vehicle camera device of the present embodiment uses the automatically extracted and saved images as learning data without cooperating with the outside, so that the problematic AI network can be made to perform additional learning, and the quality of the AI network can be improved.
- the present invention is not limited to the above-described embodiments, and includes various modifications.
- the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations.
- it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment and it is also possible to add the configuration of another embodiment to the configuration of one embodiment.
- each of the above configurations, functions, processing units, processing means, etc. may be realized in hardware, for example, by designing a part or all of them with an integrated circuit.
- each of the above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program for realizing each function.
- Information such as programs, tables, and files that implement each function can be stored in a recording device such as a memory, a hard disk, an SSD (Solid State Drive), or a recording medium such as a semiconductor memory card.
- control lines and information lines indicate what is considered necessary for explanation, and not all control lines and information lines are necessarily indicated on the product. In practice, it may be considered that almost all configurations are interconnected.
- the configuration of the functional blocks is merely an example. Some functional configurations shown as separate functional blocks may be configured integrally, or a configuration represented by one functional block diagram may be divided into two or more functions. Further, a configuration may be adopted in which part of the functions of each functional block is provided in another functional block.
- SYMBOLS 100 Vehicle-mounted camera apparatus, 1L... Left imaging part, 1R... Right imaging part, 2... Stereo matching part, 3... Monocular detection part, 4... Monocular distance measurement part, 5... Template preparation part, 6... Image storage part, 7... Similar part search part, 8... Angle-of-view specification part, 9... Stereo detection part, 10... Speed calculation part, 11... Vehicle information input part, 12... Stereo distance measurement part, 13... Type identification part, 14... Driver operation input part, 15... Identification history storage part 16 Tracking history storage unit 17 Position history storage unit 18 Speed history storage unit 19 Incorrect identification identification unit 20 Incorrect tracking identification unit 21 Incorrect control identification unit 22 Incorrect recognition image storage unit 23 Control feedback unit 24 Additional learning unit 200 Vehicle control system
Landscapes
- Engineering & Computer Science (AREA)
- Automation & Control Theory (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Transportation (AREA)
- Mechanical Engineering (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Theoretical Computer Science (AREA)
- Traffic Control Systems (AREA)
- Image Analysis (AREA)
Abstract
Provided is a vehicle-mounted camera device which automatically extracts and stores images before and after erroneous recognition of an external environment, from among images captured in an actual traveling environment. The vehicle-mounted camera device for identifying objects comprises a control feedback unit for accepting feedback relating to automated control of a host vehicle, an erroneous control determining unit for determining erroneous automated control on the basis of driving operation information relating to a driver, and an image storage unit for storing images, wherein images are stored when erroneous automated control is determined.
Description
本発明は、走行中に撮像した画像から、機械学習用の学習データを生成するする車載カメラ装置、車載カメラシステム、および、画像保存方法に関する。
The present invention relates to an in-vehicle camera device, an in-vehicle camera system, and an image storage method that generate learning data for machine learning from images captured while driving.
近年、自動運転(AD、Autonomous Driving)システムや、先進運転支援システム(ADAS、Advanced Driver-Assistance Systems)を搭載した車両が普及しつつある。これらのシステムの実現には、車両の外部環境(他車両、歩行者、自転車、静止障害物、および、それらの位置、速度、移動方向など)を正確に認識する必要がある。そのため、車両の搭載されたカメラの出力データを、画像認識処理を行い、車両の外部環境を認識している。前記画像認識処理には機械学習を用いたものもある。
In recent years, vehicles equipped with autonomous driving (AD) systems and advanced driver-assistance systems (ADAS) have become widespread. Realization of these systems requires accurate recognition of the vehicle's external environment (other vehicles, pedestrians, bicycles, stationary obstacles, their positions, speeds, directions of movement, etc.). Therefore, the output data of the camera mounted on the vehicle is subjected to image recognition processing to recognize the external environment of the vehicle. Some of the image recognition processes use machine learning.
機械学習は、多様な状況下のデータを学習に用いる。機械学習に用いるデータ(以下「学習データ」と呼ぶ)の多様性を高めれば、それらを機械学習したAIネットワークの認識精度を改善することができる。そして、学習データの多様性を高めるための技術として、運転手の手動運転操作に基づき、車載センサの出力データから学習データを取得する、特許文献1の学習データ生成装置が知られている。
Machine learning uses data under various circumstances for learning. By increasing the diversity of data used for machine learning (hereinafter referred to as "learning data"), it is possible to improve the recognition accuracy of an AI network that has machine-learned them. As a technique for increasing the diversity of learning data, there is known a learning data generating device disclosed in Patent Literature 1, which acquires learning data from output data of an in-vehicle sensor based on a driver's manual driving operation.
特許文献1の要約書には、課題として「手動運転時の運転操作を基づいて、精度の高い自動運転時の運転操作を実現させるための学習データを生成する。」と記載されており、その解決手段として「学習データ生成装置は、各種データを記憶する記憶部と、画像を表示可能な表示部と、車両の走行環境に関する情報を取得する走行環境取得部と、車両の操作に関する入力を取得する操作取得部と、走行環境に関する情報と、操作に関する入力と、を対応付けた学習データを生成する制御部と、を備え、制御部は、走行環境取得部により取得された走行環境に関する情報を表示部に表示させ、その際に操作取得部により取得された操作に関する入力と、表示させた走行環境に関する情報とを対応付けた学習データを生成する。」と記載されている。
In the abstract of Patent Document 1, the problem is described as "generating learning data for realizing highly accurate driving operations during automatic driving based on driving operations during manual driving." As a solution to this problem, "the learning data generation device includes a storage unit that stores various data, a display unit that can display images, a driving environment acquisition unit that acquires information related to the vehicle's driving environment, an operation acquisition unit that acquires input related to vehicle operation, and learning data that associates information related to the driving environment and input related to operation. and a control unit for generating, the control unit causes the display unit to display information about the driving environment acquired by the driving environment acquisition unit, and generates learning data in which the input related to the operation acquired by the operation acquisition unit is associated with the displayed information related to the driving environment.”
特許文献1の学習データ生成装置は、要約書の課題欄に「精度の高い自動運転時の運転操作を実現させるための学習データを生成する。」とあるように、精度の高い自動運転の実現を最終目的とするものである。
The learning data generation device of Patent Document 1 has the ultimate goal of realizing highly accurate automated driving, as stated in the subject column of the abstract, "Generate learning data for realizing highly accurate driving operations during automated driving."
しかしながら、特許文献1の学習データ生成装置は、同文献の段落0022に「シミュレータシステム(学習データ生成装置)1は、車両が仮想的に構築された走行環境を走行し、走行中の運転手の運転操作を取得するシステムである。」とあるように、仮想的に構築された所与の走行環境下での運転手の手動運転操作を学習データとする装置でしかなく、実走行環境下で発生する様々な状況での取得データを学習データとするものではないため、シミュレータの設計者や運用者が想定しきれない状況を学習することができず、未学習の状況下では自動運転の精度を十分に担保できない可能性があった。
However, as stated in paragraph 0022 of the same document, the learning data generation device of Patent Document 1 is a system in which "the simulator system (learning data generation device) 1 is a system in which a vehicle travels in a virtually constructed driving environment and acquires the driving operations of the driver while driving." There was a possibility that the accuracy of automated driving could not be sufficiently ensured in unlearned situations.
一方、実走行環境下で生成されるセンサ出力を学習データとすることで学習データ量を増やし、AIネットワークに様々な走行環境を機械学習させる学習方法も考えられる。この場合、膨大な学習データを容易に生成できるが、膨大な学習データの保存に膨大な記憶容量が必要となることや、学習データの正解ラベルを与えるアノテーション作業も膨大となること、更に膨大な学習データの大部分がAIネットワークの改善に寄与しない正常認識時の学習データであり、機械学習の大部分が無効な学習となることを考慮すれば、この学習方法も現実的な方法ではなかった。
On the other hand, it is also possible to increase the amount of learning data by using the sensor output generated in the actual driving environment as learning data, and to have the AI network perform machine learning of various driving environments. In this case, a large amount of learning data can be easily generated, but a large amount of storage capacity is required to store the large amount of learning data, an enormous amount of annotation work is required to give correct labels to the learning data, and most of the huge amount of learning data is learning data during normal recognition that does not contribute to the improvement of the AI network, and most of the machine learning is invalid learning.
ここで、実走行環境下で生成した膨大な学習データから、AIネットワークの改善に寄与しない学習データを除外し、改善に寄与する学習データのみを抽出することができれば、AIネットワークの機械学習時の無効な学習を抑制できるが、そのような学習データの抽出方法については、従来具体的な提案がされていなかった。
Here, if it is possible to exclude the learning data that does not contribute to the improvement of the AI network from the huge amount of learning data generated in the actual driving environment, and extract only the learning data that contributes to the improvement, it is possible to suppress invalid learning during machine learning of the AI network.
そこで、本発明は、実走行環境下で撮像された様々な画像の中から、問題のあるAIネットワークの改善に寄与する、外部環境の誤認識の発生前後の画像を自動的に抽出して保存する車載カメラ装置、車載カメラシステム、および、画像保存方法を提供することを目的とする。
Therefore, it is an object of the present invention to provide an in-vehicle camera device, an in-vehicle camera system, and an image storage method that automatically extract and store images before and after the occurrence of misrecognition of the external environment from various images captured in an actual driving environment, contributing to the improvement of problematic AI networks.
上記課題を解決するため、本発明の車載カメラ装置は、自車両の自動制御のフィードバックを受ける制御フィードバック部と、運転者の運転操作情報に基づき誤った自動制御を判定する誤制御判定部と、画像を保存する画像保存部を備え、誤った自動制御と判定された際に画像を保存するものとした。
In order to solve the above problems, the in-vehicle camera device of the present invention is provided with a control feedback unit that receives feedback on the automatic control of the own vehicle, an erroneous control determination unit that determines erroneous automatic control based on the driver's driving operation information, and an image storage unit that saves an image, and saves the image when it is determined that an erroneous automatic control has occurred.
本発明により、実走行環境下で撮像された様々な画像の中から、問題のあるAIネットワークの改善に寄与する、外部環境の誤認識の発生前後の画像を自動的に抽出して保存することができる。上記した以外の課題、構成および効果は、以下の発明を実施するための形態の説明により明らかにされる。
According to the present invention, it is possible to automatically extract and save images before and after the occurrence of misrecognition of the external environment, which contributes to the improvement of problematic AI networks, from among various images captured in the actual driving environment. Problems, configurations, and effects other than those described above will be clarified by the following description of the mode for carrying out the invention.
以下、図1から図7を参照して、本発明の車載カメラ装置100、および、同装置を備えた車載カメラシステムの実施例を説明する。
Hereinafter, embodiments of an in-vehicle camera device 100 of the present invention and an in-vehicle camera system including the same device will be described with reference to FIGS.
本実施例に係る車載カメラ装置100は、運転者の意に反するAD制御やADAS制御の原因となった画像を、運転者の手動運転操作の内容を踏まえて特定して保存するとともに、保存した画像を用いて外部環境認識用のAIネットワークに追加学習させる装置である。また、本実施例に係る車載カメラシステムは、車載カメラ装置100と、該装置の出力に基づいて自車両を制御する車両制御システム200と、を備えたシステムである。以下、各々の詳細を順次説明する。
The in-vehicle camera device 100 according to the present embodiment is a device that identifies and saves images that caused AD control or ADAS control against the driver's will based on the details of the driver's manual driving operation, and additionally uses the saved images to make the AI network for external environment recognition learn. Further, the vehicle-mounted camera system according to the present embodiment is a system that includes the vehicle-mounted camera device 100 and a vehicle control system 200 that controls the own vehicle based on the output of the device. Details of each will be described below.
図1は、自車両に搭載した車載カメラ装置100の視野画角を説明する平面図である。ここに示すように、車載カメラ装置100は、自車両のフロントガラス内面などに前方を向けて取り付けられており、左撮像部1Lと右撮像部1Rを内蔵している。左撮像部1Lは、車載カメラ装置100の左側に配された、左画像PLを撮像するための撮像部であり、その光軸ALは自車両の前方を向き、光軸ALに対して右側により広い右視野VRを有する。また、右撮像部1Rは、車載カメラ装置100の右側に配された、右画像PRを撮像するための撮像部であり、その光軸ARは自車両の前方を向き、光軸ARに対して左側により広い左視野VLを有する。
FIG. 1 is a plan view for explaining the field of view angle of view of an in-vehicle camera device 100 mounted on a vehicle. As shown here, the in-vehicle camera device 100 is attached to the inner surface of the windshield of the vehicle or the like facing forward, and incorporates a left imaging unit 1L and a right imaging unit 1R. The left imaging unit 1L is an imaging unit arranged on the left side of the on-vehicle camera device 100 for capturing the left image PL . The right imaging unit 1R is arranged on the right side of the on - vehicle camera device 100 to capture the right image PR .
なお、図示するように、右視野VRと左視野VLは自車両の前方で重複している。以下では、この重複する視野領域を「ステレオ視野領域RS」と呼び、右視野VRからステレオ視野領域RSを除いた視野領域を「右単眼視野領域RR」と呼び、左視野VLからステレオ視野領域RSを除いた視野領域を「左単眼視野領域RL」と呼ぶ。
As shown in the figure, the right visual field V R and the left visual field V L overlap in front of the host vehicle. Hereinafter, this overlapping visual field area will be referred to as "stereo visual field area R S ", the visual field area obtained by removing the stereo visual field area R S from the right visual field VR will be referred to as the "right monocular visual field area R R ", and the visual field area obtained by removing the stereo visual field area R S from the left visual field V L will be referred to as the "left monocular visual field area R L ".
図2は、上記した各視野領域の相関関係を示す図である。上段は左撮像部1Lが撮像した左画像PL、中段は右撮像部1Rが撮像した右画像PR、下段は左画像PLと右画像PRを合成した合成画像PCである。合成画像PCは、図示するように、右単眼視野領域RRと、ステレオ視野領域RSと、左単眼視野領域RLに分けられる。
FIG. 2 is a diagram showing the correlation of each visual field area described above. The upper stage is a left image P L captured by the left imaging section 1L, the middle stage is a right image P R captured by the right imaging section 1R, and the lower stage is a composite image P C obtained by synthesizing the left image PL and the right image PR . The composite image PC is divided into a right monocular viewing region RR , a stereo viewing region RS , and a left monocular viewing region RL as shown.
図3は、本実施例に係る車載カメラシステムの機能ブロック図である。ここに示すように、本実施例の車載カメラシステムは、車載カメラ装置100と、車両制御システム200を備えたシステムであり、後述する各センサの出力データを受信するものである。
FIG. 3 is a functional block diagram of the in-vehicle camera system according to this embodiment. As shown here, the in-vehicle camera system of this embodiment is a system that includes an in-vehicle camera device 100 and a vehicle control system 200, and receives output data from each sensor, which will be described later.
車載カメラ装置100は、上記した左撮像部1Lと右撮像部1Rに加え、ステレオマッチング部2と、単眼検知部3と、単眼測距部4と、テンプレート作成部5と、画像記憶部6と、類似箇所探索部7と、画角特定部8と、ステレオ検知部9と、速度算出部10と、車両情報入力部11と、ステレオ測距部12と、種別識別部13と、運転者操作入力部14と、識別履歴記憶部15と、追跡履歴記憶部16と、位置履歴記憶部17と、速度履歴記憶部18と、誤識別特定部19と、誤追跡特定部20と、誤制御判定部21と、誤認識画像保存部22と、制御フィードバック部23と、追加学習部24と、を備える。また、車載カメラ装置100は、自車両に搭載される車両速度センサ31と、車両舵角センサ32と、ヨーレートセンサ33と、ステアリングセンサ34と、アクセルペダルセンサ35と、ブレーキペダルセンサ36に接続されており、各センサの出力データを受信する。
In addition to the left imaging unit 1L and right imaging unit 1R described above, the vehicle-mounted camera device 100 includes a stereo matching unit 2, a monocular detection unit 3, a monocular distance measurement unit 4, a template creation unit 5, an image storage unit 6, a similar part search unit 7, an angle of view identification unit 8, a stereo detection unit 9, a speed calculation unit 10, a vehicle information input unit 11, a stereo distance measurement unit 12, a type identification unit 13, a driver operation input unit 14, and an identification history storage unit. 15, a tracking history storage unit 16, a position history storage unit 17, a speed history storage unit 18, an erroneous identification identification unit 19, an erroneous tracking identification unit 20, an erroneous control determination unit 21, an erroneous recognition image storage unit 22, a control feedback unit 23, and an additional learning unit 24. In addition, the vehicle-mounted camera device 100 is connected to a vehicle speed sensor 31, a vehicle steering angle sensor 32, a yaw rate sensor 33, a steering sensor 34, an accelerator pedal sensor 35, and a brake pedal sensor 36 mounted on the own vehicle, and receives output data of each sensor.
なお、車載カメラ装置100の構成のうち、左撮像部1Lと右撮像部1Rを除く構成は、具体的には、演算装置、記憶装置、通信装置などのハードウェアを備えたコンピュータである。そして、CPU(Central Processing Unit)、GPU(Graphics Processing Unit)、FPGA(Field Programmable Gate Array)やASIC(Application Specific Integrated Circuit)、CPLD(Complex Programmable Logic Device)等の演算装置が、プログラム記録媒体または車外の配布サーバから取得した所定のプログラムを実行することで、ステレオマッチング部2などの各処理部を実現し、また、半導体メモリ等の記憶装置が、識別履歴記憶部15などの各記憶部を実現するが、以下では、このようなコンピュータ分野の周知技術を適宜省略することとする。
Note that, of the configuration of the in-vehicle camera device 100, the configuration other than the left imaging unit 1L and the right imaging unit 1R is specifically a computer equipped with hardware such as an arithmetic device, a storage device, and a communication device. Then, arithmetic units such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit), and CPLD (Complex Programmable Logic Device) execute a predetermined program obtained from a program recording medium or a distribution server outside the vehicle, realizing each processing unit such as the stereo matching unit 2. Also, a storage device such as a semiconductor memory identifies Each storage unit such as the history storage unit 15 is implemented, but hereinafter, such well-known technology in the computer field will be omitted as appropriate.
車両制御システム200は、車載カメラ装置100、および、上記各センサと接続されており、車載カメラ装置100の認識結果に応じて、自車両の警報やブレーキやステアリングを、衝突回避や軽減、先行車追従、自車線走行維持などの目的で自動的に制御するシステムである。
The vehicle control system 200 is connected to the in-vehicle camera device 100 and each of the sensors described above, and is a system that automatically controls the warning, braking, and steering of the own vehicle for the purposes of collision avoidance and reduction, following the preceding vehicle, and maintaining the own lane, according to the recognition results of the in-vehicle camera device 100.
<車載カメラ装置100の詳細>
以下、本実施例の車載カメラ装置100の各部詳細を説明する。 <Details of in-vehicle camera device 100>
Details of each part of the vehicle-mountedcamera device 100 of the present embodiment will be described below.
以下、本実施例の車載カメラ装置100の各部詳細を説明する。 <Details of in-
Details of each part of the vehicle-mounted
左撮像部1Lおよび右撮像部1Rは、光を電気信号に変換する撮像センサ(CMOS(Complementary Metal Oxide Semiconductor)等)を備えた単眼カメラである。各撮像センサで電気信号に変換された情報は、さらに各撮像部内で、撮像画像を表す画像データに変換される。左撮像部1Lおよび右撮像部1Rで撮像した画像は、左画像PLおよび右画像PRとして、所定周期(例えば17ms毎)に、ステレオマッチング部2、単眼検知部3、単眼測距部4、テンプレート作成部5、画像記憶部6、類似箇所探索部7、および、画角特定部8に送信される。
The left imaging unit 1L and the right imaging unit 1R are monocular cameras equipped with imaging sensors (CMOS (Complementary Metal Oxide Semiconductor), etc.) that convert light into electric signals. The information converted into an electric signal by each imaging sensor is further converted into image data representing a captured image within each imaging unit. The images captured by the left imaging unit 1L and the right imaging unit 1R are sent as the left image PL and the right image PR to the stereo matching unit 2, the monocular detection unit 3, the monocular distance measurement unit 4, the template creation unit 5, the image storage unit 6, the similar part search unit 7, and the angle of view identification unit 8 at predetermined intervals (for example, every 17 ms).
ステレオマッチング部2は、左撮像部1Lおよび右撮像部1Rから画像データを含むデータを受信し、これを処理することにより視差を演算する。視差とは、複数の撮像部の位置の違いから生じる、同一物体の写る画像座標の差である。視差は、近距離のものは大きく、遠距離のものは小さくなり、視差から距離を算出することが可能である。また、ステレオマッチング部2では、左画像PLおよび右画像PRの画像データの歪みを補正する。例えば、中心射影や透視投影モデルと言われる、同一の高さで同一の奥行距離の物体が、画像座標の水平に並ぶように画像データの歪を補正する。なお、水平方向に並ぶように補正するのは、左撮像部1Lと右撮像部1Rが左右方向に並んで配されているためである。補正された左画像PLおよび右画像PRを用いて、これらの一方を基準となる基準画像データとし、他方を比較対象とする比較画像データとして視差を求める。
The stereo matching unit 2 receives data including image data from the left imaging unit 1L and the right imaging unit 1R, and processes the data to calculate parallax. Parallax is a difference in image coordinates in which the same object is captured, which is caused by differences in the positions of a plurality of imaging units. The parallax is large at short distances and small at long distances, and the distance can be calculated from the parallax. The stereo matching unit 2 also corrects the distortion of the image data of the left image PL and the right image PR . For example, the distortion of the image data is corrected so that objects having the same height and the same depth distance, which are called a central projection model or a perspective projection model, are arranged horizontally in the image coordinates. It should be noted that the reason why the left imaging section 1L and the right imaging section 1R are arranged side by side in the horizontal direction is that the correction is made so that they are aligned in the horizontal direction. Using the corrected left image PL and right image PR , one of these is used as reference image data as a reference, and the other is used as comparison image data for comparison to obtain parallax.
視差を求める際は、まず前述の基準画像データと、比較画像データの消失点の垂直座標を合わせる。そして、基準画像データの各座標に対して、比較画像データの同一垂直座標のどの水平座標が同一物体を映しているのかを、例えば、SSD(Sum of Squared Difference)や、SAD(Sum of Absolute Difference)などの手法で検査する。ただし、SSDやSADに限定されず、そのほかの手法でもよい、例えばコーナー特徴点抽出を行い、同一特徴点かを検査する、FAST(Features from Accelerated Segment Test)やBRIEF(Binary Robust Independent Elementary Features)などの手法を組み合わせて同一物体のマッチングを行ってもよい。
When obtaining the parallax, first match the vertical coordinates of the vanishing points of the reference image data and the comparative image data. Then, for each coordinate of the reference image data, which horizontal coordinate of the same vertical coordinates of the comparative image data shows the same object is inspected by a method such as SSD (Sum of Squared Difference) or SAD (Sum of Absolute Difference). However, it is not limited to SSD or SAD, and other methods may be used. For example, FAST (Features from Accelerated Segment Test) and BRIEF (Binary Robust Independent Elementary Features), which extract corner feature points and check whether they are the same feature points, may be combined to match the same object.
単眼検知部3は、左画像PLまたは右画像PRに映る特定の立体物を検知する。ここで、特定の立体物とは、適正なAD制御やADAS制御を実現するために検知が必要な物体であり、具体的には、自車両周囲の歩行者、他車両、自転車などである。単眼検知部3での検知は車載カメラから一定の範囲内の立体物を検知対象とする。単眼検知部3による検知結果には、検知物体の写る左画像PLまたは右画像PRの画像座標の情報を含む。例えば、検知物体を囲む長方形の枠(以下、「単眼検知枠」と呼ぶ)の左上と左下の垂直水平の画像座標として、検知結果を保持する。
The monocular detection unit 3 detects a specific three-dimensional object appearing in the left image PL or the right image PR . Here, a specific three-dimensional object is an object that needs to be detected in order to realize appropriate AD control or ADAS control, and specifically includes pedestrians, other vehicles, and bicycles around the vehicle. Detection by the monocular detection unit 3 is made to detect three-dimensional objects within a certain range from the vehicle-mounted camera. The detection result by the monocular detection unit 3 includes information on the image coordinates of the left image PL or the right image PR in which the object to be detected is captured. For example, the detection result is held as the vertical and horizontal image coordinates of the upper left and lower left of a rectangular frame (hereinafter referred to as "monocular detection frame") surrounding the detection object.
ステレオ検知部9は、ステレオマッチング部2で作成した視差画から、同一距離で一定のサイズの範囲の箇所を立体物として検知する。ステレオ検知部9による検知結果には、検知物体の視差画座標情報を含む。例えば、検知物体を囲む長方形の枠(以下、「ステレオ検知枠」と呼ぶ)の左上と左下の垂直水平の画像座標として、検知結果を保持する。
From the parallax image created by the stereo matching unit 2, the stereo detection unit 9 detects locations within a certain size range at the same distance as three-dimensional objects. The detection result by the stereo detector 9 includes parallax image coordinate information of the detected object. For example, the detection result is held as the vertical and horizontal image coordinates of the upper left and lower left of a rectangular frame (hereinafter referred to as "stereo detection frame") surrounding the detected object.
種別識別部13は、単眼検知部3やステレオ検知部9で検知した物体の種別を、テンプレート画像やパターンや機械学習の辞書などの識別ネットワークを用いて識別する。種別識別部13で識別した物体の種別が、四輪車、二輪車、歩行者である場合は、それらの移動方向となる物体の前面の向きが、何れの向きなのかも特定し、自車両の前方を横断することが想定される移動体(背景に対して動いている立体物)であるかも識別する。これらの移動体の前面の向きは、車載カメラ装置100に対しての物体の前面の向きである。本実施例では、直交に近い移動方向に限定して動作する。直交に近い移動方向かは、画角と車載カメラ装置100に対する物体の前面の向きにより判断する。例えば、車載カメラ装置100の右側に写る物体の場合、物体の進行する向きの面と側面とが写る。45°の角度の場合は、前面と側面が45°の向きで見える場合に直交に近い移動方向の物体と分かる。種別識別部13は、直交に近い移動方向の物体か否かの判断結果を、速度算出部10に伝達する。
The type identification unit 13 identifies the type of the object detected by the monocular detection unit 3 or the stereo detection unit 9 using an identification network such as a template image, pattern, or machine learning dictionary. When the types of the objects identified by the type identification unit 13 are four-wheeled vehicles, two-wheeled vehicles, and pedestrians, the directions of the front surfaces of the objects, which are moving directions thereof, are also identified, and it is also identified whether they are moving bodies (three-dimensional objects moving against the background) that are assumed to cross in front of the own vehicle. The orientation of the front surface of these mobile objects is the orientation of the front surface of the object with respect to the in-vehicle camera device 100 . In this embodiment, the operation is limited to the directions of movement that are nearly orthogonal. Whether the direction of movement is nearly orthogonal or not is determined based on the angle of view and the orientation of the front surface of the object with respect to the vehicle-mounted camera device 100 . For example, in the case of an object captured on the right side of the in-vehicle camera device 100, the surface and side of the object in the traveling direction are captured. In the case of an angle of 45°, an object with a direction of motion close to orthogonal can be identified when the front and side faces are viewed at a 45° orientation. The type identification unit 13 transmits to the speed calculation unit 10 the determination result as to whether or not the moving direction of the object is nearly orthogonal.
単眼測距部4は、単眼検知部3で検知された特定の物体の位置を特定し、左撮像部1Lまたは右撮像部1Rからの距離と方向を求める。例えば、特定された距離と方向は、自車両前方の奥行距離と横方向の横距離の平面上の位置を特定できる座標系で表す。ただし、カメラからの距離であるユーグリッド距離と方向で表す極座標系で表した情報を保持してもよく、奥行と横の2軸との相互変換は三角関数を用いればよい。また、単眼測距部4は、左画像PL、右画像PR、および合成画像PCを路面に射影した俯瞰画像を用いて、俯瞰画像の垂直座標と水平座標と、実際の路面に対する俯瞰画像の垂直と水平方向の縮尺から位置を特定する。ただし、位置の特定に俯瞰画像を用いることは必須の構成ではない。カメラの位置と姿勢の外部パラメータと、焦点距離・撮像素子の画素ピッチと光学系の歪の情報を用いて、幾何計算を行うことで特定してもよい。
The monocular distance measurement unit 4 identifies the position of a specific object detected by the monocular detection unit 3, and obtains the distance and direction from the left imaging unit 1L or the right imaging unit 1R. For example, the specified distance and direction are represented by a coordinate system that can specify the position on the plane of the depth distance in front of the own vehicle and the lateral distance in the lateral direction. However, information expressed in a polar coordinate system expressed by a Eugrid distance, which is a distance from the camera, and a direction may be held, and a trigonometric function may be used for interconversion between the depth and two horizontal axes. In addition, the monocular distance measurement unit 4 uses an overhead image obtained by projecting the left image P L , the right image P R , and the composite image P C onto the road surface, and specifies the position from the vertical and horizontal coordinates of the overhead image and the vertical and horizontal scales of the overhead image with respect to the actual road surface. However, it is not essential to use a bird's-eye view image to specify the position. It may be specified by performing geometric calculations using extrinsic parameters of the position and orientation of the camera, and information on the focal length, the pixel pitch of the imaging device, and the distortion of the optical system.
ステレオ測距部12は、ステレオ検知部9により検知された物体の位置を特定して距離と方向を特定する。例えば、特定された距離と方向は、自車両前方の奥行距離と横方向の横距離の平面上の位置を特定できる座標系で表す。ただし、カメラからの距離であるユーグリッド距離と方向で表す極座標系で表した情報を保持してもよく、奥行と横の2軸との相互変換は三角関数を用いればよい。また、ステレオ測距部12は、物体の視差から奥行方向の距離を算出する。ステレオ測距部12は、算出した物体の視差にばらつきがある場合は、平均や最頻値を用いる。視差のばらつきが大きい際は、特定の外れ値をとる手法を用いてもよい。横距離は、ステレオ検知部9の検知枠の水平画角と奥行距離から、三角関数を用いて求める。
The stereo ranging unit 12 identifies the position of the object detected by the stereo detection unit 9 and identifies the distance and direction. For example, the specified distance and direction are represented by a coordinate system that can specify the position on the plane of the depth distance in front of the own vehicle and the lateral distance in the lateral direction. However, information expressed in a polar coordinate system expressed by a Eugrid distance, which is a distance from the camera, and a direction may be held, and a trigonometric function may be used for interconversion between the depth and two horizontal axes. Also, the stereo ranging unit 12 calculates the distance in the depth direction from the parallax of the object. The stereo ranging unit 12 uses an average or a mode when there is variation in the calculated parallax of the object. When the disparity of parallax is large, a method of taking a specific outlier may be used. The horizontal distance is obtained from the horizontal angle of view of the detection frame of the stereo detection unit 9 and the depth distance using a trigonometric function.
テンプレート作成部5は、撮像画像の1つを選択し、この撮像画像(以下、「検知画像」と呼ぶ)から特定の領域を切り抜いてテンプレート画像とする。具体的にはテンプレート作成部5は、単眼検知部3で検知された物体の単眼検知枠の内部および周辺の画素情報から、類似箇所を探すためのテンプレート画像を作成する。このテンプレート画像は、あらかじめ定められた画像サイズに拡大縮小される。テンプレート画像を拡大縮小する際は、縦横の比率を維持するか、大きく変えないこととする。拡大縮小処理の後に、画素の輝度値自体か、縮小画像内を複数の小領域(以下、「カーネル」とも呼ぶ)に分割し、カーネル内の輝度値の関係を、その画像の特徴として記憶する。画像の特徴は様々な形で抽出され、カーネル内の左右や上下の平均輝度差や、周辺と中央の平均輝度差、輝度の平均や分散など、様々な特徴量が存在するが、本実施の形態ではいずれを用いてもよい。
The template creation unit 5 selects one of the captured images, cuts out a specific region from this captured image (hereinafter referred to as "detected image"), and uses it as a template image. Specifically, the template creation unit 5 creates a template image for searching for similar portions from pixel information inside and around the monocular detection frame of the object detected by the monocular detection unit 3 . This template image is scaled to a predetermined image size. When enlarging or reducing the template image, maintain the aspect ratio or do not change it significantly. After scaling, the brightness values of the pixels themselves or the reduced image is divided into a plurality of small regions (hereinafter also referred to as "kernels"), and the relationship between the brightness values in the kernels is stored as a feature of the image. Image features are extracted in various forms, and there are various feature amounts such as left-right and top-bottom average brightness differences in the kernel, average brightness differences between the periphery and the center, and brightness averages and variances.
カメラ画像では物標の周囲には背景が映り込み、画像を切り出しすると背景が混ざる。しかし、背景は時刻により、物標やカメラが動くことで、同じ物標であっても変化する。本発明では同一物体を異なる時刻で探し出したいので、前記特徴量の内、追跡したい物標種別の形状やテクスチャを、カメラ画像を用い機械学習を行い、どこの特徴量をどのような重みで保持して類似箇所を探索すべきか、追跡ネットワークとして記憶しておく。
In the camera image, the background is reflected around the target, and when the image is cut out, the background is mixed. However, the background changes depending on the time, even if the target is the same, as the target and the camera move. In the present invention, since it is desired to search for the same object at different times, machine learning is performed using camera images for the shape and texture of the type of target to be tracked among the feature quantities, and which feature quantities should be held with what weights to search for similar locations are stored as a tracking network.
画像記憶部6は、左画像PLおよび右画像PRを一定時間、または異なる時刻の左画像PLおよび右画像PRが一定枚数蓄積されるまで保持する。この保持にはDRAM(Dynamic Random Access Memory)などの一時記憶装置が用いられる。保持する記憶装置内の番地と範囲は、あらかじめ複数定めておき、撮像画像毎に転送先の番地を順に変えていき、一巡後は古い画像が記憶された領域を上書きしていくとよい。予め番地が定められているため、保持された画像を読み出す際に番地を通知する必要がない。
The image storage unit 6 holds the left image PL and the right image PR for a certain period of time, or until a certain number of left images PL and right images PR at different times are accumulated. A temporary storage device such as a DRAM (Dynamic Random Access Memory) is used for this holding. A plurality of addresses and ranges in the storage device to be held may be determined in advance, and the transfer destination address may be changed for each captured image, and after one cycle, the area where the old image is stored may be overwritten. Since the address is determined in advance, there is no need to notify the address when reading out the held image.
類似箇所探索部7は、テンプレート作成部5で作成されたテンプレート画像に類似する箇所を、画像記憶部6に格納される画像の中から探し出す。以下では、類似箇所探索部7がテンプレート画像との類似箇所を探索する画像を「探索対象画像」と呼ぶ。探索対象画像は、単眼検知部3で検知された時刻の画像とは異なる画像である。どの時刻の画像を選択するかは、検知された時刻の画像の前後の時刻に撮像された画像を選ぶことを基本とする。類似箇所は検知された時刻の単眼検知枠座標の近くに存在する可能性が高く、またテンプレート画像に対して、検知物体の向きの変化や、露光変化などによる輝度変化も少ないことが期待されるので、高精度な探索が可能である。
The similar part searching unit 7 searches the images stored in the image storage unit 6 for parts similar to the template image created by the template creating unit 5 . Hereinafter, an image for which the similarity search section 7 searches for a similarity to the template image will be referred to as a "search target image". The search target image is an image different from the image at the time detected by the monocular detection unit 3 . The selection of an image at which time is based on selecting an image captured at a time before or after the image at the detected time. Similar locations are likely to exist near the coordinates of the monocular detection frame at the time of detection, and since it is expected that there will be little change in brightness due to changes in the orientation of the detection object and changes in exposure with respect to the template image, high-precision search is possible.
前記探索された結果と単眼検知部3の検知結果と、ステレオ検知部9の検知結果と、単眼測距部4の測距結果と、ステレオ測距部12の測距結果から、同じ物標が時刻毎にどの位置に居たのか、物標の追跡を行う。この時刻の異なる画像の中で同一の物標を、ある時刻の画像からテンプレート画像を作り、該ある時刻とは異なる時刻の画像の中で類似箇所を探し出し、追跡する処理を、追跡ネットワークによる画像トラッキング処理と呼ぶ。
Based on the search result, the detection result of the monocular detection unit 3, the detection result of the stereo detection unit 9, the distance measurement result of the monocular distance measurement unit 4, and the distance measurement result of the stereo distance measurement unit 12, the position of the same target at each time is tracked. The process of creating a template image from the image at a certain time and searching for and tracking similar portions in the image at a different time is called image tracking processing by a tracking network.
ただし、撮像間隔が短い場合は、2つ以上前の時刻の画像を探索対象画像に選択してもよい。また、車速に応じて選択する画像を変えてもよい。例えば、車速が閾値以上の場合は、検知された画像の撮像時刻に対して、近い時刻の古い画像を選び、車速が前述の閾値よりも遅い場合は、前述の近い時刻の古い画像よりも更に古い画像を選んでもよい。車両速度が速い場合は、検知された時刻に対して近い時刻の画像であれば、テンプレート画像に対して大きく見え方が変わらず、探索の精度が確保しやすい。また、車速が遅い場合は、検知された時刻に対して、類似箇所を探す画像での検知物体の位置変化が大きくなり、算出速度の精度を向上できる。
However, if the imaging interval is short, an image from two or more previous times may be selected as the search target image. Also, the image to be selected may be changed according to the vehicle speed. For example, if the vehicle speed is equal to or greater than a threshold, an older image close to the imaging time of the detected image is selected, and if the vehicle speed is slower than the threshold, an older image than the close old image may be selected. When the vehicle speed is high, if the image is taken at a time close to the detected time, the appearance does not change greatly from the template image, and it is easy to ensure search accuracy. In addition, when the vehicle speed is slow, the positional change of the detected object in the image for searching for a similar part increases with respect to the detected time, and the accuracy of the calculated speed can be improved.
画角特定部8は、類似箇所探索部7が特定した類似箇所に写る特定の物体の、カメラの水平画角を特定する。ただし位置とともに高さも特定する場合は、垂直画像も特定する。また、カメラがロール回転する可能性がある場合も、水平と垂直の両方の画角を特定することで、速度を高精度に算出できる。水平画角は、奥行距離と横距離の比率から三角関数で求めればよい。
The angle-of-view specifying unit 8 specifies the horizontal angle of view of the camera for a specific object appearing in the similar location specified by the similar location searching unit 7 . However, if the height is specified along with the position, then the vertical image is also specified. Also, even if there is a possibility that the camera rolls, the speed can be calculated with high accuracy by specifying both the horizontal and vertical angles of view. The horizontal angle of view can be obtained by a trigonometric function from the ratio of the depth distance and the horizontal distance.
速度算出部10は、単眼測距部4またはステレオ測距部12から測距結果を受け取り、画角特定部8から類似箇所の画角を受け取り、車両情報入力部11から車両挙動情報を受け取り、検知された特定の物体の速度を算出する。ただし速度算出部10は、車両情報入力部11から車両速度を受け取る代わりに、算出した相対的な奥行距離の微分値を車両速度として用いてもよい。これは、奥行距離は精度が良く測距でき、横距離は精度が悪く測距される場合に有用である。また奥行速度を車両速度の代わりに用いると、直交しない横断車両との衝突予測も精度よくできる。
The speed calculation unit 10 receives the distance measurement result from the monocular distance measurement unit 4 or the stereo distance measurement unit 12, receives the angle of view of the similar location from the angle of view identification unit 8, receives vehicle behavior information from the vehicle information input unit 11, and calculates the speed of the detected specific object. However, instead of receiving the vehicle speed from the vehicle information input unit 11, the speed calculation unit 10 may use the calculated differential value of the relative depth distance as the vehicle speed. This is useful when the depth distance can be measured with high accuracy and the horizontal distance is measured with low accuracy. Also, if the depth velocity is used instead of the vehicle velocity, it is possible to accurately predict a collision with a crossing vehicle that is not orthogonal.
車両情報入力部11は、自車両の速度を測定する車両速度センサ31からの自車両速度情報と、操舵輪の舵角を測定する車両舵角センサ32からの自車両舵角情報と、自車両の旋回速度を測定するヨーレートセンサ33からの自車両旋回速度が入力される。なお、車両情報入力部11は、通信ポート(例えばIEEE802.3)に対応する通信モジュールにより実現されてもよいし、電圧や電流を読み込み可能なADコンバータにより実現されてもよい。
The vehicle information input unit 11 receives the vehicle speed information from the vehicle speed sensor 31 that measures the speed of the vehicle, the steering angle information from the vehicle steering angle sensor 32 that measures the steering angle of the steered wheels, and the turning speed of the vehicle from the yaw rate sensor 33 that measures the turning speed of the vehicle. The vehicle information input unit 11 may be realized by a communication module compatible with a communication port (for example, IEEE802.3), or may be realized by an AD converter capable of reading voltage and current.
運転者操作入力部14は、ステアリングを回した角度情報を取得するステアリングセンサ34と、アクセスペダルの踏込量情報を取得するアクセルペダルセンサ35と、ブレーキペダルの踏込量情報を取得するブレーキペダルセンサ36に接続されており、運転者がステアリング、アクセルペダル、ブレーキペダルをそれぞれ、どの程度操作したか、および、いつ操作したかの情報を受信する。
The driver operation input unit 14 is connected to a steering sensor 34 that acquires information on the angle of turning the steering wheel, an accelerator pedal sensor 35 that acquires information on the amount of depression of the access pedal, and a brake pedal sensor 36 that acquires information on the amount of depression of the brake pedal.
識別履歴記憶部15は、種別識別部13で識別した立体物(以下、「物標」と呼ぶ)を、物標の識別子と、物標の時刻とともに記憶する。なお、物標の識別子とは、物標毎に付与されるユニークな符号であり、例えば、1台の車両と3人の歩行者を物標として識別した場合であれば、車両A、歩行者A、歩行者B、歩行者C、のような識別子が各物標に付与される。
The identification history storage unit 15 stores the three-dimensional object (hereinafter referred to as "target") identified by the type identification unit 13 together with the target identifier and the time of the target. The target identifier is a unique code assigned to each target. For example, if one vehicle and three pedestrians are identified as targets, identifiers such as vehicle A, pedestrian A, pedestrian B, and pedestrian C are assigned to each target.
追跡履歴記憶部16は、テンプレート作成部5と類似箇所探索部7で、テンプレート画像の作成の成功否、各時刻で類似箇所を探索した際のテンプレート画像、類似箇所の探索結果の成功否、類似箇所として検出された画像を、物標の識別子と時刻とともに記憶する。
The tracking history storage unit 16 stores the success or failure of the template image creation by the template creation unit 5 and the similar location search unit 7, the template image when searching for similar locations at each time, the success or failure of the similar location search results, and the image detected as the similar location, together with the identifier of the target and the time.
位置履歴記憶部17は、前記探索された結果と単眼検知部3の検知結果と、ステレオ検知部9の検知結果と、単眼測距部4の測距結果と、ステレオ測距部12の測距結果から、同じ物標が時刻毎にどの位置に居たのか、物標の識別子と時刻とともに記憶する。
The position history storage unit 17 stores the location of the same target at each time, along with the target identifier and time, based on the search result, the detection result of the monocular detection unit 3, the detection result of the stereo detection unit 9, the distance measurement result of the monocular distance measurement unit 4, and the distance measurement result of the stereo distance measurement unit 12.
速度履歴記憶部18は、速度算出部10で算出された速度を、物標の識別子と時刻とともに記憶する。
The speed history storage unit 18 stores the speed calculated by the speed calculation unit 10 together with the identifier of the target and the time.
制御フィードバック部23は、車両制御システム200に接続され、車両制御システム200が制御した車両制御内容と時刻と対象物標識別子、制御判断理由、制御の取りやめ内容、制御取りやめ理由などの制御フィードバック情報を入力する。
The control feedback unit 23 is connected to the vehicle control system 200, and inputs control feedback information such as vehicle control content and time controlled by the vehicle control system 200, target target identifier, control decision reason, control cancellation content, and control cancellation reason.
誤制御判定部21は、制御フィードバック部23から自車両の制御フィードバック情報を受信し、運転者操作入力部14から運転者操作情報を受信する。また、車両制御システム200が車載カメラ装置100の出力の物標情報をもとに車両制御を行い、警報や自動ブレーキを開始した後に、運転者操作により該制御を取りやめたかを判断する。そして、そのような運転操作があった場合に、誤制御判定部21は、車両制御システム200による誤制御があったと判定する。なお、誤制御判定部21の詳細は後述する。
The erroneous control determination unit 21 receives the control feedback information of the own vehicle from the control feedback unit 23 and receives the driver's operation information from the driver's operation input unit 14 . Further, the vehicle control system 200 performs vehicle control based on the target object information output from the in-vehicle camera device 100, and after starting the alarm and automatic braking, it is determined whether or not the control is canceled by the driver's operation. Then, when there is such a driving operation, the erroneous control determination unit 21 determines that erroneous control by the vehicle control system 200 has occurred. The details of the erroneous control determination unit 21 will be described later.
誤識別特定部19は、誤制御判定部21によって誤制御と判断された後に、誤制御の対象だった物標の識別結果の内、誤識別の事例を特定する。なお、誤識別特定部19の詳細は後述する。
The erroneous identification identification unit 19 identifies cases of erroneous identification among the identification results of the targets that were the target of erroneous control after the erroneous control determination unit 21 determines that an erroneous control has occurred. The details of the erroneous identification specifying unit 19 will be described later.
誤追跡特定部20は、誤制御判定部21によって誤制御と判断された後に、誤制御の対象だった物標の追跡結果の内、誤追跡の事例を特定する。なお、誤追跡特定部20の詳細は後述する。
The erroneous tracking identification unit 20 identifies a case of erroneous tracking among the tracking results of the target that was the target of erroneous control after the erroneous control determination unit 21 determined that the target was erroneously controlled. The details of the mistracking identification unit 20 will be described later.
誤認識画像保存部22は、誤識別特定部19や誤追跡特定部20で特定された誤識別や誤追跡の前後の画像を抽出して記憶する。
The erroneous recognition image storage unit 22 extracts and stores images before and after the erroneous identification and erroneous tracking specified by the erroneous identification specifying unit 19 and the erroneous tracking specifying unit 20 .
追加学習部24は、誤認識画像保存部22に記憶された、誤識別や誤追跡の前後の画像を用いて、外部環境を認識するためのAIネットワークを追加学習する。
The additional learning unit 24 additionally learns the AI network for recognizing the external environment using the images before and after the erroneous identification and erroneous tracking stored in the erroneous recognition image storage unit 22 .
以下、上記した構成の車載カメラ装置100による、学習データの生成処理と、生成した学習データを用いた追加学習処理を、具体的な状況を例示しながら説明する。
Hereinafter, the learning data generation processing and the additional learning processing using the generated learning data by the in-vehicle camera device 100 configured as described above will be described while exemplifying specific situations.
<誤認識の一態様である、誤識別の発生状況の一例>
図4は、車載カメラ装置100の種別識別部13で、識別ネットワーク等を利用して外部環境を認識した結果、誤識別が発生した状況の一例を示す俯瞰図である。なお、ここでは、外部環境のサンプリング周期を200msとしているが、サンプリング周期はこの例に限定されるものではない。 <Example of erroneous identification, which is one aspect of erroneous recognition>
FIG. 4 is a bird's-eye view showing an example of a situation in which erroneous identification occurs as a result of recognizing the external environment using an identification network or the like in thetype identification unit 13 of the in-vehicle camera device 100 . Although the sampling period of the external environment is set to 200 ms here, the sampling period is not limited to this example.
図4は、車載カメラ装置100の種別識別部13で、識別ネットワーク等を利用して外部環境を認識した結果、誤識別が発生した状況の一例を示す俯瞰図である。なお、ここでは、外部環境のサンプリング周期を200msとしているが、サンプリング周期はこの例に限定されるものではない。 <Example of erroneous identification, which is one aspect of erroneous recognition>
FIG. 4 is a bird's-eye view showing an example of a situation in which erroneous identification occurs as a result of recognizing the external environment using an identification network or the like in the
図4(a)は、後述する時刻T0より200ms前の時刻T-1における、自車両の外部環境を例示する俯瞰図である。この時点では、自車両は直線道路を等速走行しており、自車両に取り付けた車載カメラ装置100は、自車両の前方に立体物を検知していない。
FIG. 4(a) is a bird's-eye view illustrating the external environment of the own vehicle at time T - 1 , which is 200 ms before time T0, which will be described later. At this point, the vehicle is traveling at a constant speed on a straight road, and the in-vehicle camera device 100 attached to the vehicle has not detected a three-dimensional object in front of the vehicle.
図4(b)は、自動制御(自動減速)を開始した時刻T0における、自車両の外部環境を例示する俯瞰図である。この時点で、車載カメラ装置100は、やや問題ある識別ネットワーク等を利用して撮像画像を処理した結果、実在しない歩行者を誤検知している。そのため、車載カメラ装置100の検知結果を受信した車両制御システム200は、実在しない歩行者との接触を防ぐため、自車両の制動系を制御し、自車両を急減速させる。なお、実在しない歩行者を誤検知する状況は、例えば、排気ガス塊を歩行者と誤識別するような状況である。
FIG. 4B is a bird's-eye view illustrating the external environment of the host vehicle at time T0 when automatic control (automatic deceleration) is started. At this point, the in-vehicle camera device 100 erroneously detects a non-existent pedestrian as a result of processing the captured image using a slightly problematic identification network or the like. Therefore, the vehicle control system 200 that receives the detection result of the in-vehicle camera device 100 controls the braking system of the own vehicle and decelerates the own vehicle rapidly in order to prevent contact with a nonexistent pedestrian. A situation in which a non-existing pedestrian is erroneously detected is, for example, a situation in which a lump of exhaust gas is erroneously identified as a pedestrian.
図4(c)は、時刻T0から200msを経過した時刻T1における、自車両の外部環境を例示する俯瞰図である。この時点で、運転者は、自車両を制動する理由がないのに自動減速が開始されたことから、車載カメラ装置100で何らかの異常(誤識別)が発生したことに気付く。
FIG. 4(c) is a bird's-eye view illustrating the external environment of the host vehicle at time T1 , 200 ms after time T0 . At this point, the driver notices that an abnormality (erroneous identification) has occurred in the in-vehicle camera device 100 because automatic deceleration has started without any reason to brake the vehicle.
図4(d)は、時刻T0から400msを経過した時刻T2における、自車両の外部環境を例示する俯瞰図である。この時点で、前方の安全(前方車両の不在や、周囲歩行者の不在など)を確認した運転者は、減速された速度を減速前の速度に復帰させるべくアクセルペダルを踏み込み、自車両を加速させる。
FIG. 4D is a bird's-eye view illustrating the external environment of the host vehicle at time T2 , 400 ms after time T0 . At this point, the driver who has confirmed the safety in front (no vehicle ahead, no pedestrians, etc.) depresses the accelerator pedal in order to restore the decelerated speed to the pre-deceleration speed to accelerate the own vehicle.
図4(e)は、時刻T0から600msを経過した時刻T3における、自車両の外部環境を例示する俯瞰図である。この時点で、自車両は、時刻T0の時点で誤識別した歩行者の領域に侵入しているが、そこには実際には歩行者が存在しないため、自車両はその領域を無事に通過することができる。
FIG. 4(e) is a bird's-eye view illustrating the external environment of the host vehicle at time T3 , 600 ms after time T0 . At this point, the vehicle has entered the area of the pedestrian that was misidentified at time T0 , but since there are no pedestrians there, the vehicle can safely pass through the area.
以上から明らかなように、車載カメラ装置100の識別結果に基づいて車両制御システム200が自動制御を開始した直後に、運転者がその自動制御に反する手動運転操作を実施し、かつ、識別した立体物との接触が発生しなかった場合は、車載カメラ装置100で誤識別が発生したと判定することができる。本発明の装置は、誤識別と判断された、時刻T0の時点で誤識別した歩行者物標の画像を記憶・追加学習する。
As is clear from the above, immediately after the vehicle control system 200 starts automatic control based on the identification result of the on-vehicle camera device 100, if the driver performs a manual driving operation contrary to the automatic control and there is no contact with the identified three-dimensional object, it can be determined that an erroneous identification has occurred in the on-vehicle camera device 100. The apparatus of the present invention memorizes and additionally learns images of pedestrian targets misidentified at time T0 , which are determined to be misidentified.
<誤識別の画像の保存方法と、保存画像を利用した追加学習方法>
図5は、図4で説明した機序を応用したフローチャートであり、車載カメラ装置100で誤識別が発生した時刻の画像を保存し、更に、保存した画像に基づいて識別ネットワーク等を追加学習する方法を示すものである。以下、各ステップを順次説明する。 <Method for saving misidentified images and additional learning method using saved images>
FIG. 5 is a flowchart to which the mechanism described in FIG. 4 is applied, and shows a method of saving an image at the time when an erroneous identification occurs in the on-vehicle camera device 100, and additionally learning an identification network or the like based on the saved image. Each step will be described below.
図5は、図4で説明した機序を応用したフローチャートであり、車載カメラ装置100で誤識別が発生した時刻の画像を保存し、更に、保存した画像に基づいて識別ネットワーク等を追加学習する方法を示すものである。以下、各ステップを順次説明する。 <Method for saving misidentified images and additional learning method using saved images>
FIG. 5 is a flowchart to which the mechanism described in FIG. 4 is applied, and shows a method of saving an image at the time when an erroneous identification occurs in the on-
まず、ステップS1では、誤制御判定部21は、制御フィードバック部23が入手した車両制御システム200からのデータに基づいて、衝突防止の自動制御が動作したか否かを判定する。衝突防止の自動制御とは、例えば、衝突回避や軽減の自動ブレーキや、衝突警報を鳴らす制御や、障害物を回避するステアリング制御等である。そして、車載カメラ装置100の出力に基づく自動制御が動作した場合は、ステップS2に進み、動作していない場合は、一定時間後にステップS1を再度実行する。
First, in step S1, the erroneous control determination unit 21 determines whether or not the automatic control for collision prevention has operated based on the data from the vehicle control system 200 that the control feedback unit 23 has obtained. The automatic control for collision prevention includes, for example, automatic braking for collision avoidance and mitigation, control for sounding a collision alarm, steering control for avoiding obstacles, and the like. Then, if the automatic control based on the output of the in-vehicle camera device 100 operates, the process proceeds to step S2, and if it does not operate, step S1 is executed again after a certain period of time.
次に、ステップS2では、誤制御判定部21は、運転者操作入力部14が入手したステアリングセンサ34やアクセルペダル35などからのデータに基づいて、自動制御に反する手動の運転操作がなされたか否かを判定する。図4で例示したように、検知した立体物との接触防止の自動制御が発動された後に、運転手が自動制御に反する手動操作を実施した場合、運転手は安全を確認したうえで、その手動操作を行っていると考えられるため、車両制御システム200による衝突回避の自動制御は誤りであったと判定することができる。そこで、本ステップでは、自動制御に反する手動操作が実施された場合はステップS3に進み、自動制御に反する手動操作が実施されなかった場合はステップS1に戻る。
Next, in step S2, the erroneous control determination unit 21 determines whether or not a manual driving operation contrary to automatic control has been performed, based on data from the steering sensor 34 and the accelerator pedal 35 obtained by the driver operation input unit 14. As exemplified in FIG. 4, when the driver performs a manual operation contrary to the automatic control after the automatic control for preventing contact with the detected three-dimensional object is activated, it is possible to determine that the automatic control for collision avoidance by the vehicle control system 200 has been erroneous because it is considered that the driver is performing the manual operation after confirming safety. Therefore, in this step, if the manual operation contrary to the automatic control is performed, the process proceeds to step S3, and if the manual operation contrary to the automatic control is not performed, the process returns to step S1.
本ステップで、走行制御システム200の判断が誤りであったと判定される状況を、具体例を挙げて説明する。
A specific example will be given to explain the situation in which it is determined in this step that the travel control system 200 made an error.
<<誤制御の第一例>>
車両制御システム200が物標との衝突を判断し、前記衝突点手前で停車を目標とする自動ブレーキを開始した状況を想定する。この場合、自動ブレーキが開始された後で且つ停車する前に、運転手がやや強い(具体的には、所定の閾値Th1と閾値Th2(Th1<Th2)の間の)踏込量でアクセルペダルを踏むと、自動ブレーキの解除を意図した操作として操作を受付け、自動ブレーキ制御を解除する。その結果、図4(e)の例のように、加速しながら前記衝突点前を通過することになる。このように、車両制御システム200が衝突点前で止まるように自動ブレーキを開始した後に、運転手のアクセルペダル操作で衝突点を通過した場合は、自動制御に反する運転操作が実施されたと判断してステップS3に進む。 <<First example of erroneous control>>
A situation is assumed in which thevehicle control system 200 determines that the vehicle has collided with a target and starts automatic braking with the goal of stopping the vehicle before the collision point. In this case, when the driver depresses the accelerator pedal slightly strongly (specifically, between a predetermined threshold value Th1 and a threshold value Th2 (Th1<Th2)) after the automatic braking is started and before the vehicle stops, the operation is accepted as an operation intended to release the automatic braking, and the automatic braking control is released. As a result, as shown in the example of FIG. 4(e), the collision point is passed while accelerating. In this way, after the vehicle control system 200 starts the automatic braking so as to stop before the collision point, when the collision point is passed by the driver's accelerator pedal operation, it is determined that the driving operation contrary to the automatic control is performed, and the process proceeds to step S3.
車両制御システム200が物標との衝突を判断し、前記衝突点手前で停車を目標とする自動ブレーキを開始した状況を想定する。この場合、自動ブレーキが開始された後で且つ停車する前に、運転手がやや強い(具体的には、所定の閾値Th1と閾値Th2(Th1<Th2)の間の)踏込量でアクセルペダルを踏むと、自動ブレーキの解除を意図した操作として操作を受付け、自動ブレーキ制御を解除する。その結果、図4(e)の例のように、加速しながら前記衝突点前を通過することになる。このように、車両制御システム200が衝突点前で止まるように自動ブレーキを開始した後に、運転手のアクセルペダル操作で衝突点を通過した場合は、自動制御に反する運転操作が実施されたと判断してステップS3に進む。 <<First example of erroneous control>>
A situation is assumed in which the
なお、この例では、アクセスペダルの踏込量が上記の所定範囲内にある場合のみ、その操作を停車制御解除の操作として受け付けたが、これは次の理由による。すなわち、運転者が閾値Th2を超えてアクセルペダルを強く踏み込んだ場合は、アクセスペダルをブレーキペダルと誤解して強く踏み込んでいる可能性があり、これを停車制御解除の操作として受け付けるのは不適切と考えられるからである。また、運転者が閾値Th1に達しない弱さでアクセルペダルを踏み続けていた場合は、衝突可能性に気付いていない可能性があり、これを停車制御解除の操作として受け付けるのは不適切と考えられるからである。そのため、閾値Th1と閾値Th2の間の踏込量の場合のみに、自動ブレーキの解除を行い、誤制御として扱うこととしている。
In this example, only when the amount of depression of the access pedal is within the above-mentioned predetermined range, the operation is accepted as the operation for canceling the stop control, for the following reasons. That is, when the driver strongly depresses the accelerator pedal exceeding the threshold value Th2, there is a possibility that the driver misunderstands the access pedal as the brake pedal and strongly depresses it, and it is considered inappropriate to accept this as an operation to cancel the stop control. In addition, if the driver continues to depress the accelerator pedal weakly below the threshold Th1, the driver may not be aware of the possibility of a collision, and it is considered inappropriate to accept this as an operation to cancel the stop control. Therefore, only when the depression amount is between the threshold Th1 and the threshold Th2, the automatic brake is released and treated as erroneous control.
<<誤制御の第二例>>
車両制御システム200が物標との衝突を判断し、前記衝突点手前で停車を目標とする自動ブレーキを開始した状況を想定する。この場合、自動ブレーキにより停車した後、あらかじめ定められた時間以内に、運転手がアクセルペダルを踏み再発進を行い、前記衝突点前を通過する。前記時間は例えば0.5秒間等、障害物が衝突点に存在した場合に障害物が退避する時間よりも十分短い時間である。前記障害物が退避する時間は、該障害物の横断速度から考え、衝突点が自車両の左右中央であれば、車幅分だけ該横断速度で移動する時間である。 <<Second example of erroneous control>>
A situation is assumed in which thevehicle control system 200 determines that the vehicle has collided with a target and starts automatic braking with the goal of stopping the vehicle before the collision point. In this case, after the vehicle is stopped by automatic braking, the driver depresses the accelerator pedal to restart the vehicle within a predetermined time, and the vehicle passes before the collision point. The time is, for example, 0.5 seconds, which is sufficiently shorter than the time required for the obstacle to retreat when the obstacle is present at the collision point. Considering the crossing speed of the obstacle, the time for the obstacle to retreat is the time for moving at the crossing speed by the width of the vehicle if the collision point is in the left-right center of the own vehicle.
車両制御システム200が物標との衝突を判断し、前記衝突点手前で停車を目標とする自動ブレーキを開始した状況を想定する。この場合、自動ブレーキにより停車した後、あらかじめ定められた時間以内に、運転手がアクセルペダルを踏み再発進を行い、前記衝突点前を通過する。前記時間は例えば0.5秒間等、障害物が衝突点に存在した場合に障害物が退避する時間よりも十分短い時間である。前記障害物が退避する時間は、該障害物の横断速度から考え、衝突点が自車両の左右中央であれば、車幅分だけ該横断速度で移動する時間である。 <<Second example of erroneous control>>
A situation is assumed in which the
物標が歩行者等の非車両の場合は、自車両との衝突の可能性のある状況で驚き立ち止まる可能性があるので、前記移動する時間に数秒の立ち止まる時間を加味してもよい。前記のように、車両制御システム200が衝突点前に停車した後に、運転手のアクセルペダル操作前記短時間で再発進し衝突点を通過した場合は、自動制御に反する運転操作が実施されたと判断してステップS3に進む。
If the target is a non-vehicle such as a pedestrian, there is a possibility that the vehicle will be surprised and stop in a situation where there is a possibility of collision with the own vehicle. As described above, after the vehicle control system 200 has stopped before the collision point, if the vehicle restarts and passes through the collision point in a short period of time after the driver operates the accelerator pedal, it is determined that a driving operation contrary to the automatic control has been performed, and the process proceeds to step S3.
<<誤制御の第三例>>
車両制御システム200が物標との衝突を判断したが、物標との距離が近く、速やかに自動ブレーキを開始しても、衝突点手前で停車できない場合、車両制御システム200は衝突軽減をするために自動ブレーキを行う。前記自動ブレーキを開始後に、運転手がアクセルペダルを閾値Th1と閾値Th2の間の踏込量でアクセルペダルを踏むと、車両制御システム200は運転手が自動ブレーキの解除を意図したとして自動ブレーキ制御を解除する。前記衝突点手前で停車できない状況下で、運転手がアクセルペダル操作をして、自動ブレーキを解除した場合は、自動制御に反する運転操作が実施されたと判断し、ステップS3に進む。 <<Third example of erroneous control>>
When thevehicle control system 200 determines the collision with the target, but the distance to the target is short and the vehicle cannot be stopped in front of the collision point even if automatic braking is started immediately, the vehicle control system 200 performs automatic braking to reduce the collision. After starting the automatic braking, when the driver depresses the accelerator pedal with a depression amount between the threshold Th1 and the threshold Th2, the vehicle control system 200 determines that the driver intended to release the automatic braking and releases the automatic braking control. When the driver operates the accelerator pedal to release the automatic brake under the condition that the vehicle cannot be stopped before the collision point, it is judged that the driving operation contrary to the automatic control has been performed, and the process proceeds to step S3.
車両制御システム200が物標との衝突を判断したが、物標との距離が近く、速やかに自動ブレーキを開始しても、衝突点手前で停車できない場合、車両制御システム200は衝突軽減をするために自動ブレーキを行う。前記自動ブレーキを開始後に、運転手がアクセルペダルを閾値Th1と閾値Th2の間の踏込量でアクセルペダルを踏むと、車両制御システム200は運転手が自動ブレーキの解除を意図したとして自動ブレーキ制御を解除する。前記衝突点手前で停車できない状況下で、運転手がアクセルペダル操作をして、自動ブレーキを解除した場合は、自動制御に反する運転操作が実施されたと判断し、ステップS3に進む。 <<Third example of erroneous control>>
When the
<<誤制御の第四例>>
車両制御システム200が物標との衝突を判断して、物標を回避するために、該物標の右か左に自車両を廻り込ませる自動ステアリング操作をして、衝突回避動作を制御する場合もある。この場合、運転手が前記自動ステアリング操作に対して、妨げるようなステアリング操作をした場合も、自動制御に反する運転操作が実施されたと判断し、ステップS3に進む。前記のステアリング操作は、例えば、車両制御システム200が前記自動ステアリング操作で、ステアリングを45°右回しを目標として制御したのに対して、運転手が左回し方向の力をかけて、45°よりも十分小さい、例えば22°程度までしか回らない状態にした場合などである。 <<Fourth example of erroneous control>>
In some cases, thevehicle control system 200 determines a collision with a target and performs automatic steering operation to turn the vehicle to the right or left of the target in order to avoid the target, thereby controlling the collision avoidance operation. In this case, even if the driver performs a steering operation that interferes with the automatic steering operation, it is determined that a driving operation contrary to the automatic control has been performed, and the process proceeds to step S3. The above steering operation is, for example, when the vehicle control system 200 controls the automatic steering operation to turn the steering wheel clockwise by 45°, but the driver applies a force in the counterclockwise direction to make the steering wheel turn to a state sufficiently smaller than 45°, for example, to only about 22°.
車両制御システム200が物標との衝突を判断して、物標を回避するために、該物標の右か左に自車両を廻り込ませる自動ステアリング操作をして、衝突回避動作を制御する場合もある。この場合、運転手が前記自動ステアリング操作に対して、妨げるようなステアリング操作をした場合も、自動制御に反する運転操作が実施されたと判断し、ステップS3に進む。前記のステアリング操作は、例えば、車両制御システム200が前記自動ステアリング操作で、ステアリングを45°右回しを目標として制御したのに対して、運転手が左回し方向の力をかけて、45°よりも十分小さい、例えば22°程度までしか回らない状態にした場合などである。 <<Fourth example of erroneous control>>
In some cases, the
ステップS3では、誤制御判定部21は、車両速度センサ31、車両舵角センサ32、ヨーレートセンサ33の出力等に基づいて、自動制御以後の走行軌跡(自車両の位置の軌跡)を算出し、運転操作とともに記憶する。具体的には、運転手の運転操作の結果として、アクセルペダル操作、ブレーキペダル操作、ステアリング操作、車両速度、車両舵角、車両ヨーレートの情報を各種センサから取得し、続くステップS4の判断に必要な走行軌跡を算出して推測する。
In step S3, the erroneous control determination unit 21 calculates the travel trajectory after automatic control (the trajectory of the vehicle's position) based on the outputs of the vehicle speed sensor 31, the vehicle steering angle sensor 32, and the yaw rate sensor 33, and stores it together with the driving operation. Specifically, information on accelerator pedal operation, brake pedal operation, steering operation, vehicle speed, vehicle steering angle, and vehicle yaw rate is obtained from various sensors as a result of the driver's driving operation, and the travel trajectory necessary for the subsequent determination in step S4 is calculated and estimated.
ステップS4では、誤識別特定部19は、自動制御開始時点の識別結果が正しかったと仮定した場合に、識別した物標との衝突を回避できる走行軌跡を手動運転操作で走行したかを判断することで、誤識別の発生を特定する。具体的には、識別履歴記憶部15と位置履歴記憶部17で記憶している識別履歴と位置履歴に対して、ステップS3で推測した自車両の軌跡を照らし合わせ、前記識別履歴が歩行者や自転車や車両等の衝突回避を、車両制御システム200で設定された識別結果の履歴上の位置全てに対して、衝突回避不可な走行であったか否かを判断し、衝突回避不可だと判断された前記履歴上の位置と時刻と物標の識別子を記憶する。そして、1ヶ所でも衝突回避不可の走行軌跡と判断される履歴位置があった場合は、運転者はその位置を安全に通過できると判定したうえでその走行軌跡を選択した(すなわち、車両制御システム200に本来不要な自動制御を発動させた誤識別があった)と判定し、ステップS5に進む。一方、そうでなかった場合は、車両制御システム200による自動制御は必要であった(すなわち、車両制御システム200に自動制御を発動させた識別は適切であった)と判定し、ステップS1に戻る。
In step S4, the erroneous identification identifying unit 19 identifies the occurrence of erroneous identification by determining whether the vehicle has traveled in manual operation on a travel trajectory that can avoid collision with the identified target, assuming that the identification result at the start of automatic control was correct. Specifically, the trajectory of the own vehicle estimated in step S3 is compared with the identification history and position history stored in the identification history storage unit 15 and the position history storage unit 17, and the collision avoidance of pedestrians, bicycles, vehicles, etc. is determined in the identification history for all historical positions of the identification result set by the vehicle control system 200, and whether or not the collision could not be avoided is determined. If there is even one historical position judged to be a travel locus where collision avoidance cannot be avoided, it is determined that the driver has selected the travel locus after determining that the driver can pass through that position safely (that is, it is determined that there was an erroneous identification that caused the vehicle control system 200 to activate an essentially unnecessary automatic control), and the process proceeds to step S5. On the other hand, if not, it is determined that the automatic control by the vehicle control system 200 was necessary (that is, the identification that caused the vehicle control system 200 to activate the automatic control was appropriate), and the process returns to step S1.
ステップS5では、誤識別特定部19は、誤識別の画像を、画像記憶部6から一時記憶された画像を読み取って誤認識画像保存部22に保存する。具体的には、前記記憶した履歴上の位置と時刻と物標の識別子に該当する識別画像をメモリに保存し記憶する。これにより、誤認識画像保存部22に保存する画像を、例えば、多くとも誤識別発生の数枚程度(数百KB程度のデータ量)に抑制できるため、走行中に撮像した全ての画像を学習データとして保存する場合に比べ、誤認識画像保存部22の記憶容量を大幅に削減することができる。
In step S<b>5 , the erroneous identification specifying unit 19 reads the image of the erroneous identification, which is temporarily stored in the image storage unit 6 , and saves it in the erroneous recognition image storage unit 22 . Specifically, the stored position and time in the history and the identification image corresponding to the identifier of the target object are stored in the memory. As a result, the number of images to be stored in the erroneously recognized image storage unit 22 can be reduced to, for example, at most several images (a data amount of about several hundred KB) for which erroneous identification occurs, so that the storage capacity of the erroneously recognized image storage unit 22 can be significantly reduced compared to the case where all the images captured while driving are stored as learning data.
ステップS6では、追加学習部24は、ステップS5で誤認識画像保存部22に保存された、誤識別の画像を学習データとして、識別ネットワーク等の追加機械学習を実施する。これにより、追加機械学習の頻度を抑制して演算負荷を軽減しつつ、追加機械学習により識別ネットワーク等の品質向上を図ることができる。本ステップでの追加学習の際には、誤認識画像保存部22に保存された識別画像をそのまま学習データとして用いて追加学習させても良いし、保存された識別画像を加工した学習データ(例えば、識別画像を拡大または縮小した学習データ、あるいは、識別画像を回転させた学習データ)を学習データとして用いて追加学習させても良い。これにより、誤識別した際の座標と完全一致した画像以外の状況も学習することができ、検知枠のブレが生じてもロバストに誤識別を抑制することができる。
In step S6, the additional learning unit 24 performs additional machine learning of an identification network, etc., using the erroneously identified images stored in the erroneously recognized image storage unit 22 in step S5 as learning data. As a result, the frequency of additional machine learning can be suppressed to reduce the computational load, and the additional machine learning can improve the quality of the identification network and the like. During the additional learning in this step, the identification image stored in the misrecognized image storage unit 22 may be used as learning data for additional learning, or learning data obtained by processing the stored identification image (for example, learning data obtained by enlarging or reducing the identification image, or learning data obtained by rotating the identification image) may be used as learning data for additional learning. As a result, it is possible to learn situations other than the image that completely matches the coordinates at the time of erroneous identification, and to robustly suppress erroneous identification even if the detection frame is blurred.
なお、ステップS5で識別画像を保存する際には、誤識別の態様を特定するラベルを、識別画像とともに保存することが望ましい。本実施例の車両制御システム200は、検知した物標の種別(他車両、歩行者、自転車など)に応じて、車両制御を実行するか否かの判断を変えるように設計されているため、物標種別の判断を間違ったことが誤制御要因の候補として考えられる。そこで、識別画像に付されるラベルには、誤識別した識別ネットワークを特定する識別子と、その識別ネットワークが誤識別した物標の種別を記録しておく。
It should be noted that when storing the identification image in step S5, it is desirable to store a label specifying the mode of misidentification together with the identification image. Since the vehicle control system 200 of this embodiment is designed to change the determination of whether or not to execute vehicle control according to the type of detected target (another vehicle, pedestrian, bicycle, etc.), an incorrect determination of the type of target is considered as a possible cause of erroneous control. Therefore, an identifier for identifying an erroneously identified identification network and a type of target erroneously identified by the identification network are recorded in the label attached to the identification image.
誤識別した識別ネットワークに識別が誤りであったことを学習させるためには、学習データ(画像)とともに、物標をどのように誤識別したかの種別クラス情報も必要である。そのため、ステップS5でラベルに種別クラス情報を保存しておくことで、ステップS6では、誤識別発生時の画像を用いて、ラベルが特定する問題のある識別ネットワーク等に対してピンポイントで追加の機械学習を実施させることができ、問題のある識別ネットワーク等の改善を図ることができる。また、問題のない識別ネットワーク等を追加学習の対象から除外できるため、正常な識別ネットワーク等の誤学習の発生や、正常な識別ネットワーク等にも不要な追加学習させることによる演算負荷の増大などの弊害を回避することができる。
In order to make the erroneous identification network learn that the identification was erroneous, along with the learning data (images), it is necessary to have type class information on how the target was erroneously identified. Therefore, by storing the type class information in the label in step S5, in step S6, using the image at the time of misidentification occurrence, it is possible to pinpoint additional machine learning for the problem identification network or the like specified by the label, and to improve the problem identification network or the like. In addition, because it is possible to exclude problem-free identification networks, etc., from the targets of additional learning, it is possible to avoid adverse effects such as the occurrence of erroneous learning of normal identification networks, etc., and the increase in computational load due to unnecessary additional learning of normal identification networks, etc.
<誤認識の一態様である、誤追跡の発生状況の一例>
図6Aは、車載カメラ装置100の類似箇所探索部7で、何らかの問題があるテンプレート画像や追跡ネットワーク等を利用して外部環境を認識した結果、誤追跡が発生した状況の一例を示す俯瞰図であり、図6Bは、図6Aの凡例である。なお、図示する俯瞰図は、例えば、撮像した左画像PLと右画像PRに対し、所定のアフィンテーブルを用いてアフィン変換した射影画像の経時変化と、自車両位置の経時変化を考慮して作成したものであるが、このような手法による自車両周囲の俯瞰図作成方法は周知技術であるため、その詳細については説明を省略する。 <An example of the occurrence of mistracking, which is a form of misrecognition>
FIG. 6A is a bird's-eye view showing an example of a situation in which mistracking occurs as a result of recognizing the external environment using a template image with some problem, a tracking network, or the like in the similar location search unit 7 of the vehicle-mountedcamera device 100, and FIG. 6B is a legend for FIG. 6A. The bird's-eye view shown in the figure is created, for example, by taking into account changes over time in projected images obtained by affine transforming the picked-up left image P L and right image PR using a predetermined affine table, and changes over time in the position of the vehicle. Since the method for creating the bird's-eye view of the surroundings of the vehicle using such a method is a well-known technique, the details thereof will not be described.
図6Aは、車載カメラ装置100の類似箇所探索部7で、何らかの問題があるテンプレート画像や追跡ネットワーク等を利用して外部環境を認識した結果、誤追跡が発生した状況の一例を示す俯瞰図であり、図6Bは、図6Aの凡例である。なお、図示する俯瞰図は、例えば、撮像した左画像PLと右画像PRに対し、所定のアフィンテーブルを用いてアフィン変換した射影画像の経時変化と、自車両位置の経時変化を考慮して作成したものであるが、このような手法による自車両周囲の俯瞰図作成方法は周知技術であるため、その詳細については説明を省略する。 <An example of the occurrence of mistracking, which is a form of misrecognition>
FIG. 6A is a bird's-eye view showing an example of a situation in which mistracking occurs as a result of recognizing the external environment using a template image with some problem, a tracking network, or the like in the similar location search unit 7 of the vehicle-mounted
図6A(a)~(c)はそれぞれ、後述する時刻T0より600ms、400ms、200ms前の時刻T-3、T-2、T-1における、自車両の外部環境を例示する俯瞰図である。これらの時点では、自車両は直線道路を等速走行しており、車載カメラ装置100は、右歩道を歩く歩行者の移動を、追跡ネットワーク等を利用して追跡している。時刻T-3~T-1の期間は、車両制御システム200は、自車両が歩行者と接触しないと判断しているため、これらの時点では、歩行者との接触を回避するための自動制御を実施していない。
FIGS. 6A (a) to (c) are overhead views illustrating the external environment of the host vehicle at times T −3 , T −2 , and T −1 600 ms, 400 ms, and 200 ms before time T 0 described later, respectively. At these times, the vehicle is traveling on a straight road at a constant speed, and the in-vehicle camera device 100 tracks the movement of the pedestrian walking on the right sidewalk using a tracking network or the like. Since the vehicle control system 200 determines that the own vehicle will not come into contact with the pedestrian during the period from time T -3 to T -1 , automatic control for avoiding contact with the pedestrian is not performed at these times.
図6A(d)は、自動制御(自動減速)を開始した時刻T0における、自車両の外部環境を例示する俯瞰図である。この時点で、車載カメラ装置100は、やや問題あるテンプレート画像や追跡ネットワーク等を利用して撮像画像を処理した結果、実際には歩道上にいる歩行者が、車道に飛び出して来たと誤追跡している。そのため、車載カメラ装置100の追跡結果を受信した車両制御システム200は、実在しない路上の歩行者との接触を防ぐため、自車両の制動系を制御し、自車両を急減速させる。なお、実在しない路上の歩行者を誤検知する状況は、例えば、歩行者の進路方向に存在する昇り旗や草木が突風等で揺らぎ、その揺らぎが路上に飛び出した歩行者と誤認されたような状況である。
FIG. 6A(d) is a bird's-eye view illustrating the external environment of the host vehicle at time T0 when automatic control (automatic deceleration) is started. At this point, the in-vehicle camera device 100 erroneously tracks the pedestrian who is actually on the sidewalk as having jumped out onto the roadway as a result of processing the picked-up image using a slightly problematic template image, tracking network, or the like. Therefore, the vehicle control system 200 that has received the tracking result of the in-vehicle camera device 100 controls the braking system of the own vehicle and decelerates the own vehicle rapidly in order to prevent contact with a non-existent pedestrian on the road. A situation in which a non-existing pedestrian on the road is erroneously detected is, for example, a situation in which a rising flag or plants existing in the direction of the pedestrian's path fluctuate due to a gust of wind, etc., and the fluctuation is misidentified as a pedestrian jumping out onto the road.
ここで、図6Bを用いて、車載カメラ装置100が検知した物標(歩行者)の位置の推移と、検知した物標(歩行者)との接触回避制御の開始判定方法を説明する。同図において、OB-3、OB-2、OB-1、OB0はそれぞれ、時刻T-3、T-2、T-1、T0に検知した物標(歩行者)の位置を示す。なお、位置OB0は誤検知されたものである。また、v-1、v0、v1はそれぞれ、時刻T-3~T-2、時刻T-2~T-1、時刻T-1~T0の期間の物標(歩行者)の移動ベクトルであり、車載カメラ装置100で算出された物標(歩行者)の速度ベクトルを各期間の移動量に換算したものである。EL、ERは自車進行範囲の左右端であり、自車両の舵角やヨーレートや速度に基づき、自車両が進行すると予測される範囲の左右端を示す。
Here, transition of the position of the target (pedestrian) detected by the in-vehicle camera device 100 and a method for determining the start of contact avoidance control with the detected target (pedestrian) will be described with reference to FIG. 6B. In the figure, OB -3 , OB -2 , OB -1 and OB 0 indicate the positions of targets (pedestrians) detected at times T -3 , T -2 , T -1 and T 0 respectively. Note that the position OB 0 is erroneously detected. Further, v −1 , v 0 , and v 1 are movement vectors of the target (pedestrian) during the periods of time T −3 to T −2 , time T −2 to T −1 , and time T −1 to T 0 , respectively, and are obtained by converting the velocity vector of the target (pedestrian) calculated by the in-vehicle camera device 100 into the amount of movement in each period. E L and E R are the left and right ends of the travel range of the own vehicle, and indicate the left and right ends of the range in which the own vehicle is predicted to travel based on the steering angle, yaw rate, and speed of the own vehicle.
本実施例の車両制御システム200は、移動する物標が自車進行範囲を含む所定の接触判定範囲に侵入した場合に、その物標との接触の可能性があると判断し、物標との接触を回避するための制動や操舵などの自動制御を発動する。従って、図6Bでは、接触判定範囲内の物標の位置OB0を検知した時刻T0の時点で、車両制御システム200は、例えば、自車両を急減速させる自動制御を発動する。なお、上記の自車進行範囲の長さは、自車両が所定期間内(例えば、5秒以内)に走行すると予測される範囲に限定され、仮に物標が存在しても直ちに接触回避制御を実施する必要がないような遠方は、その時点では自車進行範囲に含める必要が無い。また、上記の接触判定範囲の幅は、GNSS(Global Navigation Satellite System)等から知得した走行中道路の種別情報に応じて変化させても良く、例えば、側方からの物標(歩行者や他車両など)の接近の可能性が低い高速道路等では狭く、側方からの物標の接近の可能性が高い一般道路では広くしても良い。
The vehicle control system 200 of this embodiment determines that there is a possibility of contact with the target when a moving target enters a predetermined contact determination range including the travel range of the vehicle, and activates automatic control such as braking and steering to avoid contact with the target. Therefore, in FIG. 6B, at time T0 when the target position OB0 within the contact determination range is detected, the vehicle control system 200, for example, activates automatic control to rapidly decelerate the own vehicle. The length of the vehicle travel range is limited to a range in which the vehicle is expected to travel within a predetermined time period (for example, within 5 seconds), and even if a target exists, it is not necessary to include it in the travel range of the vehicle at that point in time where it is not necessary to immediately implement contact avoidance control. In addition, the width of the contact determination range may be changed according to the type information of the road being traveled obtained from GNSS (Global Navigation Satellite System) or the like. For example, it may be narrow on highways where the possibility of approaching targets (pedestrians, other vehicles, etc.) from the side is low, and may be widened on general roads where there is a high possibility of targets approaching from the side.
図6A(e)は、時刻T0から200msを経過した時刻T1における、自車両の外部環境を例示する俯瞰図である。この時点で、運転者は、自車両を制動する理由がないのに自動減速が開始されたことから、車載カメラ装置100で何らかの異常(誤追跡)が発生したことに気付く。そのため、前方の安全(前方車両の不在や、周囲歩行者の不在など)を確認できれば、運転者は、減速された速度を減速前の速度に復帰させるべくアクセルペダルを踏み、自車両を加速させる。その結果、自車両は、誤検知した歩行者の領域に接近するが、その領域には実際には歩行者が存在しないため(実際の歩行者は右歩道上にいるため)、自車両はその領域を無事に通過することができる。
FIG. 6A(e) is a bird's-eye view illustrating the external environment of the host vehicle at time T1 , 200 ms after time T0 . At this point, the driver notices that some abnormality (erroneous tracking) has occurred in the in-vehicle camera device 100 because automatic deceleration has started without any reason to brake the vehicle. Therefore, if the safety ahead (absence of a vehicle in front, absence of pedestrians, etc.) can be confirmed, the driver depresses the accelerator pedal in order to restore the decelerated speed to the pre-deceleration speed, thereby accelerating the own vehicle. As a result, the vehicle approaches the area of the erroneously detected pedestrian, but since there are no pedestrians in the area (the actual pedestrian is on the right sidewalk), the vehicle can safely pass through the area.
以上から明らかなように、車載カメラ装置100の追跡結果に基づいて車両制御システム200が自動制御を開始した直後に、運転者がその自動制御に反する手動運転操作を実施し、かつ、追跡した立体物への接触が発生しなかった場合は、車載カメラ装置100で誤追跡が発生したと判定することができる。
As is clear from the above, immediately after the vehicle control system 200 starts automatic control based on the tracking result of the vehicle-mounted camera device 100, if the driver performs a manual driving operation contrary to the automatic control and does not come into contact with the tracked three-dimensional object, it can be determined that the vehicle-mounted camera device 100 has mistracked.
<誤識別の発生前後の画像の保存方法と、保存画像を利用した追加学習方法>
図7は、図6Aで説明した機序を応用したフローチャートであり、車載カメラ装置100で誤追跡が発生した時刻の前後の画像を保存し、更に、保存した画像に基づいてテンプレート画像や追跡ネットワーク等を追加学習する方法を示すものである。以下、各ステップを順次説明する。なお、図5と図7の処理は並列して実施することができ、また、以下においては、図5のフローチャートとの共通点については、必要に応じて重複説明を省略する。 <Method of saving images before and after occurrence of misclassification, and method of additional learning using saved images>
FIG. 7 is a flowchart to which the mechanism described in FIG. 6A is applied, and shows a method for storing images before and after the time when mistracking occurred in the in-vehicle camera device 100, and additionally learning a template image, a tracking network, etc. based on the stored images. Each step will be described below. 5 and 7 can be performed in parallel, and redundant description of points common to the flowchart of FIG. 5 will be omitted as necessary.
図7は、図6Aで説明した機序を応用したフローチャートであり、車載カメラ装置100で誤追跡が発生した時刻の前後の画像を保存し、更に、保存した画像に基づいてテンプレート画像や追跡ネットワーク等を追加学習する方法を示すものである。以下、各ステップを順次説明する。なお、図5と図7の処理は並列して実施することができ、また、以下においては、図5のフローチャートとの共通点については、必要に応じて重複説明を省略する。 <Method of saving images before and after occurrence of misclassification, and method of additional learning using saved images>
FIG. 7 is a flowchart to which the mechanism described in FIG. 6A is applied, and shows a method for storing images before and after the time when mistracking occurred in the in-
まず、ステップS1では、誤制御判定部21は、制御フィードバック部23が入手した車両制御システム200からのデータに基づいて、衝突防止の自動制御が動作したか否かを判定する。
First, in step S1, the erroneous control determination unit 21 determines whether or not the automatic control for collision prevention has operated based on the data from the vehicle control system 200 that the control feedback unit 23 has obtained.
次に、ステップS2では、誤制御判定部21は、運転者操作入力部14が入手したステアリングセンサ34やアクセルペダル35などからのデータに基づいて、自動制御に反する手動の運転操作がなされたか否かを判定する。図6Aで説明したように、追跡中の物標との接触防止の自動制御が発動された後に、運転手が自動制御に反する手動操作を実施した場合、運転手は安全を確認したうえで、その操作を行っていると考えられるため、車両制御システム200による接触防止の自動制御は誤りであったと判定することができる。
Next, in step S2, the erroneous control determination unit 21 determines whether or not a manual driving operation contrary to automatic control has been performed, based on data from the steering sensor 34 and the accelerator pedal 35 obtained by the driver operation input unit 14. As described with reference to FIG. 6A , if the driver performs a manual operation contrary to the automatic control after the automatic control to prevent contact with the target being tracked has been activated, it is considered that the driver is performing the operation after confirming safety.
ステップS3では、誤制御判定部21は、車両速度センサ31、車両舵角センサ32、ヨーレートセンサ33の出力等に基づいて、自動制御以後の走行軌跡(自車両の位置の軌跡)を算出し、運転操作とともに記憶する。
In step S3, the erroneous control determination unit 21 calculates the travel trajectory after automatic control (the trajectory of the vehicle's position) based on the outputs of the vehicle speed sensor 31, the vehicle steering angle sensor 32, and the yaw rate sensor 33, and stores it together with the driving operation.
ステップS4aでは、誤追跡特定部20は、自動制御開始時点の追跡結果が正しかったと仮定した場合に、追跡した物標との接触を回避できる走行軌跡を手動運転操作で走行したかを判断することで、誤追跡の発生を特定する。
In step S4a, the erroneous tracking identification unit 20 identifies the occurrence of erroneous tracking by determining whether or not the vehicle has traveled manually along a trajectory that can avoid contact with the tracked target, assuming that the tracking result at the start of automatic control was correct.
ステップS5aでは、誤追跡特定部20は、誤追跡の発生前後の画像(例えば、自動制御開始時刻の前後1秒間の画像)を、画像記憶部6から読み取って誤認識画像保存部22に保存する。
In step S5a, the mistracking identification unit 20 reads images before and after the occurrence of mistracking (for example, images for one second before and after the automatic control start time) from the image storage unit 6 and stores them in the misrecognition image storage unit 22.
ステップS6aでは、追加学習部24は、ステップS5aで誤認識画像保存部22に保存された、誤追跡の発生前後の画像を学習データとして、テンプレート画像や追跡ネットワーク等の追加機械学習を実施する。これにより、追加機械学習の頻度を抑制して演算負荷を軽減しつつ、追加機械学習によりテンプレート画像や追跡ネットワーク等の品質向上を図ることができる。
In step S6a, the additional learning unit 24 performs additional machine learning of template images, tracking networks, etc., using the images before and after the occurrence of mistracking, which were stored in the misrecognized image storage unit 22 in step S5a, as learning data. As a result, it is possible to improve the quality of the template image, the tracking network, etc. by the additional machine learning while suppressing the frequency of the additional machine learning and reducing the computational load.
図7のステップS5aにおいても、図5のステップS5に倣い、誤追跡の態様を特定するラベルを、識別画像とともに保存することが望ましい。これにより、図5で説明したと同様の作用により、改善の必要なテンプレート画像や追跡ネットワーク等に対してのみ、ステップS6aでの追加学習を実行できるので、図5と同様の種々効果を得ることができる。
Also in step S5a of FIG. 7, it is desirable to follow step S5 of FIG. 5 and store a label identifying the mode of mistracking together with the identification image. As a result, the additional learning in step S6a can be executed only for template images, tracking networks, etc. that need to be improved by the same action as described in FIG. 5, so various effects similar to those in FIG. 5 can be obtained.
なお、ステップS4aで誤追跡発生と特定された箇所の近くには、テンプレートと同一の物体や、前記誤追跡箇所の前の画像に映る同一の物体が存在している可能性があるので、誤追跡箇所の前と後の画像は、追跡して検知枠をつけた領域を広げて画像を記憶することで、前記同一の物体の写る箇所も含めた画像を記憶でき、追加学習の際に正しく追跡すべき箇所を適切に追加学習することができる。
It should be noted that there is a possibility that the same object as the template or the same object that appears in the image before the mistracked portion exists near the location identified as the occurrence of mistracking in step S4a. Therefore, by storing the images before and after the mistracked portion by enlarging the area to which the detection frame is attached by tracking, an image including the location where the same object appears can be stored, and the location that should be tracked correctly can be appropriately additionally learned during additional learning.
図8は時刻T0で識別され自車の制御が発動された物標について、歩行者が移動することも加味した、誤識別発生時の自車両の挙動と誤制御・誤識別と判定する方法を説明する俯瞰図である。図8(c)の歩行者予測範囲は、図8(b)で歩行者を識別してからの経過時間に、歩行者の移動速度を掛け合わせ、歩行者の存在しうる予測範囲を示すものである。前記歩行者の移動速度は、歩行者らしい速度を予め設定した速度で、平均的な歩行速度を想定すればよく、例えば時速5kmと設定すると良い。図8の凡例の歩行者存在半径は、歩行者予測範囲の半径を示すもので、歩行者を識別してからの経過時間Tと前記歩行者の移動速度VPを掛け合わせた長さLである。
FIG. 8 is a bird's-eye view for explaining the behavior of the own vehicle at the time of misidentification and the method of determining miscontrol/misidentification, taking into account the movement of pedestrians, with respect to the target identified at time T0 and the control of the subject vehicle being activated. The pedestrian prediction range shown in FIG. 8(c) is obtained by multiplying the elapsed time after the pedestrian is identified in FIG. The moving speed of the pedestrian is a speed that is preset as a pedestrian-like speed, and may be assumed to be an average walking speed, for example, set to 5 km/h. The pedestrian existence radius in the legend of FIG. 8 indicates the radius of the pedestrian prediction range, and is the length L obtained by multiplying the elapsed time T after the pedestrian is identified and the movement speed VP of the pedestrian.
図8の例では、時刻T0で識別された物標は時刻T3で、歩行者存在予測範囲を自車が覆っているので、時刻T0で識別された物標は誤識別であったと判断される。本発明では自動制御がなされた際の衝突判定された各時刻の検知・識別物標に対して、前記歩行者予測範囲と自車範囲の重複度により誤識別した時刻の物標を特定する。
In the example of FIG. 8, the target identified at time T0 is at time T3 and the pedestrian presence prediction range is covered by the own vehicle, so it is determined that the target identified at time T0 was erroneously identified. In the present invention, a target at a time when an erroneous identification is made is specified based on the overlapping degree of the pedestrian prediction range and the own vehicle range with respect to the detection/identification target at each time when the collision is determined when the automatic control is performed.
図9は歩行者が移動することも加味した、誤識別の前後の画像の保存処理を説明するフローチャートである。なお、図5との共通点は重複説明を省略している。図9のステップS7は、ステップS3の次に実施される処理であり、自動制御以後の歩行者の予測範囲を推定する処理である。前記歩行者存在半径を、歩行者識別してからの経過時間Tと前記歩行者の移動速度VPから求めて、前記歩行者存在範囲を特定する。
FIG. 9 is a flow chart for explaining a process of storing images before and after misidentification, taking into account the movement of pedestrians. Note that redundant description of the points in common with FIG. 5 is omitted. Step S7 in FIG. 9 is a process performed after step S3, and is a process of estimating the predicted range of the pedestrian after automatic control. The pedestrian presence radius is obtained from the elapsed time T after the pedestrian is identified and the pedestrian movement speed VP to specify the pedestrian presence range.
その後のステップS4bは、識別物標を避ける手動操作か判断する処理で、ステップS3で検証した走行軌跡と、ステップS7で推定した歩行者予測範囲から、識別物標を避ける手動操作がなされたか判定する。自車の走行軌跡で車両が歩行者予測範囲を覆った場合、その歩行者の識別物標を避ける手動操作はなされなかったと判断する。
The subsequent step S4b is a process for determining whether the manual operation to avoid the identification target is performed, and it is determined whether the manual operation to avoid the identification target has been performed based on the travel trajectory verified in step S3 and the pedestrian prediction range estimated in step S7. When the vehicle covers the pedestrian prediction range on the travel path of the own vehicle, it is determined that the manual operation to avoid the identification target of the pedestrian was not performed.
このように、歩行者の移動を加味することで、誤識別の判定を歩行者移動により誤るリスクを低減し、誤識別箇所の追加学習の効果を高めることができる。
In this way, by taking into account the movement of pedestrians, it is possible to reduce the risk of misidentification errors due to pedestrian movement and increase the effect of additional learning of misidentified locations.
<本実施例の効果>
以上で説明した本実施例によれば、実走行環境下で撮像された様々な画像のなかから、問題のあるAIネットワークの改善に寄与する、外部環境の誤認識の発生前後の画像を自動的に抽出して保存することができる。これにより、本実施例の車載カメラ装置は、外部と連携することなく、自動的に抽出して保存した画像を学習データとすることで、問題のあるAIネットワークに追加学習を実行させ、そのAIネットワークの品質を改善することができる。 <Effect of this embodiment>
According to the present embodiment described above, it is possible to automatically extract and save images before and after the occurrence of misrecognition of the external environment, which contributes to the improvement of problematic AI networks, from among various images captured in an actual driving environment. As a result, the in-vehicle camera device of the present embodiment uses the automatically extracted and saved images as learning data without cooperating with the outside, so that the problematic AI network can be made to perform additional learning, and the quality of the AI network can be improved.
以上で説明した本実施例によれば、実走行環境下で撮像された様々な画像のなかから、問題のあるAIネットワークの改善に寄与する、外部環境の誤認識の発生前後の画像を自動的に抽出して保存することができる。これにより、本実施例の車載カメラ装置は、外部と連携することなく、自動的に抽出して保存した画像を学習データとすることで、問題のあるAIネットワークに追加学習を実行させ、そのAIネットワークの品質を改善することができる。 <Effect of this embodiment>
According to the present embodiment described above, it is possible to automatically extract and save images before and after the occurrence of misrecognition of the external environment, which contributes to the improvement of problematic AI networks, from among various images captured in an actual driving environment. As a result, the in-vehicle camera device of the present embodiment uses the automatically extracted and saved images as learning data without cooperating with the outside, so that the problematic AI network can be made to perform additional learning, and the quality of the AI network can be improved.
なお、本発明は上記した実施例に限定されるものではなく、様々な変形例が含まれる。例えば、上記した実施例は本発明を分かりやすく説明するために詳細に説明したものであり、必ずしも説明した全ての構成を備えるものに限定されるものではない。また、ある実施例の構成の一部を他の実施例の構成に置き換えることが可能であり、また、ある実施例の構成に他の実施例の構成を加えることも可能である。また、各実施例の構成の一部について、他の構成の追加・削除・置換をすることが可能である。
It should be noted that the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations. In addition, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Moreover, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.
また、上記の各構成、機能、処理部、処理手段等は、それらの一部又は全部を、例えば集積回路で設計する等によりハードウェアで実現してもよい。また、上記の各構成、機能等は、プロセッサがそれぞれの機能を実現するプログラムを解釈し、実行することによりソフトウェアで実現してもよい。各機能を実現するプログラム、テーブル、ファイル等の情報は、メモリや、ハードディスク、SSD(Solid State Drive)等の記録装置、または、半導体メモリカード等の記録媒体に置くことができる。
In addition, each of the above configurations, functions, processing units, processing means, etc. may be realized in hardware, for example, by designing a part or all of them with an integrated circuit. Moreover, each of the above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program for realizing each function. Information such as programs, tables, and files that implement each function can be stored in a recording device such as a memory, a hard disk, an SSD (Solid State Drive), or a recording medium such as a semiconductor memory card.
また、制御線や情報線は説明上必要と考えられるものを示しており、製品上必ずしも全ての制御線や情報線を示しているとは限らない。実際には殆ど全ての構成が相互に接続されていると考えてもよい。
In addition, the control lines and information lines indicate what is considered necessary for explanation, and not all control lines and information lines are necessarily indicated on the product. In practice, it may be considered that almost all configurations are interconnected.
上述した各実施の形態および変形例において、機能ブロックの構成は一例に過ぎない。別々の機能ブロックとして示したいくつかの機能構成を一体に構成してもよいし、1つの機能ブロック図で表した構成を2以上の機能に分割してもよい。また各機能ブロックが有する機能の一部を他の機能ブロックが備える構成としてもよい。
In each embodiment and modification described above, the configuration of the functional blocks is merely an example. Some functional configurations shown as separate functional blocks may be configured integrally, or a configuration represented by one functional block diagram may be divided into two or more functions. Further, a configuration may be adopted in which part of the functions of each functional block is provided in another functional block.
上述した各実施の形態および変形例は、それぞれ組み合わせてもよい。上記では、種々の実施の形態および変形例を説明したが、本発明はこれらの内容に限定されるものではない。本発明の技術的思想の範囲内で考えられるその他の態様も本発明の範囲内に含まれる。
Each of the above-described embodiments and modifications may be combined. Although various embodiments and modifications have been described above, the present invention is not limited to these contents. Other aspects conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention.
100…車載カメラ装置、1L…左撮像部、1R…右撮像部、2…ステレオマッチング部、3…単眼検知部、4…単眼測距部、5…テンプレート作成部、6…画像記憶部、7…類似箇所探索部、8…画角特定部、9…ステレオ検知部、10…速度算出部、11…車両情報入力部、12…ステレオ測距部、13…種別識別部、14…運転者操作入力部、15…識別履歴記憶部、16…追跡履歴記憶部、17…位置履歴記憶部、18…速度履歴記憶部、19…誤識別特定部、20…誤追跡特定部、21…誤制御特定部、22…誤認識画像保存部、23…制御フィードバック部、24…追加学習部、200…車両制御システム
DESCRIPTION OF SYMBOLS 100... Vehicle-mounted camera apparatus, 1L... Left imaging part, 1R... Right imaging part, 2... Stereo matching part, 3... Monocular detection part, 4... Monocular distance measurement part, 5... Template preparation part, 6... Image storage part, 7... Similar part search part, 8... Angle-of-view specification part, 9... Stereo detection part, 10... Speed calculation part, 11... Vehicle information input part, 12... Stereo distance measurement part, 13... Type identification part, 14... Driver operation input part, 15... Identification history storage part 16 Tracking history storage unit 17 Position history storage unit 18 Speed history storage unit 19 Incorrect identification identification unit 20 Incorrect tracking identification unit 21 Incorrect control identification unit 22 Incorrect recognition image storage unit 23 Control feedback unit 24 Additional learning unit 200 Vehicle control system
Claims (11)
- 物体を識別する車載カメラ装置であって、
自車両の自動制御のフィードバックを受ける制御フィードバック部と、
運転者の運転操作情報に基づき誤った自動制御を判定する誤制御判定部と、
画像を保存する画像保存部を備え、
誤った自動制御と判定された際に画像を保存することを特徴とする車載カメラ装置。 An in-vehicle camera device for identifying an object,
a control feedback unit that receives feedback of automatic control of the own vehicle;
an erroneous control determination unit that determines erroneous automatic control based on the driver's driving operation information;
Equipped with an image storage unit for storing images,
An in-vehicle camera device that saves an image when it is determined that an erroneous automatic control has occurred. - 請求項1に記載の車載カメラ装置において、
さらに、前記誤った自動制御が物体の種別の識別を誤った誤識別に起因することを特定する誤識別特定部を備え、
誤った自動制御後に自車両が走行する領域に位置する物体の画像を保存することを特徴とする車載カメラ装置。 The in-vehicle camera device according to claim 1,
Furthermore, an erroneous identification identification unit that identifies that the erroneous automatic control is caused by erroneous identification of the type of object,
An in-vehicle camera device, characterized in that it saves an image of an object located in an area where the own vehicle travels after an erroneous automatic control. - 請求項2に記載の車載カメラ装置において、
前記画像に誤識別を示すラベルを付けて保存することを特徴とする車載カメラ装置。 In the in-vehicle camera device according to claim 2,
An in-vehicle camera device, wherein the image is stored with a label indicating erroneous identification. - 請求項2に記載の車載カメラ装置において、
識別ネットワークが識別した種別を誤識別画像とともに記憶する車載カメラ装置。 In the in-vehicle camera device according to claim 2,
An in-vehicle camera device that stores a type identified by an identification network together with an incorrectly identified image. - 請求項2に記載の車載カメラ装置において、
歩行者予測範囲が自車進行軌跡の車両に覆われた識別物標を誤識別画像として記憶する車載カメラ装置。 In the in-vehicle camera device according to claim 2,
An in-vehicle camera device that stores, as an erroneously identified image, an identification target whose pedestrian prediction range is covered by a vehicle on the course of the own vehicle. - 請求項1に記載の車載カメラ装置において、
さらに、前記誤った自動制御が物体の追跡を誤った誤追跡に起因することを特定する誤追跡特定部を備え、
誤った自動制御後に自車両が走行する領域に位置する物体の画像を保存することを特徴とする車載カメラ装置。 The in-vehicle camera device according to claim 1,
Further comprising an erroneous tracking identification unit that identifies that the erroneous automatic control is caused by erroneous erroneous tracking of the object,
An in-vehicle camera device, characterized in that it saves an image of an object located in an area where the own vehicle travels after an erroneous automatic control. - 請求項6に記載の車載カメラ装置において、
前記画像に誤追跡を示すラベルを付けて保存することを特徴とする車載カメラ装置。 In the in-vehicle camera device according to claim 6,
An in-vehicle camera device, wherein the image is stored with a label indicating erroneous tracking. - 請求項6に記載の車載カメラ装置において、
さらに、物体を追跡する際に利用するテンプレート画像を作成するテンプレート作成部を備え、
誤追跡と判定した物体軌跡の画像と、該物体軌跡の追跡に用いたテンプレート画像の双方を保存することを特徴とする車載カメラ装置。 In the in-vehicle camera device according to claim 6,
Furthermore, it has a template creation unit that creates a template image used when tracking an object,
An in-vehicle camera device characterized by storing both an image of an object trajectory determined to be erroneously tracked and a template image used for tracking the object trajectory. - 請求項1に記載の車載カメラ装置において、
自車両の外部環境を認識するネットワークを複数有しており、
前記画像保存部に保存した画像には、何れのネットワークの機械学習で用いる画像であるかを示すラベルが付されていることを特徴とする車載カメラ装置。 The in-vehicle camera device according to claim 1,
It has multiple networks that recognize the external environment of the vehicle,
An in-vehicle camera device, wherein a label indicating which network machine learning the image is used for is attached to the image stored in the image storage unit. - 請求項1に記載の車載カメラ装置と、
前記自車両のブレーキを制御する車両制御システムと、
を有する車載カメラシステムであって、
前記車両制御システムは、前記車載カメラシステムの出力に基づいてブレーキを制御することを特徴とする車載カメラシステム。 an in-vehicle camera device according to claim 1;
a vehicle control system that controls the brakes of the host vehicle;
An in-vehicle camera system having
An in-vehicle camera system, wherein the vehicle control system controls a brake based on an output of the in-vehicle camera system. - 物体を識別するネットワークの学習に利用する画像を保存する画像保存方法であって、
自車両の自動制御のフィードバックを受ける制御フィードバック処理と、
運転者の運転操作情報に基づき誤った自動制御を判定する誤制御判定処理と、
誤った自動制御と判定された際に画像を保存する画像保存処理を備えることを特徴とする画像保存方法。 An image storage method for storing an image used for learning a network for identifying an object,
a control feedback process for receiving feedback of automatic control of the own vehicle;
erroneous control determination processing for determining erroneous automatic control based on driver's driving operation information;
An image saving method, comprising image saving processing for saving an image when it is determined that an erroneous automatic control is performed.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2022-007132 | 2022-01-20 | ||
JP2022007132A JP2023106029A (en) | 2022-01-20 | 2022-01-20 | On-vehicle camera apparatus, on-vehicle camera system, and image storing method |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2023139978A1 true WO2023139978A1 (en) | 2023-07-27 |
Family
ID=87348154
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2022/045873 WO2023139978A1 (en) | 2022-01-20 | 2022-12-13 | Vehicle-mounted camera device, vehicle-mounted camera system, and image storage method |
Country Status (2)
Country | Link |
---|---|
JP (1) | JP2023106029A (en) |
WO (1) | WO2023139978A1 (en) |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2016136332A (en) * | 2015-01-23 | 2016-07-28 | 沖電気工業株式会社 | Information processing apparatus, information processing method, and storage medium |
JP2016210285A (en) * | 2015-05-08 | 2016-12-15 | トヨタ自動車株式会社 | Erroneous recognition determination device |
JP2017138282A (en) * | 2016-02-05 | 2017-08-10 | トヨタ自動車株式会社 | Automatic operation system |
JP2019096137A (en) * | 2017-11-24 | 2019-06-20 | トヨタ自動車株式会社 | Signal apparatus recognition device |
JP2019159659A (en) * | 2018-03-12 | 2019-09-19 | パナソニックIpマネジメント株式会社 | Information processing device |
-
2022
- 2022-01-20 JP JP2022007132A patent/JP2023106029A/en active Pending
- 2022-12-13 WO PCT/JP2022/045873 patent/WO2023139978A1/en unknown
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2016136332A (en) * | 2015-01-23 | 2016-07-28 | 沖電気工業株式会社 | Information processing apparatus, information processing method, and storage medium |
JP2016210285A (en) * | 2015-05-08 | 2016-12-15 | トヨタ自動車株式会社 | Erroneous recognition determination device |
JP2017138282A (en) * | 2016-02-05 | 2017-08-10 | トヨタ自動車株式会社 | Automatic operation system |
JP2019096137A (en) * | 2017-11-24 | 2019-06-20 | トヨタ自動車株式会社 | Signal apparatus recognition device |
JP2019159659A (en) * | 2018-03-12 | 2019-09-19 | パナソニックIpマネジメント株式会社 | Information processing device |
Also Published As
Publication number | Publication date |
---|---|
JP2023106029A (en) | 2023-08-01 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Kato et al. | An obstacle detection method by fusion of radar and motion stereo | |
EP3366540B1 (en) | Information processing apparatus and non-transitory computer-readable recording medium | |
JP4755227B2 (en) | Method for recognizing objects | |
WO2017171082A1 (en) | Vehicle control device and vehicle control method | |
US11370420B2 (en) | Vehicle control device, vehicle control method, and storage medium | |
US20190276049A1 (en) | Driving assistance system | |
US12024161B2 (en) | Vehicular control system | |
Zhang et al. | A novel vehicle reversing speed control based on obstacle detection and sparse representation | |
KR20230031344A (en) | System and Method for Detecting Obstacles in Area Surrounding Vehicle | |
JP2012106735A (en) | Fork road intrusion determination device | |
CN109195849B (en) | Image pickup apparatus | |
US11715277B2 (en) | Perception system for autonomous vehicles | |
US20220234581A1 (en) | Vehicle control method, vehicle control device, and vehicle control system including same | |
JP2006004188A (en) | Obstacle recognition method and obstacle recognition device | |
WO2023139978A1 (en) | Vehicle-mounted camera device, vehicle-mounted camera system, and image storage method | |
US11893715B2 (en) | Control device and control method for mobile object, storage medium, and vehicle | |
CN116242375A (en) | High-precision electronic map generation method and system based on multiple sensors | |
CN115959109A (en) | Vehicle control device, vehicle control method, and storage medium | |
JP2003121543A (en) | Driving lane determining apparatus for vehicle | |
JP2023110364A (en) | Object tracking device, object tracking method, and program | |
Chanawangsa et al. | A novel video analysis approach for overtaking vehicle detection | |
Wang et al. | A monovision-based 3D pose estimation system for vehicle behavior prediction | |
US11938879B2 (en) | Vehicle control device, information processing apparatus, operation methods thereof, and storage medium | |
JP2019172032A (en) | Automatic braking device | |
JP7149171B2 (en) | Object recognition method and object recognition device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 22922129 Country of ref document: EP Kind code of ref document: A1 |