WO2022204867A1 - 一种车道线检测方法及装置 - Google Patents

一种车道线检测方法及装置 Download PDF

Info

Publication number
WO2022204867A1
WO2022204867A1 PCT/CN2021/083552 CN2021083552W WO2022204867A1 WO 2022204867 A1 WO2022204867 A1 WO 2022204867A1 CN 2021083552 W CN2021083552 W CN 2021083552W WO 2022204867 A1 WO2022204867 A1 WO 2022204867A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
lane lines
lane
loss function
feature
Prior art date
Application number
PCT/CN2021/083552
Other languages
English (en)
French (fr)
Inventor
晋周南
李亚敏
岳梁伟
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to CN202180095662.XA priority Critical patent/CN116997940A/zh
Priority to PCT/CN2021/083552 priority patent/WO2022204867A1/zh
Publication of WO2022204867A1 publication Critical patent/WO2022204867A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition

Definitions

  • the embodiments of the present application relate to the field of intelligent driving or automatic driving, and in particular, to a lane line detection method and device.
  • Lane line detection is an important task of advanced driver assistance systems (ADAS) and the key to realizing technologies such as adaptive cruise control (ACC) and lane departure warning system (LDWS). link.
  • Lane line detection is a complex and challenging topic in research on intelligent or unmanned vehicles. As a main part of the road, the lane line plays a role of providing reference for unmanned vehicles and guiding safe driving. At the same time, the lane line detection can also be used to realize road positioning, determine the relative position relationship between the vehicle and the road, and assist the decision-making planning of the vehicle.
  • Embodiments of the present application provide a lane line detection method and device, so as to improve the lane line detection effect in a complex scene.
  • an embodiment of the present application provides a method for detecting lane lines.
  • the method includes: acquiring a first image, where the first image is an environment image, and acquiring a first set of lane lines and a second lane according to the first image A line set, and a third set of lane lines is obtained according to the first set of lane lines and the second set of lane lines.
  • the first lane line set and the second lane line set are obtained in different ways.
  • a second image when acquiring the first set of lane lines and the second set of lane lines according to the first image, a second image may be first determined according to the first image, and the second image is used for The second set of lane lines is determined.
  • a first loss value is determined according to the first image and the second image, and the first loss value indicates the similarity of the first image and the second image.
  • a first model may be used to determine the second image according to the first image; the first model is trained according to the original image and the template image obtained, wherein the template image is associated with the original image, the lane line detection difficulty of the template image is lower than the lane line detection difficulty of the original image; the first model makes the first loss function satisfy the first As a preset condition, the first loss function is used to indicate a difference between the predicted template image determined based on the original image and the template image.
  • the first model for generating the predicted template image can be obtained by training the original image and the template image, and the first model makes the first loss function satisfy corresponding preset conditions.
  • a second model may be used to determine the first lane line detection set according to the first image, and the second model is Obtained by training based on the original image and the template image, wherein the template image is associated with the original image, and the lane line detection difficulty of the template image is lower than the lane line detection difficulty of the original image; wherein the second The model makes at least two loss functions among the second loss function, the third loss function and the fourth loss function satisfy corresponding preset conditions; wherein the second loss function is used to indicate the predicted lane determined based on the original image The difference between the line and the real lane line corresponding to the original image, the third loss function is used to indicate the difference between the first feature and the second feature, and the fourth loss function is used to indicate whether the first feature is is identified as the second feature, the first feature is a feature obtained based on the original image, the second feature is a feature obtained based on the template image, the first feature and the second feature features of the same type
  • a second model for generating predicted lane lines can be obtained by training the original image and the template image, and the second model satisfies at least two loss functions among the second loss function, the third loss function and the fourth loss function. the corresponding preset conditions.
  • the third loss function and the fourth loss function are used to constrain the features obtained based on the original image and the features obtained based on the template image to have better consistency during the training process of the second model, thereby improving the performance of the second model. predict the effect.
  • a third model may be used to determine the second image and the first lane according to the first image line set, the third model is obtained by training the original image and the template image, wherein the template image is associated with the original image, and the lane line detection difficulty of the template image is lower than that of the original image.
  • the third model makes a first loss function satisfy a first preset condition, and makes a second loss function satisfy a second preset condition, and the first loss function is used to indicate the determination based on the original image
  • the second loss function is used to indicate the difference between the predicted lane line determined based on the original image and the real lane line corresponding to the original image.
  • the third model for generating the predicted lane line and the predicted template image can be obtained by training the original image and the template image, and the third model makes the first loss function and the second loss function satisfy corresponding preset conditions respectively.
  • the third model makes the first loss function satisfy the first preset condition, makes the second loss function satisfy the second preset condition, and the third model makes the third loss function satisfy the third loss function Three preset conditions and/or making the fourth loss function satisfy the fourth preset condition, wherein the third loss function is used to indicate the difference between the first feature and the second feature, and the fourth loss function is used to indicate Whether the first feature is identified as the second feature, the first feature is a feature obtained based on the original image, the second feature is a feature obtained based on the template image, the first feature Features of the same type as the second feature.
  • the third loss function and the fourth loss function are used to constrain the features obtained based on the original image and the features obtained based on the template image to have better consistency during the training process of the second model, thereby improving the prediction effect of the second model. .
  • the first loss value is obtained by using the third loss function according to the first image and the second image; or, the first loss value is obtained according to the third loss function.
  • An image and the second image are obtained by using the fourth loss function; or, the first loss value is determined according to a second loss value and a third loss value, and the second loss value is determined according to the The first image and the second image are obtained by using the third loss function, and the third loss value is obtained by using the fourth loss function according to the first image and the second image.
  • the first loss value may characterize the similarity between the first image and the second image.
  • the third set of lane lines includes at least one of the same set of lane lines in the first set of lane lines and the second set of lane lines.
  • the first lane line in the first lane line set and the second lane line in the second lane line set can be determined Indicate the same lane line in the environment, and then use the first lane line, or the second lane line, or the third lane line as the lane line in the third lane line set.
  • the third lane line is determined according to the first lane line and the second lane line.
  • the method further includes: when it is determined that the number of lane lines included in the third set of lane lines is less than the preset number of lane lines, combining the first set of lane lines and/or the second set of lane lines The lane lines other than the lane lines already included in the third lane line set are used as the lane lines in the third lane line set.
  • the matching of the first lane may be stopped.
  • the method further includes: when it is determined that the number of lane lines included in the third set of lane lines is less than the preset number of lane lines, combining the first set of lane lines and/or the second set of lane lines.
  • the lane lines other than the lane lines already included in the third lane line set are used as the lane lines in the third lane line set, so that the number of lane lines included in the third lane line set is equal to the preset Number of lane lines. Therefore, the number of lane lines included in the third set of lane lines may be equal to the preset number of lane lines according to actual needs.
  • an embodiment of the present application provides a lane line detection device, the device includes: an acquisition module for acquiring a first image, where the first image is an environment image; and a processing module for acquiring a first image according to the first image Obtain a first set of lane lines and a second set of lane lines, wherein the first set of lane lines and the second set of lane lines are obtained in different ways; according to the first set of lane lines and the second set of lane lines Collection Gets the third lane line collection.
  • the processing module is configured to: when acquiring the first set of lane lines and the second set of lane lines according to the first image, determine a second image according to the first image, and the first set of lane lines Two images are used to determine the second set of lane lines.
  • the processing module is configured to: determine a first loss value according to the first image and the second image, the first loss value indicating the first image and the second image similarity of images.
  • the processing module is configured to: when the second image is determined according to the first image, use a first model to determine the second image according to the first image; the first model is based on The original image and the template image are obtained through training, wherein the template image is associated with the original image, and the lane line detection difficulty of the template image is lower than the lane line detection difficulty of the original image; the first model makes the A loss function satisfies a first preset condition, and the first loss function is used to indicate a difference between the predicted template image determined based on the original image and the template image.
  • the processing module is configured to: when determining the first lane line set according to the first image, use a second model to determine the first lane line detection set according to the first image,
  • the second model is obtained by training an original image and a template image, wherein the template image is associated with the original image, and the lane line detection difficulty of the template image is lower than the lane line detection difficulty of the original image;
  • the second model makes at least two loss functions among the second loss function, the third loss function and the fourth loss function satisfy corresponding preset conditions; wherein, the second loss function is used to indicate that based on the The difference between the predicted lane line determined by the original image and the real lane line corresponding to the original image, the third loss function is used to indicate the difference between the first feature and the second feature, and the fourth loss function is used to indicate Whether the first feature is identified as the second feature, the first feature is a feature obtained based on the original image, the second feature is a feature obtained based on the template image, the first feature Features of the same type
  • the processing module is configured to: when determining the second image and the first lane line set according to the first image, use a third model to determine the second image according to the first image and the first set of lane lines, the third model is obtained by training based on the original image and the template image, wherein the template image is associated with the original image, and the lane line detection difficulty of the template image is lower than that of the template image.
  • the lane line detection difficulty of the original image wherein, the third model makes the first loss function meet the first preset condition, and makes the second loss function meet the second preset condition, and the first loss function is used to indicate The difference between the predicted template image determined based on the original image and the template image, the second loss function is used to indicate the difference between the predicted lane line determined based on the original image and the real lane line corresponding to the original image value.
  • the third model makes the first loss function satisfy the first preset condition, makes the second loss function satisfy the second preset condition, and the third model makes the third loss function satisfy the third loss function Three preset conditions and/or making the fourth loss function satisfy the fourth preset condition, wherein the third loss function is used to indicate the difference between the first feature and the second feature, and the fourth loss function is used to indicate Whether the first feature is identified as the second feature, the first feature is a feature obtained based on the original image, the second feature is a feature obtained based on the template image, the first feature Features of the same type as the second feature.
  • the first loss value is obtained by using the third loss function according to the first image and the second image; or, the first loss value is obtained according to the third loss function.
  • An image and the second image are obtained by using the fourth loss function; or, the first loss value is determined according to a second loss value and a third loss value, and the second loss value is determined according to the The first image and the second image are obtained by using the third loss function, and the third loss value is obtained by using the fourth loss function according to the first image and the second image.
  • the third set of lane lines includes at least one of the same set of lane lines in the first set of lane lines and the second set of lane lines.
  • the processing module is configured to: when it is determined that the number of lane lines included in the third set of lane lines is less than the preset number of lane lines, the first set of lane lines and/or the third set of lane lines The lane lines in the second-lane line set except the lane lines already included in the third lane line set are used as the lane lines in the third lane line set.
  • the processing module is configured to: when it is determined that the number of lane lines included in the third set of lane lines is less than the preset number of lane lines, the first set of lane lines and/or the third set of lane lines The lane lines in the second lane line set except the lane lines already included in the third lane line set are used as the lane lines in the third lane line set, so that the number of lane lines included in the third lane line set is equal to the preset number of lane lines.
  • the present application further provides an apparatus.
  • the device can perform the above-mentioned method design.
  • the apparatus may be a chip or circuit capable of performing the functions corresponding to the above method, or a device including the chip or circuit.
  • the apparatus includes: a memory for storing computer-executable program code; and a processor coupled to the memory.
  • the program code stored in the memory includes instructions, and when the processor executes the instructions, the apparatus or the device installed with the apparatus is made to execute the method in the first aspect or any possible design of the first aspect.
  • the device may further include a communication interface, which may be a transceiver, or if the device is a chip or a circuit, the communication interface may be an input/output interface of the chip, such as input/output pins.
  • a communication interface which may be a transceiver, or if the device is a chip or a circuit, the communication interface may be an input/output interface of the chip, such as input/output pins.
  • the device includes corresponding functional units, which are respectively used to implement the steps in the above method.
  • the functions can be implemented by hardware, or by executing corresponding software by hardware.
  • the hardware or software includes one or more units corresponding to the above-mentioned functions.
  • embodiments of the present application provide a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program runs on an apparatus, the first aspect or the first aspect is executed. any of the possible design methods.
  • an embodiment of the present application provides a computer program product, the computer program product includes a computer program, when the computer program runs on an apparatus, executes the first aspect or any one of the possibilities of the first aspect. method in design.
  • FIG. 1 is a schematic diagram of a lane line detection scene provided by an embodiment of the present application
  • FIG. 2 is a schematic diagram of a vehicle system provided by an embodiment of the present application.
  • FIG. 3 is an overview flowchart of a lane line detection method provided by an embodiment of the present application.
  • FIG. 4 is a schematic diagram of a first image and a second image provided by an embodiment of the present application.
  • FIG. 5 is a schematic diagram of processing the first feature and the second feature provided by an embodiment of the present application.
  • FIG. 6 is a schematic diagram of a first lane line set and a second lane line set provided by an embodiment of the present application
  • FIG. 7 is one of the schematic structural diagrams of the device provided by the embodiment of the present application.
  • FIG. 8 is the second schematic diagram of the device structure provided by the embodiment of the present application.
  • the embodiments of the present application may be applied to a vehicle system, for example, the vehicle 100 in FIG. 2 .
  • the vehicle 100 may be configured in a fully or partially autonomous driving mode.
  • the vehicle 100 may control itself while in an autonomous driving mode, and may determine the current state of the vehicle and its surrounding environment through human manipulation, determine the possible behavior of at least one other vehicle in the surrounding environment, and determine that the other vehicle performs A confidence level corresponding to the likelihood of the possible behavior, and the vehicle 100 is controlled based on the determined information.
  • the vehicle 100 may be placed to operate without human interaction.
  • components coupled to or included in vehicle 100 may include propulsion system 110 , sensor system 120 , control system 130 , peripherals 140 , power supply 150 , computer system 160 , and user interface 170 .
  • the components of the vehicle 100 may be configured to operate in interconnection with each other and/or with other components coupled to the various systems.
  • power supply 150 may provide power to all components of vehicle 100 .
  • Computer system 160 may be configured to receive data from and control propulsion system 110 , sensor system 120 , control system 130 , and peripherals 140 .
  • Computer system 160 may also be configured to generate a display of images on user interface 170 and receive input from user interface 170 .
  • vehicle 100 may include more, fewer, or different systems, and each system may include more, fewer, or different components.
  • the illustrated systems and components may be combined or divided in any manner, which is not specifically limited in this application.
  • Propulsion system 110 may provide powered motion for vehicle 100 .
  • propulsion system 110 may include engine/engine 114 , energy source 113 , transmission 112 , and wheels/tires 111 . Additionally, propulsion system 110 may additionally or alternatively include other components than those shown in FIG. 2 . This application does not specifically limit this.
  • the sensor system 120 may include several sensors for sensing information about the environment in which the vehicle 100 is located. As shown in FIG. 2 , the sensors of the sensor system 120 include a global positioning system (Global Positioning System, GPS) 126, an inertial measurement unit (Inertial Measurement Unit, IMU) 125, a lidar sensor, a camera sensor 123, a millimeter wave radar sensor, and a sensor for Actuator 121 that modifies the position and/or orientation of the sensor.
  • the millimeter wave radar sensor may utilize radio signals to sense objects within the surrounding environment of the vehicle 100 . In some embodiments, in addition to sensing the target, the millimeter wave radar 124 may be used to sense the speed and/or heading of the target.
  • the lidar 122 may utilize laser light to sense objects in the environment in which the vehicle 100 is located.
  • lidar 122 may include one or more laser sources, laser scanners, and one or more detectors, among other system components.
  • the camera sensor 123 may be used to capture multiple images of the surrounding environment of the vehicle 100 .
  • the camera sensor 123 may be a still camera or a video camera.
  • GPS 126 may be any sensor used to estimate the geographic location of vehicle 100 .
  • the GPS 126 may include a transceiver that estimates the position of the vehicle 100 relative to the earth based on satellite positioning data.
  • the computer system 160 may be used to estimate the road on which the vehicle 100 is traveling using the GPS 126 in conjunction with map data.
  • the IMU 125 may be used to sense position and orientation changes of the vehicle 100 based on inertial acceleration and any combination thereof.
  • the combination of sensors in IMU 125 may include, for example, an accelerometer and a gyroscope. Additionally, other combinations of sensors in IMU 125 are possible.
  • the sensor system 120 may also include sensors that monitor the internal systems of the vehicle 100 (eg, an in-vehicle air quality monitor, a fuel gauge, an oil temperature gauge, etc.). Sensor data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, orientation, velocity, etc.). This detection and identification is a critical function for the safe operation of the vehicle 100 . Sensor system 120 may also include other sensors. This application does not specifically limit this.
  • the control system 130 controls the operation of the vehicle 100 and its components.
  • Control system 130 may include various elements including steering unit 136 , throttle 135 , braking unit 134 , sensor fusion algorithms 133 , computer vision system 132 , route control system 131 , and obstacle avoidance system 137 .
  • the steering unit 136 is operable to adjust the heading of the vehicle 100 .
  • it may be a steering wheel system.
  • the throttle 135 is used to control the operating speed of the engine 114 and thus the speed of the vehicle 100 .
  • Control system 130 may additionally or alternatively include other components than those shown in FIG. 2 . This application does not specifically limit this.
  • the braking unit 134 is used to control the deceleration of the vehicle 100 .
  • the braking unit 134 may use friction to slow the wheels 111 .
  • the braking unit 134 may convert the kinetic energy of the wheels 111 into electrical current.
  • the braking unit 134 may also take other forms to slow the wheels 111 to control the speed of the vehicle 100 .
  • Computer vision system 132 is operable to process and analyze images captured by camera sensor 123 in order to identify objects and/or features in the environment surrounding vehicle 100 .
  • the objects and/or features may include traffic signals, road boundaries and obstacles.
  • Computer vision system 132 may use object recognition algorithms, structure from motion (SFM) algorithms, video tracking, and other computer vision techniques.
  • SFM structure from motion
  • the computer vision system 132 may be used to map the environment, track objects, estimate the speed of objects, and the like.
  • the route control system 131 is used to determine the travel route of the vehicle 100 .
  • route control system 131 may combine data from sensor system 120, GPS 126, and one or more predetermined maps to determine a driving route for vehicle 100.
  • the obstacle avoidance system 137 is used to identify, evaluate, and avoid or otherwise traverse potential obstacles in the environment of the vehicle 100 .
  • control system 130 may additionally or alternatively include components other than those shown and described. Alternatively, some of the components shown above may be reduced.
  • Peripherals 140 may be configured to allow vehicle 100 to interact with external sensors, other vehicles, and/or a user.
  • peripheral devices 140 may include, for example, a wireless communication system 144 , a touch screen 143 , a microphone 142 and/or a speaker 141 .
  • Peripherals 140 may additionally or alternatively include other components than those shown in FIG. 2 . This application does not specifically limit this.
  • peripherals 140 provide a means for a user of vehicle 100 to interact with user interface 170 .
  • touch screen 143 may provide information to a user of vehicle 100 .
  • the user interface 170 may also operate the touch screen 143 to receive user input.
  • peripheral device 140 may provide a means for vehicle 100 to communicate with other devices located within the vehicle.
  • the microphone 142 may receive audio (eg, voice commands or other audio input) from a user of the vehicle 100 .
  • speakers 141 may output audio to a user of vehicle 100 .
  • Wireless communication system 144 may wirelessly communicate with one or more devices, either directly or via a communication network.
  • wireless communication system 144 may use 3G cellular communications, such as code division multiple access (CDMA), EVDO, global system for mobile communications (GSM)/general packet radio service technology (general packet) radio service, GPRS), or 4G cellular communications, such as long term evolution (LTE), or 5G cellular communications.
  • the wireless communication system 144 may communicate with a wireless local area network (WLAN) using WiFi.
  • the wireless communication system 144 may communicate directly with the device using an infrared link, Bluetooth, or ZigBee.
  • Other wireless protocols, such as various vehicle communication systems, for example, wireless communication system 144 may include one or more dedicated short range communications (DSRC) devices, which may include communication between vehicles and/or roadside stations public and/or private data communications.
  • DSRC dedicated short range communications
  • Power supply 150 may be configured to provide power to some or all components of vehicle 100 .
  • the power source 150 may include, for example, a rechargeable lithium-ion or lead-acid battery.
  • one or more battery packs may be configured to provide power.
  • Other power supply materials and configurations are also possible.
  • power source 150 and energy source 113 may be implemented together, as in some all-electric vehicles.
  • Components of the vehicle 100 may be configured to operate in interconnection with other components within and/or outside of their respective systems. To this end, the components and systems of the vehicle 100 may be communicatively linked together through a system bus, network, and/or other connection mechanisms.
  • Computer system 160 may include transceiver 162 and at least one processor 161 .
  • the transceiver 162 may be an independent receiver, an independent transmitter, a transceiver with integrated transceiver functions, or an interface circuit.
  • Processor 161 executes instructions 1631 stored in a non-transitory computer readable medium such as memory 163 .
  • Computer system 160 may also be multiple computing devices that control individual components or subsystems of vehicle 100 in a distributed fashion.
  • the processor 161 may be any conventional processor, such as a commercially available central processing unit (CPU). Alternatively, the processor may be a dedicated device such as application specific integrated circuits (ASIC) or other hardware-based processors.
  • FIG. 2 functionally illustrates a processor, memory, and other elements of the computer system 160 in the same blocks, one of ordinary skill in the art will understand that the processor, computer, or memory may actually include a processor, a computer, or a memory that may or may not Multiple processors, computers, or memories that are not stored within the same physical enclosure.
  • the memory may be a hard drive or other storage medium located within an enclosure other than computer system 160 .
  • processors or computers will be understood to include reference to a collection of processors or computers or memories that may or may not operate in parallel.
  • some components such as the steering and deceleration components may each have their own processor that only performs computations related to component-specific functions .
  • a processor may be located remotely from the vehicle and in wireless communication with the vehicle. In other aspects, some of the processes described herein are performed on a processor disposed within the vehicle while others are performed by a remote processor, including taking steps necessary to perform a single maneuver.
  • memory 163 may include instructions 1631 (eg, program logic) executable by processor 161 to perform various functions of vehicle 100 , including those described above.
  • Memory 214 may also contain additional instructions, including sending data to, receiving data from, interacting with, and/or controlling one or more of propulsion system 110 , sensor system 120 , control system 130 , and peripherals 140 . instruction.
  • memory 163 may store data such as road maps, route information, vehicle location, direction, speed, and other such vehicle data, among other information. Such information may be used by the vehicle 100 and the computer system 160 during operation of the vehicle 100 in autonomous, semi-autonomous and/or manual modes.
  • User interface 170 for providing information to or receiving information from a user of vehicle 100 .
  • user interface 170 may include one or more input/output devices within the set of peripheral devices 140 , such as wireless communication system 144 , touch screen 143 , microphone 142 and speaker 141 .
  • Computer system 160 may control the functions of vehicle 100 based on input received from various subsystems (eg, propulsion system 110 , sensor system 120 , and control system 130 ) and from user interface 170 .
  • computer system 160 may utilize input from control system 130 in order to control steering unit 136 to avoid obstacles detected by sensor system 120 and obstacle avoidance system 137 .
  • computer system 160 is operable to provide control of various aspects of vehicle 100 and its subsystems.
  • one or more of these components described above may be installed or associated with the vehicle 100 separately.
  • memory 163 may exist partially or completely separate from vehicle 100 .
  • the above-described components may be communicatively coupled together in a wired and/or wireless manner.
  • the above component is just an example.
  • components in each of the above modules may be added or deleted according to actual needs, and FIG. 2 should not be construed as a limitation on the embodiments of the present application.
  • the above-mentioned vehicle 100 can be a car, a truck, a motorcycle, a bus, a boat, an airplane, a helicopter, a lawn mower, a recreational vehicle, a playground vehicle, construction equipment, a tram, a golf cart, a train, a cart, etc.
  • the application examples are not particularly limited.
  • the lane line detection result can be determined based on the current lane line image data (that is, the image data collected for the current scene) and the lane line template data, wherein the lane line template data is from the previous frame. Lane line detection results. Therefore, the lane line detection result depends on the lane line detection result of the previous frame.
  • the lane line detection result based on the previous frame can be corrected for the lane line detection result obtained from the current lane line image data. For example, when the lane line image data indicates that a lane line is blocked by an obstacle, and the lane line detection result of the previous frame indicates that the lane line is not blocked by the obstacle, the lane line can be detected according to the lane line detection result of the previous frame.
  • the embodiments of the present application provide a lane line detection method, so as to improve the lane line detection effect in complex scenes.
  • the execution body of the following method may be the processor 161 in the vehicle 100 or a chip in the vehicle 100 . As shown in FIG. 3 , only the execution subject is the processor as an example for description.
  • Step 300 The processor acquires a first image, where the first image is an environment image.
  • the processor may acquire the first image from the camera through the communication interface or a circuit component related to the communication interface.
  • the first image may be an image of the driving environment where the vehicle is located, or the first image may also be an image of the surrounding environment when the vehicle is stationary.
  • the first image may be captured by the camera sensor 123 in the vehicle 100 described above, and the processor 161 obtains the first image from the camera sensor 123 through a communication interface or a circuit component related to the communication interface.
  • the first image may be 1-9 shown in FIG. 1 .
  • Step 310 The processor determines the first set of lane lines and the second set of lane lines according to the first image, wherein the first set of lane lines and the second set of lane lines are obtained in different ways.
  • the processor determines the first set of lane lines from the first image, the second image from the first image, and the first set of lane lines from the second image.
  • the first lane line set includes at least one lane line
  • the first lane line set is a lane line detection result determined based on the first image.
  • the second lane line set includes at least one lane line
  • the second lane line set is a lane line detection result determined based on the second image.
  • the second image may also be referred to as a template image of the first image.
  • the similarity between the second image and the first image is relatively high, for example, greater than a preset similarity threshold.
  • the similarity between the first image and the second image may be determined by a general image similarity algorithm or an artificial method, which will not be repeated here.
  • the lane line detection difficulty of the second image is lower than the lane line detection difficulty of the first image. For example, if the lane lines in the first image are blocked and the lane lines in the second image are not blocked, the difficulty of detecting lane lines in the second image is lower than that in the first image.
  • the left side of FIG. 4 is the first image.
  • the right side of Figure 4 is the second image.
  • the difficulty of detecting lane lines in the second image is lower than that in the first image.
  • the detection difficulty of the lane line of each image may be determined by the confidence level of the lane line corresponding to the image or a manual method or the like.
  • the confidence of the lane line can be used to characterize the difficulty of lane line detection.
  • the confidence level of the lane line may be formed by the sum of the probability of predicting the correct number of lane lines and the mean value of the probability corresponding to the predicted correct lane line.
  • some general lane line detection algorithms can output the number of predicted lane lines and the probability that each lane line in the predicted lane lines is the correct lane line.
  • the number of correctly predicted lane lines is n
  • the number of predicted lane lines is m
  • the probability of the correct number of predicted lane lines is represented by n/m, where n, m are positive integers greater than or equal to 1, and the above predicted correct n
  • the confidence level of the lane line is S1*n/m+S2*P', where S1 and S2 are preset coefficients.
  • the processor determining the first set of lane lines and the second set of lane lines according to the first image may adopt but not limited to the following solutions:
  • Scheme 1 The processor uses the first model to determine the second image according to the first image, uses the second model to determine the first set of lane lines according to the first image, and uses the second model to determine the second set of lane lines according to the second image.
  • the first model is used to predict the corresponding template image based on the image
  • the second model is used to predict the lane line included in the image based on the image.
  • the first model is obtained by training the original image and the template image, wherein the template image is associated with the original image, and the lane line detection difficulty of the template image is lower than the lane line detection difficulty of the original image.
  • the first model makes the first loss function satisfy the first preset condition, and the first loss function is used to indicate the difference between the predicted template image and the template image determined based on the original image.
  • the second model is obtained by training the original image and the template image, wherein the template image is associated with the original image, the similarity between the template image and the original image is high, and the lane line detection difficulty of the template image is lower than that of the original image.
  • the second model makes at least two loss functions among the second loss function, the third loss function and the fourth loss function satisfy corresponding preset conditions.
  • the second model makes the second loss function satisfy the second preset condition, and makes the third loss function satisfy the third preset condition.
  • the second model makes the second loss function satisfy the second preset condition, and makes the fourth loss function satisfy the fourth preset condition.
  • the second model makes the second loss function meet the second preset condition, the third loss function meets the third preset condition, and the fourth loss function meets the fourth preset condition.
  • the second model makes the weighted sum of the second loss function and the third loss function satisfy the fifth preset condition.
  • the second model makes the weighted sum of the second loss function and the fourth loss function satisfy the sixth preset condition.
  • the second model makes the weighted sum of the second loss function, the third loss function and the fourth loss function satisfy the seventh preset condition.
  • the second model makes the second loss function satisfy the second preset condition, and makes the weighted sum of the third loss function and the fourth loss function satisfy the eighth preset condition.
  • the second loss function is used to indicate the difference between the predicted lane line determined based on the original image and the real lane line corresponding to the original image
  • the third loss function is used to indicate the difference between the first feature and the second feature
  • the fourth loss The function is used to indicate whether the first feature is identified as the second feature, the first feature is the feature obtained based on the original image, the second feature is the feature obtained based on the template image, and the first feature and the second feature are the same type of feature.
  • the first feature may refer to a feature directly extracted from the original image, or a feature generated based on the original image.
  • the second feature may refer to a feature directly extracted from the template image, or a feature generated based on the template image.
  • Solution 2 The processor uses a third model to determine the second image and the first set of lane lines according to the first image, and uses the third model to determine the second set of lane lines according to the second image.
  • the third model is used to predict the corresponding template image and the lane lines included in the image based on the image.
  • the third model is obtained by training the original image and the template image, wherein the template image is associated with the original image, the similarity between the template image and the original image is high, and the lane line detection difficulty of the template image is lower than that of the original image. Difficulty of detection.
  • the third model makes the first loss function satisfy a first preset condition, and makes the second loss function satisfy a second preset condition.
  • the third model makes the first loss function meet the first preset condition, the second loss function meets the second preset condition, and the third model makes the third loss function meet the third preset condition.
  • the third model makes the first loss function meet the first preset condition, the second loss function meets the second preset condition, and the third model makes the fourth loss function meet the fourth preset condition.
  • the third model makes the first loss function meet the first preset condition, the second loss function meets the second preset condition, the third model makes the third loss function meet the third preset condition, and the third model makes the third loss function meet the third preset condition.
  • the four loss functions satisfy the fourth preset condition.
  • the third model makes the first loss function satisfy the first predetermined condition and the weighted sum of the second loss function satisfy the ninth predetermined condition.
  • the third model makes the weighted sum of the first loss function, the second loss function, the third loss function and the fourth loss function satisfy the tenth preset condition.
  • the third model makes the weighted sum of the first loss function and the second loss function satisfy the eleventh preset condition, and makes the weighted sum of the third loss function and the fourth loss function satisfy the eighth preset condition.
  • the third model makes the first loss function satisfy the first preset condition, makes the second loss function satisfy the second preset condition, and makes the weighted sum of the third loss function and the fourth loss function satisfy the eighth preset condition.
  • the first loss function is used to indicate the difference between the predicted template image determined based on the original image and the template image
  • the second loss function is used to indicate the difference between the predicted lane line determined based on the original image and the real lane line corresponding to the original image.
  • the third loss function is used to indicate the difference between the first feature and the second feature
  • the fourth loss function is used to indicate whether the first feature is identified as the second feature.
  • the first feature is a feature obtained based on the original image
  • the second feature is a feature obtained based on a template image
  • the first feature and the second feature are features of the same type.
  • the first feature may refer to a feature directly extracted from the original image, or a feature generated based on the original image.
  • the second feature may refer to a feature directly extracted from the template image, or a feature generated based on the template image.
  • the training data involved in the first model, the second model and the third model may be the same or different.
  • the training data may include original images and template images.
  • the original image refers to the collected environment image.
  • the original image refers to an image of the environment where the vehicle is located, and the vehicle here may refer to a designated vehicle (eg, vehicle 100 ) or any vehicle using the above-mentioned model.
  • the template image is associated with the original image, wherein the similarity between the template image and the original image is high, and the lane line detection difficulty of the template image is lower than that of the original image.
  • the similarity between the template image and the original image may be determined by a general image similarity algorithm or an artificial method, which will not be repeated here.
  • the lane line detection difficulty of each image can be determined by the confidence level of the lane line corresponding to the image or by artificial means. For details, please refer to the above description about the confidence level of the lane line.
  • the labels can determine which lane lines are correct among the predicted lane lines output by the general lane line detection algorithm, and obtain the predicted lane lines.
  • the template image can be obtained by, but not limited to, the following methods:
  • Method 1 Use the original image to generate a template image.
  • the template image can be generated by, but not limited to, the following manners: using image enhancement to eliminate the influence in the original image to generate a template image, or using image segmentation to remove interfering objects in the original image to generate a template image, or using generative confrontation to generate similar images. as a template image, etc.
  • Method 2 Manually filter out template images from multiple original images in the same geographic area.
  • images related to the GPS information can be searched to obtain multiple original images of the same geographic area.
  • the geographic area here may refer to an area determined for a certain geographic indication building (eg, a mall or a bank).
  • multiple original images of the same geographic area include N images, where N is a positive integer greater than 2, and a template image can be filtered out of the N images by manual means, for example, an image with fewer obstacles or a light source can be selected. suitable image.
  • Mode 3 Combined with the GPS information of an original image, an image related to the GPS information can be searched as a candidate image, and a candidate image with a high similarity to the original image and the least difficulty in lane line detection can be selected as the candidate image.
  • the template image associated with the original image can be selected as the candidate image.
  • GPS information referred to in Modes 2 and 3 may also be replaced with other sensor information, which is not limited in this embodiment of the present application.
  • Method 4 Using image retrieval, a large number of images similar to the original image can be obtained as candidate images, and the candidate image with high similarity to the original image and the least difficulty in lane line detection is further selected as the template associated with the original image. image.
  • the confidence of the lane line corresponding to each candidate image and the similarity between each candidate image and the original image can be calculated, and the one with the highest confidence and the similarity greater than the preset similarity threshold is selected.
  • the image to be selected is used as the template image.
  • a candidate image with a confidence greater than a preset reliability threshold and a similarity greater than a preset similarity threshold is selected as the template image.
  • the first loss function is used to indicate the difference between the predicted template image and the template image determined based on the original image.
  • the difference between the predicted template image and the template image may be determined by the difference between the matrix corresponding to the predicted template image and the matrix corresponding to the template image.
  • the first loss function can also be referred to as a template image prediction loss function, wherein the predicted template image can be represented by f template (image), and the template image can be represented by template gt , then the first loss function can be represented as:
  • the first loss function can be expressed as:
  • the second loss function is used to indicate the difference between the predicted lane line determined based on the original image and the real lane line corresponding to the original image.
  • the difference between the predicted lane line and the real lane line corresponding to the original image may be determined by the difference between the matrix corresponding to the predicted lane line and the matrix corresponding to the real lane line corresponding to the original image.
  • the second loss function can also be called the lane line detection loss function, where the predicted lane line can be represented by f lane (image), and the real lane line corresponding to the original image can be represented by lane gt , then the second loss function can be represented as:
  • the second loss function can be expressed as:
  • the first features and the second features are features of the same type.
  • the first feature may refer to a feature directly extracted from the original image, or a feature generated based on the original image.
  • G represents a generator
  • G(image) represents generating a new image based on the original image
  • features can be extracted from G(image) as the first feature.
  • the second feature may refer to a feature directly extracted from the template image, or a feature generated based on the template image.
  • the features here can be color features or texture features.
  • the third loss function can also be called the similarity metric loss function: the first feature can be represented by feature original , and the second feature can be represented by feature template , then the third loss function can be represented as:
  • the third loss function can be expressed as:
  • the fourth loss function can also be called a discriminator loss function, the discriminator can also be called a binary classifier, and the discriminator is an algorithm for classifying whether or not.
  • the fourth loss function can be expressed as:
  • the fourth loss function can be expressed as:
  • the above fourth loss function can be understood as minimizing the difference between the features generated by the generator based on the original image and the features generated based on the template image, and at the same time maximizing the features identified by the discriminator based on the original image.
  • the images are generated with different features.
  • D represents the discriminator, and D is used to determine whether the feature (ie, the first feature) generated based on G (image) is a feature template (second feature).
  • the third loss function and the fourth loss function can be used to constrain the features obtained based on the original image and the features obtained based on the template image to have better consistency during the model training process, thereby improving the prediction of the second model and the third model. Effect.
  • first feature and the second feature are input to the third loss function, and the first feature and the second feature are input to the fourth loss function, as shown in FIG. 5 .
  • the third loss function outputs the second loss value
  • the fourth loss function outputs the third loss value
  • the first loss value can be calculated according to the first loss value and the second loss value.
  • this application uses looser constraints for abnormal samples and stricter constraints for ordinary samples.
  • T lower and T high the degree of abnormality of the sample is judged by the second loss value loss s output by the third loss function, and when loss s ⁇ T lower , the fourth The weight of the third loss value output by the loss function.
  • loss s >T high the weight of loss s is reduced, or even loss s is not used.
  • the first model, the second model, and the third model involved in the embodiments of the present application may be obtained by using the above-mentioned training data through an autoencoder or other neural network systems, and the embodiments of the present application do not limit the specific methods used for training the models.
  • a general neural network system includes an input layer, a hidden layer, and an output layer, correspondingly including three types of neurons (basic units of a neural network): input layer neurons, hidden layer neurons, and output layer neurons.
  • the neurons in the input layer obtain information from the external world, and the neurons in the hidden layer have no direct connection with the external world.
  • the neurons in the hidden layer perform the calculation of preset functions and transmit information from the neurons in the input layer to the neurons in the output layer.
  • Output layer neurons are used to transmit information to the outside world.
  • the auto-encoder is a neural network with 3 layers or more, and the auto-encoder belongs to unsupervised learning.
  • Autoencoders can be used for dimensionality reduction or feature learning.
  • the function of the autoencoder network is to compress the input samples to the hidden layer, and then reconstruct the samples at the output end, where the number of neurons in the hidden layer is less than the number of neurons in the input layer, and the number of neurons in the output layer is equal to the number of neurons in the input layer, for example,
  • Example 1 The following briefly describes the training process of the third model.
  • Step 1 Get training data.
  • Training data can include original images and template images. For details, please refer to the above-mentioned related content about the training data, and the repetition will not be repeated.
  • Step 2 Input the original image and the template image into the preset system, perform calculation through the preset function corresponding to the hidden layer neurons in the preset system, and output the predicted lane line and the predicted template image.
  • the preset system here may refer to the above-described neural network system or an automatic encoder, etc., which is not limited in this embodiment of the present application.
  • Step 3 Calculate the difference between the predicted template image and the template image according to the first loss function (denoted as the first difference), and calculate the difference between the predicted lane line and the actual lane line corresponding to the original image according to the second loss function (denoted as second difference).
  • Step 4 Adjust the parameters of the preset function corresponding to the neurons of the hidden layer in the preset system multiple times, so that the first difference calculated by the first loss function satisfies the first preset condition, and the first difference calculated by the second loss function can be obtained.
  • the second difference value satisfies the second preset condition, the training process is stopped, and the third model is obtained.
  • the parameters of the preset function corresponding to the hidden layer neurons in the preset system are adjusted multiple times until the first difference is within the first preset error range and the second difference is within the second preset error range Inside.
  • the first preset error range and the second preset error range may be set according to empirical values.
  • the parameters of the preset function corresponding to the hidden layer neurons in the preset system are adjusted multiple times until the first difference is the smallest and the second difference is within the second preset error range.
  • the second preset error range may be set according to an empirical value.
  • the parameters of the preset function corresponding to the neurons in the hidden layer in the preset system are adjusted multiple times until the first difference is within the first preset error range and the second difference is the smallest.
  • the first preset error range may be set according to an empirical value.
  • Example 2 The following briefly describes the training process of the third model.
  • Step 1 Get training data.
  • Training data can include original images and template images. For details, please refer to the above-mentioned related content about the training data, and the repetition will not be repeated.
  • Step 2 Input the original image and the template image into the preset system, perform calculation through the preset function corresponding to the hidden layer neurons in the preset system, and output the predicted lane line and the predicted template image.
  • the preset system here may refer to the above-described neural network system or an automatic encoder, etc., which is not limited in this embodiment of the present application.
  • Step 3 Calculate the difference between the predicted template image and the template image according to the first loss function (denoted as the first difference), and calculate the difference between the predicted lane line and the actual lane line corresponding to the original image according to the second loss function (denoted as second difference).
  • Step 4 Obtain the first feature and the second feature, and output the second loss value and the third loss value according to the third loss function and the fourth loss function, respectively.
  • step 4 may be after step 2, or may be before step 2, which is not limited in this embodiment of the present application.
  • Step 5 Adjust the parameters of the preset function corresponding to the neurons in the hidden layer in the preset system multiple times, so that the first difference calculated by the first loss function satisfies the first preset condition, and the first difference calculated by the second loss function can be obtained.
  • the second difference value satisfies the second preset condition
  • the second loss value calculated by the third loss function satisfies the third preset condition
  • the fourth loss value calculated by the fourth loss function satisfies the fourth preset condition
  • the parameters of the preset function corresponding to a part of neurons in the hidden layer in the preset system are adjusted multiple times until the second loss value is within the third preset error range, and the third loss value is within the third loss value.
  • Four preset error ranges the parameters of the preset function corresponding to another part of the neurons in the hidden layer in the preset system are adjusted multiple times until the first difference is within the first preset error range and the second difference is within the second preset error range .
  • the parameters of the preset function corresponding to the hidden layer neurons in the preset system are adjusted multiple times until the first difference is within the first preset error range, the second difference is the smallest, and the second loss The value is within a third predetermined error range, and the third loss value is within a fourth predetermined error range.
  • the third preset error range and the fourth preset error range may be set according to empirical values.
  • the parameters of the preset function corresponding to the hidden layer neurons in the preset system are adjusted multiple times until the first difference is the smallest, the second difference is within the second preset error range, and the first loss The value is within a fifth preset error range.
  • the first loss value may be a weighted sum of the second loss value and the third loss value.
  • the fifth preset error range may be set according to an empirical value.
  • Example 3 The following briefly describes the training process of the first model.
  • Step 1 Get training data.
  • Training data can include original images and template images. For details, please refer to the above-mentioned related content about the training data, and the repetition will not be repeated.
  • Step 2 Input the original image and the template image into the preset system, perform calculation through the preset function corresponding to each neuron of the hidden layer in the preset system, and output the predicted template image.
  • the preset system here may refer to the above-described neural network system or an automatic encoder, etc., which is not limited in this embodiment of the present application.
  • Step 3 Calculate the difference between the predicted template image and the template image according to the first loss function (referred to as the first difference).
  • Step 4 Adjust the parameters of the preset function corresponding to each neuron of the hidden layer in the preset system multiple times, so that the first difference calculated by the first loss function satisfies the first preset condition, stop the training process, and obtain the first model .
  • the parameters of the preset function corresponding to each neuron in the hidden layer in the preset system are adjusted multiple times until the first difference calculated by the first loss function satisfies the first preset condition or is the smallest.
  • the first preset error range may be set according to an empirical value.
  • Example 4 The following briefly describes the training process of the second model.
  • Step 1 Get training data. If the first difference calculated by the first loss function satisfies the first preset condition, reference may be made to the above-mentioned related content about the training data, and the repetition will not be repeated.
  • Step 2 Input the original image and the template image into the preset system, perform calculation through the preset function corresponding to each neuron in the hidden layer in the preset system, and output the predicted lane line.
  • the preset system here may refer to the above-described neural network system or an automatic encoder, etc., which is not limited in this embodiment of the present application.
  • Step 3 Calculate the difference between the predicted lane line and the real lane line corresponding to the original image according to the second loss function (referred to as the second difference).
  • Step 4 Obtain the first feature and the second feature, and output the second loss value and the third loss value according to the third loss function and the fourth loss function, respectively.
  • step 4 may be after step 2, or may be before step 2, which is not limited in this embodiment of the present application.
  • Step 5 Adjust the parameters of the preset function corresponding to each neuron in the hidden layer in the preset system multiple times, so that the second difference calculated by the second loss function satisfies the second preset condition, and the third loss function is calculated.
  • the parameters of the preset function corresponding to each neuron in the hidden layer in the preset system are adjusted multiple times until the second difference is the smallest, and the second loss value is within the third preset error range, and the third The loss value is within a fourth preset error range.
  • the third preset error range and the fourth preset error range may be set according to empirical values.
  • the parameters of the preset function corresponding to each neuron in the hidden layer in the preset system are adjusted multiple times until the second difference is within the second preset error range and the first loss value is within the fifth preset Within the error range, for example, the first loss value may be a weighted sum of the second loss value and the third loss value.
  • the fifth preset error range may be set according to an empirical value.
  • Step 320 The processor determines a third set of lane lines according to the first set of lane lines and the second set of lane lines.
  • the third set of lane lines can be used as the final lane line detection result and presented on a display interface (eg, the touch screen 143 in the vehicle 100 ).
  • the processor may determine the first set of lane lines in the first set of lane lines and the third set of lane lines in the second set of lane lines.
  • the second lane line indicates the same lane line in the environment, and the processor treats the first lane line, or the second lane line, or the third lane line as the lane line in the third lane line set.
  • the third lane line is determined according to the first lane line and the second lane line.
  • the x value corresponding to lane line 1 and the x value corresponding to lane line 2 If the value is different, the difference between the x values can be calculated. Further, the difference between the x value corresponding to the lane line 1 and the x value corresponding to the lane line 2 can be obtained every preset interval, and then the lane line 1 and the lane line can be obtained. 2, and then the sum of each difference can be calculated according to the difference set to obtain the difference sum of lane line 1 and lane line 2.
  • the difference sum of lane line 1 and lane line 3 as well as the difference sum of lane line 1 and lane line 4 can be obtained, and the lane line with the smallest difference sum (that is, lane line 2) can be further selected as the
  • the lane line matched by line 1, that is, lane line 1 and lane line 2 are the same lane line in the environment.
  • the processor may take the lane line 1 or the lane line 2 as the lane line in the third lane line set, or, according to 1/1 of each difference value in the difference value set between the lane line 1 and the lane line 2 2. Determine a new lane line as the lane line in the third lane line set.
  • the process of determining the matching lanes in the first set of lanes and the second set of lanes if the number of lanes included in the third set of lanes is equal to the preset number of lanes, then Stop continuing to match lane lines in the first set of lane lines with lane lines in the second set of lane lines.
  • the processor divides the first lane line from the first set of lane lines and/or divides the second lane line from the second set of lane lines. Lane lines other than lane lines are used as lane lines in the third set of lane lines.
  • the processor may randomly select lane lines in the first set of lane lines other than the first lane lines as lane lines in the third set of lane lines.
  • the processor may randomly select lane lines in the second lane line set other than the second lane line as the lane lines in the third lane line set.
  • the processor may randomly select lane lines in the first lane line set other than the first lane line as the lane lines in the third lane line set, and randomly select lanes in the second lane line set except the second lane line lines as lane lines in the third set of lane lines.
  • the processor may determine the probability that each lane line included in the first lane line set is a correct lane line according to the second model or the third model, exclude the first lane line in the first lane line set, and select the remaining lanes The lane with the highest probability of being the correct lane in the line is used as the lane in the third set of lanes.
  • the processor may determine the probability that each lane included in the second lane line set is a correct lane line according to the second model or the third model, exclude the second lane line in the second lane line set, and select the remaining lane lines The lane line with the highest probability that the lane line is the correct lane line is used as the lane line in the third lane line set.
  • the processor may determine the probability that each lane included in the first set of lane lines is a correct lane line and the probability that each lane included in the second set of lane lines is a correct lane line according to the second model or the third model, Exclude the first lane line in the first lane line set, select the lane line with the highest probability of being the correct lane line among the remaining lane lines in the first lane line set as the lane line in the third lane line set, and at the same time, exclude For the second lane line in the second lane line set, the lane line with the highest probability of being the correct lane line among the remaining lane lines in the second lane line set is selected as the lane line in the third lane line set.
  • the third set of lane lines when it is determined that the number of lane lines included in the third set of lane lines is less than the preset number of lane lines, divide the first lane line from the first set of lane lines and/or the second lane line from the second set of lane lines.
  • the outer lane lines are used as lane lines in the third lane line set, and the number of lane lines included in the third lane line set needs to be equal to the second threshold, where the second threshold is determined according to the environment where the vehicle is located.
  • the vehicle needs to determine the lane lines on the left and right sides. Therefore, the preset number of lane lines is 2.
  • the first The lane line set or the second lane line set further determines one lane line as the lane line in the third lane line set.
  • the processor has determined that the third set of lane lines includes a lane line on the left side of the own vehicle, and further, the processor can determine the lane on the right side of the own vehicle included in the first set of lane lines according to the second model or the third model.
  • the probability that the line is the correct lane line, and the probability that the lane line on the right side of the vehicle included in the second lane line set is the correct lane line, and the lane line with the higher probability among the above two lane lines is used as the third lane line set.
  • step 320 it can be known that when the processor can determine the second set of lane lines according to the second image by using the above-mentioned third model, the processor can also obtain a third image at the same time, wherein the third image is the second set of lane lines.
  • the template image of the image, the third image is similar to the second image, and the lane line detection difficulty of the third image is lower than the lane line detection difficulty of the second image.
  • the processor may use a third model to determine a fourth set of lane lines according to the third image.
  • the processor may also determine the third set of lane lines according to the first set of lane lines and the fourth set of lane lines.
  • the processor first determines a fifth set of lane lines based on the second set of lane lines and the fourth set of lane lines, and determines a third set of lane lines according to the first set of lane lines and the fifth set of lane lines.
  • the method of determining the fifth set of lane lines based on the second set of lane lines and the fourth set of lane lines may refer to the above-mentioned method of determining the third set of lane lines according to the first set of lane lines and the second set of lane lines, and the repetition is not repeated. Repeat.
  • a set of lane lines can be determined on the basis of the further optimized template image (ie, the third image), and then the final set of lane lines can be determined according to the set of lane lines and the first set of lane lines, and then the final set of lane lines can be optimized. Obtained lane line detection results.
  • the processor may further determine the first loss value according to the second loss value and the third loss value, and the first loss value is used for Characterizing the similarity of the second image and the first image, the processor may present the first loss value on a display interface (eg, touch screen 143 in vehicle 100 ).
  • the second loss value is obtained by using the third loss function according to the first image and the second image
  • the third loss value is obtained by using the fourth loss function according to the first image and the second image.
  • the first loss value K1*second loss value+K2*third loss value, K1>0, K2>0.
  • K1+K2 1.
  • the first loss value K3*second loss value+K4*third loss value, K3 ⁇ 0, K4>0, K1>K3 .
  • the embodiments of the present application further provide an apparatus for implementing the above methods.
  • the apparatus may include hardware structures and/or software modules, and implement the above functions in the form of hardware structures, software modules, or hardware structures plus software modules. Whether one of the above functions is performed in the form of a hardware structure, a software module, or a hardware structure plus a software module depends on the specific application and design constraints of the technical solution.
  • the apparatus provided in this embodiment of the present application may be a chip or circuit capable of performing functions corresponding to the above method, and the chip or circuit may be provided in a device such as a processor. Further, the apparatuses provided in the embodiments of the present application can also be implemented in hardware or a combination of hardware and computer software. Whether a function is performed by hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Experts may use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of the embodiments of the present application.
  • the apparatus provided in this embodiment of the present application may divide functional modules, for example, each functional module may be divided into each function, or two or more functions may be integrated into one processing module.
  • the above-mentioned integrated modules can be implemented in the form of hardware, and can also be implemented in the form of software function modules. It should be noted that, the division of modules in the embodiments of the present application is schematic, and is only a logical function division, and there may be other division manners in actual implementation.
  • FIG. 7 a schematic structural diagram of a lane line detection apparatus is provided in an embodiment of the present application.
  • the apparatus may be a processor or an apparatus within a processor.
  • the apparatus 700 may include: a processing module 71 and an obtaining module 72 .
  • the apparatus 700 may also include other modules, which are not limited in the embodiments of the present application, and only main functional modules are shown.
  • the obtaining module 72 is configured to obtain a first image, where the first image is an environment image.
  • the processing module 71 is configured to obtain a first set of lane lines and a second set of lane lines according to the first image, and obtain a third set of lane lines according to the first set of lane lines and the second set of lane lines. Wherein, the first lane line set and the second lane line set are obtained in different ways.
  • the processing module 71 in this embodiment of the present application may be implemented by a processor or a circuit component related to the processor, and the acquiring module 72 may be implemented by a communication interface or a circuit component related to the communication interface.
  • the communication interface may include, for example, a transmitter and a receiver, the processor, the transmitter and the receiver being coupled to each other, wherein the transmitter and receiver are implemented, for example, by an antenna, feeder, codec, etc., or, if the device For a chip provided in the device, the transmitter and the receiver are, for example, communication interfaces in the chip, and the communication interface is connected to the radio frequency transceiver component in the device to realize information transmission and reception through the radio frequency transceiver component.
  • the apparatus 800 provided by the embodiment of the present application is shown, and the apparatus shown in FIG. 8 may be a hardware circuit implementation of the apparatus shown in FIG. 7 .
  • the apparatus may be used to perform the processor functions in the flowchart shown in FIG. 3 .
  • FIG. 8 shows only the main components of the device.
  • the apparatus shown in FIG. 8 may be a chip or circuit capable of performing functions corresponding to the foregoing method, or may be a device including the foregoing chip or circuit, which is not limited in this embodiment of the present application.
  • the apparatus 800 shown in FIG. 8 includes at least one processor 820, which is configured to implement the functions of the processor in FIG. 3 provided by the embodiment of the present application.
  • the apparatus 800 may also include at least one memory 830 for storing program instructions and/or data.
  • Memory 830 is coupled to processor 820 .
  • the coupling in the embodiments of the present application is an indirect coupling or communication connection between devices, units or modules, which may be in electrical, mechanical or other forms, and is used for information exchange between devices, units or modules.
  • Processor 820 may cooperate with memory 830 .
  • Processor 820 may execute program instructions stored in memory 830 . At least one of the at least one memory may be included in the processor.
  • the device 800 may not include the memory 830, and the processor 820 may read the instructions (programs or codes) in the memory outside the chip or circuit to realize the instructions shown in FIG. 3 . Examples of processor functionality provided.
  • the apparatus 800 may also include a communication interface 810 for communicating with other devices through a transmission medium, so that the apparatus used in the apparatus 800 may communicate with other devices.
  • the communication interface may be a transceiver, a circuit, a bus, a module, or other types of communication interfaces.
  • the transceiver may be an independent receiver, an independent transmitter, a transceiver with integrated transceiver functions, or an interface circuit.
  • the processor 820 uses the communication interface 810 to send and receive data, and is used to implement the functions of the processor in the embodiment shown in FIG. 3 . For details, reference may be made to the foregoing description, which will not be repeated here.
  • the apparatus 800 may also include a communication bus 840 .
  • the communication interface 810, the processor 820 and the memory 830 can be connected to each other through a communication bus 840; the communication bus 840 can be a peripheral component interconnect (PCI for short) bus or an extended industry standard architecture (extended industry standard architecture). , referred to as EISA) bus and so on.
  • PCI peripheral component interconnect
  • EISA extended industry standard architecture
  • the communication bus 840 can be divided into an address bus, a data bus, a control bus, and the like. For ease of presentation, only one thick line is used in FIG. 8, but it does not mean that there is only one bus or one type of bus.
  • the apparatus provided in the embodiments of the present application when implemented using software, it may be implemented in whole or in part in the form of a computer program product.
  • the computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are realized.
  • the computer may be a general purpose computer, special purpose computer, computer network, or other programmable device.
  • the computer instructions may be stored in or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions may be downloaded from a website site, computer, server or data center Transmission to another website site, computer, server, or data center by wire (eg, coaxial cable, optical fiber, digital subscriber line, DSL) or wireless (eg, infrared, wireless, microwave, etc.).
  • the computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes an integration of one or more available media.
  • the usable media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media (eg, solid state disks (SSDs)), and the like.
  • the processor included in the above-mentioned apparatus for executing the method provided by the embodiment of the present application may be a central processing unit (central processing unit, CPU), a general-purpose processor, a digital signal processor (digital signal processor, DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute the various exemplary logical blocks, modules and circuits described in connection with this disclosure.
  • the processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.
  • the steps of the method or algorithm described in conjunction with the embodiments of the present application may be implemented in a hardware manner, or may be implemented in a manner in which a processor executes software instructions.
  • Software instructions can be composed of corresponding software modules, and software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM) memory, erasable programmable read-only memory Memory (erasable programmable read-only memory, EPROM), electrically erasable programmable read-only memory (electrically erasable programmable read-only memory, EEPROM), registers, hard disks, mobile hard disks, compact disc read-only memory , CD-ROM) or any other form of storage medium known in the art.
  • RAM random access memory
  • ROM read-only memory
  • EPROM erasable programmable read-only memory
  • EEPROM electrically erasable programmable read-only memory
  • registers hard disks, mobile hard disks, compact disc read-only memory , CD-ROM or any other form of
  • An exemplary storage medium is coupled to the processor, such that the processor can read information from, and write information to, the storage medium.
  • the storage medium can also be an integral part of the processor.
  • the processor and storage medium may reside in an ASIC.
  • the ASIC may be located in the radar device or in the detection equipment in which the radar device is installed.
  • the processor and the storage medium may also be present as discrete components in the radar device or the detection device in which the radar device is mounted.
  • FIGS. 7-8 only show a simplified design of the device.
  • the apparatus provided by the embodiments of the present application may include any number of transmitters, receivers, processors, controllers, memories, and other possible elements.
  • An embodiment of the present application further provides a chip, which is connected to a memory and used to read and execute a software program stored in the memory.
  • the chip realizes The function of the processor in Figure 3.
  • Embodiments of the present application further provide a computer-readable storage medium, including instructions, which, when executed on a computer, enable the computer to implement the functions of the processor in FIG. 3 .
  • An embodiment of the present application further provides a terminal device, where the terminal device includes the lane line detection apparatus mentioned in the above embodiments of the present application.
  • the terminal device here can be a robot, a vehicle or a vehicle-mounted device.
  • An embodiment of the present application further provides a vehicle, and the vehicle includes at least one lane line detection device mentioned in the above embodiments of the present application.
  • the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
  • computer-usable storage media including, but not limited to, disk storage, CD-ROM, optical storage, etc.
  • These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions
  • the apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Traffic Control Systems (AREA)

Abstract

一种车道线检测方法及装置（700,800），适用于智能驾驶或者自动驾驶领域，方法包括：获取第一图像，第一图像为环境图像（300），根据第一图像获取第一车道线集合和第二车道线集合（310），根据第一车道线集合和第二车道线集合获取第三车道线集合（320）。其中，第一车道线集合和第二车道线集合的获取方式不同。采用车道线检测方法，在获取第一图像后，根据第一图像确定两个车道线集合，然后由两个车道线集合确定一个最终的车道线集合，可以实现在复杂场景下的车道线检测效果的提升。

Description

一种车道线检测方法及装置 技术领域
本申请实施例涉及智能驾驶或者自动驾驶领域，尤其涉及一种车道线检测方法及装置。
背景技术
车道线检测是自动驾驶辅助***(advanced driver assistance systems，ADAS)的重要任务，是实现自适应巡航控制(adaptive cruise control，ACC)、车道偏移预警(lane departure warning system，LDWS)等技术的关键环节。在针对智能车辆或无人车辆的研究中，车道线检测是一个复杂且具有挑战性的课题。车道线作为道路的一个主要部分，起到为无人驾驶车辆提供参考、指导安全驾驶的作用。同时，车道线检测还可以后续用于实现道路定位、确定车辆和道路之间的相对位置关系、以及辅助车辆的决策规划。
当前已有很多基于传统图像处理实现车道线检测算法，但是这些算法在场景明确，易于识别的道路上取得了不错的效果。但随着研究的深入，车道线检测任务所对应的场景越来越多样化，逐步已经脱离了对于车道线“白、黄色线条”这种理解。目前，当车道线是模糊的、被光照影响的、甚至是完全被障碍物遮挡时，如何提升车道线检测效果是一个难题。
如图1所示，1、2、9存在障碍物遮挡，3，5光线不足，4车道线模糊，6、8光线适宜不存在障碍物遮挡，7光线过强，其中，未能准确检测到的车道线用虚线表示，能够准确检测到的车道线用实线表示。对于图1中除上述6、8所示的场景，传统的图像处理方法均无法实现准确检测车道线。
发明内容
本申请实施例提供一种车道线检测方法及装置，用以提升在复杂场景下的车道线检测效果。
第一方面，本申请实施例提供一种车道线检测方法，该方法包括：获取第一图像，所述第一图像为环境图像，根据所述第一图像获取第一车道线集合和第二车道线集合，根据所述第一车道线集合和所述第二车道线集合获取第三车道线集合。其中，所述第一车道线集合和所述第二车道线集合的获取方式不同。采用上述方法，在获取第一图像后，根据第一图像确定两个车道线集合，然后由两个车道线集合确定一个最终的车道线集合，可以实现提升在复杂场景下的车道线检测效果。
在一种可能的设计中，在根据所述第一图像获取第一车道线集合和第二车道线集合，时，可以首先根据所述第一图像确定第二图像，所述第二图像用于确定所述第二车道线集合。采用上述设计，通过首先确定第二图像，然后由第二图像确定第二车道线检测集合，可以实现通过借助模板图像(即第二图像)提升车道线检测效果。
在一种可能的设计中，根据所述第一图像和所述第二图像确定第一损失值，所述第一损失值指示所述第一图像和所述第二图像的相似度。采用上述设计，可以输出第一图像和第二图像的相似度，进而可以为后续辅助车辆或用户的决策提供一定参考。
在一种可能的设计中，在根据第一图像确定第二图像时，可以根据所述第一图像采用 第一模型确定所述第二图像；所述第一模型是根据原始图像和模板图像训练获得的，其中，所述模板图像与所述原始图像关联，所述模板图像的车道线检测难度低于所述原始图像的车道线检测难度；所述第一模型使第一损失函数满足第一预设条件，所述第一损失函数用于指示基于所述原始图像确定的预测模板图像与所述模板图像的差值。采用上述设计可以通过原始图像和模板图像训练获得用于生成预测模板图像的第一模型，且第一模型使第一损失函数满足对应的预设条件。
在一种可能的设计中，在根据所述第一图像确定第一车道线集合时，可以根据所述第一图像采用第二模型确定所述第一车道线检测集合，所述第二模型是根据原始图像和模板图像训练获得的，其中，所述模板图像与所述原始图像关联，所述模板图像的车道线检测难度低于所述原始图像的车道线检测难度；其中，所述第二模型使第二损失函数、第三损失函数和第四损失函数中的至少两个损失函数满足对应的预设条件；其中，所述第二损失函数用于指示基于所述原始图像确定的预测车道线与所述原始图像对应的真实车道线的差值，所述第三损失函数用于指示第一特征与第二特征的差值，所述第四损失函数用于指示所述第一特征是否被鉴别为所述第二特征，所述第一特征为基于所述原始图像获取的特征，所述第二特征为基于所述模板图像获取的特征，所述第一特征与所述第二特征为相同类型的特征。
采用上述设计可以通过原始图像和模板图像训练获得用于生成预测车道线的第二模型，且第二模型使第二损失函数、第三损失函数和第四损失函数中的至少两个损失函数满足对应的预设条件。其中，第三损失函数和第四损失函数用于实现在第二模型的训练过程中约束基于原始图像获取的特征和基于模板图像获取的特征具有较好的一致性，进而可以提升第二模型的预测效果。
在一种可能的设计中，在根据所述第一图像确定第二图像和第一车道线集合时，可以根据所述第一图像采用第三模型确定所述第二图像和所述第一车道线集合，所述第三模型是根据原始图像和模板图像训练获得的，其中，所述模板图像与所述原始图像关联，所述模板图像的车道线检测难度低于所述原始图像的车道线检测难度；其中，所述第三模型使第一损失函数满足第一预设条件，以及使第二损失函数满足第二预设条件，所述第一损失函数用于指示基于所述原始图像确定的预测模板图像与所述模板图像的差值，所述第二损失函数用于指示基于所述原始图像确定的预测车道线与所述原始图像对应的真实车道线的差值。
采用上述设计可以通过原始图像和模板图像训练获得用于生成预测车道线和预测模板图像的第三模型，且第三模型使第一损失函数和第二损失函数分别满足对应的预设条件。
在一种可能的设计中，所述第三模型使第一损失函数满足第一预设条件，使第二损失函数满足第二预设条件，以及所述第三模型使第三损失函数满足第三预设条件和/或使第四损失函数满足第四预设条件，其中，所述第三损失函数用于指示第一特征与第二特征的差值，所述第四损失函数用于指示所述第一特征是否被鉴别为所述第二特征，所述第一特征为基于所述原始图像获取的特征，所述第二特征为基于所述模板图像获取的特征，所述第一特征与所述第二特征为相同类型的特征。第三损失函数和第四损失函数用于实现在第二模型的训练过程中约束基于原始图像获取的特征和基于模板图像获取的特征具有较好的一致性，进而可以提升第二模型的预测效果。
在一种可能的设计中，所述第一损失值是根据所述第一图像和所述第二图像采用所述 第三损失函数获得的；或者，所述第一损失值是根据所述第一图像和所述第二图像采用所述第四损失函数获得的；或者，所述第一损失值是根据第二损失值和第三损失值确定的，所述第二损失值是根据所述第一图像和所述第二图像采用所述第三损失函数获得的，所述第三损失值是根据所述第一图像和所述第二图像采用所述第四损失函数获得的。
第一损失值可以表征第一图像和第二图像的相似度。
在一种可能的设计中，所述第三车道线集合包括所述第一车道线集合和所述第二车道线集合中至少一条相同的车道线。采用上述设计，在根据第一车道线集合和第二车道线集合确定第三车道线集合时，可以确定第一车道线集合中的第一车道线与第二车道线集合中的第二车道线指示环境中的同一条车道线，进而将第一车道线，或第二车道线，或第三车道线作为第三车道线集合中的车道线。其中，第三车道线是根据第一车道线和第二车道线确定的。
在一种可能的设计中，还包括：确定所述第三车道线集合包括的车道线数目小于预设车道线数目时，将所述第一车道线集合和/或所述第二车道线集合中除所述第三车道线集合已包括的车道线之外的车道线作为所述第三车道线集合中的车道线。此外，在确定第一车道线集合与第二车道线集合中匹配的车道线的过程中，若第三车道线集合包括的车道线数目等于预设车道线数目，则可以停止继续匹配第一车道线集合中的车道线和第二车道线集合中的车道线。因此，可以根据实际需要确定第三车道线集合包括的车道线数目。
在一种可能的设计中，还包括：确定所述第三车道线集合包括的车道线数目小于预设车道线数目时，将所述第一车道线集合和/或所述第二车道线集合中除所述第三车道线集合已包括的车道线之外的车道线作为所述第三车道线集合中的车道线，使得所述第三车道线集合包括的车道线数目等于所述预设车道线数目。因此，可以根据实际需要使第三车道线集合包括的车道线数目等于预设车道线数目。
第二方面，本申请实施例提供一种车道线检测装置，该装置包括：获取模块，用于获取第一图像，所述第一图像为环境图像；处理模块，用于根据所述第一图像获取第一车道线集合和第二车道线集合，其中，所述第一车道线集合和所述第二车道线集合的获取方式不同；根据所述第一车道线集合和所述第二车道线集合获取第三车道线集合。
在一种可能的设计中，所述处理模块用于：在根据所述第一图像获取第一车道线集合和第二车道线集合时，根据所述第一图像确定第二图像，所述第二图像用于确定所述第二车道线集合。
在一种可能的设计中，所述处理模块用于：根据所述第一图像和所述第二图像确定第一损失值，所述第一损失值指示所述第一图像和所述第二图像的相似度。
在一种可能的设计中，所述处理模块用于：在根据第一图像确定第二图像时，根据所述第一图像采用第一模型确定所述第二图像；所述第一模型是根据原始图像和模板图像训练获得的，其中，所述模板图像与所述原始图像关联，所述模板图像的车道线检测难度低于所述原始图像的车道线检测难度；所述第一模型使第一损失函数满足第一预设条件，所述第一损失函数用于指示基于所述原始图像确定的预测模板图像与所述模板图像的差值。
在一种可能的设计中，所述处理模块用于：在根据所述第一图像确定第一车道线集合时，根据所述第一图像采用第二模型确定所述第一车道线检测集合，所述第二模型是根据原始图像和模板图像训练获得的，其中，所述模板图像与所述原始图像关联，所述模板图像的车道线检测难度低于所述原始图像的车道线检测难度；其中，所述第二模型使第二损 失函数、第三损失函数和第四损失函数中的至少两个损失函数满足对应的预设条件；其中，所述第二损失函数用于指示基于所述原始图像确定的预测车道线与所述原始图像对应的真实车道线的差值，所述第三损失函数用于指示第一特征与第二特征的差值，所述第四损失函数用于指示所述第一特征是否被鉴别为所述第二特征，所述第一特征为基于所述原始图像获取的特征，所述第二特征为基于所述模板图像获取的特征，所述第一特征与所述第二特征为相同类型的特征。
在一种可能的设计中，所述处理模块用于：在根据所述第一图像确定第二图像和第一车道线集合时，根据所述第一图像采用第三模型确定所述第二图像和所述第一车道线集合，所述第三模型是根据原始图像和模板图像训练获得的，其中，所述模板图像与所述原始图像关联，所述模板图像的车道线检测难度低于所述原始图像的车道线检测难度；其中，所述第三模型使第一损失函数满足第一预设条件，以及使第二损失函数满足第二预设条件，所述第一损失函数用于指示基于所述原始图像确定的预测模板图像与所述模板图像的差值，所述第二损失函数用于指示基于所述原始图像确定的预测车道线与所述原始图像对应的真实车道线的差值。
在一种可能的设计中，所述第三模型使第一损失函数满足第一预设条件，使第二损失函数满足第二预设条件，以及所述第三模型使第三损失函数满足第三预设条件和/或使第四损失函数满足第四预设条件，其中，所述第三损失函数用于指示第一特征与第二特征的差值，所述第四损失函数用于指示所述第一特征是否被鉴别为所述第二特征，所述第一特征为基于所述原始图像获取的特征，所述第二特征为基于所述模板图像获取的特征，所述第一特征与所述第二特征为相同类型的特征。
在一种可能的设计中，所述第一损失值是根据所述第一图像和所述第二图像采用所述第三损失函数获得的；或者，所述第一损失值是根据所述第一图像和所述第二图像采用所述第四损失函数获得的；或者，所述第一损失值是根据第二损失值和第三损失值确定的，所述第二损失值是根据所述第一图像和所述第二图像采用所述第三损失函数获得的，所述第三损失值是根据所述第一图像和所述第二图像采用所述第四损失函数获得的。
在一种可能的设计中，所述第三车道线集合包括所述第一车道线集合和所述第二车道线集合中至少一条相同的车道线。
在一种可能的设计中，所述处理模块用于：确定所述第三车道线集合包括的车道线数目小于预设车道线数目时，将所述第一车道线集合和/或所述第二车道线集合中除所述第三车道线集合已包括的车道线之外的车道线作为所述第三车道线集合中的车道线。
在一种可能的设计中，所述处理模块用于：确定所述第三车道线集合包括的车道线数目小于预设车道线数目时，将所述第一车道线集合和/或所述第二车道线集合中除所述第三车道线集合已包括的车道线之外的车道线作为所述第三车道线集合中的车道线，使得所述第三车道线集合包括的车道线数目等于所述预设车道线数目。
第三方面，本申请还提供一种装置。该装置可以执行上述方法设计。该装置可以是能够执行上述方法对应的功能的芯片或电路，或者是包括该芯片或电路的设备。
在一种可能的实现方式中，该装置包括：存储器，用于存储计算机可执行程序代码；以及处理器，处理器与存储器耦合。其中存储器所存储的程序代码包括指令，当处理器执行所述指令时，使该装置或者安装有该装置的设备执行上述第一方面或第一方面的任意一种可能的设计中的方法。
其中，该装置还可以包括通信接口，该通信接口可以是收发器，或者，如果该装置为芯片或电路，则通信接口可以是该芯片的输入/输出接口，例如输入/输出管脚等。
在一种可能的设计中，该装置包括相应的功能单元，分别用于实现以上方法中的步骤。功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。硬件或软件包括一个或多个与上述功能相对应的单元。
第四方面，本申请实施例提供一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，当所述计算机程序在装置上运行时，执行如第一方面或第一方面的任意一种可能的设计中的方法。
第五方面，本申请实施例提供一种计算机程序产品，所述计算机程序产品包括计算机程序，当所述计算机程序在装置上运行时，执行如第一方面或第一方面的任意一种可能的设计中的方法。
附图说明
图1为本申请实施例提供的车道线检测场景的示意图；
图2为本申请实施例提供的车辆***的示意图；
图3为本申请实施例提供的车道线检测方法的概述流程图；
图4为本申请实施例提供的第一图像和第二图像的示意图；
图5为本申请实施例提供的第一特征和第二特征的处理示意图；
图6为本申请实施例提供的第一车道线集合和第二车道线集合的示意图；
图7为本申请实施例提供的装置结构示意图之一；
图8为本申请实施例提供的装置结构示意图之二。
具体实施方式
下面将结合附图对本申请实施例作进一步地详细描述。
本申请实施例可以应用于车辆***，示例性地，如图2中的车辆100。
车辆100的一种示例性功能框图。在一个实施例中，车辆100可以配置为完全或部分地自动驾驶模式。例如，车辆100可以在处于自动驾驶模式中同时控制自身，并且可以通过人为操作来确定车辆及其周边环境的当前状态，确定周边环境中的至少一个其他车辆的可能行为，并确定该其他车辆执行可能行为的可能性相对应的置信水平，并基于所确定的信息来控制车辆100。在车辆100处于自动驾驶模式中时，可以将车辆100置为在没有和人交互的情况下操作。
如图2所示，耦合到车辆100或包括在车辆100中的组件可以包括推进***110、传感器***120、控制***130、***设备140、电源150、计算机***160以及用户接口170。车辆100的组件可以被配置为以与彼此互连和/或与耦合到各***的其它组件互连的方式工作。例如，电源150可以向车辆100的所有组件提供电力。计算机***160可以被配置为从推进***110、传感器***120、控制***130和***设备140接收数据并对它们进行控制。计算机***160还可以被配置为在用户接口170上生成图像的显示并从用户接口170接收输入。
需要说明的是，在其它示例中，车辆100可以包括更多、更少或不同的***，并且每 个***可以包括更多、更少或不同的组件。此外，示出的***和组件可以按任意种的方式进行组合或划分，本申请对此不做具体限定。
推进***110可以为车辆100提供动力运动。如图2所示，推进***110可以包括引擎/发动机114、能量源113、传动装置(transmission)112和车轮/轮胎111。另外，推进***110可以额外地或可替换地包括除了图2所示出的组件以外的其他组件。本申请对此不做具体限定。
传感器***120可以包括用于感测关于车辆100所位于的环境的信息的若干个传感器。如图2所示，传感器***120的传感器包括全球定位***(Global PositioningSystem，GPS)126、惯性测量单元(Inertial Measurement Unit，IMU)125、激光雷达传感器、相机传感器123、毫米波雷达传感器以及用于修改传感器的位置和/或朝向的制动器121。毫米波雷达传感器可利用无线电信号来感测车辆100的周边环境内的目标。在一些实施例中，除了感测目标以外，毫米波雷达124还可用于感测目标的速度和/或前进方向。激光雷达122可利用激光来感测车辆100所位于的环境中的目标。在一些实施例中，激光雷达122可包括一个或多个激光源、激光扫描器以及一个或多个检测器，以及其他***组件。相机传感器123可用于捕捉车辆100的周边环境的多个图像。相机传感器123可以是静态相机或视频相机。
GPS 126可以为用于估计车辆100的地理位置的任何传感器。为此，GPS 126可以包括收发器，基于卫星定位数据估计车辆100相对于地球的位置。在示例中，计算机***160可以用于结合地图数据使用GPS 126来估计车辆100行驶的道路。IMU 125可以用于基于惯性加速度及其任意组合来感测车辆100的位置和朝向变化。在一些示例中，IMU125中传感器的组合可包括例如加速度计和陀螺仪。另外，IMU 125中传感器的其它组合也是可能的。
传感器***120还可包括被监视车辆100的内部***的传感器(例如，车内空气质量监测器、燃油量表、机油温度表等)。来自这些传感器中的一个或多个的传感器数据可用于检测对象及其相应特性(位置、形状、方向、速度等)。这种检测和识别是车辆100的安全操作的关键功能。传感器***120还可以包括其它传感器。本申请对此不做具体限定。
控制***130为控制车辆100及其组件的操作。控制***130可包括各种元件，其中包括转向单元136、油门135、制动单元134、传感器融合算法133、计算机视觉***132、路线控制***131以及障碍物避免***137。转向单元136可操作来调整车辆100的前进方向。例如在一个实施例中可以为方向盘***。油门135用于控制引擎114的操作速度并进而控制车辆100的速度。控制***130可以额外地或可替换地包括除了图2所示出的组件以外的其他组件。本申请对此不做具体限定。
制动单元134用于控制车辆100减速。制动单元134可使用摩擦力来减慢车轮111。在其他实施例中，制动单元134可将车轮111的动能转换为电流。制动单元134也可采取其他形式来减慢车轮111转速从而控制车辆100的速度。计算机视觉***132可以操作来处理和分析由相机传感器123捕捉的图像以便识别车辆100周边环境中的目标和/或特征。所述目标和/或特征可包括交通信号、道路边界和障碍物。计算机视觉***132可使用目标识别算法、运动中恢复结构(structure from motion，SFM)算法、视频跟踪和其他计算机视觉技术。在一些实施例中，计算机视觉***132可以用于为环境绘制地图、跟踪目标、估计目标的速度等等。路线控制***131用于确定车辆100的行驶路线。在一些实施例中，路线控制***131可结合来自传感器***120、GPS 126和一个或多个预定地图的数据以 为车辆100确定行驶路线。障碍物避免***137用于识别、评估和避免或者以其他方式越过车辆100的环境中的潜在障碍物。当然，在一个实例中，控制***130可以增加或替换地包括除了所示出和描述的那些以外的组件。或者也可以减少一部分上述示出的组件。
***设备140可以被配置为允许车辆100与外部传感器、其它车辆和/或用户交互。为此，***设备140可以包括例如无线通信***144、触摸屏143、麦克风142和/或扬声器141。***设备140可以额外地或可替换地包括除了图2所示出的组件以外的其他组件。本申请对此不做具体限定。
在一些实施例中，***设备140提供车辆100的用户与用户接口170交互的手段。例如，触摸屏143可向车辆100的用户提供信息。用户接口170还可操作触摸屏143来接收用户的输入。在其他情况中，***设备140可提供用于车辆100与位于车内的其它设备通信的手段。例如，麦克风142可从车辆100的用户接收音频(例如，语音命令或其他音频输入)。类似地，扬声器141可向车辆100的用户输出音频。
无线通信***144可以直接地或者经由通信网络来与一个或多个设备无线通信。例如，无线通信***144可使用3G蜂窝通信，例如码分多址(code division multiple access，CDMA)、EVD0、全球移动通信***(global system for mobile communications，GSM)/通用分组无线服务技术(general packet radio service，GPRS)，或者4G蜂窝通信，例如长期演进(long term evolution，LTE)，或者5G蜂窝通信。无线通信***144可利用WiFi与无线局域网(wireless local area network，WLAN)通信。在一些实施例中，无线通信***144可利用红外链路、蓝牙或ZigBee与设备直接通信。其他无线协议，例如各种车辆通信***，例如，无线通信***144可包括一个或多个专用短程通信(dedicated short range communications，DSRC)设备，这些设备可包括车辆和/或路边台站之间的公共和/或私有数据通信。
电源150可以被配置为向车辆100的一些或全部组件提供电力。为此，电源150可以包括例如可再充电锂离子或铅酸电池。在一些示例中，一个或多个电池组可被配置为提供电力。其它电源材料和配置也是可能的。在一些示例中，电源150和能量源113可以一起实现，如一些全电动车中那样。车辆100的组件可以被配置为以与在其各自的***内部和/或外部的其它组件互连的方式工作。为此，车辆100的组件和***可以通过***总线、网络和/或其它连接机制通信地链接在一起。
车辆100的部分或所有功能受计算机***160控制。计算机***160可包括收发器162和至少一个处理器161。收发器162可以为独立的接收器、独立的发射器、集成收发功能的收发器、或者是接口电路。处理器161执行存储在例如存储器163这样的非暂态计算机可读介质中的指令1631。计算机***160还可以是采用分布式方式控制车辆100的个体组件或子***的多个计算设备。
处理器161可以是任何常规的处理器，诸如商业可获得的中央处理器(central processing unit，CPU)。替选地，该处理器可以是诸如专用集成电路(application specific integrated circuits,ASIC)或其它基于硬件的处理器的专用设备。尽管图2功能性地图示了处理器、存储器、和在相同块中的计算机***160的其它元件，但是本领域的普通技术人员应该理解该处理器、计算机、或存储器实际上可以包括可以或者可以不存储在相同的物理外壳内的多个处理器、计算机、或存储器。例如，存储器可以是硬盘驱动器或位于不同于计算机***160的外壳内的其它存储介质。因此，对处理器或计算机的引用将被理解为包括对可以或者可以不并行操作的处理器或计算机或存储器的集合的引用。不同于使用单一的处 理器来执行此处所描述的步骤，诸如转向组件和减速组件的一些组件每个都可以具有其自己的处理器，所述处理器只执行与特定于组件的功能相关的计算。
在此处所描述的各个方面中，处理器可以位于远离该车辆并且与该车辆进行无线通信。在其它方面中，此处所描述的过程中的一些在布置于车辆内的处理器上执行而其它则由远程处理器执行，包括采取执行单一操纵的必要步骤。
在一些实施例中，存储器163可包含指令1631(例如，程序逻辑)，指令1631可被处理器161执行来执行车辆100的各种功能，包括以上描述的那些功能。存储器214也可包含额外的指令，包括向推进***110、传感器***120、控制***130和***设备140中的一个或多个发送数据、从其接收数据、与其交互和/或对其进行控制的指令。
除了指令1631以外，存储器163还可存储数据，例如道路地图、路线信息，车辆的位置、方向、速度以及其它这样的车辆数据，以及其他信息。这种信息可在车辆100在自主、半自主和/或手动模式中操作期间被车辆100和计算机***160使用。
用户接口170，用于向车辆100的用户提供信息或从其接收信息。可选地，用户接口170可包括在***设备140的集合内的一个或多个输入/输出设备，例如无线通信***144、触摸屏143、麦克风142和扬声器141。
计算机***160可基于从各种子***(例如，推进***110、传感器***120和控制***130)以及从用户接口170接收的输入来控制车辆100的功能。例如，计算机***160可利用来自控制***130的输入以便控制转向单元136来避免由传感器***120和障碍物避免***137检测到的障碍物。在一些实施例中，计算机***160可操作来对车辆100及其子***的许多方面提供控制。
可选地，上述这些组件中的一个或多个可与车辆100分开安装或关联。例如，存储器163可以部分或完全地与车辆100分开存在。上述组件可以按有线和/或无线方式来通信地耦合在一起。
可选地，上述组件只是一个示例，实际应用中，上述各个模块中的组件有可能根据实际需要增添或者删除，图2不应理解为对本申请实施例的限制。
上述车辆100可以为轿车、卡车、摩托车、公共汽车、船、飞机、直升飞机、割草机、娱乐车、游乐场车辆、施工设备、电车、高尔夫球车、火车、和手推车等，本申请实施例不做特别的限定。
目前，在车道线检测的执行过程中，车道线检测结果可以基于当前的车道线图像数据(即针对当前场景采集的图像数据)和车道线模板数据确定，其中，车道线模板数据是前帧的车道线检测结果。因此，车道线检测结果依赖于前帧的车道线检测结果。基于前帧的车道线检测结果可以针对当前的车道线图像数据得到的车道线检测结果进行纠正。例如，当车道线图像数据指示某个车道线被障碍物遮挡，而前帧的车道线检测结果指示该车道线未被障碍物遮挡时，可以根据前帧的车道线检测结果对该车道线检测结果进行修正。但是，如果存在较长时间的压线遮挡或受光照影响或车道线模糊的情况，则很可能会导致车道线模板数据无法发挥修正作用，从而无法获得较好的车道线检测结果。同时，随着基于深度学习的处理方法越来越成为主流，如何在车道线被遮挡、天气情况差等复杂情况下构建合适的模型也成为了一个热点。
基于此，本申请实施例提供一种车道线检测方法，用以提升在复杂场景下的车道线检 测效果。示例性地，以下方法的执行主体可以为车辆100中的处理器161，或者车辆100中的一个芯片。如图3所示，仅以执行主体为处理器为例进行说明。
步骤300：处理器获取第一图像，第一图像为环境图像。
示例性地，处理器可以通过通信接口或通信接口相关电路组件获取来自摄像头的第一图像。
示例性地，第一图像可以为车辆所在驾驶环境的图像，或者第一图像还可以是车辆静止状态下周围环境的图像。
例如，第一图像可以由上述车辆100中的相机传感器123拍摄，处理器161通过通信接口或通信接口相关电路组件获取来自于相机传感器123的第一图像。例如，第一图像可以为图1所示的1-9。
步骤310：处理器根据第一图像确定第一车道线集合和第二车道线集合，其中，其中，第一车道线集合和第二车道线集合的获取方式不同。
在一些实施例中，处理器根据第一图像确定第一车道线集合，根据第一图像确定第二图像，根据第二图像确定第一车道线集合。其中，第一车道线集合包括至少一条车道线，第一车道线集合是基于第一图像确定的车道线检测结果。第二车道线集合包括至少一条车道线，第二车道线集合是基于第二图像确定的车道线检测结果。
其中，第二图像又可称为第一图像的模板图像。第二图像与第一图像相似度较高，例如大于预设相似度阈值。第一图像和第二图像的相似度可以通过通用图像相似度算法或人工方式等确定，此处不再赘述。第二图像的车道线检测难度低于第一图像的车道线检测难度。例如，若第一图像中的车道线被遮挡，第二图像中的车道线未被遮挡，则第二图像的车道线检测难度低于第一图像的车道线检测难度。如图4所示，图4左侧为第一图像。图4右侧为第二图像。又例如，若第一图像所示场景的光线较弱，第二图像所示场景的光线较好，则第二图像的车道线检测难度低于第一图像的车道线检测难度。
示例性地，每个图像的车道线检测难度可以通过该图像对应的车道线的置信度或人工方式等确定。其中，车道线的置信度可以用于表征车道线检测难度。其中，车道线的置信度越大，则车道线的检测难度越低，车道线的检测效果越好。例如，车道线的置信度可以由预测正确的车道线数目的概率与预测正确的车道线所对应的概率的平均值之和构成。其中，一些通用车道线检测算法均可以输出预测出的车道线的数目，以及预测出的车道线中的每个车道线为正确车道线的概率。例如，可以通过人工方法确定采用通用车道线检测算法输出的预测出的车道线中哪些车道线为正确的车道线，进一步从通用车道线检测算法输出的预测出的车道线中的每个车道线为正确车道线的概率中筛选出哪些概率是预测正确的车道线所对应的概率。具体的，预测正确的车道线数目为n，预测出的车道线数目为m，预测车道线数目正确的概率用n/m表示，n，m为大于等于1的正整数，上述预测正确的n个车道线中的每个车道线为正确车道线的概率分别为P1，P2，…，Pn，预测正确的车道线所对应的概率的平均值为(P1+P2+…+Pn)/n＝P’，则车道线的置信度为S1*n/m+S2*P’，其中，S1和S2为预设系数。
针对步骤310，处理器根据第一图像确定第一车道线集合和第二车道线集合可以采用但不限于以下方案：
方案1：处理器根据第一图像采用第一模型确定第二图像，根据第一图像采用第二模型确定第一车道线集合，根据第二图像采用第二模型确定第二车道线集合。
示例性地，第一模型用于基于图像预测相应的模板图像，第二模型用于基于图像预测该图像中包括的车道线。
其中，第一模型是根据原始图像和模板图像训练获得的，其中，模板图像与原始图像关联，模板图像的车道线检测难度低于原始图像的车道线检测难度。其中，第一模型使第一损失函数满足第一预设条件，第一损失函数用于指示基于原始图像确定的预测模板图像与模板图像的差值。
第二模型是根据原始图像和模板图像训练获得的，其中，模板图像与原始图像关联，模板图像与原始图像的相似度较高，模板图像的车道线检测难度低于原始图像的车道线检测难度。其中，第二模型使第二损失函数、第三损失函数和第四损失函数中的至少两个损失函数满足对应的预设条件。示例性地，第二模型使第二损失函数满足第二预设条件，以及使第三损失函数满足第三预设条件。或者第二模型使第二损失函数满足第二预设条件，以及使第四损失函数满足第四预设条件。或者第二模型使第二损失函数满足第二预设条件、使第三损失函数满足第三预设条件以及使第四损失函数满足第四预设条件。
又或者，第二模型使第二损失函数与第三损失函数的加权和满足第五预设条件。或者第二模型使第二损失函数与第四损失函数的加权和满足第六预设条件。或者第二模型使第二损失函数、第三损失函数和第四损失函数的加权和满足第七预设条件。或者第二模型使第二损失函数满足第二预设条件，使第三损失函数和第四损失函数的加权和满足第八预设条件。
其中，第二损失函数用于指示基于原始图像确定的预测车道线与原始图像对应的真实车道线的差值，第三损失函数用于指示第一特征与第二特征的差值，第四损失函数用于指示第一特征是否被鉴别为第二特征，第一特征为基于原始图像获取的特征，第二特征为基于模板图像获取的特征，第一特征与第二特征为相同类型的特征。示例性地，第一特征可以是指从原始图像中直接提取的特征，或者基于原始图像生成的特征。第二特征可以是指从模板图像直接提取的特征，或者基于模板图像生成的特征。
方案2：处理器根据第一图像采用第三模型确定第二图像和第一车道线集合，根据第二图像采用第三模型确定第二车道线集合。
示例性地，第三模型用于基于图像预测相应的模板图像和该图像中包括的车道线。其中，第三模型是根据原始图像和模板图像训练获得的，其中，模板图像与原始图像关联，模板图像与原始图像的相似度较高，模板图像的车道线检测难度低于原始图像的车道线检测难度。
在一些实施例中，第三模型使第一损失函数满足第一预设条件，以及使第二损失函数满足第二预设条件。
在一些实施例中，第三模型使第一损失函数满足第一预设条件，使第二损失函数满足第二预设条件，以及第三模型使第三损失函数满足第三预设条件。或者，第三模型使第一损失函数满足第一预设条件，使第二损失函数满足第二预设条件，以及第三模型使第四损失函数满足第四预设条件。或者，第三模型使第一损失函数满足第一预设条件，使第二损失函数满足第二预设条件，第三模型使第三损失函数满足第三预设条件，以及第三模型使第四损失函数满足第四预设条件。
在一些实施例中，第三模型使第一损失函数满足第一预设条件和第二损失函数的加权和满足第九预设条件。或者，第三模型使第一损失函数、第二损失函数、第三损失函数和 第四损失函数的加权和满足第十预设条件。或者，第三模型使第一损失函数与第二损失函数的加权和满足第十一预设条件，以及使第三损失函数和第四损失函数的加权和满足第八预设条件。或者，第三模型使第一损失函数满足第一预设条件，以及使第二损失函数满足第二预设条件以及使第三损失函数和第四损失函数的加权和满足第八预设条件。
第一损失函数用于指示基于原始图像确定的预测模板图像与模板图像的差值，第二损失函数用于指示基于原始图像确定的预测车道线与原始图像对应的真实车道线的差值。第三损失函数用于指示第一特征与第二特征的差值，第四损失函数用于指示第一特征是否被鉴别为第二特征。第一特征为基于原始图像获取的特征，第二特征为基于模板图像获取的特征，第一特征与第二特征为相同类型的特征。示例性地，第一特征可以是指从原始图像中直接提取的特征，或者基于原始图像生成的特征。第二特征可以是指从模板图像直接提取的特征，或者基于模板图像生成的特征。
针对上述方案1和方案2需要说明以下几点：
第一，针对上述第一模型、第二模型和第三模型所涉及的训练数据可以相同，也可以不同。其中，训练数据可以包括原始图像和模板图像。其中，原始图像是指收集环境图像。例如，原始图像是指车辆所处的环境图像，这里的车辆可以是指使用上述模型的指定车辆(例如车辆100)或者任意车辆。模板图像与原始图像关联，其中，模板图像与原始图像的相似度较高，模板图像的车道线检测难度低于原始图像的车道线检测难度。
其中，模板图像与原始图像的相似度可以通过通用图像相似度算法或人工方式等确定，此处不再赘述。每个图像的车道线检测难度可以通过该图像对应的车道线的置信度或人工方式等确定。具体可以参阅上述关于车道线的置信度的相关描述。示例性地，对于有标签的原始图像(或模板图像)，可以通过标签确定采用通用车道线检测算法输出的预测出的车道线中哪些车道线为正确的车道线，得到预测出的车道线中预测正确的车道线数目，还可以进一步从通用车道线检测算法输出的预测出的车道线中的每个车道线为正确车道线的概率中筛选出哪些概率是预测正确的车道线所对应的概率，计算原始图像(或模板图像)对应的车道线的置信度。这里的标签用于标识原始图像(或模板图像)中的车道线。对于无标签的原始图像(或模板图像)，可以通过人工方法确定采用通用车道线检测算法输出的预测出的车道线中哪些车道线为正确的车道线，进一步从通用车道线检测算法输出的预测出的车道线中的每个车道线为正确车道线的概率中筛选出哪些概率是预测正确的车道线所对应的概率，计算原始图像(或模板图像)对应的车道线的置信度。
其中，模板图像可以采用但不限于以下方法获得：
方式1：利用原始图像生成模板图像。
示例性地，生成模板图像可以采用但不限于以下方式：利用图像增强消除原始图像中的影响生成模板图像，或者利用图像分割去除原始图像中的干扰对象生成模板图像，或者利用生成对抗生成相似图像作为模板图像等。
方式2：通过人工方式从同一地理区域的多个原始图像中筛选出模板图像。示例性地，结合一个原始图像的全球定位***(global positioning syetem，GPS)信息，可以搜索与该GPS信息有关的图像，获得同一地理区域的多个原始图像。这里的地理区域可以是指针对某个地理标志建筑(例如，商城或银行)所确定的区域。
例如，同一地理区域的多个原始图像包括N个图像，N为大于2的正整数，可以通过人工方式从N个图像中筛选出一个模板图像，例如可以选择障碍物较少的图像，或光线适 合的图像。
方式3：结合一个原始图像的GPS信息，可以搜索与该GPS信息有关的图像作为待选图像，进一步地选择与该原始图像的相似度较高且车道线检测难度最低的待选图像作为与该原始图像关联的模板图像。
可以理解的是，方式2和方式3中所指的GPS信息还可以替换为其他传感器信息，本申请实施例对此不做限定。
方式4：利用图像检索可以获得大量与该原始图像相似的图像作为待选图像，并进一步选择与该原始图像相似度较高且车道线检测难度最低的待选图像作为与该原始图像关联的模板图像。
针对上述方式3和方式4，可以计算每个待选图像对应的车道线的置信度，以及每个待选图像与原始图像的相似度，选择置信度最高且相似度大于预设相似度阈值的待选图像作为模板图像。或者，选择置信度大于预设置信度阈值且相似度大于预设相似度阈值的待选图像作为模板图像。
此外，还可以采用其他方式确定与原始图像关联的模板图像，且上述方式1至方式4还可以结合使用，例如，方式1与方式2或方式3或方式4结合，本申请对此不做限定。
第二，第一损失函数用于指示基于原始图像确定的预测模板图像与模板图像的差值。示例性地，预测模板图像和模板图像的差值可以通过预测模板图像所对应的矩阵和模板图像所对应的矩阵之间的差值确定。
第一损失函数又可称为模板图像预测损失函数，其中，预测模板图像可以用f template(image)，模板图像可以用template gt表示，则第一损失函数可以表示为：
f template(image)-template gt
或者，第一损失函数可以表示为：
min(f template(image)-template gt)
第三，第二损失函数用于指示基于原始图像确定的预测车道线与原始图像对应的真实车道线的差值。示例性地，预测车道线和原始图像对应的真实车道线的差值可以通过预测的车道线所对应的矩阵和原始图像对应的真实车道线所对应的矩阵之间的差值确定。
第二损失函数又可称为车道线检测损失函数，其中，预测车道线可以用f lane(image)表示，原始图像对应的真实车道线可以用lane gt表示，则第二损失函数可以表示为：
f template(image)-template gt
或者，第二损失函数可以表示为：
min(f template(image)-template gt)
第四，第三损失函数和第四损失函数。
首先，需要首先基于原始图像获取的特征(即第一特征)，以及基于模板图像获取的特征(即第二特征)，第一特征与第二特征为相同类型的特征。示例性地，第一特征可以是指从原始图像中直接提取的特征，或者基于原始图像生成的特征。例如，G表示生成器，G(image)表示根据原始图像生成新的图像，可以从G(image)中提取特征作为第一特征。第二特征可以是指从模板图像直接提取的特征，或者基于模板图像生成的特征。例如，这里的特征可以为颜色特征或纹理特征等。
第三损失函数又可称为相似度量损失函数：第一特征可以用feature original表示，第 二特征可以用feature template表示，则第三损失函数可以表示为：
cos(feature original,feature template)，
或者，第三损失函数可以表示为：
min(cos(feature original,feature template))
第四损失函数又可称为鉴别器损失函数，鉴别器又可称为二分类器，鉴别器是一种分类是否的算法。第四损失函数可以表示为：
f(logD(feature template))+f(1-logD(G(image)))
或者，第四损失函数可以表示为：
Figure PCTCN2021083552-appb-000001
其中，上述第四损失函数可以理解为最小化生成器基于原始图像生成的特征与基于模板图像生成的特征的差别，同时最大化通过鉴别器鉴别出的生成器基于原始图像生成的特征与基于模板图像生成的特征不同。
其中，D表示鉴别器，D用于判断基于G(image)生成的特征(即第一特征)是否为feature template(第二特征)。
通过第三损失函数和第四损失函数可以实现在模型训练过程中约束基于原始图像获取的特征和基于模板图像获取的特征具有较好的一致性，进而可以提升第二模型和第三模型的预测效果。
进一步地，将第一特征和第二特征输入至第三损失函数，以及将第一特征和第二特征输入至第四损失函数，如图5所示。其中，第三损失函数输出第二损失值，第四损失函数输出第三损失值，根据第一损失值和第二损失值可以计算第一损失值。示例性地，第一损失值可以是第二损失值和第三损失值的加权和。例如，第一损失值＝K1*第二损失值+K2*第三损失值，K1＞0，K2＞0。例如，K1+K2＝1。
由于训练数据中可能存在异常样本，对异常样本的过度关注容易造成过拟合，为此对于异常样本本申请使用较宽松的约束，对于普通样本使用较严格的约束。示例性地，定义T lower与T high，其中，T lower＜T high，样本的异常程度通过第三损失函数输出的第二损失值loss s来判断，当loss s<T lower,则降低第四损失函数输出的第三损失值的权重，当loss s>T high,则降低loss s的权重，甚至可以不使用loss s。
因此，在loss s大于T high时，第一损失值＝K3*loss s+K4*第三损失值，K3≥0，K4＞0，K1＞K3，例如，K3＝0，K4＝1，K3+K4＝1。在loss s小于T lower时，第一损失值＝K5*loss s+K6*第三损失值，K5＞0，K6＞0，K2＞K6，K5+K6＝1。
以下对第一模型，第二模型和第三模型的训练过程进行简要说明。
本申请实施例所涉及的第一模型、第二模型和第三模型可以采用上述训练数据通过自动编码器或其他神经网络***获得，本申请实施例不限定训练模型采用的具体方式。
示例性地，一般的神经网络***包括输入层、隐藏层和输出层，相应的包括三种神经元(神经网络的基本单元)：输入层神经元、隐藏层神经元和输出层神经元。其中，输入层神经元从外部世界获取信息，隐藏层神经元和外部世界没有直接联系，隐藏层神经元执行预设函数的计算，并将信息从输入层神经元传递到输出层神经元。输出层神经元用于向外部世界传递信息。
示例性地，自编码器是一个3层或大于3层的神经网络，自编码器属于无监督学习，一般的自动编码可以看作由两部分组成：编码器和解码器，其中，编码器用于将输入压缩成潜在空间表征，可以用编码函数h＝f(x)表示。解码器用于重构来自潜在空间表征的输入，可以用解器码函数r＝g(h)表示。自动编码器可以被用于降维或特征学习。
自编码网络器的作用在于将输入样本压缩到隐藏层，再在输出端重建样本，其中，隐藏层神经元数目小于输入层神经元数目，输出层神经元数目等于输入层神经元数目，例如，自编码器可将n维数据压缩成m维数据，n＝输入层神经元数目＝输出层神经元数目，m＝隐藏层神经元数目，n＞m，n和m为正整数，然后再用损失尽量小的方式将n维数据恢复出来。
示例1：以下对第三模型的训练过程进行简要说明。
步骤1：获取训练数据。训练数据可以包括原始图像和模板图像。具体可以参阅上述关于训练数据的相关内容，重复之处不再赘述。
步骤2：将原始图像和模板图像输入至预设***，经过预设***中隐藏层神经元对应的预设函数进行计算，输出预测车道线和预测模板图像。这里的预设***可以是指上述介绍的神经网络***或自动编码器等，本申请实施例对此不做限定。
步骤3：根据第一损失函数计算预测模板图像和模板图像的差值(记为第一差值)，根据第二损失函数计算预测车道线和原始图像对应的真实车道线的差值(记为第二差值)。
步骤4：多次调整预设***中隐藏层神经元对应的预设函数的参数，使第一损失函数计算得到的第一差值满足第一预设条件，使第二损失函数计算得到的第二差值满足第二预设条件，停止训练过程，获得第三模型。
在一些实施例中，多次调整预设***中隐藏层神经元对应的预设函数的参数，直至第一差值在第一预设误差范围内且第二差值在第二预设误差范围内。其中，第一预设误差范围和第二预设误差范围可以根据经验值设定。
在一些实施例中，多次调整预设***中隐藏层神经元对应的预设函数的参数，直至第一差值最小且第二差值在第二预设误差范围内。其中，第二预设误差范围可以根据经验值设定。
在一些实施例中，多次调整预设***中隐藏层神经元对应的预设函数的参数，直至第一差值在第一预设误差范围内且第二差值最小。其中，第一预设误差范围可以根据经验值设定。
通过上述过程能够获得具有较好模板图像预测结果和较好车道线检测结果的第三模型。
示例2：以下对第三模型的训练过程进行简要说明。
步骤1：获取训练数据。训练数据可以包括原始图像和模板图像。具体可以参阅上述关于训练数据的相关内容，重复之处不再赘述。
步骤2：将原始图像和模板图像输入至预设***，经过预设***中隐藏层神经元对应的预设函数进行计算，输出预测车道线和预测模板图像。这里的预设***可以是指上述介绍的神经网络***或自动编码器等，本申请实施例对此不做限定。
步骤3：根据第一损失函数计算预测模板图像和模板图像的差值(记为第一差值)，根据第二损失函数计算预测车道线和原始图像对应的真实车道线的差值(记为第二差值)。
步骤4：获取第一特征和第二特征，根据第三损失函数和第四损失函数分别输出第二损失值和第三损失值。
这里步骤4可以在步骤2之后，也可以在步骤2之前，本申请实施例对此不做限定。
步骤5：多次调整预设***中隐藏层神经元对应的预设函数的参数，使第一损失函数计算得到的第一差值满足第一预设条件，使第二损失函数计算得到的第二差值满足第二预 设条件，以及使第三损失函数计算得到的第二损失值满足第三预设条件，使第四损失函数计算得到的第四损失值满足第四预设条件，停止训练过程，获得第三模型。
在一些实施例中，首先，多次调整预设***中隐藏层中一部分神经元对应的预设函数的参数，直至第二损失值在第三预设误差范围内，且第三损失值在第四预设误差范围内。然后，多次调整预设***中隐藏层中另一部分神经元对应的预设函数的参数，直至第一差值在第一预设误差范围内且第二差值在第二预设误差范围内。
在一些实施例中，多次调整预设***中隐藏层神经元对应的预设函数的参数，直至第一差值在第一预设误差范围内，且第二差值最小，且第二损失值在第三预设误差范围内，且第三损失值在第四预设误差范围内。其中，第三预设误差范围和第四预设误差范围可以根据经验值设定。
在一些实施例中，多次调整预设***中隐藏层神经元对应的预设函数的参数，直至第一差值最小，且第二差值在第二预设误差范围内，且第一损失值在第五预设误差范围内，示例性地，第一损失值可以是第二损失值和第三损失值的加权和。其中，第五预设误差范围可以根据经验值设定。
通过上述过程能够获得具有较好模板图像预测结果和较好车道线检测结果的第三模型。
示例3：以下对第一模型的训练过程进行简要说明。
步骤1：获取训练数据。训练数据可以包括原始图像和模板图像。具体可以参阅上述关于训练数据的相关内容，重复之处不再赘述。
步骤2：将原始图像和模板图像输入至预设***，经过预设***中隐藏层各神经元对应的预设函数进行计算，输出预测模板图像。这里的预设***可以是指上述介绍的神经网络***或自动编码器等，本申请实施例对此不做限定。
步骤3：根据第一损失函数计算预测模板图像和模板图像的差值(记为第一差值)。
步骤4：多次调整预设***中隐藏层各神经元对应的预设函数的参数，使第一损失函数计算得到的第一差值满足第一预设条件，停止训练过程，获得第一模型。
在一些实施例中，多次调整预设***中隐藏层各神经元对应的预设函数的参数，直至第一损失函数计算得到的第一差值满足第一预设条件或者最小。其中，第一预设误差范围可以根据经验值设定。
通过上述过程能够获得具有较好模板图像预测结果的第一模型。
示例4：以下对第二模型的训练过程进行简要说明。
步骤1：获取训练数据。第一损失函数计算得到的第一差值满足第一预设条件具体可以参阅上述关于训练数据的相关内容，重复之处不再赘述。
步骤2：将原始图像和模板图像输入至预设***，经过预设***中隐藏层各神经元对应的预设函数进行计算，输出预测的车道线。这里的预设***可以是指上述介绍的神经网络***或自动编码器等，本申请实施例对此不做限定。
步骤3：根据第二损失函数计算预测车道线和原始图像对应的真实车道线的差值(记为第二差值)。
步骤4：获取第一特征和第二特征，根据第三损失函数和第四损失函数分别输出第二损失值和第三损失值。
这里步骤4可以在步骤2之后，也可以在步骤2之前，本申请实施例对此不做限定。
步骤5：多次调整预设***中隐藏层各神经元对应的预设函数的参数，使第二损失函数计算得到的第二差值满足第二预设条件，以及使第三损失函数计算得到的第二损失值满足第三预设条件，使第四损失函数计算得到的第四损失值满足第四预设条件，停止训练过程，获得第二模型。
在一些实施例中，多次调整预设***中隐藏层各神经元对应的预设函数的参数，直至第二差值最小，且第二损失值在第三预设误差范围内，且第三损失值在第四预设误差范围内。其中，第三预设误差范围和第四预设误差范围可以根据经验值设定。
在一些实施例中，多次调整预设***中隐藏层各神经元对应的预设函数的参数，直至第二差值在第二预设误差范围内，且第一损失值在第五预设误差范围内，示例性地，第一损失值可以是第二损失值和第三损失值的加权和。其中，第五预设误差范围可以根据经验值设定。
通过上述过程能够获得具有较好车道线检测结果的第二模型。
步骤320：处理器根据第一车道线集合和第二车道线集合确定第三车道线集合。
在一些实施例中，第三车道线集合可以作为最终的车道线检测结果，并在显示界面(例如车辆100中的触摸屏143)上呈现。
具体的，在处理器根据第一车道线集合和第二车道线集合确定第三车道线集合时，处理器可以确定第一车道线集合中的第一车道线与第二车道线集合中的第二车道线指示环境中的同一条车道线，处理器将第一车道线，或第二车道线，或第三车道线作为第三车道线集合中的车道线。其中，第三车道线是根据第一车道线和第二车道线确定的。
如图6所示，对于第一车道线集合中的车道线1与第二车道线集合中的车道线2，在相同的y值下，车道线1对应的x值与车道线2对应的x值不同，进而可以计算x值的差值，进一步地，每隔预设间隔求取一次车道线1对应的x值与车道线2对应的x值的差值，可以获得车道线1和车道线2的差值集合，进而可以根据该差值集合计算各个差值之和，得到车道线1和车道线2的差值和。同理，可以求得车道线1和车道线3的差值和，以及车道线1与车道线4的差值和，并进一步选择差值和最小的车道线(即车道线2)作为与车道线1匹配的车道线，即车道线1与车道线2为环境中的同一条车道线。
可以理解的是，处理器可以将车道线1或车道线2作为第三车道线集合中的车道线，或者，根据车道线1和车道线2的差值集合中的每个差值的1/2确定一个新的车道线作为第三车道线集合中的车道线。
此外，在一些实施例中，在确定第一车道线集合与第二车道线集合中匹配的车道线的过程中，若第三车道线集合包括的车道线数目等于预设车道线数目，则可以停止继续匹配第一车道线集合中的车道线和第二车道线集合中的车道线。
或者，在确定第一车道线集合与第二车道线集合中匹配的车道线之后，若第三车道线集合包括的车道线数目小于预设车道线数目，假设第一车道线集合中的第一车道线与第二车道线集合中的第二车道线指示环境中的同一条车道线，则处理器将第一车道线集合中除第一车道线和/或第二车道线集合中除第二车道线之外的车道线作为第三车道线集合中的车道线。例如，处理器可以随机选择第一车道线集合中除第一车道线外的车道线作为第三车道线集合中的车道线。或者，处理器可以随机选择第二车道线集合中除第二车道线外的车道线作为第三车道线集合中的车道线。或者，处理器可以随机选择第一车道线集合中除 第一车道线外的车道线作为第三车道线集合中的车道线，以及随机选择第二车道线集合中除第二车道线外的车道线作为第三车道线集合中的车道线。又例如，处理器可以根据第二模型或第三模型确定第一车道线集合中包括的每个车道线为正确车道线的概率，排除第一车道线集合中的第一车道线，选择剩余车道线中车道线为正确车道线的概率最大的车道线作为第三车道线集合中的车道线。或者，处理器可以根据第二模型或第三模型确定第二车道线集合中包括的每个车道为正确车道线的概率，排除第二车道线集合中的第二车道线，选择剩余车道线中车道线为正确车道线的概率最大的车道线作为第三车道线集合中的车道线。或者，处理器可以根据第二模型或第三模型确定第一车道线集合中包括的每个车道为正确车道线的概率以及第二车道线集合中包括的每个车道为正确车道线的概率，排除第一车道线集合中的第一车道线，选择第一车道线集合中剩余车道线中车道线为正确车道线的概率最大的车道线作为第三车道线集合中的车道线，同时，排除第二车道线集合中的第二车道线，选择第二车道线集合中剩余车道线中车道线为正确车道线的概率最大的车道线作为第三车道线集合中的车道线。
进一步地，在确定第三车道线集合包括的车道线数目小于预设车道线数目时，将第一车道线集合中除第一车道线和/或第二车道线集合中除第二车道线之外的车道线作为第三车道线集合中的车道线，需要使得第三车道线集合包括的车道线数目等于第二阈值，其中，第二阈值是根据车辆所处环境确定的。
例如，在跟车的使用场景下，本车需要确定左右两侧的车道线，因此，预设车道线数目为2，当第三车道线集合仅确定一个车道线时，此时需要根据第一车道线集合或第二车道线集合再确定一条车道线作为第三车道线集合中的车道线。例如，处理器已确定第三车道线集合包括一条本车左侧的车道线，进一步地，处理器可以根据第二模型或第三模型确定第一车道线集合中包括的本车右侧的车道线为正确车道线的概率，以及第二车道线集合中包括的本车右侧的车道线为正确车道线的概率，将上述两条车道线中概率较高的车道线作为第三车道线集合中的车道线。
在一些实施例中，基于步骤320可知，在处理器可以根据第二图像采用上述第三模型确定第二车道线集合时，处理器还可以同时获得第三图像，其中，第三图像是第二图像的模板图像，第三图像与第二图像相似，第三图像的车道线检测难度低于第二图像的车道线检测难度。进一步地，处理器可以根据第三图像采用第三模型确定第四车道线集合。
因此，处理器还可以根据第一车道线集合和第四车道线集合确定第三车道线集合。或者，处理器首先基于第二车道线集合和第四车道线集合确定第五车道线集合，根据第一车道线集合和第五车道线集合确定第三车道线集合。其中，基于第二车道线集合和第四车道线集合确定第五车道线集合的方式可以参考上述根据第一车道线集合和第二车道线集合确定第三车道线集合的方式，重复之处不再赘述。
通过上述过程，可以实现在进一步优化的模板图像(即第三图像)的基础上确定一个车道线集合，然后根据该车道线集合和第一车道线集合确定最终的车道线集合，进而可以优化最终获得的车道线检测结果。
此外，基于上述对第三损失函数和第四损失函数的相关说明，在一些实施例中，处理器还可以根据第二损失值和第三损失值确定第一损失值，第一损失值用于表征第二图像和第一图像的相似度，处理器可以在显示界面(例如车辆100中的触摸屏143)上呈现第一 损失值。其中，第二损失值是根据第一图像和第二图像采用第三损失函数获得的，第三损失值是根据第一图像和第二图像采用第四损失函数获得的。
示例性地，第一损失值＝K1*第二损失值+K2*第三损失值，K1＞0，K2＞0。例如，K1+K2＝1。
此外，在第二损失值大于第一阈值(对应于上述T high)时，第一损失值＝K3*第二损失值+K4*第三损失值，K3≥0，K4＞0，K1＞K3。例如，K3＝0，K4＝1。在第二损失值小于第二阈值(对应于上述T lower)时，第一损失值＝K5*第二损失值+K6*第三损失值，K5＞0，K6＞0，K2＞K6，例如，K5+K6＝1，其中，第一阈值大于第二阈值。
上述主要从方法流程的角度对本申请实施例提供的方案进行了介绍。下面结合附图介绍本申请实施例中用来实现上述方法的装置。因此，上文中的内容均可以用于后续实施例中，重复的内容不再赘述。
为了实现上述本申请实施例提供的方法中的各功能，本申请实施例还提供一种装置用于实现上述方法。该装置可以包括硬件结构和/或软件模块，以硬件结构、软件模块、或硬件结构加软件模块的形式来实现上述各功能。上述各功能中的某个功能以硬件结构、软件模块、还是硬件结构加软件模块的方式来执行，取决于技术方案的特定应用和设计约束条件。
本申请实施例提供的装置可以是能够执行上述方法对应的功能的芯片或电路，该芯片或电路可以设置在处理器等设备中。进一步的，本申请实施例提供的装置，还能够以硬件或硬件和计算机软件的结合形式来实现。某个功能究竟以硬件还是计算机软件驱动硬件的方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请实施例的范围。
本申请实施例提供的装置可以进行功能模块的划分，例如，可对应各个功能划分各个功能模块，也可将两个或两个以上的功能集成在一个处理模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。需要说明的是，本申请实施例中对模块的划分是示意性的，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式。
一种可能的实现方式中，如图7所示，为本申请实施例提供一种车道线检测装置的结构示意图。该装置可以是处理器，也可以是处理器中的装置。该装置700可以包括：处理模块71和获取模块72。当然，该装置700还可能包括其他模块，本申请实施例并不限定，仅示出主要的功能模块。
其中，获取模块72，用于获取第一图像，所述第一图像为环境图像。
处理模块71，用于，根据所述第一图像获取第一车道线集合和第二车道线集合，根据所述第一车道线集合和所述第二车道线集合获取第三车道线集合。其中，所述第一车道线集合和所述第二车道线集合的获取方式不同。
应理解，本申请实施例中的处理模块71可以由处理器或处理器相关电路组件实现，获取模块72可以由通信接口或通信接口相关电路组件实现。应理解，通信接口可以包括例如发射器和接收器，处理器、发射器和接收器相互耦合，其中，发射器和接收器例如通过天线、馈线和编解码器等实现，或者，如果所述装置为设置在设备中的芯片，那么发射 器和接收器例如为芯片中的通信接口，该通信接口与设备中的射频收发组件连接，以通过射频收发组件实现信息的收发。
例如，如图8所示为本申请实施例提供的装置800，图8所示的装置可以为图7所示的装置的一种硬件电路的实现方式。该装置可用于执行图3所示出的流程图中的处理器功能。为了便于说明，图8仅示出了该装置的主要部件。
需要说明的是，图8所示的装置可以是能够执行上述方法对应的功能的芯片或电路，也可以是包括上述芯片或电路的设备，本申请实施例对此并不限定。
图8所示的装置800包括至少一个处理器820，用于实现本申请实施例提供的图3中处理器的功能。
装置800还可以包括至少一个存储器830，用于存储程序指令和/或数据。存储器830和处理器820耦合。本申请实施例中的耦合是装置、单元或模块之间的间接耦合或通信连接，可以是电性，机械或其它的形式，用于装置、单元或模块之间的信息交互。处理器820可能和存储器830协同操作。处理器820可能执行存储器830中存储的程序指令。所述至少一个存储器中的至少一个可以包括于处理器中。
可选地，若该装置800为芯片或电路，该装置800也可以不包括存储器830，处理器820可以读取该芯片或电路外部的存储器中的指令(程序或代码)以实现图3所示的实施例所提供的处理器的功能。
装置800还可以包括通信接口810，用于通过传输介质和其它设备进行通信，从而用于装置800中的装置可以和其它设备进行通信。在本申请实施例中，通信接口可以是收发器、电路、总线、模块或其它类型的通信接口。在本申请实施例中，收发器可以为独立的接收器、独立的发射器、集成收发功能的收发器、或者是接口电路。处理器820利用通信接口810收发数据，并用于实现图3所示实施例中处理器的功能，具体可以参考前面的描述，在此不再赘述。
装置800还可以包括通信总线840。其中，通信接口810、处理器820以及存储器830可以通过通信总线840相互连接；通信总线840可以是外设部件互连标准(peripheral component interconnect，简称PCI)总线或扩展工业标准结构(extended industry standard architecture，简称EISA)总线等。所述通信总线840可以分为地址总线、数据总线、控制总线等。为便于表示，图8中仅用一条粗线表示，但并不表示仅有一根总线或一种类型的总线。
再一种可选的方式，本申请实施例提供的装置使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行所述计算机程序指令时，全部或部分地实现本申请实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。所述计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，所述计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线(digital subscriber line，DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可用介质集成的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质，(例如软盘、硬盘、磁带)、光介质(例如DVD)、或者半导体介质(例如固态硬盘(solid  state disk，SSD))等。
需要说明的是，用于执行本申请实施例提供的方法的上述装置中所包含的处理器可以是中央处理器(central processing unit，CPU)，通用处理器、数字信号处理器(digital signal processor，DSP)、专用集成电路(application-specific integrated circuit，ASIC)，现场可编程门阵列(field programmable gate array，FPGA)或者其他可编程逻辑器件、晶体管逻辑器件，硬件部件或者其任意组合。其可以实现或执行结合本申请公开内容所描述的各种示例性的逻辑方框，模块和电路。所述处理器也可以是实现计算功能的组合，例如包含一个或多个微处理器组合，DSP和微处理器的组合等等。
结合本申请实施例所描述的方法或者算法的步骤可以硬件的方式来实现，也可以是由处理器执行软件指令的方式来实现。软件指令可以由相应的软件模块组成，软件模块可以被存放于随机存取存储器(random access memory，RAM)、闪存、只读存储器(read-only memory，ROM)存储器、可擦除可编程只读存储器(erasable programmable read-only memory，EPROM)、电可擦除可编程只读存储器(electrically erasable programmable read-only memory，EEPROM)、寄存器、硬盘、移动硬盘、只读光盘(compact disc read-only memory，CD-ROM)或者本领域熟知的任何其它形式的存储介质中。一种示例性的存储介质耦合至处理器，从而使处理器能够从该存储介质读取信息，且可向该存储介质写入信息。当然，存储介质也可以是处理器的组成部分。处理器和存储介质可以位于ASIC中。另外，该ASIC可以位于雷达装置或者安装雷达装置的探测设备中。当然，处理器和存储介质也可以作为分立组件存在于雷达装置或者安装雷达装置的探测设备中。
可以理解的是，图7～图8仅仅示出了该装置的简化设计。在实际应用中，本申请实施例提供的装置可以包含任意数量的发射器，接收器，处理器，控制器，存储器以及其他可能存在的元件。
通过以上的实施方式的描述，所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将装置的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。
本申请实施例还提供一种芯片，所述芯片与存储器相连，用于读取并执行所述存储器中存储的软件程序，当在所述芯片上运行所述软件程序时，使得所述芯片实现图3中处理器的功能。
本申请实施例还提供一种计算机可读存储介质，包括指令，当在计算机上运行所述指令时，使得计算机实现图3中处理器的功能。
本申请实施例还提供一种终端设备，所述终端设备包括本申请上述实施例提到的车道线检测装置。这里的终端设备可以为机器人，车辆或车载设备等。
本申请实施例还提供一种车辆，所述车辆包括至少一个本申请上述实施例提到的车道线检测装置。
本领域内的技术人员应明白，本申请的实施例可提供为方法、***、或计算机程序产品。因此，本申请可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本申请可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。
本申请是参照根据本申请的方法、设备(***)、和计算机程序产品的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的装置。
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。
显然，本领域的技术人员可以对本申请进行各种改动和变型而不脱离本申请的精神和范围。这样，倘若本申请的这些修改和变型属于本申请权利要求及其等同技术的范围之内，则本申请也意图包含这些改动和变型在内。

Claims (27)

  1. 一种车道线检测方法，其特征在于，该方法包括：
    获取第一图像，所述第一图像为环境图像；
    根据所述第一图像获取第一车道线集合和第二车道线集合，其中，所述第一车道线集合和所述第二车道线集合的获取方式不同；
    根据所述第一车道线集合和所述第二车道线集合获取第三车道线集合。
  2. 如权利要求1所述的方法，其特征在于，根据所述第一图像获取第一车道线集合和第二车道线集合，包括：
    根据所述第一图像确定第二图像，所述第二图像用于确定所述第二车道线集合。
  3. 如权利要求2所述的方法，其特征在于，还包括：
    根据所述第一图像和所述第二图像确定第一损失值，所述第一损失值指示所述第一图像和所述第二图像的相似度。
  4. 如权利要求2或3所述的方法，其特征在于，所述根据第一图像确定第二图像，包括：
    根据所述第一图像采用第一模型确定所述第二图像；
    所述第一模型是根据原始图像和模板图像训练获得的，其中，所述模板图像与所述原始图像关联，所述模板图像的车道线检测难度低于所述原始图像的车道线检测难度；
    所述第一模型使第一损失函数满足第一预设条件，所述第一损失函数用于指示基于所述原始图像确定的预测模板图像与所述模板图像的差值。
  5. 如权利要求1-4任一项所述的方法，其特征在于，所述根据所述第一图像确定第一车道线集合，包括：
    根据所述第一图像采用第二模型确定所述第一车道线检测集合，所述第二模型是根据原始图像和模板图像训练获得的，其中，所述模板图像与所述原始图像关联，所述模板图像的车道线检测难度低于所述原始图像的车道线检测难度；
    其中，所述第二模型使第二损失函数、第三损失函数和第四损失函数中的至少两个损失函数满足对应的预设条件；其中，所述第二损失函数用于指示基于所述原始图像确定的预测车道线与所述原始图像对应的真实车道线的差值，所述第三损失函数用于指示第一特征与第二特征的差值，所述第四损失函数用于指示所述第一特征是否被鉴别为所述第二特征，所述第一特征为基于所述原始图像获取的特征，所述第二特征为基于所述模板图像获取的特征，所述第一特征与所述第二特征为相同类型的特征。
  6. 如权利要求2或3所述的方法，其特征在于，根据所述第一图像确定第二图像和第一车道线集合，包括：
    根据所述第一图像采用第三模型确定所述第二图像和所述第一车道线集合，所述第三模型是根据原始图像和模板图像训练获得的，其中，所述模板图像与所述原始图像关联，所述模板图像的车道线检测难度低于所述原始图像的车道线检测难度；
    其中，所述第三模型使第一损失函数满足第一预设条件，以及使第二损失函数满足第二预设条件，所述第一损失函数用于指示基于所述原始图像确定的预测模板图像与所述模板图像的差值，所述第二损失函数用于指示基于所述原始图像确定的预测车道线与所述原始图像对应的真实车道线的差值。
  7. 如权利要求6所述的方法，其特征在于，所述第三模型使第一损失函数满足第一预设条件，使第二损失函数满足第二预设条件，以及所述第三模型使第三损失函数满足第三预设条件和/或使第四损失函数满足第四预设条件，其中，所述第三损失函数用于指示第一特征与第二特征的差值，所述第四损失函数用于指示所述第一特征是否被鉴别为所述第二特征，所述第一特征为基于所述原始图像获取的特征，所述第二特征为基于所述模板图像获取的特征，所述第一特征与所述第二特征为相同类型的特征。
  8. 如权利要求5或7所述的方法，其特征在于，所述第一损失值是根据所述第一图像和所述第二图像采用所述第三损失函数获得的；
    或者，所述第一损失值是根据所述第一图像和所述第二图像采用所述第四损失函数获得的；
    或者，所述第一损失值是根据第二损失值和第三损失值确定的，所述第二损失值是根据所述第一图像和所述第二图像采用所述第三损失函数获得的，所述第三损失值是根据所述第一图像和所述第二图像采用所述第四损失函数获得的。
  9. 如权利要求1-8任一项所述的方法，其特征在于，所述第三车道线集合包括所述第一车道线集合和所述第二车道线集合中至少一条相同的车道线。
  10. 如权利要求1-9任一项所述的方法，其特征在于，还包括：
    确定所述第三车道线集合包括的车道线数目小于预设车道线数目时，将所述第一车道线集合和/或所述第二车道线集合中除所述第三车道线集合已包括的车道线之外的车道线作为所述第三车道线集合中的车道线。
  11. 如权利要求10所述的方法，其特征在于，还包括：
    确定所述第三车道线集合包括的车道线数目小于预设车道线数目时，将所述第一车道线集合和/或所述第二车道线集合中除所述第三车道线集合已包括的车道线之外的车道线作为所述第三车道线集合中的车道线，使得所述第三车道线集合包括的车道线数目等于所述预设车道线数目。
  12. 一种车道线检测装置，其特征在于，该装置包括：
    获取模块，用于获取第一图像，所述第一图像为环境图像；
    处理模块，用于根据所述第一图像获取第一车道线集合和第二车道线集合，其中，所述第一车道线集合和所述第二车道线集合的获取方式不同；
    根据所述第一车道线集合和所述第二车道线集合获取第三车道线集合。
  13. 如权利要求12所述的装置，其特征在于，所述处理模块用于：在根据所述第一图像获取第一车道线集合和第二车道线集合时，根据所述第一图像确定第二图像，所述第二图像用于确定所述第二车道线集合。
  14. 如权利要求13所述的装置，其特征在于，所述处理模块用于：根据所述第一图像和所述第二图像确定第一损失值，所述第一损失值指示所述第一图像和所述第二图像的相似度。
  15. 如权利要求13或14所述的装置，其特征在于，所述处理模块用于：在根据第一图像确定第二图像时，根据所述第一图像采用第一模型确定所述第二图像；
    所述第一模型是根据原始图像和模板图像训练获得的，其中，所述模板图像与所述原始图像关联，所述模板图像的车道线检测难度低于所述原始图像的车道线检测难度；
    所述第一模型使第一损失函数满足第一预设条件，所述第一损失函数用于指示基于所 述原始图像确定的预测模板图像与所述模板图像的差值。
  16. 如权利要求12-15任一项所述的装置，其特征在于，所述处理模块用于：在根据所述第一图像确定第一车道线集合时，根据所述第一图像采用第二模型确定所述第一车道线检测集合，所述第二模型是根据原始图像和模板图像训练获得的，其中，所述模板图像与所述原始图像关联，所述模板图像的车道线检测难度低于所述原始图像的车道线检测难度；
    其中，所述第二模型使第二损失函数、第三损失函数和第四损失函数中的至少两个损失函数满足对应的预设条件；其中，所述第二损失函数用于指示基于所述原始图像确定的预测车道线与所述原始图像对应的真实车道线的差值，所述第三损失函数用于指示第一特征与第二特征的差值，所述第四损失函数用于指示所述第一特征是否被鉴别为所述第二特征，所述第一特征为基于所述原始图像获取的特征，所述第二特征为基于所述模板图像获取的特征，所述第一特征与所述第二特征为相同类型的特征。
  17. 如权利要求13或14所述的装置，其特征在于，所述处理模块用于：在根据所述第一图像确定第二图像和第一车道线集合时，根据所述第一图像采用第三模型确定所述第二图像和所述第一车道线集合，所述第三模型是根据原始图像和模板图像训练获得的，其中，所述模板图像与所述原始图像关联，所述模板图像的车道线检测难度低于所述原始图像的车道线检测难度；
    其中，所述第三模型使第一损失函数满足第一预设条件，以及使第二损失函数满足第二预设条件，所述第一损失函数用于指示基于所述原始图像确定的预测模板图像与所述模板图像的差值，所述第二损失函数用于指示基于所述原始图像确定的预测车道线与所述原始图像对应的真实车道线的差值。
  18. 如权利要求17所述的装置，其特征在于，所述第三模型使第一损失函数满足第一预设条件，使第二损失函数满足第二预设条件，以及所述第三模型使第三损失函数满足第三预设条件和/或使第四损失函数满足第四预设条件，其中，所述第三损失函数用于指示第一特征与第二特征的差值，所述第四损失函数用于指示所述第一特征是否被鉴别为所述第二特征，所述第一特征为基于所述原始图像获取的特征，所述第二特征为基于所述模板图像获取的特征，所述第一特征与所述第二特征为相同类型的特征。
  19. 如权利要求16或18所述的装置，其特征在于，所述第一损失值是根据所述第一图像和所述第二图像采用所述第三损失函数获得的；
    或者，所述第一损失值是根据所述第一图像和所述第二图像采用所述第四损失函数获得的；
    或者，所述第一损失值是根据第二损失值和第三损失值确定的，所述第二损失值是根据所述第一图像和所述第二图像采用所述第三损失函数获得的，所述第三损失值是根据所述第一图像和所述第二图像采用所述第四损失函数获得的。
  20. 如权利要求12-19任一项所述的装置，其特征在于，所述第三车道线集合包括所述第一车道线集合和所述第二车道线集合中至少一条相同的车道线。
  21. 如权利要求12-20任一项所述的装置，其特征在于，所述处理模块用于：确定所述第三车道线集合包括的车道线数目小于预设车道线数目时，将所述第一车道线集合和/或所述第二车道线集合中除所述第三车道线集合已包括的车道线之外的车道线作为所述第三车道线集合中的车道线。
  22. 如权利要求21所述的装置，其特征在于，所述处理模块用于：确定所述第三车道 线集合包括的车道线数目小于预设车道线数目时，将所述第一车道线集合和/或所述第二车道线集合中除所述第三车道线集合已包括的车道线之外的车道线作为所述第三车道线集合中的车道线，使得所述第三车道线集合包括的车道线数目等于所述预设车道线数目。
  23. 一种车道线检测装置，其特征在于，所述装置包括：
    存储器：用于存储指令；
    处理器，用于从所述存储器中调用并运行所述指令，使得所述装置或者安装有所述装置的设备执行如权利要求1～11中任意一项所述的方法。
  24. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质存储有计算机程序，当所述计算机程序在装置上运行时，执行如权利要求1～11中任意一项所述的方法。
  25. 一种计算机程序产品，其特征在于，所述计算机程序产品包括计算机程序，当所述计算机程序在装置上运行时，执行如权利要求1～11中任意一项所述的方法。
  26. 一种芯片，其特征在于，包括至少一个处理器和通信接口；
    所述通信接口，用于为所述至少一个处理器提供程序指令或者数据；
    所述至少一个处理器用于执行所述程序行指令，以实现如权利要求1～11中任意一项所述的方法。
  27. 一种终端设备，其特征在于，所述终端设备包括如权利要求12～22中任意一项所述的装置或如权利要求23所述的装置。
PCT/CN2021/083552 2021-03-29 2021-03-29 一种车道线检测方法及装置 WO2022204867A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202180095662.XA CN116997940A (zh) 2021-03-29 2021-03-29 一种车道线检测方法及装置
PCT/CN2021/083552 WO2022204867A1 (zh) 2021-03-29 2021-03-29 一种车道线检测方法及装置

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2021/083552 WO2022204867A1 (zh) 2021-03-29 2021-03-29 一种车道线检测方法及装置

Publications (1)

Publication Number Publication Date
WO2022204867A1 true WO2022204867A1 (zh) 2022-10-06

Family

ID=83456900

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/083552 WO2022204867A1 (zh) 2021-03-29 2021-03-29 一种车道线检测方法及装置

Country Status (2)

Country Link
CN (1) CN116997940A (zh)
WO (1) WO2022204867A1 (zh)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2012011713A2 (ko) * 2010-07-19 2012-01-26 주식회사 이미지넥스트 차선 인식 시스템 및 방법
CN108764187A (zh) * 2018-06-01 2018-11-06 百度在线网络技术（北京）有限公司 提取车道线的方法、装置、设备、存储介质以及采集实体
CN110766724A (zh) * 2019-10-31 2020-02-07 北京市商汤科技开发有限公司 目标跟踪网络训练、跟踪方法及装置、电子设备及介质
CN111860588A (zh) * 2020-06-12 2020-10-30 华为技术有限公司 一种用于图神经网络的训练方法以及相关设备
CN112287842A (zh) * 2020-10-29 2021-01-29 恒大新能源汽车投资控股集团有限公司 一种车道线的识别方法、装置及电子设备
CN112461257A (zh) * 2019-09-09 2021-03-09 华为技术有限公司 一种车道线信息的确定方法及装置

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110135302B (zh) * 2019-04-30 2021-08-31 百度在线网络技术（北京）有限公司 训练车道线识别模型的方法、装置、设备和存储介质
CN112115857B (zh) * 2020-09-17 2024-03-01 福建牧月科技有限公司 智能汽车的车道线识别方法、装置、电子设备及介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2012011713A2 (ko) * 2010-07-19 2012-01-26 주식회사 이미지넥스트 차선 인식 시스템 및 방법
CN108764187A (zh) * 2018-06-01 2018-11-06 百度在线网络技术（北京）有限公司 提取车道线的方法、装置、设备、存储介质以及采集实体
CN112461257A (zh) * 2019-09-09 2021-03-09 华为技术有限公司 一种车道线信息的确定方法及装置
CN110766724A (zh) * 2019-10-31 2020-02-07 北京市商汤科技开发有限公司 目标跟踪网络训练、跟踪方法及装置、电子设备及介质
CN111860588A (zh) * 2020-06-12 2020-10-30 华为技术有限公司 一种用于图神经网络的训练方法以及相关设备
CN112287842A (zh) * 2020-10-29 2021-01-29 恒大新能源汽车投资控股集团有限公司 一种车道线的识别方法、装置及电子设备

Also Published As

Publication number Publication date
CN116997940A (zh) 2023-11-03

Similar Documents

Publication Publication Date Title
CN109901574B (zh) 自动驾驶方法及装置
CN112166304B (zh) 传感器数据的误差检测
CN115206132B (zh) 基于道路使用者意图预测的潜在碰撞警告***
US10981567B2 (en) Feature-based prediction
US11966838B2 (en) Behavior-guided path planning in autonomous machine applications
WO2022001773A1 (zh) 轨迹预测方法及装置
CN108068815B (zh) 用于自动驾驶车辆的基于规划反馈的决策改进***
CN110371132B (zh) 驾驶员接管评估方法及装置
US20240127062A1 (en) Behavior-guided path planning in autonomous machine applications
WO2021102955A1 (zh) 车辆的路径规划方法以及车辆的路径规划装置
WO2021000800A1 (zh) 道路可行驶区域推理方法及装置
CN112534483B (zh) 预测车辆驶出口的方法和装置
CN113460042A (zh) 车辆驾驶行为的识别方法以及识别装置
WO2022016351A1 (zh) 一种行驶决策选择方法以及装置
CN115578711A (zh) 自动换道方法、装置及存储介质
US11721089B2 (en) Using temporal filters for automated real-time classification
WO2022178858A1 (zh) 一种车辆行驶意图预测方法、装置、终端及存储介质
WO2022017307A1 (zh) 自动驾驶场景生成方法、装置及***
CN114693540A (zh) 一种图像处理方法、装置以及智能汽车
WO2022051951A1 (zh) 车道线检测方法、相关设备及计算机可读存储介质
CN113859265B (zh) 一种驾驶过程中的提醒方法及设备
CN115546781A (zh) 一种点云数据的聚类方法以及装置
WO2021000787A1 (zh) 道路几何识别方法及装置
CN113614732A (zh) 信息处理设备及信息处理方法
WO2022204867A1 (zh) 一种车道线检测方法及装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21933553

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 202180095662.X

Country of ref document: CN

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21933553

Country of ref document: EP

Kind code of ref document: A1