CN102458964A - Camera system for use in vehicle parking - Google Patents

Camera system for use in vehicle parking Download PDF

Info

Publication number
CN102458964A
CN102458964A CN2010800250149A CN201080025014A CN102458964A CN 102458964 A CN102458964 A CN 102458964A CN 2010800250149 A CN2010800250149 A CN 2010800250149A CN 201080025014 A CN201080025014 A CN 201080025014A CN 102458964 A CN102458964 A CN 102458964A
Authority
CN
China
Prior art keywords
camera
pick
vehicle
parking
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN2010800250149A
Other languages
Chinese (zh)
Other versions
CN102458964B (en
Inventor
J·贝克尔
S·卡梅尔
B·皮策
C·杜哈德维
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Robert Bosch GmbH
Original Assignee
Robert Bosch GmbH
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Robert Bosch GmbH filed Critical Robert Bosch GmbH
Publication of CN102458964A publication Critical patent/CN102458964A/en
Application granted granted Critical
Publication of CN102458964B publication Critical patent/CN102458964B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • BPERFORMING OPERATIONS; TRANSPORTING
    • B62LAND VEHICLES FOR TRAVELLING OTHERWISE THAN ON RAILS
    • B62DMOTOR VEHICLES; TRAILERS
    • B62D15/00Steering not otherwise provided for
    • B62D15/02Steering position indicators ; Steering position determination; Steering aids
    • B62D15/027Parking aids, e.g. instruction means
    • B62D15/028Guided parking by providing commands to the driver, e.g. acoustically or optically
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B60VEHICLES IN GENERAL
    • B60RVEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R1/00Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
    • B60R1/20Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
    • B60R1/22Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle
    • B60R1/23Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle with a predetermined field of view
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B62LAND VEHICLES FOR TRAVELLING OTHERWISE THAN ON RAILS
    • B62DMOTOR VEHICLES; TRAILERS
    • B62D15/00Steering not otherwise provided for
    • B62D15/02Steering position indicators ; Steering position determination; Steering aids
    • B62D15/027Parking aids, e.g. instruction means
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/003D [Three Dimensional] image rendering
    • G06T15/10Geometric effects
    • G06T15/20Perspective computation
    • G06T15/205Image-based rendering
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00Television systems
    • H04N7/18Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H04N7/181Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B60VEHICLES IN GENERAL
    • B60RVEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R1/00Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
    • B60R1/20Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
    • B60R1/30Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles providing vision in the non-visible spectrum, e.g. night or infrared vision
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B60VEHICLES IN GENERAL
    • B60RVEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R2300/00Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
    • B60R2300/10Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used
    • B60R2300/105Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used using multiple cameras
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B60VEHICLES IN GENERAL
    • B60RVEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R2300/00Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
    • B60R2300/30Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing
    • B60R2300/304Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing using merged images, e.g. merging camera image with stored images
    • B60R2300/305Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing using merged images, e.g. merging camera image with stored images merging camera image with lines or icons
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B60VEHICLES IN GENERAL
    • B60RVEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R2300/00Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
    • B60R2300/60Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by monitoring and displaying vehicle exterior scenes from a transformed perspective
    • B60R2300/607Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by monitoring and displaying vehicle exterior scenes from a transformed perspective from a bird's eye viewpoint
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B60VEHICLES IN GENERAL
    • B60RVEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R2300/00Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
    • B60R2300/80Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement
    • B60R2300/806Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement for aiding parking

Landscapes

  • Engineering & Computer Science (AREA)
  • Mechanical Engineering (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Transportation (AREA)
  • Combustion & Propulsion (AREA)
  • Chemical & Material Sciences (AREA)
  • Physics & Mathematics (AREA)
  • Signal Processing (AREA)
  • Computing Systems (AREA)
  • Geometry (AREA)
  • Computer Graphics (AREA)
  • General Physics & Mathematics (AREA)
  • Traffic Control Systems (AREA)
  • Image Processing (AREA)
  • Closed-Circuit Television Systems (AREA)

Abstract

A method for assisting in parallel parking includes providing a vehicle with a forward-looking camera and/or a backward-looking camera. The camera is used to capture an image of a parking area including at least one unoccupied parking space and a plurality of parking spaces occupied by other vehicles. A homography of the captured image is created. The homography is used to estimate an image of the parking area from an overhead viewpoint. A portion of the overhead image including the unoccupied parking space is displayed on a display screen within a passenger compartment of the vehicle.

Description

The camera chain that is used for vehicle parking
Technical field
The present invention relates to be used for the camera chain of vehicle, and relate more specifically to be used for the camera chain of vehicle parking.
Background technology
The driver assistance system of a kind of being called " daily output panoramic monitor " has been developed in daily output (Nissan), and it adopts four ultra wide high-resolution cameras that are installed in front part of vehicle, rear portion and the both sides that the panoramic view of vehicle periphery is provided.All these images are all handled and on the screen that is installed on the instrument carrier panel, are shown in real time, and the 360 degree visuals field of the vehicle periphery obstacle that they hope crash-avoidance are provided to chaufeur.With vehicle shift to reverse gear or D Drive, can forwardly or between the visual field at rear portion replace.Chaufeur can implement to switch left side, front portion and right side view when thorny especially Parallel parking is handled." panoramic monitor " made in Japan is in Infiniti EX35 vehicle in the first Application of the U.S..
Reflection (mapping) is a kind of technology that is used for making up the map in the circumstances not known that made by robot and vehicle.This and unlike sounding so simple, because exist intrinsic uncertainty from various its relative motions of sensor identification of robot.A kind of system that is used to shine upon road surface comprises high-end laser radar, GPS and Inertial Measurement Unit (IMU) sensor.This system can produce the two-dimensional surface image of ground return property with 5 centimetres road surface resolution.
Other known systems focus on through use and are installed on the birds-eye view that pick up camera on aircraft or the satellite produces city and rural landform.Modal method is push-broom type splicing (pushbroom mosaic), and the application scenario that original visual angle is approached at its expectation visual angle for wherein final splicing is worked finely.
Some commercially produced products such as " streetscape (the Street View) " of Google or " curbside (the Street Side) " of Microsoft, have been created the static map of city scene.Focus in these systems provides the view in city, and it is similar to the scene that passenger or pedestrian possibly see when same urban activity.If any and all processing can be carried out by off-lines, the visual angle difference between captured images and the final image is very little.
(be sometimes referred to as terrestrial reference, the other system that landmark) produces map also proposes to use the sparse features diagram.Especially, several of vSLAM (visual simulation is located and mapping) are attempted developing and showing, mainly are to develop and be illustrated in to use in the indoor environment of camera chain generation based on the map of characteristic.In a word, these are disabled based on the map of characteristic for the people who attempts to locate himself or its vehicle.
Forward sight self-propelled vehicle pick up camera is used for comprising that night vision is auxiliary, lane detection, lane departur warning, the track keeps and road markings detects various occasions.In the future, this pick up camera will be the standard configuration of senior vehicle.Rear view camera is also auxiliary as the vision of reversing, and possibly be to avoid the reversing accident needed in the future.
Also there are the various auxiliary methods of parking.The something in common of these methods is that sonar is used to main sensors.The advantage of sonar is cheap.The shortcoming of sonar is that sonar to measure has noise and inaccurate, the system action that this leads to errors sometimes.Existing system is sending vision and/or acoustics warning to chaufeur before with object collision.Following system will come driver assistance person through direction of operating dish (being semiautomatic parking) or bearing circle open out/brake (being that full automaticity is parked).
BMW has proposed a kind of system based on pick up camera, and it uses motion three-dimensional (motion-stereo) to be used to park auxiliary.This system needs other side direction pick up camera with the detection zone of parking.
Prior art be not both open, do not advised that a kind of forward direction pick up camera and/or back of utilizing came birds-eye view to chaufeur to be provided to pick up camera, with the method for auxiliary Parallel parking yet.
Summary of the invention
The present invention can provide a kind of forward direction and/or back of comprising to pick up camera, with the system of service vehicle's chaufeur during Parallel parking.
In its a kind of form, the present invention includes a kind of method of auxiliary Parallel parking, it comprises to vehicle provides forward direction pick up camera and/or back to pick up camera.Pick up camera is used for catching the area image of parking that comprises at least one vacant park space and a plurality of spaces of parking that taken by other vehicles.The homograph of image is caught in generation.Homograph is used for estimating the image in the zone of parking from getting a bird's eye view the visual angle.Comprise on the read-out of a part of birds-eye view in the passenger carriage of vehicle in the vacant space of parking and showing.
In its another kind of form, the present invention includes a kind of electronic parking auxiliary device that is used for vehicle.This device comprises that the forward direction that is arranged in the vehicle and/or back are to pick up camera.Treater is bonded at least one pick up camera.Treater receives first image in the zone of parking.Image is caught from substantially horizontal visual angle by pick up camera.Treater is second image from the perpendicular view with first image transitions.Discern the vacant space of parking in second image.Contiguous basically vacant the parking during the space of vehicle, the diagram of vehicle is overlapped onto on second image.The vehicle diagram is upgraded with respect to the position in the vacant space of parking constantly in second image, with the actual position of indication vehicle with respect to the vacant space of parking.Read-out is arranged in the passenger carriage of vehicle, and is bonded to treater.Read-out receives and shows has overlapping graphic second image of vehicle.
In its another form, the present invention includes a kind of method of displayed map information, it comprises to vehicle provides forward direction and/or back to pick up camera.Pick up camera is used for catching the image of peripheral region.The homograph of image is caught in generation.Homograph is used to from getting a bird's eye view this regional image of visual angle estimation.Confirm vehicle position with respect to birds-eye view when moving.Show on the read-out of birds-eye view in the passenger carriage of vehicle.The birds-eye view that is shown comprises the real-time graphic of vehicle.The graphic position of vehicle in the birds-eye view is based on definite step and upgrades constantly basically.
An advantage of the present invention is, it compares sensor that known system need be still less and therefore obvious more cheap than known system.Particularly, the present invention can only utilize forward direction and/or back to pick up camera.
Another advantage of the present invention is that its utilization will be the equipment of the standard facility on the following vehicle, and need not other hardware.More specifically, forward direction will be the standard facility on the following automobile with the back to pick up camera.The forward direction pick up camera is used for night vision, lane detection, road markings detection etc.The back is generally used for avoiding the reversing accident to pick up camera, and possibly be that following automobile is required.For example, referring to by of the guidance of national highway traffic safety administration about avoiding the reversing accident to be provided.Thereby the present invention can provide other valuable benefit through software, and can only use existing hardware.
Description of drawings
Through with reference to below in conjunction with the description of accompanying drawing to the embodiment of the invention, of the present inventionly above-mentionedly will become obvious more and invent self and will be better understood with other characteristics and target and obtain manner thereof, wherein:
Fig. 1 is the vehicle schematic plan view that includes an embodiment of parking assist apparatus of the present invention.
Fig. 2 is the example image of being caught by the forward direction pick up camera of the parking assist apparatus of Fig. 1.
Fig. 3 be according to an embodiment of the invention, image shown in Figure 2 projected on the three-dimensional ground level of estimation through using homograph.
Fig. 4 is the exemplary map by an embodiment generation of method of the present invention.
Fig. 5 is the exemplary map by another embodiment generation of method of the present invention.
Fig. 6 is the situation of the image overlay that is mapped in vehicle shown in Figure 1 on it time of Fig. 4.
Fig. 7 is the exemplary map by the another embodiment generation of method of the present invention.
Fig. 8 is the exemplary map by another embodiment generation of method of the present invention.
Fig. 9 is tire that the signal of Fig. 8 is mapped in the chaufeur vehicle situation when being in linear position.
Figure 10 is that the exemplary map of Fig. 8 must be with the bearing circle anti-clockwise rotation to accompany or follow the park situation at some place in path of the best at the chaufeur vehicle.
Figure 11 illustrates the diagram of circuit that is used for a kind of method of auxiliary Parallel parking of the present invention.
Figure 12 illustrates the diagram of circuit that is used for a kind of method of displayed map information of the present invention.
Corresponding reference symbol is shown corresponding parts in several views.Though the example of setting forth here shows embodiments of the invention with several forms, below disclosed embodiment be not the exhaustive disclosed precise forms that perhaps is configured to limit the scope of the present invention to.
The specific embodiment
Referring now to accompanying drawing; And Fig. 1 especially; An embodiment who comprises the vehicle of parking assist apparatus 10 of the present invention shown in it, this parking assist apparatus comprise that forward direction pick up camera 12, back are to pick up camera 14, treater 16, telltale 18, rotation direction sensor 19 and audio speaker 20.Forward direction pick up camera 12 can be installed in the grid of vehicle so that pick up camera 12 towards and point to forward direction 13.The back can be installed in the rear bumper of vehicle to pick up camera 14 so that pick up camera 14 towards and point to the back to 15.In certain embodiments, parking assist apparatus only comprises in pick up camera 14 one of forward direction pick up camera 12 and back, rather than two.
Telltale 18 can comprise the for example screen or the monitoring device of automatic computer.In one embodiment, screen or monitoring device can be installed on the instrument carrier panel of vehicle (it can be an automobile).
Treater 16 can be the form of standard micro treater, and can comprise the memory device (not shown).Treater 16 may command are shown on screen or the monitoring device of telltale 18 and from the information or the content of loud speaker 20 audio frequency ground output.
The present invention's frozen image capable of using.Pick up camera 12 and 14 can be when vehicle moves the image of capturing ambient always.The typical image of when parked in the space in the chaufeur searching, being caught by forward direction pick up camera 12 and on telltale 18, being shown is shown in Fig. 2.This image comprises the unappropriated space 21 of parking by chance.The relative position of given pick up camera 12 in vehicle, homograph (homography) or " projective transformation (projective transformation) " can be at the planes of delineation and are calculated between the plane practically.Homograph can be used to each image (such as the image shown in Fig. 2) is projected on the three-dimensional ground level of estimation, shown in the projection camera review of Fig. 3.
In order to collect the data that stride across a plurality of frames with fine resolution, discrete grid block (as 22 places become the fragment ground shown in the position) can be aimed at and overlap onto on it with ground level.In a specific embodiment, each grid cell is measured 5 centimetres of 5 cm x, and collects its center and project the whole pixels on the unit in its border.Grid self can have limited height and width, and in can be when pick up camera moves through scenery fixed again always on pick up camera.
What in memory device, stored in one embodiment, is the incidence relation the lists of links of each unit and the pixel value that is observed from all images at this cell position place recently.Observed pixel value can be stored in the memory device in predetermined period formerly, and the oldest pixel value can be substituted by nearest observed pixel value in memory device.The much higher observed value of number of comparing with those unit of both sides can be collected near the unit picture centre (and therefore the center of grid).In one embodiment, near the unit the picture centre can be collected up to about 700 observed values, and 4 or 5 observed values are only collected near the unit left-hand side periphery or the right-hand side periphery.
Then, the end value of each grid cell can be with any calculating in several different modes, and these unit can be combined to form the single birds-eye view of observed environment.The simplest aggregation technique is in each unit, to select up-to-date observed value.Shown in Fig. 4, this is the map that is produced by single forward direction pick up camera by this technological exemplary map that produces.
In other embodiments, use the aviation value of observed value or mode can reduce owing to be positioned at the caused distortion of obstacle (such as other vehicles) of ground level outside (for example top).The variation that can also use value collected in individual unit is to detect and possibly remove this distortion.Fig. 5 is and the similar map of Fig. 4, has wherein removed some distortions that caused by vehicle 24 and 26 at least.
The map that can be used by chaufeur is shown in Fig. 6, and this is image or graphic situation that being mapped in of Fig. 4 has vehicle, and wherein parking assist apparatus 10 is installed to be overlapping or covers above that.Yet vehicle can also overlap on the scheme of distortion reduction of Fig. 5.Overlapping vehicle diagram can be drawn in proportion, and can be aspect geometric aspects or position ground accurately so that let chaufeur exactly estimating vehicle with respect to the current location and the size of any track on the road or the mark of parking.
As stated, the present invention can be applicable to park auxiliary.When chaufeur was sought the parking site of Parallel parking, parking assisting system can be always and is drawn the map of vehicle environmental constantly.Use mapping and described object detection, might be from the map measurement size a little of parking.Park the space measurement size can with the size compared of chaufeur vehicle, and can provide the space of parking whether enough big to chaufeur to hold the indication of chaufeur vehicle.
Parking assisting system can provide vision and/or the audible signal that has been found that the space of parking to chaufeur, and can the position in the space of parking and the vision and/or the audible signal of size be provided to chaufeur.The size in space of parking can be expressed with respect to the size of vehicle.
In one embodiment, chaufeur can be indicated a little available threshold size that provides the space of parking to have before of parking to chaufeur at parking assisting system.Threshold size can be expressed as the space of the parking distance bigger than vehicle.
In another embodiment, parking assisting system can keep vehicle successfully Parallel parking in the size record in wherein the space of formerly parking, but and the not size in the space of formerly parking of success of attempting to park therein.Then, parking assisting system can be provided at the indication of formerly attempting the success ratio of Parallel parking in the space of similar size to chaufeur.Because some chaufeurs maybe be better in the technology aspect the Parallel parking than other chaufeurs, parking assisting system can additionally be preserved this record of each chaufeur Parallel parking formerly.Parking assisting system provides its individual who in the space identical with current identification place general size, parks success ratio formerly also can for each chaufeur.
Park a little and size can show on map, as indicating by the dotted line among Fig. 7.In this specific embodiment, park a little and size is represented by the rectangle that on telltale 18, provides.Vehicle also can show on telltale 18 with respect to the position of parking a little.
Parking assisting system can provide about recommending the audio frequency and/or the visual instructions of the technology of parking to chaufeur.For example, how far this system can should go before vehicle is placed reverse gear, on the street to the chaufeur indication.The suggestion that is used for the optimal path of Parallel parking can provide with vision appreciiable mode on telltale 18.Optimal path can be that vehicle can sail the path in the space of parking under the situation that has maximum clearance between vehicle and the obstacle (such as other parked vehicle) thus.
Except optimal path, telltale 18 current location according to bearing circle and/or tire (as by 19 sensings of rotation direction sensor) also can be shown and vehicle with the path of adopting.Telltale 18 is accompanyed or follow sustainably and is shown moving of vehicle, thereby feeds back to chaufeur.This system also can when steering wheel rotation and rotation be how far when reverse gear is driven to chaufeur indication.For example, the exemplary inventive screen display of Fig. 8 is based on the expectation path 28 (illustrating with solid line) that the chaufeur vehicle is shown like the steering position by the tire 30a-b of rotation direction sensor 19 sensings.At point 32 places along expectation path 28, chaufeur is left-handed turning hastily, to accompany or follow and expectation path 28 optimal path 34 (being shown in broken lines) of difference mutually.
In Fig. 9, the expectation path 28 of chaufeur vehicle can be overlapping and consistent with optimal path 34.Yet in one embodiment, expectation path 28 and optimal path can not have in these two paths when overlapping or consistent dividually shown in the telltale.As an example, the read-out of Fig. 9 illustrate produced by the tire 30a-b that is in linear position with optimal path 34 comparatively speaking be the expectation path 28 of linearity or straight line.Because difference or deviation between expectation path 28 and the optimal path 34 can provide literal order " to turn right rapidly " as illustrated in fig. 9 on screen display.Additionally, or alternatively, loud speaker 20 can send the audio frequency spoken command and " sharply turns right ".These instructions of turning right are sufficiently turned right bearing circle rapidly at chaufeur and are stopped when causing that expectation path 28 coincides with optimal path 34 as shown in Figure 8.
Example screen display when Figure 10 is illustrated in vehicle destination point 32.Literal order " turns left rapidly " and can when vehicle destination point 32, be provided on the screen display as shown.Additionally, or alternatively, loud speaker 20 can send the audio frequency spoken command and " turns left rapidly ".
When parked in the zone in pedestrian's entering, can audio frequency or visual alert be provided to chaufeur.Pedestrian's existence can detect through sonar system (not shown) in the pick up camera 12,14 or the car.
The embodiment of the method 110 that is used for auxiliary Parallel parking of the present invention is shown in Figure 11.In first step 1102, vehicle set has forward direction pick up camera and/or back to pick up camera.For example, the vehicle shown in Fig. 1 comprises that forward direction pick up camera 12 and back are to pick up camera 14.
In next procedure 1104, pick up camera is used for catching the regional image of parking, and comprises at least one vacant space and a plurality of space of parking that is taken by other vehicles of parking.For example, by the image shown in Figure 2 that pick up camera is caught, be the zone of parking that comprises vacant other spaces of parking of parking space 21 and taking by vehicle (such as vehicle 24 and 26).
Then, in step 1106, produce the homograph of captured images.Just, projection conversion can be calculated between the plane of delineation of Fig. 2 and ground level.In this article, ground level can be thought the plane of the supported ground surface on it of parked vehicle.
In step 1108, homograph is used to estimate the image in the zone of parking from getting a bird's eye view the visual angle.For example, homograph can be used to catching of Fig. 2 area image of parking is transformed on the ground level, thus the birds-eye view of estimation Fig. 3.
In the end in the step 1110, comprise on the read-out of a part of birds-eye view in the passenger carriage of vehicle in the vacant space of parking showing.Just, the birds-eye view of Fig. 3 can further be handled, to discern the vacant any image shown in space 21 and generation Fig. 4-10 of parking.Each of Fig. 4-10 comprises the vacant space 21 of parking, and any one of Fig. 4-10 can show on read-out 18.
Another embodiment of the method 1200 that is used for displayed map information of the present invention is shown in Figure 12.In first step 1202, vehicle set has forward direction pick up camera and/or back to pick up camera.For example, the vehicle shown in Fig. 1 comprises that forward direction pick up camera 12 and back are to pick up camera 14.
In next procedure 1204, pick up camera is used to catch the image of peripheral region.The image shown in Figure 2 of for example, being caught by pick up camera is the image in the zone of parking of chaufeur vehicle periphery.
Then, in step 1206, produce the homograph of captured images.Just, projection conversion can be calculated between the plane of delineation of Fig. 2 and ground level.In this article, ground level can be thought the plane of the supported ground surface on it of parking automobile.
In step 1208, homograph is used to estimate the image in the zone of parking from getting a bird's eye view the visual angle.For example, homograph can be used to catching of Fig. 2 area image of parking is transformed on the ground level, thus the birds-eye view of estimation Fig. 3.
In next procedure 1210, confirm that vehicle is with respect to the position of birds-eye view when vehicle moves.Vehicle can be confirmed with any variety of way with respect to the position of birds-eye view, comprise: based on calculating current vehicle location by the current image of catching of pick up camera; The service range sensor, such as sonar on the chaufeur vehicle or radar, its sensing is the position of parking automobile; And/or export the travel distance and the direction of following the trail of the chaufeur vehicle based on mileage meter and rotation direction sensor, because the current region in the birds-eye view is maintained in the visual field of pick up camera (the initial acquisition image that supposition chaufeur vehicle and vacant initial distance of parking between the space are based on sonar and/or the vacant space of parking calculates).
In the end in the step 1212, show birds-eye view on the read-out in the passenger carriage of vehicle, the birds-eye view of demonstration comprises that the position that the real-time graphic of vehicle, vehicle are shown in the birds-eye view is based on the renewal constantly basically of definite step.Just, birds-eye view (such as shown in Fig. 6 and the 8-10 those) shows on the read-out 18 of the passenger carriage of vehicle.In one embodiment, read-out 18 is installed on the instrument carrier panel.Each of Fig. 6 and 8-10 shows that birds-eye view can comprise the real-time graphic of chaufeur vehicle.Vehicle is shown in the interior position of birds-eye view of Fig. 6 and 8-10, can upgrade constantly using the true timer-operated up-to-date actual position of up-to-date input data based on the chaufeur vehicle.
The present invention has been disclosed as at this should be used for driver assistance person's manipulation of implementing to park.Yet, will appreciate that the present invention can be applicable to other occasions, such as near the pedestrian's alerting driver actual position, or prevent that automatically chaufeur from knocking the pedestrian.
Though the present invention has been described as having the design of example, however the present invention can be in spirit of the present disclosure and scope further modification.Therefore, the application will cover any variation, use or the change of principle as the use one of which of the present invention.

Claims (20)

1. method that service vehicle parks, it may further comprise the steps:
To vehicle forward direction pick up camera and at least one in pick up camera of back are provided;
Use pick up camera to catch the regional image of parking, the said zone of parking comprises at least one vacant space and a plurality of space of parking that is taken by other vehicles of parking;
The homograph of image is caught in generation;
Use said homograph to estimate the image in the zone of parking from getting a bird's eye view the visual angle; And
Show a part of birds-eye view that comprises the vacant space of parking on the read-out in the passenger carriage of vehicle.
2. according to the method for claim 1, it comprises the another step that detects the vacant space of parking in the birds-eye view.
3. according to the method for claim 1, it comprises the graphic another step that the current location of vehicle in birds-eye view is provided.
4. according to the method for claim 3, wherein the diagram of the current location of vehicle in birds-eye view is upgraded constantly.
5. according to the process of claim 1 wherein that said forward direction pick up camera and in pick up camera at least one of back comprise at least one the forward direction pick up camera that is used for that night vision, lane detection and road markings detect.
6. be used for avoiding the back of reversing accident according to the process of claim 1 wherein that at least one to pick up camera of said forward direction pick up camera and back comprises to pick up camera.
7. according to the method for claim 1, further comprising the steps of:
Calculating is used for the optimal path that vehicle sails the vacant space of parking into; And
On read-out, show optimal path.
8. electronic parking auxiliary device that is used for vehicle, said device comprises:
Be arranged at forward direction pick up camera and at least one in pick up camera of back in the vehicle;
Be bonded to the treater on said at least one pick up camera, said treater is configured to:
Reception park the zone first image, this image is caught from substantially horizontal visual angle by pick up camera;
With first image transitions is second image from the perpendicular view;
Discern the vacant space of parking in second image; And
Overlap onto on second image in the contiguous basically vacant diagram of parking during the space vehicle of vehicle; Vehicle diagram in second image is upgraded with respect to the position in the vacant space of parking constantly, with the actual position of indication vehicle with respect to the vacant space of parking; And
Read-out, it is arranged in the passenger carriage of vehicle and is bonded to treater, and this read-out is constructed to receive and shows has overlapping graphic second image of vehicle.
9. according to Claim 8 device, wherein treater is constructed to:
Produce the homograph of first image; And
In the process that with first image transitions is second image, use said homograph.
10. according to Claim 8 device; Wherein said forward direction pick up camera comprises the forward direction pick up camera with at least one in pick up camera of back, and said treater is constructed to use the forward direction pick up camera to be used at least one that night vision, lane detection and road markings detect.
11. device according to Claim 8, wherein said forward direction pick up camera comprises the back to pick up camera with at least one in pick up camera of back, and said treater is constructed to use the back to be used to avoid the reversing accident to pick up camera.
12. device according to Claim 8 also comprises audio speaker, said treater is constructed to audible signal is transferred to audio speaker, and audible signal comprises the spoken command that is used for Parallel parking.
13. device according to Claim 8 also comprises the rotation direction sensor that is bonded to treater, this rotation direction sensor is constructed to:
The steering direction of senses vehicle tire; And
The direction signal of direction indication is sent to treater;
Said treater is constructed to:
Calculate the expectation path of vehicle based on direction signal; And
The indication of the vehicle expectation path that is calculated is provided in second image.
14. according to the device of claim 13, wherein said treater is configured to:
Calculate and the expectation path optimal path of difference mutually; And
The indication of optimal path is provided in second image.
15. the method for a displayed map information, it comprises step:
To vehicle forward direction pick up camera and at least one in pick up camera of back are provided;
Use pick up camera to catch the image of peripheral region;
The be hunted down homograph of image of generation;
Use said homograph to come from getting a bird's eye view the image that the visual field estimates that this is regional;
Confirm vehicle position with respect to birds-eye view when moving; And
Show birds-eye view on the read-out in the passenger carriage of vehicle, the birds-eye view that is shown comprises the real-time graphic of vehicle, and the vehicle shown position in the said birds-eye view is based on definite step and upgrades constantly basically.
16. according to the method for claim 15, it comprises the another step that homograph is divided into substantially the same grid cell, said step display depends on one of aviation value, mode and variance of the pixel value in the corresponding grid cell.
17. according to the method for claim 15, it comprises the another step that detects the vacant space of parking in the birds-eye view.
18. according to the method for claim 15, wherein said forward direction pick up camera and in pick up camera at least one of back comprise at least one the forward direction pick up camera that is used for that night vision, lane detection and road markings detect.
19. according to the method for claim 15, wherein said forward direction pick up camera comprises with in pick up camera at least one of back and is used to avoid the back to pick up camera of reversing accident.
20., further comprising the steps of according to the method for claim 15:
Calculate vehicle and sail the optimal path in the vacant space of parking into; And
On read-out, show optimal path.
CN201080025014.9A 2009-05-11 2010-05-07 Camera system for use in vehicle parking Active CN102458964B (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US12/463,699 US8289189B2 (en) 2009-05-11 2009-05-11 Camera system for use in vehicle parking
US12/463,699 2009-05-11
PCT/US2010/034141 WO2010132314A2 (en) 2009-05-11 2010-05-07 Camera system for use in vehicle parking

Publications (2)

Publication Number Publication Date
CN102458964A true CN102458964A (en) 2012-05-16
CN102458964B CN102458964B (en) 2015-01-28

Family

ID=43062036

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201080025014.9A Active CN102458964B (en) 2009-05-11 2010-05-07 Camera system for use in vehicle parking

Country Status (5)

Country Link
US (1) US8289189B2 (en)
EP (1) EP2429877B1 (en)
JP (1) JP5620472B2 (en)
CN (1) CN102458964B (en)
WO (1) WO2010132314A2 (en)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103728624A (en) * 2012-10-12 2014-04-16 现代摩比斯株式会社 Parking assist apparatus and parking assist method and parking assist system using the same
CN103723194A (en) * 2013-12-13 2014-04-16 浙江吉利控股集团有限公司 Automobile reverse-driving tire position monitoring device and reverse driving assisting device
CN105355083A (en) * 2015-12-14 2016-02-24 宁波裕兰信息科技有限公司 Vision-based 360-degree parking assist intelligent guiding system
CN105745122A (en) * 2013-09-23 2016-07-06 大众汽车有限公司 Driver assistance system for displaying surroundings of a vehicle
CN107298096A (en) * 2016-04-15 2017-10-27 株式会社万都 Driving assistance device
CN108275143A (en) * 2016-12-30 2018-07-13 现代自动车株式会社 Automated parking system and automatic parking method
CN108528441A (en) * 2018-03-27 2018-09-14 广州汽车集团股份有限公司 It parks method and device

Families Citing this family (52)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102005030838A1 (en) * 2005-07-01 2007-01-04 Siemens Ag Night Vision System
US10878646B2 (en) 2005-12-08 2020-12-29 Smartdrive Systems, Inc. Vehicle event recorder systems
US8996240B2 (en) 2006-03-16 2015-03-31 Smartdrive Systems, Inc. Vehicle event recorders with integrated web server
US9201842B2 (en) 2006-03-16 2015-12-01 Smartdrive Systems, Inc. Vehicle event recorder systems and networks having integrated cellular wireless communications systems
US8649933B2 (en) 2006-11-07 2014-02-11 Smartdrive Systems Inc. Power management systems for automotive video event recorders
US8989959B2 (en) 2006-11-07 2015-03-24 Smartdrive Systems, Inc. Vehicle operator performance history recording, scoring and reporting systems
US8868288B2 (en) 2006-11-09 2014-10-21 Smartdrive Systems, Inc. Vehicle exception event management systems
US8239092B2 (en) 2007-05-08 2012-08-07 Smartdrive Systems Inc. Distributed vehicle event recorder systems having a portable memory data transfer system
US20100328464A1 (en) * 2009-06-30 2010-12-30 Tech-Cast Mfg. Corp. Car-mount traffic condition surveillance device
JP5696872B2 (en) * 2010-03-26 2015-04-08 アイシン精機株式会社 Vehicle periphery monitoring device
US20120056995A1 (en) * 2010-08-31 2012-03-08 Texas Instruments Incorporated Method and Apparatus for Stereo-Based Proximity Warning System for Vehicle Safety
US8766818B2 (en) 2010-11-09 2014-07-01 International Business Machines Corporation Smart spacing allocation
US9926008B2 (en) 2011-04-19 2018-03-27 Ford Global Technologies, Llc Trailer backup assist system with waypoint selection
US20150197281A1 (en) * 2011-04-19 2015-07-16 Ford Global Technologies, Llc Trailer backup assist system with lane marker detection
US9854209B2 (en) 2011-04-19 2017-12-26 Ford Global Technologies, Llc Display system utilizing vehicle and trailer dynamics
EP2723069B1 (en) * 2011-06-16 2017-10-18 Aisin Seiki Kabushiki Kaisha Vehicle periphery monitoring device
KR20130019639A (en) * 2011-08-17 2013-02-27 엘지이노텍 주식회사 Camera apparatus of vehicle
WO2013042686A1 (en) * 2011-09-22 2013-03-28 日産自動車株式会社 Vehicle control apparatus
US8996234B1 (en) 2011-10-11 2015-03-31 Lytx, Inc. Driver performance determination based on geolocation
US9298575B2 (en) 2011-10-12 2016-03-29 Lytx, Inc. Drive event capturing based on geolocation
US9418552B2 (en) 2011-12-14 2016-08-16 Hi-Park Solutions Ltd. Method and system for automatically locating vacant parking places
KR101478067B1 (en) * 2012-03-21 2015-01-02 주식회사 만도 Smart Parking Assist System of Vehicle and Control Method Thereof
US9598836B2 (en) 2012-03-29 2017-03-21 Harnischfeger Technologies, Inc. Overhead view system for a shovel
KR101877570B1 (en) 2012-04-04 2018-07-11 현대자동차주식회사 Apparatus for setting parking position based on around view image and method thereof
US9728228B2 (en) 2012-08-10 2017-08-08 Smartdrive Systems, Inc. Vehicle event playback apparatus and methods
US9344683B1 (en) * 2012-11-28 2016-05-17 Lytx, Inc. Capturing driving risk based on vehicle state and automatic detection of a state of a location
US9165208B1 (en) * 2013-03-13 2015-10-20 Hrl Laboratories, Llc Robust ground-plane homography estimation using adaptive feature selection
US9501878B2 (en) 2013-10-16 2016-11-22 Smartdrive Systems, Inc. Vehicle event playback apparatus and methods
US9610955B2 (en) 2013-11-11 2017-04-04 Smartdrive Systems, Inc. Vehicle fuel consumption monitor and feedback systems
US8892310B1 (en) 2014-02-21 2014-11-18 Smartdrive Systems, Inc. System and method to detect execution of driving maneuvers
JP6303784B2 (en) * 2014-05-09 2018-04-04 日産自動車株式会社 Parking assistance device and parking assistance method
US9663127B2 (en) 2014-10-28 2017-05-30 Smartdrive Systems, Inc. Rail vehicle event detection and recording system
US11069257B2 (en) 2014-11-13 2021-07-20 Smartdrive Systems, Inc. System and method for detecting a vehicle event and generating review criteria
US20160176340A1 (en) * 2014-12-17 2016-06-23 Continental Automotive Systems, Inc. Perspective shifting parking camera system
KR102227855B1 (en) * 2015-01-22 2021-03-15 현대모비스 주식회사 Parking guide system and method for controlling the same
US9725116B2 (en) 2015-03-27 2017-08-08 Ford Global Technologies, Llc Vehicle and vehicle parking system
US9679420B2 (en) 2015-04-01 2017-06-13 Smartdrive Systems, Inc. Vehicle event recording system and method
IL240831A (en) 2015-08-25 2017-03-30 Elbit Systems Ltd System and method for identifying a deviation of an operator of a vehicle from a doctrine
MY188579A (en) * 2015-10-22 2021-12-22 Nissan Motor Parking space line detection method and device
US10683035B2 (en) 2015-12-08 2020-06-16 Panasonic Intellectual Property Management Co., Ltd. Parking assistance device, parking assistance method, and non-transitory computer readable medium
US10325502B2 (en) 2016-02-10 2019-06-18 Ford Global Technologies, Llc Parallel parking assistant
US10368036B2 (en) * 2016-11-17 2019-07-30 Vivotek Inc. Pair of parking area sensing cameras, a parking area sensing method and a parking area sensing system
CN106740632A (en) * 2016-11-28 2017-05-31 芜湖市吉安汽车电子销售有限公司 A kind of vehicle-mounted reverse image system
KR102046468B1 (en) * 2017-07-26 2019-11-18 엘지전자 주식회사 Side mirror for vehicle
JP6958117B2 (en) * 2017-08-29 2021-11-02 株式会社アイシン Parking support device
US10737725B2 (en) * 2017-09-27 2020-08-11 Gentex Corporation System and method for assisting parallel parking using orthogonal projection
CN108297794B (en) * 2018-01-11 2022-11-18 阿尔派株式会社 Parking assistance device and travel prediction line display method
US10900857B2 (en) * 2018-06-06 2021-01-26 Ford Global Technologies, Llc Methods and systems for fluid leak determination
DE102018131106A1 (en) * 2018-12-06 2020-06-10 Valeo Schalter Und Sensoren Gmbh Method for recognizing a parking area for a vehicle by separately recognizing elements of a parking area marking, computing device and driver assistance system
US10713509B1 (en) 2019-01-24 2020-07-14 Ford Global Technologies, Llc Parking assist system with parking spot occupancy readout
US11301948B2 (en) 2019-04-01 2022-04-12 International Business Machines Corporation Parking continuity with unused duration between automated vehicles
CN113034963B (en) * 2021-03-02 2022-08-02 英博超算(南京)科技有限公司 Vision parking stall tracking system

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7069128B2 (en) * 2004-09-30 2006-06-27 Clarion Co., Ltd. Parking-assist system using image information from an imaging camera and distance information from an infrared laser camera
US20060287825A1 (en) * 1999-06-25 2006-12-21 Fujitsu Ten Limited Vehicle drive assist system
US20080136673A1 (en) * 2006-09-14 2008-06-12 Mando Corporation Method and apparatus for recognizing parking slot by using bird's eye view and parking assist system using the same
WO2009050987A1 (en) * 2007-10-18 2009-04-23 Sanyo Electric Co., Ltd. Camera calibration device and method, and vehicle
CN101425181A (en) * 2008-12-15 2009-05-06 浙江大学 Panoramic view vision auxiliary parking system demarcating method

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4114292B2 (en) * 1998-12-03 2008-07-09 アイシン・エィ・ダブリュ株式会社 Driving support device
JP2001213253A (en) * 2000-01-31 2001-08-07 Equos Research Co Ltd Auxiliary apparatus for parking vehicle
JP2006341641A (en) * 2005-06-07 2006-12-21 Nissan Motor Co Ltd Image display apparatus and image display method
JP2008187564A (en) * 2007-01-31 2008-08-14 Sanyo Electric Co Ltd Camera calibration apparatus and method, and vehicle
JP5182545B2 (en) * 2007-05-16 2013-04-17 アイシン精機株式会社 Parking assistance device
JP5003946B2 (en) * 2007-05-30 2012-08-22 アイシン精機株式会社 Parking assistance device
JP2009129001A (en) * 2007-11-20 2009-06-11 Sanyo Electric Co Ltd Operation support system, vehicle, and method for estimating three-dimensional object area
US20090187300A1 (en) * 2008-01-22 2009-07-23 David Wayne Everitt Integrated vehicle computer system
KR20090088210A (en) * 2008-02-14 2009-08-19 주식회사 만도 Method and apparatus for detecting target parking location by using two reference point and parking assist system using same

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060287825A1 (en) * 1999-06-25 2006-12-21 Fujitsu Ten Limited Vehicle drive assist system
US7069128B2 (en) * 2004-09-30 2006-06-27 Clarion Co., Ltd. Parking-assist system using image information from an imaging camera and distance information from an infrared laser camera
US20080136673A1 (en) * 2006-09-14 2008-06-12 Mando Corporation Method and apparatus for recognizing parking slot by using bird's eye view and parking assist system using the same
WO2009050987A1 (en) * 2007-10-18 2009-04-23 Sanyo Electric Co., Ltd. Camera calibration device and method, and vehicle
CN101425181A (en) * 2008-12-15 2009-05-06 浙江大学 Panoramic view vision auxiliary parking system demarcating method

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103728624A (en) * 2012-10-12 2014-04-16 现代摩比斯株式会社 Parking assist apparatus and parking assist method and parking assist system using the same
CN103728624B (en) * 2012-10-12 2017-09-26 现代摩比斯株式会社 Parking assist apparatus and method and the utilization parking assist apparatus and the parking assisting system of method
CN105745122A (en) * 2013-09-23 2016-07-06 大众汽车有限公司 Driver assistance system for displaying surroundings of a vehicle
CN105745122B (en) * 2013-09-23 2017-10-20 大众汽车有限公司 Driver assistance system for showing vehicle-periphery
CN103723194A (en) * 2013-12-13 2014-04-16 浙江吉利控股集团有限公司 Automobile reverse-driving tire position monitoring device and reverse driving assisting device
CN105355083A (en) * 2015-12-14 2016-02-24 宁波裕兰信息科技有限公司 Vision-based 360-degree parking assist intelligent guiding system
CN107298096A (en) * 2016-04-15 2017-10-27 株式会社万都 Driving assistance device
CN108275143A (en) * 2016-12-30 2018-07-13 现代自动车株式会社 Automated parking system and automatic parking method
CN108275143B (en) * 2016-12-30 2022-12-06 现代自动车株式会社 Automatic parking system and automatic parking method
CN108528441A (en) * 2018-03-27 2018-09-14 广州汽车集团股份有限公司 It parks method and device

Also Published As

Publication number Publication date
US8289189B2 (en) 2012-10-16
WO2010132314A3 (en) 2011-10-06
EP2429877B1 (en) 2020-03-11
CN102458964B (en) 2015-01-28
US20100283633A1 (en) 2010-11-11
JP5620472B2 (en) 2014-11-05
JP2012526707A (en) 2012-11-01
WO2010132314A2 (en) 2010-11-18
EP2429877A2 (en) 2012-03-21

Similar Documents

Publication Publication Date Title
CN102458964B (en) Camera system for use in vehicle parking
KR102052313B1 (en) Device for assisting a driver driving a vehicle or for independently driving a vehicle
US8559674B2 (en) Moving state estimating device
US8319618B2 (en) Image processing apparatus, image processing method, and recording medium
US6330511B2 (en) Danger deciding apparatus for motor vehicle and environment monitoring apparatus therefor
US8988525B2 (en) System and method for providing guidance information to a driver of a vehicle
JP2007228448A (en) Imaging environment recognition apparatus
JP2007210458A (en) Display device for vehicle and image display control method for vehicle
JP2006279752A (en) Undervehicle image display controlling apparatus and its display controlling program
JP5680436B2 (en) Foreign matter adhesion determination device for in-vehicle camera lens
JP4590962B2 (en) Vehicle periphery monitoring device
US20190073541A1 (en) Lane detection system for vehicle
JP2008097279A (en) Vehicle exterior information display device
JP5134608B2 (en) Vehicle periphery display device, vehicle periphery display method and program
CA3020813C (en) Mobile body surroundings display method and mobile body surroundings display apparatus
JP3988551B2 (en) Vehicle perimeter monitoring device
US11100353B2 (en) Apparatus of controlling region of interest of image and method for controlling the same
CN114902309A (en) Driving support device, driving support method, and program
US11967007B2 (en) Vehicle surroundings information displaying system and vehicle surroundings information displaying method
CN112449625B (en) Method, system and trailer combination for assisting in scheduling operation of trailer combination
JP2022056153A (en) Temporary stop detection device, temporary stop detection system, and temporary stop detection program
WO2014090957A1 (en) Method for switching a camera system to a supporting mode, camera system and motor vehicle
JP2023055204A (en) Vehicle recording device
CN118264872A (en) Image pickup setting apparatus and method, image pickup apparatus, vehicle, and storage medium
JP2024023076A (en) Drive recorder device

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
C14 Grant of patent or utility model
GR01 Patent grant