EP2887315B1 - Dispositif de calibration de caméra, procédé pour la mise en oeuvre de la calibration, programme et caméra pour corps mobile - Google Patents
Dispositif de calibration de caméra, procédé pour la mise en oeuvre de la calibration, programme et caméra pour corps mobile Download PDFInfo
- Publication number
- EP2887315B1 EP2887315B1 EP14196761.2A EP14196761A EP2887315B1 EP 2887315 B1 EP2887315 B1 EP 2887315B1 EP 14196761 A EP14196761 A EP 14196761A EP 2887315 B1 EP2887315 B1 EP 2887315B1
- Authority
- EP
- European Patent Office
- Prior art keywords
- point
- image
- straight line
- camera
- coordinate
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims description 59
- 238000009434 installation Methods 0.000 claims description 35
- 238000010586 diagram Methods 0.000 description 20
- 239000000284 extract Substances 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000004590 computer program Methods 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N17/00—Diagnosis, testing or measuring for television systems or their details
- H04N17/002—Diagnosis, testing or measuring for television systems or their details for television cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
Definitions
- This disclosure relates to a calibration technique for calculating installation parameters of a camera on the basis of images captured by the camera installed on a movable body.
- the installation angle of the camera is calculated using at least two vanishing points: a vanishing point (a deep vanishing point) which is calculated using lines parallel to a travelling direction of the vehicle and a vanishing point which is calculated using other lines parallel to another direction.
- the installation angle of the camera with respect to the vehicle is referred to as an installation parameter of the camera.
- a calibration technique utilizing a vanishing point to calculate an installation parameter of the camera there is a technique where two pairs of parallel lines are extracted using a road marking of which graphical features are stored in advance, and then two vanishing points are calculated from the extracted two pairs of parallel lines (see, for example, Japanese Unexamined Patent Application Publication No. 2008-11174 ).
- a technique where a plurality of images are successively captured and accumulated, then an edge is extracted from the accumulated image, whereby a set of lines parallel to the travelling direction of a vehicle is extracted and a deep vanishing point is calculated using the extracted parallel lines (see, for example, International Publication No. 2010/146695 ).
- a non-limiting exemplary embodiment of the present disclosure provides a calibration device, a method for implementing calibration, and a camera for a movable body and a storage medium with a calibration function which are capable of calculating an installation parameter of the camera without storing graphical features of road markings in advance or without requiring another technique.
- an installation parameter of a camera can be calculated without storing graphical features of road markings in advance or without requiring another technique.
- a vehicle is given as an example of a movable body in the Embodiments of this disclosure, the movable body is not limited to a vehicle.
- Fig. 1 is a block diagram illustrating a configuration example of the calibration device according to Embodiment 1 of this disclosure.
- a calibration device 100 includes an acquiring unit 101 which acquires an image captured by a camera 110 installed on a vehicle, an extracting unit 102 which extracts a point through an image process from the image acquired by the acquiring unit 101, and the calculating unit 103 which calculates an installation parameter of a camera using coordinates of the point extracted by the extracting unit 102.
- Fig. 2 is a flowchart illustrating an operation of the calibration device 100.
- Fig. 3 is an explanatory diagram illustrating a situation where the acquiring unit 101 in the calibration device 100 acquires an image from the camera 110.
- a first point a and a second point b are on a motion plane 300.
- a straight line 301 which connects the two points is not parallel to a moving direction 302 of a vehicle.
- the term "motion plane" refers to a flat plane on which a vehicle travels. The motion plane may be inclined from the horizontal.
- the acquiring unit 101 acquires an image captured by the camera 110 from the camera 110, in accordance with an externally provided image acquiring signal (not illustrated).
- an image acquiring signal is input to the acquiring unit 101 at a first image acquiring position illustrated in Fig. 3 .
- the acquiring unit 101 acquires an image (a first image), which is an image including the first point a and the second point b captured by the camera 110 at the first image acquiring position (Step S201).
- Fig. 4 is an explanatory diagram of the first image acquired by the acquiring unit 101 in the Step S201.
- a point a1 corresponds to the first point a in Fig. 3 and a point b1 corresponds to the second point b in Fig. 3 .
- the image acquired by the acquiring unit 101 is input to the extracting unit 102.
- the extracting unit 102 extracts the first point a and the second point b from the input image through an image process, such as a comparison of pixel brightness in the image.
- the extracting unit 102 calculates coordinates of the first point a (first coordinates) and coordinates of the second point b (second coordinates) in the first image (Step S202).
- the image acquiring signal is input to the acquiring unit 101 and the acquiring unit 101 acquires an image (a second image), which is an image including the first point a and the second point b captured by the camera 110 at the second image acquiring position (Step S203).
- Fig. 5 is an explanatory diagram of the second image acquired by the acquiring unit 101 in the Step S203.
- a point a2 corresponds to the first point a in Fig. 3 and a point b2 corresponds to the second point b in Fig. 3 .
- the image acquired by the acquiring unit 101 is input to the extracting unit 102.
- the extracting unit 102 performs a similar process to that in the Step S202 and calculates coordinates of the first point a (third coordinates) and coordinates of the second point b (fourth coordinates) in the second image (Step S204).
- the coordinates of the first point a (the coordinates of the point a1 in Fig. 4 ) and the coordinates of the second point b (the coordinates of the point b1 in Fig. 4 ), which are extracted in the Step S202, and the coordinates of the first point a (the coordinates of the point a2 in Fig. 5 ) and the coordinates of the second point b (the coordinates of the point b2 in Fig. 5 ), which are extracted in the Step S204, are input to the calculating unit 103.
- the calculating unit 103 calculates an installation parameter of a camera on the basis of a positional relationship between the coordinates of the point a1 and the coordinates of the point b1, a positional relationship between the coordinates of the point a2 and the coordinates of the point b2, a positional relationship between the coordinates of the point a1 and the coordinates of the point a2, and a positional relationship between the coordinates of the point b1 and the coordinates of the point b2.
- the calculating unit 103 calculates straight lines from the coordinates of the point a1, the point b1, the point a2, and the point b2 which are extracted by the extracting unit 102.
- the point a1, the point b1, the point a2, and the point b2, which are extracted by the extracting unit 102 are illustrated in one image as an example.
- Fig. 6 is a diagram for illustrating a process of the calculating unit 103.
- the calculating unit 103 calculates a straight line 610 which passes through the coordinates of the point a1 and the point a2, a straight line 611 which passes through the coordinates of the point b1 and the point b2, a straight line 620 which passes through the coordinates of the point a1 and the point b1, and a straight line 621 which passes through the coordinates of the point a2 and the point b2 (Step S205).
- the calculating unit 103 calculates an installation parameter of a camera on the basis of the positional relationships of the straight line 610, the straight line 611, the straight line 620, and the straight line 621 (the Step S206) and outputs a calculation result (Step S207).
- Fig. 7 is a flowchart illustrating a detailed operation of a camera installation parameter calculating process in the Step S206.
- the calculating unit 103 compares the inclination of the straight line 610 with that of the straight line 611 (Step S701) and the inclination of the straight line 620 with that of the straight line 621 (Step S702 or Step S703), and then the camera installation parameter calculating process is executed on the basis of the positional relationships of the straight lines.
- Described hereinafter are examples of the camera installation parameter calculating process depending on whether the inclination of the straight line 610 is equal to or not equal to the inclination of the straight line 611 and whether the inclination of the straight line 620 is equal to or not equal to the inclination of the straight line 621.
- the calculating unit 103 calculates an intersection of the straight line 610 and the straight line 611 as a deep vanishing point (the Step S704). In addition, the calculating unit 103 calculates an intersection of the straight line 620 and the straight line 621 as a vanishing point (the Step S705). Then, the calculating unit 103 calculates a straight line which passes through the deep vanishing point and the vanishing point as a vanishing line (the Step S706).
- the calculating unit 103 calculates a camera installation parameter using a focal length of the camera 110 on the basis of the positional relationship between the coordinates of the point a1 and the coordinates of the point b1, the positional relationship between the coordinates of the point a2 and the coordinates of the point b2, the positional relationship between the coordinates of the point a1 and the coordinates of the point a2, and the positional relationship between the coordinates of the point b1 and the coordinates of the point b2.
- definitions of the camera installation parameter are as follows: the term “roll angle” refers to a rotation angle whose axis is an optical axis of the camera, the term “pitch angle” refers to a rotation angle whose axis is parallel to the motion plane and perpendicular to the optical axis of the camera, and the term “yawing angle” refers to a rotation angle whose axis is perpendicular to the motion plane.
- the original point is in the upper left, the X-axis extends rightward and the Y-axis extends downward.
- the calculating unit 103 calculates an angle between a straight line parallel to the X-axis and the vanishing line calculated in the Step S706 as the roll angle (the Step S707). Then, the calculating unit 103 calculates the pitch angle using the focal length of the camera 110, the coordinates of the vanishing point calculated in the Step S705, and the roll angle calculated in the Step S707 (the Step S708). Finally, the calculating unit 103 calculates the yawing angle using the focal length of the camera 110, the coordinates of the deep vanishing point calculated in the Step S704, the vanishing line calculated in the Step S706, and the pitch angle calculated in the Step S708 (the Step S709).
- the calibration device can obtain the deep vanishing point and the vanishing point at the same time and then calculate the camera installation parameter using the vanishing line which passes through the deep vanishing point and the vanishing point.
- Step S710 to Step S714 A case where the inclination of the straight line 610 is not equal to the inclination of the straight line 611 and the inclination of the straight line 620 is equal to the inclination of the straight line 621 (Step S710 to Step S714)
- the calculating unit 103 calculates an intersection of the straight line 610 and the straight line 611 as a deep vanishing point (the Step S710). Then, the calculating unit 103 calculates a straight line which passes through the deep vanishing point and is parallel to the straight line 620 and the straight line 621 as a vanishing line (the Step S711).
- the calculating unit 103 calculates an angle between a straight line parallel to the X-axis and the vanishing line calculated in the Step S711 as the roll angle (the Step S712). Then, the calculating unit 103 calculates the pitch angle using the focal length of the camera 110, the coordinates of the deep vanishing point calculated in the Step S710, and the roll angle calculated in the Step S712 (the Step S713). Finally, the calculating unit 103 calculates the yawing angle using the focal length of the camera 110, the coordinates of the deep vanishing point calculated in the Step S710, the vanishing line calculated in the Step S711, and the pitch angle calculated in the Step S713 (the Step S714). (3) A case where the inclination of the straight line 610 is equal to the inclination of the straight line 611 and the inclination of the straight line 620 is not equal to the inclination of the straight line 621 (Step S715 to Step S719)
- the calculating unit 103 calculates an intersection of the straight line 620 and the straight line 621 as a vanishing point (the Step S715). Then, the calculating unit 103 calculates a straight line which passes through the vanishing point and is parallel to the straight line 610 and the straight line 611 as a vanishing line (the Step S716).
- the calculating unit 103 calculates an angle between a straight line parallel to the X-axis and the vanishing line calculated in the Step S716 as the roll angle (the Step S717). Then, the calculating unit 103 calculates the pitch angle using the focal length of the camera 110, the coordinates of the vanishing point calculated in the Step S715, and the roll angle calculated in the Step S717 (the Step S718).
- the calculating unit 103 calculates the yawing angle using the focal length of the camera 110, the coordinates of a deep vanishing point which is defined to lie at an infinite distance along the vanishing line calculated in the Step S716 in the direction from the point a1 toward the point a2, the vanishing line calculated in the Step S716, and the pitch angle calculated in the Step S718 (the Step S719).
- the inclination of the straight line 610 is equal to the inclination of the straight line 611 and the inclination of the straight line 620 is equal to the inclination of the straight line 621 (Step S720 to Step S722)
- FIG. 11 An example of the positional relationship of the point a1, the point b1, the point a2, and the point b2 is illustrated in Fig. 11 .
- the calculating unit 103 defines the pitch angle as 90 degrees (the Step S720). Further, when the pitch angle is 90 degrees, it is impossible to distinguish changes in the roll angle from changes in the yawing angle; therefore, the roll angle is defined as 0 degrees in this case (the Step S721).
- the calculating unit 103 calculates an angle 1101 formed by a straight line 1100 parallel to the Y-axis, the straight line 610, and the straight line 611, as the yawing angle (the Step S722).
- two points on the motion plane are used in the above description, two points on another plane parallel to the motion plane, such as tops of poles having the same height, may be similarly used to obtain the camera installation parameter.
- the calibration device according to Embodiment 1 of this disclosure may be realized by using dedicated hardware.
- the calibration device may be realized by storing a program which executes the function in a computer-readable recording medium and making a computer read and execute the program.
- Such a storage medium may be mounted on the camera 110 to provide a camera having a calibration function.
- the first point a and the second point b on the motion plane 300 may be markers having a specific shape and color to enhance the process accuracy of the extracting unit 102.
- the distance between the first image acquiring position and the second image acquiring position is not particularly limited, as long as both the first point a and the second point b are captured in each of the first image and the second image.
- the distance may be 50 cm or 10 m.
- the movable body moves in a straight line on a level plane in this embodiment, a perfectly straight trajectory is not essential for this disclosure.
- the calculation accuracy of the camera installation parameter depends on the degree of straightness and levelness. Therefore, the degree of straightness and levelness is not limited, as long as the calculation accuracy of the camera installation parameter reaches a certain degree required by an application or the like; for example, the movable body may zigzag or move over an uneven or curved surface.
- a straight line which passes through two feature points on an image captured before the vehicle moves straight and a straight line which passes through two feature points on an image captured after the vehicle moves straight, which correspond to the two points before the vehicle moves straight are parallel to each other in the real space.
- a straight line which passes through one of the two feature points before the vehicle moves straight and the corresponding feature point after the vehicle moves straight are also parallel to a straight line which passes through another of the two feature points before the vehicle moves straight and the corresponding feature point after the vehicle moves straight in the real space.
- the installation parameter of a camera can be calculated using the calibration device according to Embodiment 1 of this disclosure, without storing graphical features of road markings or the like in advance or without using another technique.
- Fig. 12 is a block diagram illustrating a configuration example of the calibration device according to Embodiment 2 of this disclosure.
- a calibration device 1200 includes an acquiring unit 1201 which acquires an image captured by the camera 110 installed on a vehicle, an extracting unit 1202 which extracts a point through an image process from the image acquired by the acquiring unit 1201, a calculating unit 1203 which calculates an installation parameter of a camera using coordinates of the point extracted by the extracting unit 1202, and a driving status determining unit 1204 which determines a driving status of the vehicle.
- the acquiring unit 1201 acquires an image captured by the camera 110 from the camera 110, in accordance with an externally provided image acquiring signal.
- the extracting unit 1202 extracts a feature point from the image and performs an image process to extract corresponding feature points from different images by, for example, block matching.
- the calculating unit 1203 calculates an installation parameter of a camera on the basis of the corresponding feature points in different images which are output from the extracting unit 1202.
- the driving status determining unit 1204 determines a driving status of the vehicle on the basis of driving distance information, travelling direction information, and motion surface information, which are input from the vehicle.
- driving distance information refers to information for determining whether the vehicle moves a certain distance or not. Such information is, for example, provided by a driving distance meter of the vehicle.
- traveling direction information refers to information for determining whether the vehicle moves straight or not. Such information is, for example, provided by a steering angle sensor of the vehicle.
- motion surface information refers to information for determining whether the motion surface is plane or not. Such information is, for example, provided by an angle sensor, a gyro, or the like of the vehicle.
- Fig. 13 is a flowchart illustrating an operation of the calibration device 1200 according to Embodiment 2 of this disclosure.
- the driving status determining unit 1204 determines whether the status is suitable for executing calibration or not (Step S1301). In other words, the driving status determining unit 1204 determines whether the image which the acquiring unit 1201 acquires from the camera 110 is an image capturing the motion plane or not. Specifically, the driving status determining unit 1204 determines whether a driving condition 1, "the vehicle drives a certain distance, such as 5 meters, and the moving surface information does not change during the drive" is satisfied or not. This determination step is repeated until the driving condition 1 is satisfied. When the driving condition 1 is satisfied, an input image process is then performed (the Step S1302).
- Fig. 14 is a flowchart illustrating details of the input image process of the Step S1302.
- the acquiring unit 1201 acquires a first image (hereinafter, referred to as an image 1) (Step S1401).
- the image acquired by the acquiring unit 1201 is input to the extracting unit 1202.
- the extracting unit 1202 divides the image 1 acquired from the acquiring unit 1201 into a plurality of regions (Step S1402).
- one feature point is extracted from each of the divided regions through an image process (Step S1403).
- Harris method may be employed and a pixel which has the largest value in each divided region is extracted as a feature point of the region.
- Fig. 15 illustrates the image 1 where the image is divided into 16 regions and from which 16 feature points are extracted.
- the feature points are represented by black dots such as the point a1 and the point b1.
- the acquiring unit 1201 acquires at least one image (hereinafter, referred to as a sub-image) from the camera 110 and the extracting unit 1202 performs an image process to extract feature points from the sub-image which correspond to those extracted from the image 1 by, for example, block matching (Step S1404 to Step S1406).
- a sub-image at least one image
- the extracting unit 1202 performs an image process to extract feature points from the sub-image which correspond to those extracted from the image 1 by, for example, block matching (Step S1404 to Step S1406).
- the driving status determining unit 1204 determines whether the vehicle drives the predetermined distance or not.
- the extracting unit 1202 defines the last sub-image which is acquired by the acquiring unit 1201, as the image 2 (Step S1407), and outputs the feature points extracted from the image 1 and the feature points extracted from the image 2 to the calculating unit 1203 (Step S1408), whereby the input image process ends.
- Fig. 16 illustrates the image 2 obtained through the input image process.
- White squares in Fig. 16 represent the feature points corresponding to the feature points extracted from the image 1.
- the feature point a2 corresponds to the point a1 in the image 1
- the feature point b2 corresponds to the point b1 in the image 1.
- Each one of the feature points in Fig. 16 corresponds to respective feature point in Fig. 15 .
- the driving status determining unit 1204 determines whether to start a vanishing line calculating process or not (Step S1303). In other words, whether the vehicle moves straight over the motion plane after the driving condition 1 is satisfied or not is determined. Specifically, whether a driving condition 2 "the motion surface information does not change from the value at the time when the driving condition 1 is satisfied and a value of the steering angle sensor shows the vehicle moves straight after the driving condition 1 is satisfied" is satisfied or not is determined.
- Step S1304 When the driving condition 2 is not satisfied, the process returns to the determination of whether the driving condition 1 is satisfied or not.
- the calculating unit 1203 then performs a vanishing line calculating process (Step S1304).
- Figs. 17, 18 , and 19 illustrate an example of the vanishing line calculating process.
- Fig. 17 the feature points of the image 1 and the feature points of the image 2 corresponding to the feature points of the image 1 are illustrated in one image.
- the calculating unit 1203 calculates straight lines each of which passes through one of the feature points in the image 1 ( Fig. 15 ) and one of the feature points in the image 2 ( Fig. 16 ) corresponding to the one feature point in the image 1 and calculates the intersection of the straight lines as a deep vanishing point.
- a white circle represents the calculated deep vanishing point.
- the calculating unit 1203 selects two feature points in the image 1 and calculates a straight line which passes through the two feature points.
- the calculating unit 1203 also calculates a straight line which passes through two feature points in the image 2 corresponding to the two feature points selected in the image 1. Then, the calculating unit 1203 calculates the intersection of the straight line calculated in the image 1 and the straight line calculated in the image 2 as a vanishing point. This process is performed for all pairs of the feature points. In Fig. 19 , triangles represent the calculated vanishing points.
- the calculating unit 1203 calculates a straight line which passes through the deep vanishing point and the largest number of the vanishing points as the vanishing line, using Hough transform.
- Fig. 20 is an explanatory diagram illustrating the calculated deep vanishing point, the calculated vanishing points, and the vanishing line calculated through Hough transform.
- the vanishing points on the vanishing line in Fig. 20 are calculated from feature points which are on the motion plane or on a plane parallel to the motion plane and which are located in a manner such that the line segment connecting the feature points is not parallel to the moving direction of the vehicle.
- the calculating unit 1203 calculates a roll angle, a pitch angle, and a yawing angle, which are camera installation parameters, using the coordinates of the deep vanishing point, the vanishing line, and the focal length of the camera 110, as described in Embodiment 1 (Step S1305).
- the calculating unit 1203 calculates an angle between a straight line parallel to the X-axis and the vanishing line calculated in the Step S1304 as a roll angle. Then, the calculating unit 1203 calculates a pitch angle using the focal length of the camera 110, the coordinates of the deep vanishing point, and the roll angle. Finally, the calculating unit 1203 calculates a yawing angle using the focal length of the camera 110, the coordinates of the deep vanishing point, the vanishing line calculated in the Step S1304, and the pitch angle.
- the driving status determining unit 1204 may calculate the driving distance, the travelling direction, and the levelness of the motion surface on the basis of information from the GPS (global positioning system).
- the calibration device 1200 may include a CPU (central processing unit), a RAM (random access memory), and a ROM (read only memory) and be realized as a program stored in a CPU-readable storage medium performing processes of the units.
- a CPU central processing unit
- RAM random access memory
- ROM read only memory
- the calculating unit 1203 may calculate the vanishing points only in a part of the image which captures the motion plane.
- the calculating unit 1203 illustrated in Fig. 12 may be mounted on the camera to provide a camera having a calibration function.
- the calibration device can calculate a camera installation even when the motion plane in the image is not identified.
- This disclosure can be applied to, for example, an on-vehicle camera which is installed on a vehicle and assists driving.
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Biomedical Technology (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Image Analysis (AREA)
- Studio Devices (AREA)
- Image Processing (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
Claims (4)
- Dispositif d'étalonnage comprenant:un dispositif d'acquisition (101) conçu pour acquérir une image capturée par une caméra installée sur un corps mobile;un extracteur (102) conçu pour extraire de l'image un premier point (a) et un deuxième point (b);le dispositif d'étalonnage étant caractérisé en ce qu'il comprend en outreune calculatrice (103) conçue pour calculer le paramètre d'installation de la caméra sur la base de (i) une première ligne droite (620) passant par une première coordonnée (a1) du premier point (a) dans une première image capturée à un premier instant et une deuxième coordonnée (b1) du deuxième point (b) dans la première image, (ii) une deuxième ligne droite (621) passant par une troisième coordonnée (a2) du premier point (a) dans une deuxième image capturée à un deuxième instant après que le corps mobile se soit déplacé dans la direction de déplacement après le premier instant et une quatrième coordonnée (b2) du deuxième point (b) dans la deuxième image, (iii) une troisième ligne droite (610) passant à travers la première coordonnée (a1) et la troisième coordonnée (a2), et (iv) une quatrième ligne droite (611) passant par la deuxième coordonnée (b1) et la quatrième coordonnée (b2); dans lequelle premier point (a) et le deuxième point (b) sont situés sur un plan de mouvement du corps mobile ou sur un plan parallèle au plan de mouvement, une ligne droite reliant le premier point (a) au deuxième point (b) étant non parallèle à une direction de déplacement du corps mobile et le plan de mouvement est un plan plat sur lequel le corps mobile se déplace; etau moins l'un du dispositif d'acquisition (101), de l'extracteur (102) et du calculateur (103) est inclus dans un processeur.
- Caméra pour corps mobile comprenant le dispositif d'étalonnage selon la revendication 1.
- Procédé d'étalonnage pour un dispositif d'étalonnage, le procédé comprenant les opérations suivantes:acquérir (S201, S203) une image capturée par une caméra installée sur un corps mobile;extraire (S202, S204) de l'image un premier point (a) et un deuxième point (b);le procédé étant caractérisé en ce qu'il consiste en outre àcalculer (S205) le paramètre d'installation de la caméra sur la base de (i) une première ligne droite (620) passant par une première coordonnée (a1) du premier point (a) dans une première image capturée à un premier instant et une deuxième coordonnée (b1) du deuxième point (b) à partir de la première image, (ii) une deuxième ligne droite (621) passant par une troisième coordonnée (a2) du premier point (a) dans une deuxième image capturée à un deuxième instant après que le corps mobile se soit déplacé dans la direction de déplacement après le premier instant et une quatrième coordonnée (b4) du deuxième point (b) à partir de la deuxième image, (iii) une troisième ligne droite (610) passant à travers la première coordonnée (a1) et la troisième coordonnée (a2), et (iv) une quatrième ligne droite (611) passant par la deuxième coordonnée (b1) et la quatrième coordonnée (b2); dans lequelle premier point (a) et le deuxième point (b) sont situés sur un plan de mouvement du corps mobile ou sur un plan parallèle au plan de mouvement, une ligne droite reliant le premier point (a) au deuxième point (b) étant non parallèle à une direction de déplacement du corps mobile et le plan de mouvement est un plan plat sur lequel le corps mobile se déplace; etau moins l'un des processus d'acquisition (S201, S203), d'extraction (S202, S204) et de calcul (S205) est effectué par un circuit.
- Support d'enregistrement lisible par ordinateur stockant un programme qui conduit un ordinateur à exécuter le procédé selon la revendication 3.
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2013263698A JP6201148B2 (ja) | 2013-12-20 | 2013-12-20 | キャリブレーション装置、キャリブレーション方法、キャリブレーション機能を備えた移動体搭載用カメラ及びプログラム |
Publications (2)
Publication Number | Publication Date |
---|---|
EP2887315A1 EP2887315A1 (fr) | 2015-06-24 |
EP2887315B1 true EP2887315B1 (fr) | 2019-03-27 |
Family
ID=52015945
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP14196761.2A Active EP2887315B1 (fr) | 2013-12-20 | 2014-12-08 | Dispositif de calibration de caméra, procédé pour la mise en oeuvre de la calibration, programme et caméra pour corps mobile |
Country Status (3)
Country | Link |
---|---|
US (1) | US9336595B2 (fr) |
EP (1) | EP2887315B1 (fr) |
JP (1) | JP6201148B2 (fr) |
Families Citing this family (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP6326624B2 (ja) * | 2014-04-02 | 2018-05-23 | パナソニックIpマネジメント株式会社 | キャリブレーション装置、キャリブレーション方法、キャリブレーション機能を備えたカメラ及びプログラム |
DE102016104729A1 (de) * | 2016-03-15 | 2017-09-21 | Connaught Electronics Ltd. | Verfahren zur extrinsischen Kalibrierung einer Kamera, Rechenvorrichtung, Fahrerassistenzsystem sowie Kraftfahrzeug |
US10008000B2 (en) * | 2016-05-18 | 2018-06-26 | Conduent Business Services, Llc | Camera calibration based on moving vehicle line segments |
US20180068459A1 (en) * | 2016-09-08 | 2018-03-08 | Ford Global Technologies, Llc | Object Distance Estimation Using Data From A Single Camera |
JP6536529B2 (ja) * | 2016-10-17 | 2019-07-03 | 株式会社デンソー | 車載カメラのキャリブレーション装置及び車載カメラのキャリブレーション方法 |
JP6905173B2 (ja) * | 2016-10-31 | 2021-07-21 | キヤノンマーケティングジャパン株式会社 | 情報処理システム、情報処理方法 |
JP6782895B2 (ja) * | 2017-03-09 | 2020-11-11 | 国立研究開発法人農業・食品産業技術総合研究機構 | 車載カメラの取付方向パラメータ算出装置および取付方向パラメータ算出方法 |
EP3392801A1 (fr) * | 2017-04-21 | 2018-10-24 | Harman International Industries, Incorporated | Systèmes et procédés d'assistance au conducteur |
WO2019225681A1 (fr) * | 2018-05-23 | 2019-11-28 | パナソニックIpマネジメント株式会社 | Dispositif d'étalonnage et procédé d'étalonnage |
JP2020051932A (ja) * | 2018-09-27 | 2020-04-02 | 京セラ株式会社 | 較正装置、カメラ、移動体、および較正値調整方法 |
CN112132902B (zh) * | 2019-06-24 | 2024-01-16 | 上海安亭地平线智能交通技术有限公司 | 车载摄像头外参调整方法、装置、电子设备以及介质 |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20030144813A1 (en) * | 2002-01-31 | 2003-07-31 | Canon Kabushiki Kaisha | Position and orientation determination method and apparatus and storage medium |
EP1352363A1 (fr) * | 2000-12-21 | 2003-10-15 | Robert Bosch Gmbh | Procede et dispositif permettant de compenser un defaut d'alignement d'un dispositif de production d'images |
EP2405399A1 (fr) * | 2010-07-06 | 2012-01-11 | Imajing | Méthode de calibration de l'orientation d'une caméra vidéo embarquée |
Family Cites Families (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6192145B1 (en) * | 1996-02-12 | 2001-02-20 | Sarnoff Corporation | Method and apparatus for three-dimensional scene processing using parallax geometry of pairs of points |
JP4820221B2 (ja) | 2006-06-29 | 2011-11-24 | 日立オートモティブシステムズ株式会社 | 車載カメラのキャリブレーション装置およびプログラム |
JP4870546B2 (ja) * | 2006-12-27 | 2012-02-08 | 株式会社岩根研究所 | レイヤー生成・選択機能を備えたcvタグ映像表示装置 |
US8373763B2 (en) * | 2008-05-22 | 2013-02-12 | GM Global Technology Operations LLC | Self calibration of extrinsic camera parameters for a vehicle camera |
WO2010146695A1 (fr) | 2009-06-18 | 2010-12-23 | 富士通株式会社 | Dispositif de traitement d'image et procédé de traitement d'image |
US8866901B2 (en) * | 2010-01-15 | 2014-10-21 | Honda Elesys Co., Ltd. | Motion calculation device and motion calculation method |
KR101758058B1 (ko) * | 2011-01-20 | 2017-07-17 | 삼성전자주식회사 | 깊이 정보를 이용한 카메라 모션 추정 방법 및 장치, 증강 현실 시스템 |
JP6141601B2 (ja) * | 2012-05-15 | 2017-06-07 | 東芝アルパイン・オートモティブテクノロジー株式会社 | 車載カメラ自動キャリブレーション装置 |
-
2013
- 2013-12-20 JP JP2013263698A patent/JP6201148B2/ja not_active Expired - Fee Related
-
2014
- 2014-12-08 EP EP14196761.2A patent/EP2887315B1/fr active Active
- 2014-12-10 US US14/565,941 patent/US9336595B2/en active Active
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP1352363A1 (fr) * | 2000-12-21 | 2003-10-15 | Robert Bosch Gmbh | Procede et dispositif permettant de compenser un defaut d'alignement d'un dispositif de production d'images |
US20030144813A1 (en) * | 2002-01-31 | 2003-07-31 | Canon Kabushiki Kaisha | Position and orientation determination method and apparatus and storage medium |
EP2405399A1 (fr) * | 2010-07-06 | 2012-01-11 | Imajing | Méthode de calibration de l'orientation d'une caméra vidéo embarquée |
Also Published As
Publication number | Publication date |
---|---|
JP6201148B2 (ja) | 2017-09-27 |
EP2887315A1 (fr) | 2015-06-24 |
US9336595B2 (en) | 2016-05-10 |
US20150178922A1 (en) | 2015-06-25 |
JP2015122547A (ja) | 2015-07-02 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP2887315B1 (fr) | Dispositif de calibration de caméra, procédé pour la mise en oeuvre de la calibration, programme et caméra pour corps mobile | |
EP3407294B1 (fr) | Procédé, dispositif et terminal de traitement d'informations | |
CN110807350B (zh) | 用于面向扫描匹配的视觉slam的***和方法 | |
CN104833370B (zh) | 用于映射、定位和位姿校正的***和方法 | |
CN112292711A (zh) | 关联lidar数据和图像数据 | |
CN107735797B (zh) | 用于确定第一坐标系与第二坐标系之间的运动的方法 | |
JP6299291B2 (ja) | 道路エッジ検出方法及び道路エッジ検出装置 | |
CN107844750A (zh) | 一种水面全景图像目标检测识别方法 | |
CN104700414A (zh) | 一种基于车载双目相机的前方道路行人快速测距方法 | |
CN108597009B (zh) | 一种基于方向角信息进行三维目标检测的方法 | |
CN110930459A (zh) | 灭点提取方法、相机标定方法以及存储介质 | |
EP3686775B1 (fr) | Procédé de détection d'une boîte de limitation en pseudo-3d à base de cnn pouvant convertir des modes en fonction de poses d'objets à l'aide de segmentation d'instances | |
CN111612728B (zh) | 一种基于双目rgb图像的3d点云稠密化方法和装置 | |
JP7077910B2 (ja) | 区画線検出装置及び区画線検出方法 | |
Fiala et al. | Visual odometry using 3-dimensional video input | |
CN113393524B (zh) | 一种结合深度学习和轮廓点云重建的目标位姿估计方法 | |
CN111738033A (zh) | 基于平面分割的车辆行驶信息确定方法及装置、车载终端 | |
Rehder et al. | Submap-based SLAM for road markings | |
Manivannan et al. | Vision based intelligent vehicle steering control using single camera for automated highway system | |
CN104166995A (zh) | 一种基于马步测度的Harris-SIFT双目视觉定位方法 | |
JPH07103715A (ja) | 視覚に基く三次元位置および姿勢の認識方法ならびに視覚に基く三次元位置および姿勢の認識装置 | |
CN106682584B (zh) | 无人机障碍物检测方法及装置 | |
KR102003387B1 (ko) | 조감도 이미지를 이용한 교통 장애물의 검출 및 거리 측정 방법, 교통 장애물을 검출하고 거리를 측정하는 프로그램을 저장한 컴퓨터 판독가능 기록매체 | |
Kiran et al. | Automatic hump detection and 3D view generation from a single road image | |
CN115239902A (zh) | 移动设备的周边地图建立方法、装置、设备及存储介质 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
17P | Request for examination filed |
Effective date: 20141208 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
R17P | Request for examination filed (corrected) |
Effective date: 20151201 |
|
RBV | Designated contracting states (corrected) |
Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
17Q | First examination report despatched |
Effective date: 20171201 |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
INTG | Intention to grant announced |
Effective date: 20180323 |
|
GRAJ | Information related to disapproval of communication of intention to grant by the applicant or resumption of examination proceedings by the epo deleted |
Free format text: ORIGINAL CODE: EPIDOSDIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
INTC | Intention to grant announced (deleted) | ||
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
INTG | Intention to grant announced |
Effective date: 20180910 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE PATENT HAS BEEN GRANTED |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602014043556 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: REF Ref document number: 1113947 Country of ref document: AT Kind code of ref document: T Effective date: 20190415 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190327 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190327 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190327 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190627 |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20190327 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190327 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190628 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190327 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190327 Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190327 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190627 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 1113947 Country of ref document: AT Kind code of ref document: T Effective date: 20190327 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: AL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190327 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190327 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190327 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190327 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190327 Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190327 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190327 Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190727 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190327 Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190327 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190727 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190327 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602014043556 Country of ref document: DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190327 |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190327 |
|
26N | No opposition filed |
Effective date: 20200103 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190327 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
REG | Reference to a national code |
Ref country code: BE Ref legal event code: MM Effective date: 20191231 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190327 |
|
GBPC | Gb: european patent ceased through non-payment of renewal fee |
Effective date: 20191208 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: GB Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20191208 Ref country code: FR Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20191231 Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20191208 Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20191208 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20191231 Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20191231 Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20191231 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190327 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190327 Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20141208 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190327 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: DE Payment date: 20231214 Year of fee payment: 10 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R081 Ref document number: 602014043556 Country of ref document: DE Owner name: PANASONIC AUTOMOTIVE SYSTEMS CO., LTD., YOKOHA, JP Free format text: FORMER OWNER: PANASONIC INTELLECTUAL PROPERTY MANAGEMENT CO., LTD., OSAKA, JP |