CN111861891A - Method for realizing panoramic image system picture splicing display based on checkerboard calibration - Google Patents

Method for realizing panoramic image system picture splicing display based on checkerboard calibration Download PDF

Info

Publication number
CN111861891A
CN111861891A CN202010668193.3A CN202010668193A CN111861891A CN 111861891 A CN111861891 A CN 111861891A CN 202010668193 A CN202010668193 A CN 202010668193A CN 111861891 A CN111861891 A CN 111861891A
Authority
CN
China
Prior art keywords
calibration
coordinate
pixel
point
points
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202010668193.3A
Other languages
Chinese (zh)
Inventor
王瑞琳
马文峰
节忠海
王亮
白一迪
李成浩
黄晓慧
赵一
杨冉
张诗宇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
FAW Bestune Car Co Ltd
Original Assignee
FAW Bestune Car Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by FAW Bestune Car Co Ltd filed Critical FAW Bestune Car Co Ltd
Priority to CN202010668193.3A priority Critical patent/CN111861891A/en
Publication of CN111861891A publication Critical patent/CN111861891A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00Geometric image transformations in the plane of the image
    • G06T3/60Rotation of whole images or parts thereof
    • G06T3/604Rotation of whole images or parts thereof using coordinate rotation digital computer [CORDIC] devices
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00Geometric image transformations in the plane of the image
    • G06T3/40Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4038Image mosaicing, e.g. composing plane images from plane sub-images
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/10Segmentation; Edge detection
    • G06T7/13Edge detection
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/80Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/90Determination of colour characteristics
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00Indexing scheme for image data processing or generation, in general
    • G06T2200/32Indexing scheme for image data processing or generation, in general involving image mosaicing

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Processing (AREA)

Abstract

The invention belongs to the technical field of automotive electronics, and particularly relates to a method for realizing splicing display of panoramic image system pictures based on checkerboard calibration. The method can enable a driver to master the situation around the automobile body through real-time display of scenes in a certain range of 360-degree panorama in the external space of the automobile, greatly reduces blind areas, improves the degree of safe driving, reduces the driving technology required to be mastered by the driver, meanwhile, through simulation of the automobile on a screen, the driver can quickly master the states of the automobile (such as opening and closing doors, steering lamps and the like), the driving safety is guaranteed, the driver can obtain better experience, and the problem of visual blind areas existing in the situation that information around the automobile body is obtained through rearview mirrors or backing images is solved.

Description

Method for realizing panoramic image system picture splicing display based on checkerboard calibration
Technical Field
The invention belongs to the technical field of automotive electronics, and particularly relates to a method for realizing splicing display of panoramic image system pictures based on checkerboard calibration.
Background
With the development of science and technology and the improvement of the living standard of people, automobiles become indispensable tools for riding instead of walk in the life of people. In the use process of the vehicle, a driver needs to know the conditions inside the vehicle and the conditions around the outside of the vehicle so as to realize safe driving.
At present, information around a vehicle body is obtained through rearview mirrors or reverse images in the industry, but the method has a very large visual blind area, has high requirements on the level of a driver, seriously influences safe driving, and has high requirements on the technical level of the driver due to low user experience.
Disclosure of Invention
The invention provides a simple method for realizing the splicing display of the camera pictures of the panoramic images of the automobile, which can enable a driver to master the conditions around the automobile body by displaying the scenes in a certain range of 360-degree panorama in the external space of the automobile in real time, greatly reduce blind areas, improve the degree of safe driving and reduce the driving technology which needs to be mastered by the driver.
The technical scheme of the invention is described as follows by combining the attached drawings:
a method for realizing the picture splicing display of a panoramic image system based on checkerboard calibration comprises the following steps:
Step one, after the vehicle stops at the correction position of the calibration platform, turning on a high beam long lamp and pressing a touch screen switch for ten seconds to start calibration;
secondly, respectively capturing a picture from four cameras installed on a vehicle as an original picture for calibration;
thirdly, unfolding the original picture into a 2D aerial view by using the internal reference matrix and the initial external reference matrix, and unfolding the 2D aerial view from the world coordinate to the image coordinate for calibration;
step four, carrying out corner searching on the 2D aerial view;
step five, mapping the found corner points back to the original image and mapping the corner points back to the original image for realizing the mapping relation in the inverse process of the conversion formula from the world coordinate to the image coordinate;
and step six, utilizing the found actual angular points and four black and fast vertexes on the theoretical angular point calibration field to carry out iterative approximation so as to obtain an optimal RT matrix, and completing calibration.
In the third step, the reference matrix is:
assuming that the pixel of the CCD camera after digital discretization is a rectangle, the length and width of the rectangle are dx and dy, respectively, the pixel coordinate is (u, v,1)TI.e. by
Figure BDA0002581227300000021
Figure BDA0002581227300000022
Wherein u is the abscissa of the pixel; v is the ordinate of the pixel; x is the abscissa of the point under the world coordinate; y is the vertical coordinate of the point under the world coordinate; f. of xIs a scale factor of the CCD camera in the direction of the u axis; f. ofyThe scale factor of the CCD camera in the V-axis direction is obtained; u. of0An abscissa which is a point on the pixel coordinate; v. of0Is the ordinate of a point on the pixel coordinate; k is an internal reference matrix.
In step three, the external parameter matrix is:
Figure BDA0002581227300000031
wherein u is the abscissa of the pixel; v is the ordinate of the pixel; x is the number ofwThe horizontal coordinate of the point under the world coordinate; y iswIs the ordinate of the world coordinate lower point; zwVertical coordinates of points under world coordinates; f. ofxIs a scale factor of the CCD camera in the direction of the u axis; f. ofyThe scale factor of the CCD camera in the V-axis direction is obtained; u. of0An abscissa which is a point on the pixel coordinate; v. of0Is the ordinate of a point on the pixel coordinate; f is the focal length of the camera; r is a rotation matrix, and T is a translation matrix; dx is the pixel width and dy is the pixel height. The angular points in the fourth step are points where the black blocks are in contact with each other, 34 points are arranged before and after the calibration cloth, and 42 points are arranged left and right.
The specific method for finding the corner point in the fourth step is as follows:
1) converting the color image into a gray image;
2) binarizing the gray level image by using a self-adaptive threshold value method;
3) detecting a square block in the picture by using edge detection in the binarized picture;
4) Selecting a square meeting the condition from the picture by utilizing the characteristic that the squares on the calibration field are continuously connected together; the square blocks are four points, and the points with the overlapped square blocks are screened out to be the needed angular points.
The invention has the beneficial effects that:
1) the invention is based on checkerboard calibration, the calibration mode is simple, excessive operation is not needed, one-click type fool operation is carried out after the vehicle reaches the designated area, and no threshold is used;
2) the invention has good splicing effect and can lead the driver to obtain better experience;
3) the invention has simple calibration conditions and can complete better calibration without too many working procedures and materials.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the contents of the embodiments of the present invention and the drawings without creative efforts.
FIG. 1 is a flow chart of the present invention;
fig. 2a and 2b are schematic diagrams of basic models of pinhole cameras.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, a method for implementing a tiled display of panoramic image system pictures based on checkerboard calibration includes the following steps:
step one, after the vehicle stops at the correction position of the calibration platform, turning on a high beam long lamp and pressing a touch screen switch for ten seconds to start calibration;
secondly, respectively capturing a picture from four cameras installed on a vehicle as an original picture for calibration;
thirdly, unfolding the original picture into a 2D aerial view by using the internal reference matrix and the initial external reference matrix, and unfolding the 2D aerial view from the world coordinate to the image coordinate for calibration;
referring to fig. 2, the plane pi is referred to as the image plane of the camera, the point Oc is referred to as the center (or optical center) of the camera, f is the focal length of the camera, Oc is the end point and the ray perpendicular to the image plane becomes the optical axis or principal axis, and the intersection point p of the principal axis and the image plane is the principal point of the camera.
The image coordinate system is O-xy, and the camera coordinate system is Oc-xcyczc. Recording space point XcThe homogeneous coordinates in the camera coordinate system are:
Xc=(xc,yc,zc,1)T
its homogeneous coordinate of the image point m in the image coordinate system is recorded as:
m=(x,y,1)T
according to the triangle similarity principle, the following can be obtained:
Figure BDA0002581227300000051
namely:
Figure BDA0002581227300000052
being homogeneous coordinates, it can be written as:
m=PXc
P=diag(f,f,1)(I,0)
the above discussion is a theoretical case, but in practice the principal point may not be the image coordinate system origin (defined herein as the image center). If the principal point has the coordinate of
p=(x0,y0,1)T
Therefore, it is not only easy to use
Figure BDA0002581227300000053
With reference to internal reference matrix
Typically, the image that we acquire is a digital image captured by a CCD camera, which digitally discretizes points in the image plane. Assuming that the pixel of the CCD camera after digital discretization is a rectangle, the length and width of the rectangle are dx and dy, respectively, the pixel coordinate is (u, v,1)TI.e. by
Figure BDA0002581227300000054
Figure BDA0002581227300000061
Wherein u is the abscissa of the pixel; v is the ordinate of the pixel; x is the abscissa of the point under the world coordinate; y is the vertical coordinate of the point under the world coordinate; f. ofxIs a scale factor of the CCD camera in the direction of the u axis; f. ofyThe scale factor of the CCD camera in the V-axis direction is obtained; u. of0An abscissa which is a point on the pixel coordinate; v. of0Is the ordinate of a point on the pixel coordinate; k is an internal reference matrix.
With respect to external reference matrix
We generally describe a three-dimensional point, and we do not describe it based on the camera coordinate system since the camera may be moving all the time. We describe it generally in the world coordinate system. The relationship between the world coordinate system and the camera coordinate system can be described by a rotation matrix R and a translation vector T.
The formula for the transformation from world coordinates to image coordinates:
Figure BDA0002581227300000062
wherein u is the abscissa of the pixel; v is the ordinate of the pixel; x is the number ofwThe horizontal coordinate of the point under the world coordinate; y iswIs the ordinate of the world coordinate lower point; zwVertical coordinates of points under world coordinates; f. ofxIs a scale factor of the CCD camera in the direction of the u axis; f. ofyThe scale factor of the CCD camera in the V-axis direction is obtained; u. of0An abscissa which is a point on the pixel coordinate; v. of0Is the ordinate of a point on the pixel coordinate; f is the focal length of the camera; r is a rotation matrix, and T is a translation matrix; dx is the pixel width and dy is the pixel height.
Step four, carrying out corner searching on the 2D aerial view; the angular points are the points where the black blocks are contacted, and the front and the back of the calibration cloth are 34, and the left and the right are 42. The specific method comprises the following steps:
1) converting the color image into a gray image;
2) binarizing the gray level image by using a self-adaptive threshold value method;
3) Detecting a square block in the picture by using edge detection in the binarized picture;
4) selecting a square meeting the condition from the picture by utilizing the characteristic that the squares on the calibration field are continuously connected together; the square blocks are four points, and the points with the overlapped square blocks are screened out to be the needed angular points.
Step five, mapping the found corner points back to the original image and mapping the corner points back to the original image for realizing the mapping relation in the inverse process of the conversion formula from the world coordinate to the image coordinate;
step six, calibrating four vertexes of the black block on the field by using the found actual angular point and the theoretical angular point (the theoretical angular point is a coordinate value of a point connecting the black block and the black block under an ideal splicing effect in a coordinate system taking the center of the vehicle as an original point), performing iterative approximation (namely, the actual angular point is changed through multiple times of rotation and translation to enable the value of the actual angular point to approach the value of the theoretical angular point), obtaining an optimal RT matrix, and completing calibration
The preferred embodiments of the present invention have been described in detail with reference to the accompanying drawings, however, the present invention is not limited to the specific details of the above embodiments, and various simple modifications can be made to the technical solution of the present invention within the technical idea of the present invention, and these simple modifications are within the protective scope of the present invention.
It should be noted that the various technical features described in the above embodiments can be combined in any suitable manner without contradiction, and the invention is not described in any way for the possible combinations in order to avoid unnecessary repetition.
In addition, any combination of the various embodiments of the present invention is also possible, and the same should be considered as the disclosure of the present invention as long as it does not depart from the spirit of the present invention.

Claims (5)

1. The method for realizing the picture splicing display of the panoramic image system based on checkerboard calibration is characterized by comprising the following steps of:
step one, after the vehicle stops at the correction position of the calibration platform, turning on a high beam long lamp and pressing a touch screen switch for ten seconds to start calibration;
secondly, respectively capturing a picture from four cameras installed on a vehicle as an original picture for calibration;
thirdly, unfolding the original picture into a 2D aerial view by using the internal reference matrix and the initial external reference matrix, and unfolding the 2D aerial view from the world coordinate to the image coordinate for calibration;
step four, carrying out corner searching on the 2D aerial view;
step five, mapping the found corner points back to the original image and mapping the corner points back to the original image for realizing the mapping relation in the inverse process of the conversion formula from the world coordinate to the image coordinate;
And step six, utilizing the found actual angular points and four black and fast vertexes on the theoretical angular point calibration field to carry out iterative approximation so as to obtain an optimal RT matrix, and completing calibration.
2. The method for realizing tiled display of panoramic image system pictures based on checkerboard calibration as claimed in claim 1, wherein said internal reference matrix in step three is:
assuming that the pixel of the CCD camera after digital discretization is a rectangle, the length and width of the rectangle are dx and dy, respectively, the pixel coordinate is (u, v,1)TI.e. by
Figure FDA0002581227290000011
Figure FDA0002581227290000012
Wherein u is the abscissa of the pixel; v is the ordinate of the pixel; x is the abscissa of the point under the world coordinate; y is the vertical coordinate of the point under the world coordinate; f. ofxIs a scale factor of the CCD camera in the direction of the u axis; f. ofyThe scale factor of the CCD camera in the V-axis direction is obtained; u. of0An abscissa which is a point on the pixel coordinate; v. of0Is the ordinate of a point on the pixel coordinate; k is an internal reference matrix.
3. The method for realizing tiled display of panoramic image system pictures based on checkerboard calibration as claimed in claim 1, wherein said external reference matrix in step three is:
Figure FDA0002581227290000021
wherein u is the abscissa of the pixel; v is the ordinate of the pixel; x is the number of wThe horizontal coordinate of the point under the world coordinate; y iswIs the ordinate of the world coordinate lower point; zwVertical coordinates of points under world coordinates; f. ofxIs a scale factor of the CCD camera in the direction of the u axis; f. ofyThe scale factor of the CCD camera in the V-axis direction is obtained; u. of0An abscissa which is a point on the pixel coordinate; v. of0Is the ordinate of a point on the pixel coordinate; f is the focal length of the camera; r is a rotation matrix, and T is a translation matrix; dx is the pixel width and dy is the pixel height.
4. The method for realizing splicing display of panoramic image system pictures based on checkerboard calibration as claimed in claim 1, wherein said corner points in step four are the points where the black blocks are contacted, the number of calibration cloth is 34 before and after, and the number of calibration cloth is 42.
5. The method for realizing splicing display of panoramic image system pictures based on checkerboard calibration as claimed in claim 1, wherein the specific method for finding the corner in step four is as follows:
1) converting the color image into a gray image;
2) binarizing the gray level image by using a self-adaptive threshold value method;
3) detecting a square block in the picture by using edge detection in the binarized picture;
4) selecting a square meeting the condition from the picture by utilizing the characteristic that the squares on the calibration field are continuously connected together; the square blocks are four points, and the points with the overlapped square blocks are screened out to be the needed angular points.
CN202010668193.3A 2020-07-13 2020-07-13 Method for realizing panoramic image system picture splicing display based on checkerboard calibration Pending CN111861891A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010668193.3A CN111861891A (en) 2020-07-13 2020-07-13 Method for realizing panoramic image system picture splicing display based on checkerboard calibration

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010668193.3A CN111861891A (en) 2020-07-13 2020-07-13 Method for realizing panoramic image system picture splicing display based on checkerboard calibration

Publications (1)

Publication Number Publication Date
CN111861891A true CN111861891A (en) 2020-10-30

Family

ID=72984459

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010668193.3A Pending CN111861891A (en) 2020-07-13 2020-07-13 Method for realizing panoramic image system picture splicing display based on checkerboard calibration

Country Status (1)

Country Link
CN (1) CN111861891A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112802109A (en) * 2021-02-07 2021-05-14 的卢技术有限公司 Method for generating automobile aerial view panoramic image

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011076104A1 (en) * 2009-12-21 2011-06-30 华为终端有限公司 Method and device for splicing images
CN107507131A (en) * 2017-08-08 2017-12-22 广州市安晓科技有限责任公司 360 degree of panorama reverse image generation methods based on single camera
CN108765496A (en) * 2018-05-24 2018-11-06 河海大学常州校区 A kind of multiple views automobile looks around DAS (Driver Assistant System) and method
CN109190628A (en) * 2018-08-15 2019-01-11 东北大学 A kind of plate camber detection method based on machine vision
CN109767473A (en) * 2018-12-30 2019-05-17 惠州华阳通用电子有限公司 A kind of panorama parking apparatus scaling method and device
CN110264395A (en) * 2019-05-20 2019-09-20 深圳市森国科科技股份有限公司 A kind of the camera lens scaling method and relevant apparatus of vehicle-mounted monocular panorama system
CN111223038A (en) * 2019-12-02 2020-06-02 上海赫千电子科技有限公司 Automatic splicing method and display device for vehicle-mounted all-around images
CN111243032A (en) * 2020-01-10 2020-06-05 大连理工大学 Full-automatic checkerboard angular point detection method

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011076104A1 (en) * 2009-12-21 2011-06-30 华为终端有限公司 Method and device for splicing images
CN107507131A (en) * 2017-08-08 2017-12-22 广州市安晓科技有限责任公司 360 degree of panorama reverse image generation methods based on single camera
CN108765496A (en) * 2018-05-24 2018-11-06 河海大学常州校区 A kind of multiple views automobile looks around DAS (Driver Assistant System) and method
CN109190628A (en) * 2018-08-15 2019-01-11 东北大学 A kind of plate camber detection method based on machine vision
CN109767473A (en) * 2018-12-30 2019-05-17 惠州华阳通用电子有限公司 A kind of panorama parking apparatus scaling method and device
CN110264395A (en) * 2019-05-20 2019-09-20 深圳市森国科科技股份有限公司 A kind of the camera lens scaling method and relevant apparatus of vehicle-mounted monocular panorama system
CN111223038A (en) * 2019-12-02 2020-06-02 上海赫千电子科技有限公司 Automatic splicing method and display device for vehicle-mounted all-around images
CN111243032A (en) * 2020-01-10 2020-06-05 大连理工大学 Full-automatic checkerboard angular point detection method

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112802109A (en) * 2021-02-07 2021-05-14 的卢技术有限公司 Method for generating automobile aerial view panoramic image
CN112802109B (en) * 2021-02-07 2024-05-10 的卢技术有限公司 Method for generating aerial view panorama of automobile

Similar Documents

Publication Publication Date Title
US9858639B2 (en) Imaging surface modeling for camera modeling and virtual view synthesis
CN108263283B (en) Method for calibrating and splicing panoramic all-round looking system of multi-marshalling variable-angle vehicle
CN109741455B (en) Vehicle-mounted stereoscopic panoramic display method, computer readable storage medium and system
CN111062873B (en) Parallax image splicing and visualization method based on multiple pairs of binocular cameras
US9445011B2 (en) Dynamic rearview mirror adaptive dimming overlay through scene brightness estimation
CN103770706B (en) Dynamic reversing mirror indicating characteristic
JP5739584B2 (en) 3D image synthesizing apparatus and method for visualizing vehicle periphery
JP4695167B2 (en) Method and apparatus for correcting distortion and enhancing an image in a vehicle rear view system
CN109407547A (en) Multi-cam assemblage on-orbit test method and system towards panoramic vision perception
US20150042799A1 (en) Object highlighting and sensing in vehicle image display systems
JP2009129001A (en) Operation support system, vehicle, and method for estimating three-dimensional object area
CN112712037A (en) Vehicle-mounted environment sensing method and system based on panoramic image and target detection
CN104859538A (en) Vision-based object sensing and highlighting in vehicle image display systems
JP2008048345A (en) Image processing unit, and sight support device and method
CN113362228A (en) Method and system for splicing panoramic images based on improved distortion correction and mark splicing
US20220101021A1 (en) Image colorization for vehicular camera images
JP4344860B2 (en) Road plan area and obstacle detection method using stereo image
JP3453960B2 (en) Vehicle periphery monitoring device
CN115936995A (en) Panoramic splicing method for four-way fisheye cameras of vehicle
CN111861891A (en) Method for realizing panoramic image system picture splicing display based on checkerboard calibration
CN116245722A (en) Panoramic image stitching system and method applied to heavy high-speed vehicle
JP2007134961A (en) Vehicle detection device and display device for vehicle using the same
CN113330480A (en) Modular image restoration method
CN113468991B (en) Parking space detection method based on panoramic video
JP2009077022A (en) Driving support system and vehicle

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination