CN111447431A - Naked eye 3D display method and system applied to vehicle-mounted all-around camera shooting - Google Patents
Naked eye 3D display method and system applied to vehicle-mounted all-around camera shooting Download PDFInfo
- Publication number
- CN111447431A CN111447431A CN202010256550.5A CN202010256550A CN111447431A CN 111447431 A CN111447431 A CN 111447431A CN 202010256550 A CN202010256550 A CN 202010256550A CN 111447431 A CN111447431 A CN 111447431A
- Authority
- CN
- China
- Prior art keywords
- vehicle
- image
- display
- visual angle
- naked eye
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/275—Image signal generators from 3D object models, e.g. computer-generated stereoscopic image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/128—Adjusting depth or disparity
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/302—Image reproducers for viewing without the aid of special glasses, i.e. using autostereoscopic displays
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Closed-Circuit Television Systems (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
Abstract
The invention relates to a naked eye 3D display method and system applied to vehicle-mounted all-around photography, belonging to the field of vehicle-mounted 3D image display, and being applied to a naked eye 3D display system of vehicle-mounted all-around photography, and being characterized in that: the method comprises the following steps: step S1, collecting video images; step S2, generating a panoramic image; step S3, embedding the 3D model pair of the vehicle into the left view panoramic image and the right view panoramic image respectively; step S4, loading the embedded pavement marker 3D model; step S5, synthesizing a 3D image; step S6, 3D image display. The invention provides a naked eye 3D display method and a naked eye 3D display system applied to vehicle-mounted all-around camera shooting, wherein a 3D model is embedded on the basis of left and right eye views with parallax, so that a driving picture presents more vivid three-dimensional effect and the feeling that an indication icon is suspended on a screen, the indication information is more striking, and a human-computer interaction interface is more friendly.
Description
Technical Field
The invention relates to the technical field of vehicle-mounted 3D image display, in particular to a naked eye 3D display method and system applied to vehicle-mounted all-around camera shooting.
Background
With the development of naked eye 3D technology, human eyes can see a vivid three-dimensional image from display equipment without an auxiliary means, the technical principle of naked eye 3D is to generate two left and right eye images with parallax, the left eye view and the right eye view can be separated by the light splitting of the parallax image by the display equipment and respectively reach the left eye and the right eye of a human, the visual nerve of the brain senses the depth and the distance through the difference between the visual angles of the two eyes, and a vivid virtual three-dimensional picture with a space depth sense is obtained.
The vehicle-mounted display system increasingly becomes an important component of the vehicle-mounted auxiliary driving and vehicle-mounted entertainment system, the all-round camera system displays that the driving condition of the vehicle and surrounding road conditions can be checked by a driver through the visual angle of the display screen, the current all-round camera system displays the images in a plane, the plane display looks visual and striking relatively, the human-computer interaction interface is not friendly enough, and the efficient and convenient three-dimensional guiding information for auxiliary driving is lacked.
Therefore, a new technical solution is needed to solve the above technical problems.
Disclosure of Invention
The invention aims to provide a naked eye 3D display method and system applied to vehicle-mounted all-around camera shooting, wherein a 3D model is embedded on the basis of left and right eye views with parallax, so that a driving picture presents a more vivid three-dimensional effect and a feeling that an indication icon is suspended on a screen, indication information is more striking, and a human-computer interaction interface is more friendly.
The above object of the present invention is achieved by the following technical solutions:
a naked eye 3D display method applied to vehicle-mounted all-around camera shooting is applied to a naked eye 3D display system of vehicle-mounted all-around camera shooting, and comprises the following steps:
s1, acquiring video images, namely acquiring eight paths of video image signals in the circumferential direction of the vehicle in real time through four groups of vehicle-mounted panoramic binocular cameras;
step S2, generating a panoramic image, rendering and merging 4 paths of left view images into a left view panoramic image, rendering and merging 4 paths of right view images into a right view panoramic image;
step S3, embedding the 3D model pair of the vehicle into the left view panoramic image and the right view panoramic image respectively;
s4, loading embedded pavement marker 3D models, detecting pavement markers of the left visual angle panoramic image and the right visual angle panoramic image through the detector models, loading corresponding pavement marker 3D models from the storage unit module according to the types of the detected pavement markers, and respectively embedding the pavement markers into corresponding positions of the left visual angle panoramic image and the right visual angle panoramic image;
step S5, synthesizing the 3D image, and performing rendering, arrangement and synthesis processing on the two paths of image pixels by the image processing module to generate a 3D image with parallax information;
and S6, displaying the 3D image, wherein the 3D image is split by the vehicle-mounted naked eye 3D display to generate a virtual stereo picture with a naked eye 3D effect.
By adopting the technical scheme, the left view and the right view of four points are respectively synthesized into the left view and the right view panoramic image, the 3D model of the vehicle is respectively embedded into the left view panoramic image and the right view panoramic image, the pavement marks of the two views are simultaneously detected through the detector model, the corresponding mark 3D model is loaded from the storage unit module according to the detected pavement mark type, the corresponding positions of the left view panoramic image and the right view panoramic image are respectively embedded, then the 3D image with parallax information is rendered and synthesized, and the 3D image is displayed on the naked eye 3D display to obtain the driving environment three-dimensional picture which is watched from the above two views and has auxiliary driving operation guidance.
The invention is further configured to: in the step S1, video image acquisition acquires eight video image signals of the vehicle circumferential direction in real time through four groups of vehicle-mounted binocular cameras around the vehicle, and includes the following steps:
s101, acquiring original image information, and shooting by four groups of vehicle-mounted binocular looking around cameras in real time to acquire scene image information;
and S102, image processing, wherein the vehicle-mounted all-round-looking binocular camera is electrically connected with a video image acquisition module, and the video image acquisition module is used for carrying out parallel analog-to-digital conversion and compression processing on the eight paths of video signals shot and transmitted by the vehicle-mounted all-round-looking binocular camera and converting the eight paths of video signals into digital image frames.
By adopting the technical scheme, the video stream shot by the vehicle-mounted panoramic binocular camera is converted into the data stream through the video image acquisition module and is output so as to facilitate later data transmission and processing.
The invention also aims to provide a naked eye 3D display system applied to vehicle-mounted all-around camera shooting, which has the characteristics of generating vivid driving three-dimensional pictures, visually and obviously providing the conditions of vehicles and surrounding spaces and enabling drivers to more quickly acquire important pavement marking information.
Another technical object of the present invention is to provide a naked eye 3D display system applied to vehicle-mounted panoramic photography, including four sets of vehicle-mounted binocular panoramic cameras, a vehicle-mounted naked eye 3D display, and a display processing device connecting the vehicle-mounted binocular panoramic cameras and the vehicle-mounted naked eye 3D display, the display processing device including:
the video image acquisition module is used for acquiring a vehicle-mounted all-round binocular camera video signal and converting the video signal into digital image frame data to be output;
the image processing module is used for merging the four square point images at the left visual angle into a left visual angle panoramic image, merging the four square point images at the right visual angle into a right visual angle panoramic image, and respectively embedding the pavement marker 3D model into the left visual angle panoramic image and the right visual angle panoramic image;
the AI module is electrically connected with the image processing module, is used for the neural network computing module with the accelerated computing function of image processing, and is used for carrying out target detection on the two panoramic images;
the display control module is used for controlling the signal driving and controlling the display arrangement of the 3D image pixels on the vehicle-mounted naked eye 3D display;
the storage unit module is used for storing the operating system and the pavement marker 3D model;
and the general computing module is electrically connected with the video image acquisition module, the AI module, the image processing module, the display control module and the storage unit module and controls the cooperative work among the modules.
By adopting the technical scheme, eight paths of video images shot by the four groups of vehicle-mounted around-looking binocular cameras are processed through mutual matching of the modules to obtain two panoramic images, and then pixels of the two panoramic images are rendered and arranged to be synthesized into a panoramic 3D image with parallax information; compared with a mode of simultaneously shooting from a plurality of angles by directly adopting expensive stereo camera equipment, the stereo driving picture manufactured by adopting the system has low manufacturing cost and is not limited by spatial layout; compared with a mode of shooting a 2D video source by a single camera and converting a 2D video picture into a 3D video picture by a software algorithm, the manufacturing mode of the system is higher in cost than the mode of shooting by the single camera, but the problem that the depth information is highly uncertain in the process of converting the 2D video picture into the 3D video picture due to the fact that the depth information is inferred from a single picture is solved, and for an invisible visual area, the technical problem that the image of the area is difficult to fill in a synthesized 3D image is lost due to the fact that the information of the area is not available in a single-view image or a corresponding depth map.
The invention is further configured to: the four groups of vehicle-mounted all-round binocular cameras are fixedly arranged on the vehicle body in a surrounding mode around the circumferential direction of the vehicle body, the four groups of vehicle-mounted all-round binocular cameras are respectively arranged in four directions of the front direction, the left direction, the rear direction and the right direction of the vehicle body in a surrounding mode, and the vehicle-mounted all-round binocular cameras are respectively a front-direction binocular camera arranged on a vehicle head safety lever, a left-direction binocular camera arranged on a left front vehicle door of the vehicle body, a rear-direction binocular camera arranged on a vehicle tail safety lever and a right-direction binocular camera arranged on a right front vehicle door of the; and each group of vehicle-mounted all-around binocular cameras is designed and placed according to the visual characteristics of human eyes.
Through adopting above-mentioned technical scheme, encircle setting up on the automobile body through four groups of on-vehicle binocular cameras of looking around to realize panoramic image's collection and shoot, the binocular camera adopts human eye visual characteristic to arrange can directly produce the required parallax information of 3D image, need not estimate the unseen regional image of height uncertainty through the depth information, can establish more lifelike three-dimensional scene through the image information that obtains from different shooting angles.
The invention is further configured to: every group on-vehicle binocular camera of looking around all includes left visual angle camera and right visual angle camera, left side visual angle camera and right visual angle camera deflect the same angle to the inboard jointly.
Through adopting above-mentioned technical scheme, avoid introducing extra perpendicular parallax between two visual angles through setting up the same angle of left visual angle camera and the inward sloping of right visual angle camera to reduce the tired sense that people's eye watched.
The invention is further configured to: and the left visual angle camera and the right visual angle camera both adopt ultra-wide angle fisheye cameras.
Through adopting above-mentioned technical scheme, through adopting super wide angle flake camera to make the scope of shooing wider, the picture of shooing is more clear.
The invention is further configured to: the vehicle-mounted naked eye 3D displayer comprises a vehicle-mounted liquid crystal display panel and a naked eye 3D grating film pasted on the vehicle-mounted liquid crystal display panel, wherein the naked eye 3D grating film is a lenticular lens grating film.
Through adopting above-mentioned technical scheme, thereby form on-vehicle bore hole 3D display through pasting bore hole 3D grating film on-vehicle liquid crystal display panel, be convenient for reform transform on current on-vehicle liquid crystal display panel, thereby reduce the cost that the system was used, bore hole 3D grating film adopts the lenticular lens grating film, utilize the lenticular lens grating film to refract liquid crystal screen light pixel and realize the beam split, throw the light of left eye image picture and right eye image picture respectively to viewer's left eye and right eye, make the viewer see two images that have parallax error information from two different angles, produce the stereoscopic vision effect that has distance sense and depth sense.
The invention is further configured to: the AI module is a neural network computing module, the neural network computing module is configured by a pre-trained detector model based on an artificial intelligence technology, and the detector model adopts a self-coding deep neural network based on a VGG framework.
By adopting the technical scheme, the self-coding depth neural network of the VGG framework has a simple structure, adopts a stacked 3x3 convolution kernel structure, increases the network depth through a plurality of layers of nonlinear layers, and has stronger learning capability on characteristics; the processing speed is high, and the requirement of real-time detection can be met.
The invention is further configured to: the training scheme for the detector model includes initialization training based on ImageNet data sets, optimization training based on collected and self-made data sets.
The third purpose of the invention is to provide a computer storage medium which can store corresponding programs and has the characteristic of being convenient for realizing the purpose of obtaining vivid virtual stereo pictures with space depth feeling.
The third object of the invention is realized by the following technical scheme:
a computer program is stored which can be loaded by a processor and which executes any of the above-described naked-eye 3D display methods applied to vehicle-mounted panoramic photography.
By adopting the technical scheme, the collected and self-made data set is used for training the detector model, the semi-supervised learning training process comprises two parts of unsupervised learning and supervised learning, the unsupervised learning clustered data is combined with the label data for supervised learning training, and the label classification is carried out on the classified data by using the supervised learning on the basis of classifying the data by the unsupervised learning society, so that a large-scale labeled training data sample set can be obtained at lower cost, the capability of extracting target characteristics and identification modes by a network can be enhanced on the basis of massive characteristic distribution training, the detection accuracy of the detector model and the generalization capability of the detector model to various changeable environments are improved, and the purpose of accurately applying the detector model to detecting road surface marks with various shapes and angles in an actual driving scene is achieved.
In conclusion, the beneficial technical effects of the invention are as follows: the arrangement of the binocular cameras can be used for directly shooting and generating left and right parallax images, the left and right parallax images are combined into a panoramic image picture, a 3D left and right eye image is rendered and combined and displayed on a vehicle-mounted naked eye 3D display, a driving environment stereoscopic picture which is watched at an upper view angle and has auxiliary driving operation guidance is obtained, meanwhile, a more vivid stereoscopic scene is constructed through image information obtained from different shooting angles, and the cost of 3D video production is reduced; the detector model based on the deep learning technology is designed and trained for detecting road traffic signs, and corresponding 3D images are embedded according to the road signs detected based on the artificial intelligence detector model, so that the indication icons are 'suspended' on the screen, the indication information is more striking, and the human-computer interaction interface is more friendly.
Drawings
FIG. 1 is a flow chart illustrating the steps of a display method according to the present invention;
FIG. 2 is a flowchart of the operation of the display method algorithm of the present invention;
FIG. 3 is a general block diagram of the display system of the present invention;
FIG. 4 is a front view of the vehicle mounted panoramic binocular camera arrangement of the present invention;
FIG. 5 is a plan cross-sectional view of a vehicle mounted panoramic binocular camera arrangement of the present invention;
FIG. 6 is the mounting positions of four sets of vehicle-mounted binocular looking around cameras of the present invention on the vehicle body;
FIG. 7 is a diagram of the overlapping area and the concatenation of two adjacent images of the vehicle-mounted panoramic binocular camera of the present invention;
FIG. 8 is a schematic diagram of a left eye view pixel arrangement to be extracted in accordance with the present invention;
FIG. 9 is a schematic diagram of the arrangement of pixels to be extracted for the right eye according to the present invention;
FIG. 10 is a schematic diagram of a pixel arrangement of a composite image according to the present invention;
FIG. 11 is a schematic diagram of the operation of the naked eye 3D display of the present invention;
fig. 12 is a schematic structural diagram of the vehicle-mounted naked eye 3D display of the present invention.
In the figure: 1. vehicle-mounted around-looking binocular camera; 11. a left view camera; 12. a right view camera; 2. a video image acquisition module; 3. an AI module; 4. an image processing module; 5. a display control module; 6. a memory cell module; 7. a general purpose computing module; 8. a vehicle-mounted naked eye 3D display; 81. a vehicle-mounted liquid crystal display panel; 82. naked eye 3D grating film.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings.
The first embodiment is as follows:
referring to fig. 1 and 2, the invention discloses a naked eye 3D display method applied to a vehicle-mounted panoramic photography, and a naked eye 3D display system applied to a vehicle-mounted panoramic photography, which comprises the following steps:
and S1, acquiring video images, and acquiring eight paths of video image signals in the circumferential direction of the vehicle in real time through four groups of vehicle-mounted panoramic binocular cameras 1.
The acquisition of the video image is obtained by the following specific steps:
s101, acquiring original image information, and acquiring scene image information by real-time shooting through four groups of vehicle-mounted panoramic binocular cameras 1;
and S102, image processing, wherein the four groups of vehicle-mounted all-around binocular cameras 1 are electrically connected to the video image acquisition module 2, and the video image acquisition module 2 is used for carrying out parallel analog-to-digital conversion and compression processing on eight paths of video signals shot and transmitted by the vehicle-mounted all-around binocular cameras 1 and converting the eight paths of video signals into digital image frames suitable for being processed by the image processing module 4 and the AI module 3.
And step S2, generating a panoramic image, rendering and combining the 4 paths of left view images into a left view panoramic image, and rendering and combining the 4 paths of right view images into a right view panoramic image.
And step S3, embedding the 3D model pair of the vehicle into the left perspective panoramic image and the right perspective panoramic image respectively.
And S4, loading the embedded pavement marker 3D model, detecting the pavement markers of the left visual angle panoramic image and the right visual angle panoramic image through the detector model, loading the corresponding pavement marker 3D model from the storage unit module 6 according to the type of the detected pavement marker, and respectively embedding the pavement marker 3D model into the corresponding positions of the left visual angle panoramic image and the right visual angle panoramic image.
And step S5, synthesizing the 3D image, and the image processing module 4 performs rendering, layout and synthesis processing on the two paths of image pixels to generate a 3D image with parallax information.
And step S6, displaying the 3D image, wherein the display control module 5 drives and controls the display layout of the image pixels on the vehicle-mounted naked eye 3D display 8, and the generated 3D image is split by the vehicle-mounted naked eye 3D display 8 to generate a virtual stereo picture with a naked eye 3D effect.
Example two:
referring to fig. 3, the naked eye 3D display system applied to vehicle-mounted panoramic photography comprises two parts, namely hardware system design and software algorithm implementation, wherein the hardware system mainly comprises a vehicle-mounted panoramic binocular camera 1, a vehicle-mounted naked eye 3D display 8 and a hardware computing system, and the hardware computing system is designed by adopting an SoC system; the software algorithm implementation part comprises an artificial intelligence-based detector model applied to target detection and identification, and a rendering algorithm for performing parallel fusion of panoramic images, embedding of 3D model images and synthesizing binocular 2D images into naked eye 3D images; the software algorithm implementation part is stored in the hardware computing system in an operating system mode, and the software algorithm implementation part and the hardware computing system are combined into a display processing device which is connected with the vehicle-mounted all-round binocular camera 1 and the vehicle-mounted naked eye 3D display 8. The display processing apparatus includes:
the video image acquisition module 2 is used for acquiring video signals of the vehicle-mounted all-round binocular camera 1 and converting the video signals into digital image frame data to be output;
the image processing module 4 is used for merging the four square point images at the left view angle into a left view angle panoramic image, merging the four square point images at the right view angle into a right view angle panoramic image, and respectively embedding the pavement marker 3D model into the left view angle panoramic image and the right view angle panoramic image;
the AI module 3 is electrically connected with the image processing module 4, is used for the neural network computing module with the accelerated computing function of image processing, and is used for carrying out target detection on the two panoramic images;
the display control module 5 is used for controlling the signal to drive and control the display arrangement of the 3D image pixels on the vehicle-mounted naked eye 3D display 8;
the storage unit module 6 is used for storing the operating system and the pavement marker 3D model;
and the general computing module 7 is electrically connected with the video image acquisition module 2, the AI module 3, the image processing module 4, the display control module 5 and the storage unit module 6 and is used for sending instruction signals to the modules and controlling the cooperative work among the modules.
Referring to fig. 4 and 5, in a human vision system, because two eyes have a distance, when viewing an object, object images on the retina of the left eye and the retina of the right eye have a certain level difference, so that the scene viewed by the human eyes has a distance and a distance sense, and the vehicle-mounted binocular looking around camera 1 in the scheme is designed and placed according to the visual characteristics of the human eyes. Every group vehicle-mounted all-around binocular camera 1 includes left visual angle camera 11 and right visual angle camera 12, and the interval between two cameras is calculated according to the interval between the camera central points, and the interval between the central points of left visual angle camera 11 and right visual angle camera 12 in this embodiment is close the distance between people's eye binocular to be 65 mm.
Referring to fig. 4 and 5, the left view camera 11 and the right view camera 12 are collectively deflected inward by the same angle, and no additional vertical parallax is introduced between the two views by using such a tilt-shift method of deflecting the same angle. The inward inclination angles of the visual axes of the cameras of the left visual angle camera 11 and the right visual angle camera 12 are (a/2) degrees, the visual lines of the two cameras are intersected in the forward direction, the internal angle of the two intersecting lines is a degrees, the specific numerical value of a is determined by debugging and calibrating according to the type selection of the cameras and the effect of vehicle-mounted naked eye 3D display when the vehicle is placed on a specific vehicle, and the cameras are fixedly placed on a vehicle body after being calibrated.
Referring to fig. 6, for realizing the look around effect, four groups of vehicle-mounted look around binocular cameras 1 encircle automobile body circumference fixed mounting on the automobile body, four groups of vehicle-mounted look around binocular cameras 1 encircle respectively and set up in four directions of the preceding of automobile body, the left direction, the back, the right direction, divide according to its position of installation, four groups of vehicle-mounted look around binocular cameras 1 are respectively for setting up the preceding binocular cameras on the locomotive bumper bar, install the left direction binocular camera on the left front door of automobile body, install the back binocular camera on the rear of a vehicle bumper bar and install the right direction binocular camera on the right front door of automobile body, make four groups of vehicle-mounted look around binocular cameras 1 scan the scene around the automobile body, thereby realize the original material that vehicle-mounted look around image formed. For guaranteeing better all-round looking effect, left visual angle camera 11 and right visual angle camera 12 all adopt super wide angle fisheye camera to make the scope of shooting wider, the picture of shooting is more clear.
Referring to fig. 3, the video image acquisition module 2 is electrically connected to the four groups of vehicle-mounted binocular cameras 1, and is configured to perform parallel analog-to-digital conversion and compression processing on the eight paths of video signals transmitted by the binocular cameras, and convert the eight paths of video signals into digital image frames suitable for processing by the image processing module 4 and the AI module 3. Eight paths of image frame data output by the video image acquisition processing module are transmitted to the image processing module 4 through electric connection, the image processing module 4 respectively connects four square point images of a left visual angle into a left visual angle panoramic image, connects four square point images of a right visual angle into a right visual angle panoramic image, then calls a 3D model pair of a vehicle from the storage unit module 6, the left visual angle panoramic image and the right visual angle panoramic image are respectively embedded, the left visual angle panoramic image and the right visual angle panoramic image which are well connected are transmitted to the AI module 3 through electric connection, a neural unit is configured by loading a pre-trained neural network model, and target detection is carried out on the two panoramic images.
Referring to fig. 7, in the cameras in two adjacent directions of the panoramic camera, there are two regions that intersect each other in their visual field ranges, that is, overlapping regions, where the overlapping regions are an overlapping region between a forward camera and a left camera, an overlapping region between a left camera and a backward camera, an overlapping region between a backward camera and a right camera, and an overlapping region between a right camera and a forward camera, and in order to ensure that the spliced panoramic image is smoother and more natural, the algorithm of the image processing module 4 includes a SIFT algorithm and a multiband fusion algorithm.
Firstly, respectively carrying out feature point matching of adjacent overlapping areas on a front left two adjacent camera images, a left rear two adjacent camera images, a rear right two adjacent camera images and a right front two adjacent camera images of each visual angle (a left eye visual angle and a right eye visual angle) of four points of the vehicle body around view through an SIFT algorithm (a scale invariant feature transform algorithm), estimating a projection transformation matrix of every two adjacent images, respectively splicing the four images of the left eye visual angle and the right eye visual angle together in sequence according to the projection transformation relation of every two adjacent images to respectively obtain panoramic images of the left eye visual angle and the right eye visual angle, then respectively carrying out fusion processing on the panoramic images of the left visual angle and the panoramic images of the right visual angle through a multiband fusion algorithm, eliminating the problems of boundary gaps and ghost in the two images, and enabling the spliced images to be smoother and natural.
Referring to fig. 2 and 3, a pair of 3D models of a prefabricated vehicle, the pair of 3D models including a left-view 3D model and a right-view 3D model, is pre-stored in a storage unit module 6 of the system. When the left perspective panorama and the right perspective panorama are generated, the image processing module 4 calls a pair of 3D models of the vehicle prestored in the storage unit module 6, and the pair of 3D models of the vehicle are respectively embedded and synthesized in the central positions of the left and right eye perspective panorama, so that the formed 3D panorama is more vivid.
Referring to fig. 2 and 3, the AI module 3 is a neural network computing module, and the neural network computing module is configured by a pre-trained detector model based on artificial intelligence technology, and the detector model adopts a self-coding deep neural network based on a VGG architecture. Wherein the training scheme for the detector model includes initialization training based on the ImageNet dataset, optimization training based on the collected and self-fabricated dataset. Firstly, performing initialization training on an ImageNet data set to obtain an initial model as the weight initialization of the transfer learning, and then performing the transfer learning training by using the collected and self-made data set. The collected and self-made data set comprises two parts, wherein one part is large-scale label-free data and is used for unsupervised learning; the other part is labeled data of larger scale for supervised learning training. The collected data are a large number of road traffic identification pictures with changeable shapes, the self-made road traffic identification pictures with label data as mark numbers, and json format files for recording the picture numbers and the key value pairs corresponding to the classification numbers.
The method comprises the steps of firstly carrying out characteristic noise reduction processing on a large amount of unlabeled data through a Principal Component Analysis (PCA) algorithm in the unsupervised learning, then carrying out dimensionality reduction on large-scale data through a t-SNE (t-distribution random neighborhood embedding) algorithm, extracting main characteristic distribution, finally clustering the data according to the characteristic distribution through a k-Means clustering algorithm, realizing characteristic learning through self-coding reconstruction data, learning an implicit characteristic mode from the data characteristic distribution, and classifying and dividing the data according to the characteristics under the condition of no label learning supervision. The method has the advantages that supervised learning training is carried out by combining the clustered data of unsupervised learning with the label data, and label classification is carried out on the classified data by using the supervised learning on the basis of classifying the data by the unsupervised learning society, so that a large-scale labeled training data sample set can be obtained at a low cost, the capability of extracting target characteristics and recognition modes by a network can be enhanced based on massive characteristic distribution training, the detection accuracy of a detector model and the generalization capability of the detector model to various changeable environments are improved, and the aim of accurately detecting various shape and angle changeable pavement markers in actual driving scenes is fulfilled.
Referring to fig. 2 and 3, signals of detection results of the AI module 3 are transmitted to the image processing module 4 through electric connection, the image processing module 4 loads corresponding road traffic sign 3D models according to the detection results, renders and embeds the road traffic sign 3D models into the left-view panoramic image and the right-view panoramic image, and renders and arranges image pixels of the two panoramic images to synthesize a panoramic 3D image with parallax information. The prefabricated road traffic sign 3D model comprises: the system comprises a straight driving indication model for straight driving, a left turning indication model for turning to the left, a right turning indication model for turning to the right, a turning indication model allowing turning around, a straight driving left turning indication model, a straight driving right turning indication model, a deceleration driving indication model of a deceleration strip and a guiding mark indication model for driving guidance.
The 3D models of various road traffic marks are numbered, the numbers are stored in a list of storage addresses of two mapping storage unit modules 6 and respectively correspond to mapping relations of the left-view 3D models and the right-view 3D models which are prestored in the storage unit modules 6. The output of the detector is a vector table, the number of the vector table corresponds to the 3D model number of the road traffic identification, when the numerical value of a certain number in the vector table is greater than a threshold value, the detection of the corresponding road traffic identification is indicated, and the image processing module 4 extracts the pre-stored road traffic identification 3D model pair through a list and embeds the pre-stored road traffic identification 3D model pair into corresponding positions of the left eye visual angle panoramic image and the right eye visual angle panoramic image. The 3D image display format adopts a layered depth video image format, namely, on the basis of a left-eye disparity map and a right-eye disparity map shot by a real scene, a layer of synthesized disparity map is added, a 3D scene is divided into a background and a foreground, the background is a real driving environment disparity map shot by a binocular surround-view camera, the foreground is a 3D virtual animation of a vehicle 3D model and a driving indication icon which are embedded and synthesized, and the synthesized virtual 3D scene is described by two layers of disparity maps.
Referring to fig. 8 and 9, the display control module 5 performs sampling rearrangement on the image pixels at the left and right viewing angles, so that the size of the synthesized image is consistent with the size of the vehicle-mounted naked eye 3D display 8, and after light splitting of the vehicle-mounted naked eye 3D display 8, the light pixels at the left viewing angle are projected to the left eye direction, and the light pixels at the right viewing angle are projected to the right eye direction. And obtaining a left eye view and a right eye view after image rendering processing, wherein the resolution of the left eye view and the resolution of the right eye view are both M × N, and both M and N are even numbers. Referring to fig. 10, the in-vehicle naked-eye 3D display 8 creates a blank canvas with a resolution of M × N, extracts pixels from pixels of even columns counted from the left of the left-eye view by columns, fills the corresponding even columns of the blank canvas column by corresponding column number, extracts pixels from pixels of odd columns counted from the left of the right-eye view by columns, and fills the corresponding odd columns of the blank canvas column by column number, thus combining a left-eye view with a resolution of (M/2) × N and a right-eye view with a resolution of (M/2) × N into a new left-eye image with a resolution of M × N.
Referring to fig. 11 and 12, in order to facilitate the modification of a general vehicle that is shipped and used, the on-vehicle naked-eye 3D display 8 may use a naked-eye 3D panel display, or may use a combination of an on-vehicle liquid crystal display panel 81 and a naked-eye 3D grating film 82 adhered to the on-vehicle liquid crystal display panel 81. In this embodiment, the naked eye 3D grating film 82 adopts a lenticular grating film, and the liquid crystal display pixels are refracted by the lenticular grating film to split light, so that the light of the left-eye image and the right-eye image is projected to the left eye and the right eye of the viewer, respectively, so that the viewer sees two images with parallax information from two different angles, and a stereoscopic vision effect with distance and depth is generated. The size of the naked eye 3D grating film 82 is 2 times of the size of the basic unit of the pixel, parameters such as the parameter period, the caliber, the focal length and the like of the micro-column grating are customized, so that the parameters of the grating film and the image of the vehicle-mounted liquid crystal display panel 81 are matched with each other, and vivid three-dimensional picture display is realized. In this embodiment, the image pixel arrangement on the on-vehicle liquid crystal display panel 81 adopts a full-pixel light splitting structure, that is, a pixel composed of three RGB sub-pixels is used as a basic unit, and light splitting is performed by refraction of a cylindrical mirror.
Example three:
in one embodiment, a computer-readable storage medium is provided, which stores a computer program that can be loaded by a processor and executes the above-mentioned naked-eye 3D display method applied to vehicle-mounted panoramic photography, and when executed by the processor, the computer program realizes the following steps:
s1, acquiring video images, namely acquiring eight video image signals of the vehicle circumference in real time through four groups of vehicle-mounted panoramic binocular cameras 1;
step S2, generating a panoramic image, rendering and merging 4 paths of left view images into a left view panoramic image, rendering and merging 4 paths of right view images into a right view panoramic image;
step S3, embedding the 3D model pair of the vehicle into the left view panoramic image and the right view panoramic image respectively;
step S4, loading a 3D model embedded with a pavement marker, detecting the pavement markers of the left visual angle panoramic image and the right visual angle panoramic image through a detector model, loading a corresponding 3D model of the pavement marker from the storage unit module 6 according to the type of the detected pavement marker, and respectively embedding the pavement marker into corresponding positions of the left visual angle panoramic image and the right visual angle panoramic image;
step S5, synthesizing the 3D image, and the image processing module 4 carries out rendering, arrangement and synthesis processing on the two paths of image pixels to generate a 3D image with parallax information;
and S6, displaying the 3D image, wherein the 3D image is split by the vehicle-mounted naked eye 3D display 8 to generate a virtual stereo picture with a naked eye 3D effect.
The computer-readable storage medium includes, for example: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The embodiments of the present invention are preferred embodiments of the present invention, and the scope of the present invention is not limited by these embodiments, so: all equivalent changes made according to the structure, shape and principle of the invention are covered by the protection scope of the invention.
Claims (10)
1. The utility model provides a bore hole 3D display method for on-vehicle looking around makes a video recording, is applied to bore hole 3D display system that on-vehicle looking around made a video recording which characterized in that: the method comprises the following steps:
s1, acquiring video images, namely acquiring eight paths of video image signals in the circumferential direction of the vehicle in real time through four groups of vehicle-mounted panoramic binocular cameras (1);
step S2, generating a panoramic image, rendering and merging 4 paths of left view images into a left view panoramic image, rendering and merging 4 paths of right view images into a right view panoramic image;
step S3, embedding the 3D model pair of the vehicle into the left view panoramic image and the right view panoramic image respectively;
s4, loading embedded pavement marker 3D models, detecting pavement markers of the left visual angle panoramic image and the right visual angle panoramic image through the detector models, loading corresponding pavement marker 3D models from the storage unit module (6) according to the types of the detected pavement markers, and respectively embedding the pavement markers into corresponding positions of the left visual angle panoramic image and the right visual angle panoramic image;
s5, synthesizing the 3D image, and performing rendering, arranging and synthesizing processing on the two paths of image pixels by the image processing module (4) to generate a 3D image with parallax information;
and step S6, displaying the 3D image, wherein the 3D image is split by the vehicle-mounted naked eye 3D display (8) to generate a virtual stereo picture with a naked eye 3D effect.
2. The naked-eye 3D display method applied to vehicle-mounted all-around camera shooting according to claim 1, characterized in that: in the step S1, video image acquisition acquires eight video image signals of the vehicle circumferential direction in real time through four groups of vehicle-mounted binocular looking around cameras (1), and includes the following steps:
s101, acquiring original image information, and shooting by four groups of vehicle-mounted binocular looking around cameras (1) in real time to acquire scene image information;
and S102, image processing, wherein the vehicle-mounted all-round binocular camera (1) is electrically connected with a video image acquisition module (2), and the video image acquisition module (2) is used for carrying out parallel analog-to-digital conversion and compression processing on the eight paths of video signals shot and transmitted by the vehicle-mounted all-round binocular camera (1) and converting the eight paths of video signals into digital image frames.
3. The utility model provides a bore hole 3D display system for on-vehicle looking around makes a video recording which characterized in that: including four on-vehicle binocular cameras of looking around (1), on-vehicle bore hole 3D display (8) and connect on-vehicle binocular cameras of looking around (1) and on-vehicle bore hole 3D display's (8) display processing apparatus, display processing apparatus includes:
the video image acquisition module (2) is used for acquiring video signals of the vehicle-mounted all-round binocular camera (1) and converting the video signals into digital image frame data to be output;
the image processing module (4) is used for merging the four square point images at the left visual angle into a left visual angle panoramic image, merging the four square point images at the right visual angle into a right visual angle panoramic image, and respectively embedding the pavement marker 3D model into the left visual angle panoramic image and the right visual angle panoramic image;
the AI module (3) is electrically connected with the image processing module (4), is used for the neural network computing module with the accelerated computing function of image processing, and is used for carrying out target detection on the two panoramic images;
the display control module (5) is used for controlling the signal to drive and control the display arrangement of the 3D image pixels on the vehicle-mounted naked eye 3D display (8);
the storage unit module (6) is used for storing the operating system and the pavement marker 3D model;
and the general computing module (7) is electrically connected with the video image acquisition module (2), the AI module (3), the image processing module (4), the display control module (5) and the storage unit module (6) and controls the cooperative work among the modules.
4. The naked eye 3D display system applied to vehicle-mounted all-around camera shooting according to claim 3, characterized in that: the four groups of vehicle-mounted all-round binocular cameras (1) are fixedly arranged on the vehicle body in a circumferential direction around the vehicle body, the four groups of vehicle-mounted all-round binocular cameras (1) are respectively arranged in four directions of the front direction, the left direction, the rear direction and the right direction of the vehicle body in a surrounding mode, and the vehicle-mounted all-round binocular cameras (1) are respectively a front-direction binocular camera arranged on a vehicle head safety lever, a left-direction binocular camera arranged on a left front vehicle door of the vehicle body, a rear-direction binocular camera arranged on a vehicle tail safety lever and a right-direction binocular camera arranged on a right front vehicle door of the vehicle body; and each group of vehicle-mounted all-round binocular cameras (1) is designed and placed according to the visual characteristics of human eyes.
5. The naked eye 3D display system applied to vehicle-mounted all-around camera shooting according to claim 4, characterized in that: every group on-vehicle binocular camera of looking around (1) all includes left visual angle camera (11) and right visual angle camera (12), same angle of inboard deflection is jointly gone to left visual angle camera (11) and right visual angle camera (12).
6. The naked eye 3D display system applied to vehicle-mounted all-around camera shooting according to claim 5, characterized in that: and the left visual angle camera (11) and the right visual angle camera (12) both adopt ultra-wide angle fisheye cameras.
7. The naked eye 3D display system applied to vehicle-mounted all-around camera shooting according to claim 3, characterized in that: the vehicle-mounted naked eye 3D display (8) comprises a vehicle-mounted liquid crystal display panel (81) and a naked eye 3D grating film (82) pasted on the vehicle-mounted liquid crystal display panel (81), wherein the naked eye 3D grating film (82) adopts a lenticular grating film.
8. The naked eye 3D display system applied to vehicle-mounted all-around camera shooting according to claim 3, characterized in that: the AI module (3) is a neural network computing module, the neural network computing module is configured by a pre-trained detector model based on an artificial intelligence technology, and the detector model adopts a self-coding deep neural network based on a VGG framework.
9. The naked eye 3D display system applied to vehicle-mounted all-around camera shooting according to claim 8, characterized in that: the training scheme for the detector model includes initialization training based on ImageNet data sets, optimization training based on collected and self-made data sets.
10. A computer-readable storage medium characterized by: a computer program which can be loaded by a processor and which executes the method according to any of claims 1 to 2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010256550.5A CN111447431A (en) | 2020-04-02 | 2020-04-02 | Naked eye 3D display method and system applied to vehicle-mounted all-around camera shooting |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010256550.5A CN111447431A (en) | 2020-04-02 | 2020-04-02 | Naked eye 3D display method and system applied to vehicle-mounted all-around camera shooting |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111447431A true CN111447431A (en) | 2020-07-24 |
Family
ID=71655141
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010256550.5A Pending CN111447431A (en) | 2020-04-02 | 2020-04-02 | Naked eye 3D display method and system applied to vehicle-mounted all-around camera shooting |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111447431A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112801880A (en) * | 2021-03-08 | 2021-05-14 | 广州敏视数码科技有限公司 | Vehicle-mounted panoramic image imaging and target detection fusion display method |
| CN113066003A (en) * | 2021-03-12 | 2021-07-02 | 深圳市唯特视科技有限公司 | Panoramic image generation method and device, electronic equipment and storage medium |
| CN115471925A (en) * | 2022-03-21 | 2022-12-13 | 北京罗克维尔斯科技有限公司 | Driving video recording method and device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN205395928U (en) * | 2016-01-19 | 2016-07-27 | 华晨汽车集团控股有限公司 | On -vehicle three -dimensional panorama monitored control system |
| CN109246416A (en) * | 2018-09-21 | 2019-01-18 | 福州大学 | The panorama mosaic method of vehicle-mounted six road camera |
| CN109525790A (en) * | 2017-09-20 | 2019-03-26 | 杭州海康威视数字技术股份有限公司 | Video file generation method and system, playback method and device |
| CN110341597A (en) * | 2018-04-02 | 2019-10-18 | 杭州海康威视数字技术股份有限公司 | A kind of vehicle-mounted panoramic video display system, method and Vehicle Controller |
-
2020
- 2020-04-02 CN CN202010256550.5A patent/CN111447431A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN205395928U (en) * | 2016-01-19 | 2016-07-27 | 华晨汽车集团控股有限公司 | On -vehicle three -dimensional panorama monitored control system |
| CN109525790A (en) * | 2017-09-20 | 2019-03-26 | 杭州海康威视数字技术股份有限公司 | Video file generation method and system, playback method and device |
| CN110341597A (en) * | 2018-04-02 | 2019-10-18 | 杭州海康威视数字技术股份有限公司 | A kind of vehicle-mounted panoramic video display system, method and Vehicle Controller |
| CN109246416A (en) * | 2018-09-21 | 2019-01-18 | 福州大学 | The panorama mosaic method of vehicle-mounted six road camera |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112801880A (en) * | 2021-03-08 | 2021-05-14 | 广州敏视数码科技有限公司 | Vehicle-mounted panoramic image imaging and target detection fusion display method |
| CN112801880B (en) * | 2021-03-08 | 2024-06-07 | 广州敏视数码科技有限公司 | Method for fusion display of vehicle-mounted panoramic image imaging and target detection |
| CN113066003A (en) * | 2021-03-12 | 2021-07-02 | 深圳市唯特视科技有限公司 | Panoramic image generation method and device, electronic equipment and storage medium |
| CN113066003B (en) * | 2021-03-12 | 2024-03-29 | 深圳市唯特视科技有限公司 | Method and device for generating panoramic image, electronic equipment and storage medium |
| CN115471925A (en) * | 2022-03-21 | 2022-12-13 | 北京罗克维尔斯科技有限公司 | Driving video recording method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10560687B2 (en) | LED-based integral imaging display system as well as its control method and device | |
| CN110758243B (en) | Surrounding environment display method and system in vehicle running process | |
| CN111447431A (en) | Naked eye 3D display method and system applied to vehicle-mounted all-around camera shooting | |
| JP4764305B2 (en) | Stereoscopic image generating apparatus, method and program | |
| JP5036132B2 (en) | Critical alignment of parallax images for autostereoscopic display | |
| CN108513123B (en) | Image array generation method for integrated imaging light field display | |
| CN100576251C (en) | Display unit, rendering method and image processing equipment | |
| CN102209254B (en) | One-dimensional integrated imaging method and device | |
| CN102124749B (en) | Stereoscopic image display apparatus | |
| KR100901866B1 (en) | 3D floating-image display System having a background and Method thereof | |
| US20110211043A1 (en) | Method and system for encoding a 3d image signal, encoded 3d image signal, method and system for decoding a 3d image signal | |
| CN103913845A (en) | Method for displaying three-dimensional integral images using mask and time division multiplexing | |
| CN113763301B (en) | Three-dimensional image synthesis method and device for reducing miscut probability | |
| CN102555905A (en) | Method for producing image of e.g. post in concrete parking bay of ego vehicle, involves installing graphical element in object region to produce image in surrounding from virtual camera position, where region is not visible for sensor | |
| KR20110113923A (en) | Image converting device and three dimensional image display device including the same | |
| CN104635337A (en) | Design method of honeycomb type lens array capable of improving stereo image display resolution | |
| CN109764888A (en) | Display system and display methods | |
| CN111447429B (en) | Vehicle-mounted naked eye 3D display method and system based on binocular camera shooting | |
| CN107948631A (en) | It is a kind of based on cluster and the bore hole 3D systems that render | |
| CN113467214B (en) | Holographic imaging device and imaging method thereof | |
| JP2018160853A (en) | Video conversion system, video conversion method, and video conversion program | |
| CN112866676A (en) | Naked eye three-dimensional display algorithm based on single-pixel multi-view reconstruction | |
| CN101655747B (en) | Three-dimensional mouse | |
| CN116708746A (en) | Naked eye 3D-based intelligent display processing method | |
| KR101790720B1 (en) | Method for generating integrated image using terrain rendering of real image, and recording medium thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200724 |