CN108055472A - A kind of real time panoramic live broadcast system and method - Google Patents
A kind of real time panoramic live broadcast system and method Download PDFInfo
- Publication number
- CN108055472A CN108055472A CN201711392097.5A CN201711392097A CN108055472A CN 108055472 A CN108055472 A CN 108055472A CN 201711392097 A CN201711392097 A CN 201711392097A CN 108055472 A CN108055472 A CN 108055472A
- Authority
- CN
- China
- Prior art keywords
- video
- live
- camera
- image
- panoramic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 238000012545 processing Methods 0.000 claims abstract description 40
- 238000004891 communication Methods 0.000 claims abstract description 19
- 238000013507 mapping Methods 0.000 claims description 15
- 230000004927 fusion Effects 0.000 claims description 7
- 238000000205 computational method Methods 0.000 claims description 6
- 238000012937 correction Methods 0.000 claims description 6
- 238000009434 installation Methods 0.000 claims description 6
- 241000251468 Actinopterygii Species 0.000 claims description 5
- 238000003384 imaging method Methods 0.000 claims description 5
- 238000005538 encapsulation Methods 0.000 claims description 4
- 230000008569 process Effects 0.000 claims description 3
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims 1
- 230000006870 function Effects 0.000 description 6
- 238000005265 energy consumption Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 239000011521 glass Substances 0.000 description 4
- 238000007654 immersion Methods 0.000 description 4
- 230000009467 reduction Effects 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000000155 melt Substances 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 238000010899 nucleation Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/2187—Live feed
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
- H04N21/643—Communication protocols
- H04N21/6437—Real-time Transport Protocol [RTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
- H04N21/816—Monomedia components thereof involving special video data, e.g 3D video
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/698—Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/265—Mixing
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
- Studio Devices (AREA)
Abstract
A kind of real time panoramic live broadcast system and method, wherein system include：Lens of panoramic camera module, for gathering video data；Fpga chip is connected with lens of panoramic camera module, to receive the video data that lens of panoramic camera module is gathered, and carries out image mosaic processing to video data；Microphone, for gathering voice data；Video frequency processing chip is connected respectively with fpga chip and microphone, for receiving the video data of voice data and image mosaic processing and being compressed coding respectively, to be packaged into the multimedia stream file for the reference format for including video and audio；And communication module；It is connected with video frequency processing chip, for multimedia stream file to be pushed out realization live streaming.Real time panoramic live broadcast system of the present invention and method can set the parameters such as live video resolution ratio, code check according to the protocol self-adapting that platform is broadcast live, and carry out secondary transcoding without panorama live streaming Platform Server, greatly reduce the delay during live streaming.
Description
Technical field
The invention belongs to panoramic shootings and direct seeding technique field, and in particular to a kind of real time panoramic live broadcast system and method.
Background technology
With the continuous development of network technology, net cast is increasingly welcome by masses, the field being related to include game,
Physical culture, social activity, investment in finance and economics, amusement scene, stock finance, campus, tourism, music scene, wedding celebration, online education, Venture Road
Drill, recruit, parent-offspring, body-building, e-commerce, medical treatment and all kinds of direct broadcast services provide etc. dozens ofs field.By internet and
Stream media technology is broadcast live, so as in real time, comprehensively propagate video.
At present, VR live streamings are mostly using panoramic camera, and the field range of panoramic camera is bigger than common camera, usually
Think that horizontal field of view reaches 360 degree, vertical visual field reaches and more than 180 degree, can meet the videorama of a place.Panorama
Camera needs complete video acquisition, video processing in machine, and the processing such as Video coding and plug-flow when carrying out panorama live streaming, need
A high performance home server is wanted, and is pushed to live streaming Platform Server and is broadcast live；Some even needs to take in platform
Business device end carries out secondary transcoding, could realize 3D or VR live streamings in client.
In addition, during viewing panorama live streaming, since panoramic picture is to be mapped on a spherical surface to realize that 3D or VR is shown, show
Area is than traditional 4:3 or 16:9 plane is much bigger, and high definition resolution ratio can not provide enough picture element densities and ensure picture
Clarity.
The content of the invention
In the prior art, panorama live broadcast system access device is more, system architecture is complicated, live streaming flow is complicated, and there are videos
The problems such as relatively low resolution ratio, limited viewing angle, real-time are poor, live streaming is unstable is broadcast live, this in order to solve the problems, such as, the present invention carries
For a kind of real time panoramic live broadcast system, concrete scheme is as follows：
A kind of real time panoramic live broadcast system, the system include：
Lens of panoramic camera module, for gathering video data；
Fpga chip is connected with the lens of panoramic camera module, is regarded with receive that lens of panoramic camera module gathered
Frequency evidence, and image mosaic processing is carried out to the video data；
Microphone, for gathering voice data；
Video frequency processing chip is connected respectively with the fpga chip and microphone, is spelled for receiving voice data and image
It connects the video data of processing and is compressed coding respectively, to be packaged into the media stream for the reference format for including video and audio
File；And
Communication module；It is connected with the video frequency processing chip, for multimedia stream file to be pushed out realization live streaming.
Wherein, in above system, the lens of panoramic camera module include 4 fish-eye cameras and with each fish
The corresponding imaging sensor of Eye imaging head；Four 90 degree of fish-eye camera intervals are fixed on same level, and each flake is taken the photograph
As the relative position of head immobilizes.
Wherein, in above system, the system also includes gyroscope, the gyroscope is used to gather lens of panoramic camera
Module is sent to the video frequency processing chip to calculate the current sky of panoramic camera around default axial turning rate
Between posture, and calculate panorama live video.
Wherein, in above system, the system also includes HDMI processing modules, the HDMI processing modules are for real-time
Ground exports 4K high-definition images, and the HDMI processing modules are also accompanied with audio output.
Wherein, in above system, the network connection of the communication module is WLAN and/or wired wide area network；
When the network connection is WLAN, the communication module leads to being equipped with the user terminal of camera APP
It crosses WLAN and carries out communication connection, so that by camera APP panoramic camera is adjusted setting in terminal user；
When the network connection is wired wide area network, the communication module connects router by cable and accesses wide area
Net, and communication connection is established with the live streaming Platform Server in wide area network, so that terminal user passes through browser plug-in or straight
It broadcasts APP and panorama live streaming viewing is carried out using 3D/VR modes in client.
The real time panoramic live broadcast system of the present invention, is handled original video data using fpga chip, compared with embedding
Enter formula processor, fpga chip is functioning in parallel, and single clock cycle interior energy is largely operated, especially suitable for video
Image procossing, and FPAG have the characteristics that it is low in energy consumption, will be in power consumption and cost all as the core of processing using fpga chip
To reduction.
The real time panoramic live broadcast system of the present invention, it is more to solve existing live broadcast system access device, and system architecture is complicated, straight
The problems such as flow is complicated, net cast resolution ratio is relatively low, real-time is poor, live streaming is unstable is broadcast, fuselage is compact, convenient for carry-on
It carries, there is ultra high-definition image quality, be suitble to the various occasions such as enterprise, individual, live news coverage.
The real time panoramic live broadcast system of the present invention sets live video to differentiate according to the protocol self-adapting of live streaming platform
The parameters such as rate, code check carry out secondary transcoding without panorama live streaming Platform Server, greatly reduce the delay during live streaming,
Terminal user directly can carry out panorama live streaming in client with its existing browser plug-in or live streaming APP with 3D/VR modes
Viewing if wearing VR glasses, will obtain the hearing experience of the presence with impact, feeling of immersion.
According to another aspect of the present invention, the present invention also provides a kind of real time panoramic live broadcasting method, including walking as follows
Suddenly：
Initialize installation is broadcast live in step S1；
Step S2 reads the video data that every road fish-eye camera is gathered by fpga chip in real time, and to video counts
According to progress image mosaic；
Step S3, video, audio are encoded in real time, and the audio and video after coding is packaged to be converted into more matchmakers
Body stream；
Step S4, real time propelling movement for panorama live streaming media stream to CDN high in the clouds so that user terminal passes through connection
Streaming media server in CDN high in the clouds is to realize that panorama plays.
Wherein, in the above-mentioned methods, Initialize installation is broadcast live in the step S1 to specifically include：
Panoramic camera and cable router are connected by cable, and passes through router and is linked into wide area network, to establish
Procotol connection needed for live streaming；
Camera APP and panoramic camera are connected by WLAN, according to streaming media server agreement, in camera APP
Live streaming parameter inside middle setting panorama camera, the live streaming parameter include the resolution ratio of live streaming panoramic video, code check, Yi Ji
Unique live streaming ID and the streaming media server address that CDN high in the clouds provides are inputted in camera APP, establish panoramic camera with it is described
The agreement connection of streaming media server.
Wherein, in the above-mentioned methods, carrying out image mosaic to video data by fpga chip in the step S2 includes：
Step S21 carries out pattern distortion correction using coordinate map；
It specifically includes：Step S211 to given real time panoramic live broadcast system, is calculated respectively using gridiron pattern scaling method
Spatial relationship parameters between the intrinsic parameter and distortion factor and adjacent fish-eye camera of a fish-eye camera；
Step S212：Utilize the spatial relationship parameters meter between the intrinsic parameter of fish-eye camera and distortion factor and camera lens
The coordinate map between original image and target image is calculated, coordinate map saves what each fish-eye camera was gathered
One-to-many correspondence in the target image that original image and process image split-joint method are formed between pixel point coordinates,
In, target image is the image of distortion；
Step S213：The original image that each fish eye lens takes is mapped to by target according to the coordinate map
On image, using the pixel in the target image as target pixel points, using the preset coordinate mapping table, determine described
Target pixel points and the correspondence of the source image vegetarian refreshments on the pending original image；
Wherein, searched in original image as follows with the computational methods of the corresponding source image vegetarian refreshments of target pixel points：
Dst (x, y)=Src (Lut_x (x, y), Lut_y (x, y))
Wherein, Dst (x, y) denotation coordination is the target pixel points of (x, y), and Lut_x (x, y) represents object pixel point coordinates
(x, y) is mapped in source images the coordinate value in X-direction by preset coordinate mapping table, and Lut_y (x, y) represents target pixel points
Coordinate (x, y) is mapped in source images the coordinate value in Y-direction, Src (Lut_x (x, y), Lut_y by preset coordinate mapping table
(x, y)) represent the position that object pixel point coordinates (x, y) is mapped to by preset coordinate mapping table in source images；
The coordinate map calculated between adjacent two pictures in each original image due to existing centainly
Overlapping region, for overlapping region, preset coordinate mapping table preserves two groups of correspondences for it；
Judge whether target pixel points are located at overlapping region, the source image vegetarian refreshments determined using the target pixel points
Number it is whether unique；If the number is unique, judge that the target pixel points are not located at the overlapping region；Otherwise sentence
The fixed target pixel points are located at the overlapping region；
Step S22, the image of correction carry out linear weighted function fusion；
It specifically includes：α is using the method for linear weighted function to the pixel value of two source image vegetarian refreshments of overlapping region to mix
To mixed pixel value, computational methods are as follows:
Idst(x, y)=α × Isrc1(x, y)+(1- α) × Isrc2(x, y)
Wherein, Idst(x, y) represents the pixel value at coordinate (x, y), I in target imagesrc1(x, y) represents original image 1
Pixel value at middle coordinate (x, y), Isrc2(x, y) represents the pixel value at coordinate (x, y) in original image 2, and α represents weighting system
Number.
Wherein, in the above-mentioned methods, video, audio are encoded in real time in the step S3, and by the sound after coding
Video is packaged to be specifically included with converting multimedia stream：
After panoramic camera obtains video data progress Panorama Mosaic, video is regarded using video frequency processing chip
Frequency coding carries out audio coding to the audio of microphone acquisition and audio and video is packaged；
The Video coding encodes panorama live video using the video code model of mainstream；
The audio coding encodes audio using the audio coding formats of mainstream；
Described be packaged to audio and video is to turn the video after coding and audio according to the reference format of MP4 (MPEG-4)
Change media stream into.
Wherein, in the above-mentioned methods, in the step S4 real time propelling movement for panorama live streaming media stream to CDN clouds
End, so that user terminal realizes that panorama broadcasting specifically includes by connecting the streaming media server in CDN high in the clouds：
It is that the media stream of encapsulation is pushed to CDN high in the clouds in real time by agreements such as RTMP/RTSP to push Streaming Media
According to live streaming platform protocol, live video resolution ratio and code check are set adaptively by camera APP, and in phase
Uniquely live streaming ID and streaming media server address, user terminal that CDN high in the clouds provides is inputted in machine APP to take to the Streaming Media
Device of being engaged in sends connection request, by default and establish after mutual authentication the association of panoramic camera and the streaming media server
View connection.
The real time panoramic live broadcasting method of the present invention, is handled original video data using fpga chip, compared with embedding
Enter formula processor, fpga chip is functioning in parallel, and single clock cycle interior energy is largely operated, especially suitable for video
Image procossing, and FPAG have the characteristics that it is low in energy consumption, will be in power consumption and cost all as the core of processing using fpga chip
To reduction.
The real time panoramic live broadcasting method of the present invention, solves that existing live broadcast system access device is more, system architecture is complicated, directly
Broadcast flow is complicated, net cast resolution ratio is relatively low, limited viewing angle, the problems such as real-time is poor, live streaming is unstable, and fuselage is small
Ingeniously, convenient for carrying, ultra high-definition image quality is suitble to the various occasions such as enterprise, individual, live news coverage.
The real time panoramic live broadcasting method of the present invention sets live video to differentiate according to the protocol self-adapting of live streaming platform
The parameters such as rate, code check carry out secondary transcoding without panorama live streaming Platform Server, greatly reduce the delay during live streaming,
Terminal user directly can carry out panorama live streaming viewing in client with its existing browser plug-in or APP with 3D/VR modes,
If wearing VR glasses, the hearing experience of the presence with impact, feeling of immersion will be obtained.
Description of the drawings
Fig. 1 is that the system structure for the example that real time panoramic live broadcast system of the present invention provides shows block diagram；
Fig. 2 is the method flow diagram for the example that real time panoramic live broadcasting method of the present invention provides.
Specific embodiment
Understand to make the object, technical solutions and advantages of the present invention clearer, With reference to embodiment and join
According to attached drawing, the present invention is described in more detail.It should be understood that these descriptions are merely illustrative, and it is not intended to limit this hair
Bright scope.In addition, in the following description, the description to known features and technology is omitted, to avoid this is unnecessarily obscured
The concept of invention.
At present, VR live streamings are mostly using panoramic camera, and the field range of panoramic camera is bigger than common camera, usually
Think that horizontal field of view reaches 360 degree, vertical visual field reaches and more than 180 degree, can meet the videorama of a place.Panorama
Camera needs complete video acquisition, video processing in machine, and the processing such as Video coding and plug-flow when carrying out panorama live streaming, need
A high performance home server is wanted, and is pushed to live streaming Platform Server and is broadcast live；Some even needs to take in platform
Business device end carries out secondary transcoding, could realize 3D or VR live streamings in client.
In addition, during viewing panorama live streaming, since panoramic picture is to be mapped on a spherical surface to realize that 3D or VR is shown, show
Area is than traditional 4:3 or 16:9 plane is much bigger, and high definition resolution ratio can not provide enough picture element densities and ensure picture
Clarity.
In order to solve, existing live broadcast system access device is more, system architecture is complicated, live streaming flow is complicated, net cast is differentiated
The problems such as rate is relatively low, limited viewing angle, real-time are poor, live streaming is unstable, the present invention provide a kind of real time panoramic live broadcast system and
Method.
Real time panoramic live broadcast system provided by the invention, as shown in Figure 1, specifically including：
Lens of panoramic camera module, for gathering video data；
Fpga chip is connected with the lens of panoramic camera module, is regarded with receive that lens of panoramic camera module gathered
Frequency evidence, and image mosaic processing is carried out to the video data；
Microphone, for gathering voice data；
Video frequency processing chip is connected respectively with the fpga chip and microphone, is spelled for receiving voice data and image
It connects the video data of processing and is compressed coding respectively, to be packaged into the media stream for the reference format for including video and audio
File；And
Communication module；It is connected with the video frequency processing chip, for multimedia stream file to be pushed out realization live streaming.
In above system, the lens of panoramic camera module include 4 fish-eye cameras and with each fish-eye camera
Corresponding imaging sensor；Four 90 degree of fish-eye camera intervals are fixed on same level, and the phase of each fish-eye camera
It immobilizes to position, therefore, the system acquired image quality is higher, and is not in rotation, not in same level
Etc. image mosaics situation about not getting up.
In above system, the system also includes gyroscope, the gyroscope encloses for gathering lens of panoramic camera module
Around default axial turning rate, and the video frequency processing chip is sent to calculate the current space appearance of panoramic camera
State, and calculate panorama live video.
In above system, the system also includes HDMI processing modules, the HDMI processing modules are used to export in real time
4K high-definition images, the HDMI processing modules are also accompanied with audio output, in specific implementation, the 4K sounds of HDMI processing modules output
Video data can be as the original input video stream of live streaming.
In above system, the network connection of the communication module is WLAN and/or wired wide area network；
When the network connection is WLAN, the communication module passes through with being equipped with the user terminal of camera APP
WLAN carries out communication connection, so that by camera APP panoramic camera is adjusted setting in terminal user, it is shown
WLAN is WIFI wireless networks；
When the network connection is wired wide area network, the communication module connects router by cable and accesses wide area
Net, and communication connection is established with the live streaming Platform Server in wide area network, so that terminal user passes through browser plug-in or straight
It broadcasts APP and panorama live streaming viewing is carried out using 3D/VR modes in client.
The real time panoramic live broadcast system of the present invention, is handled original video data using fpga chip, compared with embedding
Enter formula processor, fpga chip is functioning in parallel, and single clock cycle interior energy is largely operated, especially suitable for video
Image procossing, and FPAG have the characteristics that it is low in energy consumption, will be in power consumption and cost all as the core of processing using fpga chip
To reduction.
The real time panoramic live broadcast system of the present invention, it is more to solve existing live broadcast system access device, and system architecture is complicated, straight
The problems such as flow is complicated, net cast resolution ratio is relatively low, real-time is poor, live streaming is unstable is broadcast, fuselage is compact, convenient for carry-on
It carries, there is ultra high-definition image quality, be suitble to the various occasions such as enterprise, individual, live news coverage.
The real time panoramic live broadcast system of the present invention sets live video resolution ratio according to the protocol self-adapting of live streaming platform
The parameters such as (4K or 1080P), code check (2M, 4M, 8M) carry out secondary transcoding without panorama live streaming Platform Server, drop significantly
Delay during low live streaming, terminal user is with its existing browser plug-in or APP is broadcast live can directly be used in client
3D/VR modes carry out panorama live streaming viewing, if wearing VR glasses, will obtain the audiovisual of presence, feeling of immersion with impact
Impression.
Real time panoramic live broadcasting method provided by the invention, as shown in Fig. 2, including the following steps：
Initialize installation is broadcast live in step S1；
Step S2 reads the video data that every road fish-eye camera is gathered by fpga chip in real time, and to video counts
According to progress image mosaic；
Step S3, video, audio are encoded in real time, and the audio and video after coding is packaged to be converted into more matchmakers
Body stream；
Step S4, real time propelling movement for panorama live streaming media stream to CDN high in the clouds so that user terminal passes through connection
Streaming media server in CDN high in the clouds is to realize that panorama plays.
Initialize installation is broadcast live in the above method, in the step S1 to specifically include：
Panoramic camera and cable router are connected by cable, and passes through router and is linked into wide area network, to establish
Procotol connection needed for live streaming；
Camera APP and panoramic camera are connected by WLAN, according to streaming media server agreement, in camera APP
Live streaming parameter inside middle setting panorama camera, the live streaming parameter include the resolution ratio of live streaming panoramic video, code check, Yi Ji
Unique live streaming ID and the streaming media server address that CDN high in the clouds provides are inputted in camera APP, establish panoramic camera with it is described
The agreement connection of streaming media server.
Further, more Splicing models of panoramic camera, image ISP processing, operation can also be set by camera APP
The other parameters such as prompt tone.
In the above method, wherein, in the above-mentioned methods, figure is carried out to video data by fpga chip in the step S2
As splicing includes：
Step S21 carries out pattern distortion correction using coordinate map；
Step S22, the image of correction carry out linear weighted function fusion.
Wherein, in the above-mentioned methods, step S21 is specifically included：
Step S211 to given real time panoramic live broadcast system, calculates each flake using gridiron pattern scaling method and takes the photograph
As the spatial relationship parameters between the intrinsic parameter and distortion factor of head and adjacent fish-eye camera；
Step S212：Utilize the spatial relationship parameters meter between the intrinsic parameter of fish-eye camera and distortion factor and camera lens
The coordinate map between original image and target image is calculated, coordinate map saves what each fish-eye camera was gathered
One-to-many correspondence in the target image that original image and process image split-joint method are formed between pixel point coordinates,
In, target image is the image of distortion；
Step S213：The original image that each fish eye lens takes is mapped to by target according to the coordinate map
On image, using the pixel in the target image as target pixel points, using the preset coordinate mapping table, determine described
Target pixel points and the correspondence of the source image vegetarian refreshments on the pending original image；
Wherein, searched in original image as follows with the computational methods of the corresponding source image vegetarian refreshments of target pixel points：
Dst (x, y)=Src (Lut_x (x, y), Lut_y (x, y))
Wherein, Dst (x, y) denotation coordination is the target pixel points of (x, y), and Lut_x (x, y) represents object pixel point coordinates
(x, y) is mapped in source images the coordinate value in X-direction by preset coordinate mapping table, and Lut_y (x, y) represents target pixel points
Coordinate (x, y) is mapped in source images the coordinate value in Y-direction, Src (Lut_x (x, y), Lut_y by preset coordinate mapping table
(x, y)) represent the position that object pixel point coordinates (x, y) is mapped to by preset coordinate mapping table in source images；
The coordinate map calculated between adjacent two pictures in each original image due to existing centainly
Overlapping region, for overlapping region, preset coordinate mapping table preserves two groups of correspondences for it；
Judge whether target pixel points are located at overlapping region, the source image vegetarian refreshments determined using the target pixel points
Number it is whether unique；If the number is unique, judge that the target pixel points are not located at the overlapping region；Otherwise sentence
The fixed target pixel points are located at the overlapping region.
Wherein, in the above-mentioned methods, step S22 is specifically included；
α is done to the pixel value of two source image vegetarian refreshments of overlapping region using the method for linear weighted function and is mixed to get mixed pixel
Value, computational methods are as follows:
Idst(x, y)=α × Isrc1(x, y)+(1- α) × Isrc2(x, y)
Wherein, Idst(x, y) represents the pixel value at coordinate (x, y), I in target imagesrc1(x, y) represents original image 1
Pixel value at middle coordinate (x, y), Isrc2(x, y) represents the pixel value at coordinate (x, y) in original image 2, and α represents weighting system
Number.
In the present invention, lap left image " gradually going out " can be caused using the method for linear weighted function, and lap
Right image " is fade-in ", and which achieves gentle excessive so that image mosaic seems nature.
The method of linear weighted function needs to determine the region of fusion before image co-registration, and that chooses in embodiments of the present invention melts
Conjunction width is 128 pixels, then selects a fusion line as preferable as possible, in embodiments of the present invention, using dynamic
The method of state planning calculates fusion line, and certainly, fusion width can also choose other pixel numbers.
In the above method, video, audio are encoded in real time in the step S3, and the audio and video after coding are carried out
Encapsulation is specifically included with converting multimedia stream：
After panoramic camera obtains video data progress Panorama Mosaic, video is regarded using video frequency processing chip
Frequency coding carries out audio coding to the audio of microphone acquisition and audio and video is packaged；
The Video coding encodes panorama live video using the video code model of mainstream；
The audio coding encodes audio using the audio coding formats of mainstream；
Described be packaged to audio and video is to turn the video after coding and audio according to the reference format of MP4 (MPEG-4)
Change media stream into.
In the above method, in the step S4 real time propelling movement for panorama live streaming media stream to CDN high in the clouds, to use
Family terminal realizes that panorama broadcasting specifically includes by connecting the streaming media server in CDN high in the clouds：
It is that the media stream of encapsulation is pushed to CDN high in the clouds in real time by agreements such as RTMP/RTSP to push Streaming Media
According to live streaming platform protocol, by camera APP be set adaptively live video resolution ratio (4K or 1080P) and
Code check (2M, 4M, 8M), and with inputting in camera APP unique live streaming ID and the streaming media server that CDN high in the clouds provides
Location, user terminal send connection request to the streaming media server, by default and establish panorama after mutual authentication and take the photograph
Camera is connected with the agreement of the streaming media server.
Further, when needing to stop live streaming, the agreement connection needed for live streaming is terminated in camera APP, is specifically
Refer to when needing to stop live streaming, sent to the streaming media server in CDN high in the clouds and stop request, interrupt panoramic camera and Streaming Media
The agreement connection of server.
The real time panoramic live broadcasting method of the present invention, is handled original video data using fpga chip, compared with embedding
Enter formula processor, fpga chip is functioning in parallel, and single clock cycle interior energy is largely operated, especially suitable for video
Image procossing, and FPAG have the characteristics that it is low in energy consumption, will be in power consumption and cost all as the core of processing using fpga chip
To reduction.
The real time panoramic live broadcasting method of the present invention, solves that existing live broadcast system access device is more, system architecture is complicated, directly
Broadcast flow is complicated, net cast resolution ratio is relatively low, limited viewing angle, the problems such as real-time is poor, live streaming is unstable, and fuselage is small
Ingeniously, convenient for carrying, ultra high-definition image quality is suitble to the various occasions such as enterprise, individual, live news coverage.
The real time panoramic live broadcasting method of the present invention sets live video to differentiate according to the protocol self-adapting of live streaming platform
The parameters such as rate, code check carry out secondary transcoding without panorama live streaming Platform Server, greatly reduce the delay during live streaming,
Terminal user directly can carry out panorama live streaming viewing in client with its existing browser plug-in or APP with 3D/VR modes,
If wearing VR glasses, the hearing experience of the presence with impact, feeling of immersion will be obtained.
It should be appreciated that the above-mentioned specific embodiment of the present invention is used only for exemplary illustration or explains the present invention's
Principle, without being construed as limiting the invention.Therefore, that is done without departing from the spirit and scope of the present invention is any
Modification, equivalent substitution, improvement etc., should all be included in the protection scope of the present invention.In addition, appended claims purport of the present invention
Covering the whole variations fallen into scope and border or this scope and the equivalents on border and repairing
Change example.
Claims (10)
1. a kind of real time panoramic live broadcast system, which is characterized in that the system includes：
Lens of panoramic camera module, for gathering video data；
Fpga chip is connected with the lens of panoramic camera module, to receive the video counts that lens of panoramic camera module is gathered
According to, and image mosaic processing is carried out to the video data；
Microphone, for gathering voice data；
Video frequency processing chip is connected respectively with the fpga chip and microphone, for receiving at voice data and image mosaic
The video data of reason is simultaneously compressed coding respectively, to be packaged into the media stream for the reference format for including video and audio text
Part；And
Communication module；It is connected with the video frequency processing chip, for multimedia stream file to be pushed out realization live streaming.
2. system according to claim 1, which is characterized in that the lens of panoramic camera module includes 4 flakes and images
Head and with the corresponding imaging sensor of each fish-eye camera；Four 90 degree of fish-eye camera intervals are fixed on same water
Plane, and the relative position of each fish-eye camera immobilizes.
3. system according to claim 2, which is characterized in that the system also includes gyroscope, the gyroscope is used for
Lens of panoramic camera module is gathered around default axial turning rate, and it is complete to calculate to be sent to the video frequency processing chip
The current spatial attitude of scape video camera, and calculate panorama live video.
4. system according to claim 3, which is characterized in that the system also includes HDMI processing modules, the HDMI
For processing module for exporting 4K high-definition images in real time, the HDMI processing modules are also accompanied with audio output.
5. system according to claim 4, which is characterized in that the network connection of the communication module is WLAN
And/or wired wide area network；Wherein,
When the network connection is WLAN, the communication module passes through nothing with being equipped with the user terminal of camera APP
Line LAN carries out communication connection, so that by camera APP panoramic camera is adjusted setting in terminal user；
When the network connection is wired wide area network, the communication module connects router by cable and accesses wide area network,
And communication connection is established with the live streaming Platform Server in wide area network, so that terminal user passes through browser plug-in or live streaming
APP carries out panorama live streaming viewing in client using 3D/VR modes.
6. a kind of real time panoramic live broadcasting method, which is characterized in that include the following steps：
Initialize installation is broadcast live in step S1；
Step S2, reads the video data that every road fish-eye camera is gathered by fpga chip in real time, and to video data into
Row image mosaic；
Step S3 in real time encodes video, audio, and the audio and video after coding are packaged with converting multimedia
Stream；
Step S4, real time propelling movement is used for the media stream of panorama live streaming to CDN high in the clouds, so that user terminal is by connecting CDN clouds
Streaming media server in end is to realize that panorama plays.
7. it according to the method described in claim 6, is specifically included it is characterized in that, Initialize installation is broadcast live in the step S1：
Panoramic camera and cable router are connected by cable, and passes through router and is linked into wide area network, to establish live streaming
Required procotol connection；
Camera APP and panoramic camera are connected by WLAN, according to streaming media server agreement, set in camera APP
The live streaming parameter inside panorama camera is put, the live streaming parameter includes the resolution ratio of panoramic video, code check being broadcast live and in camera
Unique live streaming ID and the streaming media server address that CDN high in the clouds provides are inputted in APP, establishes panoramic camera and the stream matchmaker
The agreement connection of body server.
8. the method according to the description of claim 7 is characterized in that in the step S2 by fpga chip to video data into
Row image mosaic includes：
Step S21 carries out pattern distortion correction using coordinate map；
It specifically includes：To given real time panoramic live broadcast system, each fish is calculated using gridiron pattern scaling method by step S211
Spatial relationship parameters between the intrinsic parameter and distortion factor and adjacent fish-eye camera of Eye imaging head；
Step S212：It is calculated using the spatial relationship parameters between the intrinsic parameter of fish-eye camera and distortion factor and camera lens
Coordinate map between original image and target image, coordinate map save each fish-eye camera gathered it is original
One-to-many correspondence in the target image that image and process image split-joint method are formed between pixel point coordinates, wherein,
Target image is the image of distortion；
Step S213：The original image that each fish eye lens takes is mapped to by target image according to the coordinate map
On, using the pixel in the target image as target pixel points, using the preset coordinate mapping table, determine the target
The correspondence of pixel and the source image vegetarian refreshments on the pending original image；
Wherein, searched in original image as follows with the computational methods of the corresponding source image vegetarian refreshments of target pixel points：
Dst (x, y)=Src (Lut_x (x, y), Lut_y (x, y))
Wherein, Dst (x, y) denotation coordination is the target pixel points of (x, y), Lut_x (x, y) represent object pixel point coordinates (x,
Y) coordinate value being mapped to by preset coordinate mapping table in source images in X-direction, Lut_y (x, y) represent that target pixel points are sat
Mark (x, y) is mapped in source images the coordinate value in Y-direction, Src (Lut_x (x, y), Lut_y by preset coordinate mapping table
(x, y)) represent the position that object pixel point coordinates (x, y) is mapped to by preset coordinate mapping table in source images；
The coordinate map calculated is since there are certain overlappings between adjacent two pictures in each original image
Region, for overlapping region, preset coordinate mapping table preserves two groups of correspondences for it；
Judge whether target pixel points are located at overlapping region, utilize the number for the source image vegetarian refreshments that the target pixel points are determined
Whether mesh is unique；If the number is unique, judge that the target pixel points are not located at the overlapping region；Otherwise institute is judged
It states target pixel points and is located at the overlapping region；
Step S22, the image of correction carry out linear weighted function fusion；
It specifically includes：
α is done to the pixel value of two source image vegetarian refreshments of overlapping region using the method for linear weighted function and is mixed to get mixed pixel value,
Computational methods are as follows:
Idst(x, y)=α × Isrc1(x, y)+(1- α) × Isrc2(x, y)
Wherein, Idst(x, y) represents the pixel value at coordinate (x, y), I in target imagesrc1(x, y) represents to sit in original image 1
Mark the pixel value at (x, y), Isrc2(x, y) represents the pixel value at coordinate (x, y) in original image 2, and α represents weighting coefficient.
9. according to the method described in claim 8, it is characterized in that, encoded in real time to video, audio in the step S3,
And the audio and video after coding are packaged and are specifically included with converting multimedia stream：
After panoramic camera obtains video data progress Panorama Mosaic, video volume is carried out to video using video frequency processing chip
Code carries out audio coding to the audio of microphone acquisition and audio and video is packaged；
The Video coding encodes panorama live video using the video code model of mainstream；
The audio coding encodes audio using the audio coding formats of mainstream；
Described be packaged to audio and video is to be converted into the video after coding and audio according to the reference format of MP4 (MPEG-4)
Media stream.
10. according to the method described in claim 9, it is characterized in that, real time propelling movement is for panorama live streaming in the step S4
Media stream is to CDN high in the clouds, so that user terminal realizes that panorama plays tool by connecting the streaming media server in CDN high in the clouds
Body includes：
It is that the media stream of encapsulation is pushed to CDN high in the clouds in real time by agreements such as RTMP/RTSP to push Streaming Media
According to live streaming platform protocol, live video resolution ratio and code check are set adaptively by camera APP, and in camera
Input unique live streaming ID and the streaming media server address that CDN high in the clouds provides in APP, user terminal is to the streaming media service
Device sends connection request, by default and establish after mutual authentication the agreement of panoramic camera and the streaming media server
Connection.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711392097.5A CN108055472A (en) | 2017-12-21 | 2017-12-21 | A kind of real time panoramic live broadcast system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711392097.5A CN108055472A (en) | 2017-12-21 | 2017-12-21 | A kind of real time panoramic live broadcast system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108055472A true CN108055472A (en) | 2018-05-18 |
Family
ID=62131153
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711392097.5A Pending CN108055472A (en) | 2017-12-21 | 2017-12-21 | A kind of real time panoramic live broadcast system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108055472A (en) |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109275010A (en) * | 2018-11-21 | 2019-01-25 | 北京未来媒体科技股份有限公司 | A kind of 4K panorama is super to merge video terminal adaptation method and device |
| CN109391802A (en) * | 2018-12-20 | 2019-02-26 | 北京伊神华虹***工程技术有限公司 | A kind of method and apparatus for realizing real-time VR function |
| CN109741404A (en) * | 2019-01-10 | 2019-05-10 | 奥本未来(北京)科技有限责任公司 | A kind of mobile device-based optical field acquisition method |
| CN109788366A (en) * | 2018-12-30 | 2019-05-21 | 广东彼雍德云教育科技有限公司 | A kind of 3D interaction live broadcast system |
| CN110113628A (en) * | 2019-05-14 | 2019-08-09 | 宁夏融媒科技有限公司 | A kind of panorama live broadcast system under video-splicing server and moving condition |
| CN110557662A (en) * | 2018-05-30 | 2019-12-10 | 视联动力信息技术股份有限公司 | Audio and video data processing method and device |
| CN111405298A (en) * | 2020-02-17 | 2020-07-10 | 重庆邮电大学 | Android end-to-end live broadcast method based on KCP protocol |
| CN111416989A (en) * | 2020-04-28 | 2020-07-14 | 北京金山云网络技术有限公司 | Video live broadcast method and system and electronic equipment |
| CN112383788A (en) * | 2020-11-11 | 2021-02-19 | 成都威爱新经济技术研究院有限公司 | Live broadcast real-time image extraction system and method based on intelligent AI technology |
| CN113891171A (en) * | 2021-12-06 | 2022-01-04 | 杭州天宽科技有限公司 | Portable audio/video acquisition, storage and forwarding system based on RTMP/RTSP protocol |
| CN114449245A (en) * | 2022-01-28 | 2022-05-06 | 上海瞳观智能科技有限公司 | Real-time two-way video processing system and method based on programmable chip |
| CN114513675A (en) * | 2022-01-04 | 2022-05-17 | 桂林电子科技大学 | Construction method of panoramic video live broadcast system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106101503A (en) * | 2016-07-18 | 2016-11-09 | 优势拓展（北京）科技有限公司 | Real time panoramic Living Network video camera and system and method |
| CN106815808A (en) * | 2017-01-20 | 2017-06-09 | 长沙全度影像科技有限公司 | A kind of image split-joint method of utilization piecemeal computing |
| CN106875339A (en) * | 2017-02-22 | 2017-06-20 | 长沙全度影像科技有限公司 | A kind of fish eye images joining method based on strip scaling board |
| CN106954029A (en) * | 2017-02-24 | 2017-07-14 | 中国水利水电科学研究院 | A kind of panoramic video joining method and system |
-
2017
- 2017-12-21 CN CN201711392097.5A patent/CN108055472A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106101503A (en) * | 2016-07-18 | 2016-11-09 | 优势拓展（北京）科技有限公司 | Real time panoramic Living Network video camera and system and method |
| CN106815808A (en) * | 2017-01-20 | 2017-06-09 | 长沙全度影像科技有限公司 | A kind of image split-joint method of utilization piecemeal computing |
| CN106875339A (en) * | 2017-02-22 | 2017-06-20 | 长沙全度影像科技有限公司 | A kind of fish eye images joining method based on strip scaling board |
| CN106954029A (en) * | 2017-02-24 | 2017-07-14 | 中国水利水电科学研究院 | A kind of panoramic video joining method and system |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110557662A (en) * | 2018-05-30 | 2019-12-10 | 视联动力信息技术股份有限公司 | Audio and video data processing method and device |
| CN109275010A (en) * | 2018-11-21 | 2019-01-25 | 北京未来媒体科技股份有限公司 | A kind of 4K panorama is super to merge video terminal adaptation method and device |
| CN109391802A (en) * | 2018-12-20 | 2019-02-26 | 北京伊神华虹***工程技术有限公司 | A kind of method and apparatus for realizing real-time VR function |
| CN109788366A (en) * | 2018-12-30 | 2019-05-21 | 广东彼雍德云教育科技有限公司 | A kind of 3D interaction live broadcast system |
| CN109741404A (en) * | 2019-01-10 | 2019-05-10 | 奥本未来(北京)科技有限责任公司 | A kind of mobile device-based optical field acquisition method |
| CN110113628A (en) * | 2019-05-14 | 2019-08-09 | 宁夏融媒科技有限公司 | A kind of panorama live broadcast system under video-splicing server and moving condition |
| CN111405298A (en) * | 2020-02-17 | 2020-07-10 | 重庆邮电大学 | Android end-to-end live broadcast method based on KCP protocol |
| CN111416989A (en) * | 2020-04-28 | 2020-07-14 | 北京金山云网络技术有限公司 | Video live broadcast method and system and electronic equipment |
| CN112383788A (en) * | 2020-11-11 | 2021-02-19 | 成都威爱新经济技术研究院有限公司 | Live broadcast real-time image extraction system and method based on intelligent AI technology |
| CN113891171A (en) * | 2021-12-06 | 2022-01-04 | 杭州天宽科技有限公司 | Portable audio/video acquisition, storage and forwarding system based on RTMP/RTSP protocol |
| CN113891171B (en) * | 2021-12-06 | 2022-04-01 | 杭州天宽科技有限公司 | Portable audio/video acquisition, storage and forwarding system based on RTMP/RTSP protocol |
| CN114513675A (en) * | 2022-01-04 | 2022-05-17 | 桂林电子科技大学 | Construction method of panoramic video live broadcast system |
| CN114449245A (en) * | 2022-01-28 | 2022-05-06 | 上海瞳观智能科技有限公司 | Real-time two-way video processing system and method based on programmable chip |
| CN114449245B (en) * | 2022-01-28 | 2024-04-05 | 上海瞳观智能科技有限公司 | Real-time two-way video processing system and method based on programmable chip |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108055472A (en) | A kind of real time panoramic live broadcast system and method | |
| WO2018014495A1 (en) | Real-time panoramic live broadcast network camera and system and method | |
| Zhou et al. | Video coding optimization for virtual reality 360-degree source | |
| US11044455B2 (en) | Multiple-viewpoints related metadata transmission and reception method and apparatus | |
| US11303826B2 (en) | Method and device for transmitting/receiving metadata of image in wireless communication system | |
| US20150055937A1 (en) | Aggregating images and audio data to generate virtual reality content | |
| EP4287637A1 (en) | Information processing method and apparatus | |
| CN109889855A (en) | Intelligent panoramic net cast networked shopping system and method based on mobile APP | |
| CN104980697A (en) | Video transmission method for web camera | |
| WO2019041351A1 (en) | Real-time aliasing rendering method for 3d vr video and virtual three-dimensional scene | |
| CN106210525A (en) | For realizing camera and the method for net cast | |
| KR101851338B1 (en) | Device for displaying realistic media contents | |
| KR102214085B1 (en) | Method and apparatus for transmitting and receiving metadata for a plurality of viewpoints | |
| US10869017B2 (en) | Multiple-viewpoints related metadata transmission and reception method and apparatus | |
| CN104536763A (en) | Method for implementing online game simulating reality | |
| US11431901B2 (en) | Aggregating images to generate content | |
| CN105978926A (en) | Data transmission method and device | |
| CN106993177A (en) | A kind of 720 degree of panorama acquisition systems of binocular | |
| CN106604087A (en) | Rendering method for panoramic live streaming | |
| CN106657977A (en) | Virtual reality equipment with panoramic camera shooting function and panoramic video call method | |
| WO2013067898A1 (en) | Method and terminal for transmitting information | |
| CN109756683A (en) | Panorama audio-video method for recording, device, storage medium and computer equipment | |
| US20230086988A1 (en) | Method and apparatus for processing multi-view video, device and storage medium | |
| US20230007067A1 (en) | Bidirectional presentation datastream | |
| CN116016977A (en) | Live broadcast-based virtual same-platform wheat connection interaction method, computer equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20180518 |