WO2019128342A1 - Video processing method and apparatus - Google Patents

Video processing method and apparatus Download PDF

Info

Publication number
WO2019128342A1
WO2019128342A1 PCT/CN2018/107236 CN2018107236W WO2019128342A1 WO 2019128342 A1 WO2019128342 A1 WO 2019128342A1 CN 2018107236 W CN2018107236 W CN 2018107236W WO 2019128342 A1 WO2019128342 A1 WO 2019128342A1
Authority
WO
WIPO (PCT)
Prior art keywords
video
dimensional
frame
dimensional image
dimensional video
Prior art date
Application number
PCT/CN2018/107236
Other languages
French (fr)
Chinese (zh)
Inventor
涂成义
尹海生
朱兴昌
Original Assignee
中兴通讯股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中兴通讯股份有限公司 filed Critical 中兴通讯股份有限公司
Publication of WO2019128342A1 publication Critical patent/WO2019128342A1/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/218—Source of audio or video content, e.g. local disk arrays
    • H04N21/21805—Source of audio or video content, e.g. local disk arrays enabling multiple viewpoints, e.g. using a plurality of cameras
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/23418—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
    • H04N21/2365—Multiplexing of several video streams
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/816—Monomedia components thereof involving special video data, e.g 3D video
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
    • H04N21/845—Structuring of content, e.g. decomposing content into time segments
    • H04N21/8455—Structuring of content, e.g. decomposing content into time segments involving pointers to the content, e.g. pointers to the I-frames of the video stream

Definitions

  • the present invention relates to the field of computers, and in particular to a video processing method and apparatus.
  • VR Virtual Reality
  • VR video leads the continuous breakthrough and application of VR technology.
  • the VR video can be viewed by using the helmet and the mobile terminal, the device is simple, the operation is convenient, and the market is recognized.
  • VR video is spliced into a 360-degree panoramic video, pushed to the terminal, and equipped with a display peripheral such as a helmet to provide an immersive viewing experience.
  • a display peripheral such as a helmet
  • 360-degree viewing without dead ends can be achieved.
  • the VR video in the related art also has limitations.
  • the VR video can not reflect the immersion, and the VR video cannot be played normally, but it also takes up the bandwidth and traffic of the VR video.
  • the embodiment of the invention provides a video processing method and device, so as to at least solve the problem that the VR video cannot be played when the video playing terminal does not support the VR in the related art.
  • a video processing method comprising: restoring a picture of a virtual reality VR video to a two-dimensional image; and generating a two-dimensional video of the plurality of views using the two-dimensional image.
  • restoring the picture of the virtual reality VR video to the two-dimensional image comprises: parsing each frame of the VR video, and restoring each frame picture from the tile mode to the panorama mode by the spherical model rendering; Each frame of the panoramic mode performs view segmentation to obtain a two-dimensional image of a plurality of views.
  • dividing each frame of the panoramic mode into a view angle to obtain a two-dimensional image of multiple views including one of: splitting a spherical surface of each frame in the panoramic mode into a hexahedron, Using the texture mapping method with the center of the circle as the center point, using a 120-degree angle of view, projecting in different directions, and obtaining images of six angles of up, down, left, and right; and customizing the spherical surface of each frame in the panoramic mode
  • generating the two-dimensional video of the plurality of viewing angles by using the two-dimensional image comprises: merging the two-dimensional image into a multiplexed code stream according to the location information of the two-dimensional image; and performing the multiple multiplexed code streams Packing slices produces a two-dimensional video of multiple perspectives.
  • the two-dimensional video of the multiple views is described by different index numbers.
  • outputting the two-dimensional video includes: receiving perspective selection information; selecting a specified two-dimensional video corresponding to the perspective in all the two-dimensional videos according to the perspective selection information; and outputting the code stream of the specified two-dimensional video.
  • the method before the outputting the two-dimensional video, the method further includes: determining that the video playing terminal meets at least one of the following conditions: VR play is not supported, and the VR wearable device is not connected.
  • the method further includes: outputting the two-dimensional video.
  • outputting the two-dimensional video comprises: outputting a two-dimensional video of one or more viewing angles.
  • the two-dimensional video of the multiple viewing angles is output, the two-dimensional video of the multiple viewing angles is played on the screen in the video playing terminal.
  • a video playback apparatus comprising: a restoration module configured to restore a picture of a virtual reality VR video to a two-dimensional image; and a generation module configured to generate the plurality of images using the two-dimensional image Two-dimensional video of a perspective.
  • the restoring module includes: a parsing unit configured to parse each frame of the VR video, and restore each frame from a tile mode to a pan mode by a spherical model rendering; dividing the unit, setting In order to divide the viewing angle of each frame of the panoramic mode, a two-dimensional image of a plurality of viewing angles is obtained.
  • the dividing unit is further configured to: divide a spherical surface of each frame in the panoramic mode into a hexahedron, and adopt a texture mapping manner with a center coordinate as a center point, and adopt a 120 degree angle of view.
  • the generating module includes: a convergence unit configured to aggregate the two-dimensional image into multiple streams according to position information of the two-dimensional image; and a processing unit configured to perform the multiple streams Packing slices produces a two-dimensional video of multiple perspectives.
  • the two-dimensional video of the multiple views is described by different index numbers.
  • the device further includes: an output module configured to output the two-dimensional video.
  • the output module includes: a receiving unit configured to receive view selection information; and a selecting unit configured to select a specified two-dimensional video corresponding to the view angle among all the two-dimensional videos according to the view selection information; an output unit, setting To output the code stream of the specified two-dimensional video.
  • the device further includes: a determining module, configured to determine, before the output module outputs the two-dimensional video, that the video playing terminal meets at least one of the following conditions: the VR playing is not supported, and the VR wearing device is not connected.
  • a determining module configured to determine, before the output module outputs the two-dimensional video, that the video playing terminal meets at least one of the following conditions: the VR playing is not supported, and the VR wearing device is not connected.
  • a storage medium is also provided.
  • the storage medium is set to store program code set to perform the following steps:
  • a two-dimensional video of a plurality of viewing angles is generated using the two-dimensional image.
  • the video of the two-dimensional (2D) view is regenerated by the reverse reduction of the video, which solves the problem that the VR video cannot be played when the video play terminal does not support the VR in the related art, and reduces the transmission bandwidth of the video play terminal. Requirements to reduce traffic consumption and storage capacity.
  • FIG. 1 is a flow chart of a video processing method according to an embodiment of the present invention.
  • FIG. 2 is a block diagram showing the structure of a video processing apparatus according to an embodiment of the present invention.
  • FIG. 3 is a schematic view of a spherical projection in an embodiment of the present invention.
  • FIG. 4 is a schematic diagram of a hexahedron division in an embodiment of the present invention.
  • FIG. 5 is a schematic diagram of texture mapping in an embodiment of the present invention.
  • FIG. 6 is a schematic diagram of a free view splitting in an embodiment of the present invention.
  • FIG. 7 is a schematic diagram of code stream re-generation in an embodiment of the present invention.
  • FIG. 8 is a flowchart of a client selection perspective in an embodiment of the present invention.
  • FIG. 9 is a schematic diagram of a VR live broadcast in an embodiment of the present invention.
  • FIG. 10 is a schematic diagram of VR on-demand in an embodiment of the present invention.
  • FIG. 11 is a schematic diagram of comparison of two players in an embodiment of the present invention.
  • FIG. 13 is a flowchart of automatically switching a viewing angle according to a connection of an external device according to an embodiment of the present invention
  • FIG. 14 is a schematic diagram of clicking a video selection view video in an embodiment of the present invention.
  • 15 is a schematic diagram of a video frame selection angle of view in an embodiment of the present invention.
  • FIG. 1 is a flowchart of a video processing method according to an embodiment of the present invention. As shown in FIG. 1, the process includes the following steps:
  • Step S102 restoring the picture of the virtual reality VR video to a two-dimensional image
  • Step S104 generating a two-dimensional video of a plurality of viewing angles using the two-dimensional image.
  • the video of the two-dimensional (2D) view is regenerated by the reverse reduction of the video, which solves the problem that the VR video cannot be played when the video play terminal does not support the VR in the related art, and reduces the transmission bandwidth of the video play terminal. Requirements to reduce traffic consumption and storage capacity.
  • the execution body of the foregoing steps may be a video distribution platform, such as a content delivery network (CDN), or a video presentation platform, such as a playback terminal, but is not limited thereto.
  • a video distribution platform such as a content delivery network (CDN)
  • CDN content delivery network
  • a video presentation platform such as a playback terminal
  • restoring the picture of the virtual reality VR video to the two-dimensional image comprises:
  • S12 Performing a view angle division on each frame of the panoramic mode to obtain a two-dimensional image of a plurality of viewing angles.
  • the angle of view can be up, down, left, and right, front and rear, or angles of view according to different shooting positions.
  • each frame of the panoramic mode is divided into a view angle to obtain a two-dimensional image of multiple viewing angles, which may be, but is not limited to, the following:
  • the spherical surface of each frame in the panoramic mode is divided into hexahedrons, and the center of the center coordinates is used as a center point by using texture mapping, and the image is projected in different directions by using a 120-degree angle of view, and images of six angles of up, down, left, and right are obtained.
  • the spherical surface of each frame in the panoramic mode is automatically divided.
  • the texture mapping is used as the center point of the center of the circle, the Y axis is the positive direction, and the positive X direction is the starting point of 0 degrees, and is sequentially divided around the Z axis.
  • M maps are generated and rotated around the X axis to generate M maps, and images of different viewing angles of 2M frames are obtained, and M is an integer greater than 1.
  • Taking a custom angle of view as a 90-degree angle of view taking the origin as the center, the Y-axis is the positive direction, and the X-axis positive direction is the 0-degree starting point, which is sequentially divided around the Z-axis.
  • the principle of segmentation also uses texture mapping. Generate 4 graphs. At the same time, four pictures are generated by rotating around the X axis, for a total of 8 frames.
  • generating a two-dimensional video of the plurality of views using the two-dimensional image includes:
  • S21 converge the two-dimensional image into a multi-stream according to the position information of the two-dimensional image; each frame is marked according to the position after the segmentation is completed, and the position information of the image frame of the same view has relevance;
  • the two-dimensional video of multiple viewing angles is described by different index numbers, and the index numbers that can be generated in the order of cutting are sequentially.
  • the method further includes: outputting the two-dimensional video.
  • Output 2D video including:
  • S31 Receive view selection information; the view selection information is used to indicate a viewing angle selected by the user or a preset policy selection (a presentation angle of the video playing terminal), and may also be a switch when the video is currently being played;
  • the outputting the two-dimensional video comprises: outputting two-dimensional video of one or more viewing angles, and when simultaneously outputting the two-dimensional video of the multiple viewing angles, the video playing terminal may play and display by using a split screen or a multi-frequency manner.
  • the method before the outputting the two-dimensional video, the method further includes: determining that the video playing terminal meets at least one of the following conditions: the VR playing is not supported, and the VR wearing device is not connected. This saves network bandwidth and storage space and prevents the terminal from downloading video types that cannot be played.
  • the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware, but in many cases, the former is A better implementation.
  • the technical solution of the present invention which is essential or contributes to the prior art, may be embodied in the form of a software product stored in a storage medium (such as ROM/RAM, disk,
  • the optical disc includes a number of instructions for causing a terminal device (which may be a cell phone, a computer, a server, or a network device, etc.) to perform the methods described in various embodiments of the present invention.
  • a video processing device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and has not been described again.
  • the term "module” may implement a combination of software and/or hardware of a predetermined function.
  • the apparatus described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware, is also possible and contemplated.
  • FIG. 2 is a structural block diagram of a video processing apparatus according to an embodiment of the present invention. As shown in FIG. 2, the apparatus includes:
  • the restoration module 20 is configured to restore the picture of the virtual reality VR video to a two-dimensional image
  • the generating module 22 is configured to generate a two-dimensional video of a plurality of viewing angles using the two-dimensional image.
  • the restoration module includes: a parsing unit configured to parse each frame of the VR video, and restore each frame from the tile mode to the panorama mode through the spherical model rendering; the segmentation unit is set to the pan mode
  • a parsing unit configured to parse each frame of the VR video, and restore each frame from the tile mode to the panorama mode through the spherical model rendering; the segmentation unit is set to the pan mode
  • Each frame of the picture is divided into a view angle to obtain a two-dimensional image of a plurality of angles of view.
  • the dividing unit is further configured to: divide the spherical surface of each frame in the panoramic mode into a hexahedron, adopt a texture mapping manner with the center of the circle as a center point, adopt a 120 degree angle of view, and perform in different directions.
  • Projection obtain images of six perspectives from top to bottom, left and right, and left and right; separate the spherical surface of each frame in the panorama mode, using texture mapping as the center point of the center of the circle, the positive direction of the Y axis, and the positive direction of the X axis It is a starting point of 0 degrees, and is sequentially divided around the Z axis to generate M maps, and rotates around the X axis to generate M maps, and obtain images of different viewing angles of 2M frames.
  • the generating module includes: a convergence unit configured to aggregate the two-dimensional image into multiple streams according to the position information of the two-dimensional image; and the processing unit is configured to package the multiple streams to generate multiple perspectives. Dimensional video.
  • the two-dimensional video of multiple viewing angles is described by different index numbers.
  • the device further includes: an output module configured to output the two-dimensional video.
  • the output module may include: a receiving unit configured to receive the viewing angle selection information; and a selecting unit configured to select the specified two-dimensional video of the corresponding viewing angle among all the two-dimensional videos according to the viewing angle selection information; and the output unit is configured to output the specified two-dimensional video Code stream.
  • the device further includes: a determining module, configured to determine, before the output module outputs the two-dimensional video, that the video playing terminal meets at least one of the following conditions: the VR playing is not supported, and the VR wearing device is not connected.
  • a determining module configured to determine, before the output module outputs the two-dimensional video, that the video playing terminal meets at least one of the following conditions: the VR playing is not supported, and the VR wearing device is not connected.
  • each of the above modules may be implemented by software or hardware.
  • the foregoing may be implemented by, but not limited to, the foregoing modules are all located in the same processor; or, the above modules are in any combination.
  • the forms are located in different processors.
  • the invention relates to a video processing scheme for VR video playing, and particularly relates to a method for selecting a viewing angle.
  • the invention provides a method for multi-view selection of VR video based on reverse synthesis.
  • the VR video is generally combined with a multi-channel video signal to generate a 360-degree panoramic video.
  • the splicing seam is sharpened and the like to eliminate the traces to the greatest extent.
  • the plane projection is performed to form a tiled image for transmission.
  • the video stream is distributed to the video inverse synthesizer, the video is first restored to a spherical video by a reverse algorithm, and the video is divided into multiple 2D video streams according to a defined perspective and pushed to the CDN side.
  • the terminal presents the video of the VR, the related view video is selected and pulled down to play, so that different viewing angle videos can be viewed without the related viewing device such as a helmet.
  • This embodiment can be divided into three parts from the flow.
  • the first part is the reverse synthesis of the video inverse synthesizer.
  • the VR video is restored by the inverse algorithm and segmented into 2D images.
  • the second part is to regenerate the code stream, slice the 2D image, package it into a new code stream, and describe it in the main m3u8 file.
  • the third part is to select the perspective of the client and present the 2D video. The following instructions are presented:
  • FIG. 3 is a schematic diagram of a spherical projection in the embodiment of the present invention, wherein the planar video forms a spherical texture VR video through spherical projection.
  • FIG. 4 is a schematic diagram of the hexahedron segmentation in the embodiment of the present invention, and finally forms six figures of up, down, left, and right.
  • FIG. 5 is a schematic diagram of texture mapping in the embodiment of the present invention.
  • a 120-degree angle of view is used to shoot in different directions, that is, to project in different directions, and finally an image of each face is formed.
  • Customized viewing angle segmentation you can customize the segmentation angle of view, taking the 90-degree angle of view as an example, taking the origin as the center, the Y-axis is the positive direction, and the X-axis positive direction is the 0-degree starting point, which is sequentially divided around the Z-axis.
  • the principle of segmentation is the same.
  • Four maps are generated by texture mapping.
  • four pictures are generated by rotating around the X axis, for a total of 8 frames.
  • FIG. 6 is a schematic diagram of free viewing angle division in the embodiment of the present invention.
  • FIG. 7 is a schematic diagram of re-generation of a code stream in an embodiment of the present invention. After each frame is divided, the frame is marked according to the position, and the frame of the same position is aggregated and coded to form a single block.
  • the description file of the code stream such as index1.m3u8, index2.m3u8....
  • the index numbers generated in the order in which they are cut are described in the main index.m3u8:
  • FIG. 8 is a flowchart of a client selection perspective in the embodiment of the present invention.
  • the client first downloads the index.m3u8 file of the VR video for parsing, and obtains the angle between the angle and the sub-m3u8 file. Correspondence.
  • the sub-m3u8 description file of the corresponding perspective is downloaded, and the stream is pulled down for playback.
  • FIG. 9 is a schematic diagram of a VR live broadcast in an embodiment of the present invention.
  • the head end collects images through a multi-channel camera, transmits the image to a synthesizer of the panoramic video, completes the stitching and composite projection of the panoramic video, and pushes the real-time message transmission protocol (
  • the code of the Real Time Message Protocol (RTMP) is forwarded to the transcoder.
  • the transcoder converts the rtmp protocol stream into a hls (HTTP Live Streaming) format stream and pushes it to the inverse synthesis server.
  • RTMP Real Time Message Protocol
  • the reverse synthesis of the VR panoramic video is completed in real time on the server of the video reverse synthesis, and the segmented 2D video is buffered into the cache and pushed to the CDN node.
  • FIG. 10 is a schematic diagram of VR on-demand in the embodiment of the present invention.
  • the VR panoramic video is injected into the encoder in the format of mp4, and the encoder is transcoded into the HLS format and pushed to the video inverse synthesis server.
  • the reverse synthesis of the VR panoramic video is completed on the video inverse synthesis node, and the segmented 2D video is buffered into the cache and pushed to the CDN node. Compared with VR live reverse synthesis, no real-time considerations are needed.
  • Example 3 the client perspective selection of VR video playback is not supported.
  • FIG. 11 is a schematic diagram of comparison of two players in the embodiment of the present invention.
  • FIG. 12 is a flow chart of client selection that does not support VR video in the embodiment of the present invention, including:
  • the user selects one of the videos to play in full screen.
  • the viewing angle is automatically switched according to the connection of the external device.
  • FIG. 13 is a flowchart of automatically switching a viewing angle according to a connection of an external device according to an embodiment of the present invention, including:
  • the user selects one of the videos to play in full screen.
  • Example 5 When playing VR video, switch to the corresponding perspective 2D video playback.
  • the VR video can be adjusted by gestures and gyroscopes.
  • FIG. 14 is a schematic diagram of clicking a video selection view video in an embodiment of the present invention.
  • FIG. 15 is a schematic diagram of the selection of the viewing angle video in the embodiment of the present invention. .
  • the code stream of the 2D view of the code stream is regenerated by the reverse synthesis of the video, and the client downloads and plays the game through different modes, which effectively solves the problem that the terminal does not support VR video playback, and has no peripherals.
  • the 2D video playback corresponding to the selection of the viewing angle reduces the bandwidth requirement and reduces the consumption of the traffic.
  • Embodiments of the present invention also provide a storage medium.
  • the above storage medium may be configured to store program code for performing the following steps:
  • the foregoing storage medium may include, but not limited to, a USB flash drive, a Read-Only Memory (ROM), a Random Access Memory (RAM), a mobile hard disk, and a magnetic memory.
  • ROM Read-Only Memory
  • RAM Random Access Memory
  • a mobile hard disk e.g., a hard disk
  • magnetic memory e.g., a hard disk
  • the processor executes according to the stored program code in the storage medium:
  • a two-dimensional image of multiple perspectives is generated using a two-dimensional image.
  • modules or steps of the present invention described above can be implemented by a general-purpose computing device that can be centralized on a single computing device or distributed across a network of multiple computing devices. Alternatively, they may be implemented by program code executable by the computing device such that they may be stored in the storage device by the computing device and, in some cases, may be different from the order herein.
  • the steps shown or described are performed, or they are separately fabricated into individual integrated circuit modules, or a plurality of modules or steps thereof are fabricated as a single integrated circuit module.
  • the invention is not limited to any specific combination of hardware and software.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Databases & Information Systems (AREA)
  • Television Signal Processing For Recording (AREA)
  • Image Generation (AREA)
  • Controls And Circuits For Display Device (AREA)

Abstract

Provided are a video processing method and apparatus. The method comprises: restoring a picture of a virtual reality (VR) video back to two-dimensional images; and using the two-dimensional images to generate a two-dimensional video of multiple viewing angles. The problem in the relevant art that VR video cannot be played when a video playing terminal does not support VR is solved.

Description

视频处理方法及装置Video processing method and device 技术领域Technical field
本发明涉及计算机领域，具体而言，涉及一种视频处理方法及装置。The present invention relates to the field of computers, and in particular to a video processing method and apparatus.
背景技术Background technique
相关技术中的虚拟现实(Virtual Reality，简称为VR)技术深刻影响我们未来的生活，包括医学，娱乐，教育等。VR视频作为VR娱乐行业的第一梯队，引领VR技术的不断突破和应用。特别是在谷歌，三星等巨头在移动终端的VR布局，利用头盔与手机终端配合，即可观看VR视频，设备简单，操作便利，得到了市场的认可。Virtual Reality (VR) technology in related technologies profoundly affects our future life, including medicine, entertainment, and education. As the first echelon of VR entertainment industry, VR video leads the continuous breakthrough and application of VR technology. Especially in the VR layout of giants such as Google and Samsung in the mobile terminal, the VR video can be viewed by using the helmet and the mobile terminal, the device is simple, the operation is convenient, and the market is recognized.
VR视频通过拼接成360度全景视频，推送到终端后，配合头盔等显示外设，提供沉浸式的观看体验，与普通的视频相比较，可以实现360度无死角的观赏。但是相关技术中的VR视频也存在局限性，在无法支持VR视频播放的设备上，或者无法利用外设的情况下，比如在电视屏上将会呈现拉伸平面的视频效果，在没有配套VR头盔的手机上，VR视频也无法体现出沉浸感，不能正常播放VR视频，但同时占用了VR视频的带宽和流量。VR video is spliced into a 360-degree panoramic video, pushed to the terminal, and equipped with a display peripheral such as a helmet to provide an immersive viewing experience. Compared with ordinary video, 360-degree viewing without dead ends can be achieved. However, the VR video in the related art also has limitations. In the case of a device that cannot support VR video playback, or in the case where the peripheral device cannot be used, for example, a video effect of the stretched plane will be presented on the television screen, and there is no supporting VR. On the mobile phone of the helmet, the VR video can not reflect the immersion, and the VR video cannot be played normally, but it also takes up the bandwidth and traffic of the VR video.
针对相关技术中存在的上述问题，目前尚未发现有效的解决方案。In view of the above problems in the related art, no effective solution has been found yet.
发明内容Summary of the invention
本发明实施例提供了一种视频处理方法及装置，以至少解决相关技术中在视频播放终端不支持VR时不能播放VR视频的问题。The embodiment of the invention provides a video processing method and device, so as to at least solve the problem that the VR video cannot be played when the video playing terminal does not support the VR in the related art.
根据本发明的一个实施例，提供了一种视频处理方法，包括：将虚拟现实VR视频的画面还原成二维图像；使用所述二维图像生成多个视角的二维视频。According to an embodiment of the present invention, there is provided a video processing method comprising: restoring a picture of a virtual reality VR video to a two-dimensional image; and generating a two-dimensional video of the plurality of views using the two-dimensional image.
可选地，将虚拟现实VR视频的画面还原成二维图像包括：解析所述VR视频中的每一帧画面，并通过球面模型渲染将每一帧画面从平铺模式 还原成全景模式；将所述全景模式中的每一帧画面进行视角分割，得到多个视角的二维图像。Optionally, restoring the picture of the virtual reality VR video to the two-dimensional image comprises: parsing each frame of the VR video, and restoring each frame picture from the tile mode to the panorama mode by the spherical model rendering; Each frame of the panoramic mode performs view segmentation to obtain a two-dimensional image of a plurality of views.
可选地，将所述全景模式中的每一帧画面进行视角分割，得到多个视角的二维图像，包括以下之一：将所述全景模式中的每一帧画面的球面分割成六面体，采用纹理映射的方式以圆心坐标为中心点，采用120度视角，在不同的方向进行投影，得到上下左右前后六个视角的图像；将所述全景模式中的每一帧画面的球面进行自定义分割，采用纹理映射的方式以圆心坐标为中心点，Y轴为正方向，X轴正方向为0度起始点，绕Z轴依次分割，生成M个图，并绕X轴旋转生成M个图，得到2M帧不同视角的图像。Optionally, dividing each frame of the panoramic mode into a view angle to obtain a two-dimensional image of multiple views, including one of: splitting a spherical surface of each frame in the panoramic mode into a hexahedron, Using the texture mapping method with the center of the circle as the center point, using a 120-degree angle of view, projecting in different directions, and obtaining images of six angles of up, down, left, and right; and customizing the spherical surface of each frame in the panoramic mode Dividing, using the texture mapping method with the center of the circle as the center point, the Y axis is the positive direction, the positive X direction is the 0 degree starting point, and the Z axis is sequentially divided, and M pictures are generated, and the M pictures are generated by rotating around the X axis. , get 2M frames of different perspective images.
可选地，使用所述二维图像生成多个视角的二维视频包括：按照所述二维图像的位置信息将所述二维图像汇聚成多路码流；对所述多路码流进行打包切片生成多个视角的二维视频。Optionally, generating the two-dimensional video of the plurality of viewing angles by using the two-dimensional image comprises: merging the two-dimensional image into a multiplexed code stream according to the location information of the two-dimensional image; and performing the multiple multiplexed code streams Packing slices produces a two-dimensional video of multiple perspectives.
可选地，所述多个视角的二维视频通过不同的索引号进行描述。Optionally, the two-dimensional video of the multiple views is described by different index numbers.
可选地，输出所述二维视频包括：接收视角选择信息；根据所述视角选择信息在所有二维视频中选择对应视角的指定二维视频；输出所述指定二维视频的码流。Optionally, outputting the two-dimensional video includes: receiving perspective selection information; selecting a specified two-dimensional video corresponding to the perspective in all the two-dimensional videos according to the perspective selection information; and outputting the code stream of the specified two-dimensional video.
可选地，在输出所述二维视频之前，所述方法还包括：确定视频播放终端满足以下条件至少之一：不支持VR播放，未连接VR穿戴设备。Optionally, before the outputting the two-dimensional video, the method further includes: determining that the video playing terminal meets at least one of the following conditions: VR play is not supported, and the VR wearable device is not connected.
可选的，在使用所述二维图像生成多个视角的二维视频之后，所述方法还包括：输出所述二维视频。Optionally, after generating the two-dimensional video of the plurality of views using the two-dimensional image, the method further includes: outputting the two-dimensional video.
可选地，输出所述二维视频包括：输出一个或多个视角的二维视频。Optionally, outputting the two-dimensional video comprises: outputting a two-dimensional video of one or more viewing angles.
可选地，在输出多个视角的二维视频时，所述多个视角的二维视频在视频播放终端分屏播放。Optionally, when the two-dimensional video of the multiple viewing angles is output, the two-dimensional video of the multiple viewing angles is played on the screen in the video playing terminal.
根据本发明的另一个实施例，提供了一种视频播放装置，包括：还原模块，设置为将虚拟现实VR视频的画面还原成二维图像；生成模块，设置为使用所述二维图像生成多个视角的二维视频。According to another embodiment of the present invention, there is provided a video playback apparatus comprising: a restoration module configured to restore a picture of a virtual reality VR video to a two-dimensional image; and a generation module configured to generate the plurality of images using the two-dimensional image Two-dimensional video of a perspective.
可选地，所述还原模块包括：解析单元，设置为解析所述VR视频中的每一帧画面，并通过球面模型渲染将每一帧画面从平铺模式还原成全景模式；分割单元，设置为将所述全景模式中的每一帧画面进行视角分割，得到多个视角的二维图像。Optionally, the restoring module includes: a parsing unit configured to parse each frame of the VR video, and restore each frame from a tile mode to a pan mode by a spherical model rendering; dividing the unit, setting In order to divide the viewing angle of each frame of the panoramic mode, a two-dimensional image of a plurality of viewing angles is obtained.
可选地，所述分割单元还设置为以下之一：将所述全景模式中的每一帧画面的球面分割成六面体，采用纹理映射的方式以圆心坐标为中心点，采用120度视角，在不同的方向进行投影，得到上下左右前后六个视角的图像；将所述全景模式中的每一帧画面的球面进行自定义分割，采用纹理映射的方式以圆心坐标为中心点，Y轴为正方向，X轴正方向为0度起始点，绕Z轴依次分割，生成M个图，并绕X轴旋转生成M个图，得到2M帧不同视角的图像。Optionally, the dividing unit is further configured to: divide a spherical surface of each frame in the panoramic mode into a hexahedron, and adopt a texture mapping manner with a center coordinate as a center point, and adopt a 120 degree angle of view. Projecting in different directions, obtaining images of six viewing angles from top to bottom, left and right; and performing custom segmentation on the spherical surface of each frame in the panoramic mode, using texture mapping as the center point of the center of the circle, and the Y axis is positive Direction, the positive direction of the X-axis is 0 degree starting point, and is sequentially divided around the Z-axis, and M pictures are generated, and M pictures are rotated around the X-axis to obtain images of different viewing angles of 2M frames.
可选地，所述生成模块包括：汇聚单元，设置为按照所述二维图像的位置信息将所述二维图像汇聚成多路码流；处理单元，设置为对所述多路码流进行打包切片生成多个视角的二维视频。Optionally, the generating module includes: a convergence unit configured to aggregate the two-dimensional image into multiple streams according to position information of the two-dimensional image; and a processing unit configured to perform the multiple streams Packing slices produces a two-dimensional video of multiple perspectives.
可选地，所述多个视角的二维视频通过不同的索引号进行描述。Optionally, the two-dimensional video of the multiple views is described by different index numbers.
可选的，所述装置还包括：输出模块，设置为输出所述二维视频。Optionally, the device further includes: an output module configured to output the two-dimensional video.
可选地，所述输出模块包括：接收单元，设置为接收视角选择信息；选择单元，设置为根据所述视角选择信息在所有二维视频中选择对应视角的指定二维视频；输出单元，设置为输出所述指定二维视频的码流。Optionally, the output module includes: a receiving unit configured to receive view selection information; and a selecting unit configured to select a specified two-dimensional video corresponding to the view angle among all the two-dimensional videos according to the view selection information; an output unit, setting To output the code stream of the specified two-dimensional video.
可选地，所述装置还包括：确定模块，设置为在所述输出模块输出所述二维视频之前，确定视频播放终端满足以下条件至少之一：不支持VR播放，未连接VR穿戴设备。Optionally, the device further includes: a determining module, configured to determine, before the output module outputs the two-dimensional video, that the video playing terminal meets at least one of the following conditions: the VR playing is not supported, and the VR wearing device is not connected.
根据本发明的又一个实施例，还提供了一种存储介质。该存储介质设置为存储设置为执行以下步骤的程序代码：According to still another embodiment of the present invention, a storage medium is also provided. The storage medium is set to store program code set to perform the following steps:
将虚拟现实VR视频的画面还原成二维图像；Restore the picture of the virtual reality VR video to a two-dimensional image;
使用所述二维图像生成多个视角的二维视频。A two-dimensional video of a plurality of viewing angles is generated using the two-dimensional image.
通过本发明，通过视频的逆向还原，重新生成二维(2D)视角的视频，解决了相关技术中在视频播放终端不支持VR时不能播放VR视频的问题，并降低了视频播放终端对传输带宽的要求，减少流量的消耗和存储容量。Through the invention, the video of the two-dimensional (2D) view is regenerated by the reverse reduction of the video, which solves the problem that the VR video cannot be played when the video play terminal does not support the VR in the related art, and reduces the transmission bandwidth of the video play terminal. Requirements to reduce traffic consumption and storage capacity.
附图说明DRAWINGS
此处所说明的附图用来提供对本发明的进一步理解，构成本申请的一部分，本发明的示意性实施例及其说明用于解释本发明，并不构成对本发明的不当限定。在附图中：The drawings described herein are intended to provide a further understanding of the invention, and are intended to be a part of the invention. In the drawing:
图1是根据本发明实施例的视频处理方法的流程图；1 is a flow chart of a video processing method according to an embodiment of the present invention;
图2是根据本发明实施例的视频处理装置的结构框图；2 is a block diagram showing the structure of a video processing apparatus according to an embodiment of the present invention;
图3是本发明实施例中球面投影示意图；3 is a schematic view of a spherical projection in an embodiment of the present invention;
图4是本发明实施例中六面体分割示意图；4 is a schematic diagram of a hexahedron division in an embodiment of the present invention;
图5是本发明实施例中纹理映射示意图；FIG. 5 is a schematic diagram of texture mapping in an embodiment of the present invention; FIG.
图6是本发明实施例中自由视角分割示意图；6 is a schematic diagram of a free view splitting in an embodiment of the present invention;
图7是本发明实施例中码流重新生成示意图；7 is a schematic diagram of code stream re-generation in an embodiment of the present invention;
图8是本发明实施例中客户端选择视角流程图；8 is a flowchart of a client selection perspective in an embodiment of the present invention;
图9是本发明实施例中VR直播的示意图；9 is a schematic diagram of a VR live broadcast in an embodiment of the present invention;
图10是本发明实施例中VR点播的示意图；FIG. 10 is a schematic diagram of VR on-demand in an embodiment of the present invention; FIG.
图11是本发明实施例中两种播放器比较的示意图；11 is a schematic diagram of comparison of two players in an embodiment of the present invention;
图12是本发明实施例中不支持VR视频的客户端选择流程图；12 is a flowchart of client selection that does not support VR video in the embodiment of the present invention;
图13是本发明实施例中根据外接设备的连接自动切换视角的流程图；FIG. 13 is a flowchart of automatically switching a viewing angle according to a connection of an external device according to an embodiment of the present invention; FIG.
图14是本发明实施例中点击视频选择视角视频的示意图；14 is a schematic diagram of clicking a video selection view video in an embodiment of the present invention;
图15是本发明实施例中宫格选择视角视频的示意图。15 is a schematic diagram of a video frame selection angle of view in an embodiment of the present invention.
具体实施方式Detailed ways
下文中将参考附图并结合实施例来详细说明本发明。需要说明的是， 在不冲突的情况下，本申请中的实施例及实施例中的特征可以相互组合。The invention will be described in detail below with reference to the drawings in conjunction with the embodiments. It should be noted that the embodiments in the present application and the features in the embodiments may be combined with each other without conflict.
需要说明的是，本发明的说明书和权利要求书及上述附图中的术语“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。It is to be understood that the terms "first", "second" and the like in the specification and claims of the present invention are used to distinguish similar objects, and are not necessarily used to describe a particular order or order.
实施例1Example 1
在本实施例中提供了一种视频处理方法，图1是根据本发明实施例的视频处理方法的流程图，如图1所示，该流程包括如下步骤：A video processing method is provided in this embodiment. FIG. 1 is a flowchart of a video processing method according to an embodiment of the present invention. As shown in FIG. 1, the process includes the following steps:
步骤S102，将虚拟现实VR视频的画面还原成二维图像；Step S102, restoring the picture of the virtual reality VR video to a two-dimensional image;
步骤S104，使用二维图像生成多个视角的二维视频。Step S104, generating a two-dimensional video of a plurality of viewing angles using the two-dimensional image.
通过上述步骤，通过视频的逆向还原，重新生成二维(2D)视角的视频，解决了相关技术中在视频播放终端不支持VR时不能播放VR视频的问题，并降低了视频播放终端对传输带宽的要求，减少流量的消耗和存储容量。Through the above steps, the video of the two-dimensional (2D) view is regenerated by the reverse reduction of the video, which solves the problem that the VR video cannot be played when the video play terminal does not support the VR in the related art, and reduces the transmission bandwidth of the video play terminal. Requirements to reduce traffic consumption and storage capacity.
可选地，上述步骤的执行主体可以为视频分发平台，如内容分发网络(Content Delivery Network，简称为CDN)，也可以是视频呈现平台、如播放终端等，但不限于此。Optionally, the execution body of the foregoing steps may be a video distribution platform, such as a content delivery network (CDN), or a video presentation platform, such as a playback terminal, but is not limited thereto.
可选地，将虚拟现实VR视频的画面还原成二维图像包括：Optionally, restoring the picture of the virtual reality VR video to the two-dimensional image comprises:
S11，解析VR视频中的每一帧画面，并通过球面模型渲染将每一帧画面从平铺模式还原成全景模式；S11, parsing each frame of the VR video, and restoring each frame from the tile mode to the panorama mode through the spherical model rendering;
S12，将全景模式中的每一帧画面进行视角分割，得到多个视角的二维图像。视角可以是上下左右前后视角，或者是按照不同拍摄机位的视角。S12: Performing a view angle division on each frame of the panoramic mode to obtain a two-dimensional image of a plurality of viewing angles. The angle of view can be up, down, left, and right, front and rear, or angles of view according to different shooting positions.
在本实施例中，将全景模式中的每一帧画面进行视角分割，得到多个视角的二维图像，可以但不限于为以下方式：In this embodiment, each frame of the panoramic mode is divided into a view angle to obtain a two-dimensional image of multiple viewing angles, which may be, but is not limited to, the following:
将全景模式中的每一帧画面的球面分割成六面体，采用纹理映射的方式以圆心坐标为中心点，采用120度视角，在不同的方向进行投影，得到上下左右前后六个视角的图像；The spherical surface of each frame in the panoramic mode is divided into hexahedrons, and the center of the center coordinates is used as a center point by using texture mapping, and the image is projected in different directions by using a 120-degree angle of view, and images of six angles of up, down, left, and right are obtained.
将全景模式中的每一帧画面的球面进行自定义分割，采用纹理映射的方式以圆心坐标为中心点，Y轴为正方向，X轴正方向为0度起始点，绕Z轴依次分割，生成M个图，并绕X轴旋转生成M个图，得到2M帧不同视角的图像，M是大于1的整数。以自定义的分割视角为90度视角为例，以原点为中心，Y轴为正方向，X轴正方向为0度起始点，绕Z轴依次分割，分割的原理同样采用纹理映射的方式，生成4个图。同时，绕X轴旋转也生成4个图，一共8帧图。The spherical surface of each frame in the panoramic mode is automatically divided. The texture mapping is used as the center point of the center of the circle, the Y axis is the positive direction, and the positive X direction is the starting point of 0 degrees, and is sequentially divided around the Z axis. M maps are generated and rotated around the X axis to generate M maps, and images of different viewing angles of 2M frames are obtained, and M is an integer greater than 1. Taking a custom angle of view as a 90-degree angle of view, taking the origin as the center, the Y-axis is the positive direction, and the X-axis positive direction is the 0-degree starting point, which is sequentially divided around the Z-axis. The principle of segmentation also uses texture mapping. Generate 4 graphs. At the same time, four pictures are generated by rotating around the X axis, for a total of 8 frames.
可选的，使用二维图像生成多个视角的二维视频包括：Optionally, generating a two-dimensional video of the plurality of views using the two-dimensional image includes:
S21，按照二维图像的位置信息将二维图像汇聚成多路码流；每帧图在分割完成后，会按照位置进行标记，同一视角的图像帧的位置信息具备关联性；S21: converge the two-dimensional image into a multi-stream according to the position information of the two-dimensional image; each frame is marked according to the position after the segmentation is completed, and the position information of the image frame of the same view has relevance;
S22，对多路码流进行打包切片生成多个视角的二维视频。S22, packetizing and slicing the multiplexed code stream to generate a two-dimensional video of multiple viewing angles.
可选的，多个视角的二维视频通过不同的索引号进行描述，可以以切割的依次顺序生成的索引号。Optionally, the two-dimensional video of multiple viewing angles is described by different index numbers, and the index numbers that can be generated in the order of cutting are sequentially.
可选的，在使用二维图像生成多个视角的二维视频之后，方法还包括：输出二维视频。输出二维视频包括：Optionally, after generating the two-dimensional video of the multiple views using the two-dimensional image, the method further includes: outputting the two-dimensional video. Output 2D video including:
S31，接收视角选择信息；视角选择信息用于指示用户选择的或者预设策略择优选择的观看视角(视频播放终端的呈现视角)，在当前正在播放视频时，也可以是切换；S31: Receive view selection information; the view selection information is used to indicate a viewing angle selected by the user or a preset policy selection (a presentation angle of the video playing terminal), and may also be a switch when the video is currently being played;
S32，根据视角选择信息在所有二维视频中选择对应视角的指定二维视频；S32. Select a specified two-dimensional video corresponding to the view angle among all the two-dimensional videos according to the view selection information.
S33，输出指定二维视频的码流。可选的，输出二维视频包括：输出一个或多个视角的二维视频，在同时输出多个视角的二维视频时，视频播放终端可以通过分屏或者多频的方式进行播放呈现。S33. Output a code stream specifying the two-dimensional video. Optionally, the outputting the two-dimensional video comprises: outputting two-dimensional video of one or more viewing angles, and when simultaneously outputting the two-dimensional video of the multiple viewing angles, the video playing terminal may play and display by using a split screen or a multi-frequency manner.
可选的，在输出二维视频之前，方法还包括：确定视频播放终端满足以下条件至少之一：不支持VR播放，未连接VR穿戴设备。由此可以节省网络带宽和存储空间，避免终端下载不能播放的视频类型。Optionally, before the outputting the two-dimensional video, the method further includes: determining that the video playing terminal meets at least one of the following conditions: the VR playing is not supported, and the VR wearing device is not connected. This saves network bandwidth and storage space and prevents the terminal from downloading video types that cannot be played.
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到根据上述实施例的方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本发明的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台终端设备(可以是手机，计算机，服务器，或者网络设备等)执行本发明各个实施例所述的方法。Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware, but in many cases, the former is A better implementation. Based on such understanding, the technical solution of the present invention, which is essential or contributes to the prior art, may be embodied in the form of a software product stored in a storage medium (such as ROM/RAM, disk, The optical disc includes a number of instructions for causing a terminal device (which may be a cell phone, a computer, a server, or a network device, etc.) to perform the methods described in various embodiments of the present invention.
实施例2Example 2
在本实施例中还提供了一种视频处理装置，该装置用于实现上述实施例及优选实施方式，已经进行过说明的不再赘述。如以下所使用的，术语“模块”可以实现预定功能的软件和/或硬件的组合。尽管以下实施例所描述的装置较佳地以软件来实现，但是硬件，或者软件和硬件的组合的实现也是可能并被构想的。In the embodiment, a video processing device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and has not been described again. As used below, the term "module" may implement a combination of software and/or hardware of a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware, is also possible and contemplated.
图2是根据本发明实施例的视频处理装置的结构框图，如图2所示，该装置包括：FIG. 2 is a structural block diagram of a video processing apparatus according to an embodiment of the present invention. As shown in FIG. 2, the apparatus includes:
还原模块20，设置为将虚拟现实VR视频的画面还原成二维图像；The restoration module 20 is configured to restore the picture of the virtual reality VR video to a two-dimensional image;
生成模块22，设置为使用二维图像生成多个视角的二维视频。The generating module 22 is configured to generate a two-dimensional video of a plurality of viewing angles using the two-dimensional image.
可选的，还原模块包括：解析单元，设置为解析VR视频中的每一帧画面，并通过球面模型渲染将每一帧画面从平铺模式还原成全景模式；分割单元，设置为将全景模式中的每一帧画面进行视角分割，得到多个视角的二维图像。Optionally, the restoration module includes: a parsing unit configured to parse each frame of the VR video, and restore each frame from the tile mode to the panorama mode through the spherical model rendering; the segmentation unit is set to the pan mode Each frame of the picture is divided into a view angle to obtain a two-dimensional image of a plurality of angles of view.
可选的，分割单元还设置为以下之一：将全景模式中的每一帧画面的球面分割成六面体，采用纹理映射的方式以圆心坐标为中心点，采用120度视角，在不同的方向进行投影，得到上下左右前后六个视角的图像；将全景模式中的每一帧画面的球面进行自定义分割，采用纹理映射的方式以圆心坐标为中心点，Y轴为正方向，X轴正方向为0度起始点，绕Z轴依 次分割，生成M个图，并绕X轴旋转生成M个图，得到2M帧不同视角的图像。Optionally, the dividing unit is further configured to: divide the spherical surface of each frame in the panoramic mode into a hexahedron, adopt a texture mapping manner with the center of the circle as a center point, adopt a 120 degree angle of view, and perform in different directions. Projection, obtain images of six perspectives from top to bottom, left and right, and left and right; separate the spherical surface of each frame in the panorama mode, using texture mapping as the center point of the center of the circle, the positive direction of the Y axis, and the positive direction of the X axis It is a starting point of 0 degrees, and is sequentially divided around the Z axis to generate M maps, and rotates around the X axis to generate M maps, and obtain images of different viewing angles of 2M frames.
可选的，生成模块包括：汇聚单元，设置为按照二维图像的位置信息将二维图像汇聚成多路码流；处理单元，设置为对多路码流进行打包切片生成多个视角的二维视频。Optionally, the generating module includes: a convergence unit configured to aggregate the two-dimensional image into multiple streams according to the position information of the two-dimensional image; and the processing unit is configured to package the multiple streams to generate multiple perspectives. Dimensional video.
可选的，多个视角的二维视频通过不同的索引号进行描述。Optionally, the two-dimensional video of multiple viewing angles is described by different index numbers.
可选的，装置还包括：输出模块，设置为输出二维视频。输出模块可以包括：接收单元，设置为接收视角选择信息；选择单元，设置为根据视角选择信息在所有二维视频中选择对应视角的指定二维视频；输出单元，设置为输出指定二维视频的码流。Optionally, the device further includes: an output module configured to output the two-dimensional video. The output module may include: a receiving unit configured to receive the viewing angle selection information; and a selecting unit configured to select the specified two-dimensional video of the corresponding viewing angle among all the two-dimensional videos according to the viewing angle selection information; and the output unit is configured to output the specified two-dimensional video Code stream.
可选的，装置还包括：确定模块，设置为在输出模块输出二维视频之前，确定视频播放终端满足以下条件至少之一：不支持VR播放，未连接VR穿戴设备。Optionally, the device further includes: a determining module, configured to determine, before the output module outputs the two-dimensional video, that the video playing terminal meets at least one of the following conditions: the VR playing is not supported, and the VR wearing device is not connected.
需要说明的是，上述各个模块是可以通过软件或硬件来实现的，对于后者，可以通过以下方式实现，但不限于此：上述模块均位于同一处理器中；或者，上述各个模块以任意组合的形式分别位于不同的处理器中。It should be noted that each of the above modules may be implemented by software or hardware. For the latter, the foregoing may be implemented by, but not limited to, the foregoing modules are all located in the same processor; or, the above modules are in any combination. The forms are located in different processors.
实施例3Example 3
本实施例是本申请的可选实施例，用于结合具体的实例对本申请进行详细说明：This embodiment is an optional embodiment of the present application, and is used to describe the present application in detail with reference to specific examples:
本发明涉VR视频播放时的视频处理方案，特别涉及一种视角选择的方法。The invention relates to a video processing scheme for VR video playing, and particularly relates to a method for selecting a viewing angle.
本发明提供一种基于逆向合成的VR视频多视角选择的方法，相关技术中，VR视频一般是通过多路视频信号采集后，经过合成拼接，生成360度的全景视频。在拼接的过程中，对于拼接缝通过锐化等多种处理，最大程度消除痕迹，拼接完成后，进行平面投影，形成平铺的图像进行传输。在本实施例的方案中，当视频码流分发到视频逆向合成器，通过逆向算法，首先将视频还原成球面视频，根据定义的视角将视频分割成多路2D视频 码流推送到CDN侧。终端在呈现VR的视频时，选择相关的视角视频并下拉播放，实现在没有头盔等相关观看设备的情况下也能观看不同的视角视频。The invention provides a method for multi-view selection of VR video based on reverse synthesis. In the related art, the VR video is generally combined with a multi-channel video signal to generate a 360-degree panoramic video. In the process of splicing, the splicing seam is sharpened and the like to eliminate the traces to the greatest extent. After the splicing is completed, the plane projection is performed to form a tiled image for transmission. In the solution of this embodiment, when the video stream is distributed to the video inverse synthesizer, the video is first restored to a spherical video by a reverse algorithm, and the video is divided into multiple 2D video streams according to a defined perspective and pushed to the CDN side. When the terminal presents the video of the VR, the related view video is selected and pulled down to play, so that different viewing angle videos can be viewed without the related viewing device such as a helmet.
本实施例从流程上可以分为三部分，第一部分为视频逆向合成器的逆向合成，通过逆向算法将VR视频还原，并分割成2D图像。第二部分为重新生成码流，将2D图像切片，打包形成新的码流，并在主m3u8文件中描述。第三部分为客户端选择视角，进行2D视频的呈现。下面进行展示说明书：This embodiment can be divided into three parts from the flow. The first part is the reverse synthesis of the video inverse synthesizer. The VR video is restored by the inverse algorithm and segmented into 2D images. The second part is to regenerate the code stream, slice the 2D image, package it into a new code stream, and describe it in the main m3u8 file. The third part is to select the perspective of the client and present the 2D video. The following instructions are presented:
视频逆向合成器的逆向合成：Reverse synthesis of video inverse synthesizer:
VR视频是以平面的模式分发到视频逆向合成器节点，其接收到码流后，解析其每一帧的画面，其画面为平铺模式，通过球面模型渲染，还原成VR全景模式，如图3所示，图3是本发明实施例中球面投影示意图，平面视频通过球面投影形成球面纹理的VR视频。The VR video is distributed to the video inverse synthesizer node in a planar mode. After receiving the code stream, it parses the picture of each frame. The picture is tiled and is rendered by the spherical model and restored to the VR panoramic mode. 3, FIG. 3 is a schematic diagram of a spherical projection in the embodiment of the present invention, wherein the planar video forms a spherical texture VR video through spherical projection.
在视角分割过程中，采用以下两种方式。In the process of viewing angle segmentation, the following two methods are used.
六面体分割，将球面默认分割成六面体，如图4所示，图4是本发明实施例中六面体分割示意图，最终形成上下左右前后六个图。The hexahedron segmentation divides the spherical surface into a hexahedron by default. As shown in FIG. 4, FIG. 4 is a schematic diagram of the hexahedron segmentation in the embodiment of the present invention, and finally forms six figures of up, down, left, and right.
分割采用纹理映射的方式，其原理如图5所示，图5是本发明实施例中纹理映射示意图。以圆心坐标为观看者位置，为了保证其图片有余量，采用120度视角，对不同方向进行拍摄，即在不同的方向进行投影，最终形成每个面的图像。The method of texture mapping is adopted, and the principle is shown in FIG. 5. FIG. 5 is a schematic diagram of texture mapping in the embodiment of the present invention. With the coordinates of the center of the circle as the position of the viewer, in order to ensure that the picture has a margin, a 120-degree angle of view is used to shoot in different directions, that is, to project in different directions, and finally an image of each face is formed.
自定义视角分割，可以自定义的分割视角，以90度视角为例，以原点为中心，Y轴为正方向，X轴正方向为0度起始点，绕Z轴依次分割，分割的原理同样采用纹理映射的方式，生成4个图。同时，绕X轴旋转也生成4个图，一共8帧图。如图6所示，图6是本发明实施例中自由视角分割示意图。Customized viewing angle segmentation, you can customize the segmentation angle of view, taking the 90-degree angle of view as an example, taking the origin as the center, the Y-axis is the positive direction, and the X-axis positive direction is the 0-degree starting point, which is sequentially divided around the Z-axis. The principle of segmentation is the same. Four maps are generated by texture mapping. At the same time, four pictures are generated by rotating around the X axis, for a total of 8 frames. As shown in FIG. 6, FIG. 6 is a schematic diagram of free viewing angle division in the embodiment of the present invention.
重新生成码流：Regenerate the code stream:
如图7所示，图7是本发明实施例中码流重新生成示意图，每帧图分 割完成后，按照位置进行标记，相同位置的帧图汇聚一路码流，并进行切片打包，形成一个个子码流的描述文件，如index1.m3u8,index2.m3u8…。以切割的依次顺序生成的索引号在主index.m3u8中进行描述:As shown in FIG. 7, FIG. 7 is a schematic diagram of re-generation of a code stream in an embodiment of the present invention. After each frame is divided, the frame is marked according to the position, and the frame of the same position is aggregated and coded to form a single block. The description file of the code stream, such as index1.m3u8, index2.m3u8.... The index numbers generated in the order in which they are cut are described in the main index.m3u8:
#以下定义切割的视角#以下defined cutting perspective
View angle＝90View angle=90
#以下定义子m3u8的#以下的名字m3u8
v1＝index1.m3u8V1=index1.m3u8
v2＝index2.m3u8V2=index2.m3u8
v3＝index3.m3u8V3=index3.m3u8
v4＝index4.m3u8V4=index4.m3u8
…...
客户端视角选择：Client perspective selection:
如图8所示，图8是本发明实施例中客户端选择视角流程图，客户端在播放的过程中，首先下载VR视频的index.m3u8文件进行解析，获取其角度与子m3u8文件之间的对应关系。当用户选择某个视角的视频观看，下载对应的视角的子m3u8描述文件，并下拉码流进行播放。As shown in FIG. 8, FIG. 8 is a flowchart of a client selection perspective in the embodiment of the present invention. In the process of playing, the client first downloads the index.m3u8 file of the VR video for parsing, and obtains the angle between the angle and the sub-m3u8 file. Correspondence. When the user selects a video view of a certain perspective, the sub-m3u8 description file of the corresponding perspective is downloaded, and the stream is pulled down for playback.
下面结合具体的实例对本发明的技术方案做进一步描述。The technical solutions of the present invention are further described below in conjunction with specific examples.
实例一，VR直播逆向合成Example 1, VR live reverse synthesis
如图9所示，图9是本发明实施例中VR直播的示意图，头端通过多路摄像头采集图像，传输到全景视频的合成器，完成全景视频的拼接合成投影，推送实时消息传输协议(Real Time Message Protocol，简称为rtmp)的码流至转码器，转码器将rtmp协议码流转码成hls(HTTP Live Streaming)格式码流推送到逆合成服务器。As shown in FIG. 9, FIG. 9 is a schematic diagram of a VR live broadcast in an embodiment of the present invention. The head end collects images through a multi-channel camera, transmits the image to a synthesizer of the panoramic video, completes the stitching and composite projection of the panoramic video, and pushes the real-time message transmission protocol ( The code of the Real Time Message Protocol (RTMP) is forwarded to the transcoder. The transcoder converts the rtmp protocol stream into a hls (HTTP Live Streaming) format stream and pushes it to the inverse synthesis server.
在视频逆向合成的服务器上实时完成VR全景视频的逆向合成，并将分割的2D视频缓存到cache中，并推送到CDN节点。The reverse synthesis of the VR panoramic video is completed in real time on the server of the video reverse synthesis, and the segmented 2D video is buffered into the cache and pushed to the CDN node.
实例二，VR点播逆向合成Example 2, VR on-demand reverse synthesis
如图10所示，图10是本发明实施例中VR点播的示意图，VR全景视频以mp4的格式注入到编码器，编码器转码成HLS格式，推送到视频逆合成服务器。As shown in FIG. 10, FIG. 10 is a schematic diagram of VR on-demand in the embodiment of the present invention. The VR panoramic video is injected into the encoder in the format of mp4, and the encoder is transcoded into the HLS format and pushed to the video inverse synthesis server.
在视频逆合成节点上完成VR全景视频的逆向合成，并将分割的2D视频缓存到cache中，并推送到CDN节点。与VR直播逆向合成相比较，无需实时性考虑。The reverse synthesis of the VR panoramic video is completed on the video inverse synthesis node, and the segmented 2D video is buffered into the cache and pushed to the CDN node. Compared with VR live reverse synthesis, no real-time considerations are needed.
实例三，不支持VR视频播放的客户端视角选择。Example 3, the client perspective selection of VR video playback is not supported.
VR视频的播放时需要播放器支持的，VR播放器在播放的过程中，将平铺的视频图像还原成全景视频，并进行分屏。如果播放器不支持，即普通播放器，那么看到的是原始的平铺画面。目前大部分的机顶盒是不支持VR播放的，如图11所示，图11是本发明实施例中两种播放器比较的示意图。When the VR video is played, the player needs to support it. During the playback process, the VR player restores the tiled video image to the panoramic video and splits the screen. If the player does not support it, that is, the normal player, then the original tile is seen. At present, most of the set top boxes do not support VR playback, as shown in FIG. 11, FIG. 11 is a schematic diagram of comparison of two players in the embodiment of the present invention.
在此种情况下，通过判断是否支持VR播放来进行视角的选择。流程如图12所示，图12是本发明实施例中不支持VR视频的客户端选择流程图，包括：In this case, the selection of the angle of view is made by judging whether or not VR playback is supported. The process is shown in FIG. 12, which is a flow chart of client selection that does not support VR video in the embodiment of the present invention, including:
下载主m3u8文件，并解析Download the main m3u8 file and parse it
判断是否包含视角的子m3u8文件，如果不包含，则为普通视频，直接播放。如果包含子文件，则为VR视频。Determine whether the sub-m3u8 file containing the view is included. If it is not included, it is a normal video and played directly. If it contains subfiles, it is a VR video.
判断当前的设备是否支持VR视频的播放，如果支持，则调用播放器呈现VR视频，否则下载多路分割的2D视频，并以宫格的模式呈现，如四宫格。Determine whether the current device supports playback of VR video. If it supports, call the player to render VR video. Otherwise, download the multi-channel 2D video and display it in the mode of the palace, such as the four-square grid.
用户选择其中某一路视频进行全屏播放。The user selects one of the videos to play in full screen.
实例四，根据外接设备的连接自动切换视角。In the fourth example, the viewing angle is automatically switched according to the connection of the external device.
对于支持VR视频播放的设备，如果体验VR的沉浸感，需要外接头盔等设备。此案例在外接头盔设备是，切换到VR模式，当断连后，切换 到视角选择模式。流程如图13所示，图13是本发明实施例中根据外接设备的连接自动切换视角的流程图，包括：For devices that support VR video playback, if you experience the immersion of VR, you need an external helmet and other equipment. In this case, the external helmet device is switched to VR mode, and when disconnected, switches to the angle selection mode. FIG. 13 is a flowchart of automatically switching a viewing angle according to a connection of an external device according to an embodiment of the present invention, including:
下载主m3u8文件，并解析Download the main m3u8 file and parse it
判断是否包含视角的子m3u8文件，如果不包含，则为普通视频，直接播放。如果包含子m3u8文件，则为VR视频。Determine whether the sub-m3u8 file containing the view is included. If it is not included, it is a normal video and played directly. If the subm3u8 file is included, it is a VR video.
判断是否外接VR设备，如果外接，则直接下载并播放VR视频，如果没有外接设备，则下载多路分割的2D视频，并以宫格的模式呈现，如四宫格。Determine whether the external VR device is externally connected. If there is no external device, download and play the VR video directly. If there is no external device, download the multi-channel 2D video and display it in the mode of the palace, such as the four-square grid.
用户选择其中某一路视频进行全屏播放。The user selects one of the videos to play in full screen.
实例五，播放VR视频时，切换到对应的视角2D视频播放。Example 5: When playing VR video, switch to the corresponding perspective 2D video playback.
在支持VR视频播放的设备上，裸眼观看时，可以通过手势，陀螺仪调整VR视频的视角，但此时需要下载全景的VR视频，带宽要求高，流量大。可以通过以下两种方式快速切换到对应的2D视角视频。On a device that supports VR video playback, when viewing with naked eyes, the VR video can be adjusted by gestures and gyroscopes. However, it is necessary to download panoramic VR video, which requires high bandwidth and large traffic. You can quickly switch to the corresponding 2D view video in the following two ways.
点击视频区域，通过当前触点的角度，计算其对应的2D视频索引，并下载对应的视频播放。示意图如图14所示，图14是本发明实施例中点击视频选择视角视频的示意图。Click on the video area, calculate the corresponding 2D video index through the angle of the current contact, and download the corresponding video playback. FIG. 14 is a schematic diagram of clicking a video selection view video in an embodiment of the present invention.
在播放VR视频时，同时展示2D视频的宫格播放，用户可以通过点击宫格自动切换到相关的2D视频，如图15所示，图15是本发明实施例中宫格选择视角视频的示意图。When the VR video is played, the 2G video can be displayed at the same time. The user can automatically switch to the related 2D video by clicking the grid, as shown in FIG. 15. FIG. 15 is a schematic diagram of the selection of the viewing angle video in the embodiment of the present invention. .
通过本实施例，对于全景视频，通过视频的逆向合成，重新生成码流2D视角的码流，客户端通过不同的方式下载播放，有效的解决了在终端不支持VR视频播放，以及没有外设情况下无法观看VR视频的问题，通过视角的选择对应的2D视频播放，降低了对带宽的要求，减少流量的消耗。With the embodiment, for the panoramic video, the code stream of the 2D view of the code stream is regenerated by the reverse synthesis of the video, and the client downloads and plays the game through different modes, which effectively solves the problem that the terminal does not support VR video playback, and has no peripherals. In the case that the VR video cannot be viewed, the 2D video playback corresponding to the selection of the viewing angle reduces the bandwidth requirement and reduces the consumption of the traffic.
实施例4Example 4
本发明的实施例还提供了一种存储介质。可选地，在本实施例中，上 述存储介质可以被设置为存储用于执行以下步骤的程序代码：Embodiments of the present invention also provide a storage medium. Optionally, in the embodiment, the above storage medium may be configured to store program code for performing the following steps:
S1，将虚拟现实VR视频的画面还原成二维图像；S1, reducing the picture of the virtual reality VR video into a two-dimensional image;
S2，使用二维图像生成多个视角的二维视频。S2, using a two-dimensional image to generate a two-dimensional video of multiple viewing angles.
可选地，在本实施例中，上述存储介质可以包括但不限于：U盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、移动硬盘、磁碟或者光盘等各种可以存储程序代码的介质。Optionally, in this embodiment, the foregoing storage medium may include, but not limited to, a USB flash drive, a Read-Only Memory (ROM), a Random Access Memory (RAM), a mobile hard disk, and a magnetic memory. A variety of media that can store program code, such as a disc or a disc.
可选地，在本实施例中，处理器根据存储介质中已存储的程序代码执行：Optionally, in this embodiment, the processor executes according to the stored program code in the storage medium:
将虚拟现实VR视频的画面还原成二维图像；Restore the picture of the virtual reality VR video to a two-dimensional image;
使用二维图像生成多个视角的二维视频。A two-dimensional image of multiple perspectives is generated using a two-dimensional image.
可选地，本实施例中的具体示例可以参考上述实施例及可选实施方式中所描述的示例，本实施例在此不再赘述。For example, the specific examples in this embodiment may refer to the examples described in the foregoing embodiments and the optional embodiments, and details are not described herein again.
显然，本领域的技术人员应该明白，上述的本发明的各模块或各步骤可以用通用的计算装置来实现，它们可以集中在单个的计算装置上，或者分布在多个计算装置所组成的网络上，可选地，它们可以用计算装置可执行的程序代码来实现，从而，可以将它们存储在存储装置中由计算装置来执行，并且在某些情况下，可以以不同于此处的顺序执行所示出或描述的步骤，或者将它们分别制作成各个集成电路模块，或者将它们中的多个模块或步骤制作成单个集成电路模块来实现。这样，本发明不限制于任何特定的硬件和软件结合。It will be apparent to those skilled in the art that the various modules or steps of the present invention described above can be implemented by a general-purpose computing device that can be centralized on a single computing device or distributed across a network of multiple computing devices. Alternatively, they may be implemented by program code executable by the computing device such that they may be stored in the storage device by the computing device and, in some cases, may be different from the order herein. The steps shown or described are performed, or they are separately fabricated into individual integrated circuit modules, or a plurality of modules or steps thereof are fabricated as a single integrated circuit module. Thus, the invention is not limited to any specific combination of hardware and software.
以上所述仅为本发明的优选实施例而已，并不用于限制本发明，对于本领域的技术人员来说，本发明可以有各种更改和变化。凡在本发明的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。The above description is only the preferred embodiment of the present invention, and is not intended to limit the present invention, and various modifications and changes can be made to the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and scope of the present invention are intended to be included within the scope of the present invention.

Claims (18)

  1. 一种视频处理方法，包括：A video processing method comprising:
    将虚拟现实VR视频的画面还原成二维图像；Restore the picture of the virtual reality VR video to a two-dimensional image;
    使用所述二维图像生成多个视角的二维视频。A two-dimensional video of a plurality of viewing angles is generated using the two-dimensional image.
  2. 根据权利要求1所述的方法，其中，将虚拟现实VR视频的画面还原成二维图像包括：The method of claim 1 wherein restoring the picture of the virtual reality VR video to the two-dimensional image comprises:
    解析所述VR视频中的每一帧画面，并通过球面模型渲染将每一帧画面从平铺模式还原成全景模式；Parsing each frame of the VR video and restoring each frame from the tile mode to the panorama mode through spherical model rendering;
    将所述全景模式中的每一帧画面进行视角分割，得到多个视角的二维图像。Each frame of the panoramic mode is divided into a view angle to obtain a two-dimensional image of a plurality of views.
  3. 根据权利要求2所述的方法，其中，将所述全景模式中的每一帧画面进行视角分割，得到多个视角的二维图像，包括以下之一：The method according to claim 2, wherein each frame of the panoramic mode is subjected to view segmentation to obtain a two-dimensional image of a plurality of views, including one of the following:
    将所述全景模式中的每一帧画面的球面分割成六面体，采用纹理映射的方式以圆心坐标为中心点，采用120度视角，在不同的方向进行投影，得到上下左右前后六个视角的图像；The spherical surface of each frame in the panoramic mode is divided into hexahedrons, and the center of the center coordinates is used as a center point by using texture mapping, and the image is projected in different directions by using a 120-degree angle of view, and images of six angles of up, down, left, and right are obtained. ;
    将所述全景模式中的每一帧画面的球面进行自定义分割，采用纹理映射的方式以圆心坐标为中心点，Y轴为正方向，X轴正方向为0度起始点，绕Z轴依次分割，生成M个图，并绕X轴旋转生成M个图，得到2M帧不同视角的图像。Customizing the spherical surface of each frame in the panoramic mode, using texture mapping as the center point of the center of the circle, the Y axis is the positive direction, the positive X direction is the 0 degree starting point, and the Z axis is sequentially Splitting, generating M maps, and rotating around the X-axis to generate M maps, and obtaining images of different viewing angles of 2M frames.
  4. 根据权利要求1所述的方法，其中，使用所述二维图像生成多个视角的二维视频包括：The method of claim 1, wherein generating the two-dimensional video of the plurality of views using the two-dimensional image comprises:
    按照所述二维图像的位置信息将所述二维图像汇聚成多路码流；Converging the two-dimensional images into multiple streams according to position information of the two-dimensional image;
    对所述多路码流进行打包切片生成多个视角的二维视频。The multiplexed code stream is packetized and sliced to generate a two-dimensional video of multiple viewing angles.
  5. 根据权利要求1所述的方法，其中，所述多个视角的二维视频通过不同的索引号进行描述。The method of claim 1, wherein the two-dimensional video of the plurality of views is described by a different index number.
  6. 根据权利要求1所述的方法，其中，在使用所述二维图像生成多个视角的二维视频之后，所述方法还包括：The method of claim 1, wherein after generating the two-dimensional video of the plurality of views using the two-dimensional image, the method further comprises:
    输出所述二维视频。The two-dimensional video is output.
  7. 根据权利要求6所述的方法，其中，输出所述二维视频包括：The method of claim 6 wherein outputting the two-dimensional video comprises:
    接收视角选择信息；Receiving perspective selection information;
    根据所述视角选择信息在所有二维视频中选择对应视角的指定二维视频；Selecting a specified two-dimensional video corresponding to the view angle among all the two-dimensional videos according to the view selection information;
    输出所述指定二维视频的码流。Outputting the code stream of the specified two-dimensional video.
  8. 根据权利要求1所述的方法，其中，输出所述二维视频包括：The method of claim 1 wherein outputting the two-dimensional video comprises:
    输出一个或多个视角的二维视频。Output two-dimensional video of one or more views.
  9. 根据权利要求8所述的方法，其中，在输出多个视角的二维视频时，所述多个视角的二维视频在视频播放终端分屏播放。The method according to claim 8, wherein the two-dimensional video of the plurality of viewing angles is played on a split screen of the video playing terminal when the two-dimensional video of the plurality of viewing angles is output.
  10. 一种视频处理装置，包括：A video processing device comprising:
    还原模块，设置为将虚拟现实VR视频的画面还原成二维图像；a restoration module configured to restore a virtual reality VR video to a two-dimensional image;
    生成模块，设置为使用所述二维图像生成多个视角的二维视频。A generating module is configured to generate a two-dimensional video of a plurality of viewing angles using the two-dimensional image.
  11. 根据权利要求10所述的装置，其中，所述还原模块包括：The apparatus of claim 10 wherein said restoration module comprises:
    解析单元，设置为解析所述VR视频中的每一帧画面，并通过球面模型渲染将每一帧画面从平铺模式还原成全景模式；a parsing unit configured to parse each frame of the VR video and restore each frame from a tile mode to a pan mode by spherical model rendering;
    分割单元，设置为将所述全景模式中的每一帧画面进行视角分割，得到多个视角的二维图像。The dividing unit is configured to divide the viewing angle of each frame of the panoramic mode to obtain a two-dimensional image of a plurality of viewing angles.
  12. 根据权利要求11所述的装置，其中，所述分割单元还设置为以下之一：The apparatus according to claim 11, wherein said dividing unit is further set to one of:
    将所述全景模式中的每一帧画面的球面分割成六面体，采用纹理映射的方式以圆心坐标为中心点，采用120度视角，在不同的方向进行投影，得到上下左右前后六个视角的图像；The spherical surface of each frame in the panoramic mode is divided into hexahedrons, and the center of the center coordinates is used as a center point by using texture mapping, and the image is projected in different directions by using a 120-degree angle of view, and images of six angles of up, down, left, and right are obtained. ;
    将所述全景模式中的每一帧画面的球面进行自定义分割，采用纹理映射的方式以圆心坐标为中心点，Y轴为正方向，X轴正方向为0度起始点，绕Z轴依次分割，生成M个图，并绕X轴旋转生成M个图，得到2M帧不同视角的图像。Customizing the spherical surface of each frame in the panoramic mode, using texture mapping as the center point of the center of the circle, the Y axis is the positive direction, the positive X direction is the 0 degree starting point, and the Z axis is sequentially Splitting, generating M maps, and rotating around the X-axis to generate M maps, and obtaining images of different viewing angles of 2M frames.
  13. 根据权利要求10所述的装置，其中，所述生成模块包括：The apparatus of claim 10, wherein the generating module comprises:
    汇聚单元，设置为按照所述二维图像的位置信息将所述二维图像汇聚成多路码流；a convergence unit, configured to aggregate the two-dimensional images into multiple streams according to position information of the two-dimensional image;
    处理单元，设置为对所述多路码流进行打包切片生成多个视角的二维视频。The processing unit is configured to perform packet slicing on the multiple code streams to generate a two-dimensional video of multiple viewing angles.
  14. 根据权利要求10所述的装置，其中，所述多个视角的二维视频通过不同的索引号进行描述。The apparatus of claim 10, wherein the two-dimensional video of the plurality of views is described by a different index number.
  15. 根据权利要求10所述的装置，其中，所述装置还包括：The device of claim 10, wherein the device further comprises:
    输出模块，设置为输出所述二维视频。An output module configured to output the two-dimensional video.
  16. 根据权利要求15所述的装置，其中，所述输出模块包括：The apparatus of claim 15 wherein said output module comprises:
    接收单元，设置为接收视角选择信息；a receiving unit configured to receive viewing angle selection information;
    选择单元，设置为根据所述视角选择信息在所有二维视频中选择对应视角的指定二维视频；a selecting unit, configured to select a specified two-dimensional video corresponding to the viewing angle among all the two-dimensional videos according to the viewing angle selection information;
    输出单元，设置为输出所述指定二维视频的码流。An output unit configured to output a code stream of the specified two-dimensional video.
  17. 一种存储介质，所述存储介质包括存储的程序，其中，所述程序运行时执行权利要求1至9中任一项所述的方法。A storage medium, the storage medium comprising a stored program, wherein the program is executed to perform the method of any one of claims 1 to 9.
  18. 一种处理器，所述处理器设置为运行程序，其中，所述程序运行时执行权利要求1至9中任一项所述的方法。A processor, the processor being arranged to run a program, wherein the program is executed to perform the method of any one of claims 1 to 9.
PCT/CN2018/107236 2017-12-25 2018-09-25 Video processing method and apparatus WO2019128342A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201711423373.XA CN108124193A (en) 2017-12-25 2017-12-25 Method for processing video frequency and device
CN201711423373.X 2017-12-25

Publications (1)

Publication Number Publication Date
WO2019128342A1 true WO2019128342A1 (en) 2019-07-04

Family

ID=62231481

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/107236 WO2019128342A1 (en) 2017-12-25 2018-09-25 Video processing method and apparatus

Country Status (2)

Country Link
CN (1) CN108124193A (en)
WO (1) WO2019128342A1 (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108124193A (en) * 2017-12-25 2018-06-05 中兴通讯股份有限公司 Method for processing video frequency and device
CN109729338B (en) * 2018-11-28 2021-10-01 北京虚拟动点科技有限公司 Display data processing method, device and system
CN112188303A (en) * 2020-09-03 2021-01-05 北京火眼目测科技有限公司 VR (virtual reality) streaming media playing method and device based on visual angle
CN113489935A (en) * 2020-09-11 2021-10-08 青岛海信电子产业控股股份有限公司 Video interaction method and device

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030063816A1 (en) * 1998-05-27 2003-04-03 Industrial Technology Research Institute, A Taiwanese Corporation Image-based method and system for building spherical panoramas
CN106162207A (en) * 2016-08-25 2016-11-23 北京字节跳动科技有限公司 A kind of panoramic video parallel encoding method and apparatus
CN107169924A (en) * 2017-06-14 2017-09-15 歌尔科技有限公司 The method for building up and system of three-dimensional panoramic image
CN107197135A (en) * 2016-03-21 2017-09-22 成都理想境界科技有限公司 A kind of video generation method, player method and video-generating device, playing device
CN107205140A (en) * 2017-07-12 2017-09-26 赵政宇 A kind of panoramic video segmentation projecting method and apply its system
CN108124193A (en) * 2017-12-25 2018-06-05 中兴通讯股份有限公司 Method for processing video frequency and device

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030063816A1 (en) * 1998-05-27 2003-04-03 Industrial Technology Research Institute, A Taiwanese Corporation Image-based method and system for building spherical panoramas
CN107197135A (en) * 2016-03-21 2017-09-22 成都理想境界科技有限公司 A kind of video generation method, player method and video-generating device, playing device
CN106162207A (en) * 2016-08-25 2016-11-23 北京字节跳动科技有限公司 A kind of panoramic video parallel encoding method and apparatus
CN107169924A (en) * 2017-06-14 2017-09-15 歌尔科技有限公司 The method for building up and system of three-dimensional panoramic image
CN107205140A (en) * 2017-07-12 2017-09-26 赵政宇 A kind of panoramic video segmentation projecting method and apply its system
CN108124193A (en) * 2017-12-25 2018-06-05 中兴通讯股份有限公司 Method for processing video frequency and device

Also Published As

Publication number Publication date
CN108124193A (en) 2018-06-05

Similar Documents

Publication Publication Date Title
US11870967B2 (en) Methods and apparatus for supporting content generation, transmission and/or playback
US11303881B2 (en) Method and client for playing back panoramic video
JP7223106B2 (en) Method, device and computer program for adaptive streaming of virtual reality media content
WO2019128342A1 (en) Video processing method and apparatus
JP6979035B2 (en) How to Improve Streaming of Virtual Reality Media Content, Devices and Computer Programs
CN109076255B (en) Method and equipment for sending and receiving 360-degree video
US10897646B2 (en) Video stream transmission method and related device and system
KR20220081390A (en) Methods and apparatus for delivering content and/or playing back content
JP2020503792A (en) Information processing method and apparatus
RU2733218C2 (en) Method, apparatus and a stream for formatting an immersive video image for traditional and immersive playback devices
TWI786572B (en) Immersive media providing method and acquiring method, device, equipment and storage medium
Romero-Rondón et al. Foveated streaming of virtual reality videos
JP2023529748A (en) Support for multi-view video motion with disocclusion atlas
Podborski et al. Virtual reality and DASH
CN107438203B (en) Method for establishing and receiving list, network equipment and terminal
KR102243666B1 (en) How to transmit 360 degree video, how to receive 360 degree video, 360 degree video transmission device, 360 degree video receiving device
JP7054351B2 (en) System to play replay video of free viewpoint video
CN114513702B (en) Web-based block panoramic video processing method, system and storage medium
US10593122B1 (en) Mirroring edge pixels
KR20220054283A (en) Methods for transmitting and rendering a 3D scene, a method for generating patches, and corresponding devices and computer programs
US20220329886A1 (en) Methods and devices for handling media data streams
JP2022036123A (en) System and method for reproducing replay video of free-viewpoint video
JP2023544049A (en) Bidirectional presentation data stream
CN116235429A (en) Bi-directional presentation of data streams using control plane channels and data plane channels

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18896429

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18896429

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 17.11.2020)

122 Ep: pct application non-entry in european phase

Ref document number: 18896429

Country of ref document: EP

Kind code of ref document: A1