WO2016192413A1 - 码流对齐、同步处理方法及发送、接收终端和通信*** - Google Patents

码流对齐、同步处理方法及发送、接收终端和通信*** Download PDF

Info

Publication number
WO2016192413A1
WO2016192413A1 PCT/CN2016/073823 CN2016073823W WO2016192413A1 WO 2016192413 A1 WO2016192413 A1 WO 2016192413A1 CN 2016073823 W CN2016073823 W CN 2016073823W WO 2016192413 A1 WO2016192413 A1 WO 2016192413A1
Authority
WO
WIPO (PCT)
Prior art keywords
alignment
code stream
information
video
layer
Prior art date
Application number
PCT/CN2016/073823
Other languages
English (en)
French (fr)
Inventor
吴钊
李明
尚国强
Original Assignee
中兴通讯股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中兴通讯股份有限公司 filed Critical 中兴通讯股份有限公司
Priority to EP16802336.4A priority Critical patent/EP3306945A4/en
Priority to US15/579,310 priority patent/US10582234B2/en
Publication of WO2016192413A1 publication Critical patent/WO2016192413A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
    • H04N21/23611—Insertion of stuffing data into a multiplex stream, e.g. to obtain a constant bitrate
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234327—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by decomposing into layers, e.g. base layer and one or more enhancement layers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
    • H04N21/23602—Multiplexing isochronously with the video sync, e.g. according to bit-parallel or bit-serial interface formats, as SDI
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/187—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a scalable video layer
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/65—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using error resilience
    • H04N19/68—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using error resilience involving the insertion of resynchronisation markers into the bitstream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/70—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
    • H04N21/23614—Multiplexing of additional data and video streams
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
    • H04N21/2365—Multiplexing of several video streams
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/434—Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/434—Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream
    • H04N21/4342—Demultiplexing isochronously with video sync, e.g. according to bit-parallel or bit-serial interface formats, as SDI
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/434—Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream
    • H04N21/4346—Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream involving stuffing data, e.g. packets or bytes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/434—Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream
    • H04N21/4348—Demultiplexing of additional data and video streams
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
    • H04N21/845—Structuring of content, e.g. decomposing content into time segments

Definitions

  • the present invention relates to the field of communications, and in particular, to a code stream alignment, synchronization processing method, and a transmitting and receiving terminal and a communication system.
  • H.265/HEVC High Efficiency Video Coding
  • ITU-T/VCEG Video Coding Experts Group
  • ISO/IEC MPEG continues the style of the previous generation video standard H.264/AVC.
  • H.264/AVC One aspect different from H.264/AVC is the need for multiple Coding Standards with Scalability.
  • the currently published H.265/HEVC standard (version 2014-10) is called the "H.265/HEVC Version 2" standard, and in Version 2, a hybrid layer (Hybrid Scalability) is provided to indicate the base layer (Base Layer).
  • the information about how to place the base layer image into the reference frame management of the Enhancement Layer is implemented by the external method (External Means), and the external method is not described in Version 2.
  • H.264/AVC will be the main video coding protocol for digital video worldwide. Therefore, the development of a new generation of video scalable coding standards compatible with H.264/AVC will be very useful in the standard migration process.
  • the scalable coding of multiple coding standards is that two different video standard code streams are combined according to certain rules, also called hybrid protocol hybrid coding (Hybrid Coding).
  • Hybrid Coding For example, Spatial Scalability, where the base layer is the H.264/AVC standard Constrained Baseline Profile, the Main Profile and High Profile compatible streams, and the Enhancement Layer contains the HEVC elementary stream.
  • the size of the two layers of images may be the same or different, that is, H.264/AVC uses low resolution, and H.265/HEVC uses high resolution.
  • the base layer of spatial scalable coding can also use the MPEG-2 code stream, combined with the enhancement layer code stream of H.265/HEVC for hybrid coding.
  • the enhancement layer is another viewpoint, using the H.265/HEVC format with the base layer as a reference.
  • the basic layer of hybrid coding combined with multiple coding standards is not limited to H.264/AVC or MPEG-2.
  • a scalable code stream composed of the same protocol stream can be synchronized in various ways. For example, a video image obtained at the same time in H.264/AVC and its corresponding coded bits form an access unit (AU). Therefore, different layer images can be synchronized by the AU range.
  • syntax information in a video stream such as POC (Picture Order Count), etc., can be used.
  • the codecs are independent codecs, and the syntax information contained in different video streams is slightly different, so in what form Stream package, or provide Which grammar information can ensure synchronization is a problem that Hybrid Coding needs to solve.
  • an existing method is to encapsulate the code streams of H.264/AVC and H.265/HEVC, and as a whole, use an internal means to transmit. That is to say, some syntax elements are added in H.265/HEVC, which can indicate the form in which the H.264/AVC code stream exists.
  • the JCT-VC standard conference has many related proposals.
  • the disadvantage of this method is that a standard H.264/AVC decoder cannot decode such a code stream, and the NAL unit header information corresponding to H.265/HEVC needs to be unpacked before it can be correctly decoded.
  • JCTVC-O0190 is a different packaging strategy for JCTVC-N0050. It also encapsulates the code streams of H.264/AVC and H.265/HEVC as a whole, but the base layer H.264/AVC NAL unit does not change, and the enhancement layer H.265/HEVC NAL unit adds a prefix word.
  • the section (0x16) is used to allow the H.264/AVC decoder to recognize that this part of the stream cannot be decoded and discarded, and its stream portion still uses the in-band method. It is also recommended to add a variable HevcPrefixBytePresent to indicate to the decoder that the prefix byte needs to be parsed.
  • This method requires the addition of information that identifies different streams in or before the NAL header, and the decoder must be able to identify its own stream and its own stream. Therefore, whether it is an H.264/AVC decoder or an H.265/HEVC decoder, if you want to support the hybrid encoding (Hybrid Coding) function, you need to increase the corresponding processing. For the decoder in the existing device, it is impossible to implement the function without changing or changing it.
  • the existing methods are based on the in-band method for transmitting the code streams of different video standards, and the alignment or synchronization process needs to modify the existing codec devices, the implementation process is complicated, the versatility is poor, and the implementation cost is relatively low. high.
  • the main technical problem to be solved by the present invention is to provide a code stream alignment, synchronization processing method, and a transmitting and receiving terminal and a communication system, which solves the complicated implementation of the existing alignment and synchronization processing for a plurality of different video code streams, and has poor versatility.
  • the problem of high cost is to provide a code stream alignment, synchronization processing method, and a transmitting and receiving terminal and a communication system, which solves the complicated implementation of the existing alignment and synchronization processing for a plurality of different video code streams, and has poor versatility.
  • the present invention provides a code stream alignment method, including:
  • the transmitting terminal sets the alignment auxiliary information in the multiple code stream when the multiple code streams need to be aligned;
  • the transmitting terminal stores or sends the multiple code stream to the receiving terminal.
  • the code stream is a video code stream;
  • the sending terminal setting the alignment assistance information in the multiple code stream includes: a video layer and a system layer in the multiple code stream
  • the alignment assistance information is set with at least one of the application layers.
  • the transmitting terminal sets the alignment in a video layer of the multiple code stream
  • the auxiliary information includes: setting the alignment assistance information in at least one of the following fields in the video layer:
  • the method when the sending terminal sets the alignment assistance information in a system layer of the multiple code stream, the method includes: in a system layer, a field setting corresponding to a video media attribute. Align auxiliary information.
  • the method when the sending terminal sets the alignment assistance information in an application layer of the multiplexed code stream, the method includes: in a media file, a field corresponding to a video media attribute, and The alignment auxiliary information is set in a field corresponding to the video media attribute in the application layer.
  • the sending terminal sets the location and/or content of the alignment assistance information in the multiple code stream to be the same.
  • the sending terminal sets the number and/or content of the alignment assistance information in the multiple code stream to be different.
  • the sending, by the sending terminal, the multiplexed code stream to the receiving terminal includes: sending, by the sending terminal, the multiplexed code stream separately to the receiving terminal; or Transmitting, by the transmitting terminal, at least two channels of the multiple code streams into one path and then sending the signal to the receiving terminal; or the sending terminal reassembling the multiple code streams into multiple streams and sending the information to the station Said receiving terminal.
  • the method when the sending terminal stores the multiple code streams, the method includes: writing the multiple code streams into a file for storage, or writing the multiple code streams separately.
  • the file is stored in a separate file, or the multi-stream is split and recombined and written into different files for storage.
  • the alignment assistance information includes aligning scalar content or, when the scalar content is included, alignment indication information.
  • the aligned scalar content includes at least one of image absolute time information, image relative time information, image decoding order information, image playback order information, image separator, and image specific delimiter.
  • the alignment indication information when the alignment assistance information includes alignment indication information, includes alignment identification information and/or type information.
  • the multiple streams are video streams of two different video standards, and one of them is an H.265/HEVC video stream.
  • the present invention further provides a code stream receiving method, including:
  • the alignment assistance information is used to synchronize the multiple code streams.
  • the code stream is a video code stream
  • the receiving terminal searches for the alignment auxiliary information from at least one of an application layer, a system layer, and a video layer of the multiple code stream.
  • the alignment auxiliary information of one layer is selected according to a preset rule for synchronization processing.
  • the preset rule includes:
  • the video layer has the alignment auxiliary information
  • select the alignment auxiliary information in the video layer otherwise, select the alignment auxiliary information in the layer closest to the video layer;
  • the complete alignment auxiliary information exists in a layer with relatively low channel transmission reliability of one code stream, and the selected channel transmission is reliable when the alignment auxiliary information of the layer with high channel transmission reliability is missing.
  • the alignment auxiliary information of the relatively low-level layer is synchronized;
  • the complete alignment auxiliary information exists in other layers outside the video layer of the one-stream stream, and when the alignment auxiliary information of the video layer is missing, the alignment auxiliary information of other layers outside the video layer is selected for synchronization processing.
  • the alignment assistance information includes at least aligning scalar content; and the receiving terminal performs synchronization processing on the multiplexed code stream according to the aligned scalar content.
  • the present invention also provides a code stream processing method, including:
  • the transmitting terminal sets the alignment auxiliary information in the multiple code stream when the multiple code streams need to be aligned;
  • the transmitting terminal stores or sends the multiple code stream to the receiving terminal;
  • the receiving terminal when the receiving terminal carries the alignment assistance information in the multiple code stream, the receiving terminal performs synchronization processing on the multiple code streams according to the alignment assistance information.
  • the present invention also provides a processing module and a sending module
  • the processing module is configured to set alignment assistance information in the multiple code stream when the multiple code streams need to be aligned;
  • the sending module is configured to store or send the multiple code stream to the receiving terminal.
  • the code stream is a video code stream
  • the processing module includes a write submodule for at least a video layer, a system layer, and an application layer of the multiple code stream.
  • the alignment assistance information is set on one layer.
  • the present invention also provides an acquisition module and an execution module
  • the obtaining module is configured to acquire a multiple code stream
  • the execution module is configured to perform synchronization processing on the multiple code streams according to the alignment auxiliary information when the alignment auxiliary information set by the sending terminal is carried in the multiple code stream.
  • the execution module includes a lookup submodule for using the multiple stream At least one of the application layer, the system layer, and the video layer searches for the alignment assistance information.
  • the present invention also provides a communication system including the transmitting terminal as described above and the receiving terminal as described above;
  • the transmitting terminal sets aligning auxiliary information in the multiplexed code stream, and stores or sends the multiplexed code stream to the receiving terminal;
  • the receiving terminal acquires the multiplexed code stream, and when the multiplexed code stream carries the aligning auxiliary information, performs synchronization processing on the multiplexed code stream according to the aligning auxiliary information.
  • the transmitting terminal sets the alignment auxiliary information in the multiplexed code stream, and then stores or transmits to the receiving.
  • the receiving terminal may perform synchronization processing on each standard code stream according to the alignment auxiliary information carried in each code stream. It can be seen that the present invention directly sets the alignment auxiliary information in each code stream, which is simple to implement, and basically does not require additional modification to the codec, has good versatility, and has lower implementation cost.
  • FIG. 1 is a schematic flowchart of a method for completing a code stream alignment by a transmitting end according to Embodiment 1 of the present invention
  • FIG. 2 is a schematic flowchart of a method for completing a code stream synchronization at a receiving end according to Embodiment 2 of the present invention
  • FIG. 3 is a schematic diagram of a communication system according to Embodiment 3 of the present invention.
  • FIG. 4 is a schematic structural diagram of a transmitting terminal according to Embodiment 3 of the present invention.
  • FIG. 5 is a schematic structural diagram of a receiving terminal according to Embodiment 3 of the present invention.
  • the transmitting end will align the auxiliary information when determining that the code stream needs to be aligned (the alignment auxiliary information is relative to the transmitting end, and is used by the receiving end. In the synchronization, therefore, it can also be referred to as synchronization auxiliary information for the receiving end.) It is set to the code streams of different standards of each channel, and after the receiving end acquires, the synchronization processing can be completed according to the alignment auxiliary information in each code stream.
  • the invention directly sets the alignment auxiliary information in each code stream, which is simple to implement, and basically does not need to additionally modify the codec, has good versatility, and has lower implementation cost.
  • Embodiment 1 is a diagrammatic representation of Embodiment 1:
  • the transmitting end determines that the current multiple stream needs to be aligned
  • the positioning auxiliary information is separately set in the multiple stream, and then the multiple stream is stored or transmitted to the receiving terminal.
  • Step 101 The transmitting terminal determines whether the current multiple code stream needs to be aligned, if yes, go to step 102, otherwise, go to step 103;
  • Step 102 The sending terminal sets the alignment assistance information in the multiplexed code stream.
  • Step 103 The sending terminal stores or sends the multiplexed code stream to the receiving terminal.
  • the storage here may be stored locally at the sending terminal, or may be stored on the corresponding server, or stored locally and on the server. And can be stored in the form of a file.
  • the foregoing step 101 is performed by the sending terminal when the current multiplexed code stream needs to be sent, and the sending terminal may specifically determine whether the current multiplexed code stream needs to be aligned according to the application requirement.
  • the application requirements may be determined according to factors such as the specific needs of the user or a specific application environment, such as whether the encoder uses the base layer AVC image as a reference for the enhancement layer HEVC image, or the AVC image and the HEVC image need to be output synchronously.
  • the code stream in this embodiment may be a video code stream; or may be other code streams on the system layer; this embodiment does not limit the specific code stream type; it should be understood that this
  • the multiplexed code stream in the embodiment may be a multiplexed code stream of the same standard, or may be a multiplexed code stream of different standards, which may be determined according to a specific application scenario.
  • the alignment or synchronization operation of the multiplexed stream in the above case can be realized by the scheme shown in FIG. 1 above.
  • the present invention is exemplarily described with the video stream as an example.
  • the sending terminal when the sending terminal sets the alignment auxiliary information in the multiplexed code stream, it needs to determine in which position of each video code stream the aligning auxiliary information needs to be set.
  • the sending terminal may specifically Alignment auxiliary information is set in at least one of a video layer, a system layer, and an application layer of the multiplexed stream.
  • the sending terminal sets the alignment auxiliary information in each code stream, the existing information in the code stream may be directly used as the alignment auxiliary information, or the new content may be added as the alignment auxiliary information in the code stream.
  • the sending terminal when the sending terminal sets the alignment auxiliary information in the video layer of the multiplexed stream, the sending terminal includes setting the alignment auxiliary information in at least one of the following fields of the video layer:
  • a field corresponding to the enhanced information the image in the code stream in the valid range of the field is aligned or synchronized with the code stream image corresponding to the corresponding identifier and content information; for example, for H.265/HEVC video
  • the field corresponding to the code stream and the enhanced information is specifically a Supplemental Enhancement Information (SEI) field, which is used to indicate an image in the code stream within the valid range of the SEI information, and correspondingly possess corresponding identifier and content information.
  • SEI Supplemental Enhancement Information
  • the code stream image is aligned or synchronized; correspondingly, the field corresponding to the H.264/AVC video stream enhancement information is specifically an SEI field; it should be understood that for other formats of the code stream, a similar function field is taken as the enhanced information.
  • the corresponding field can be used, for example, the user-defined corresponding field (User Data) is used in MPEG-2 and MPEG-4 to perform the same function;
  • the field corresponding to the video available information is specifically a Video Usability Information (VUI) field.
  • VUI Video Usability Information
  • the field corresponding to the video available information is specifically a VUI field; correspondingly, for For code streams in other formats, take a similar function field as a video.
  • the field corresponding to the information can be used, for example, the user-defined corresponding field (User Data) is used in MPEG-2 and MPEG-4 to perform the same function;
  • the corresponding field of the video common information includes a video parameter set (VPS), a sequence parameter set (SPS), and a picture parameter set (PPS), corresponding to H.264/AVC video stream, the parameter set includes a sequence parameter set (SPS) and a picture parameter set (PPS).
  • the video information common field is a VOS (Visual Object Sequences) header, VO.
  • VOM Video Object Layer
  • GOV Group of Video Object Plane
  • VOP Video Object Plane
  • the video information has a Video Sequence header of the field, The group of pictures header and the Picture header; correspondingly, for the code streams of other formats, a similar function field is taken as the field corresponding to the video common information;
  • the code stream is a Slice Header field, which is a Slice Header field for the H.264/AVC video stream, and a Video Packet Header field for the MPGE-4 video stream; correspondingly, for other formats, the code stream has a similar function.
  • the field may be a slice header information field; it is noted that, in this embodiment, the same video image may be set only in the first slice header, or may be set simultaneously in the other at least one slice header.
  • the alignment assistance information set in the plurality of slice headers is consistent;
  • the stream stream video layer includes, in addition to the above fields, a field containing at least one image (frame image or field image), indicating whether the image layer to which the corresponding data structure of the code stream is subjected to performing alignment or synchronization operations, for example, user-specific Define a data field (User Data) or an extension field (Extension Data);
  • the specific field setting in the system layer is included:
  • (6) a field corresponding to the video media attribute in the system layer, used to indicate an image included in the system code stream, and aligned or synchronized with the code stream image corresponding to the corresponding identifier and the content information;
  • the sending terminal When the sending terminal sets the alignment auxiliary information in the application layer of the multiplexed stream, the sending terminal includes setting in at least one of the following fields:
  • a field corresponding to the video media attribute in the media file used to indicate an image in the code stream included in the media file, aligned or synchronized with the code stream image corresponding to the corresponding identifier and the content information;
  • the field corresponding to the video media attribute in the application layer is used to indicate the code stream image included in the application, and is aligned or synchronized with the code stream image corresponding to the corresponding identifier and the content information.
  • the decoder can discard according to standard rules.
  • the sending terminal may be set in any one of the above fields of each video code stream, or may be set in multiple fields in the above field of each video code stream, for one video stream,
  • the location in which the transmitting terminal in this embodiment sets the alignment assistance information in each video code stream may be the same or different; for example, when the multiple code streams are two different standard video streams, Two video streams can be set in the same position, for example, all of the fields corresponding to the enhanced information of the video layer; or can be set at different positions, for example, the field corresponding to the enhanced information of the video layer Set, the other way is set in the field corresponding to the video sharing information of the video layer, and for example, one of the fields is used in the system layer to describe the field setting of the video media attribute, and the other way is used to describe the field of the video media attribute in the application layer. Settings.
  • the content of the alignment assistance information set by the transmitting terminal in the multiplexed code stream may be the same or different, as long as the content can be characterized and needs to be aligned and can be correctly identified.
  • the number of the positioning auxiliary information set by the transmitting terminal in the multiplexed code stream may be the same or different.
  • the video stream can set one alignment auxiliary information in one code stream and multiple alignment auxiliary information in another code stream.
  • the content of multiple alignment auxiliary information set on the same layer for one code stream is the same.
  • the alignment assistance information set by the transmitting terminal at least includes aligning the scalar content, or may further include the alignment indication information when the scalar content is included.
  • the alignment scalar content in this embodiment includes at least image absolute time information (ie, system time when hybrid encoding is performed), image relative time information, image decoding order information, image playback order information, image separator, and image specific delimiter.
  • image absolute time information ie, system time when hybrid encoding is performed
  • image relative time information image decoding order information
  • image decoding order information image playback order information
  • image separator image specific delimiter.
  • the alignment indication information may specifically be alignment identification information and/or type information.
  • An exemplary illustration is given below with a specific example:
  • the alignment identification information and the type information of the alignment indication information in the alignment assistance information for the alignment or synchronization operation used in the following example are respectively as shown in Table 1 and/or Table 2. Align the scalar content with the structure shown in Table 3.
  • the code stream shown in Table 1 contains identification information identifying whether the whole or part of the code stream uses alignment or synchronization operations.
  • the code stream shown in Table 2 contains the type of identification information that identifies the overall or partial stream alignment or synchronization operation.
  • the code stream shown in Table 3 contains the content information required for alignment or synchronization.
  • pic_alignment_flag takes a value equal to 1 to indicate that video images of different encoding formats in the code stream need to be aligned or synchronized.
  • a pic_alignment_flag value equal to 0 is a video image representing a different encoding format in the code stream that does not require alignment or synchronization operations.
  • the values of the pic_alignment_flag and the meanings thereof can be used in other manners, which are not limited in this embodiment.
  • Pic_alignment_flag uses the corresponding codec method in different encoding protocols.
  • the value of the corresponding variable pic_alignment_flag is 0 by default.
  • pic_alignment_type() indicates the acquisition type information.
  • the type referred to here refers to the information used to store the alignment or synchronization information fields placed under the same type of data segment. Further extract the aligned or synchronized content information.
  • the internal form of the pic_alignment_type() and the meaning of the representation thereof may also be used in other manners, which is not limited in this embodiment.
  • pic_alignment_type() does not exist in the code stream, the corresponding type information does not exist.
  • alignment_code() contains the data information required for alignment or synchronization.
  • the fields in Table 3 can be used in combination with the fields in Table 1 and/or Table 2.
  • the alignment_code() in Table 3 may use one or more of the following information, including but not limited to the following information: image absolute time information, image relative time information, image decoding order information, image playback order information, image separator , image special delimiters, etc.
  • the syntax elements in Table 1, Table 2, and Table 3 may be set to at least one field or a plurality of fields mixed in the above (1) to (8); the syntax elements in Table 1, Table 2, and Table 3 correspond to The bit field is used in the code stream corresponding to the data structure.
  • the transmitting terminal when it sends the multiplexed code stream to the receiving terminal, it may flexibly select to separately send the multiplexed code streams to the receiving terminal according to the specific application scenario; or select at least one of the multiplexed code streams.
  • Two The road is encapsulated into one way and sent to the receiving terminal.
  • the code stream is three-way (A, B, and C respectively)
  • the three-way package may be selected to be sent in one way; or two of the three ways (for example, A and B channels) may be selected to be packaged.
  • the one-way code stream and the C-way code stream obtained after the encapsulation are separately sent separately sent separately; the processing for the case of three or more channels is deduced by analogy.
  • the multiplexed stream splitting may also be reassembled into a multiplexed stream and sent to the receiving terminal.
  • the multiplexed code stream when it is stored locally or stored on the server, the multiplexed code stream can be written into a file for storage, or the multiplexed code stream can be separately written. Store in a separate file, or split and reassemble multiple streams to be written to different files for storage.
  • the multiplexed code stream in this embodiment is specifically configured as a video code stream of two different video standards, and one of the paths (for example, the enhancement layer) is an H.265/HEVC video code stream, and the other path (for example, the base layer) ) is a non-H.265/HEVC video stream, such as H.264/AVC video stream, MPGE-2 video stream, MPGE-4 video stream, VC-1 video stream, VP8 video stream, AVS video stream, etc. .
  • the multiplexed code stream in this embodiment may also be specifically configured as two video streams of the same video standard, and one of the paths (for example, the base layer) is an H.265/HEVC video code stream, and the other path (for example, enhancement) Layer) is also the H.265/HEVC video stream.
  • one of the paths for example, the base layer
  • the other path for example, enhancement
  • Embodiment 2 is a diagrammatic representation of Embodiment 1:
  • the operation of the receiving terminal to perform code stream synchronization is shown in FIG. 2, and includes:
  • Step 201 The receiving terminal acquires a multi-path code stream; the acquiring manner may receive the multiple code stream from the sending terminal or the server, or directly read from the corresponding storage medium;
  • the transmitting terminal when the transmitting terminal separately sends the multiple code streams to the receiving terminal, the receiving terminal separately receives each code stream; when the transmitting terminal encapsulates the multiple code streams into one way, it sends the received to the receiving.
  • the receiving terminal accepts the encoded code stream;
  • Step 202 When the receiving terminal carries the alignment auxiliary information set by the transmitting terminal in the multiplexed code stream, the receiving terminal performs synchronization processing on the multiplexed code stream according to the alignment auxiliary information.
  • the code stream in this embodiment may be a video code stream; or may be other code streams on the system layer; this embodiment does not limit the specific code stream type; it should be understood that this
  • the multiplexed code stream in the embodiment may be a multiplexed code stream of the same standard, or may be a multiplexed code stream of different standards, which may be determined according to a specific application scenario.
  • the present invention is exemplarily described with the video stream as an example.
  • the receiving terminal after receiving the code stream, the receiving terminal searches for the alignment auxiliary information from at least one of the application layer, the system layer, and the video layer of the multiplexed code stream. For example, when there is an agreement between the transmitting and receiving terminals to which layer or layers of auxiliary information is specifically added when alignment or synchronization is required, it is only necessary to search at the layer; otherwise, it is necessary to The application layer, system layer, and video layer of the code stream are searched in turn.
  • the alignment assistance information of one layer is selected according to a preset rule for synchronization processing.
  • the preset rules in the embodiment include but are not limited to:
  • Selecting alignment assistance information in a layer with high channel transmission reliability for example, when both the application layer and the video layer are present
  • the auxiliary information is inconsistent, but the channel transmission reliability of the application layer is higher than the channel transmission reliability of the video layer, and the alignment assistance information of the application layer is selected;
  • the alignment auxiliary information in the video layer is selected, otherwise, the alignment auxiliary information in the layer closest to the video layer is selected; for example, when the video layer and the application layer have alignment auxiliary information and In case of inconsistency, the alignment auxiliary information of the video layer is selected; when there is no alignment auxiliary information in the video layer, but the system layer and the application layer have inconsistent alignment auxiliary information, the alignment auxiliary information of the video layer closer to the video layer is selected;
  • the alignment assistance information in the active layer supported by the receiving terminal is selected; in this embodiment, the active layer supported by the receiving terminal refers to a layer that the receiving terminal needs to decode, and the receiving terminal does not need to decode and directly discard the layer to be relatively received.
  • the terminal is an invalid layer. For example, when the processing mode of the SEI in the video layer is directly discarded, when the alignment auxiliary information exists in the SEI field in the application layer and the video layer, the receiving terminal directly selects the alignment auxiliary information in the application layer. If there is no active layer supported by the receiving terminal in the layer with the auxiliary information, the receiving terminal may not perform the synchronization processing.
  • the user may customize other rules according to the application scenario or other factors, and may select effective alignment assistance information to complete alignment or synchronization operations. Just fine.
  • the selected channel transmission reliability is relatively
  • the alignment assistance information of the lower layer is synchronized; for example, when the alignment auxiliary information of the application layer is missing and the alignment auxiliary information of the video layer is complete, although the channel transmission reliability of the application layer is higher than that of the video layer. However, the alignment assistance information of the video layer is still selected at this time.
  • the alignment auxiliary information of other layers outside the video layer is selected for synchronization processing. For example, when the alignment auxiliary information of the video layer is missing and the alignment auxiliary information of the application layer is complete, the alignment auxiliary information of the application layer is selected.
  • the alignment assistance information set by the transmitting terminal at least includes aligning the scalar content, or may further include the alignment indication information when the scalar content is included.
  • the alignment scalar content in this embodiment includes at least image absolute time information (ie, system time when hybrid encoding is performed), image relative time information, image decoding order information, image playback order information, image separator, and image specific delimiter.
  • image absolute time information ie, system time when hybrid encoding is performed
  • image relative time information image decoding order information
  • image playback order information image playback order information
  • image separator image specific delimiter.
  • image specific delimiter One.
  • the alignment indication information may specifically be alignment identification information and/or type information.
  • the receiving terminal performs synchronization processing on the multiplexed code stream according to the aligned scalar content.
  • the multiplexed code stream in this embodiment is specifically configured as a code stream of two different video standards, and one of the paths is a H.265/HEVC video code stream, and the other path is a non-H.265/HEVC video stream.
  • H.265/HEVC video code stream For example, H.263/AVC video stream, MPGE-2 video stream, MPGE-4 video stream, VC-1 video stream, VP8 video stream, AVS video stream, and the like.
  • the multiple code streams in this embodiment may also be specifically configured as two video codes of the same video standard.
  • the stream, and one of the paths (eg, the base layer) is the H.265/HEVC video code stream, and the other path (eg, the enhancement layer) is also the H.265/HEVC video code stream.
  • Embodiment 3 is a diagrammatic representation of Embodiment 3
  • the present embodiment provides a communication system, including a transmitting terminal 1 and a receiving terminal 2.
  • the structure of the transmitting terminal 1 and the receiving terminal 2 is shown in FIG. 4 and FIG. 5, respectively.
  • the transmitting terminal 1 that is, the source device, includes a processing module 11 and a sending module 12;
  • the processing module 11 is configured to set the alignment auxiliary information in the multiple code stream when the current multiple code stream needs to be aligned; when the alignment auxiliary information is set in each code stream, the existing information in the code stream can be directly used as the alignment. Auxiliary information, you can also choose to add new content in the code stream as alignment assistance information.
  • the sending module 12 is configured to store or send the multiplexed code stream to the receiving terminal; when storing, it may be stored locally in the sending terminal, may also be stored on the server, etc.; and the multiplexed code stream may be written into one file.
  • it may flexibly select to separately send the multiple streams to the receiving terminal according to the specific application scenario; or select to encapsulate the multiple streams into one channel and then send the signals to the receiving terminal; or send the multiple streams
  • the split is reassembled into a multiplexed stream and sent to the receiving terminal.
  • the processing module 11 specifically includes a determining sub-module 111 and a writing sub-module 112.
  • the determining sub-module 111 is configured to determine whether it is necessary to perform alignment processing on the current multi-stream code stream according to an application requirement.
  • the application requirements may be determined according to specific needs of the user or a specific application environment, such as whether the encoder uses the base layer AVC image as a reference for the enhancement layer HEVC image, or the AVC image and the HEVC image need to be output synchronously; 112 is for writing alignment assistance information at a corresponding position of the code stream.
  • the code stream in this embodiment may be a video code stream; or may be other code streams on the system layer; this embodiment does not limit the specific code stream type; it should be understood that this
  • the multiplexed code stream in the embodiment may be a multiplexed code stream of the same standard, or may be a multiplexed code stream of different standards, which may be determined according to a specific application scenario.
  • the present invention is exemplarily described with the video stream as an example.
  • the writing sub-module 112 is configured to respectively set the alignment assistance information in at least one of a video layer, a system layer, and an application layer of the multi-stream. Specifically, when the sending terminal sets the alignment auxiliary information in the video layer of the multiplexed stream, the sending terminal includes setting the alignment auxiliary information in at least one of the following fields in the video layer:
  • a stream stream video layer other than the above fields which contains at least one image (frame image or field image);
  • the sending terminal When the sending terminal sets the alignment auxiliary information in the application layer of the multiplexed stream, the sending terminal includes setting in at least one of the following fields:
  • the writing sub-module 112 may be set in any one of the above fields of each video code stream, or may be set in multiple fields in the above-mentioned fields of each video code stream, for one video stream, theory
  • the alignment auxiliary information may be set in any one of the above layers; however, in order to further improve reliability, avoid transmission errors and the like, the loss or absence of the alignment auxiliary information may be set in multiple fields at the same time, for example, at the same time.
  • the writing sub-module 112 in this embodiment may have the same or different positions for setting the alignment auxiliary information in each video code stream; for example, when the multiple code streams are two different standard video streams.
  • alignment auxiliary information can be set at the same position, for example, all of the fields corresponding to the enhanced information of the video layer; or can be set at different positions, for example, one of the enhanced information in the video layer
  • the corresponding field is set, and the other path is set in the field corresponding to the video common information of the video layer, and for example, one of the fields used to describe the video media attribute in the system layer, and the other way is used to describe the video medium in the application layer.
  • the field settings for the property are used to describe the video medium in the application layer.
  • the content of the alignment auxiliary information set by the writing sub-module 112 in the multiplexed code stream may be the same or different, as long as the content can be characterized as requiring an alignment operation and can be correctly identified.
  • the number of the alignment auxiliary information set by the write sub-module 112 in the multiple code streams may be the same or different.
  • the multiple code streams are two different standard video streams
  • one alignment auxiliary information may be set in one of the code streams
  • a plurality of alignment auxiliary information may be set in the other code stream.
  • the content of multiple alignment auxiliary information set on the same layer for one code stream is the same.
  • the alignment assistance information set by the writing sub-module 112 includes at least aligning the scalar content, or may further include the alignment indication information when the scalar content is included.
  • the alignment scalar content in this embodiment includes at least image absolute time information (ie, system time when hybrid encoding is performed), image relative time information, image decoding order information, image playback order information, image separator, and image specific delimiter.
  • image absolute time information ie, system time when hybrid encoding is performed
  • image relative time information ie, system time when hybrid encoding is performed
  • image decoding order information ie, image relative time information
  • image decoding order information image playback order information
  • image separator image specific delimiter.
  • image specific delimiter ie, image relative time information
  • the alignment indication information may specifically be alignment identification information and/or type information.
  • the receiving terminal 2 includes an obtaining module 21 and an executing module 22;
  • the obtaining module 21 is configured to obtain a multiplex code stream; the acquiring manner may be performed by receiving the multiplexed code stream from the sending terminal or the server, or directly reading from the corresponding storage medium;
  • the execution module 22 is configured to: when the multiplexed code stream carries the alignment auxiliary information set by the sending terminal, according to the pair The auxiliary information is synchronized to the multiplexed code stream.
  • the execution module 22 includes a lookup submodule 221 for searching alignment assistance information from at least one of an application layer, a system layer, and a video layer of the multiplexed code stream.
  • the search sub-module 221 finds that there is inconsistent alignment auxiliary information in the multi-layer of one-way code stream, the alignment auxiliary information of one layer is selected according to the preset rule for synchronization processing.
  • the preset rules in the embodiment include but are not limited to:
  • Selecting alignment assistance information in a layer with high channel transmission reliability for example, when both the application layer and the video layer have alignment assistance information and are inconsistent, but the application layer channel transmission reliability is higher than the video layer channel transmission reliability, then the selection is Alignment auxiliary information of the application layer;
  • the alignment auxiliary information in the video layer is selected, otherwise, the alignment auxiliary information in the layer closest to the video layer is selected; for example, when the video layer and the application layer have alignment auxiliary information and In case of inconsistency, the alignment auxiliary information of the video layer is selected; when there is no alignment auxiliary information in the video layer, but the system layer and the application layer have inconsistent alignment auxiliary information, the alignment auxiliary information of the video layer closer to the video layer is selected;
  • the alignment assistance information in the active layer supported by the receiving terminal is selected; in this embodiment, the active layer supported by the receiving terminal refers to a layer that the receiving terminal needs to decode, and the receiving terminal does not need to decode and directly discard the layer to be relatively received.
  • the terminal is an invalid layer. For example, when the processing mode of the SEI in the video layer is directly discarded, when the alignment auxiliary information exists in the SEI field in the application layer and the video layer, the receiving terminal directly selects the alignment auxiliary information in the application layer. If there is no active layer supported by the receiving terminal in the layer with the auxiliary information, the receiving terminal may not perform the synchronization processing.
  • the user may customize other rules according to the application scenario or other factors, as long as the effective alignment assistance information can be selected to complete the alignment or synchronization operation. Just fine.
  • the search sub-module 221 finds that the layer with the relatively low channel transmission reliability of the one-way code stream has complete alignment auxiliary information, and the alignment auxiliary information of the layer with high channel transmission reliability is missing, the selection is performed.
  • the alignment auxiliary information of the layer with relatively low channel transmission reliability is synchronously processed; for example, when the alignment auxiliary information of the application layer is missing and the alignment auxiliary information of the video layer is complete, although the channel transmission reliability of the application layer is higher than that of the video layer Channel transmission reliability is high, but the alignment assistance information of the video layer is still selected at this time.
  • the alignment auxiliary information of other layers outside the video layer is selected for synchronization processing. For example, when the alignment auxiliary information of the video layer is missing and the alignment auxiliary information of the application layer is complete, the alignment auxiliary information of the application layer is selected.
  • the execution module 22 further includes a synchronization sub-module 222 for performing synchronization processing on the multiplexed code stream according to the aligned scalar content.
  • a processor should not be understood to refer to hardware capable of executing software, but may implicitly include, without limitation, digital signal processor (DSP) hardware, read-only memory (ROM) for storing software, random Access memory (RAM), as well as non-volatile storage devices.
  • DSP digital signal processor
  • ROM read-only memory
  • RAM random Access memory
  • the terminal in this embodiment may be a related stream receiving and playing device in a video communication application, for example, a mobile phone, a computer, a server, a set top box, a portable mobile terminal, a digital video camera, a television broadcasting system device, or the like.
  • the apparatus can be located in a sink device that can process at least one of the following multi-layer video signals: scalable video, multi-view video, multi-view depth, multi-view video + multi-view depth.
  • stereoscopic video is a special form in which the number of viewpoints of multi-view video is equal to 2.
  • Embodiment 4 is a diagrammatic representation of Embodiment 4:
  • the Pay type of the SEI is pic_alignment_type, and its control range is an image using the SEI.
  • the SEI unit of the time type is time information
  • the bit field of Table 3 is used to further give time information.
  • the transmission of video streams of different encoding formats mainly includes the following steps:
  • Step 1 The source device (ie, the transmitting terminal) determines whether it is necessary to use different encoding formats according to application requirements (such as whether the encoder uses the base layer AVC image as a reference for the enhancement layer HEVC image, or the AVC image and the HEVC image need to be output synchronously).
  • the video is aligned.
  • the HEVC encoder in the source device encodes a time code SEI message, that is, the payload type of the SEI is equal to 136.
  • the AVC encoder in the source device encodes a Pic_timing type SEI message, that is, the payloadType of the SEI is equal to 1.
  • the SEI unit is not set.
  • Step 2 The source device uses the encoding method corresponding to u(n) to write the values of the payloadType in different encoding protocols to the SEI header of the AVC video stream and the SEI header of the HEVC video stream.
  • Step 3 The source device acquires the system time (ie, the absolute time information of the image) when the current encoder performs hybrid coding, and writes the time information as the alignment information into the payload field of the SEI of the AVC video code stream, and the HEVC video stream. SEI in the payload field.
  • Step 4 After receiving the code stream, the sink device (ie, the receiving terminal) searches for the SEI data in the AVC video code stream and the HEVC video code stream, respectively.
  • Step 5 The sink device uses the decoding method corresponding to u(n) respectively from the SEI header of the AVC video bitstream, and The value of the payloadType is obtained in the SEI header of the HEVC video stream to determine whether there is time information (pic_timing type in AVC, payloadType equal to 1, time_code type in HEVC, payloadType equal to 136).
  • Step 6 If the time information exists, the sink device acquires the corresponding field shown in Table 3 according to the payloadType, and obtains the time information of the AVC image and the HEVC image from the code stream by using the corresponding decoding method.
  • Step 7 The sink device determines, according to the obtained time information, whether the images of different encoding formats need to be synchronized.
  • Step 8 If synchronization is required, that is, the time information acquired by AVC and HEVC is consistent, the sink device puts the reconstructed image obtained by decoding the current AVC decoder into the reference frame management list of HEVC for use by the HEVC decoder. For multi-view applications, the sink device uses this information to perform synchronous output display operations on images of different viewpoints.
  • the vui_timing_info_present_flag in the payload type of the VUI is pic_alignment_flag, and its control range is an image using the VUI.
  • vui_timing_info_present_flag 1
  • the bit field of Table 3 is used to further give time information.
  • the transmission of video streams of different encoding formats mainly includes the following steps:
  • Step 1 The source device determines whether it is necessary to perform alignment operations on videos of different encoding formats according to application requirements (such as whether the encoder uses the base layer AVC image as a reference for the enhancement layer HEVC image, or the AVC image and the HEVC image need to be output synchronously).
  • the HEVC encoder in the source device encodes a VUI unit, where vui_timing_info_present_flag in the VUI is equal to one.
  • the AVC encoder in the source device encodes a VUI unit in which the vui_timing_info_present_flag in the VUI is equal to one.
  • vui_timing_info_present_flag is equal to zero.
  • Step 2 The source device uses the encoding method corresponding to u(1) to write the values of vui_timing_info_present_flag to the VUI of the AVC video code stream and the VUI of the HEVC video code stream.
  • Step 3 The source device obtains the system time when the current encoder performs hybrid coding, and writes the time information as the alignment information into the time information field of the VUI of the AVC video code stream, and the time information field of the VUI of the HEVC video code stream. in.
  • Step 4 After receiving the code stream, the sink device searches for VUI data in the AVC video code stream and the HEVC video code stream, respectively.
  • Step 5 The sink device uses the decoding method corresponding to u(1) to obtain the value of vui_timing_info_present_flag from the VUI of the AVC video code stream and the VUI of the HEVC video code stream, respectively, to determine whether there is time information (whether vui_timing_info_present_flag is equal to 1).
  • Step 6 If there is time information (vui_timing_info_present_flag is equal to 1), the sink device acquires time information of the AVC image and the HEVC image from the code stream respectively according to the corresponding field shown in Table 3.
  • Step 7 The sink device determines, according to the obtained time information, whether the images of different encoding formats need to be synchronized.
  • Step 8 If synchronization is required, that is, the time information acquired by AVC and HEVC is consistent, the sink device puts the reconstructed image obtained by decoding the current AVC decoder into the reference frame management list of HEVC for use by the HEVC decoder. For multi-view applications, the sink device uses this information to perform synchronous output display operations on images of different viewpoints.
  • the extension_flag in the field corresponding to the video sharing information is pic_alignment_flag
  • the control range is an image using the parameter set.
  • the extension_flag of the corresponding parameter set is 1
  • the extension_data is the alignment_code
  • the bit field of Table 3 is used to further give the time information.
  • the source device and the sink device operate on the pic_alignment_flag and alignment_code similarly to the example 2.
  • This example differs from the example 2 in that the user can define the time information by itself, and the source device and the sink device can negotiate a unified rule.
  • the control range of the video parameter set is a multi-frame image, and the video parameter set can store single-frame image time information, and can also store multi-frame image time information;
  • the control range of the sequence parameter set is a multi-frame image, and the sequence parameter set can store single frame image time information, and can also store multi-frame image time information;
  • the control range of the image parameter set is a single frame image, and then the image parameter set stores the single frame image time information.
  • the position at which the time information appears in the video parameter set, the sequence parameter set, and the image parameter set may be any combination.
  • the set time information can be the same or combined.
  • the slice_segement_header_extension_present_flag in the field corresponding to the slice header information is pic_alignment_flag.
  • slice_segement_header_extension_present_flag 1
  • slice_segment_header_extension_data is alignment_code
  • the bit field of Table 3 is used to further give time information.
  • the source device and the sink device operate on the pic_alignment_flag and alignment_code similarly to the example 2.
  • This example differs from Example 2 in that the position at which the time information is located is not the entire frame image control range, but a slice. Requires slices in the same frame image, if there is time information, it must be consistent. And the user can define the time information by itself, and the source device and the sink device can negotiate a unified rule.
  • the alignment assistance information is located in a data structure in which at least one image (frame image and/or field image) is included in the other range of action, and at least one image (frame image and/or field image) is included for the other range of action.
  • the data structure if the data structure contains data information that must be used in other decoding processes, the data structure is a data structure necessary for the decoding process. If the data structure uses type information similar to pic_alignment_type, the source device and the sink device operate on the pic_alignment_type and alignment_code similarly to the example 1. If the data structure uses a pic_alignment_flag-like identifier, the source device and the sink device operate on the pic_alignment_flag and alignment_code similarly to the example 2.
  • This example differs from Example 1 and Example 2 in that the source device writes pic_alignment_flag and/or pic_alignment_type, alignment_code to the correspondence of the data structure containing at least one image (frame image and/or field image) using a corresponding encoding method.
  • the sink device parses the pic_alignment_flag and/or pic_alignment_type, alignment_code corresponding fields from the data structure corresponding code stream including at least one image (frame image and/or field image) using a corresponding decoding method to obtain their values.
  • the alignment or synchronization information is located in the system layer, such as the MPEG-2 TS (Transport Stream) layer, and PTS_DTS_flags is pic_alignment_flag, which is located in the PES_packet of the TS.
  • the control range is the Transport Stream to which the current PES belongs.
  • PTS_DTS_flags is equal to 2, indicating that there is time information PTS (Presentation Time Stamps), and PTS_DTS_flags is equal to 3, indicating that there is time information PTS and DTS (Decoding Time Stamps).
  • the PTS_DTS_flags value is equal to 0, indicating that there is no time information PTS or DTS.
  • the bit information of Table 3 is used to further give time information.
  • the transmission of video streams of different encoding formats mainly includes the following steps:
  • Step 1 The source device determines whether it is necessary to perform alignment operations on videos of different encoding formats according to application requirements (such as whether the encoder uses the base layer AVC image as a reference for the enhancement layer HEVC image, or the AVC image and the HEVC image need to be output synchronously).
  • the source device sets PTS_DTS_flags to 2 or 3, otherwise sets PTS_DTS_flags to 0.
  • Step 2 The source device uses the corresponding encoding method to write the values of PTS_DTS_flags to the PES header of the MPEG-2 TS encapsulating the AVC video stream, and the PES header of the MPEG-2 TS encapsulating the HEVC video stream.
  • the video streams of non-HEVC and HEVC are in different TS packets.
  • Step 3 The source device obtains the system time when the current encoder performs hybrid coding, and writes the time information as the alignment information into the PES header of the MPEG-2 TS encapsulating the AVC video stream, and the MPEG-encapsulated HEVC video stream. 2TS PES header.
  • Step 4 After receiving the code stream, the sink device searches for PTS_DTS_flags in the PES header of the MPEG-2 TS encapsulating the AVC video stream and the PES header of the MPEG-2 TS encapsulating the HEVC video stream.
  • Step 5 The sink device uses the corresponding decoding method to separately encode the AVC video stream from the MPEG-2 TS.
  • the value of PTS_DTS_flags is obtained in the PES header of the MPEG-2 TS encapsulating the HEVC video stream, and it is judged whether or not there is time information (whether PTS_DTS_flags is equal to 2 or 3).
  • Step 6 If there is time information (PTS_DTS_flags is equal to 2 or 3), the sink device obtains the PES header of the MPEG-2 TS encapsulating the AVC video stream from the code stream according to the corresponding field shown in Table 3, using the corresponding decoding method. And time information in the PES header of the MPEG-2 TS that encapsulates the HEVC video stream.
  • Step 7 The sink device determines, according to the obtained time information, whether the images of different encoding formats need to be synchronized.
  • Step 8 If synchronization is required, that is, the time information acquired by AVC and HEVC is consistent, the sink device puts the reconstructed image obtained by decoding the current AVC decoder into the reference frame management list of HEVC for use by the HEVC decoder. For multi-view applications, the sink device uses this information to perform synchronous output display operations on images of different viewpoints.
  • the foregoing alignment assistance information is located in a field corresponding to the file video media attribute, and the field of the video media attribute including the pic_alignment_flag and/or pic_alignment_type ranges from the video stream of the entire different encoding format in the media file code stream.
  • the operation method of the source device and the sink device for pic_alignment_flag and/or pic_alignment_type is similar to that of the example 1 and the example 2.
  • This example differs from Example 1 and Example 2 in that the source device writes the values of pic_alignment_flag and/or pic_alignment_type, alignment_code into the corresponding system code stream of the field of the video media attribute using a corresponding encoding method.
  • the sink device uses the corresponding decoding method to parse the pic_alignment_flag and/or pic_alignment_type, alignment_code corresponding fields from the corresponding system code stream describing the field of the video media attribute, and obtain their values.
  • the foregoing alignment assistance information is located in the application layer media data area, and the data area containing the pic_alignment_flag and/or pic_alignment_type ranges from the video code stream of the entire different coding format in the application layer code stream, the source device and the sink device.
  • the operation methods for pic_alignment_flag and/or pic_alignment_type are similar to those of Example 1 and Example 2.
  • the present example differs from the example 1 and the example 2 in that the source device writes the values of pic_alignment_flag and/or pic_alignment_type, alignment_code into the corresponding system code stream of the media data area using a corresponding encoding method.
  • the sink device parses the pic_alignment_flag and/or pic_alignment_type, alignment_code corresponding fields from the corresponding system code stream in the media data area by using a corresponding decoding method, and obtains their values.
  • the above alignment or synchronization information is written into Private Data Bytes in the DVB application layer code stream containing video data of different encoding formats.
  • Step 1 The source device determines whether it is necessary or not according to the application requirements (such as whether the encoder uses the base layer AVC image as a reference for the enhancement layer HEVC image, or the AVC image and the HEVC image need to be output synchronously) Align with video in encoded format.
  • the application requirements such as whether the encoder uses the base layer AVC image as a reference for the enhancement layer HEVC image, or the AVC image and the HEVC image need to be output synchronously
  • the source device sets the private data flag private_data_flag to 1, and the private packet header type data_field_tag to one of the user-defined values (0xA0 to 0xFF, not limited to this value range, and the specific definition value is agreed with the sink device). Otherwise set private_data_flag to 0.
  • Step 2 The source device uses the corresponding encoding method to write the value of the data_field_tag into the DVB private data area of the encapsulated AVC video code stream, and the DVB private data area encapsulating the HEVC video code stream.
  • Step 3 The source device uses the corresponding encoding method to write the values of the data_field_tag into the DVB private data area encapsulated by the non-HEVC video stream, and the DVB private data area encapsulated by the HEVC video stream.
  • Step 4 The source device acquires the system time when the current encoder performs hybrid coding, and writes the time information as the alignment information into the DVB private data area of the encapsulated AVC video code stream, and the DVB private data area encapsulating the HEVC video code stream. in.
  • Step 5 After receiving the code stream, the sink device searches for a private_data_flag in the DVB private data area of the encapsulated AVC video code stream and the DVB private data area of the encapsulated HEVC video code stream.
  • Step 6 The sink device uses the decoding method corresponding to u(1) to obtain the value of the private_data_flag from the DVB code stream encapsulating the non-HEVC video data and the DVB code stream encapsulating the HEVC video data, to determine whether the private data information exists.
  • Step 7 If there is time information (private_data_flag is equal to 1), the sink device acquires the private data, and uses the corresponding decoding method to respectively obtain the PES header of the MPEG-2 TS encapsulating the AVC video bitstream from the code stream, and encapsulate the HEVC video bitstream.
  • the data_field_tag is decoded in the PES header of the MPEG-2 TS. If it is a user-defined value (a value in 0xA0 to 0xFF, which is not limited to this value range) agreed with the source device, time information exists. Otherwise, there is no time information. .
  • Step 8 If there is time information, the sink device uses the corresponding decoding method to obtain time information from the DVB private data area containing the non-HEVC video data and the DVB private data area containing the HEVC video data according to the field shown in Table 3.
  • Step 9 The sink device determines, according to the obtained time information, whether the images of different encoding formats need to be synchronized.
  • Step 10 If synchronization is required, that is, the time information acquired by AVC and HEVC is consistent, the sink device puts the reconstructed image obtained by decoding the current AVC decoder into the reference frame management list of HEVC for use by the HEVC decoder. For multi-view applications, the sink device uses this information to perform synchronous output display operations on images of different viewpoints.
  • the above examples 1-8 show the processing when the multiplexed code streams are different standard code streams.
  • This example illustrates the case where the multiplexed code streams are the same standard code stream.
  • the synchronization process is similar to the multiplexed code stream being different standard code streams.
  • the following two streams are used as an example.
  • both the base layer and the enhancement layer are H.265/HEVC video streams.
  • the source device determines the form of adding alignment or synchronization information to the two H.265/HEVC video streams according to the application requirements.
  • the sink device gets the required alignment or synchronization information.
  • the source device may use the methods described in Examples 1 to 8, to set the video stream layer, the pic_alignment_flag and/or pic_alignment_type, alignment_code values in the upper layer data structure and the lower layer data structure in the system layer and the application layer, and use the corresponding encoding.
  • the method writes it to the code stream.
  • the sink device processes the received code stream, and obtains the values of pic_alignment_flag and/or pic_alignment_type, alignment_code from the application layer, the system layer, and the video stream layer layer using the methods described in Examples 1 to 8, according to a unified agreement with the source device. , converted to a uniform time format.
  • the sink device uses the above methods to set an error control and playback control module during reception and decoding.
  • the locations of the different video standards or the same video standard carrying the above-mentioned alignment assistance information may be different.
  • the source device determines the location of each video stream to add alignment or synchronization information according to the application requirements, and writes the same information format.
  • the sink device performs a search lookup in the order of the application layer, the system layer, and the video coding layer until the required alignment or synchronization information is obtained.
  • the source device first determines the alignment of each code stream of different video standards or the same video standard according to the application requirements, including determining the layer where the alignment information is located, and the data format.
  • the source device sets the video stream layer, the pic_alignment_flag and/or the pic_alignment_type in the upper layer data structure and the lower layer data structure in the system layer and the application layer according to the information determined in the above, and the value of the alignment_code in the lower layer data structure. And write it to the code stream using the appropriate encoding method.
  • the sink device processes the received code stream, and obtains the values of pic_alignment_flag and/or pic_alignment_type, alignment_code from the application layer, the system layer, and the video stream layer layer using the methods described in Examples 1 to 8, and the sink device uses the above method settings. Error control and playback control module during receiving and decoding.
  • the form of the above-mentioned alignment assistance information may be different in each code stream of different video standards or the same video standard.
  • the source device determines the form of alignment or synchronization information for each code stream of different video standards or the same video standard according to application requirements. Uniform rules are negotiated with the sink device, such as converting time information into system time or a unified sequence number.
  • the sink device performs a search lookup in the order of the application layer, the system layer, and the video coding layer until the required alignment or synchronization information is obtained.
  • the source device first determines the alignment of each code stream of different video standards or the same video standard according to the application requirements, including determining the layer where the alignment or synchronization information is located, and the data format.
  • the source device sets the video stream layer, the pic_alignment_flag and/or the pic_alignment_type in the upper layer data structure and the lower layer data structure in the system layer and the application layer according to the information determined in the above, and the value of the alignment_code in the lower layer data structure. And write it to the code stream using the appropriate encoding method.
  • the sink device processes the received code stream, and obtains the values of pic_alignment_flag and/or pic_alignment_type, alignment_code from the application layer, the system layer, and the video stream layer layer using the methods described in Examples 1 to 8, according to a unified agreement with the source device. , converted to a uniform time format.
  • the sink device uses the above methods to set an error control and playback control module during reception and decoding.
  • the above-described alignment assistance information is carried by a mixed use method.
  • the application layer is higher than the data structure of the system layer, and the system layer is higher than the data structure of the video coding layer.
  • alignment_code may be encoded in different levels of data structures.
  • the high-level data structure differs from the pic_alignment_flag and/or pic_alignment_type
  • alignment_code values in the lower-level data structure cover the values in the higher-level data structure.
  • the pic_alignment_flag and/or pic_alignment_type alignment_code in the lower layer data structure is missing, these values in the higher layer data structure can be borrowed for alignment or synchronization.
  • the source device first determines the alignment of each code stream of different video standards or the same video standard according to the application requirements, including determining the layer where the alignment or synchronization information is located, and the data format.
  • the source device sets the video stream layer, the pic_alignment_flag and/or the pic_alignment_type in the upper layer data structure and the lower layer data structure in the system layer and the application layer according to the information determined in the above, and the value of the alignment_code in the lower layer data structure. And write it to the code stream using the appropriate encoding method.
  • the sink device processes the received code stream, and obtains the values of pic_alignment_flag and/or pic_alignment_type, alignment_code from the application layer, the system layer, and the video code stream layer using the methods described in Examples 1 to 8, and the sink device sets the reception using the above methods. Error control and playback control module during decoding.
  • the sink device can use different layers to obtain the alignment or synchronization information as its code stream error detection and playback operation control conditions. If the alignment or synchronization information obtained by different layers is different, the sink device performs an error control mechanism to report an error to the source device through the feedback information. The sink device can directly perform output and playback control between the layer image and other layer images according to the alignment or synchronization information.
  • the method provided by the embodiment of the present invention can add a description of video alignment of different coding formats in a code stream upper layer, a system layer, a video layer description, and the like.
  • a layered description mechanism is adopted on the high-level structure of the code stream, which is beneficial to the flexible control in the code stream generation process.
  • modules or steps of the present invention described above can be implemented by a general-purpose computing device that can be centralized on a single computing device or distributed across a network of multiple computing devices. Alternatively, they may be implemented by program code executable by the computing device such that they may be stored in the storage device by the computing device and, in some cases, may be different from the order herein.
  • the steps shown or described are performed, or they are separately fabricated into individual integrated circuit modules, or a plurality of modules or steps thereof are fabricated as a single integrated circuit module.
  • the invention is not limited to any specific combination of hardware and software.
  • the invention is applicable to a related code stream sending, storing or receiving playing device in a video communication application, and is used for aligning and synchronizing the video code stream.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)

Abstract

本发明公开了一种码流对齐、同步处理方法及发送、接收终端和通信***，发送终端在多路码流需要对齐时，在该多路码流中设置对齐辅助信息后进行存储或发送给接收终端；接收终端获取到该多路码流后，则可根据各路码流中携带的对齐辅助信息对各路不同标准码流进行同步处理。可见，本发明直接在各路码流中设置对齐辅助信息，实现简单，且基本不需要对编解码器进行额外的改动，通用性好，且实现成本更低。 (图1)

Description

码流对齐、同步处理方法及发送、接收终端和通信*** 技术领域
本发明涉及通信领域，具体涉及一种码流对齐、同步处理方法及发送、接收终端和通信***。
背景技术
由ITU-T/VCEG(Video Coding Experts Group)和ISO/IEC MPEG联合制定的新一代视频压缩标准H.265/HEVC(High Efficiency Video Coding)，延续了上一代视频标准H.264/AVC的风格，兼容可伸缩视频编码(Scalable Video Coding)，与H.264/AVC不同的一个方面是，提出了多个编码标准联合的可伸缩编码(Different Coding Standards with Scalability)的需求。目前已发布的H.265/HEVC标准(2014-10版本)称为“H.265/HEVC Version 2”标准，在Version 2中为混合编码(Hybrid Scalability)提供一个用于指示基本层(Base Layer)的信息，至于如何将基本层(Base Layer)的图像放置进入增强层(Enhancement Layer)的参考帧管理中，则采用外部方法(External Means)实现，而外部方法在Version 2中不做说明。
预计未来几年使用H.264/AVC的设备将广泛应用，在全世界范围内，H.264/AVC将是数字视频主要采用的视频编码协议。因此，发展能够兼容H.264/AVC的新一代视频可伸缩编码标准，在标准的迁移过程中就会非常有用。
多个编码标准联合的可伸缩编码是指两个不同的视频标准码流根据一定的规则组合，也称为多协议混合编码(Hybrid Coding)。例如，空间可伸缩(Spatial Scalability)，其中基本层(Base Layer)是H.264/AVC标准Constrained Baseline Profile，Main Profile和High Profile兼容的码流，增强层(Enhancement Layer)包含HEVC基本码流。这两层图像的尺寸可以相同，也可以不相同，即H.264/AVC采用低分辨率，H.265/HEVC采用高分辨率。空间可伸缩编码的基本层也可以使用MPEG-2的码流，与H.265/HEVC的增强层码流组合为混合编码。再如，视点可伸缩(View Scalability)，其中基本层是一个视点，采用H.264/AVC编码方式，增强层是另一个视点，采用H.265/HEVC格式，以基本层作为参考。当然，多个编码标准联合的混合编码的基本层不止限于H.264/AVC或MPEG-2。
相同协议码流组成的可伸缩码流，可以使用多种方式进行同步，比如在H.264/AVC中同一时刻获得的视频图像及其对应的编码比特组成一个接入单元(Access Unit，AU)，因此，可以通过AU范围来同步不同层图像。在H.265/HEVC中，可以使用视频码流中的语法信息，如POC(Picture Order Count)等。对于两层或者多层不同视频编码协议的码流来说，编解码器是各自独立的编解码器，而且不同的视频码流中包含的语法信息也略有不同，那么以何种形式进行码流封装，或者提供 哪些语法信息，才能够确保同步，是混合编码(Hybrid Coding)需要解决的问题。
对此，现有的一种方法是将H.264/AVC和H.265/HEVC的码流封装起来，作为一个整体，采用带内方法(internal means)传输。也就是说在H.265/HEVC中增加一些语法元素，可以指示H.264/AVC码流存在的形式。JCT-VC标准会议有很多相关提案，如JCTVC-N0050针对AVC-HEVC混合码流提出了解决方案，在H.264/AVC NAL单元添加一个H.265/HEVC的新增NAL头(nal_unit_type＝ENC_NUT)，封装为HEVC NAL单元，在VPS扩展中增加avc_base_layer_flag来指示avc_base_profile_level_idc是否存在，而avc_base_profile_level_idc指示H.264/AVC的能力集(profile)。这种方法的缺点是一个标准的H.264/AVC解码器无法解码此种码流，需要先解包去掉对应H.265/HEVC的NAL单元头信息，才能够正确解码。
JCTVC-O0190是针对JCTVC-N0050提出了另一个不同的封装策略。其也是将H.264/AVC和H.265/HEVC的码流封装起来，作为一个整体，但基本层H.264/AVC NAL单元不改变，增强层H.265/HEVC NAL单元增加一个前缀字节(0x16)用来让H.264/AVC解码器识别这部分码流无法解码并且丢弃，其码流部分仍采用带内方法。同时建议增加一个变量HevcPrefixBytePresent，给解码器指示需要解析这个前缀字节。这种方法需要在NAL头中或者前面添加可标识不同码流的信息，而解码器必须能够识别属于自己的码流和不属于自己的码流。因此，无论是H.264/AVC解码器还是H.265/HEVC解码器，如果想支持混合编码(Hybrid Coding)的功能，都需要增加相应的处理。这个对于已有的设备中的解码器来说，就无法做到不改动或者改动很少即可实现功能。
可见，现有的方法都是基于带内方法对不同视频标准的码流进行传输，其对齐或同步处理过程都需要对现有的编解码设备进行改动，实现过程繁杂、通用性差且实现成本较高。
发明内容
本发明要解决的主要技术问题是，提供一种码流对齐、同步处理方法及发送、接收终端和通信***，解决现有针对多种不同视频码流对齐、同步处理实现过程繁杂，通用性差、成本高的问题。
为解决上述技术问题，本发明提供一种码流对齐方法，包括：
发送终端在多路码流需要对齐时，在所述多路码流中设置对齐辅助信息；
所述发送终端将所述多路码流进行存储或发送给接收终端。
在本发明的一种实施例中，所述码流为视频码流；所述发送终端在所述多路码流中设置对齐辅助信息包括：在所述多路码流的视频层、***层和应用层中的至少一层设置所述对齐辅助信息。
在本发明的一种实施例中，所述发送终端在所述多路码流的视频层设置所述对齐 辅助信息时，包括：在视频层以下字段中的至少一个字段设置所述对齐辅助信息：
增强信息所对应的字段；
视频可用信息所对应的字段；
视频共有信息所对应的字段；
片头信息所对应的字段；
以及码流视频层除上述字段外的其他至少包含一个图像的字段。
在本发明的一种实施例中，所述发送终端在所述多路码流的***层设置所述对齐辅助信息时，包括：在***层中用于描述视频媒体属性所对应的字段设置所述对齐辅助信息。
在本发明的一种实施例中，所述发送终端在所述多路码流的应用层设置所述对齐辅助信息时，包括：在媒体文件中用于描述视频媒体属性所对应的字段、和/或应用层中用于描述视频媒体属性所对应的字段设置所述对齐辅助信息。
在本发明的一种实施例中，所述发送终端在所述多路码流中设置所述对齐辅助信息的位置和/或内容相同。
在本发明的一种实施例中，所述发送终端在所述多路码流中设置所述对齐辅助信息的个数和/或内容不同。
在本发明的一种实施例中，所述发送终端将所述多路码流发送给接收终端包括：所述发送终端将所述多路码流分别单独发送给所述接收终端；或所述发送终端将所述多路码流中的至少两路封装成一路后发送给所述接收终端；或所述发送终端将所述多路码流拆分重新组合为多路码流后发送给所述接收终端。
在本发明的一种实施例中，所述发送终端将所述多路码流进行存储时，包括将所述多路码流写入一个文件进行存储，或将所述多路码流分别写入单独的文件进行存储，或将所述多路码流拆分重组后写入到不同的文件中进行存储。
在本发明的一种实施例中，所述对齐辅助信息包括对齐标量内容，或在包括所述对齐标量内容时，还包括对齐指示信息。
在本发明的一种实施例中，所述对齐标量内容包括图像绝对时间信息、图像相对时间信息、图像解码顺序信息、图像播放顺序信息、图像分隔符和图像特定界定符中的至少一种。
在本发明的一种实施例中，所述对齐辅助信息包括对齐指示信息时，所述对齐指示信息包括对齐标识信息和/或类型信息。
在本发明的一种实施例中，其中，所述多路码流为两路不同视频标准的视频码流，且其中一路为H.265/HEVC视频码流。
为了解决上述问题，本发明还提供了一种码流接收方法，包括：
接收终端获取多路码流；
接收终端在所述多路码流中携带有所述发送终端设置的对齐辅助信息时，根据所 述对齐辅助信息对所述多路码流进行同步处理。
在本发明的一种实施例中，所述码流为视频码流，所述接收终端从所述多路码流的应用层、***层、视频层中的至少一层查找所述对齐辅助信息。
在本发明的一种实施例中，当一路码流的多层中存在不一致的对齐辅助信息时，按照预设规则从中选择一层的对齐辅助信息进行同步处理。
在本发明的一种实施例中，所述预设规则包括：
选择信道传输可靠性高的层中的对齐辅助信息；
或，当视频层存在对齐辅助信息时，选择视频层中的对齐辅助信息，否则，选择离视频层最近的层中的对齐辅助信息；
或，选择所述接收终端需解码的有效层中的对齐辅助信息。
在本发明的一种实施例中，在一路码流的信道传输可靠性相对较低的层存在完整的对齐辅助信息，而信道传输可靠性高的层的对齐辅助信息缺失时，选择信道传输可靠性相对较低的层的对齐辅助信息进行同步处理；
或在一路码流的视频层外的其他层存在完整的对齐辅助信息，而视频层的对齐辅助信息缺失时，选择视频层外的其他层的对齐辅助信息进行同步处理。
在本发明的一种实施例中，所述对齐辅助信息至少包括对齐标量内容；所述接收终端根据所述对齐标量内容对所述多路码流进行同步处理。
为了解决上述问题，本发明还提供了一种码流处理方法，包括：
发送终端在多路码流需要对齐时，在所述多路码流中设置对齐辅助信息；
所述发送终端将所述多路码流进行存储或发送给接收终端；
所述接收终端获取所述多路码流；
所述接收终端在所述多路码流中携带有所述对齐辅助信息时，根据所述对齐辅助信息对所述多路码流进行同步处理。
为了解决上述问题，本发明还提供了处理模块和发送模块；
所述处理模块用于在多路码流需要对齐时，在所述多路码流中设置对齐辅助信息；
所述发送模块用于将所述多路码流进行存储或发送给接收终端。
在本发明的一种实施例中，所述码流为视频码流，所述处理模块包括写入子模块，用于在所述多路码流的视频层、***层和应用层中的至少一层设置所述对齐辅助信息。
为了解决上述问题，本发明还提供了获取模块和执行模块；
所述获取模块用于获取多路码流；
所述执行模块用于在所述多路码流中携带有所述发送终端设置的对齐辅助信息时，根据所述对齐辅助信息对所述多路码流进行同步处理。
在本发明的一种实施例中，所述执行模块包括查找子模块，用于从所述多路码流 的应用层、***层、视频层中的至少一层查找所述对齐辅助信息。
为了解决上述问题，本发明还提供了一种通信***，包括如上所述的发送终端和如上所述的接收终端；
所述发送终端在多路码流需要对齐时，分别在所述多路码流中设置对齐辅助信息，并将所述多路码流进行存储或发送给所述接收终端；
所述接收终端获取所述多路码流，在所述多路码流中携带有所述对齐辅助信息时，根据所述对齐辅助信息对所述多路码流进行同步处理。
本发明的有益效果是：
根据本发明提供的码流对齐、同步处理方法及发送、接收终端和通信***，发送终端在多路码流需要对齐时，在该多路码流中设置对齐辅助信息后进行存储或发送给接收终端；接收终端接收到该多路码流后，则可根据各路码流中携带的对齐辅助信息对各路不同标准码流进行同步处理。可见，本发明直接在各路码流中设置对齐辅助信息，实现简单，且基本不需要对编解码器进行额外的改动，通用性好，且实现成本更低。
附图说明
图1为本发明实施例一中发送端完成码流对齐方法的流程示意图；
图2为本发明实施例二中接收端完成码流同步方法的流程示意图；
图3为本发明实施例三中通信***示意图；
图4为本发明实施例三中发送终端结构示意图；
图5为本发明实施例三中接收终端结构示意图。
具体实施方式
本发明为了尽可能减少对现有的编解码设备进行改动，发送端在确定码流需要进行对齐处理时，将对齐辅助信息(该对齐辅助信息是相对发送端而言，相对接收端则为用于同步，因此对于接收端而言其也可称为同步辅助信息)设置到各路不同标准的码流中，接收端获取到后即可根据各路码流中的对齐辅助信息完成同步处理。本发明直接分别在各路码流中设置对齐辅助信息，实现简单，且基本不需要对编解码器进行额外的改动，通用性好，且实现成本更低。下面通过具体实施方式结合附图对本发明作进一步详细说明。
实施例一：
在本实施例中，发送端在确定当前的多路码流需要对齐时，分别在多路码流中设置对齐辅助信息，然后将该多路码流进行存储或发送给接收终端。该过程具体请参见图1所示，包括，
步骤101：发送终端判断当前的多路码流是否需要对齐，如是，转至步骤102，否则，转至步骤103；
步骤102：发送终端在多路码流中设置对齐辅助信息；
步骤103：发送终端将多路码流进行存储或发送给接收终端；此处的存储可以是在发送终端本地存储，也可以是存储到对应的服务器上，或者同时在本地和服务器上进行存储，且可以文件的形式进行存储。
上述步骤101是发送终端在需要发送当前多路码流时进行的，发送终端具体可根据应用需求判断是否需要对当前的多路码流进行对齐处理。该应用需求可以根据用户的具体需求或者具体的应用环境等因素确定，例如编码器是否使用基本层AVC图像作为增强层HEVC图像的参考，或AVC图像与HEVC图像需要同步输出等。
应当理解的是，本实施例中的码流可以是视频码流；也可以是***层上的其他码流；本实施例对具体的码流类型不做任何限定；同时应当理解的是，本实施例中的多路码流可以是相同标准的多路码流，也可以是不同标准的多路码流，具体可视具体应用场景而定。以上情况的多路码流的对齐或同步操作都可通过上述图1所示的方案实现。下面本实施例仅以视频码流作为示例对本发明进行示例性的说明。
在上述步骤102中，发送终端在多路码流中设置对齐辅助信息时，其需先确定具体需要在各路视频码流的哪个位置设置该对齐辅助信息；本实施例中，发送终端具体可以在多路码流的视频层、***层和应用层中的至少一层设置对齐辅助信息。发送终端在各路码流中设置对齐辅助信息时，可以直接利用码流中现有的信息作为对齐辅助信息，也可以选择在码流中增加新的内容作为对齐辅助信息。
具体的，发送终端在多路码流的视频层设置对齐辅助信息时，包括在视频层的以下字段中的至少一个字段设置对齐辅助信息：
(1)增强信息所对应的字段，用于该字段有效范围内的所述码流中的图像，与对应拥有相应标识及内容信息的码流图像对齐或同步；例如对于H.265/HEVC视频码流，增强信息所对应的字段具体为补充增强信息(Supplemental Enhancement Information，SEI)字段，用于指示该SEI信息有效范围内的所述码流中的图像，与对应拥有相应标识及内容信息的码流图像对齐或同步；对应的，H.264/AVC视频码流增强信息所对应的字段具体为SEI字段；应当理解的是，对于其他格式的码流，则取具有类似功能字段作为增强信息所对应的字段即可，例如，在MPEG-2和MPEG-4中使用用户自定义所对应字段(User Data)来完成同样的功能；
(2)视频可用信息所对应的字段，指示该字段有效范围内的所述码流中的图像，与对应拥有相应标识及内容信息的码流图像对齐或同步；例如对于H.265/HEVC视频码流，视频可用信息所对应的字段具体为视频可用性信息(Video Usability Information，VUI)字段，对于H.264/AVC视频码流，视频可用信息所对应的字段具体为VUI字段；对应的，对于其他格式的码流，则取具有类似功能字段作为视频可 用信息所对应的字段即可，例如，在MPEG-2和MPEG-4中使用用户自定义所对应字段(User Data)来完成同样的功能；
(3)视频共有信息所对应的字段，指示参数集信息有效范围内的所述码流中的图像，与对应拥有相应标识及内容信息的码流图像对齐或同步；例如对于H.265/HEVC视频码流，该视频共有信息所对应的字段包括视频参数集(Video Parameter Set，VPS)、序列参数集(Sequence Parameter Set，SPS)及图像参数集(Picture Parameter Set，PPS)，对应的，对于H.264/AVC视频码流，该参数集包括序列参数集(SPS)和图像参数集(PPS)，对于MPEG-4视频码流，该视频信息共有字段是VOS(Visual Object Sequences)头、VO(Visual Object)头、VOL(Video Object Layer)头、GOV(Group of Video Object Plane)头和VOP(Video Object Plane)头，对MPEG-2视频码流，该视频信息共有字段的Video Sequence头、Group of pictures header和Picture header；对应的，对于其他格式的码流，则取类似功能字段作为视频共有信息所对应字段即可；
(4)片头信息所对应的字段，指示该片头信息有效范围内的所述码流中的图像，与对应拥有相同标识及内容信息的码流图像对齐或同步；例如对于H.265/HEVC视频码流为Slice Header字段，对于H.264/AVC视频码流为Slice Header字段，对于MPGE-4视频码流为Video Packet Header字段；对应的，对于其他格式的码流，则取具有类似功能的字段作为片头信息字段即可；值得注意的是，本实施例中，对于同一幅视频图像，可以仅在第一个片头中设置，也可以在此基础上在其他至少一个片头中同时进行设置，在多个片头中设置的对齐辅助信息一致；
(5)码流视频层除上述字段外的其他至少包含一个图像(帧图像或场图像)的字段，指示码流的对应数据结构所作用的图像层是否执行对齐或同步操作，例如，用户自定义数据字段(User Data)或者扩展字段(Extension Data)；
发送终端在多路码流的***层设置对齐辅助信息时，具体包括在***层的以下字段设置：
(6)***层中用于描述视频媒体属性所对应的字段，用于指示包含在***码流中的图像，与对应拥有相应标识及内容信息的码流图像对齐或同步；
发送终端在多路码流的应用层设置所述对齐辅助信息时，包括在以下字段中的至少一个字段进行设置：
(7)媒体文件中用于描述视频媒体属性所对应的字段，用于指示包含在媒体文件中的码流中的图像，与对应拥有相应标识及内容信息的码流图像对齐或同步；
(8)应用层中用于描述视频媒体属性所对应的字段，用于指示包含在应用中的码流图像，与对应拥有相应标识及内容信息的码流图像对齐或同步。
对于以上字段中为编解码器非必要的内容的，解码器可以按照标准规则进行丢弃。
本实施例中发送终端可以在各路视频码流的上述字段中的任意一个字段进行设置，也可以同时在各路视频码流的上述字段中的多个字段进行设置，对于一路视频码流，理论上在上述任意层的任意一个字段设置对齐辅助信息即可；但是为了进一步提高可靠性，避免传输错误等因素导致对齐辅助信息的丢失或缺失，也可同时在多个字段进行设置，例如同时在一路码流的视频层、***层和应用层中的至少两层中的至少一个字段中设置。
另外，本实施例中的发送终端在各路视频码流中设置对齐辅助信息的位置可以相同，也可以不同；例如当所述多路码流为两路不同标准的视频码流时，对于这两路视频码流，可在同一位置设置对齐辅助信息，例如都在视频层的增强信息所对应的字段设置；也可在不同位置进行设置，例如其中一路在视频层的增强信息所对应的字段设置，另一路则在视频层的视频共有信息所对应的字段设置，又例如其中一路在***层中用于描述视频媒体属性的字段设置，另外一路在应用层中用于描述视频媒体属性的字段设置。
本实施例中，发送终端在多路码流中设置的对齐辅助信息的内容可以相同，也可以不同，只要该内容能表征需要进行对齐操作且可被正确识别即可。
本实施例中，发送终端在多路码流中设置对齐辅助信息的个数可以相同，也可以不同，例如当所述多路码流为两路不同标准的视频码流时，对于这两路视频码流，可在其中一路码流中设置一个对齐辅助信息，在另外一路码流中设置多个对齐辅助信息。对于一路码流在同一层设置的多个对齐辅助信息的内容相同。
本实施例中，发送终端所设置的对齐辅助信息至少包括对齐标量内容，或在包括对齐标量内容时，还可进一步包括对齐指示信息。本实施例中的对齐标量内容包括图像绝对时间信息(即进行混合编码时的***时间)、图像相对时间信息、图像解码顺序信息、图像播放顺序信息、图像分隔符和图像特定界定符中的至少一种。正如上所述，当对齐标量内容为图像解码顺序信息时，当码流中本身就存在图像解码顺序信息时，则针对该码流即可直接利用其原本就存在的图像解码顺序信息作为对齐辅助信息，对于其他路码流则可增加该图像解码顺序信息作为对齐辅助信息。
对齐辅助信息包括对齐指示信息时，对齐指示信息具体可为对齐标识信息和/或类型信息。下面以一种具体示例进行示例性说明：
以下示例所使用的针对对齐或同步操作的对齐辅助信息中的对齐指示信息的对齐标识信息和类型信息分别采用如表1和/或表2所示的结构。对齐标量内容采用如表3所示的结构。表1所示的码流里包含标识整体或部分码流是否使用对齐或同步操作的标识信息。表2所示的码流里包含标识整体或部分码流对齐或同步操作的标识信息所属类型。表3所示码流中包含对齐或同步所需的内容信息。
表1
…… Descriptor
pic_alignment_flag u(1)
……  
表2
…… Descriptor
pic_alignment_type()  
……  
表3
…… Descriptor
alignment_code()  
……  
其中，表1中各字段的语义如下：pic_alignment_flag取值等于1表示码流中的不同编码格式的视频图像需要对齐或同步操作。pic_alignment_flag取值等于0是表示码流中的不同编码格式的视频图像不需要对齐或同步操作。对于本领域技术人员来说pic_alignment_flag取值及其所表示的含义也可以采用其它方式，具体本实施例中不做限定。pic_alignment_flag使用不同编码协议中对应的编解码方法。
可选的，如果码流中不存在pic_alignment_flag对应比特字段，则其对应变量pic_alignment_flag的取值默认为0。
表2中字段的语义(对应的控制操作)如下：pic_alignment_type()表示获取类型信息，这里所指类型是指用于存储对齐或同步信息字段放置在同一类型数据段下，需要通过获取类型信息才能进一步提取对齐或同步的内容信息。对于本领域技术人员来说pic_alignment_type()的内部形式及其所表示的含义也可以采用其它方式，具体本实施例中不做限定。
可选的，如果码流中不存在pic_alignment_type()，则其对应的类型信息是不存在的。
表3中字段语义(对应的控制操作)如下：alignment_code()包含的是对齐或同步所需的数据信息。表3中的字段可以与表1和/或表2中的字段组合使用。表3中alignment_code()可以使用以下所列信息中的一种或者多种，包括但不限于以下信息：图像绝对时间信息，图像相对时间信息，图像解码顺序信息，图像播放顺序信息，图像分隔符，图像特殊界定符等。
表1、表2和表3中的语法元素可以设置在上述(1)－(8)所示的至少一个字段或者多个字段混合的位置；表1、表2和表3中的语法元素对应的比特字段使用在对应于该数据结构的码流中。
上述步骤103中，发送终端将多路码流发送给接收终端时，其根据具体应用场景，可以灵活地选择将多路码流分别单独发送给接收终端；或选择将多路码流中的至少两 路封装成一路后发送给接收终端。例如，当码流为三路(分别为A、B、C)时，可以选择将该三路封装成一路发送；也可以选择将该三路中的其中两路(例如A和B路)封装成一路，然后将封装后得到的一路码流和C路码流分别单独发送；对于三路以上的情况的处理则以此类推。还可以将多路码流拆分重新组合为多路码流后发送给所述接收终端。
上述步骤103中，发送终端将多路码流进行存储时，不管其是在本地存储还是存储到服务器上，都可以将多路码流写入一个文件进行存储，或将多路码流分别写入单独的文件进行存储，或将多路码流拆分重组后写入到不同的文件中进行存储。
优选的，本实施例中的多路码流具体设定为两路不同视频标准的视频码流，且其中一路(例如增强层)为H.265/HEVC视频码流，另一路(例如基本层)为非H.265/HEVC视频码流，例如H.264/AVC视频码流、MPGE-2视频码流、MPGE-4视频码流、VC-1视频流、VP8视频流、AVS视频流等。当然，本实施例中的多路码流也可具体设定为两路相同视频标准的视频码流，且其中一路(例如基本层)为H.265/HEVC视频码流，另一路(例如增强层)也为H.265/HEVC视频码流。
实施例二：
在本实施例中，接收终端进行码流同步的操作请参见图2所示，包括：
步骤201：接收终端获取多路码流；其获取方式可以从发送终端或服务器接收该多路码流，或者直接从对应存储介质中读取；
应当理解的是，该步骤中，发送终端将多路码流分别单独发送给接收终端时，则接收终端分别单独接收各路码流；当发送终端将多路码流封装成一路后发送给接收终端时，接收终端则接受封装后的一路码流即可；
步骤202：接收终端在多路码流中携带有发送终端设置的对齐辅助信息时，根据对齐辅助信息对多路码流进行同步处理。
应当理解的是，本实施例中的码流可以是视频码流；也可以是***层上的其他码流；本实施例对具体的码流类型不做任何限定；同时应当理解的是，本实施例中的多路码流可以是相同标准的多路码流，也可以是不同标准的多路码流，具体可视具体应用场景而定。下面本实施例仅以视频码流作为示例对本发明进行示例性的说明。
本实施例中，接收终端接收到码流后，从多路码流的应用层、***层、视频层中的至少一层中查找对齐辅助信息。例如，当发送和接收终端二者之间有达成协议当需要对齐或同步时对其辅助信息具体添加到哪一层或几层时，则只需在该层查找即可；否则，则需要从码流的应用层、***层、视频层依次查找。
本实施例中，当一路码流的多层中存在不一致的对齐辅助信息时，则按照预设规则从中选择一层的对齐辅助信息进行同步处理。实施例中的预设规则包括但不限于：
选择信道传输可靠性高的层中的对齐辅助信息；例如当应用层和视频层都存在对 齐辅助信息且不一致，但应用层的信道传输可靠性要比视频层的信道传输可靠性高，则选择应用层的对齐辅助信息；
或，当视频层存在对齐辅助信息时，则选择视频层中的对齐辅助信息，否则，选择离视频层最近的层中的对齐辅助信息；例如：当视频层和应用层都存在对齐辅助信息且不一致时，则选择视频层的对齐辅助信息；当视频层不存在对齐辅助信息，但***层和应用层存在不一致的对齐辅助信息时，则选择离视频层较近的视频层的对齐辅助信息；
或，选择接收终端所支持的有效层中的对齐辅助信息；本实施例中接收终端所支持的有效层是指接收终端需进行解码的层，对于接收终端不需要解码直接丢弃的层则相对接收终端为无效层。例如，当接收终端对于视频层中的SEI的处理方式是直接丢弃时，则当应用层和视频层中的SEI字段中都存在对齐辅助信息时，则接收终端直接选择应用层中的对齐辅助信息；如果具有对齐辅助信息的层中根本就没有接收终端所支持的有效层，则接收终端可以不做同步处理。
应当理解的是，本实施例中的预设规则除了上述示例的两种规则外，用户还可根据应用场景或其他因素自定义其他规则，主要能选择出有效的对齐辅助信息完成对齐或同步操作即可。
另外，在本实施例中，当一路码流的信道传输可靠性相对较低的层存在完整的对齐辅助信息，而信道传输可靠性高的层的对齐辅助信息缺失时，选择信道传输可靠性相对较低的层的对齐辅助信息进行同步处理；例如，当应用层的对齐辅助信息缺失，视频层的对齐辅助信息完整时，尽管应用层的信道传输可靠性要比视频层的信道传输可靠性高，但此时仍选择视频层的对齐辅助信息。
或当在一路码流的视频层外的其他层存在完整的对齐辅助信息，而视频层的对齐辅助信息缺失时，选择视频层外的其他层的对齐辅助信息进行同步处理。例如当视频层的对齐辅助信息缺失，而应用层的对齐辅助信息完整时，则选择应用层的对齐辅助信息。
本实施例中，发送终端所设置的对齐辅助信息至少包括对齐标量内容，或在包括对齐标量内容时，还可进一步包括对齐指示信息。本实施例中的对齐标量内容包括图像绝对时间信息(即进行混合编码时的***时间)、图像相对时间信息、图像解码顺序信息、图像播放顺序信息、图像分隔符和图像特定界定符中的至少一种。对齐辅助信息包括对齐指示信息时，对齐指示信息具体可为对齐标识信息和/或类型信息。接收终端则具体根据对齐标量内容进行对多路码流进行同步处理。
优选的，本实施例中的多路码流具体设定为两路不同视频标准的码流，且其中一路为H.265/HEVC视频码流，另一路为非H.265/HEVC视频码流，例如H.263/AVC视频码流、MPGE-2视频码流、MPGE-4视频码流、VC-1视频流、VP8视频流、AVS视频流等。当然，本实施例中的多路码流也可具体设定为两路相同视频标准的视频码 流，且其中一路(例如基本层)为H.265/HEVC视频码流，另一路(例如增强层)也为H.265/HEVC视频码流。
实施例三：
请参见图3所示，本实施例提供了一种通信***，包括发送终端1和接收终端2，其中，发送终端1和接收终端2的结构请分别参见图4和图5所示。
请参见图4所示，发送终端1，即源设备，包括处理模块11和发送模块12；
处理模块11用于在当前多路码流需要对齐时，在多路码流中设置对齐辅助信息；在各路码流中设置对齐辅助信息时，可以直接利用码流中现有的信息作为对齐辅助信息，也可以选择在码流中增加新的内容作为对齐辅助信息。
发送模块12用于将多路码流进行存储或发送给接收终端；进行存储时，其可以在发送终端本地存储，也可以在服务器上进行存储等；且可以将多路码流写入一个文件进行存储，或将多路码流分别写入单独的文件进行存储，或将多路码流拆分重组后写入到不同的文件中进行存储。进行发送时，其根据具体应用场景，可以灵活地选择将多路码流分别单独发送给接收终端；或选择将多路码流封装成一路后发送给接收终端；或将所述多路码流拆分重新组合为多路码流后发送给所述接收终端。
处理模块11具体包括判断子模块111和写入子模块112；判断子模块111用于根据应用需求判断是否需要对当前的多路码流进行对齐处理。该应用需求可以根据用户的具体需求或者具体的应用环境等因素确定，例如编码器是否使用基本层AVC图像作为增强层HEVC图像的参考，或AVC图像与HEVC图像需要同步输出等；写入子模块112用于在码流的对应位置写入对齐辅助信息。
应当理解的是，本实施例中的码流可以是视频码流；也可以是***层上的其他码流；本实施例对具体的码流类型不做任何限定；同时应当理解的是，本实施例中的多路码流可以是相同标准的多路码流，也可以是不同标准的多路码流，具体可视具体应用场景而定。下面本实施例仅以视频码流作为示例对本发明进行示例性的说明。
写入子模块112用于分别在多路码流的视频层、***层和应用层中的至少一层设置所述对齐辅助信息。具体的，发送终端在多路码流的视频层设置对齐辅助信息时，包括在视频层以下字段中的至少一个字段设置对齐辅助信息：
(1)增强信息所对应的字段；
(2)视频可用信息所对应的字段；
(3)视频共有信息所对应的字段；
(4)片头信息所对应的字段；
(5)码流视频层除上述字段外的其他至少包含一个图像(帧图像或场图像)的字段；
发送终端在多路码流的***层设置对齐辅助信息时，具体包括在***层的以下字 段设置：
(6)***层中用于描述视频媒体属性所对应的字段；
发送终端在多路码流的应用层设置所述对齐辅助信息时，包括在以下字段中的至少一个字段进行设置：
(7)媒体文件中用于描述视频媒体属性所对应的字段；
(8)应用层中用于描述视频媒体属性所对应的字段。
写入子模块112可以在各路视频码流的上述字段中的任意一个字段进行设置，也可以同时在各路视频码流的上述字段中的多个字段进行设置，对于一路视频码流，理论上在上述任意层的任意一个字段设置对齐辅助信息即可；但是为了进一步提高可靠性，避免传输错误等因素导致对齐辅助信息的丢失或缺失，也可同时在多个字段进行设置，例如同时在一路码流的视频层、***层和应用层中的至少两层中的至少一个字段中设置。
另外，本实施例中的写入子模块112在各路视频码流中设置对齐辅助信息的位置可以相同，也可以不同；例如当所述多路码流为两路不同标准的视频码流时，对于这两路视频码流，可在同一位置设置对齐辅助信息，例如都在视频层的增强信息所对应的字段设置；也可在不同位置进行设置，例如其中一路在视频层的增强信息所对应的字段设置，另一路则在是视频层的视频共有信息所对应的字段设置，又例如其中一路在***层中用于描述视频媒体属性的字段，另外一路在应用层中用于描述视频媒体属性的字段设置。
本实施例中，写入子模块112在多路码流中设置的对齐辅助信息的内容可以相同，也可以不同，只要该内容能表征需要进行对齐操作且可被正确识别即可。
本实施例中，写入子模块112在多路码流中设置对齐辅助信息的个数可以相同，也可以不同，例如当所述多路码流为两路不同标准的视频码流时，对于这两路视频码流，可在其中一路码流中设置一个对齐辅助信息，在另外一路码流中设置多个对齐辅助信息。对于一路码流在同一层设置的多个对齐辅助信息的内容相同。
本实施例中，写入子模块112所设置的对齐辅助信息至少包括对齐标量内容，或在包括对齐标量内容时，还可进一步包括对齐指示信息。本实施例中的对齐标量内容包括图像绝对时间信息(即进行混合编码时的***时间)、图像相对时间信息、图像解码顺序信息、图像播放顺序信息、图像分隔符和图像特定界定符中的至少一种。对齐辅助信息包括对齐指示信息时，对齐指示信息具体可为对齐标识信息和/或类型信息。
请参见图5所示，接收终端2包括获取模块21和执行模块22；
获取模块21用于获取多路码流；其获取方式可以从发送终端或服务器接收该多路码流，或者直接从对应存储介质中读取；
执行模块22用于在多路码流中携带有发送终端设置的对齐辅助信息时，根据对 齐辅助信息对多路码流进行同步处理。
其中，执行模块22包括查找子模块221，用于从多路码流的应用层、***层、视频层中的至少一层查找对齐辅助信息。当查找子模块221发现一路码流的多层中存在不一致的对齐辅助信息时，则按照预设规则从中选择一层的对齐辅助信息进行同步处理。实施例中的预设规则包括但不限于：
选择信道传输可靠性高的层中的对齐辅助信息；例如当应用层和视频层都存在对齐辅助信息且不一致，但应用层的信道传输可靠性要比视频层的信道传输可靠性高，则选择应用层的对齐辅助信息；
或，当视频层存在对齐辅助信息时，则选择视频层中的对齐辅助信息，否则，选择离视频层最近的层中的对齐辅助信息；例如：当视频层和应用层都存在对齐辅助信息且不一致时，则选择视频层的对齐辅助信息；当视频层不存在对齐辅助信息，但***层和应用层存在不一致的对齐辅助信息时，则选择离视频层较近的视频层的对齐辅助信息；
或，选择接收终端所支持的有效层中的对齐辅助信息；本实施例中接收终端所支持的有效层是指接收终端需进行解码的层，对于接收终端不需要解码直接丢弃的层则相对接收终端为无效层。例如，当接收终端对于视频层中的SEI的处理方式是直接丢弃时，则当应用层和视频层中的SEI字段中都存在对齐辅助信息时，则接收终端直接选择应用层中的对齐辅助信息；如果具有对齐辅助信息的层中根本就没有接收终端所支持的有效层，则接收终端可以不做同步处理。
应当理解的是，本实施例中的预设规则除了上述示例的两种规则外，用户还可根据应用场景或其他因素自定义其他规则，只要能选择出有效的对齐辅助信息完成对齐或同步操作即可。
另外，在本实施例中，当查找子模块221发现一路码流的信道传输可靠性相对较低的层存在完整的对齐辅助信息，而信道传输可靠性高的层的对齐辅助信息缺失时，选择信道传输可靠性相对较低的层的对齐辅助信息进行同步处理；例如，当应用层的对齐辅助信息缺失，视频层的对齐辅助信息完整时，尽管应用层的信道传输可靠性要比视频层的信道传输可靠性高，但此时仍选择视频层的对齐辅助信息。
或当在一路码流的视频层外的其他层存在完整的对齐辅助信息，而视频层的对齐辅助信息缺失时，选择视频层外的其他层的对齐辅助信息进行同步处理。例如当视频层的对齐辅助信息缺失，而应用层的对齐辅助信息完整时，则选择应用层的对齐辅助信息。
执行模块22还包括同步子模块222，用于根据对齐标量内容进行对多路码流进行同步处理。
应当理解的是，本实施例中的各个模块和子模块的功能可以通过使用专用硬件、软件或者软硬件结合来实现。该硬件或专用硬件可以包括专用集成电路(ASIC)、 各种其它电路、各种处理器等。当由处理器实现时，该功能可以由单个专用处理器、单个共享处理器、或者多个独立的处理器(其中某些可能被共享)来提供。另外，处理器不应该被理解为专指能够执行软件的硬件，而是可以隐含地包括、而不限于数字信号处理器(DSP)硬件、用来存储软件的只读存储器(ROM)、随机存取存储器(RAM)、以及非易失存储设备。
本实施例的终端可以是视频通信应用中相关码流接收播放设备，例如，手机、计算机、服务器、机顶盒、便携式移动终端、数字摄像机，电视广播***设备等。该装置可以位于宿设备中，该处理装置可以处理以下至少一种多层视频信号：可伸缩视频，多视点视频，多视点深度，多视点视频+多视点深度。其中，立体视频是多视点视频的一种视点数等于2的特殊形式。
实施例四：
为了更好地理解本发明，下面以几种具体的应用场景作为示例，对本发明做进一步说明。
示例1：
本示例中，SEI的负载类型(Payload Type)即为pic_alignment_type，其控制范围是使用该SEI的图像。在负载类型为时间信息的SEI单元时，使用表3的比特字段，进一步给出时间信息。
本示例中，不同编码格式视频码流的传输主要包括以下步骤：
步骤1：源设备(即发送终端)根据应用需求(如编码器是否使用基本层AVC图像作为增强层HEVC图像的参考，或AVC图像与HEVC图像需要同步输出)，判断是否需要对不同编码格式的视频进行对齐操作。
若需要使用对齐信息，源设备中HEVC编码器编码一个time_code类型SEI单元(Time code SEI message)，即SEI的payloadType等于136。源设备中AVC编码器编码一个pic_timing类型SEI单元(Picture timing SEI message)，即SEI的payloadType等于1。
若不需要使用对齐信息，则不设置SEI单元。
步骤2：源设备使用u(n)对应的编码方法将payloadType在不同编码协议里的取值分别写入到AVC视频码流的SEI包头，和HEVC视频码流的SEI包头。
步骤3：源设备获取当前编码器进行混合编码时的***时间(即图像绝对时间信息)，将此时间信息作为对齐信息分别写入AVC视频码流的SEI的payload字段中，和HEVC视频码流的SEI的payload字段中。
步骤4：宿设备(即接收终端)接收到码流后，分别在AVC视频码流和HEVC视频码流中寻找SEI数据。
步骤5：宿设备使用u(n)对应的解码方法分别从AVC视频码流的SEI包头，和 HEVC视频码流的SEI包头中获取payloadType的值，判断是否存在时间信息(AVC中pic_timing类型，payloadType等于1，HEVC中time_code类型，payloadType等于136)。
步骤6：如果存在时间信息，宿设备根据payloadType，获取对应的表3所示字段，使用对应解码方法从码流中分别获取AVC图像和HEVC图像的时间信息。
步骤7：宿设备根据得到的时间信息，确定不同编码格式的图像是否需要同步。
步骤8：如果需要同步，即AVC和HEVC获取的时间信息一致，宿设备将当前AVC解码器解码获得的重建图像放入HEVC的参考帧管理列表中，供HEVC解码器使用。对于多视点应用，宿设备利用此信息对不同视点的图像，进行同步输出显示操作。
示例2：
本示例中，VUI的负载类型中的vui_timing_info_present_flag即为pic_alignment_flag，其控制范围是使用该VUI的图像。当vui_timing_info_present_flag为1时，使用表3的比特字段，进一步给出时间信息。
本示例中，不同编码格式视频码流的传输主要包括以下步骤：
步骤1：源设备根据应用需求(如编码器是否使用基本层AVC图像作为增强层HEVC图像的参考，或AVC图像与HEVC图像需要同步输出)，判断是否需要对不同编码格式的视频进行对齐操作。
若需要使用对齐信息，源设备中HEVC编码器编码一个VUI单元，其中VUI中的vui_timing_info_present_flag等于1。源设备中AVC编码器编码一个VUI单元，其中VUI中的vui_timing_info_present_flag等于1。
若不需要使用对齐信息，则vui_timing_info_present_flag等于0。
步骤2：源设备使用u(1)对应的编码方法将vui_timing_info_present_flag的取值分别写入到AVC视频码流的VUI中，和HEVC视频码流的VUI中。
步骤3：源设备获取当前编码器进行混合编码时的***时间，将此时间信息作为对齐信息分别写入AVC视频码流的VUI的时间信息字段中，和HEVC视频码流的VUI的时间信息字段中。
步骤4：宿设备接收到码流后，分别在AVC视频码流和HEVC视频码流中寻找VUI数据。
步骤5：宿设备使用u(1)对应的解码方法分别从AVC视频码流的VUI中，和HEVC视频码流的VUI中获取vui_timing_info_present_flag的值，判断是否存在时间信息(vui_timing_info_present_flag是否等于1)。
步骤6：如果存在时间信息(vui_timing_info_present_flag等于1)，宿设备根据对应的表3所示字段，使用对应解码方法从码流中分别获取AVC图像和HEVC图像的时间信息。
步骤7：宿设备根据得到的时间信息，确定不同编码格式的图像是否需要同步。
步骤8：如果需要同步，即AVC和HEVC获取的时间信息一致，宿设备将当前AVC解码器解码获得的重建图像放入HEVC的参考帧管理列表中，供HEVC解码器使用。对于多视点应用中，宿设备利用此信息对不同视点的图像，进行同步输出显示操作。
示例3：
本示例中，视频共有信息所对应的字段(即参数集对应的字段)中的extension_flag即为pic_alignment_flag，其控制范围是使用该参数集的图像。当对应参数集的extension_flag为1时，extension_data即alignment_code，使用表3的比特字段，进一步给出时间信息。源设备和宿设备对pic_alignment_flag和alignment_code的操作方法与示例2相似。
本示例与示例2不同的是用户可以自行定义时间信息，源设备与宿设备协商统一规则即可。
例如：
视频参数集(VPS)的控制范围是多帧图像，那么视频参数集里可以存储单帧图像时间信息，也可以存储多帧图像时间信息；
序列参数集(SPS)的控制范围是多帧图像，那么序列参数集里可以存储单帧图像时间信息，也可以存储多帧图像时间信息；
图像参数集(PPS)的控制范围是单帧图像，那么图像参数集里存储的即是单帧图像时间信息。
时间信息在视频参数集、序列参数集和图像参数集中出现的位置可以是任意组合。设置的时间信息可以相同，也可以组合生成。
示例4：
本示例中，片头信息所对应字段中的slice_segement_header_extension_present_flag即pic_alignment_flag。当slice_segement_header_extension_present_flag为1时，slice_segment_header_extension_data即alignment_code，使用表3的比特字段，进一步给出时间信息。源设备和宿设备对pic_alignment_flag和alignment_code的操作方法与示例2相似。
本示例与示例2不同的是时间信息所处的位置不是整帧图像控制范围，而是片(Slice)。要求同一帧图像里的片，如果出现时间信息，那么必须一致。并且用户可以自行定义时间信息，源设备与宿设备协商统一规则即可。
示例5：
本示例中，对齐辅助信息位于其它作用范围至少包含一个图像(帧图像和/或场图像)的数据结构，对于所述其它作用范围至少包含一个图像(帧图像和/或场图像) 的数据结构，若该数据结构中包含其它解码过程中必须使用的数据信息，则该数据结构是解码过程中所必需的数据结构。如果该数据结构使用的是类似pic_alignment_type的类型信息，则源设备和宿设备对pic_alignment_type和alignment_code的操作方法与示例1相似。如果该数据结构使用的是类似pic_alignment_flag标识，则源设备和宿设备对pic_alignment_flag和alignment_code的操作方法与示例2相似。
本示例与示例1和示例2的不同之处在于源设备使用对应的编码方法将pic_alignment_flag和/或pic_alignment_type，alignment_code写入所述至少包含一个图像(帧图像和/或场图像)的数据结构的对应码流中，宿设备使用对应的解码方法从所述至少包含一个图像(帧图像和/或场图像)的数据结构对应码流中解析pic_alignment_flag和/或pic_alignment_type，alignment_code对应字段，获取它们的值。
示例6：
本示例中，对齐或同步信息位于***层，如MPEG-2TS(Transport Stream)层，PTS_DTS_flags即为pic_alignment_flag，位于TS的PES_packet中。其控制范围是当前PES所属Transport Stream。PTS_DTS_flags等于2，表示后续有时间信息PTS(Presentation Time Stamps)，PTS_DTS_flags等于3，表示后续有时间信息PTS和DTS(Decoding Time Stamps)。PTS_DTS_flags值等于0，表示后续没有时间信息PTS或DTS。
在PTS_DTS_flags字段后，使用表3的比特字段，进一步给出时间信息。
本示例中，不同编码格式视频码流的传输主要包括以下步骤：
步骤1：源设备根据应用需求(如编码器是否使用基本层AVC图像作为增强层HEVC图像的参考，或AVC图像与HEVC图像需要同步输出)，判断是否需要对不同编码格式的视频进行对齐操作。
若需要使用对齐信息，源设备设置PTS_DTS_flags为2或3，否则设置PTS_DTS_flags为0。
步骤2：源设备使用对应编码方法将PTS_DTS_flags的取值分别写入到封装AVC视频码流的MPEG-2TS的PES包头中，和封装HEVC视频码流的MPEG-2TS的PES包头中。此时非HEVC和HEVC的视频码流在不同的TS包中。
步骤3：源设备获取当前编码器进行混合编码时的***时间，将此时间信息作为对齐信息分别写入封装AVC视频码流的MPEG-2TS的PES包头中，和封装HEVC视频码流的MPEG-2TS的PES包头中。
步骤4：宿设备接收到码流后，分别在封装AVC视频码流的MPEG-2TS的PES包头中，和封装HEVC视频码流的MPEG-2TS的PES包头中寻找PTS_DTS_flags。
步骤5：宿设备使用对应解码方法分别从封装AVC视频码流的MPEG-2TS的 PES包头中，和封装HEVC视频码流的MPEG-2TS的PES包头中获取PTS_DTS_flags的值，判断是否存在时间信息(PTS_DTS_flags是否等于2或3)。
步骤6：如果存在时间信息(PTS_DTS_flags等于2或3)，宿设备根据对应的表3所示字段，使用对应解码方法从码流中分别获取封装AVC视频码流的MPEG-2TS的PES包头中，和封装HEVC视频码流的MPEG-2TS的PES包头中的时间信息。
步骤7：宿设备根据得到的时间信息，确定不同编码格式的图像是否需要同步。
步骤8：如果需要同步，即AVC和HEVC获取的时间信息一致，宿设备将当前AVC解码器解码获得的重建图像放入HEVC的参考帧管理列表中，供HEVC解码器使用。对于多视点应用中，宿设备利用此信息对不同视点的图像，进行同步输出显示操作。
示例7：
本示例中，上述对齐辅助信息位于文件视频媒体属性对应的字段中，当包含有pic_alignment_flag和/或pic_alignment_type的视频媒体属性的字段的作用范围是媒体文件码流中的整个不同编码格式的视频码流，源设备和宿设备对pic_alignment_flag和/或pic_alignment_type的操作方法与示例1和示例2的操作方法相似。
本示例与示例1和示例2的不同之处在于源设备使用对应的编码方法将pic_alignment_flag和/或pic_alignment_type，alignment_code的取值写入所述视频媒体属性的字段的对应***码流中。宿设备使用对应的解码方法从所述描述视频媒体属性的字段的对应的***码流中解析pic_alignment_flag和/或pic_alignment_type，alignment_code对应字段，获取它们的取值。
示例8：
本示例中，上述对齐辅助信息位于应用层媒体数据区，当包含有pic_alignment_flag和/或pic_alignment_type的数据区的作用范围是应用层码流中的整个不同编码格式的视频码流，源设备和宿设备对pic_alignment_flag和/或pic_alignment_type的操作方法与示例1和示例2的操作方法相似。
本示例与示例1和示例2的不同之处在于源设备使用对应的编码方法将pic_alignment_flag和/或pic_alignment_type，alignment_code的取值写入所述媒体数据区的对应***码流中。宿设备使用对应的解码方法从所述媒体数据区的对应的***码流中解析pic_alignment_flag和/或pic_alignment_type，alignment_code对应字段，获取它们的取值。
如把上述对齐或同步信息写入到包含不同编码格式视频数据的DVB应用层码流中私有数据字节区(Private Data Bytes)。
主要包括以下步骤：
步骤1：源设备根据应用需求(如编码器是否使用基本层AVC图像作为增强层HEVC图像的参考，或AVC图像与HEVC图像需要同步输出)，判断是否需要对不 同编码格式的视频进行对齐操作。
若需要使用对齐信息，源设备将私有数据标志private_data_flag设为1，私有数据包头类型data_field_tag设置为用户自定义值之一(0xA0至0xFF，不限于此数值范围，与宿设备约定具体定义值)。否则设置private_data_flag为0。
步骤2：源设备使用对应编码方法将data_field_tag的取值分别写入到封装AVC视频码流的DVB私有数据区中，和封装HEVC视频码流的DVB私有数据区中。
步骤3：源设备使用对应的编码方法将data_field_tag的取值分别写入到非HEVC视频码流封装的DVB私有数据区中，和HEVC视频码流封装的DVB私有数据区中。
步骤4：源设备获取当前编码器进行混合编码时的***时间，将此时间信息作为对齐信息分别写入封装AVC视频码流的DVB私有数据区中，和封装HEVC视频码流的DVB私有数据区中。
步骤5：宿设备接收到码流后，分别在封装AVC视频码流的DVB私有数据区中，和封装HEVC视频码流的DVB私有数据区中寻找private_data_flag是否存在。
步骤6：宿设备使用u(1)对应的解码方法从封装非HEVC视频数据的DVB码流中，和封装HEVC视频数据的DVB码流中获取private_data_flag的值，判断是否存在私有数据信息。
步骤7：如果存在时间信息(private_data_flag等于1)，宿设备获取私有数据，使用对应解码方法从码流中分别获取封装AVC视频码流的MPEG-2TS的PES包头中，和封装HEVC视频码流的MPEG-2TS的PES包头中解码出data_field_tag，如果是与源设备约定的用户自定义值(0xA0至0xFF中的某个值，不限于此数值范围)，则存在时间信息，否则，不存在时间信息。
步骤8：如果存在时间信息，宿设备根据表3所示字段，使用对应解码方法从包含非HEVC视频数据的DVB私有数据区，和包含HEVC视频数据的DVB私有数据区获取的时间信息。
步骤9：宿设备根据得到的时间信息，确定不同编码格式的图像是否需要同步。
步骤10：如果需要同步，即AVC和HEVC获取的时间信息一致，宿设备将当前AVC解码器解码获得的重建图像放入HEVC的参考帧管理列表中，供HEVC解码器使用。对于多视点应用中，宿设备利用此信息对不同视点的图像，进行同步输出显示操作。
示例9：
上述示例1-8所示都是针对多路码流为不同标准码流时的处理，本示例则对多路码流为相同标准码流的情况进行示例说明，此时的多路码流对齐或同步的处理过程与多路码流为不同标准码流类似，下面以两路相同码流为例进行说明，例如：基本层和增强层都是H.265/HEVC视频码流。此时源设备根据应用需求，确定两个H.265/HEVC视频码流添加对齐或同步信息的形式。宿设备获取所需的对齐或同步信息。
源设备可以使用示例1至示例8所述的方法，设置视频码流层，***层和应用层中高层数据结构与低层数据结构中的pic_alignment_flag和/或pic_alignment_type，alignment_code取值，并使用相应的编码方法将其写入码流。
宿设备对接收码流进行处理，使用示例1至示例8所述的方法，从应用层、***层和视频码流层查找获得pic_alignment_flag和/或pic_alignment_type，alignment_code的值，根据与源设备的统一约定，转换为统一的时间格式。宿设备使用上述各方法设置接收、解码过程中的差错控制、播放控制模块。
示例10：
本示例中，不同视频标准或相同视频标准的各码流携带上述对齐辅助信息的位置可以不同。源设备根据应用需求，确定各视频码流添加对齐或同步信息的位置，写入相同信息格式。宿设备按照应用层、***层和视频编码层的顺序进行搜索查找，直到获取所需的对齐或同步信息。
使用混合方法中，源设备首先根据应用需求，确定不同视频标准或相同视频标准的各码流的对齐方式，包括确定对齐信息所在层，以及数据形式。源设备根据以上所确定的信息，使用示例1至示例8所述的方法，设置视频码流层，***层和应用层中高层数据结构与低层数据结构中的pic_alignment_flag和/或pic_alignment_type，alignment_code取值，并使用相应的编码方法将其写入码流。
宿设备对接收码流进行处理，使用示例1至示例8所述的方法，从应用层、***层和视频码流层查找获得pic_alignment_flag和/或pic_alignment_type，alignment_code的值，宿设备使用上述各方法设置接收、解码过程中的差错控制、播放控制模块。
示例11：
本示例中，不同视频标准或相同视频标准的各码流中携带上述对齐辅助信息的形式可不同。源设备根据应用需求，确定不同视频标准或相同视频标准的各码流添加对齐或同步信息的形式。通过与宿设备协商统一规则，如将时间信息统一转换为***时间，或者统一序号。宿设备按照应用层、***层和视频编码层的顺序进行搜索查找，直到获取所需的对齐或同步信息。
使用混合方法中，源设备首先根据应用需求，确定不同视频标准或相同视频标准的各码流对齐方式，包括确定对齐或同步信息所在层，以及数据形式。源设备根据以上所确定的信息，使用示例1至示例8所述的方法，设置视频码流层，***层和应用层中高层数据结构与低层数据结构中的pic_alignment_flag和/或pic_alignment_type，alignment_code取值，并使用相应的编码方法将其写入码流。
宿设备对接收码流进行处理，使用示例1至示例8所述的方法，从应用层、***层和视频码流层查找获得pic_alignment_flag和/或pic_alignment_type，alignment_code的值，根据与源设备的统一约定，转换为统一的时间格式。宿设备使用上述各方法设置接收、解码过程中的差错控制、播放控制模块。
示例12：
本示例中，采用混合使用的方法携带上述对齐辅助信息。对于不同视频标准或相同视频标准的各码流来说，应用层是高于***层的数据结构，***层是高于视频编码层的数据结构。
在示例中，pic_alignment_flag和/或pic_alignment_type，alignment_code可以在不同层次的数据结构中进行编码。当高层数据结构与低层数据结构中的pic_alignment_flag和/或pic_alignment_type，alignment_code取值不同时，低层数据结构中的值覆盖高层数据结构中的值。当低层数据结构中的pic_alignment_flag和/或pic_alignment_type，alignment_code有缺失时，可以借用高层数据结构中的这些值用于对齐或同步。
使用混合方法中，源设备首先根据应用需求，确定不同视频标准或相同视频标准的各码流对齐方式，包括确定对齐或同步信息所在层，以及数据形式。源设备根据以上所确定的信息，使用示例1至示例8所述的方法，设置视频码流层，***层和应用层中高层数据结构与低层数据结构中的pic_alignment_flag和/或pic_alignment_type，alignment_code取值，并使用相应的编码方法将其写入码流。
宿设备对接收码流进行处理，使用示例1至示例8所述的方法，从应用层、***层和视频码流层获得pic_alignment_flag和/或pic_alignment_type，alignment_code的值，宿设备使用上述各方法设置接收、解码过程中的差错控制、播放控制模块。
宿设备可使用不同层获取得到的对齐或同步信息，作为其码流检错和播放操作控制条件。若不同层获取的对齐或同步信息有差异，宿设备执行差错控制机制，通过反馈信息向源设备报告错误。宿设备可直接根据对齐或同步信息进行该层图像与其它层图像间的输出和播放控制。
综上所述，通过本发明实施例提供的方法，可以在码流高层、***层、视频层描述等增加不同编码格式的视频对齐的描述。同时，在码流高层结构上采用分层描述机制，有利于码流生成过程中的灵活控制。
显然，本领域的技术人员应该明白，上述的本发明的各模块或各步骤可以用通用的计算装置来实现，它们可以集中在单个的计算装置上，或者分布在多个计算装置所组成的网络上，可选地，它们可以用计算装置可执行的程序代码来实现，从而，可以将它们存储在存储装置中由计算装置来执行，并且在某些情况下，可以以不同于此处的顺序执行所示出或描述的步骤，或者将它们分别制作成各个集成电路模块，或者将它们中的多个模块或步骤制作成单个集成电路模块来实现。这样，本发明不限制于任何特定的硬件和软件结合。
以上内容是结合具体的实施方式对本发明所作的进一步详细说明，不能认定本发明的具体实施只局限于这些说明。对于本发明所属技术领域的普通技术人员来说，在 不脱离本发明构思的前提下，还可以做出若干简单推演或替换，都应当视为属于本发明的保护范围。
工业实用性
本发明适用于视频通信应用中相关码流发送、存储或接收播放设备，用以对视频码流进行对齐、同步处理。

Claims (25)

  1. 一种码流对齐方法，包括：
    发送终端在多路码流需要对齐时，在所述多路码流中设置对齐辅助信息；
    所述发送终端将所述多路码流进行存储或发送给接收终端。
  2. 如权利要求1所述的码流对齐方法，其中，所述码流为视频码流；所述发送终端在所述多路码流中设置对齐辅助信息包括：在所述多路码流的视频层、***层和应用层中的至少一层设置所述对齐辅助信息。
  3. 如权利要求2所述的码流对齐方法，其中，所述发送终端在所述多路码流的视频层设置所述对齐辅助信息时，包括：在视频层以下字段中的至少一个字段设置所述对齐辅助信息：
    增强信息所对应的字段；
    视频可用信息所对应的字段；
    视频共有信息所对应的字段；
    片头信息所对应的字段；
    以及码流视频层除上述字段外的其他至少包含一个图像的字段。
  4. 如权利要求2所述的码流对齐方法，其中，所述发送终端在所述多路码流的***层设置所述对齐辅助信息时，包括：在***层中设置为描述视频媒体属性所对应的字段设置所述对齐辅助信息。
  5. 如权利要求2所述的码流对齐方法，其中，所述发送终端在所述多路码流的应用层设置所述对齐辅助信息时，包括：在媒体文件中设置为描述视频媒体属性所对应的字段、和/或应用层中设置为描述视频媒体属性所对应的字段设置所述对齐辅助信息。
  6. 如权利要求2-5任一项所述的码流对齐方法，其中，所述发送终端在所述多路码流中设置所述对齐辅助信息的位置和/或内容相同。
  7. 如权利要求2-5任一项所述的码流对齐方法，其中，所述发送终端在所述多路码流中设置所述对齐辅助信息的个数和/或内容不同。
  8. 如权利要求1-5任一项所述的码流对齐方法，其中，所述发送终端将所述多路码流发送给接收终端时，包括：所述发送终端将所述多路码流分别单独发送给所述接收终端；或所述发送终端将所述多路码流中的至少两路封装成一路后发送给所述接收终端；或所述发送终端将所述多路码流拆分重新组合为多路码流后发送给所述接收终端。
  9. 如权利要求1-5任一项所述的码流对齐方法，其中，所述发送终端将所述多路码流进行存储时，包括将所述多路码流写入一个文件进行存储，或将所述多路码流分别写入单独的文件进行存储，或将所述多路码流拆分重组后写入到不同的文件中进行存储。
  10. 如权利要求2-5任一项所述的码流对齐方法，其中，所述对齐辅助信息包括对齐标量内容，或在包括所述对齐标量内容时，还包括对齐指示信息。
  11. 如权利要求10所述的码流对齐方法，其中，所述对齐标量内容包括图像绝对时间信息、图像相对时间信息、图像解码顺序信息、图像播放顺序信息、图像分隔符和图像特定界定符中的至少一种。
  12. 如权利要求10所述的码流对齐方法，其中，所述对齐辅助信息包括对齐指示信息时，所述对齐指示信息包括对齐标识信息和/或类型信息。
  13. 如权利要求2-5任一项所述的码流对齐方法，其中，所述多路码流为两路不同视频标准的视频码流，且其中一路为H.265/HEVC视频码流。
  14. 一种码流接收方法，包括：
    接收终端获取多路码流；
    接收终端在所述多路码流中携带有发送终端设置的对齐辅助信息时，根据所述对齐辅助信息对所述多路码流进行同步处理。
  15. 如权利要求14所述的码流接收方法，其中，所述码流为视频码流，所述接收终端从所述多路码流的应用层、***层、视频层中的至少一层查找所述对齐辅助信息。
  16. 如权利要求15所述的码流接收方法，其中，当一路码流的多层中存在不一致的对齐辅助信息时，按照预设规则从中选择一层的对齐辅助信息进行同步处理。
  17. 如权利要求16所述的码流接收方法，其中，所述预设规则包括：
    选择信道传输可靠性高的层中的对齐辅助信息；
    或，当视频层存在对齐辅助信息时，选择视频层中的对齐辅助信息，当视频层不存在对齐辅助信息时，选择离视频层最近的层中的对齐辅助信息；
    或，选择所述接收终端所支持的有效层中的对齐辅助信息。
  18. 如权利要求17所述的码流接收方法，其中，在一路码流的信道传输可靠性相对较低的层存在完整的对齐辅助信息，而信道传输可靠性高的层的对齐辅助信息缺失时，选择信道传输可靠性相对较低的层的对齐辅助信息进行同步处理；
    或在一路码流的视频层外的其他层存在完整的对齐辅助信息，而视频层的对齐辅助信息缺失时，选择视频层外的其他层的对齐辅助信息进行同步处理。
  19. 如权利要求14-18任一项所述的码流接收方法，其中，所述对齐辅助信息至少包括对齐标量内容；所述接收终端根据所述对齐标量内容对所述多路码流进行同步处理。
  20. 一种码流处理方法，包括：
    发送终端在多路码流需要对齐时，在所述多路码流中设置对齐辅助信息；
    所述发送终端将所述多路码流进行存储或发送给接收终端；
    所述接收终端获取所述多路码流；
    所述接收终端在所述多路码流中携带有所述对齐辅助信息时，根据所述对齐辅助信息对所述多路码流进行同步处理。
  21. 一种发送终端，包括处理模块和发送模块；
    所述处理模块设置为在多路码流需要对齐时，在所述多路码流中设置对齐辅助信息；
    所述发送模块设置为将所述多路码流进行存储或发送给接收终端。
  22. 如权利要求21所述的发送终端，其中，所述码流为视频码流，所述处理模块包括写入子模块，设置为在所述多路码流的视频层、***层和应用层中的至少一层设置所述对齐辅助信息。
  23. 一种接收终端，包括获取模块和执行模块；
    所述获取模块设置为获取多路码流；
    所述执行模块设置为在所述多路码流中携带有所述发送终端设置的对齐辅助信息时，根据所述对齐辅助信息对所述多路码流进行同步处理。
  24. 如权利要求23所述的接收终端，其中，所述执行模块包括查找子模块，设置为从所述多路码流的应用层、***层、视频层中的至少一层查找所述对齐辅助信息。
  25. 一种通信***，包括如权利要求21或22所述的发送终端和如权利要求23或24所述的接收终端；
    所述发送终端在多路码流需要对齐时，分别在所述多路码流中设置对齐辅助信息，并将所述多路码流进行存储或发送给所述接收终端；
    所述接收终端获取所述多路码流，在所述多路码流中携带有所述对齐辅助信息时，根据所述对齐辅助信息对所述多路码流进行同步处理。
PCT/CN2016/073823 2015-06-04 2016-02-15 码流对齐、同步处理方法及发送、接收终端和通信*** WO2016192413A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP16802336.4A EP3306945A4 (en) 2015-06-04 2016-02-15 Bitstream alignment and synchronous processing method and system, receiving terminal and communication system
US15/579,310 US10582234B2 (en) 2015-06-04 2016-02-15 Bitstream alignment and synchronous processing method transmitting terminal, receiving terminal and communication system

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510304991.7A CN106303673B (zh) 2015-06-04 2015-06-04 码流对齐、同步处理方法及发送、接收终端和通信***
CN201510304991.7 2015-06-04

Publications (1)

Publication Number Publication Date
WO2016192413A1 true WO2016192413A1 (zh) 2016-12-08

Family

ID=57440003

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/073823 WO2016192413A1 (zh) 2015-06-04 2016-02-15 码流对齐、同步处理方法及发送、接收终端和通信***

Country Status (4)

Country Link
US (1) US10582234B2 (zh)
EP (1) EP3306945A4 (zh)
CN (1) CN106303673B (zh)
WO (1) WO2016192413A1 (zh)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112019852A (zh) * 2019-05-29 2020-12-01 杭州海康威视数字技术股份有限公司 一种目标类型帧识别方法、装置及电子设备
CN110798731A (zh) * 2019-11-15 2020-02-14 北京字节跳动网络技术有限公司 视频数据的处理方法、装置、电子设备及计算机可读介质
CN110958466A (zh) * 2019-12-17 2020-04-03 杭州当虹科技股份有限公司 一种基于rtmp传输的sdi信号同步回传方法
CN115695883A (zh) * 2022-09-27 2023-02-03 北京奇艺世纪科技有限公司 一种视频数据处理方法、装置、设备以及存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101951506A (zh) * 2010-09-17 2011-01-19 中兴通讯股份有限公司 实现可伸缩视频编码业务同步发送、接收的***和方法
CN102480634A (zh) * 2010-11-24 2012-05-30 中兴通讯股份有限公司 移动多媒体广播中分级业务同步的方法、设备和***
WO2015037373A1 (ja) * 2013-09-12 2015-03-19 ソニー株式会社 符号化装置、送信装置および受信装置
CN104754347A (zh) * 2013-12-26 2015-07-01 中兴通讯股份有限公司 视频图像序号的编码、解码方法及装置、电子设备
CN104754358A (zh) * 2013-12-27 2015-07-01 中兴通讯股份有限公司 码流的生成和处理方法、装置及***

Family Cites Families (34)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5784528A (en) * 1995-09-29 1998-07-21 Matsushita Electric Industrial Co. Ltd. Method and an apparatus for interleaving bitstream to record thereof on a recording medium, and reproducing the interleaved bitstream therefrom
US6188987B1 (en) * 1998-11-17 2001-02-13 Dolby Laboratories Licensing Corporation Providing auxiliary information with frame-based encoded audio information
US7010037B2 (en) * 2002-08-06 2006-03-07 Koninklijke Philips Electronics N.V. System and method for rate-distortion optimized data partitioning for video coding using backward adaptation
EP1618742A1 (en) * 2003-04-18 2006-01-25 Koninklijke Philips Electronics N.V. System and method for rate-distortion optimized data partitioning for video coding using parametric rate-distortion model
US8351514B2 (en) * 2004-01-16 2013-01-08 General Instrument Corporation Method, protocol, and apparatus for transporting advanced video coding content
US9071344B2 (en) * 2005-08-22 2015-06-30 Qualcomm Incorporated Reverse link interference cancellation
US8436889B2 (en) * 2005-12-22 2013-05-07 Vidyo, Inc. System and method for videoconferencing using scalable video coding and compositing scalable video conferencing servers
US7882442B2 (en) * 2007-01-05 2011-02-01 Eastman Kodak Company Multi-frame display system with perspective based image arrangement
US8289998B2 (en) * 2009-02-13 2012-10-16 Samsung Electronics Co., Ltd. Method and apparatus for generating three (3)-dimensional image data stream, and method and apparatus for receiving three (3)-dimensional image data stream
CA2765570C (en) * 2009-06-16 2016-05-03 Lg Electronics Inc. Broadcast transmitter, broadcast receiver and 3d video processing method thereof
US8948241B2 (en) * 2009-08-07 2015-02-03 Qualcomm Incorporated Signaling characteristics of an MVC operation point
US8976871B2 (en) * 2009-09-16 2015-03-10 Qualcomm Incorporated Media extractor tracks for file format track selection
US9485546B2 (en) * 2010-06-29 2016-11-01 Qualcomm Incorporated Signaling video samples for trick mode video representations
US9131033B2 (en) * 2010-07-20 2015-09-08 Qualcomm Incoporated Providing sequence data sets for streaming video data
US9596447B2 (en) * 2010-07-21 2017-03-14 Qualcomm Incorporated Providing frame packing type information for video coding
TW201210325A (en) * 2010-07-21 2012-03-01 Nokia Corp Method and apparatus for indicating switching points in a streaming session
US20120069927A1 (en) * 2010-09-17 2012-03-22 Intel Corporation CQI feedback mechanisms for distortion-aware link adaptation toward enhanced multimedia communications
KR101831775B1 (ko) * 2010-12-07 2018-02-26 삼성전자주식회사 멀티미디어 컨텐츠를 송수신하는 송신 장치 및 수신 장치와, 그 재생 방법
WO2012177041A2 (ko) * 2011-06-20 2012-12-27 엘지전자 주식회사 미디어 컨텐트 송수신 방법 및 그를 이용한 송수신 장치
US9357275B2 (en) * 2011-09-06 2016-05-31 Qualcomm Incorporated Network streaming of coded video data
US20130195172A1 (en) * 2012-01-30 2013-08-01 Qualcomm Incorporated Method of coding video and storing video content
US9313486B2 (en) * 2012-06-20 2016-04-12 Vidyo, Inc. Hybrid video coding techniques
US20140098883A1 (en) * 2012-10-09 2014-04-10 Nokia Corporation Method and apparatus for video coding
US9774927B2 (en) * 2012-12-21 2017-09-26 Telefonaktiebolaget L M Ericsson (Publ) Multi-layer video stream decoding
CN104885466A (zh) * 2012-12-26 2015-09-02 索尼公司 图像处理装置和方法
KR101466574B1 (ko) * 2013-03-19 2014-12-02 삼성전자주식회사 스케일러블 비디오 부호화 방법 및 장치, 스케일러블 비디오 복호화 방법 및 장치
US9596486B2 (en) * 2013-04-05 2017-03-14 Qualcomm Incorporated IRAP access units and bitstream switching and splicing
US9591321B2 (en) * 2013-04-07 2017-03-07 Dolby International Ab Signaling change in output layer sets
US9602822B2 (en) * 2013-04-17 2017-03-21 Qualcomm Incorporated Indication of cross-layer picture type alignment in multi-layer video coding
US20150016503A1 (en) * 2013-07-15 2015-01-15 Qualcomm Incorporated Tiles and wavefront processing in multi-layer context
GB2516824A (en) * 2013-07-23 2015-02-11 Nokia Corp An apparatus, a method and a computer program for video coding and decoding
US9641862B2 (en) * 2013-10-15 2017-05-02 Nokia Technologies Oy Video encoding and decoding
WO2015082763A1 (en) * 2013-12-02 2015-06-11 Nokia Technologies Oy Video encoding and decoding
US20150264404A1 (en) * 2014-03-17 2015-09-17 Nokia Technologies Oy Method and apparatus for video coding and decoding

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101951506A (zh) * 2010-09-17 2011-01-19 中兴通讯股份有限公司 实现可伸缩视频编码业务同步发送、接收的***和方法
CN102480634A (zh) * 2010-11-24 2012-05-30 中兴通讯股份有限公司 移动多媒体广播中分级业务同步的方法、设备和***
WO2015037373A1 (ja) * 2013-09-12 2015-03-19 ソニー株式会社 符号化装置、送信装置および受信装置
CN104754347A (zh) * 2013-12-26 2015-07-01 中兴通讯股份有限公司 视频图像序号的编码、解码方法及装置、电子设备
CN104754358A (zh) * 2013-12-27 2015-07-01 中兴通讯股份有限公司 码流的生成和处理方法、装置及***

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3306945A4 *

Also Published As

Publication number Publication date
US10582234B2 (en) 2020-03-03
EP3306945A4 (en) 2018-05-23
CN106303673A (zh) 2017-01-04
US20180176617A1 (en) 2018-06-21
CN106303673B (zh) 2021-01-22
EP3306945A1 (en) 2018-04-11

Similar Documents

Publication Publication Date Title
CN108702503B (zh) 用于提供视频比特流的方法及装置
US10171842B2 (en) HRD descriptor and buffer model of data streams for carriage of HEVC extensions
EP3363204B1 (en) Support of random access and switching of layers and sub-layers in multi-layer video files
US9602802B2 (en) Providing frame packing type information for video coding
EP3363205B1 (en) Alignment of operation point sample group in multi-layer bitstreams file format
US10567804B2 (en) Carriage of HEVC extension bitstreams and buffer model with MPEG-2 systems
US10999605B2 (en) Signaling of important video information in file formats
DK2904798T3 (en) VIDEO DATA FILE FORMAT
JP5553945B2 (ja) ビットストリームサブセットの指示
JP2018510595A (ja) 画像コーディング・デコーディングのための装置、方法およびコンピュータ・プログラム
BR112013001332B1 (pt) Prover conjuntos de dados de seqüência para transmissão contínua de dados de vídeo
WO2016192413A1 (zh) 码流对齐、同步处理方法及发送、接收终端和通信***
CN109600616B (zh) 一种基于h.264视频压缩标准的码流封装方法
US10291922B2 (en) Method and apparatus for decoding an enhanced video stream
KR20160149150A (ko) 스테레오스코픽 비디오 데이터를 처리하기 위한 mmt 장치 및 방법
JP6442067B2 (ja) Ｈｅｖｃ拡張の搬送のための動作点のシグナリング
US20130250059A1 (en) Method and apparatus for transmitting stereoscopic video information
CN106507112B (zh) 码流处理方法、装置及***
FI124520B (fi) Menetelmä ja järjestely digitaalisten multimediasignaalien synkronoimiseksi

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16802336

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 15579310

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

WWE Wipo information: entry into national phase

Ref document number: 2016802336

Country of ref document: EP