WO2012106898A1 - Method, device and system for transmitting and processing multi-channel audio-video - Google Patents

Method, device and system for transmitting and processing multi-channel audio-video Download PDF

Info

Publication number
WO2012106898A1
WO2012106898A1 PCT/CN2011/077259 CN2011077259W WO2012106898A1 WO 2012106898 A1 WO2012106898 A1 WO 2012106898A1 CN 2011077259 W CN2011077259 W CN 2011077259W WO 2012106898 A1 WO2012106898 A1 WO 2012106898A1
Authority
WO
WIPO (PCT)
Prior art keywords
video
audio
code stream
channel
composite
Prior art date
Application number
PCT/CN2011/077259
Other languages
French (fr)
Chinese (zh)
Inventor
李昭早
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to PCT/CN2011/077259 priority Critical patent/WO2012106898A1/en
Priority to CN201180001226.8A priority patent/CN102301730B/en
Publication of WO2012106898A1 publication Critical patent/WO2012106898A1/en

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
    • H04N21/2365—Multiplexing of several video streams

Definitions

  • Multi-channel audio and video transmission and processing method device and system
  • Embodiments of the present invention relate to data processing technologies, and in particular, to a multi-channel audio and video transmission and processing method, apparatus, and system. Background technique
  • the common method is to separately process each audio and video separately, from data acquisition, encoding, network transmission, to decoding and output, all of which are controlled by themselves, but are played in the output stage. Synchronous processing, making a stitching at the output for multi-screen output on one display. This scheme makes each channel have good independence and simple control. The failure of one channel does not affect the normal operation of other channels, and the recorded audio and video streams can be conveniently stored for other general-purpose players.
  • Embodiments of the present invention provide a multi-channel audio and video transmission and processing method, apparatus, and system to optimize utilization of network transmission resources.
  • the embodiment of the invention provides a multi-channel audio and video transmission method, including:
  • each audio and video code stream is combined into at least one composite code stream, where each audio and video code stream combined in the composite code stream is correspondingly provided with a code stream identifier for identifying the associated audio and video code stream. ;
  • the embodiment of the invention further provides a multi-channel audio and video processing method, including:
  • the split audio and video streams are processed separately.
  • the embodiment of the invention further provides a multi-channel audio and video transmission device, including:
  • the code stream is obtained by using an ear block for respectively acquiring a multi-channel audio and video code stream;
  • the code stream merging module is configured to combine the audio and video code streams in the at least one composite code stream in the merging process, wherein each audio and video code stream combined in the composite code stream is correspondingly configured to identify the associated sound
  • the code stream identifier of the video code stream
  • a code stream transmission module configured to transmit the composite code stream to an audio/video receiving end according to a network transmission protocol, to indicate that the audio and video receiving end splits the composite code stream into audio and video channels according to the code stream identifier Code stream.
  • the embodiment of the invention further provides a multi-channel audio and video processing device, including:
  • a code stream receiving module configured to receive at least one composite code stream transmitted by the audio and video transmitting end according to a network transmission protocol
  • a code stream splitting module configured to split the composite code stream into audio and video code streams according to a code stream identifier corresponding to each audio and video code stream in a split processing link;
  • the code stream processing module is configured to separately process the split audio and video code streams.
  • the embodiment of the present invention further provides a multi-channel audio and video processing system, including: a multi-channel audio and video transmission device and a multi-channel audio and video processing device according to any embodiment of the present invention.
  • the multi-channel audio and video transmission and processing method, device and system provided by the invention solve the problem of remotely playing multi-channel audio and video, and reduce the audio and video code stream by integrating the multi-channel audio and video code stream into at least one composite code stream. Therefore, when the code stream is transmitted through the network, there is no need to occupy the network channel independently, and some channels are busy and some channels are idle, so that the network resources can be fully utilized to improve the transmission efficiency.
  • FIG. 1 is a flowchart of a multi-channel audio and video transmission method according to Embodiment 1 of the present invention
  • FIG. 2 is a flowchart of a multi-channel audio and video transmission method according to Embodiment 2 of the present invention.
  • FIG. 3A is a schematic diagram of data packet merging in Embodiment 2 of the present invention.
  • FIG. 3B is a schematic structural diagram of a data packet according to Embodiment 2 of the present invention.
  • 3C is a schematic structural diagram of a data message frame header according to Embodiment 2 of the present invention.
  • FIG. 4 is a flowchart of a multi-channel audio and video processing method according to Embodiment 3 of the present invention.
  • FIG. 5 is a flowchart of a multi-channel audio and video processing method according to Embodiment 4 of the present invention.
  • FIG. 6 is a flowchart of a multi-channel audio and video processing method according to Embodiment 5 of the present invention.
  • FIG. 7 is a flow chart showing the processing of multi-channel audio and video in the fifth embodiment of the present invention.
  • FIG. 8 is a schematic structural diagram of a multi-channel audio and video transmission apparatus according to Embodiment 6 of the present invention
  • FIG. 9 is a schematic structural diagram of a multi-channel audio and video transmission apparatus according to Embodiment 7 of the present invention. detailed description
  • FIG. 1 is a flowchart of a multi-channel audio and video transmission method according to Embodiment 1 of the present invention.
  • the method in this embodiment is applicable to a playback application of a multi-channel audio and video code stream, and is particularly suitable for a method performed by an audio-video transmitting end side when a multi-channel audio and video code stream needs to be transmitted through a network, and the following steps can be performed as follows: Step 110: The audio and video transmitting end respectively acquires a multi-channel audio and video code stream;
  • the method for obtaining the audio and video code stream by the audio and video transmitting end may be directly obtained from the camera device, or may be the received audio and video code stream, or the audio and video code stream may be retrieved from the stored data block, and the specific acquisition form is not limit.
  • Step 120 The audio and video transmitting end combines the audio and video code streams in the at least one composite code stream in the merge processing link;
  • each audio and video code stream combined in the composite code stream is correspondingly provided with a code stream identifier for identifying the associated audio and video code stream.
  • the audio and video code stream may be set to multiple processing links as needed before being transmitted through the network, such as camera acquisition, digital-to-analog (AD) conversion, encoding, buffering, and message encapsulation, etc.
  • the merge processing link may be one of them or Several links.
  • the code stream identifier corresponding to each audio and video code stream may be included in the composite code stream, or may be independent of the composite code stream, and can distinguish the audio and video code streams in the composite code stream.
  • Step 1 30 The audio and video transmitting end transmits the composite code stream to the audio and video receiving end according to the network transmission protocol, to indicate that the audio and video receiving end splits the composite code stream into the audio and video code streams according to the code stream identifier.
  • the network transmission protocol applied to transmit the composite code stream may be various existing protocols.
  • the technical solution of the embodiment is to integrate the multi-channel audio and video code streams into at least one composite code stream, and is particularly preferably integrated into one composite code stream, and avoids imbalance of network channel occupation when transmitting through the network. There are no cases where some channels are busy and some channels are idle, so network usage can be optimized and data transmission efficiency can be improved.
  • the technical solution of the embodiment can also improve system performance. When integrated into a small number of composite code streams, the load increased by the system invoking processing resources between multiple audio and video code streams is reduced. At the same time, the technical solution transmits the audio and video code stream by using less or even one composite code stream, so that the network transmission is synchronized, and the multi-channel audio and video is transmitted independently. The problem of synchronous output during the processing.
  • FIG. 2 is a flowchart of a multi-channel audio and video transmission method according to Embodiment 2 of the present invention.
  • the operation of combining the audio and video code streams in the at least one composite code stream is optimized, and specifically, the audio and video transmitting end sets the code stream identifier corresponding to each road sound.
  • Each unit data block of the video code stream is combined and each unit data block is combined into at least one composite code stream.
  • an audio and video code stream is composed of a plurality of unit data blocks, and the unit data block of each processing link may behave differently depending on the processing link.
  • the code stream identifier can be added to each unit data block, so that each unit data block can be split at the audio and video receiving end, and then combined in each processing link.
  • This embodiment provides a preferred solution, specifically, selecting a packet encapsulation link as a merge processing link, and the unit data block processed by the packet encapsulation link is a data packet, and the code stream identifier is correspondingly set in each audio and video code.
  • each unit data block of the stream, and combining the unit data blocks into at least one composite code stream may include the following steps:
  • Step 210 The audio and video transmitting end sets the channel number of each audio and video code stream correspondingly in the frame header of each data packet of each audio and video video stream;
  • Step 220 The audio and video sending end encapsulates each data packet in at least one composite message stream.
  • each data message typically includes a header containing information for several fields.
  • the technical solution of this embodiment utilizes this point, and occupies a field in the frame header of the data packet, and carries a channel number that can identify each audio and video code stream.
  • it may generally correspond to a set number of audio and video code stream collecting devices, and can receive audio and video code streams of several channels, so that channel and video streams can be uniquely distinguished by channel numbers. .
  • each audio and video code stream may also be identified by a sequential unique numbering or the like.
  • FIG. 3A is a schematic diagram of data packet merging according to Embodiment 2 of the present invention.
  • a four-channel audio and video code stream is assumed, which is encoded by an encoder to form a data message in the form of a frame code stream.
  • a header file is usually added to the data packet.
  • FIG. 3A after adding a frame header to each of the data packets of the four-channel audio and video stream, an overall is formed in the cache.
  • Composite stream Each data packet in the composite code stream may also be referred to as a code stream.
  • the first audio and video code stream includes "stream 1 , code stream ⁇ , code stream 1 " as shown in FIG. 3A .
  • the second, third, and fourth audio and video streams similarly include respective code streams.
  • the four-channel audio and video code stream is combined into one composite code stream, and the order of the audio and video code streams in the composite code stream is not limited, and may be combined according to the order of acquisition.
  • Each data packet includes a frame header and a code stream frame data.
  • the frame header is mainly for parsing each code stream frame from the composite code stream.
  • the structure is as shown in FIG. 3A, and preferably includes a frame type identifier and a frame length. And private information, etc.
  • the frame type identifier is not only used to represent the starting position of a frame, but also represents the specific type of the frame.
  • the frame length may be the length of the effective code stream frame data representing the immediately following face. Private information can carry alarm information in the monitoring system, and can carry program information and the like in the field of set-top boxes and the like.
  • the channel number can be used as a byte of a field or as a separate byte in the header.
  • the channel number can be used as a byte in the frame type identifier.
  • the frame type identifier includes four bytes, which are an audio and video frame identifier, a frame type, a specific frame format, and a channel number.
  • the frame type of the video frame is identified as "Vxxx".
  • the first byte “V” indicates the video frame identifier; the second byte indicates the video type, "H” for H264 video, "M” for MPEG4 video; the third byte indicates the video format, taking "0””Represents a video I frame, "1" represents a video P frame; the fourth byte represents the channel number, a value of "0" indicates a 0th channel, and a 1 indicates a 1st channel.
  • the H264 video I frame of the 15th channel is "VH0E” o
  • the frame type of the audio frame is identified as "Axxx".
  • the first byte “A” indicates the audio frame identifier; the second byte indicates the audio type, taking "A” for AMR, “G” for G-series audio, and "P” for PCM, depending on the application.
  • the third byte represents the specific frame format, which is combined with the second byte to represent the specific audio format. It takes “6” for G726 audio, "A” for ADPCM audio, and the fourth byte. Indicates the channel number. For example, if the 10th channel audio format is G71 1 , it is "AG 1A".
  • the content of the frame header is not limited, but the overall principle is to provide an identifier that distinguishes the frame boundary and provide information about the frame type for use by the terminal for parsing.
  • the technical solution of the embodiment utilizes a processing process of packet encapsulation of an existing audio and video code stream before the network transmission, and carries a code stream identifier in the frame header to distinguish each audio and video code stream, thereby realizing the code stream combining. The effect is to minimize the modification of the existing code stream processing method, and adapt to the promotion application.
  • the processing of the audio and video code stream on the transmitting end may include multiple required links.
  • the operations of separately acquiring the multi-channel audio and video code streams in the audio and video transmitting end may include: The device separately collects multiple audio and video code streams; performs digital-to-analog conversion and encoding on the collected multi-channel audio and video code streams to obtain multi-channel audio and video code streams.
  • the actual application is not limited to this.
  • the multi-channel audio and video transmission method has many advantages, such as optimizing network transmission resources and system scheduling resources.
  • other processing can be performed for each audio and video stream and composite stream.
  • each of the audio and video code streams may be separately stored, and/or the composite code stream may be stored. This is equivalent to recording and recording audio and video.
  • the separately recorded audio and video streams can be played independently, and the unified stored composite stream can then be split and played in the parsing device.
  • Embodiment 4 is a flowchart of a multi-channel audio and video processing method according to Embodiment 3 of the present invention.
  • the present embodiment is preferably implemented in conjunction with the multi-channel audio and video transmission method provided by the present invention, and may be specifically implemented by an audio/video receiving end.
  • the method includes the following steps:
  • Step 410 The audio and video receiving end receives at least one composite code stream transmitted by the audio and video transmitting end according to a network transmission protocol.
  • Step 420 The audio and video receiving end splits the composite code stream into audio and video code streams according to the code stream identifier corresponding to each audio and video code stream in the split processing link.
  • Step 430 The audio and video receiving end processes the split audio and video code streams separately.
  • the technical solution of the embodiment splits the composite code stream transmitted by the network, specifically, according to the code stream identifier, and the code stream identifier may be carried in the composite code stream, or may be transmitted independently of the composite code stream, as long as It is possible to split and parse each audio and video code stream.
  • the technical solution of this embodiment is implemented by integrating multiple audio and video code streams into at least one composite code stream.
  • the imbalance of the network channel is avoided, and some channels are busy and some channels are idle, so that the use of network resources can be optimized and the data transmission efficiency can be improved.
  • the technical solution of the embodiment can also improve system performance, and when processing a small amount or even a composite code stream, the load increased by the system to call processing resources between multiple audio and video code streams is reduced.
  • the technical solution transmits the audio and video code stream by using less or even one composite code stream, so that the network transmission is synchronized, and the problem of synchronous output when the multi-channel audio and video transmission processing is solved is solved.
  • FIG. 5 is a flowchart of a multi-channel audio and video processing method according to Embodiment 4 of the present invention.
  • an operation of splitting a composite code stream is specifically optimized, and according to a code stream identifier corresponding to each audio and video code stream,
  • the operation of splitting the composite code stream into the audio and video code streams of each channel may perform the following steps: extracting each unit data block from the composite code stream; and setting the code corresponding to each audio and video code stream carried in each unit data block according to each unit data block
  • the stream identifier merges the extracted unit data blocks into a stream of audio and video streams.
  • the code stream identifier can be added to each unit data block, so that the audio and video receiver can split each unit data block, and then combine the audio and video code streams in subsequent processing steps.
  • This embodiment provides a preferred solution, specifically, selecting a packet parsing link as a split processing link, and the unit data block processed by the packet parsing link is a data packet, and the corresponding data carried in each unit data block is corresponding to
  • combining the extracted unit data blocks into each audio and video code stream may include the following steps:
  • Step 510 The audio and video receiving end identifies a channel number corresponding to each channel of the audio and video code stream carried in each data message frame header;
  • Step 520 The audio and video receiving end combines the extracted data packets into channel audio and video streams according to the channel number.
  • each data message typically includes a frame header containing information for several fields.
  • the technical solution of this embodiment utilizes this point, and occupies a field in the frame header of the data packet, and carries a channel number that can identify each channel of the audio and video code stream.
  • the code stream identifier capable of identifying the audio and video code streams of each channel is not limited to the channel number.
  • the technical solution of the present embodiment utilizes the processing of the existing audio and video code stream after the network transmission, that is, the processing of the packet decapsulation, and the code stream identifier carried in the frame header is used to distinguish the audio and video code streams. It not only realizes the effect of code stream splitting, but also minimizes the modification of the existing code stream processing method, and is suitable for popularization and application.
  • the operations performed on the split audio and video code streams performed by the receiving end may be refined into: decoding and decoding the split audio and video code streams respectively. Cache processing; and splicing each audio and video code stream processed by the cache, and playing on the same display.
  • the processing links in practical applications are not limited to this.
  • the multi-channel audio and video processing method provided by each embodiment of the present invention can process the audio and video code streams independently, and can process the composite code stream uniformly. For example, after receiving at least one composite code stream transmitted by the audio and video transmitting end based on the network transmission protocol, the received composite code stream may also be stored. It is equivalent to recording and recording the composite stream, and then splitting it when it needs to be played.
  • FIG. 6 is a flowchart of a multi-channel audio and video processing method according to Embodiment 5 of the present invention.
  • This embodiment specifically provides a whole process of processing a multi-channel audio and video code stream, as shown in FIG. 7 , which is a fifth embodiment of the present invention.
  • the flow block diagram of the multi-channel audio and video processing, the multi-channel audio and video processing method typically includes the following steps: Step 610: Each camera device in the audio and video transmitting end separately collects multiple audio and video code streams; It can be a camera, or it can be broadly interpreted as a disc that stores audio and video streams, or other devices that can provide audio and video streams.
  • Step 620 The audio and video transmitting end performs digital-to-analog conversion and encoding on the collected multi-channel audio and video code streams to obtain a multi-channel audio and video code stream.
  • a plurality of AD converters may be used to perform digital-to-analog conversion on each audio and video code stream, and then multiple encoders are used to respectively perform audio and video code streams to form a frame code composed of data packets. flow.
  • Step 630 The audio and video transmitting end sets the channel number of each audio and video code stream correspondingly in each channel of audio and video.
  • the frame header of each data packet of the frequency code stream is uniformly sent to the cache for storage;
  • the buffer unit shown in FIG. 7 can complete the operation of the above step 630, adding both a frame header and a buffer, and a sequenced composite code stream is actually formed in the cache.
  • the data packets are cached in a first-in, first-out queue according to the received priority order.
  • Step 640 The audio and video sending end extracts each data packet from the cache, and encapsulates it in at least one composite packet stream, preferably encapsulated in a composite message stream;
  • Step 650 The audio and video transmitting end transmits the composite code stream to the audio and video receiving end according to a network transmission protocol.
  • Step 660 The audio and video receiving end identifies a channel number corresponding to each audio and video code stream carried in each data message frame header in the composite code stream;
  • Step 670 The audio and video receiving end combines the extracted data packets into channel audio and video code streams according to the channel number.
  • the data packet received by the network may be cached, and then the parsing unit performs a packet parsing operation.
  • the frame type identifier is first searched first byte by byte, and then the subsequent frame data is read according to the subsequent frame data length, according to the parsed frame information, such as channel number, audio and video specific format, etc.
  • the code stream can be sent to the decoder of the corresponding channel for decoding and display.
  • Step 680 The audio and video receiving end separately decodes and buffers the split audio and video code streams.
  • the multiple audio and video code streams can be separately obtained from the parsing unit by the plurality of decoding units, decoded independently, and then cached.
  • the time synchronization control can be performed during the caching process so that the subsequent playback pictures are kept in time synchronization.
  • Step 690 The audio and video receiving end splicing the audio and video code streams processed by the buffering and playing in the same display.
  • the technical solution of this embodiment actually provides a technical solution of multiple codecs and single buffer splicing, and the technical solution is more suitable for the general single channel audio and video codec technology.
  • Multiple image sequence The columns respectively generate respective video code streams by the encoder, and the multiple code streams are added to the common buffer by adding the special identifiers of the respective channels to form a multi-way composite code stream.
  • Video recording can be based on this composite stream recording, or it can be based on single-stream encoded stream recording.
  • the network transmission is processed based on the composite code stream, and the network terminal receives the data into the buffer, and parses out each video code stream through the parsing unit, and sends them to the respective decoders to solve the image sequence; after the picture is spliced, it is sent to the display.
  • FIG. 8 is a schematic structural diagram of a multi-channel audio and video transmission apparatus according to Embodiment 6 of the present invention.
  • the apparatus may be specifically integrated in an audio/video transmitting end, and may specifically include a code stream acquiring module 810, a code stream combining module 820, and a code stream transmission. Module 830.
  • the code stream obtaining module 810 is configured to respectively acquire the multi-channel audio and video code streams;
  • the code stream combining module 820 is configured to combine the audio and video code streams in the at least one composite code stream in the merge processing link, where Each audio and video code stream in the composite code stream is correspondingly provided with a code stream identifier for identifying the associated audio and video code stream;
  • the code stream transmission module 830 is configured to transmit the composite code stream to the audio and video receiver according to the network transmission protocol, The audio and video receiving end is instructed to split the composite code stream into each audio and video code stream according to the code stream identifier.
  • the packet encapsulation link is selected as the merge processing link
  • the stream merge module 820 can include: a header add unit 821 and a message encapsulation unit 822.
  • the frame header adding unit 821 is configured to set the channel number of each audio and video code stream correspondingly in the frame header of each data packet of each audio and video code stream; the message encapsulating unit 822 is configured to use each datagram.
  • the text is encapsulated in at least one composite message stream.
  • the technical solution of this embodiment may perform the multi-channel audio and video transmission method provided by any embodiment of the present invention, and has a corresponding functional module.
  • the multi-channel audio and video code stream By integrating the multi-channel audio and video code stream into at least one composite code stream, in particular, it is preferably integrated into A composite code stream avoids the imbalance of the network channel when transmitting through the network, and there is no situation that some channels are busy and some channels are idle, so that the use of network resources can be optimized, and the data transmission efficiency is improved.
  • the technical solution of this embodiment can also improve system performance. When integrated into a small number of composite code streams, the load increased by the system invoking processing resources between multiple audio and video code streams is reduced.
  • FIG. 9 is a schematic structural diagram of a multi-channel audio and video transmission apparatus according to Embodiment 7 of the present invention.
  • the apparatus may be specifically integrated in an audio/video receiving end, and specifically includes a code stream receiving module 910, a code stream splitting module 920, and a code stream. Processing module 930.
  • the code stream receiving module 910 is configured to receive at least one composite code stream transmitted by the audio and video transmitting end according to the network transmission protocol
  • the code stream splitting module 920 is configured to perform, according to each audio and video code stream setting, in the split processing link.
  • the code stream identifier is used to split the composite code stream into respective audio and video code streams; the code stream processing module 930 is configured to separately process the split audio and video code streams.
  • the message parsing step is selected as the split processing link
  • the code stream splitting module 920 preferably includes: a header identifying unit 921 and a stream combining unit 922.
  • the frame header identifying unit 921 is configured to identify a channel number corresponding to each audio and video code stream carried in each data message frame header; the code stream merging unit 922 is configured to extract the datagram according to the channel number. The text is merged into a stream of audio and video streams.
  • the technical solution of this embodiment can implement the multi-channel audio and video processing method provided by any embodiment of the present invention, and has corresponding functional modules.
  • the technical solution of the embodiment is integrated into the multi-channel audio and video code stream of at least one composite code stream, and avoids the imbalance of the network channel when transmitting through the network, and some channels are not busy and some channels are idle. In this case, it is possible to optimize the use of network resources and improve data transmission efficiency.
  • the technical solution of the embodiment can also improve the performance of the system. When processing a small number or even a composite code stream, the load increased by the system invoking processing resources between the multiple audio and video code streams is reduced.
  • the embodiment of the present invention further provides a multi-channel audio and video processing system, including the multi-channel audio and video transmission device and the multi-channel audio and video processing device provided by any embodiment of the present invention.
  • the technical solution of the embodiments of the present invention solves the problem of remotely playing multi-channel audio and video.
  • the technical solution of the embodiment of the present invention reduces the number of audio and video code streams by integration. Therefore, when the code stream is transmitted through the network, there is no need to occupy the network channel independently, and some channels are busy and some channels are idle, so that the channel stream can be fully utilized.
  • audio and video streams have many processing links, such as digital-to-analog conversion, codec, etc.
  • the technical solution of the embodiment of the present invention does not excessively increase the load added by the system when scheduling processing resources in each processing link, and the processing resource balance can be more easily realized.
  • the technical solution of the embodiment of the present invention can uniformly balance the data transmission rate by using the unified transmission of the composite code stream, and reduce the phenomenon that the data transmitted after the data to be transmitted is transmitted and delayed.
  • the foregoing storage medium includes: a medium that can store program codes, such as a ROM, a RAM, a magnetic disk, or an optical disk.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)

Abstract

Disclosed in the present invention are a method, a device and a system for transmitting and processing multi-channel AV (audio-video). The method comprises the following steps: obtaining respectively multi-channel AV code streams; in a merging processing step, merging each channel AV code stream into at least one channel composite code stream, wherein each channel AV code stream merged in the composite code stream is correspondingly provided with a code stream identification used for identifying the AV code stream belonged to; and transmitting the composite code stream to an AV receiving terminal based on a network transport protocol for indicating the AV receiving terminal to divide the composite code stream into each channel AV code stream according to the code stream identification. In the present invention, the multi-channel AV code streams are integrated into at least one channel composite code stream so as to reduce the number of the AV code streams; thus when the code stream is transmitted by the network, the network channel is no need to be occupied independently, and the phenomenon that some channels are busy and some channels are free will not occur, thereby taking fully advantage of network resource and raising transmitting efficiency.

Description

多路音视频传输和处理方法、 装置及*** 技术领域  Multi-channel audio and video transmission and processing method, device and system
本发明实施例涉及数据处理技术， 尤其涉及一种多路音视频传输和处理 方法、 装置及***。 背景技术  Embodiments of the present invention relate to data processing technologies, and in particular, to a multi-channel audio and video transmission and processing method, apparatus, and system. Background technique
随着流媒体技术的发展， 多路音视频技术得到广泛应用。 特别在视频监 控领域， 人们通过多路视频监控***， 对管辖范围进行统一安防监控。 随着 集成化程度的发展， 视频的路数越来越多， 对编解码能力、 网传能力与整套 ***性能都提出了挑战； 且随着路数的增多， 多路同步播放也成为当前的一 大难题。  With the development of streaming media technology, multi-channel audio and video technology has been widely used. Especially in the field of video surveillance, people use a multi-channel video surveillance system to conduct unified security monitoring of jurisdictions. With the development of integration, the number of video channels is increasing, which challenges the codec, network transmission capability and overall system performance. With the increase of the number of channels, multi-channel synchronous playback has become the current A big problem.
现有技术在实现多路同步播放时， 常用方法是将各路音视频各自单独处 理， 从数据采集、 编码、 网络传输、 再到解码以及输出， 都是独自控制， 只 是在输出阶段进行播放的同步处理， 在输出时做一个拼接以供多画面在一个 显示器上输出。 该方案使得各个通道具有良好的独立性， 控制简单。 某一个 通道出故障不影响其他通道的正常运行， 且独立存储记录下来的音视频码流 可以方便地供其他通用播放器播放。  In the prior art, when implementing multi-channel synchronous playback, the common method is to separately process each audio and video separately, from data acquisition, encoding, network transmission, to decoding and output, all of which are controlled by themselves, but are played in the output stage. Synchronous processing, making a stitching at the output for multi-screen output on one display. This scheme makes each channel have good independence and simple control. The failure of one channel does not affect the normal operation of other channels, and the recorded audio and video streams can be conveniently stored for other general-purpose players.
但是， 在实现本发明的过程中， 发明人发现现有技术至少存在如下缺陷： 网络传输需要占用多个通道独立地传输各路音视频码流， 浪费网络资源。 发明内容  However, in the process of implementing the present invention, the inventors have found that the prior art has at least the following drawbacks: The network transmission needs to occupy multiple channels to independently transmit audio and video code streams, and waste network resources. Summary of the invention
本发明实施例提供一种多路音视频传输和处理方法、 装置及***， 以优 化网络传输资源的利用。  Embodiments of the present invention provide a multi-channel audio and video transmission and processing method, apparatus, and system to optimize utilization of network transmission resources.
本发明实施例提供了一种多路音视频传输方法， 包括：  The embodiment of the invention provides a multi-channel audio and video transmission method, including:
分别获取多路音视频码流； 在合并处理环节， 将各路音视频码流合并在至少一路复合码流中， 其中， 合并在复合码流中的每路音视频码流对应设置有用于标识所属音视频码流的 码流标识； Obtaining multiple audio and video streams respectively; In the merging process, each audio and video code stream is combined into at least one composite code stream, where each audio and video code stream combined in the composite code stream is correspondingly provided with a code stream identifier for identifying the associated audio and video code stream. ;
基于网络传输协议向音视频接收端传输所述复合码流， 以指示所述音视 频接收端根据所述码流标识将所述复合码流拆分成为各路音视频码流。  Transmitting the composite code stream to the audio and video receiver according to the network transmission protocol, to instruct the audio and video receiver to split the composite code stream into audio and video code streams according to the code stream identifier.
本发明实施例还提供了一种多路音视频处理方法， 包括：  The embodiment of the invention further provides a multi-channel audio and video processing method, including:
基于网络传输协议接收音视频发送端传输的至少一路复合码流； 在拆分处理环节， 根据对应于每路音视频码流设置的码流标识， 将所述 复合码流拆分成为各路音视频码流；  Receiving at least one composite code stream transmitted by the audio and video transmitting end according to the network transmission protocol; in the splitting processing step, splitting the composite code stream into respective road sounds according to the code stream identifier corresponding to each audio and video code stream set Video stream
对拆分后的各路音视频码流分别进行处理。  The split audio and video streams are processed separately.
本发明实施例还提供了一种多路音视频传输装置， 包括：  The embodiment of the invention further provides a multi-channel audio and video transmission device, including:
码流获耳 莫块， 用于分别获取多路音视频码流；  The code stream is obtained by using an ear block for respectively acquiring a multi-channel audio and video code stream;
码流合并模块， 用于在合并处理环节， 将各路音视频码流合并在至少一 路复合码流中， 其中， 合并在复合码流中的每路音视频码流对应设置有用于 标识所属音视频码流的码流标识；  The code stream merging module is configured to combine the audio and video code streams in the at least one composite code stream in the merging process, wherein each audio and video code stream combined in the composite code stream is correspondingly configured to identify the associated sound The code stream identifier of the video code stream;
码流传输模块， 用于基于网络传输协议向音视频接收端传输所述复合码 流， 以指示所述音视频接收端根据所述码流标识将所述复合码流拆分成为各 路音视频码流。  a code stream transmission module, configured to transmit the composite code stream to an audio/video receiving end according to a network transmission protocol, to indicate that the audio and video receiving end splits the composite code stream into audio and video channels according to the code stream identifier Code stream.
本发明实施例还提供了一种多路音视频处理装置， 包括：  The embodiment of the invention further provides a multi-channel audio and video processing device, including:
码流接收模块， 用于基于网络传输协议接收音视频发送端传输的至少一 路复合码流；  a code stream receiving module, configured to receive at least one composite code stream transmitted by the audio and video transmitting end according to a network transmission protocol;
码流拆分模块， 用于在拆分处理环节， 根据对应于每路音视频码流设置 的码流标识， 将所述复合码流拆分成为各路音视频码流；  a code stream splitting module, configured to split the composite code stream into audio and video code streams according to a code stream identifier corresponding to each audio and video code stream in a split processing link;
码流处理模块， 用于对拆分后的各路音视频码流分别进行处理。  The code stream processing module is configured to separately process the split audio and video code streams.
本发明实施例还提供了一种多路音视频处理***， 包括： 本发明任意实 施例所提供的多路音视频传输装置以及多路音视频处理装置。 本发明提供的多路音视频传输和处理方法、 装置及***， 解决了远程播 放多路音视频的问题， 通过将多路音视频码流整合为至少一路复合码流， 减 少了音视频码流的数量， 因此在通过网络传输码流时， 无需独立占用网络通 道而出现有些通道忙碌有些通道空闲的现象， 因而能够充分地利用网络资源， 提高传输效率。 附图说明 The embodiment of the present invention further provides a multi-channel audio and video processing system, including: a multi-channel audio and video transmission device and a multi-channel audio and video processing device according to any embodiment of the present invention. The multi-channel audio and video transmission and processing method, device and system provided by the invention solve the problem of remotely playing multi-channel audio and video, and reduce the audio and video code stream by integrating the multi-channel audio and video code stream into at least one composite code stream. Therefore, when the code stream is transmitted through the network, there is no need to occupy the network channel independently, and some channels are busy and some channels are idle, so that the network resources can be fully utilized to improve the transmission efficiency. DRAWINGS
图 1为本发明实施例一提供的多路音视频传输方法的流程图；  1 is a flowchart of a multi-channel audio and video transmission method according to Embodiment 1 of the present invention;
图 2为本发明实施例二提供的多路音视频传输方法的流程图；  2 is a flowchart of a multi-channel audio and video transmission method according to Embodiment 2 of the present invention;
图 3A为本发明实施例二中进行数据报文合并的示意图；  3A is a schematic diagram of data packet merging in Embodiment 2 of the present invention;
图 3B为本发明实施例二中数据报文的结构示意图；  3B is a schematic structural diagram of a data packet according to Embodiment 2 of the present invention;
图 3C为本发明实施例二中数据报文帧头的结构示意图；  3C is a schematic structural diagram of a data message frame header according to Embodiment 2 of the present invention;
图 4为本发明实施例三提供的多路音视频处理方法的流程图；  4 is a flowchart of a multi-channel audio and video processing method according to Embodiment 3 of the present invention;
图 5为本发明实施例四提供的多路音视频处理方法的流程图；  FIG. 5 is a flowchart of a multi-channel audio and video processing method according to Embodiment 4 of the present invention; FIG.
图 6为本发明实施例五提供的多路音视频处理方法的流程图；  6 is a flowchart of a multi-channel audio and video processing method according to Embodiment 5 of the present invention;
图 7为本发明实施例五中多路音视频处理的流程框图；  7 is a flow chart showing the processing of multi-channel audio and video in the fifth embodiment of the present invention;
图 8为本发明实施例六提供的多路音视频传输装置的结构示意图； 图 9为本发明实施例七提供的多路音视频传输装置的结构示意图。 具体实施方式  FIG. 8 is a schematic structural diagram of a multi-channel audio and video transmission apparatus according to Embodiment 6 of the present invention; FIG. 9 is a schematic structural diagram of a multi-channel audio and video transmission apparatus according to Embodiment 7 of the present invention. detailed description
为使本发明实施例的目的、 技术方案和优点更加清楚， 下面将结合本发 明实施例中的附图， 对本发明实施例中的技术方案进行清楚、 完整地描述， 显然， 所描述的实施例是本发明一部分实施例， 而不是全部的实施例。 基于 本发明中的实施例， 本领域普通技术人员在没有作出创造性劳动前提下所获 得的所有其他实施例， 都属于本发明保护的范围。  The technical solutions in the embodiments of the present invention are clearly and completely described in the following with reference to the accompanying drawings in the embodiments of the present invention. It is a partial embodiment of the invention, and not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts are within the scope of the present invention.
实施例一 图 1为本发明实施例一提供的多路音视频传输方法的流程图。本实施例的 方法适用于多路音视频码流的播放应用中， 尤其适用于多路音视频码流需要 通过网络传输时， 由音视频发送端侧所执行的方法， 具体可执行如下步骤： 步骤 110、 音视频发送端分别获取多路音视频码流； Embodiment 1 FIG. 1 is a flowchart of a multi-channel audio and video transmission method according to Embodiment 1 of the present invention. The method in this embodiment is applicable to a playback application of a multi-channel audio and video code stream, and is particularly suitable for a method performed by an audio-video transmitting end side when a multi-channel audio and video code stream needs to be transmitted through a network, and the following steps can be performed as follows: Step 110: The audio and video transmitting end respectively acquires a multi-channel audio and video code stream;
上述步骤中音视频发送端获取音视频码流的方式可以是从摄像装置中直 接获取， 也可以是接收音视频码流， 或者从存储的数据块中调取音视频码流， 具体获取形式不限。  In the above step, the method for obtaining the audio and video code stream by the audio and video transmitting end may be directly obtained from the camera device, or may be the received audio and video code stream, or the audio and video code stream may be retrieved from the stored data block, and the specific acquisition form is not limit.
步骤 120、 音视频发送端在合并处理环节， 将各路音视频码流合并在至少 一路复合码流中；  Step 120: The audio and video transmitting end combines the audio and video code streams in the at least one composite code stream in the merge processing link;
在步骤 120中，合并在复合码流中的每路音视频码流对应设置有用于标识 所属音视频码流的码流标识。 由于音视频码流在通过网络传输之前可能根据 需要设置多个处理环节， 例如摄像采集、 数模（AD )转换、 编码、 緩存和报 文封装环节等， 所以合并处理环节可以是其中的一个或几个环节。 对应于每 路音视频码流设置的码流标识可以包含在复合码流中， 也可以独立于复合码 流， 能够起到区分复合码流中各路音视频码流即可。  In step 120, each audio and video code stream combined in the composite code stream is correspondingly provided with a code stream identifier for identifying the associated audio and video code stream. Since the audio and video code stream may be set to multiple processing links as needed before being transmitted through the network, such as camera acquisition, digital-to-analog (AD) conversion, encoding, buffering, and message encapsulation, etc., the merge processing link may be one of them or Several links. The code stream identifier corresponding to each audio and video code stream may be included in the composite code stream, or may be independent of the composite code stream, and can distinguish the audio and video code streams in the composite code stream.
步骤 1 30、音视频发送端基于网络传输协议向音视频接收端传输该复合码 流，以指示音视频接收端根据码流标识将复合码流拆分成为各路音视频码流。  Step 1 30: The audio and video transmitting end transmits the composite code stream to the audio and video receiving end according to the network transmission protocol, to indicate that the audio and video receiving end splits the composite code stream into the audio and video code streams according to the code stream identifier.
上述步骤中， 传输复合码流所应用的网络传输协议可以为已有的各种协 议。  In the above steps, the network transmission protocol applied to transmit the composite code stream may be various existing protocols.
本实施例的技术方案通过将多路音视频码流整合为至少一路复合码流， 特别是优选整合为一路复合码流， 在通过网络进行传输时， 避免了对网络通 道占用的不均衡性， 不会出现有些通道忙碌而有些通道空闲的情况， 因而能 够优化网络资源的使用， 提高数据传输效率。 本实施例的技术方案， 还能够 提升***性能， 当整合为少量复合码流时， 减少了***在多路音视频码流之 间调用处理资源而增加的负荷。 同时， 该技术方案由于采用较少甚至是一条 复合码流传输音视频码流， 因而使得网络传输同步， 解决多路音视频独自传 输处理时的同步输出的问题。 The technical solution of the embodiment is to integrate the multi-channel audio and video code streams into at least one composite code stream, and is particularly preferably integrated into one composite code stream, and avoids imbalance of network channel occupation when transmitting through the network. There are no cases where some channels are busy and some channels are idle, so network usage can be optimized and data transmission efficiency can be improved. The technical solution of the embodiment can also improve system performance. When integrated into a small number of composite code streams, the load increased by the system invoking processing resources between multiple audio and video code streams is reduced. At the same time, the technical solution transmits the audio and video code stream by using less or even one composite code stream, so that the network transmission is synchronized, and the multi-channel audio and video is transmitted independently. The problem of synchronous output during the processing.
实施例二  Embodiment 2
图 2为本发明实施例二提供的多路音视频传输方法的流程图。本实施例可 以以上述实施例为基础， 对将各路音视频码流合并在至少一路复合码流中的 操作进行了优化， 具体可以是音视频发送端将码流标识对应设置在各路音视 频码流的每个单位数据块中，并将各单位数据块合并成为至少一路复合码流。  FIG. 2 is a flowchart of a multi-channel audio and video transmission method according to Embodiment 2 of the present invention. In this embodiment, based on the foregoing embodiment, the operation of combining the audio and video code streams in the at least one composite code stream is optimized, and specifically, the audio and video transmitting end sets the code stream identifier corresponding to each road sound. Each unit data block of the video code stream is combined and each unit data block is combined into at least one composite code stream.
通常， 音视频码流由多个单位数据块组成， 根据处理环节的不同， 每个 处理环节的单位数据块的表现形式会有不同。 本实施例可以将码流标识添加 到每个单位数据块中， 从而在音视频接收端能够拆分各个单位数据块， 而后 再在各处理环节进行组合。 本实施例提供了一优选方案， 具体是选择报文封 装环节作为合并处理环节，则报文封装环节所处理的单位数据块是数据报文， 则将码流标识对应设置在各路音视频码流的每个单位数据块中， 并将各单位 数据块合并成为至少一路复合码流可包括如下步骤：  Generally, an audio and video code stream is composed of a plurality of unit data blocks, and the unit data block of each processing link may behave differently depending on the processing link. In this embodiment, the code stream identifier can be added to each unit data block, so that each unit data block can be split at the audio and video receiving end, and then combined in each processing link. This embodiment provides a preferred solution, specifically, selecting a packet encapsulation link as a merge processing link, and the unit data block processed by the packet encapsulation link is a data packet, and the code stream identifier is correspondingly set in each audio and video code. In each unit data block of the stream, and combining the unit data blocks into at least one composite code stream may include the following steps:
步骤 210、音视频发送端将各路音视频码流的通道号对应设置在各路音视 频码流的每个数据报文的帧头中；  Step 210: The audio and video transmitting end sets the channel number of each audio and video code stream correspondingly in the frame header of each data packet of each audio and video video stream;
步骤 220、 音视频发送端将各数据报文封装在至少一路复合报文流中。 在实际应用中， 每个数据报文典型地都包括一个帧头， 其中包含若干字 段的信息。 本实施例的技术方案即利用此点， 在数据报文的帧头中占用一个 字段， 携带能够标识各路音视频码流的通道号。 对于某一个音视频发送端而 言， 其通常可能对应着设定数量的音视频码流采集装置， 能够接收数个通道 的音视频码流， 因此可以通过通道号唯一区分各路音视频码流。 或者， 在通 常的音视频发送端， 也可以用顺序唯一编号等方式来标识各路音视频码流。  Step 220: The audio and video sending end encapsulates each data packet in at least one composite message stream. In practical applications, each data message typically includes a header containing information for several fields. The technical solution of this embodiment utilizes this point, and occupies a field in the frame header of the data packet, and carries a channel number that can identify each audio and video code stream. For a certain audio and video transmitting end, it may generally correspond to a set number of audio and video code stream collecting devices, and can receive audio and video code streams of several channels, so that channel and video streams can be uniquely distinguished by channel numbers. . Alternatively, in the normal audio and video transmitting end, each audio and video code stream may also be identified by a sequential unique numbering or the like.
图 3A为本发明实施例二中进行数据报文合并的示意图， 如图 3A所示， 假设有四路音视频码流， 其通过编码器进行编码后形成帧码流形式的数据报 文。 在进行报文封装时， 通常会为数据报文添加帧头文件， 如图 3A所示， 为 四路音视频码流的数据报文各自添加帧头之后， 在緩存中组成了一个整体的 复合码流。 复合码流中的每个数据报文， 又可称为码流， 第一路音视频码流 如图 3A所示包括 "码流 1、 码流 Γ、 码流 1 " ... ...，，， 第二路、 第三路和第四 路音视频码流类似包括各自的码流。四路音视频码流复合成为一路复合码流， 且各路音视频码流在复合码流中的排列顺序不限， 可以按照获取的先后顺序 合并。 FIG. 3A is a schematic diagram of data packet merging according to Embodiment 2 of the present invention. As shown in FIG. 3A, a four-channel audio and video code stream is assumed, which is encoded by an encoder to form a data message in the form of a frame code stream. When packet encapsulation is performed, a header file is usually added to the data packet. As shown in FIG. 3A, after adding a frame header to each of the data packets of the four-channel audio and video stream, an overall is formed in the cache. Composite stream. Each data packet in the composite code stream may also be referred to as a code stream. The first audio and video code stream includes "stream 1 , code stream 码 , code stream 1 " as shown in FIG. 3A . , , , The second, third, and fourth audio and video streams similarly include respective code streams. The four-channel audio and video code stream is combined into one composite code stream, and the order of the audio and video code streams in the composite code stream is not limited, and may be combined according to the order of acquisition.
每个数据报文均包括帧头和码流帧数据， 帧头主要是方便从复合码流中 解析出各个码流帧， 其结构如图 3Β所示， 优选是可包括帧类型标识、 帧长度 和私有信息等。 其中帧类型标识不仅用来代表一个帧的起始位置， 而且还代 表该帧具体的类型。 帧长度可以为代表紧随后面的有效码流帧数据的长度。 私有信息在监控***中可以携带报警信息， 在机顶盒等领域中， 可以携带节 目信息等。 通道号可以作为某个字段的一个字节也可以作为帧头中的独立字 节。  Each data packet includes a frame header and a code stream frame data. The frame header is mainly for parsing each code stream frame from the composite code stream. The structure is as shown in FIG. 3A, and preferably includes a frame type identifier and a frame length. And private information, etc. The frame type identifier is not only used to represent the starting position of a frame, but also represents the specific type of the frame. The frame length may be the length of the effective code stream frame data representing the immediately following face. Private information can carry alarm information in the monitoring system, and can carry program information and the like in the field of set-top boxes and the like. The channel number can be used as a byte of a field or as a separate byte in the header.
例如， 通道号可作为帧类型标识中的一个字节， 如图 3C所示， 帧类型标 识包括四个字节， 分别为音视频帧标识、 帧类型、 具体帧格式和通道号。 如 视频帧的帧类型标识为 "Vxxx"。 第一个字节 "V" 表示视频帧标识； 第二个 字节表示视频类型， 取 "H" 代表 H264视频， "M" 代表 MPEG4视频； 第三个 字节表示视频具体格式， 取 " 0" 代表视频 I帧， " 1 " 代表视频 P帧； 第四个 字节表示通道号， 取值为 " 0" 表示第 0通道、 为 1表示第 1通道。 比如第 1 5通道的 H264视频 I帧， 则为 "VH0E" o 音频帧的帧类型标识为 "Axxx"。 第一个字节 "A" 表示音频帧标识； 第二个字节表示音频类型， 取 "A" 表示 AMR , 取 "G" 表示 G系列音频， 取 "P" 表示 PCM, 还可根据应用进行相应扩 充； 第三个字节表示具体帧格式， 它跟第二个字节搭配起来代表具体的音频 格式， 它取 "6" 表示 G726音频， 取 "A" 表示 ADPCM音频； 第四个字节表示 通道号。 比如第 1 0通道音频格式为 G71 1 , 则为 "AG 1A"。 For example, the channel number can be used as a byte in the frame type identifier. As shown in FIG. 3C, the frame type identifier includes four bytes, which are an audio and video frame identifier, a frame type, a specific frame format, and a channel number. For example, the frame type of the video frame is identified as "Vxxx". The first byte "V" indicates the video frame identifier; the second byte indicates the video type, "H" for H264 video, "M" for MPEG4 video; the third byte indicates the video format, taking "0""Represents a video I frame, "1" represents a video P frame; the fourth byte represents the channel number, a value of "0" indicates a 0th channel, and a 1 indicates a 1st channel. For example, the H264 video I frame of the 15th channel is "VH0E" o The frame type of the audio frame is identified as "Axxx". The first byte "A" indicates the audio frame identifier; the second byte indicates the audio type, taking "A" for AMR, "G" for G-series audio, and "P" for PCM, depending on the application. The third byte represents the specific frame format, which is combined with the second byte to represent the specific audio format. It takes "6" for G726 audio, "A" for ADPCM audio, and the fourth byte. Indicates the channel number. For example, if the 10th channel audio format is G71 1 , it is "AG 1A".
当然， 帧头中对放置的内容不限制， 但总体原则是要求提供区分帧边界 的标识、 提供帧类型的信息， 以供终端解析使用。 本实施例的技术方案利用了已有音视频码流在网络传输之前进行报文封 装的处理环节， 在帧头中携带码流标识来区分各路音视频码流， 既实现了码 流合并的效果， 又尽量减少了对已***流处理方法的改动， 适应于推广应用。 Of course, the content of the frame header is not limited, but the overall principle is to provide an identifier that distinguishes the frame boundary and provide information about the frame type for use by the terminal for parsing. The technical solution of the embodiment utilizes a processing process of packet encapsulation of an existing audio and video code stream before the network transmission, and carries a code stream identifier in the frame header to distinguish each audio and video code stream, thereby realizing the code stream combining. The effect is to minimize the modification of the existing code stream processing method, and adapt to the promotion application.
在上述各实施例的基础上， 音视频码流在发送端的处理过程可包括多个 所需环节， 典型地在音视频发送端中分别获取多路音视频码流的操作可以包 括： 从各摄像装置中分别采集多路音视频码流； 对采集到的多路音视频码流 分别进行数模转换和编码， 以获取多路音视频码流。 实际应用中并不限于此。  On the basis of the foregoing embodiments, the processing of the audio and video code stream on the transmitting end may include multiple required links. The operations of separately acquiring the multi-channel audio and video code streams in the audio and video transmitting end may include: The device separately collects multiple audio and video code streams; performs digital-to-analog conversion and encoding on the collected multi-channel audio and video code streams to obtain multi-channel audio and video code streams. The actual application is not limited to this.
本发明实施例所提供的多路音视频传输方法具有诸多优势， 例如优化了 网络传输资源和***调度资源等。 此外， 对各路音视频码流和复合码流都可 以进行其他处理。例如在将各路音视频码流合并在至少一路复合码流中之后 , 还可以分别存储各路音视频码流， 和 /或存储复合码流。 即相当于对音视频进 行录像记录操作。 单独记录的音视频码流可以独立播放， 统一存储的复合码 流也可以随后在解析装置中进行拆分播放。  The multi-channel audio and video transmission method provided by the embodiments of the present invention has many advantages, such as optimizing network transmission resources and system scheduling resources. In addition, other processing can be performed for each audio and video stream and composite stream. For example, after combining the audio and video code streams in at least one composite code stream, each of the audio and video code streams may be separately stored, and/or the composite code stream may be stored. This is equivalent to recording and recording audio and video. The separately recorded audio and video streams can be played independently, and the unified stored composite stream can then be split and played in the parsing device.
实施例三  Embodiment 3
图 4为本发明实施例三提供的多路音视频处理方法的流程图，本实施例优 选与本发明提供的多路音视频传输方法配合执行， 且具体可以由音视频接收 端来执行， 该方法包括如下步骤：  4 is a flowchart of a multi-channel audio and video processing method according to Embodiment 3 of the present invention. The present embodiment is preferably implemented in conjunction with the multi-channel audio and video transmission method provided by the present invention, and may be specifically implemented by an audio/video receiving end. The method includes the following steps:
步骤 410、音视频接收端基于网络传输协议接收音视频发送端传输的至少 一路复合码流；  Step 410: The audio and video receiving end receives at least one composite code stream transmitted by the audio and video transmitting end according to a network transmission protocol.
步骤 420、 音视频接收端在拆分处理环节， 根据对应于每路音视频码流设 置的码流标识， 将复合码流拆分成为各路音视频码流；  Step 420: The audio and video receiving end splits the composite code stream into audio and video code streams according to the code stream identifier corresponding to each audio and video code stream in the split processing link.
步骤 430、 音视频接收端对拆分后的各路音视频码流分别进行处理。  Step 430: The audio and video receiving end processes the split audio and video code streams separately.
本实施例的技术方案对经网络传输的复合码流进行拆分处理， 具体是依 据码流标识进行拆分， 码流标识可以携带在复合码流中， 也可以独立于复合 码流传输， 只要能将各路音视频码流进行拆分解析即可。  The technical solution of the embodiment splits the composite code stream transmitted by the network, specifically, according to the code stream identifier, and the code stream identifier may be carried in the composite code stream, or may be transmitted independently of the composite code stream, as long as It is possible to split and parse each audio and video code stream.
本实施例的技术方案通过整合为至少一路复合码流的多路音视频码流， 在通过网络进行传输时， 避免了对网络通道占用的不均衡性， 不会出现有些 通道忙碌而有些通道空闲的情况， 因而能够优化网络资源的使用， 提高数据 传输效率。 本实施例的技术方案， 还能够提升***性能， 当处理少量甚至一 条复合码流时， 减少了***在多路音视频码流之间调用处理资源而增加的负 荷。 同时， 该技术方案由于采用较少甚至是一条复合码流传输音视频码流， 因而使得网络传输同步，解决多路音视频独自传输处理时的同步输出的问题。 The technical solution of this embodiment is implemented by integrating multiple audio and video code streams into at least one composite code stream. When transmitting over the network, the imbalance of the network channel is avoided, and some channels are busy and some channels are idle, so that the use of network resources can be optimized and the data transmission efficiency can be improved. The technical solution of the embodiment can also improve system performance, and when processing a small amount or even a composite code stream, the load increased by the system to call processing resources between multiple audio and video code streams is reduced. At the same time, the technical solution transmits the audio and video code stream by using less or even one composite code stream, so that the network transmission is synchronized, and the problem of synchronous output when the multi-channel audio and video transmission processing is solved is solved.
实施例四  Embodiment 4
图 5为本发明实施例四提供的多路音视频处理方法的流程图，本实施例具 体优化了拆分复合码流的操作，根据对应于每路音视频码流设置的码流标识， 将复合码流拆分成为各路音视频码流的操作可以执行如下步骤： 从复合码流 中提取各单位数据块； 根据每个单位数据块中携带的对应于每路音视频码流 设置的码流标识， 将提取的单位数据块合并成为各路音视频码流。  FIG. 5 is a flowchart of a multi-channel audio and video processing method according to Embodiment 4 of the present invention. In this embodiment, an operation of splitting a composite code stream is specifically optimized, and according to a code stream identifier corresponding to each audio and video code stream, The operation of splitting the composite code stream into the audio and video code streams of each channel may perform the following steps: extracting each unit data block from the composite code stream; and setting the code corresponding to each audio and video code stream carried in each unit data block according to each unit data block The stream identifier merges the extracted unit data blocks into a stream of audio and video streams.
本实施例可以将码流标识添加到每个单位数据块中， 从而音视频接收端 能够拆分各个单位数据块， 而后再在后续处理环节进行各路音视频码流的组 合。 本实施例提供了一优选方案， 具体是选择报文解析环节作为拆分处理环 节， 则报文解析环节所处理的单位数据块是数据报文， 则根据每个单位数据 块中携带的对应于每路音视频码流设置的码流标识， 将提取的单位数据块合 并成为各路音视频码流可包括如下步骤：  In this embodiment, the code stream identifier can be added to each unit data block, so that the audio and video receiver can split each unit data block, and then combine the audio and video code streams in subsequent processing steps. This embodiment provides a preferred solution, specifically, selecting a packet parsing link as a split processing link, and the unit data block processed by the packet parsing link is a data packet, and the corresponding data carried in each unit data block is corresponding to For the code stream identifier set by each audio and video code stream, combining the extracted unit data blocks into each audio and video code stream may include the following steps:
步骤 510、音视频接收端识别每个数据报文帧头中携带的对应于每路音视 频码流设置的通道号；  Step 510: The audio and video receiving end identifies a channel number corresponding to each channel of the audio and video code stream carried in each data message frame header;
步骤 520、音视频接收端根据通道号将提取的数据报文合并成为各路音视 频码流。  Step 520: The audio and video receiving end combines the extracted data packets into channel audio and video streams according to the channel number.
在实际应用中， 每个数据报文典型地都包括一个帧头， 其中包含若干字 段的信息。 本实施例的技术方案即利用此点， 在数据报文的帧头中占用一个 字段， 携带能够标识各路音视频码流的通道号。 当然， 能够标识区分各路音 视频码流的码流标识并不限于通道号。 本实施例的技术方案利用了已有音视频码流在网络传输之后进行报文解 析， 即报文解封装的处理环节， 以帧头中携带的码流标识来区分各路音视频 码流， 既实现了码流拆分的效果， 又尽量减少了对已***流处理方法的改动， 适应于推广应用。 In practical applications, each data message typically includes a frame header containing information for several fields. The technical solution of this embodiment utilizes this point, and occupies a field in the frame header of the data packet, and carries a channel number that can identify each channel of the audio and video code stream. Of course, the code stream identifier capable of identifying the audio and video code streams of each channel is not limited to the channel number. The technical solution of the present embodiment utilizes the processing of the existing audio and video code stream after the network transmission, that is, the processing of the packet decapsulation, and the code stream identifier carried in the frame header is used to distinguish the audio and video code streams. It not only realizes the effect of code stream splitting, but also minimizes the modification of the existing code stream processing method, and is suitable for popularization and application.
在上述各实施例的基础上， 接收端所执行的对拆分后的各路音视频码流 分别进行处理的操作可以细化为： 对拆分后的各路音视频码流分别进行解码 和緩存处理； 并将緩存处理的各路音视频码流进行画面拼接， 在同一台显示 器中进行播放。 当然实际应用中的处理环节并不限于此。  On the basis of the foregoing embodiments, the operations performed on the split audio and video code streams performed by the receiving end may be refined into: decoding and decoding the split audio and video code streams respectively. Cache processing; and splicing each audio and video code stream processed by the cache, and playing on the same display. Of course, the processing links in practical applications are not limited to this.
本发明各实施例所提供的多路音视频处理方法， 既能够独立地对各路音 视频码流进行处理， 也能够统一对复合码流进行处理。 例如， 基于网络传输 协议接收音视频发送端传输的至少一路复合码流之后， 还可以对接收到的复 合码流进行存储。 相当于对复合码流进行录像记录， 在需要播放时再拆分即 可。  The multi-channel audio and video processing method provided by each embodiment of the present invention can process the audio and video code streams independently, and can process the composite code stream uniformly. For example, after receiving at least one composite code stream transmitted by the audio and video transmitting end based on the network transmission protocol, the received composite code stream may also be stored. It is equivalent to recording and recording the composite stream, and then splitting it when it needs to be played.
实施例五  Embodiment 5
图 6为本发明实施例五提供的多路音视频处理方法的流程图，本实施例具 体提供了多路音视频码流的整个处理过程， 如图 7所示， 为本发明实施例五中 多路音视频处理的流程框图， 多路音视频处理方法典型地是包括如下步骤： 步骤 610、 音视频发送端中的各摄像装置分别采集多路音视频码流； 摄像装置如图 7所示， 可以为摄像头， 也可以宽泛地解释为存储音视频码 流的影碟， 或者其他能够提供音视频码流的装置。  FIG. 6 is a flowchart of a multi-channel audio and video processing method according to Embodiment 5 of the present invention. This embodiment specifically provides a whole process of processing a multi-channel audio and video code stream, as shown in FIG. 7 , which is a fifth embodiment of the present invention. The flow block diagram of the multi-channel audio and video processing, the multi-channel audio and video processing method typically includes the following steps: Step 610: Each camera device in the audio and video transmitting end separately collects multiple audio and video code streams; It can be a camera, or it can be broadly interpreted as a disc that stores audio and video streams, or other devices that can provide audio and video streams.
步骤 620、音视频发送端对采集到的多路音视频码流分别进行数模转换和 编码， 以获取多路音视频码流；  Step 620: The audio and video transmitting end performs digital-to-analog conversion and encoding on the collected multi-channel audio and video code streams to obtain a multi-channel audio and video code stream.
上述步骤 620中，具体可以采用多个 AD转换器对各路音视频码流进行数模 转换， 而后采用多个编码器分别对各路音视频码流进行， 形成由数据报文组 成的帧码流。  In the foregoing step 620, a plurality of AD converters may be used to perform digital-to-analog conversion on each audio and video code stream, and then multiple encoders are used to respectively perform audio and video code streams to form a frame code composed of data packets. flow.
步骤 630、音视频发送端将各路音视频码流的通道号对应设置在各路音视 频码流的每个数据报文的帧头中， 再统一送入緩存中存储； Step 630: The audio and video transmitting end sets the channel number of each audio and video code stream correspondingly in each channel of audio and video. The frame header of each data packet of the frequency code stream is uniformly sent to the cache for storage;
图 7所示的緩存单元即可完成上述步骤 630的操作， 既添加帧头又进行緩 存， 在緩存中实际就形成了顺序排列的复合码流。 对数据报文进行緩存可以 按照接收的优先顺序， 以先入先出的队列进行存储。  The buffer unit shown in FIG. 7 can complete the operation of the above step 630, adding both a frame header and a buffer, and a sequenced composite code stream is actually formed in the cache. The data packets are cached in a first-in, first-out queue according to the received priority order.
步骤 640、 音视频发送端从緩存中提取各数据报文， 封装在至少一路复合 报文流中， 优选是封装在一路复合报文流中；  Step 640: The audio and video sending end extracts each data packet from the cache, and encapsulates it in at least one composite packet stream, preferably encapsulated in a composite message stream;
步骤 650、音视频发送端基于网络传输协议向音视频接收端传输该复合码 流;  Step 650: The audio and video transmitting end transmits the composite code stream to the audio and video receiving end according to a network transmission protocol.
步骤 660、音视频接收端识别复合码流中每个数据报文帧头中携带的对应 于每路音视频码流设置的通道号；  Step 660: The audio and video receiving end identifies a channel number corresponding to each audio and video code stream carried in each data message frame header in the composite code stream;
步骤 670、音视频接收端根据通道号将提取的数据报文合并成为各路音视 频码流；  Step 670: The audio and video receiving end combines the extracted data packets into channel audio and video code streams according to the channel number.
上述步骤 660和步骤 670中，具体可以将网络接收到的数据报文进行緩存， 而后在解析单元执行报文解析操作。 解析复合码流时， 首先逐个字节先搜索 到帧类型标识， 再根据后面的帧数据长度， 读出后面的帧数据， 根据解析出 的帧信息， 诸如通道号、 音视频具体格式等， 该码流可送入对应通道的解码 器进行解码显示。  In the foregoing steps 660 and 670, the data packet received by the network may be cached, and then the parsing unit performs a packet parsing operation. When parsing the composite code stream, the frame type identifier is first searched first byte by byte, and then the subsequent frame data is read according to the subsequent frame data length, according to the parsed frame information, such as channel number, audio and video specific format, etc. The code stream can be sent to the decoder of the corresponding channel for decoding and display.
步骤 680、音视频接收端对拆分后的各路音视频码流分别进行解码和緩存 处理；  Step 680: The audio and video receiving end separately decodes and buffers the split audio and video code streams.
上述步骤 680中，可以由多个解码单元从解析单元分别获取多路音视频码 流， 独立进行解码， 而后緩存。 在緩存的过程中可以执行时间同步控制， 以 便后续播放的画面保持时间同步。  In the above step 680, the multiple audio and video code streams can be separately obtained from the parsing unit by the plurality of decoding units, decoded independently, and then cached. The time synchronization control can be performed during the caching process so that the subsequent playback pictures are kept in time synchronization.
步骤 690、 音视频接收端将緩存处理的各路音视频码流进行画面拼接， 在 同一台显示器中进行播放。  Step 690: The audio and video receiving end splicing the audio and video code streams processed by the buffering and playing in the same display.
本实施例的技术方案， 实际上提供了多个编解码器、 单緩存拼接的技术 方案， 该技术方案更加适用于通用的单路音视频编解码器技术。 多路图像序 列分别通过编码器生成各自的视频码流， 多路码流加上各自通道的特殊标识 之后送入公共緩存中， 形成多路的复合码流。 录像存储可基于此复合码流录 像， 也可基于单路编码出来的码流录像。 网络发送基于此复合码流来处理， 网络终端接收数据送入緩存， 通过解析单元解析出各路视频码流， 分别送入 各自的解码器， 解出图像序列； 画面拼接之后就送显示器显示。 The technical solution of this embodiment actually provides a technical solution of multiple codecs and single buffer splicing, and the technical solution is more suitable for the general single channel audio and video codec technology. Multiple image sequence The columns respectively generate respective video code streams by the encoder, and the multiple code streams are added to the common buffer by adding the special identifiers of the respective channels to form a multi-way composite code stream. Video recording can be based on this composite stream recording, or it can be based on single-stream encoded stream recording. The network transmission is processed based on the composite code stream, and the network terminal receives the data into the buffer, and parses out each video code stream through the parsing unit, and sends them to the respective decoders to solve the image sequence; after the picture is spliced, it is sent to the display.
实施例六  Embodiment 6
图 8为本发明实施例六提供的多路音视频传输装置的结构示意图，该装置 具体可以集成在音视频发送端中， 具体可包括码流获取模块 810、码流合并模 块 820和码流传输模块 830。 其中， 码流获取模块 810用于分别获取多路音视频 码流； 码流合并模块 820用于在合并处理环节， 将各路音视频码流合并在至少 一路复合码流中， 其中， 合并在复合码流中的每路音视频码流对应设置有用 于标识所属音视频码流的码流标识；码流传输模块 830用于基于网络传输协议 向音视频接收端传输所述复合码流， 以指示音视频接收端根据码流标识将复 合码流拆分成为各路音视频码流。  FIG. 8 is a schematic structural diagram of a multi-channel audio and video transmission apparatus according to Embodiment 6 of the present invention. The apparatus may be specifically integrated in an audio/video transmitting end, and may specifically include a code stream acquiring module 810, a code stream combining module 820, and a code stream transmission. Module 830. The code stream obtaining module 810 is configured to respectively acquire the multi-channel audio and video code streams; the code stream combining module 820 is configured to combine the audio and video code streams in the at least one composite code stream in the merge processing link, where Each audio and video code stream in the composite code stream is correspondingly provided with a code stream identifier for identifying the associated audio and video code stream; the code stream transmission module 830 is configured to transmit the composite code stream to the audio and video receiver according to the network transmission protocol, The audio and video receiving end is instructed to split the composite code stream into each audio and video code stream according to the code stream identifier.
优选是选择报文封装环节作为合并处理环节，则码流合并模块 820可以包 括： 帧头添加单元 821和报文封装单元 822。 其中， 帧头添加单元 821用于将各 路音视频码流的通道号对应设置在各路音视频码流的每个数据报文的帧头 中； 报文封装单元 822用于将各数据报文封装在至少一路复合报文流中。  Preferably, the packet encapsulation link is selected as the merge processing link, and the stream merge module 820 can include: a header add unit 821 and a message encapsulation unit 822. The frame header adding unit 821 is configured to set the channel number of each audio and video code stream correspondingly in the frame header of each data packet of each audio and video code stream; the message encapsulating unit 822 is configured to use each datagram. The text is encapsulated in at least one composite message stream.
本实施例的技术方案可执行本发明任意实施例所提供多路音视频传输方 法， 并具有相应的功能模块， 通过将多路音视频码流整合为至少一路复合码 流， 特别是优选整合为一路复合码流， 在通过网络进行传输时， 避免了对网 络通道占用的不均衡性， 不会出现有些通道忙碌而有些通道空闲的情况， 因 而能够优化网络资源的使用， 提高数据传输效率。 本实施例的技术方案， 还 能够提升***性能， 当整合为少量复合码流时， 减少了***在多路音视频码 流之间调用处理资源而增加的负荷。  The technical solution of this embodiment may perform the multi-channel audio and video transmission method provided by any embodiment of the present invention, and has a corresponding functional module. By integrating the multi-channel audio and video code stream into at least one composite code stream, in particular, it is preferably integrated into A composite code stream avoids the imbalance of the network channel when transmitting through the network, and there is no situation that some channels are busy and some channels are idle, so that the use of network resources can be optimized, and the data transmission efficiency is improved. The technical solution of this embodiment can also improve system performance. When integrated into a small number of composite code streams, the load increased by the system invoking processing resources between multiple audio and video code streams is reduced.
实施例七 图 9为本发明实施例七提供的多路音视频传输装置的结构示意图，该装置 具体可以集成在音视频接收端中， 具体可包括码流接收模块 910、码流拆分模 块 920和码流处理模块 930。 其中， 码流接收模块 910用于基于网络传输协议接 收音视频发送端传输的至少一路复合码流；码流拆分模块 920用于在拆分处理 环节， 根据对应于每路音视频码流设置的码流标识， 将复合码流拆分成为各 路音视频码流；码流处理模块 930用于对拆分后的各路音视频码流分别进行处 理。 Example 7 FIG. 9 is a schematic structural diagram of a multi-channel audio and video transmission apparatus according to Embodiment 7 of the present invention. The apparatus may be specifically integrated in an audio/video receiving end, and specifically includes a code stream receiving module 910, a code stream splitting module 920, and a code stream. Processing module 930. The code stream receiving module 910 is configured to receive at least one composite code stream transmitted by the audio and video transmitting end according to the network transmission protocol, and the code stream splitting module 920 is configured to perform, according to each audio and video code stream setting, in the split processing link. The code stream identifier is used to split the composite code stream into respective audio and video code streams; the code stream processing module 930 is configured to separately process the split audio and video code streams.
优选是选择报文解析环节作为拆分处理环节，则码流拆分模块 920优选包 括： 帧头识别单元 921和码流合并单元 922。 其中， 帧头识别单元 921用于识别 每个数据报文帧头中携带的对应于每路音视频码流设置的通道号； 码流合并 单元 922用于根据所述通道号将提取的数据报文合并成为各路音视频码流。  Preferably, the message parsing step is selected as the split processing link, and the code stream splitting module 920 preferably includes: a header identifying unit 921 and a stream combining unit 922. The frame header identifying unit 921 is configured to identify a channel number corresponding to each audio and video code stream carried in each data message frame header; the code stream merging unit 922 is configured to extract the datagram according to the channel number. The text is merged into a stream of audio and video streams.
本实施例的技术方案可执行本发明任意实施例所提供多路音视频处理方 法， 并具有相应的功能模块。 本实施例的技术方案通过整合为至少一路复合 码流的多路音视频码流， 在通过网络进行传输时， 避免了对网络通道占用的 不均衡性， 不会出现有些通道忙碌而有些通道空闲的情况， 因而能够优化网 络资源的使用， 提高数据传输效率。 本实施例的技术方案， 还能够提升*** 性能， 当处理少量甚至一条复合码流时， 减少了***在多路音视频码流之间 调用处理资源而增加的负荷。  The technical solution of this embodiment can implement the multi-channel audio and video processing method provided by any embodiment of the present invention, and has corresponding functional modules. The technical solution of the embodiment is integrated into the multi-channel audio and video code stream of at least one composite code stream, and avoids the imbalance of the network channel when transmitting through the network, and some channels are not busy and some channels are idle. In this case, it is possible to optimize the use of network resources and improve data transmission efficiency. The technical solution of the embodiment can also improve the performance of the system. When processing a small number or even a composite code stream, the load increased by the system invoking processing resources between the multiple audio and video code streams is reduced.
本发明实施例还提供了一种多路音视频处理***， 包括本发明任意实施 例所提供的多路音视频传输装置以及多路音视频处理装置。  The embodiment of the present invention further provides a multi-channel audio and video processing system, including the multi-channel audio and video transmission device and the multi-channel audio and video processing device provided by any embodiment of the present invention.
本发明各实施例的技术方案解决了远程播放多路音视频的问题， 通过将 多路音视频码流整合为至少一路复合码流，特别是优选整合为一路复合码流， 从而具有如下优势： 一方面， 本发明实施例的技术方案通过整合减少了音视 频码流的数量， 因此在通过网络传输码流时， 无需独立占用网络通道而出现 有些通道忙碌有些通道空闲的现象， 因而能够充分地利用网络资源， 提高传 输效率； 另一方面， 音视频码流的处理环节较多， 例如数模转换、 编解码等， 当音视频码流达到十几路甚至几十路时， 本发明实施例的技术方案不会过多 地增加***在各处理环节调度处理资源时所增加的负荷， 能够更容易实现处 理资源的平衡利用； 再一方面， 本发明实施例的技术方案通过统一传输复合 码流， 能够尽量均衡数据传输速率， 减少先传输到的数据等待后传输到的数 据而播放延迟的现象。 The technical solution of the embodiments of the present invention solves the problem of remotely playing multi-channel audio and video. By integrating the multi-channel audio and video code streams into at least one composite code stream, and particularly preferably integrating into one composite code stream, the following advantages are obtained: On the one hand, the technical solution of the embodiment of the present invention reduces the number of audio and video code streams by integration. Therefore, when the code stream is transmitted through the network, there is no need to occupy the network channel independently, and some channels are busy and some channels are idle, so that the channel stream can be fully utilized. Using network resources to improve transmission efficiency; on the other hand, audio and video streams have many processing links, such as digital-to-analog conversion, codec, etc. When the audio and video code stream reaches a dozen or even dozens of channels, the technical solution of the embodiment of the present invention does not excessively increase the load added by the system when scheduling processing resources in each processing link, and the processing resource balance can be more easily realized. In another aspect, the technical solution of the embodiment of the present invention can uniformly balance the data transmission rate by using the unified transmission of the composite code stream, and reduce the phenomenon that the data transmitted after the data to be transmitted is transmitted and delayed.
本领域普通技术人员可以理解： 实现上述方法实施例的全部或部分步骤 可以通过程序指令相关的硬件来完成， 前述的程序可以存储于一计算机可读 取存储介质中， 该程序在执行时， 执行包括上述方法实施例的步骤； 而前述 的存储介质包括： R0M、 RAM, 磁碟或者光盘等各种可以存储程序代码的介质。  A person skilled in the art can understand that all or part of the steps of implementing the above method embodiments may be completed by using hardware related to program instructions, and the foregoing program may be stored in a computer readable storage medium, and the program is executed when executed. The steps of the foregoing method embodiments are included; and the foregoing storage medium includes: a medium that can store program codes, such as a ROM, a RAM, a magnetic disk, or an optical disk.
最后应说明的是： 以上实施例仅用以说明本发明的技术方案， 而非对其 限制； 尽管参照前述实施例对本发明进行了详细的说明， 本领域的普通技术 人员应当理解： 其依然可以对前述各实施例所记载的技术方案进行修改， 或 者对其中部分技术特征进行等同替换； 而这些修改或者替换， 并不使相应技 术方案的本质脱离本发明各实施例技术方案的精神和范围。  It should be noted that the above embodiments are only for explaining the technical solutions of the present invention, and are not intended to be limiting; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those skilled in the art that: The technical solutions described in the foregoing embodiments are modified, or some of the technical features are equivalently replaced. The modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

权 利 要 求 书 Claim
1、 一种多路音视频传输方法， 其特征在于， 包括：  A multi-channel audio and video transmission method, comprising:
分别获取多路音视频码流；  Obtaining multiple audio and video streams respectively;
在合并处理环节， 将各路音视频码流合并在至少一路复合码流中， 其中， 合并在复合码流中的每路音视频码流对应设置有用于标识所属音视频码流的 码流标识；  In the merging process, each audio and video code stream is combined into at least one composite code stream, where each audio and video code stream combined in the composite code stream is correspondingly provided with a code stream identifier for identifying the associated audio and video code stream. ;
基于网络传输协议向音视频接收端传输所述复合码流， 以指示所述音视 频接收端根据所述码流标识将所述复合码流拆分成为各路音视频码流。  Transmitting the composite code stream to the audio and video receiver according to the network transmission protocol, to instruct the audio and video receiver to split the composite code stream into audio and video code streams according to the code stream identifier.
2、 根据权利要求 1所述的多路音视频传输方法， 其特征在于， 将各路 音视频码流合并在至少一路复合码流中包括：  2. The multiplex audio and video transmission method according to claim 1, wherein combining the audio and video code streams in the at least one composite code stream comprises:
将所述码流标识对应设置在各路音视频码流的每个单位数据块中， 并将 各单位数据块合并成为至少一路复合码流。  The code stream identifiers are correspondingly arranged in each unit data block of each audio and video code stream, and each unit data block is combined into at least one composite code stream.
3、 根据权利要求 2所述的多路音视频传输方法， 其特征在于， 所述合 并处理环节为报文封装环节， 则将所述码流标识对应设置在各路音视频码流 的每个单位数据块中， 并将各单位数据块合并成为至少一路复合码流包括： 将各路音视频码流的通道号对应设置在各路音视频码流的每个数据报文 的帧头中；  The multi-channel audio and video transmission method according to claim 2, wherein the merge processing link is a packet encapsulation step, and the code stream identifier is correspondingly set in each of the audio and video code streams of each channel. In the unit data block, combining the unit data blocks into the at least one composite code stream includes: setting a channel number of each audio and video code stream correspondingly in a frame header of each data message of each audio and video code stream;
将各数据报文封装在至少一路复合报文流中。  Each data packet is encapsulated in at least one composite message stream.
4、 根据权利要求 1-3任一所述的多路音视频传输方法， 其特征在于， 分别获取多路音视频码流包括：  The multi-channel audio and video transmission method according to any one of claims 1-3, wherein acquiring the multi-channel audio and video code streams separately comprises:
从各摄像装置中分别采集多路音视频码流；  Collecting multiple audio and video code streams from each camera device;
对采集到的多路音视频码流分别进行数模转换和编码，以获取多路音视 频码流。  The collected multi-channel audio and video streams are respectively digital-to-analog converted and encoded to obtain a multi-channel audio and video stream.
5、 根据权利要求 1-3任一所述的多路音视频传输方法， 其特征在于， 在将各路音视频码流合并在至少一路复合码流中之后， 还包括： 分别存储所 述各路音视频码流， 和 /或存储所述复合码流。 The multiplex audio and video transmission method according to any one of claims 1-3, further comprising: separately storing the audio and video code streams in the at least one composite code stream a road audio video stream, and/or storing the composite code stream.
6、 一种多路音视频处理方法， 其特征在于， 包括： 6. A multi-channel audio and video processing method, comprising:
基于网络传输协议接收音视频发送端传输的至少一路复合码流； 在拆分处理环节， 根据对应于每路音视频码流设置的码流标识， 将所述 复合码流拆分成为各路音视频码流；  Receiving at least one composite code stream transmitted by the audio and video transmitting end according to the network transmission protocol; in the splitting processing step, splitting the composite code stream into respective road sounds according to the code stream identifier corresponding to each audio and video code stream set Video stream
对拆分后的各路音视频码流分别进行处理。  The split audio and video streams are processed separately.
7、 根据权利要求 6所述的多路音视频处理方法， 其特征在于，根据对 应于每路音视频码流设置的码流标识，将所述复合码流拆分成为各路音视频 码流包括：  The multiplex audio and video processing method according to claim 6, wherein the composite code stream is split into audio and video streams according to a code stream identifier corresponding to each audio and video code stream. Includes:
从所述复合码流中提取各单位数据块；  Extracting each unit data block from the composite code stream;
根据每个单位数据块中携带的对应于每路音视频码流设置的码流标识， 将提取的单位数据块合并成为各路音视频码流。  The extracted unit data blocks are merged into respective audio and video code streams according to the code stream identifiers corresponding to each audio and video code stream carried in each unit data block.
8、 根据权利要求 7所述的多路音视频处理方法， 其特征在于， 所述拆 分处理环节为报文解析环节， 则根据每个单位数据块中携带的对应于每路音 视频码流设置的码流标识，将提取的单位数据块合并成为各路音视频码流包 括：  The multi-channel audio and video processing method according to claim 7, wherein the split processing link is a message parsing step, and is corresponding to each audio and video code stream carried in each unit data block. The code stream identifier is set, and the extracted unit data blocks are merged into each audio and video code stream, including:
识别每个数据报文帧头中携带的对应于每路音视频码流设置的通道号； 根据所述通道号将提取的数据报文合并成为各路音视频码流。  Identifying a channel number corresponding to each audio and video code stream carried in each data message frame header; and combining the extracted data packets according to the channel number into a channel audio and video code stream.
9、 根据权利要求 6-8任一所述的多路音视频处理方法， 其特征在于， 对拆分后的各路音视频码流分别进行处理包括：  The multi-channel audio and video processing method according to any one of claims 6-8, wherein separately processing the split audio and video code streams comprises:
对拆分后的各路音视频码流分别进行解码和緩存处理；  Decoding and buffering the respective audio and video code streams after splitting;
并将緩存处理的各路音视频码流进行画面拼接， 在同一台显示器中进行 播放。  The video and audio code streams processed by the buffer are spliced and played on the same display.
10、 根据权利要求 6-8任一所述的多路音视频处理方法，其特征在于，基 于网 专输协议接收音视频发送端传输的至少一路复合码流之后， 还包括： 对接收到的所述复合码流进行存储。  The multi-channel audio and video processing method according to any one of claims 6-8, wherein after receiving at least one composite code stream transmitted by the audio and video transmitting end, the network special transmission protocol further comprises: The composite code stream is stored.
11、 一种多路音视频传输装置， 其特征在于， 包括： 码流获耳 莫块， 用于分别获取多路音视频码流； A multi-channel audio and video transmission device, comprising: The code stream is obtained by using an ear block for respectively acquiring a multi-channel audio and video code stream;
码流合并模块， 用于在合并处理环节， 将各路音视频码流合并在至少一 路复合码流中， 其中， 合并在复合码流中的每路音视频码流对应设置有用于 标识所属音视频码流的码流标识；  The code stream merging module is configured to combine the audio and video code streams in the at least one composite code stream in the merging process, wherein each audio and video code stream combined in the composite code stream is correspondingly configured to identify the associated sound The code stream identifier of the video code stream;
码流传输模块， 用于基于网络传输协议向音视频接收端传输所述复合码 流， 以指示所述音视频接收端根据所述码流标识将所述复合码流拆分成为各 路音视频码流。  a code stream transmission module, configured to transmit the composite code stream to an audio/video receiving end according to a network transmission protocol, to indicate that the audio and video receiving end splits the composite code stream into audio and video channels according to the code stream identifier Code stream.
12、 根据权利要求 11所述的多路音视频传输装置， 其特征在于， 所述 合并处理环节为报文封装环节， 则所述码流合并模块包括：  The multi-channel audio and video transmission device according to claim 11, wherein the merge processing link is a packet encapsulation step, and the code stream merging module comprises:
帧头添加单元， 用于将各路音视频码流的通道号对应设置在各路音视频 码流的每个数据报文的帧头中；  a frame header adding unit, configured to set a channel number of each audio and video code stream in a frame header of each data packet of each audio and video code stream;
报文封装单元， 用于将各数据报文封装在至少一路复合报文流中。  A packet encapsulating unit is configured to encapsulate each data packet in at least one composite packet stream.
13、 一种多路音视频处理装置， 其特征在于， 包括：  13. A multi-channel audio and video processing apparatus, comprising:
码流接收模块， 用于基于网络传输协议接收音视频发送端传输的至少一 路复合码流；  a code stream receiving module, configured to receive at least one composite code stream transmitted by the audio and video transmitting end according to a network transmission protocol;
码流拆分模块， 用于在拆分处理环节， 根据对应于每路音视频码流设置 的码流标识， 将所述复合码流拆分成为各路音视频码流；  a code stream splitting module, configured to split the composite code stream into audio and video code streams according to a code stream identifier corresponding to each audio and video code stream in a split processing link;
码流处理模块， 用于对拆分后的各路音视频码流分别进行处理。  The code stream processing module is configured to separately process the split audio and video code streams.
14、 根据权利要求 13所述的多路音视频处理装置， 其特征在于， 所述 拆分处理环节为报文解析环节， 则所述码流拆分模块包括：  The multi-channel audio and video processing device according to claim 13, wherein the split processing link is a message parsing step, and the code stream splitting module comprises:
帧头识别单元， 用于识别每个数据报文帧头中携带的对应于每路音视频 码流设置的通道号；  a frame header identifying unit, configured to identify a channel number corresponding to each audio and video code stream carried in each data message frame header;
码流合并单元， 用于根据所述通道号将提取的数据报文合并成为各路音 视频码流。  And a code stream merging unit, configured to merge the extracted data packets into respective audio and video code streams according to the channel number.
15、 一种多路音视频处理***， 其特征在于， 包括： 权利要求 11 或 12 所述的多路音视频传输装置以及权利要求 13或 14所述的多路音视频处理装置。  A multi-channel audio and video processing system, comprising: the multi-channel audio and video transmission device according to claim 11 or 12; and the multi-channel audio and video processing device according to claim 13 or 14.
PCT/CN2011/077259 2011-07-18 2011-07-18 Method, device and system for transmitting and processing multi-channel audio-video WO2012106898A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/CN2011/077259 WO2012106898A1 (en) 2011-07-18 2011-07-18 Method, device and system for transmitting and processing multi-channel audio-video
CN201180001226.8A CN102301730B (en) 2011-07-18 2011-07-18 Method, device and system for transmitting and processing multichannel AV

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2011/077259 WO2012106898A1 (en) 2011-07-18 2011-07-18 Method, device and system for transmitting and processing multi-channel audio-video

Publications (1)

Publication Number Publication Date
WO2012106898A1 true WO2012106898A1 (en) 2012-08-16

Family

ID=45360541

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2011/077259 WO2012106898A1 (en) 2011-07-18 2011-07-18 Method, device and system for transmitting and processing multi-channel audio-video

Country Status (2)

Country Link
CN (1) CN102301730B (en)
WO (1) WO2012106898A1 (en)

Families Citing this family (41)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103199865B (en) * 2012-01-10 2016-06-15 武汉凯默电气有限公司 A kind of light adaptive serial port decoding circuit
CN102682741B (en) * 2012-05-30 2014-12-03 华为技术有限公司 Multi-display control system and implementation method of multi-display control system
CN102739777B (en) * 2012-06-01 2015-04-01 苏州阔地网络科技有限公司 Data stream disconnection recovering method and system
CN103514225B (en) * 2012-06-27 2016-12-21 浙江大华技术股份有限公司 The multichannel multiplexing method of a kind of data source and device thereof
CN103517027B (en) * 2012-06-29 2017-05-24 联想(北京)有限公司 Video data processing method and electronic equipment
CN103581609B (en) * 2012-07-23 2018-09-28 中兴通讯股份有限公司 A kind of method for processing video frequency and device, system
CN102801969A (en) * 2012-07-25 2012-11-28 华为技术有限公司 Method, device and system of processing multimedia data
CN103856784B (en) * 2012-12-03 2019-02-05 联想(北京)有限公司 Data sending, receiving method and electronic equipment
CN103179435B (en) * 2013-02-27 2016-09-28 北京视博数字电视科技有限公司 A kind of multi-path video data multiplexing method and device
CN103686047A (en) * 2013-12-18 2014-03-26 电子科技大学 Multi-channel video data transmission method
CN104703051B (en) * 2014-01-06 2018-06-05 杭州海康威视数字技术股份有限公司 Code stream sending method and device
CN105227242A (en) * 2014-06-17 2016-01-06 中兴通讯股份有限公司 Opto-electronic receiver, launching technique, device, optoelectronic transceiver method, module, equipment
CN105847825A (en) 2015-01-16 2016-08-10 杭州海康威视数字技术股份有限公司 Encoding, index storage and access methods for video encoding code stream and corresponding apparatus
CN104734992B (en) * 2015-03-30 2018-04-20 华为技术有限公司 A kind of service data transmission method and device
CN105979289A (en) * 2015-12-03 2016-09-28 乐视致新电子科技（天津）有限公司 Video generation and play method and device
CN105451067A (en) * 2015-12-04 2016-03-30 珠海全志科技股份有限公司 Audio and video data transmission method and device
CN105898381A (en) * 2015-12-15 2016-08-24 乐视致新电子科技（天津）有限公司 Content transmission method, content play method, content server and intelligent equipment
CN105681682B (en) * 2016-01-19 2019-06-14 广东威创视讯科技股份有限公司 Method of transmitting video data and system
CN106028137A (en) * 2016-06-22 2016-10-12 北京小米移动软件有限公司 Live streaming processing method and apparatus
CN106341619A (en) * 2016-09-27 2017-01-18 北京乐景科技有限公司 Video data synchronous transmission method, device and system
CN106331650A (en) * 2016-09-27 2017-01-11 北京乐景科技有限公司 Video data transmission method and apparatus
CN106791902A (en) * 2016-12-23 2017-05-31 北京多视角文化传播有限公司 To the audio-video switching method and its cut bank of mobile internet device
CN106961571B (en) * 2017-03-07 2020-09-04 深圳市广和通无线通信软件有限公司 Method and system for simultaneously receiving multiple paths of image data
CN107645647A (en) * 2017-09-21 2018-01-30 京信通信***（中国）有限公司 A kind of multichannel audio-video frequency transmission method and device
CN107888567B (en) 2017-10-23 2020-08-04 浙江大华技术股份有限公司 Transmission method and device for composite multimedia signal
CN109922330B (en) * 2017-12-13 2021-11-16 北京传送科技有限公司 Data sending method, data receiving method and device
CN109618110A (en) * 2018-12-24 2019-04-12 大陆汽车车身电子***（芜湖）有限公司 Multi-channel video Transmission system
CN111372035A (en) * 2018-12-25 2020-07-03 杭州海康威视数字技术股份有限公司 Multimedia data processing method and device, electronic equipment and readable storage medium
CN110049341B (en) * 2019-03-08 2022-02-18 视联动力信息技术股份有限公司 Video processing method and device
CN110349584A (en) * 2019-07-31 2019-10-18 北京声智科技有限公司 A kind of audio data transmission method, device and speech recognition system
CN112740708B (en) * 2020-05-21 2022-07-22 华为技术有限公司 Audio data transmission method and related device
CN111698386B (en) * 2020-05-26 2021-07-09 中国科学院上海微***与信息技术研究所 Multi-channel image data synchronous transmitting device, receiving device and transmission system
CN112383701A (en) * 2020-10-14 2021-02-19 深圳海翼智新科技有限公司 Control method and system for image acquisition equipment and hardware controller
CN112492348A (en) * 2020-10-20 2021-03-12 广州视源电子科技股份有限公司 Multi-code stream processing and output method, equipment and storage medium
CN114520925A (en) * 2020-11-19 2022-05-20 西安诺瓦星云科技股份有限公司 Video stream processing method, device and system and cloud server system
CN112839240B (en) * 2020-12-31 2022-03-22 福州大学 Bandwidth detection method and system based on video stream
CN113068008B (en) * 2021-03-24 2022-09-27 苏州科达科技股份有限公司 Decoding method, encoding method, code stream transmission method, electronic equipment and system
CN113840166A (en) * 2021-08-31 2021-12-24 南京巨鲨显示科技有限公司 Method and system for synchronizing audio and video mixing of multi-path streaming media
CN114222166B (en) * 2021-09-29 2024-02-13 重庆创通联达智能技术有限公司 Multi-channel video code stream real-time processing and on-screen playing method and related system
CN115802074B (en) * 2022-11-10 2024-03-29 中国联合网络通信集团有限公司 Multipath video transmission method, device, equipment and medium
CN116886912B (en) * 2022-12-06 2024-02-13 广州开得联软件技术有限公司 Multipath video coding method, device, equipment and storage medium

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101068366A (en) * 2007-06-05 2007-11-07 南京大学 H.264 based multichannel video transcoding multiplexing method and multiplexer
CN101662637A (en) * 2009-09-25 2010-03-03 深圳创维数字技术股份有限公司 Method for playing all packaged programs in transmission code stream and set-top box
CN101977305A (en) * 2010-10-27 2011-02-16 北京中星微电子有限公司 Video processing method, device and system

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2006067124A (en) * 2004-08-25 2006-03-09 Nec Corp Method and device for switching image encoded data, system, and program
CN201226562Y (en) * 2008-03-27 2009-04-22 北京汉邦高科数字技术有限公司 Multipath video playback apparatus for embedded digital picture-recording equipment
KR20100121384A (en) * 2009-05-08 2010-11-17 삼성전자주식회사 System and method for providing service related to telephone to a plurality of devices using upnp in the home network

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101068366A (en) * 2007-06-05 2007-11-07 南京大学 H.264 based multichannel video transcoding multiplexing method and multiplexer
CN101662637A (en) * 2009-09-25 2010-03-03 深圳创维数字技术股份有限公司 Method for playing all packaged programs in transmission code stream and set-top box
CN101977305A (en) * 2010-10-27 2011-02-16 北京中星微电子有限公司 Video processing method, device and system

Also Published As

Publication number Publication date
CN102301730B (en) 2014-02-19
CN102301730A (en) 2011-12-28

Similar Documents

Publication Publication Date Title
WO2012106898A1 (en) Method, device and system for transmitting and processing multi-channel audio-video
JP6422527B2 (en) Data receiving method and apparatus in multimedia system
US10177958B2 (en) Method for synchronously taking audio and video in order to proceed one-to-multi multimedia stream
CN108877820B (en) Audio data mixing method and device
CN101272500B (en) Transmission method and system for video/audio data flow
CN107147916B (en) Method for transmitting H.265 coding video data on transmission layer
JP2009177447A (en) Moving image transmitting and receiving system
CN109640162B (en) Code stream conversion method and system
JP2018117259A (en) One-to-many audio video streaming method by audio video synchronous take in
CN100455027C (en) Signal processing apparatus and stream processing method
CN1534503A (en) Method of realizing real time image sound talks in network game, system and storage medium thereof
WO2012068940A1 (en) Method for monitoring terminal through ip network and mcu
CN108122558A (en) A kind of LATM AAC audio streams turn appearance implementation method and device in real time
CN100479529C (en) Conversion method of multiplexing protocols in broadcast network
CN108632679A (en) A kind of method of multi-medium data transmission and a kind of regarding networked terminals
CN100416689C (en) Reproducing apparatus and method, and recording medium
JP2004537226A5 (en)
KR20170003904A (en) Broadcasting signal transmitting device, broadcasting signal receiving device, broadcasting signal transmitting method, and broadcasting signal receiving method
WO2013040996A1 (en) Sending end, terminal, system and method for hierarchical coding multiplexing
CN108124183B (en) Method for synchronously acquiring video and audio to perform one-to-many video and audio streaming
US10567852B2 (en) Transmission device, transmission method, reception device, reception method, and program
CN110392276B (en) Live broadcast recording and broadcasting method based on Real Time Messaging Protocol (RTMP) synchronous transmission MIDI
WO2016107174A1 (en) Method and system for processing multimedia file data, player and client
JP6641344B2 (en) Encoding device
CN110324666B (en) Method and system for playing security video stream

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 201180001226.8

Country of ref document: CN

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 11858166

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 11858166

Country of ref document: EP

Kind code of ref document: A1