AU2012201576A1 - Improved systems and methods for error resilience in video communication systems - Google Patents

Abstract

C:\NRPonnDCC\KNM\1 212 _ I )O0C.-1, 112 Systems and methods for error resilient transmission and for random access in video communication systems are provided. The video communication systems arc based on 5 single-layer, scalable video, or simulcast video coding with temporal scalahility. which may be used in video communication systems. A set of video frames or pictures in a video signal transmission is designated for reliable or guaranteed delivery to receivers using secure or high reliability links, or by retransmission techniques. The reliably-delivered video frames are used as reference pictures for resynchronization of receivers with the 10 transmitted video signal after error incidence and for random access. WO 2008/086423 PCT/US2008/050640 4* cJ n I-D C,,N C=( C-14

Description

Australian Patents Act 1990 - Regulation 3.2 ORIGINAL COMPLETE SPECIFICATION STANDARD PATENT Invention Title "Improved systems and methods for error resilience in video communication systems" The following statement is a full description of this invention, including the best method of performing it known to me/us: P/00/0 Il C.\NRPonbl\DCC\KXM'4212 i_ DOC- 15/1U21112 - 1 IMPROVED SYSTEMS AND METHODS FOR ERROR RESI lI IENCE IN VIDEO COMMUNICATION SYSTEMS Cross-Reference to Related Applications 5 This application claims the benefit of United States provisional patent application Serial No. 60/884,148 filed January 9, 2007. Further. this application is related to International patent application Nos. PCT/US06/028365, PCT/US06/028366. PCT/US06/0618 15. PCT/US06/62569, PCT/US07/80089, PCT/JS07/062357. PCTUIJS07/065554. 10 PCT/US07/065003, PCT/US06/028367, PCT/US07/063335. lCT/US07/08 1217. PCTI/US07/080089, PCT/US07/083351, PCT/US07/086958. and PCT/1US07/089076. All of the aforementioned applications. which arc commonly assigned. are herehy incorporated by reference herein in their entireties. The disclosure of the complete specification of Australian Patent Application No. 2008204833. as originally filed. is incorporated herein by reference. 15 Field of the Invention The present invention relates to video data communication systems. In particular. the invention relates to techniques for providing error resilience in videoconferencing applications. 20 Background of the Invention Providing high quality digital video communications between senders and receivers over packet-based modern communication networks (e.g.. a network based on the Internet Protocol 25 (IP)) is technically challenging, at least due to the fact that data transport on such networks is typically carried out on a best-effort basis. Transmission errors in modern communication networks generally manifest themselves as packet losses and not as hit errors. which were characteristic of earlier communication systems. The packet losses often are the restilt of congestion in intermediary routers. and not the result of physical layer errors. 30 When a transmission error occurs in a digital video comilLinication system. it is important to ensure that the receiver can quickly recover from the error and turn to an error-free display of the incoming video signal. However, in typical digital video communication svstels. the receiver's robustness is reduced by the fact that the incoming data is heavily compressed in 35 order to conserve bandwidth. Further. the WO 2008/086423 PCT/US2008/050640 video compression techniques employed in the communication systems (e.g., state-of the-art codecs ITU-T H.264 and H.263 or ISO MPEG-2 and MPEG-4 codecs) can create a strong temporal dependency between sequential video packets or frames. In particular, use of motion compensated prediction (e.g., involving the use of P or B 5 frames) codecs creates a chain of frame dependencies in which a displayed frame depends on past frame(s). The chain of dependencies can extend all the way to the beginning of the video sequence. As a result of the chain of dependencies, the loss of a given packet can affect the decoding of a number of the subsequent packets at the receiver. Error propagation due to the loss of the given packet terminates only at an 10 "intra" (I) refresh point, or at a frame which does not use any temporal prediction at all. [00051 Error resilience in digital video communication systems requires having at least some level of redundancy in the transmitted signals. However, this requirement is contrary to the goals of video compression techniques, which strive to eliminate or 15 minimize redundancy in the transmitted signals. [00061 On a network that offers differentiated services (e.g., DiffServ IP-based networks, private networks over leased lines, etc.), a video data communication application may exploit network features to deliver some or all of video signal data in a lossless or nearly lossless manner to a receiver. However, in an arbitrary best-effort 20 network (such as the Internet) that has no provision for differentiated services, a data communication application has to rely on its own features for achieving error resilience. Known techniques (e.g., the Transmission Control Protocol - TCP) that are useful in text or alpha-numeric data communications are not appropriate for video or audio communications, which have the added constraint of low end-to-end delay 25 arising out of human interface requirements. For example, TCP techniques may be used for error resilience in text or alpha-numeric data transport. TCP keeps on retransmitting data until confirmation that all data is received, even if it involves a delay of several seconds. However, TCP is inappropriate for video data transport in a live or interactive videoconferencing application because the end-to-end delay, which 30 is unbounded, would be unacceptable to participants. [0007] An aspect of error resilience in video communication systems relates to random access (e.g., when a receiver joins an existing transmission of a video signal), which has a considerable impact on compression efficiency. Instances of random access are, for example, a user who joins a videoconference, or a user who tunes in to NY02:608243.1 2 WO 2008/086423 PCT/US2008/050640 a broadcast. Such a user would have to find a suitable point in the incoming bitstream signal to start decoding and be synchronized with the encoder. A random access point is effectively an error resilience feature since at that point any error propagation terminates (or is an error recovery point). Thus, a particular coding scheme, which 5 provides good random access support, will generally have an error resilience technique that provides for faster error recovery. However, the converse depends on the specific assumptions about the duration and extent of the errors that the error resilience technique is designed to address. The error resilience technique may assume that some state information is available at the receiver at the time an error 10 occurs. In such case, the error resilience technique does not assure good random access support. 100081 In MPEG-2 video codecs for digital television systems (digital cable TV or satellite TV), I pictures are used at periodic intervals (typically 0.5 sec) to enable fast switching into a stream. The I pictures, however, are considerably larger than their P 15 or B counterparts (typically by 3-6 times) and are thus to be avoided, especially in low bandwidth and/or low delay applications. [00091 In interactive applications such as videoconferencing, the concept of requesting an intra update is often used for error resilience. In operation, the update involves a request from the receiver to the sender for an intra picture transmission, 20 which enables the decoder to be synchronized. The bandwidth overhead of this operation is significant. Additionally, this overhead is also incurred when packet errors occur. If the packet losses are caused by congestion, then the use of the intra pictures only exacerbates the congestion problem. [0010] Another traditional technique for error robustness, which has been used in the 25 past to mitigate drift caused by mismatch in IDCT implementations (e.g., in the H.261 standard), is to periodically code each macroblock intra mode. The H.261 standard requires forced intra coding every 132 times a macroblock is transmitted. [0011] The coding efficiency decreases with increasing percentage of macroblocks that are forced to be coded as intra in a given frame. Conversely, when this 30 percentage is low, the time to recover from a packet loss increases. The forced intra coding process requires extra care to avoid motion-related drift, which further limits the encoder's performance since some motion vector values have to be avoided, even if they are the most effective. NY02:608243.I 3 WO 2008/086423 PCT/US2008/050640 [00121 In addition to traditional, single-layer codecs, layered or scalable coding is a well-known technique in multimedia data encoding. Scalable coding is used to generate two or more "scaled" bitstreams collectively representing a given medium in a bandwidth-efficient manner. Scalability can be provided in a number of different 5 dimensions, namely temporally, spatially, and quality (also referred to as SNR "Signal-to-Noise Ratio" scalability). For example, a video signal may be scalably coded in different layers at CIF and QCIF resolutions, and at frame rates of 7.5, 15, and 30 frames per second (fps). Depending on the codec's structure, any combination of spatial resolutions and frame rates may be obtainable from the codec bitstream. 10 The bits corresponding to the different layers can be transmitted as separate bitstreams (i.e., one stream per layer) or they can be multiplexed together in one or more bitstreams. For convenience in description herein, the coded bits corresponding to a given layer may be referred to as that layer's bitstream, even if the various layers are multiplexed and transmitted in a single bitstream. Codecs specifically designed to 15 offer scalability features include, for example, MPEG-2 (ISO/IEC 13818-2, also known as ITU-T H.262) and the currently developed H.264 Scalable Video Coding extension (known as ITU-T H.264 Annex G or MPEG-4 Part 10 SVC). Scalable video coding (SVC) techniques specifically designed for video communication are described in commonly assigned international patent application No. 20 PCT/US06/028365 "SYSTEM AND METHOD FOR SCALABLE AND LOW DELAY VIDEOCONFERENCING USING SCALABLE VIDEO CODING". It is noted that even codecs that are not specifically designed to be scalable can exhibit scalability characteristics in the temporal dimension. For example, consider an MPEG-2 Main Profile code, a non-scalable codec, which is used in DVDs and 25 digital TV environments. Further, assume that the codec is operated at 30 fps and that a GOP structure of IBBPBBPBBPBBPBB (period N=15 frames) is used. By sequential elimination of the B pictures, followed by elimination of the P pictures, it is possible to derive a total of three temporal resolutions: 30 fps (all picture types included), 10 fps (I and P only), and 2 fps (I only). The sequential elimination 30 process results in a decodable bitstream because the MPEG-2 Main Profile codec is designed so that coding of the P pictures does not rely on the B pictures, and similarly coding of the I pictures does not rely on other P or B pictures. In the following, single-layer codecs with temporal scalability features are considered to be NY02:608243.1 4 WO 2008/086423 PCT/US2008/050640 a special case of scalable video coding, and are thus included in the term scalable video coding, unless explicitly indicated otherwise. [00131 Scalable codecs typically have a pyramidal bitstream structure in which one of the constituent bitstreams (called the "base layer") is essential in recovering the 5 original medium at some basic quality. Use of one or more the remaining bitstream(s) (called "the enhancement layer(s)") along with the base layer increases the quality of the recovered medium. Data losses in the enhancement layers may be tolerable, but data losses in the base layer can cause significant distortions or complete loss of the recovered medium. 10 [00141 Scalable codecs pose challenges similar to those posed by single layer codecs for error resilience and random access. However, the coding structures of the scalable codecs have unique characteristics that are not present in single layer video codecs. Further, unlike single layer coding, scalable coding may involve switching from one scalability layer to another (e.g., switching back and forth between CIF and QCIF 15 resolutions). 100151 Simulcasting is a coding solution for videoconferencing that is less complex than scalable video coding but has some of the advantages of the latter. In simulcasting, two different versions of the source are encoded (e.g., at two different spatial resolutions) and transmitted. Each version is independent, in that its decoding 20 does not depend on reception of the other version. Like scalable and single-layer coding, simulcasting poses similar random access and robustness issues. In the following, simulcasting is considered a special case of scalable coding (where no inter layer prediction is performed) and both are referred to simply as scalable video coding techniques unless explicitly indicated otherwise. 25 [0016] Specific techniques for providing error resilience and random access in video communication systems are described in commonly assigned International patent application Nos. PCT/US06/061815, "SYSTEMS AND METHODS FOR ERROR RESILIENCE AND RANDOM ACCESS IN VIDEO COMMUNICATIONS SYSTEMS," and PCT/US07/063335, "SYSTEM AND METHOD FOR PROVIDING 30 ERROR RESILIENCE, RANDOM ACCESS, AND RATE CONTROL IN SCALABLE VIDEO COMMUNICATIONS." Among other things, these patent applications disclose the concept of LR pictures, i.e., pictures that constitute the lowest temporal layer of a scalably coded video signal (at the lowest spatial or quality resolution) and which are transmitted reliably from a sender to a receiver. Reliable NY02:608243.1 5 WO 2008/086423 PCT/US2008/050640 transmission of the LR pictures ensures a minimum level of quality at a receiving decoder. A receiver can immediately detect if an LR picture has been lost and take steps to obtain the lost picture (e.g., by requesting its retransmission from the sender) using, for example, a "key picture indices" mechanism, which is also disclosed in 5 International patent application No. PCTIUS06/061815. It is noted that the sender and receiver are not necessarily the encoder and decoder, respectively, but may be a Scalable Video Communication Server (SVCS) as disclosed in commonly assigned International patent application No. PCT/US06/028366, a Compositing SVCS (CSVCS) as disclosed in commonly assigned International patent application No. 10 PCT/US06/62569, or a Multicast SVCS (MSVCS) as disclosed in commonly assigned International patent application No. PCT/US07/80089. [00171 A potential limitation of the systems and methods described in International patent application No. PCT/US06/061815 occurs when the lowest temporal level pictures are transported over more than one packets. This may occur, for example, in 15 coding high-definition video, where each frame may be transported using more than one transport-layer packets, or when a picture is coded using more than one slices and each slice is transported in its own packet. In these cases, all packets belonging to the same frame will have the same key picture index. If all slices are lost due to packet losses in the network, then a receiver can properly detect the loss of the entire picture 20 and initiate corrective action. If, however, few or all of the slices are received, then a receiver can not immediately infer if the received slices contain the entire or only a partial picture, unless it proceeds to decode the slice data. This inference is straightforward in a receiver that decodes the received data, but it presents significant complexity for an intermediate receiver (e.g., an SVCS, CSVCS, or MSCVS, or any 25 Media-Aware Network Element - MANE) that is normally not equipped to perform decoding of the video data. 100181 Consideration is now being given to improving error resilience to the coded bitstreams in video communications systems. Attention is directed towards developing error resilience techniques which have a minimal impact on end-to-end 30 delay and the bandwidth used by the system, and address the possibility of fragmentation of coded video data in multiple slices. Desirable error resilience techniques will be applicable to both scalable and single-layer video coding. NY02:608243.1 6 WO 2008/086423 PCT/US2008/050640 SUMMARY OF THE INVENTION [00191 The present invention provides systems and methods to increase error resilience in video communication systems based on single-layer as well as scalable video coding. Specifically, the present invention provides a mechanism for a receiver 5 to detect if portions of a picture that is intended to be transmitted reliably have been lost due to packet losses, so that corrective action can be initiated with minimal delay. Specific techniques are provided for transmission over RTP as well as when using H.264 Annex G (SVC) NAL units. 10 BRIEF DESCRIPTION OF THE DRAWINGS [0020] FIG. 1 is a block diagram illustrating an exemplary video conferencing system for delivering scalably coded video data, in accordance with the principles of the present invention; [0021] FIG. 2 is a block diagram illustrating an exemplary end-user terminal 15 compatible with the use of single layer video coding, in accordance with the principles of the present invention; [00221 FIG. 3 is a block diagram illustrating an exemplary end-user terminal compatible with the use of scalable or simulcast coding, in accordance with the principles of the present invention; 20 100231 FIG. 4 is a block diagram illustrating the internal switching structure of a multipoint SVCS, in accordance with the principles of the present invention; 100241 FIG. 5 is a block diagram illustrating the principles of operation of an SVCS; 25 100251 FIG. 6 is a block diagram illustrating the structure of an exemplary video encoder, in accordance with the principles of the present invention; [0026] FIG. 7 is a block diagram illustrating an exemplary architecture of a video encoder for encoding base and temporal enhancement layers, in accordance with the principles of the present invention; 30 [00271 FIG. 8 is a block diagram illustrating an exemplary architecture of a video encoder for a spatial enhancement layer, in accordance with the principles of the present invention; 100281 FIG. 9 is a block diagram illustrating an exemplary layered picture coding structure, in accordance with the principles of the present invention; NY02:608243.1 7 WO 2008/086423 PCT/US2008/050640 [00291 FIG. 10 is a block diagram illustrating another exemplary layered picture coding structure, in accordance with the principles of the present invention; [00301 FIG. 11 is a block diagram illustrating an exemplary picture coding structure including temporal and spatial scalability, in accordance with the principles of the 5 present invention; [00311 FIG. 12 is a block diagram illustrating an exemplary layered picture coding structure used for error resilient video communications, in accordance with the principles of the present invention; 100321 FIG. 13 is a block diagram illustrating an exemplary picture coding structure 10 used for error resilient video communications with spatial/quality scalability, in accordance with the principles of the present invention. [0033] FIG. 14 is a block diagram illustrating an exemplary architecture of the transmitting terminal's LRP (Snd) module when the R-packets technique is used for transmission over RTP, in accordance with the principles of the present invention. 15 [00341 FIG. 15 is a block diagram illustrating an exemplary architecture of the receiving terminal's LRP (Rcv) module when the R-packets technique is used for transmission over RTP, in accordance with the principles of the present invention. [0035] FIG. 16 is a block diagram illustrating an exemplary architecture of the server's LRP Snd and Rcv modules when the R-packets technique is used for 20 transmission over RTP, in accordance with the principles of the present invention. [00361 FIG. 17 illustrates an exemplary structure for the named RTP header extension for RTP packets, in accordance with the principles of the present invention. [00371 FIG. 18 illustrates an exemplary structure for the feedback control information field of RNACK packets, in accordance with the principles of the present invention. 25 [00381 FIG. 19 illustrates a modified H.264 Annex G (SVC) NAL header extension syntax with frame indices and start/end flags, in accordance with the principles of the present invention. [00391 Throughout the figures the same reference numerals and characters, unless otherwise stated, are used to denote like features, elements, components or portions of 30 the illustrated embodiments. Moreover, while the present invention will now be described in detail with reference to the Figures, it is done so in connection with the illustrative embodiments. NY02:608243.1 8 WO 2008/086423 PCT/US2008/050640 DETAILED DESCRIPTION OF THE INVENTION [0040] The present invention provides systems and methods for error resilient transmission in video communication systems. The mechanisms are compatible with scalable video coding techniques as well as single-layer and simulcast video coding 5 with temporal scalability, which may be used in video communication systems. 10041] The system and methods involve designating a set of video frames or pictures in a video signal transmission for reliable or guaranteed delivery to receivers. Reliable delivery of the designated set video frames may be accomplished by using secure or high reliability links, or by retransmission techniques. The reliably 10 delivered video frames are used as reference pictures for resynchronization of receivers with the transmitted video signal after error incidence or for random access. [00421 In a preferred embodiment, an exemplary video communication system may be a multi-point videoconferencing system 10 operated over a packet-based network. (See e.g., FIG. 1). Multi-point videoconferencing system may include optional 15 bridges 120a and 120b (e.g., Multipoint Control Unit (MCU) or Scalable Video Communication Server (SVCS)) to mediate scalable multilayer or single layer video communications between endpoints (e.g., users 1-k and 1 -m) over the network. The operation of the exemplary video communication system is the same and as advantageous for a point-to-point connection with or without the use of optional 20 bridges 120a and 120b. [0043] A detailed description of scalable video coding techniques and videoconferencing systems based on scalable video coding is provided in commonly assigned International patent application No. PCT/US06/028365 "SYSTEM AND METHOD FOR SCALABLE AND LOW-DELAY VIDEOCONFERENCING 25 USING SCALABLE VIDEO CODING", No. PCT/US06/028266 "SYSTEM AND METHOD FOR A CONFERENCE SERVER ARCHITECTURE FOR LOW DELAY AND DISTRIBUTED CONFERENCING APPLICATIONS", No. PCT/US/06/062569 "SYSTEM AND METHOD FOR VIDEOCONFERENCING USING SCALABLE VIDEO CODING AND COMPOSITING SCALABLE VIDEO 30 SERVERS", and No. PCT/US07/80089 "SYSTEM AND METHOD FOR MULTIPOINT CONFERENCING WITH SCALABLE VIDEO CODING SERVERS AND MULTICAST". Further, descriptions of error resilience, random access, and rate control techniques are provided in commonly assigned International patent applications No. PCT/US06/061815 "SYSTEMS AND METHODS FOR ERROR NY02:608243.1 9 WO 2008/086423 PCT/US2008/050640 RESILIENCE AND RANDOM ACCESS IN VIDEO COMMUNICATION SYSTEMS" and No. PCTIUS07/063335 "SYSTEM AND METHOD FOR PROVIDING ERROR RESILIENCE, RANDOM ACCESS, AND RATE CONTROL IN SCALABLE VIDEO COMMUNICATIONS". All of the aforementioned 5 International patent applications are incorporated by reference herein in their entireties. The systems and methods of the present invention improve upon the systems and methods described in International patent application No. PCTIUS06/61815. [00441 FIG. I shows the general structure of a videoconferencing system 10. 10 Videoconferencing system 10 includes a plurality of end-user terminals (e.g., users I k and users 1-m) that are linked over a network 100 via LANS I and 2 and servers 120a and 120b. The servers may be traditional MCUs, Scalable Video Communication Servers (SVCS), Compositing Scalable Video Communication Servers (CSVCS), or Multicast Scalable Video Communication Servers (MSVCS). 15 The latter servers have the same purpose as traditional MCUs, but with significantly reduced complexity and improved functionality. (See e.g., International patent application No. PCT/US06/28366). In the description herein, the term "server" may be used generically to refer to any of the SVCS types. [00451 FIG. 2 shows the architecture of an end-user terminal 140, which is designed 20 for use with videoconferencing systems (e.g., system 100) based on single layer coding. Similarly, FIG. 3 shows the architecture of an end-user terminal 140, which is designed for use with videoconferencing systems (e.g., system 10) based on multi layer coding. Terminal 140 includes human interface input/output devices (e.g., a camera 210A, a microphone 210B, a video display 250C, a speaker 250D), and one or 25 more network interface controller cards (NICs) 230 coupled to input and output signal multiplexer and demultiplexer units (e.g., packet MUX 220A and packet DMUX 220B). NIC 230 may be a standard hardware component, such as an Ethernet LAN adapter, or any other suitable network interface device, or a combination thereof. 30 [00461 Camera 210A and microphone 210B are designed to capture participant video and audio signals, respectively, for transmission to other conferencing participants. Conversely, video display 250C and speaker 250D are designed to display and play back video and audio signals received from other participants, respectively. Video display 250C may also be configured to optionally display NY02:608243.1 10 WO 2008/086423 PCT/US2008/050640 participant/terminal 140's own video. Camera 210A and microphone 210B outputs are coupled to video and audio encoders 210G and 210H via analog-to-digital converters 210E and 210F, respectively. Video and audio encoders 210G and 210H are designed to compress input video and audio digital signals in order to reduce the 5 bandwidths necessary for transmission of the signals over the electronic communications network. The input video signal may be live, or pre-recorded and stored video signals. The encoders compress the local digital signals in order to minimize the bandwidth necessary for transmission of the signals. [0047] In an exemplary embodiment of the present invention, the audio signal may be 10 encoded using any suitable technique known in the art (e.g., G.711, G.729, G.729EV, MPEG-1, etc.). In a preferred embodiment of the present invention, the scalable audio codec G.729EV is employed by audio encoder 21 OG to encode audio signals. The output of audio encoder 21 OG is sent to multiplexer MUX 220A for transmission over network 100 via NIC 230. 15 100481 Packet MUX 220A may perform traditional multiplexing using the RTP protocol. Packet MUX 220A may also perform any related Quality of Service (QoS) processing that may be offered by network 100. Each stream of data from terminal 140 is transmitted in its own virtual channel or "port number" in IP terminology. [00491 FIG. 3 shows the end-user terminal 140, which is configured for use with 20 videoconferencing systems in which scalable or simulcast video coding is used. In this case, video encoder 210G has multiple outputs. FIG. 3 shows, for example, two layer outputs, which are labeled as "base" and "enhancement". The outputs of terminal 140 (i.e., the single layer output (FIG. 2) or the multiple layer outputs (FIG. 3)) are connected to Packet MUX 220A via an LRP processing module 270A. LRP 25 processing module 270A (and modules 270B) are designed for error resilient communications ("error resilience LRP operation") by processing transmissions of special types of frames (e.g. "R" frames, FIGS. 12 and 13) as well as any other information that requires reliable transmission such as video sequence header data. If video encoder 21 OG produces more than one enhancement layer output, then each 30 may be connected to LRP processing module 270A in the same manner as shown in FIG. 3. Similarly, in this case, the additional enhancement layers will be provided to video decoders 230A via LRP processing modules 270B. Alternatively, one or more of the enhancement layer outputs may be directly connected to Packet MUX 220A, and not via LRP processing module 270A. NY02:608243.1 1I WO 2008/086423 PCT/US2008/050640 [0050] Terminal 140 also may be configured with a set of video and audio decoder pairs 230A and 230B, with one pair for each participant that is seen or heard at terminal 140 in a videoconference. It will be understood that although several instances of decoders 230A and 230B are shown in FIGS. 2 and 3, it is possible to use 5 a single pair of decoders 230A and 230B to sequentially process signals from multiple participants. Thus, terminal 140 may be configured with a single pair or a fewer number of pairs of decoders 230A and 230B than the number of participants. [0051] The outputs of audio decoders 230B are connected to an audio mixer 240, which in turn is connected with a digital-to-analog converter (DA/C) 250A, which 10 drives speaker 250B. The audio mixer combines the individual signals into a single output signal for playback. If the audio signals arrive pre-mixed, then audio mixer 240 may not be required. Similarly, the outputs of video decoders 230A may be combined in the frame buffer 250B of video display 250C via compositor 260. Compositor 260 is designed to position each decoded picture at an appropriate area of 15 the output picture display. For example, if the display is split into four smaller areas, then compositor 260 obtains pixel data from each of video decoders 230A and places it in the appropriate frame buffer position (e.g., by filling up the lower right picture). To avoid double buffering (e.g., once at the output of decoder 230A and once at frame buffer 250B), compositor 260 may be implemented as an address generator that drives 20 the placement of the output pixels of decoder 230A. Other techniques for optimizing the placement of the individual video outputs to display 250C can also be used to similar effect. [00521 For example, in the H.264 standard specification, it is possible to combine views of multiple participants in a single coded picture by using a flexible macroblock 25 ordering (FMO) scheme. In this scheme, each participant occupies a portion of the coded image, comprising one of its slices. Conceptually, a single decoder can be used to decode all participant signals. However, from a practical view, the receiver/terminal will have to decode four smaller independently coded slices. Thus, terminal 140 shown in FIG. 2 and FIG. 3 with decoders 230A may be used in 30 applications of the H.264 specification. It is noted that the server for forwarding slices is an CSVCS. [0053] In terminal 140, demultiplexer DMUX 220B receives packets from NIC 320 and redirects them to the appropriate decoder unit 230A via receiving LRP modules NY02:608243.1 12 WO 2008/086423 PCT/US2008/050640 270B as shown in FIGS. 2 and 3. LRP modules 270B at the inputs of video decoders 230A terminate the error resilience LRP at the receiving terminal end. 100541 The MCU or SERVER CONTROL block 280 coordinates the interaction between the server (SVCS/CSVCS) and the end-user terminals. In a point-to-point 5 communication system without intermediate servers, the SERVER CONTROL block is not needed. Similarly, in non-conferencing applications, only a single decoder is needed at a receiving end-user terminal. For applications involving stored video (e.g., broadcast of pre-recorded, pre-coded material), the transmitting end-user terminal may not involve the entire functionality of the audio and video encoding blocks or of 10 all the terminal blocks preceding them (e.g., camera, microphone, etc.). Specifically, only the portions related to selective transmission of video packets, as explained below, need to be provided. [00551 It will be understood that the various components of terminal 140 may be physically separate software and hardware devices or units that are interconnected to 15 each other (e.g., integrated in a personal computer), or may be any combination thereof. [00561 FIG. 4 shows the structure of an exemplary SVCS 400 for use in error resilient processing applications. The core of the SVCS 400 is a switch 410 that determines which packet from each of the possible sources is transmitted to which destination 20 and over what channel. (See e.g., PCT/US06/028366). [00571 The principles of operation of an exemplary SVCS 400 can be understood with refrence to FIG. 5. A SVC Encoder 510 at a transmitting terminal or endpoint in this example produces three spatial layers in addition to a number of temporal layers (not shown pictorially). The individual coded video layers are transmitted from 25 the transmitting endpoint (SVC Encoder) to SVCS 400 in individual packets. SVCS 400 decides which packets to forward to each of the three recipient/decoders 520 shown, depending on network conditions or user preferences. In the example shown in FIG. 5, SVCS 400 forwards only the first and second spatial layers to SVC Decoder 520(0), all three spatial layers to SVC Decoder 520(1), and only the first 30 (base) layer to SVC Decoder 520(2). 100581 With renewed reference to FIG. 4, in addition to the switch, which is described in PCT/US06/028366, SVCS 400 includes LRP units 470A and 470B, which are disposed at the switch inputs and outputs, respectively. SVCS 400 is configured to terminate error resilience LRP processing at its incoming switch connection, and to NY02:608243.1 13 WO 2008/086423 PCT/US2008/050640 initiate error resilience LRP processing at its outgoing switch connections. In implementations of the invention using SVCS 400, error resilience LRP processing is not performed end-to-end over the network, but only over each individual connection segment (e.g., sender-to-SVCS, SVCS-to-SVCS, and SVCS-to-recipient). It will, 5 however, be understood that the inventive error resilience LRP processing may be executed in an end-to-end fashion over the network, with or without the use an SVCS. An SVCS 400 without LRP units 470A and 470B can be used for end-to-end LRP processing in networks in which an SVCS is used. Further, SVCS 400 may be equipped with more than one NIC 230, as would typically be the case if SVCS 400 10 connects users across different networks. [00591 FIG. 6 shows the architecture of an exemplary video encoder 600 that may be used for in error resilient video communication systems. Video encoder 600 may, for example, be a motion-compensated, block-based transform coder. An H.264 design is a preferred design for video encoder 600. However, other codec designs may be used. 15 For example, FIG. 7 shows the architecture of an exemplary video encoder 600' for encoding base and temporal enhancement layers based on SVC designs, whereas FIG. 8 shows the architecture of an exemplary video encoder 600" for encoding spatial enhancement layers. (See e.g., PCT/US06/28365 and PCT/US06/028366). Video encoder 600'and 600" include an optional input downsampler 640, which can be 20 utilized to reduce the input resolution (e.g., from CIF to CIF) in systems using spatial scalability. 10060] FIG. 6 also shows a coding process, which may be implemented using video encoder 600. ENC REF CONTROL 620 in encoder 600 is used to create a "threaded" coding structure. (See e.g., PCT/US06/28365 and PCT/US06/028366). Standard 25 block-based motion compensated codecs have a regular structure of I, P, and B frames. For example, in a picture sequence (in display order) such as IBBPBBP, the 'P' frames are predicted from the previous P or I frame, whereas the B pictures are predicted using both the previous and next P or I frame. Although the number of B pictures between successive I or P pictures can vary, as can the rate in which I 30 pictures appear, it is not possible, for example, for a P picture to use as a reference for prediction another P picture that is earlier in time than the most recent one. H.264 is an exception in that the encoder and decoder maintain two reference picture lists. It is possible to select which pictures are used for references and also which references are used for a particular picture that is to be coded. The FRAME BUFFERS block 610 in NY02:608243.1 14 WO 2008/086423 PCT/US2008/050640 FIG. 6 represents the memory that stores the reference picture list(s), whereas ENC REF CONTROL 620 determines - at the encoder side - which reference picture is to be used for the current picture. 100611 The operation of ENC REF CONTROL 520 can be better understood with 5 reference to FIG. 9, which shows an exemplary layered picture coding structure 900. In order to enable multiple temporal resolutions, the codec used in the video communications system may generate a number of separate picture "threads." A thread at a given level is defined as a sequence of pictures that are motion compensated using pictures either from the same thread, or pictures from a lower 10 level thread. The use of threads allows the implementation of temporal scalability, since one can eliminate any number of top-level threads without affecting the decoding process of the remaining threads. [00621 In a preferred embodiment of the present invention, a coding structure with a set of three threads is used (e.g., structure 900, FIG. 9). In FIG. 9, the letter 'L' in the 15 picture labels indicates an arbitrary scalability layer. The numbers (0, 1 and 2) following L identify the temporal layer, for example, with "0" corresponding to the lowest, or coarsest temporal layer and "2" corresponding the highest or finest temporal layer. The arrows shown in FIG. 9 indicate the direction, source, and target of prediction. In most applications only P pictures will be used, as the use of B 20 pictures increases the coding delay by the time it takes to capture and encode the reference pictures used for the B pictures. However, in applications that are not delay sensitive, some or all of the pictures could be B pictures with the possible exception of LO pictures. Similarly, the LO pictures may be I pictures forming a traditional group of pictures (GOP). 25 [00631 With continued reference to FIG. 9, layer LO is simply a series of regular P pictures spaced four pictures apart. Layer LI has the same frame rate as LO, but prediction is only allowed from the previous LO frame. Layer L2 frames are predicted from the most recent LO or LI frame. LO provides one fourth (1:4) of the full temporal resolution, LI doubles the LO frame rate (1:2), and L2 doubles the LO+Ll 30 frame rate (1:1). 100641 More or fewer layers than the three LO, LI and L2 layers discussed above may be similarly constructed in coding structures designed to accommodate the different bandwidth/scalability requirements of specific implementations of the present invention. FIG. 11: shows an example of a threaded coding structure 1000 with only NY02:608243.1 15 WO 2008/086423 PCT/US2008/050640 two layers LO and LI. Further, FIG. I1 shows an example of a threaded coding structure 1100 for spatial scalability. Coding structure 1100 includes threads for enhancement layers, which are denoted by the letter 'S'. It will be noted that the enhancement layer frames may have a different threading structure than the base layer 5 frames. [00651 Video encoder 600' (FIG. 7) for encoding temporal layers may be augmented to encode spatial and/or quality enhancement layers. (See e.g., PCT/US06/028365 and PCT/US06/028366). FIG. 8 shows an exemplary encoder 600" for the spatial enhancement layer. The structure and functions of encoder 600" are similar to that of 10 the base layer codec 600', except in that base layer information is also available to the encoder 600". This information may consist of motion vector data, macroblock mode data, coded prediction error data, or reconstructed pixel data. Encoder 600" can re use some or all of this data in order to make coding decisions for the enhancement layers S. The data has to be scaled to the target resolution of the enhancement layer 15 (e.g., by factor of 2 if the base layer is QCIF and the enhancement layer is CIF). Although spatial scalability typically requires two coding loops to be maintained, it is possible,for example, under theH.264 Annex G (SVC) draft standard, to perform single-loop decoding by limiting the data of the base layer that is used for enhancement layer coding to only values that are computable from the information 20 encoded in the current picture's base layer. (See e.g., T. Wiegand, G. Sullivan, J. Reichel, H. Schwarz, M. Wien, eds., "Joint Draft 8 of SVC Amendment," Joint Video Team, Doc. JVT-U201, Hangzhou, October 2006, incorporated herein by reference in its entirety). For example, if a base layer macroblock is inter-coded, then the enhancement layer cannot use the reconstructed pixels of that macroblock as a 25 basis for prediction. It can, however, use its motion vectors and the prediction error values since they are obtainable by just decoding the information contained in the current base layer picture. Single-loop decoding is desirable since the complexity of the decoder is significantly decreased. [00661 Quality or SNR scalability enhancement layer codecs may be constructed in 30 the manner as spatial scalability codecs. For quality scalability, instead of building the enhancement layer on a higher resolution version of the input, the codecs code the residual prediction error at the same spatial resolution. As with spatial scalability, all the macroblock data of the base layer can be re-used at the enhancement layer, in either single- or dual-loop coding configurations. For brevity, the description herein NY02:608243.1 16 WO 2008/086423 PCT/US2008/050640 is generally directed to techniques using spatial scalability. It will, however, be understood that the same techniques are applicable to quality scalability. 100671 International patent application PCT/US06/028365 describes the distinct advantages that threading coding structures (e.g., coding structure 900) have in terms 5 of their robustness to the presence of transmission errors. In traditional state-of-the art video codecs based on motion-compensated prediction, temporal dependency is inherent. Any packet losses at a given picture not only affects the quality of that particular picture, but also affects all future pictures for which the given picture acts as a reference, either directly or indirectly. This is because the reference frame that 10 the decoder can construct for future predictions will not be the same as the one used at the encoder. The ensuing difference, or drift, can have tremendous impact on the visual quality produced by traditional state-of-the-art video codecs. 100681 In contrast, the threading structure shown in FIG. 9 creates three self contained threads or chains of dependencies. A packet loss occurring for an L2 15 picture will only affect L2 pictures; the LO and Ll pictures can still be decoded and displayed. Similarly, a packet loss occurring at an LI picture will only affect LI and L2 pictures; the LO pictures can still be decoded and displayed. Further, threading structures may be created to include threads or chains of dependencies for S pictures (e.g., FIG. 11). The exemplary S packets threading structure 1100 shown in FIG. 11 20 has similar properties as the L picture threading structure 900 shown in FIG. 9. A loss occurring at an S2 picture only affects the particular picture, whereas a loss at an SI picture will also affect the following S2 picture. In either case, drift will terminate upon decoding of the next SO picture. [00691 With renewed reference to FIG. 9, a packet loss occurring at an LO picture can 25 be catastrophic in terms of picture quality, since all picture types will be affected. As previously noted, a traditional solution to this problem is to periodically code LO pictures as intra or I pictures. However, the bandwidth overhead for implementing this solution can be considerable as the I pictures are typically 3-6 times larger than P pictures. Furthermore, the packet loss, which gives rise to the need to use an I picture, 30 is often the result of network congestion. Attempting to send an I picture over the network to remedy the packet loss only exacerbates the congestion problem. [00701 If the base layer LO and some enhancement layer pictures are transmitted in a way that guarantees their delivery, the remaining layers can be transmitted on a best effort basis without catastrophic results in the case of a packet loss. Such guaranteed NY02:608243.l 17 WO 2008/086423 PCT/US2008/050640 transmissions can be performed using known techniques such as DiffServ, and FEC, etc. In the description herein, reference also may be made to a High Reliability Channel (HRC) and Low Reliability Channel (LRC) as the two actual or virtual channels that offer such differentiated quality of service (FIG. 1). (See e.g., 5 PCT/US06/028365 and PCT/US06/028366). In video communication systems which use scalable video coded structures (e.g., structure 1100, FIG. 11), layers LO-L2 and SO may, for example, be reliably transmitted on the HRC, while SI and S2 are transmitted on the LRC. Although the loss of an SI or S2 packet would cause limited drift, it is still desirable to be able to conceal as much of the loss of information as 10 possible. 10071] The error resilience techniques described in International patent application No. PCT/US06/061815 overcome the limitations of traditional techniques for compensating for packet loss by utilizing reliable transmission of a subset of the LO layer or the entire LO layer. Error resilience or reliability is ensured by 15 retransmissions. These error resilience techniques are designed not merely to recover a lost picture for display purposes, but are designed to create the correct reference picture for the decoding of future pictures that depend on the one that was contained (in whole or in part) in a lost packet. The present invention improves on these techniques by ensuring their proper operation in the case where pictures are 20 transmitted over multiple transport layer (e.g., RTP) packets. In system implementations of the present invention, the reliable transmission of the LO pictures may be performed by LRP modules (e.g., FIG. 2, modules 270A and 270B, and FIG. 4, modules 470A and 470B) using positive or negative acknowledgments between the sending and receiving counterparts according to a suitable protection protocol. 25 10072] FIG. 12 shows an exemplary picture coding structure 1200 (which is also described in International patent application No. PCT/US06/061815) in which the LO base and L 1 -L2 temporal enhancement layers are coupled with at least one reliably transmitted base layer picture for error resilient video communications. In coding structure 1200, in addition to conventional base and enhancement picture types that 30 are labeled as LO-L2 pictures, there is a new picture type called LR ('R' for reliable). It is noted that in coding structure 1200 shown in FIG. 12, the layers LR and LO-L2 can equivalently have been labeled as LO-L3, respectively, since the LR pictures always are the lowest temporal layer of the coded video signal. In accordance with NY02:608243.1 18 WO 2008/086423 PCT/US2008/050640 the present invention for error resilient video communications, the LR pictures, which may be P pictures, are designated to be reliably delivered to receiver destinations. [0073] The operation of the inventive error resilient techniques can be understood by consideration of an example in which one of the LO pictures is damaged or lost due to 5 packet loss. As previously noted, in traditional communication systems the effect of loss of the LO picture is severe on all subsequent LO-L2 pictures. With the picture coding structure 1200, the next "reliably-delivered" LR picture after a lost LO picture offers a resynchronization point, after which point the receiver/decoder can continue decoding and display without distortion. 10 [0074] In the coding structure 1200 shown in FIG. 12, the temporal distance between the LR pictures is, for example, 12 frames. The reliable delivery of the LR pictures exploits the fact that P pictures with very long temporal distances (6 frames or more) are about half the size of an I picture, and that the reliable delivery is not intended to ensure timely display of the relevant picture, but instead is intended for creation of a 15 suitable reference picture for future use. As a result the delivery of an LR picture can be accomplished by a very slight bandwidth increase in the system during the period between successive LR pictures. [00751 Coding structure 1200 may be implemented using the existing H.264 standard under which the LR pictures may, for example, be stored at a decoder as long-term 20 reference pictures and be replaced using MMCO commands. [00761 FIG. 13 shows an exemplary picture coding structure 1300 where the LR picture concept is applied to enhancement layer pictures (either spatial or quality scalability). Here, the pictures to be reliably transmitted are labeled SR, and as with LR pictures, they constitute the lowest temporal layer of the spatial or quality 25 enhancement layer. 100771 It is noted that although the LR pictures concept is generally described herein for purposes of illustration, as applied to the lowest temporal layer of the coded video signal, the concept can also be extended and applied to additional layers in accordance with the principles of the present invention. This extended application will result in 30 additional pictures being transported in a reliable fashion. For example, with reference to FIG. 12 in addition to the LR pictures, the LO pictures could also be included in the reliable (re)transmission mechanism. Similarly, pictures of any spatial/quality enhancement layers (from the lowest or additional temporal layers) may be included. Further, video sequence header or other data may be treated or NY02:608243.1 19 WO 2008/086423 PCT/US2008/050640 considered to be equivalent to LR pictures in the system so that they (header or other data) are reliably transmitted. In the following, for simplicity in description we assume that only LR pictures are reliably transmitted, unless explicitly specified otherwise. However, it will be readily understood that additional layers or data can be 5 reliably transmitted in exactly the same way. [00781 It is desirable that the bandwidth overhead for the reliable delivery of the LR frames is zero or negligible, when there are no packet losses. This implies that a dynamic, closed-loop algorithm should be used for the reliable delivery mechanism. It may also be possible to use open loop algorithms, where, for example, an LR frame 10 is retransmitted proactively a number of times. [00791 International patent application No. PCT/US06/061825 describes several mechanisms to notify a sender (e.g., SENDER, SVCS 1, or SVCS2) that a particular LR picture has been received by an intended receiver, and also techniques for dynamically establishing LR pictures. Using RTCP or other feedback mechanisms, 15 the sender can be notified that a particular receiver is experiencing lost packets using, for example, the positive and negative acknowledgment techniques described therein. The feedback can be as detailed as individual ACK/NACK messages for each individual packet. Use of feedback enables the encoder to calculate (exactly or approximately) the state of the decoder(s), and act accordingly. This feedback is 20 generated and collected byReliability and Random access Control (RRC) modules 530 (FIG. 6). 100801 An important aspect of these sender-notification mechanisms is the technique by which a receiver (receiving endpoint or SVCS) detects the loss of an LR picture with minimal delay. The technique used in the aforementioned patent application 25 relies on LR picture numbers and picture number references. [00811 The LR picture numbers technique operates by assigning sequential numbers to LR pictures, which are carried together with the LR picture packets. The receiver maintains a list of the numbers of the LR pictures it has received. Non-LR pictures, on the other hand, contain the sequence number of the most recent LR picture in 30 decoding order. This sequence number reference allows a receiver to detect a lost LR picture even before receipt of the following LR picture. When a receiver receives an LR picture, it can detect if it has lost (i.e. not received) one or more of the previous LR pictures by comparing the picture number of the received LR picture with the list of picture numbers it maintains. The picture number of the received LR picture NY02:608243.1 20 WO 2008/086423 PCT/US2008/050640 should be one more than that of the previous one, or 0 if the count has restarted. When a receiver receives a non-LR picture, it tests to see if the referenced LR picture number is present in its number list. If it is not, the referenced LR picture is assumed to be lost and corrective action may be initiated (e.g., a NACK message is transmitted 5 back to the sender). It is noted that detection of lost LR pictures using the LR picture number technique can be performed both at a receiving endpoint as well as an intermediate SVCS. The operation is performed, e.g., at the LRP (Rcv) module 270B in FIG. 2 and FIG. 3, or 470B in FIG. 4. 100821 A potential limitation of the picture numbers technique can manifest itself 10 when a single LR picture is transported using more than one packet. Such transport may occur, for example, if encoding is done using multiple slices, but can occur whenever the coded bits of a given picture exceed the maximum transport layer packet size. When multiple packets are used to transport a picture, all the packets will have the same picture index value since they belong to the same picture. If all such 15 packets are lost in transit, then the receiver can properly detect the loss upon the next successful reception of picture data. If, however, in the case of partial data reception in which only some of the picture's packets are lost (but a few of the packets are received) a receiver will not be able to detect the loss, unless it examines the data to determine if all macroblocks contained in the picture are included in the received data. 20 This determination, which requires that the receiver parse coded video data, is a computationally demanding task. In the H.264 or H.264 SVC cases, for example, determining if a set of slices includes data for an entire packet requires parsing of the entire slice header. The parsing operation can be performed in a receiver that is equipped with a decoder. However, such is not the case when the receiver is an 25 SVCS or any other type of MANE. [00831 To address error resilience in the case of partial data reception, it is noted that a receiver can detect packet losses using the sequence number associated with every packet (e.g., RTP sequence numbers in a preferred embodiment where RTP is used as the transport protocol). Successive packets of an LR picture will contain successive 30 RTP sequence numbers. If partial data is received, a receiver knows from the gap in the received RTP sequence numbers that some data was lost, but it cannot determine if the lost data correspond to portion of the LR picture or data from a following picture. As a result, from the RTP sequence numbers alone, it is not possible to detect if the received data contains the entire LR picture. To enable a receiver to detect NY02:608243.1 21 WO 2008/086423 PCT/US2008/050640 receipt of the entire picture, the present invention introduces two flags, a start bit flag and an end bit flag, that respectively indicate the first and last packets containing data of an LR picture. [0084] Upon reception of packet of an LR picture, a receiver can examine its RTP 5 sequence number and check if it has received all previous packets with successively smaller RTP sequence numbers until reaching a packet that has the same picture index value and in which the 'start' bit is set. Similarly, it can continue checking that successive packets with successively larger RTP sequence numbers are received, until reaching a packet that has the same picture index value and in which the 'last' bit set. 10 With this modification, frame indices can be used to detect losses of lowest temporal level pictures in both cases when no data is received and when partial data is received. [00851 The two flags also may be introduced in temporal levels higher than the lowest temporal level to enable integrity detection for pictures belonging to higher temporal levels. This - coupled with RTP sequence numbers - would allow a receiver to 15 quickly determine if it has received all needed data for a particular picture, regardless of its temporal level. 100861 It is noted that RTP marker bit has a usual definition for use in video transport as "the last packet of a picture." Use of the RTP marker bit may be considered in lieu of the 'last' flag. However, in the context of SVC, such use of the RTP marker bit is 20 not sufficient to solve the problem this invention addresses, since a picture may include several 'pictures' (base and enhancement layers). Furthermore, such a change would create problems in existing RTP systems that already incorporate the usual interpretation of the RTP marker bit. [00871 Two different embodiments of the modified LR picture numbering technique 25 are described herein. One embodiment (hereinafter referred to as the 'R packets' technique) is appropriate when the RTP protocol is used by the system for transmission. The other embodiment is applicable when the H.264 SVC draft standard is used for the system. [0088] For the R packets technique, assume that the RTP protocol (over UDP and IP) 30 is used for communication between two terminals, possibly through one or more intermediate servers. Note that the media transmitting terminal may perform real time encoding, or may access media data from local or other storage (RAM, hard disk, a storage area network, a file server, etc.). Similarly, the receiving terminal may perform real-time decoding, and it may be storing the received data in local or other NY02:608243.1 22 WO 2008/086423 PCT/US2008/050640 storage for future playback, or both. For the description herein, it is assumed, without limitation, that real-time encoding and decoding are taking place. [00891 FIG. 14 shows the architecture of the transmitting terminal's LRP Snd module (e.g., module270A, FIG. 2). LRP Snd module includes a packet processor (R-Packet 5 Controller 1610) with local storage (e.g.,. buffer 1605)for packets that may require retransmission). R-Packet Controller 1610 marks the R packets and also responds to RNACKs. The R-Packet Controller is connected to a multiplexer MUX 1620 and a demultiplexer DMUX 1630 implementing the RTPIUDP/IP protocol stack. Although MUX 1620 and DMUX 1630 are shown in FIG. 14 as separate entities, they may be 10 combined in the same unit. MUX 1620 and DMUX 1630 are connected to one or more network interface controllers (NICs) which provide the physical layer interface. In a preferred embodiment, the NIC is an Ethernet adapter, but any other NICs can be used as will be obvious to persons skilled in the art. 100901 Similarly, FIG. 15 shows an exemplary architecture of the receiving terminal's 15 LRP Rcv module (e.g., module 270B, FIG. 2). The R-Packet Controller here (e.g., controller 1610') is responsible for packet loss detection and generation of appropriate NACK messages. Further, FIG. 16 shows the structure of the server's LRP Snd and Rcv modules (e.g., modules 420A and 420B, FIG. 4), which may be the same as components of a receiving terminal and that of a transmitting terminal connected 20 back-to-back. [0091] In a preferred embodiment, the transmitting terminal packetizes media data according to the RTP specification. It is noted that that although different packetization (called "payload") formats are defined for RTP, they all share the same common header. This invention introduces a named header extension mechanism 25 (see Singer, D., "A general mechanism for RTP Header Extensions," draft-ietf-avt rtp-hdrext-0I (work in progress), February 2006) for RTP packets so that R packets can be properly handled. 100921 According to the present invention, in an RTP session containing R packets, individual packets are marked with the named header extension mechanism. The R 30 packet header extension element identifies both R packets themselves and previously sent R packets. This header extension element has the name "com.layeredmedia.avt.r-packet/200606". Every R packet includes, and every non-R packet should include, a header extension element of this form. NY02:608243.1 23 WO 2008/086423 PCT/US2008/050640 [00931 FIG. 17 shows an exemplary data field format of the inventive named header extension, in which the fields are defined as follows. [00941 ID: 4 bits The local identifier negotiated for this header extension element, as 5 defined, for example, in Singer, D., "A general mechanism for RTP Header Extensions," draft-ietf-avt-rtp-hdrext-0 1 (work in progress), February 2006. [00951 Length (len): 4 bits The length minus one of the data bytes of this header extension 10 element, not counting the header byte (ID and len). This will have the value 6 if the second word (the superseded range) is present, and 2 if it is not. Thus, its value must either be 2 or 6. [00961 R: I bit A bit indicating that the packet containing this header extension 15 element is an R packet in series SER with R sequence number RSEQ. If this bit is not set, the header extension element instead indicates that the media stream's most recent R packet in series SER had R sequence number RSEQ. If this bit is not set, the superseded range should not be present (i.e. the len field should be 2) and must be ignored if present. 20 100971 Reserved, Must Be Zero (0): 1 bit Reserved bit. This must be set to zero on transmit and ignored on receive. 100981 Start (S): 1 bit This must be set to one if this is the first packet containing data from a 25 given picture. [00991 End (E): 1 bit This must be set to one if this is the last packet containing data from a given picture. 1001001 Series ID (SER): 4 bits 30 An identifier of which series of R packets is being described by this header extension element. If a media encoder is describing only a single series of R packets, this should have the value 0. For example, using the scalable video picture coding structure depicted in FIG. 13, L packets (base spatial enhancement layer, all threads) would have SER NY02:608243.1 24 WO 2008/086423 PCT/US2008/050640 set to, say, 0, and S packets (spatial enhancement layer, all threads) would have SER set to 1. 1001011 R Packet Sequence Number (RSEQ): 16 bits An unsigned sequence number indicating the number of this R packet 5 within the series SER. This value is incremented by 1 (modulo 2^16) for every R packet sent in a given series. RSEQ values for separate sequences are independent. [001021 Start of Superseded Range (SUPERSEDESTART): 16 bits The R sequence number of the earliest R packet, inclusive, superseded 10 by this R packet, calculated modulo 2^16. (Since this value uses modulo arithmetic, the value RSEQ + I may be used for SUPERSEDESTART to indicate that all R packets prior to the end of the superseded range have been superseded.) This field is optional, and is only present when len=6. 15 [001031 End of Superseded Range (SUPERSEDEEND): 16 bits The R sequence number of the final R packet, inclusive, superseded by this R packet, calculated modulo 2^16. This value must lie in the closed range [SUPERSEDESTART.. RSEQ] modulo 2^16. This field is optional, and is only present when len=6. 20 [001041 The operation of an error resilient video communication system in accordance with the present invention is the same or similar to the operation described in International patent application No. PCT/US06/61815, except from the use of the 'S' and 'E' flags. These flags are used at the receiver in combination with RTP sequence numbers to detect if an LR picture has been received in its entirety (in which 25 case no corrective action is needed) or partially (in which case corrective action must be initiated). All other aspects of the system's operation including the various retransmission techniques (e.g., positive or negative acknowledgments) remain the same. [001051 An RTP packet may contain multiple R packet mark elements, so long 30 as each of these elements has a different value for SER. However, an RTP packet must not contain more than one of these header extension elements with the R bit set, i.e. an R packet may not belong to more than one series. [001061 All RTP packets in a media stream using R packets should include a mark element for all active series. NY02:608243.1 25 WO 2008/086423 PCT/US2008/050640 [00107] When the second word of this header extension element is present, it indicates that this R packet supersedes some previously-received R packets, meaning that these packets are no longer necessary in order to reconstruct stream state. This second word must only appear in a header extension element which has its R bit set. 5 [001081 An R packet can only supersede R packets in the series identified by the element's SER field. R packets cannot supersede packets in other series. [001091 It is valid for a superseded element to have SUPERSEDEEND=RSEQ. This indicates that the R packet supersedes itself, i.e., that this R packet immediately becomes irrelevant to the stream state. In practice, the 10 most common reason to do this would be to end a series; this can be done by sending an empty packet (e.g. an RTP No-op packet, see Andreasen, F., "A No-Op Payload Format for RTP," draft-ietf-avt-rtp-no-op-00 (work in progress), May 2005.) with the superseded range (SUPERSEDESTART, SUPERSEDEEND) = (RSEQ+l, RSEQ), so that the series no longer contains any non-superseded packets. 15 [001101 The first R packet sent in a series should be sent with the superseded range (SUPERSEDESTART, SUPERSEDEEND) = (RSEQ+1, RSEQ-1), to make it clear that no other R packets are present in the range. [001111 R packets may redundantly include already-superseded packets in the range of packets to be superseded. 20 [00112] The loss of R packets is detected by the receiver, and is indicated by the receiver to the sender using an RTCP feedback message. The R Packet Negative Acknowledgement (RNACK) Message is an RTCP Feedback message (see e.g., Ott, J. et al., "Extended RTP Profile for RTCP-based Feedback(RTP/AVPF)," RFC 4585, July 2006) identified, as an example, by PT=RTPFB and FMT=4. Other values can 25 be chosen, in accordance with the present invention. The FCI field must contain at least one and may contain more than one RNACK. [00113] The RNACK packet is used to indicate the loss of one or more R packets. The lost packet(s) are identified by means of a packet sequence number, the series identifier, and a bit mask. 30 [00114] The structure and semantics of the RNACK message are similar to that of the AVPF Generic NACK message. [001151 FIG. 18 shows the exemplary syntax of the RNACK Feedback Control Information (FCI) field in which individual fields are defined as follows: [001161 R Packet Sequence Number (RSEQ): 16 bits NY02:608243.1 26 WO 2008/086423 PCT/US2008/050640 The RSEQ field indicates a RSEQ value that the receiver has not received. [001171 Series ID (SER): 4 bits An identifier of which sequence of R packets is being described as 5 being lost by this header extension element. [00118] Bitmask of following Lost R Packets (BLR): 12 bits The BLR allows for reporting losses of any of the 12 R Packets immediately following the RTP packet indicated by RSEQ. Denoting the BLP's least significant bit as bit 1, and its most significant bit as bit 10 12, then bit i of the bit mask is set to 1 if the receiver has not received R packet number (RSEQ+i) in the series SER (modulo 2^16) and indicates this packet is lost; bit i is set to 0 otherwise. Note that the sender must not assume that a receiver has received an R packet because its bit mask was set to 0. For example, the least significant bit 15 of the BLR would be set to 1 if the packet corresponding to RSEQ and the following R packet in the sequence had been lost. However, the sender cannot infer that packets RSEQ+2 through RSEQ+16 have been received simply because bits 2 through 15 of the BLR are 0; all the sender knows is that the receiver has not reported them as lost at this 20 time. [001191 The structure of the RNACK message shown in FIG. 18 is identical to the one described in International patent application No. PCT/US06/061815. [001201 The second exemplary detection technique, which allows a receiver to detect that an LR picture (including SR pictures) has been lost with a minimal delay, 25 is applicable to the systems based on the H.264 SVC draft standard. In such case H.264 SVC NAL units are used as the basis for transmission. International patent application No. PCTIUS06/61815 describes how the LR picture index technique may be applied in this case as well. As with the RTP embodiment, the present invention introduces two single-bit flags to address the case where multiple packets are used for 30 transport of a given LR picture. [00121] FIG. 19 shows the structure of the inventive H.264 SVC NAL header extension modified to include the start and end flags, using as the basis the syntax of the H.264 SVC draft (see e.g., T. Wiegand, G. Sullivan, J. Reichel, H. Schwarz, M. Wien, eds., "Joint Scalable Video Model 8: Joint Draft 8 with proposed changes," NY02:608243.1 27 WO 2008/086423 PCT/US2008/050640 Joint Video Team, Doc. JVT-U202, Hangzhou, October 2006, incorporated herein by reference in its entirety). The start and end flags are the picstartflag and pic-endflag, whereas the picture index is the tl0_pic idx parameter. The dependencyid (D), temporallevel (T), and qualitylevel (Q) fields indicate points in 5 the spatial/coarse grain quality, temporal, and fine-grain quality dimensions respectively. In other words, they indicate the position of the NAL's payload in the set of resolutions provided by the scalable encoder. It is noted that the base layer in this scheme is identified by D=Q=T=0. [001221 While there have been described what are believed to be the preferred 10 embodiments of the present invention, those skilled in the art will recognize that other and further changes and modifications may be made thereto without departing from the spirit of the invention, and it is intended to claim all such changes and modifications as fall within the true scope of the invention. For example, alternative mechanisms for indicating the LR picture frame index value and referring to it in non 15 LR pictures may be used in accordance with the present invention both within an RTP transmission context and an H.264 SVC NAL transmission context. Similarly, alternative mechanisms for indicating the start and end flags may be used in both RTP and H.264 SVC. For example, the tl0_pic idx parameter and associated picstart flag and pic_end_flag parameters may be carried in an SEI message. 20 1001231 It also will be understood that the systems and methods of the present invention can be implemented using any suitable combination of hardware and software. The software (i.e., instructions) for implementing and operating the aforementioned systems and methods can be provided on computer-readable media, which can include without limitation, firmware, memory, storage devices, 25 microcontrollers, microprocessors, integrated circuits, ASICS, on-line downloadable media, and other available media. NY02:608243.1 28

Claims (17)

1. A system for media communications between a transmitting endpoint or server and one or more receiving endpoint(s) or server(s) over a packet-based communication network, the system comprising: 5 an encoder which encodes transmitted media as pictures in a threaded coding structure having a number of different layers including a lowest temporal layer, wherein each picture is associated with a picture index number wherein data corresponding to a single picture is portioned and transmitted in one or 10 more individual data packets, wherein an individual data packet packet comprises data elements that indicate: for the lowest temporal level pictures, a sequence number identifying said pictures, for other temporal level pictures, a reference to the sequence number of 15 the most recent, in decoding order, lowest temporal level picture, and for all pictures, a 'start' flag and an 'end' flag that respectively indicate if the individual data packet contains the first or last data portions of the picture, so that a receiving endpoint or server on receipt of the individual data packet can detect a loss of portions of a lowest temporal level picture by examining if all data 20 portions of the picture corresponding to the referenced picture index have been received at the receiving endpoint or server such that their packet sequence numbers contain no gaps, and where the first such data portion has the 'start' bit set and the last such data portion has have the 'end' bit set. 25

2. The system of claim I wherein the data elements additionally indicate a series number associated with each spatial or quality layer, wherein the receiving endpoint or server detects if a lowest temporal level picture of a particular spatial or quality layer is lost by determining if the picture corresponding to the referenced series number and sequence number has been received at the receiving 30 endpoint or server.

3. The system of claim 1 comprising a codec conforming to H.264, wherein the lowest temporal level pictures include pictures that are marked as long term reference pictures, and wherein the decoding of at least a portion of the received NY02:608243.1 29 WO 2008/086423 PCT/US2008/050640 media based on a lowest temporal level picture by the decoder is controlled by MMCO commands.

4. The system of claim 1 wherein the communication network uses the Internet Protocol, media transport is performed using the real-time protocol (RTP), 5 and the data elements include data to indicate presence of a lowest temporal layer frame or fragment thereof in the packet.

5. The system of claim 4 wherein a receiving endpoint or server sends a negative acknowledgment message to the transmitting endpoint or server in response to the receiving endpoint's or server's detection of a lost lowest temporal level picture 10 or portion of such picture.

6. The system of claim 5 wherein the transmitting endpoint or server upon receiving the negative acknowledgment message retransmit the lost picture or portion of such picture.

7. The system of claim 1, wherein the encoder conforms to H.264 SVC, 15 and the data elements are carried in NAL unit header extension for SVC elements.

8. The system of claim 1, wherein the encoder conforms to H.264 SVC, and the data elements are carried in an SEI message.

9. A method for media communications between a transmitting endpoint or server and one or more receiving endpoint(s) or server(s) over a packet-based 20 communication network, wherein an encoder encodes transmitted media as pictures in a threaded coding structure having a number of different layers including a lowest temporal layer, and wherein data corresponding to a single picture is portioned and transmitted in one or more individual data packets, the method comprising: placing in in each individual data packet data elements that indicate: 25 for the lowest temporal level pictures, a sequence or index number identifying said pictures, for other temporal level pictures, a reference to the sequence number of the most recent, in decoding order, lowest temporal level picture, and for all pictures, a 'start' flag and an 'end' flag that indicate if the individual 30 data packet contains the first or last, respectively, data portions of the picture, so that a receiving endpoint or server on receipt of the individual data packet can detect a loss of portions of a lowest temporal level picture by examining if all data portions of the picture corresponding to the referenced picture index have been received at the receiving endpoint or server such that their packet sequence numbers NY02:608243.1 30 WO 2008/086423 PCT/US2008/050640 contain no gaps, and where the first such data portion has the 'start' bit set and the last such data portion has the 'end' bit set.

10. The method of claim 9 wherein the data elements additionally indicate a series number associated with each spatial or quality layer, 5 so that the receiving endpoint or server detects if a lowest temporal level picture of a particular spatial or quality layer is lost by determining if the picture corresponding to the referenced series number and sequence number has been received at the receiving endpoint or server. 10

11. The method of claim 9, wherein the tranmsitted media is coded using a codec conforming to H.264, wherein the lowest temporal level pictures include pictures that are marked as long term reference pictures, and wherein the decoding of at least a portion of the received media based on a lowest temporal level picture by the decoder is controlled by MMCO commands. 15

12. The method of claim 9 wherein the communication network uses the Internet Protocol, media transport is performed using the real-time protocol (RTP), and the data elements include data to indicate presence of a lowest temporal layer picture or fragment thereof in the packet. 20

13. The method of claim 12 wherein a receiving endpoint or server sends a negative acknowledgment message to the transmitting endpoint or server in response to the receiving endpoint's or server's detection of a lost lowest temporal level picture or portion of such picture.

14. The method of claim 13 wherein the transmitting endpoint or server 25 upon receiving the negative acknowledgment message retransmit the lost picture or portion of such picture.

15. The method of claim 14, wherein the encoder conforms to H.264 SVC, and the data elements are carried in NAL unit header extension for SVC elements.

16. The method of claim 9,wherein the encoder conforms to H.264 SVC, 30 and the data elements are carried in an SEI message.

17. Computer readable media comprising a set of instructions to perform the steps recited in at least one of the method claims 9-16. NY02:608243.1 31