MX2008008309A - Method for checking of video encoder and decoder state integrity - Google Patents

Abstract

The present invention provides a method and a system for verifying a match between states of a first video processor and a second video processor, wherein one of said first and second video processors is a video encoder utilizing predictive video encoding and the other one of said first and second video processors is a video decoder capable of reproducing a decoded bit stream from an encoded bit stream generated by said video encoder.

Description

METHOD FOR CHECKING THE INTEGRITY OF STATUS OF VIDEO ENCODER AND DECODER FIELD OF THE INVENTION The present invention relates to the field of predictive video coding, particularly with the assurance of an adequate match between the states of the encoder and the decoder.

BACKGROUND OF THE INVENTION The algorithms of. Most current video compression requires maintaining state information between decoding images. An obvious example is the reference image (s) used for inter-image prediction. When errors have occurred during the streaming of bits from the encoder to the decoder, normally this state of the decoder is corrupted. However, only in a few cases, a decoder can determine the presence of corruption from the bit stream itself. In other cases, external means (for example, sequence numbering of transmission packets) may also be used to determine the presence of possible corruption. Decoder state corruption can occur because of erroneous streaming of the bit stream. In the decoder, while most transport stacks contain indication of possible continuous bitstream corruption, it is not clear in which data structure corruption occurs (if there is any possibility to determine that corruption occurred). For example, data with corrupted segments can lead to a corrupted reference image, used for future prediction, and a transmission of lost or damaged group of parameters can lead to a group of lost or damaged parameters, which can be referenced more ahead . Prior to this invention, no mechanism was known to allow any encoder to inform a decoder of its internal state in a simple way. Also, previously no mechanism was known that would allow a decoder to inform an encoder about its internal state, without inferring corruption (or corruption). In other words, there were no means through which a decoder could (for example periodically) inform an encoder about its status without having to perform error detection and explicitly send information that would positively or negatively inform the encoder regarding corruption . The following feedback messages from the decoder to the encoder are known from the prior art. All of them inflict corruption or non-corruption implicitly. In order to generate these prior art messages, error detection must be performed by the message transmitter (the decoder). The feedback messages of reference images NEWPRED (see for example patent US 6,621,868 and Kimata, H., et al, "Study on Adaptive Reference Picture Selection Coding Scheme for the NEWPRED Receiver-Oriented Mobile Visual Communication," Conference of IEEE Global Telecommunications, Nov. 8-12, 1998 (8 pages), Rec. ITU-T, Annex N H.263): The decoder can inform an encoder about known corruption or known non-corruption of an image of reference with a certain ID. The encoder can use this information to use an older reference image for inter-image prediction, which is known to be uncorrupted (in the decoder). However, the decoder can not simply send information regarding its state and let the encoder decide whether corruption exists or not. corruption In other words, the error detection load is in the decoder, not in the encoder. Reverse channel signa for packet loss (for example, ARQ): These techniques check the non-arrival of a packet in order to activate the forwarding. This is information where the status of a receiver ("packages x, and, z are lost ") is transported, however, this is not a" state "in the sense of a video decoder state that is linked to data entities other than packets. of image, indication of segment loss and similar mechanisms related to video indicate corruption: These are available in many different standards, for example in the "Extended RTP Profile for RTCP-based Feedback (RTP / AVPF)" Internet project (AVPF) , please see http: // http: // www. ietf.org/ internet drafts / draft-ietf-avt-rtcp-feedback-11. txt), or in document ITU-T Rec. H.245. All of these require that the error detection be made in the decoder The object of the present invention is to provide means for validating an appropriate match between the states of the encoder and the decoder In the present invention error detection is made possible so that be handled by the receiver of the messages, in contrast to the previously identified prior art which requires that error detection be performed on the message transmitter. In addition, the state of the art feeds back messages that infer corruption or non-corruption, while the messages according to the invention are "neutral".

SUMMARY OF THE INVENTION The invention provides means for calculating checksum information in a video encoder, a signal for conveying the summation control information from the video encoder to a video decoder, means for checking the integrity of the information of state in the video decoder and means for signaling the status information of the video decoder to the video encoder. According to one aspect of the present invention, there is provided a method for verifying a match between states of a first video processor and a second video processor. One of the first and the second video processors is a video encoder that uses predictive video encoding and the other of the first and the second video processors is a video decoder, capable of playing a decoded video stream from a stream continuous bit, encoded, generated by the video encoder. The method comprises: generating, in the first video processor, a first indication of one or more properties of a first state of the first video processor; and - transmitting a message comprising the indication to the second video processor. The message also comprises additional information that helps the second video processor identify which properties have been generated by the indication. This method according to the invention makes it possible to correspond the states of video encoders and decoders, in order to perform the appropriate error handling procedures in case of corruption related to transmission or other type of continuous bit stream encoded. According to an exemplary embodiment, the method further comprises: generating, in the second video processor, a second indication of the properties of a second state of the second video processor; and verify if the first and second states of the first and second video processors match, by comparing the first and second indications. According to an exemplary embodiment, the method further comprises: carrying out error handling procedures in case the states do not coincide. According to an exemplary embodiment, the indication comprises a control sum and wherein the generation step comprises: calculating the control sum. It will be necessary to standardize this calculation in some way, to ensure that the encoders / decoders of different suppliers are interoperable. According to an exemplary mode, the checksum is. calculates by at least one algorithm of the group comprising Binary Copy, Exclusive "0" (XOR), Cyclical Redundancy Code (CRC), Secure Hash Algorithm (SHAl), defined in the National Institute of Standards and Technology of the United States (NIST), FIPS 180 Publication: Safe Hash Standard (SHS), May 1993), or Message Compendium 5 (MD5, defined in IETF RFC 1321). According to an exemplary embodiment, the indication comprises a coded copy of the properties of the first state according to at least one coding method between the group comprising Binary, Base 16, Base 64 (see IETF RFC 3548) and Notation of Abstract Syntax 1 (ASN.l, see ITU-T Rec. X.680) According to an exemplary modality, the properties of the first state include at least one of the group comprising: - Active Image Parameter groups ( PPS); Groups of Active Sequence Parameters (SPS); groups of Image Parameters (PPS) with index x; - Groups of Sequence Parameters (SPS) with index x; groups of Image Parameters (PPS) with index x and groups of referred Sequence Parameters (SPS); Groups of Parameters of all images (PPS); Groups of Parameters of all Sequences (SPS); most recent reference image in the Decoded Image Buffer (DPB); - all reference images in the Decoded Image Buffer (DPB) used for prediction; all the reference images in the Decoded Image Buffer (DPB). According to an exemplary embodiment, the first video processor is the video encoder and the second video processor is the video decoder. According to an exemplary embodiment, the message is transmitted within the continuous stream of bits and can be a Supplementary Information Information (SEI) message. According to an exemplary embodiment, the message is transmitted separately from the continuous stream of bits and the message may also comprise synchronization information to associate the indication, for example a checksum with the point of generation of the indication. That is, such a message sent in an out-of-band manner has to be synchronized with the bit stream in some way. According to an exemplary embodiment, the first video processor is the video decoder and the second video processor is the video encoder. According to an exemplary embodiment, the message is transmitted using at least one protocol between the group comprising Real-Time Control Protocol (RTCP, see IETF RFC 3550), ITU-T Rec. H.225 and ITU-T Rec. H.245. According to an exemplary embodiment, the message also comprises synchronization information to associate the indication, for example a checksum, with the point of generation of the indication. According to another aspect of the invention, a computer-readable medium, comprising sections of code stored thereon, is provided for instructing a processor to perform the steps of: generating, in a first video processor, a first indication of one or more properties of a first state of the first video processor; Y and transmitting a message comprising the indication to a second video processor. In an exemplary embodiment, the computer readable medium further comprises sections of code stored therein, to instruct a processor to perform the additional steps of: generating, in the second video processor, a second indication of the properties of a second state of the second video processor; and - verifying whether the first and second states of the first and second video processors match, by comparing the first and second indications. According to yet another aspect of the invention, there is provided a system for verifying a match between states of a first video processor and a second video processor, wherein one of the first and the second video processors is a video encoder that it uses predictive video encoding and the other of the first and the second video processors is a video decoder capable of reproducing a decoded video sequence from a coded continuous stream of bits, generated by the video encoder. In the system, the. The first video processor comprises: a component for generating a first indication of one or more properties of a first state of the first video processor; and a component for transmitting a message comprising the indication to the second video processor; and the second video processor comprises: a component for generating a second indication of the properties of a second state of the second video processor; and a component to verify if the first and second states of the first and second video processors match, by comparing the first and second indications. In an exemplary embodiment, the second video processor further comprises: a component for carrying out error handling procedures in case the states do not match.

BRIEF DESCRIPTION OF THE DRAWINGS The drawings are provided in an exemplary manner only, to illustrate the principles of the present invention. In the drawings: Figure 1 illustrates a step flow diagram of one embodiment of the inventive method; and Figure 2 illustrates one embodiment of the system of the present invention.

DETAILED DESCRIPTION OF THE INVENTION The main differentiation between messages according to this invention and feedback messages of the prior art lies in two aspects: a) conventional feedback messages infer corruption or non-corruption, while the messages according to the invention are "neutral"; and b) in order to generate prior art messages, error detection always has to be performed by the message transmitter, while in the present invention, the error detection is handled by the message receiver. In the following detailed description, terminology used in conjunction with the ITU-T Rec. H.264 video compression standard is used. However, it should be obvious to a person skilled in the art that the present invention will operate in an equivalent manner with other standardized and non-standardized video compression algorithms using any form of predictive coding. In this way, the invention is not limited to the use of H.264. In Figure 1, a step flow diagram of an exemplary embodiment is illustrated. After the start, the first video processor calculates a control sum, according to one of different algorithms that will be mentioned later on a state of the first video processor in step (102). There are several possible properties that are useful in this case, which will also be mentioned later. In step (104), the first video processor transmits a message comprising the calculated checksum to a second video processor. The second video processor can now decide, in step (106), if you want to ignore the message, that is, do not perform any error checking. In case this happens, the process begins again. If you decide that the message will be used to check for error, the second video processor calculates a checksum (step (108)) on the same property in the first video processor. It will be necessary to make sure that both video processors do not use the same standard / algorithm to calculate this sum of control. In step (110), it is determined whether the two control sums coincide. In case of a positive match, no additional action is required (step (112)), the process starts again. However, in the event that the checksums do not correspond, the appropriate error handling procedures can be performed in step (114). When the first video processor is a video encoder (forward transmission of the checksum information), for example a prior art feedback message may be sent, a request to resend a reference image or the like. When the first video processor is a video decoder (reverse transmission of the checksum information), the video encoder as the checksum receiver may refrain from using a state information known as corrupt in the decoder, for example. example, not using certain reference images known as corrupted for prediction. Now two exemplary cases of use to use the present invention will be presented: Use case 1: The video transmitter wishes to facilitate the detection of error in the video receiver. The video transmitter creates checksums on its status information at a given time and sends the status information as a H.264 SEI message to the receiver. The receiver can react to this message by calculating identical control sums about its state, and if the two control sums do not match, it performs an appropriate procedure (which could for example consist of sending a feedback message of the prior art as a Reference Image Selection Request, Full InRequest, or the like See for example ITU-T Rec. 245 for these prior art feedback messages). Use case 2: The video decoder wants to check its status integrity on its own initiative (for example because it considers possible corruption, but does not have the secure knowledge of it). The video decoder generates a sum of control over the parts of its state that it wishes to check and sends this sum of control together with the timing information to the video encoder. The video encoder checks if the state of the decoder is accurate. If so, no other action is needed. If it is not, the video encoder knows in which data structure the problem exists and can react accordingly. Possible reactions include: During the process of encoding future images, refrain from referencing reference images that are known (after receipt of the checksum) for being corrupted in the decoder; Send groups of parameters that are known to be corrupt and / or refrain from referencing groups of parameters known to be corrupt in the process of encoding future images; - When large parts of the state are corrupted (the worst case), restore the complete status information in the decoder by sending all the parameter groups and a Regeneration Point Image from the Independent Decoder (IDR). First, it will be described how a message from an encoder to a decoder can facilitate error handling in the decoder ("use case 1"). For purposes of simplicity, groups of fixed parameters and only a simple reference image are assumed in this part of the description. In this case, most of the state of the H.264 decoder that may be corrupted by errors of the bitstream is found in the reference image. Parts of the codex state not related to the simple reference image will be described below. According to this aspect of the invention, the encoder first calculates a sum of control over the YUV data (color space) of the reference image. The precise algorithm for this calculation needs to be standardized, since both the encoder and the decoder need to implement equivalent operations to arrive at identical results. A simple way to implement the checksum would be to use a 16-bit CRC and order the CRC to be calculated on the least significant 8 bits of the samples of the Y, U, V color planes, in scanning order, respectively. However, many other forms of calculation are also possible. For example, more advanced control sums such as SHA1, MD5 or other frequently used integrity checkers may be used. It is also possible to modify the order of the samples. As mentioned, what matters is that both the encoder and the decoder have a common understanding and an equivalent implementation of the checksum calculation. Once the checksum is calculated, it becomes a suitable representation (for example, binary, Base 16, Base 64, etc.) and is placed in the continuous stream of bits, for example in the form of a SEI message . Alternatively, the message may also be sent out of band, ie, out of the continuous stream of bits, using an appropriate protocol. An example for an appropriate protocol would be an RTCP Transmitter Report (see IETF RFC 3550). In this case, means are used to synchronize the message with the continuous streams of video bits; When using SEI messages, synchronization is implicit. Subsequently, the encoder encodes the image and sends it. When the decoder receives the checksum (from the SEI message or in out-of-band media) and the encoded image, it may choose to ignore the checksum, for example when it is in the underfeed cycle. However, you can also check the received checksum against a checksum calculated from your own reference image buffer. If the two control sums coincide, the decoder knows for certain that its reference image is not corrupted, in the time it starts with the decoding, the image in question. If the checksum test fails, however, it can react accordingly, for example by sending a prior art feedback message to the encoder because it indicates corruption of the reference image (e.g. in the form of a message). Full InRequest), and not completely decode the continuous stream of bits. There are many advantages to this mechanism: integrity checking can occur while the bit stream of the new image is still being received - leading to an early sending of a prior art feedback message. And, as mentioned, in some environments it is not possible to detect corruption at all from the continuous flow of bits and / or the transport environment, while it is possible to do so with the present invention. The in-band SEI message (or an out-of-band equivalent message) can be sent as frequently as the encoder desires, thus allowing an optimization of the trade-off of advantages and disadvantages between the error detection capability and the width overload of band . Now it will be described how a message coming from a decoder to the encoder can facilitate an appropriate reaction of the encoder that responds to the state of the decoder (once it knows it) ("use case 2"). For simplicity, it is again assumed that the parameter groups are fixed and that only a simple reference image is used. According to this aspect of the invention, the decoder calculates a sum of control over the data of the reference image, as described above. How often this calculation is performed is a matter of the decoder, for example after each image decoding, at fixed intervals, at varying intervals determined by the needs of external protocols (for example RTCP receiver report intervals, see discussion in IETF RFC 3550), etc. Once the checksum is generated, the decoder sends it to the encoder, using an appropriate protocol. An example for such a protocol would be the RTCP receiver reports, see IETF RFC 3550. Some of these protocols inherently can support synchronization with the video transmitter (for example RTCP in the form of the chronological mark); In other environments, the message may need to contain an identification. from the generation point of the image on which the control sum was generated. The encoder, once it has received the message, can choose to ignore it. However, you could also try to check the integrity of your local reference image using the received checksum. It should be noted that this process may require the encoder to calculate checksums for all the images it sends during the round trip delay time of the image transmission (and reverse channel). When the encoder identifies that the reference image in the decoder is uncorrupted, it can continue its normal operation, which usually consists of encoding and sending only predictively encoded images. If the * encoder identifies that the decoder reference image is corrupted, it can react accordingly, ie by sending an intra-coded image. Other properties of a state are very dependent on the codee technology used. Again, using H.264 with an example, at least the following categories of status information can be identified: Parameter groups; and Values of reference (multiple) image samples. All these will be described briefly. H.264 introduces, with its parameter group concept, the decoupling of data belonging to more than one simple segment between block / block segment / macro data. H.264 requires that the "active" image and sequence parameter be available when it starts segment decoding, but does not include a mechanism to check the integrity of the parameter groups. Therefore, additional mechanisms that support integrity checking are useful under conditions of error-proneness. There are two types of parameter groups: Image Parameters Groups (PPS) and Sequence Parameters Groups (SPS). Both are stored in numbered sites. The heading of the segment, of each segment, contains indexing information to reference the active PPS, and each PPS contains indexing information with respect to the related SPS. PPS can grow greatly - several KB - under certain conditions. Taking into account the complexity scalability, according to the invention, control sums can be generated: Active PPS (s) - H.264 (2005) only allows a simple active PPS, but may allow future extensions of H.264 greater than one; Active SPS (s) - H.264 (2005) only allows a simple active SPS, but may allow future extensions of H.264 greater than one, active PPS (s) and active SPS (s) (thus covering all the groups of relevant parameters for decoding the current image - the most common use case), PPS with index x, SPS with index x, PPS with index x and the SPS (s) that is referenced in the PPS with index x, all PPS, all SPS, and all PPS and SPS. H.264, in its 2005 version, does not contain initialization information for parameter groups. In other words, the value of an individual parameter in a group of parameters is undefined before it is transmitted first. The checksum calculation algorithm for parameter groups has to be designed to take this property into account. A possible solution is to infer all the parameters in a group of parameters without initializing, with the value zero. Other solutions may also be possible and obvious to a person skilled in the art. The precise design of the inference is irrelevant to the invention, as long as the encoder and the decoder use the same design. In H.264, but also in some older video compression standards such as H.263 when using Annex N or U or part 2 of MPEG-4 (please see ISO / IEC 14496-2) with the " Advanced Simple Real Time Profile (ARTS profile) enabled, more than one reference image can be used. The reference image in use is indicated by what could be termed the "temporal" component of a three-dimensional motion vector. Depending on the standard, this vector component can be part of the macroblock image, segment or syntactic layers. H.264 also allows biprediction from two reference images. When multiple reference images are allowed, obviously an integrity check on all, or a subgroup, of these reference images may be useful. The most common cases are mentioned here explicitly, although a person skilled in the art is aware of other cases that are equally valid for certain applications. According to this invention, the following cases are mainly considered useful: the most recent reference image in the Decoded Image Buffer (DPB); it is in most cases (unless it is moved by a command of Memory Management Control Operation (MMCO, H.264) the previously decoded image, where it is most likely corruption; all the reference images in the DPB used for prediction; mainly useful for the encoder to decoder address of the method of the invention; the control sum is calculated on all the reference images used for prediction of the image that is encoded; All the reference images in the DPB. H.264 in its 2005 version contains support for color spaces other than YUV 4: 2: 0. A possible mechanism to generate a checksum for the YUV 4: 2: 0 color space has already been introduced previously. A person skilled in the art is capable of designing checksum algorithms for other color spaces as well. The design of these checksum algorithms is in principle irrelevant to this invention, as long as the encoder and the decoder use the same design. Figure 2 describes one embodiment of the system of the present invention. A first video processor (2) comprises a component (4) for generating a first indication of one or more properties of a first state of the first video processor, and a component (6) for transmitting a message (14) comprising the indication to a second video processor (12). The second video processor (12) comprises a component (8) for generating a second indication of the properties of a second state of the second video processor, and a component (10) for verifying whether the first and second states of the first and the second video processors match, comparing the first and second indications. In a further embodiment, an additional component is provided to perform the error handling procedures in case the states do not match.

Claims (21)

1. CLAIMS: 1. A method to verify a match between states of a first video processor and a second video processor, wherein one of the first and the second video processors is a video encoder that uses predictive video encoding and the other of the first and the second video processors is a video decoder capable of reproducing a decoded video sequence from a continuous stream of bits encoded and generated by the video encoder, the method comprises: generating, in the first video processor , a first indication of one or more properties of a first state of the first video processor; and transmitting a message comprising the indication to the second video processor. The method according to claim 1, further comprising: generating, in the second video processor, a second indication of the properties of a second state of the second video processor; and check if the first and second states of the first and second video processors match, comparing the first and second indications. 3. The method according to claim 2, further comprising performing error handling procedures in case the states do not match. The method according to claim 2, wherein the indication comprises a control sum, and wherein the generation step comprises calculating the control sum. The method according to claim 4, wherein the checksum is calculated by at least one algorithm of the group comprising: Binary Copy, Exclusive 0 (XOR), Cyclic Redundancy Code (CRC), Secure Hash Algorithm (SHA1) , Compendium of Messages 5 (MD5). The method according to claim 1, wherein the indication comprises a checksum, and wherein the step of generating comprises calculating the checksum. The method according to claim 6, wherein the checksum is calculated by at least one algorithm of the group comprising: Binary Copy, Exclusive O (XOR), Cyclic Redundancy Code (CRC), Secure Hash Algorithm (SHA1) , Compendium of Messages 5 (MD5). The method according to claim 1, wherein the indication comprises a coded copy of the properties of the first state according to at least one encoded method of a group comprising: Binary, Base 16, Base 64 and Abstract Syntax Notation 1 (ASN.l). The method according to claim 1, wherein the properties of the first state include at least one of the group comprising: groups of Active Image Parameters (PPS); Groups of Active Sequence Parameters (SPS); Groups of Image Parameters (PPS) with index x; groups of Sequence Parameters (SPS) with index x; groups of Image Parameters (PPS) with index x and groups of referred Sequence Parameters (SPS); - Groups of Parameters of all the images (PPS); Groups of Parameters of all Sequences (SPS); most recent reference image in the Decoded Image Buffer (DPB); all reference images in the Decoded Image Buffer (DPB) used for prediction; and all the reference images in the Decoded Image Buffer (DPB). The method according to claim 1, wherein the first video processor is the video encoder and the second video processor is the video decoder. The method according to claim 10, wherein the message is transmitted within the continuous stream of bits. 12. The method according to claim 11, wherein the message is a Supplementary Information Information (SEI) message. The method according to claim 10, wherein the message is transmitted separately from the bit stream. The message according to claim 13, wherein the message also comprises synchronization information to associate the indication with the point of generation of the indication. The message according to claim 1, wherein the first video processor is the video decoder and the second video processor is the video encoder. 16. The method, according to claim 15, wherein the message is transmitted using at least one protocol between the group comprising Real Time Control Protocol (RTCP), H.225 and H.245. The method according to claim 16, wherein the message also comprises synchronization information to associate the indication with the point of generation of the indication. 18. Computer-readable medium, comprising sections of code stored therein, for instructing a processor to carry out the steps of: - generating, in a first video processor, a first indication of one or more properties of a first state of the first video processor; and transmitting a message comprising the indication to a second video processor. The computer readable medium of claim 18, further comprising sections of code stored therein, for instructing a processor to perform the additional steps of: generating, in the second video processor, a second indication of the properties of a second state of the second video processor; and checking whether the first and second states of the first and second video processors match, by comparing the first and second indications. 20. System for verifying a match between states of a first video processor and a second video processor, wherein one of the first and the second video processors is a video encoder that uses predictive video encoding and the other of the first and the second video processor is a video decoder capable of reproducing a decoded video sequence from a coded continuous stream of bits, generated by the video encoder, wherein the first video processor comprises: a component for generating a video first indication of one or more properties of a first state of the first video processor; and v a component for transmitting a message comprising the indication to the second video processor; and wherein the second video processor comprises: a component for generating a second indication of, the properties of a second state of the second video processor; and a component to verify if the first and second states of the first and second video processors match, by comparing the first and second indications. The system according to claim 20, wherein the second video processor further comprises a component for carrying out error handling procedures in case the states do not match.