CN101176353B - Decoder architecture for optimized error management in streaming multimedia - Google Patents
Decoder architecture for optimized error management in streaming multimedia Download PDFInfo
- Publication number
- CN101176353B CN101176353B CN200680014271.6A CN200680014271A CN101176353B CN 101176353 B CN101176353 B CN 101176353B CN 200680014271 A CN200680014271 A CN 200680014271A CN 101176353 B CN101176353 B CN 101176353B
- Authority
- CN
- China
- Prior art keywords
- mistake
- layer
- agreement
- described detected
- medium data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Images
Landscapes
- Detection And Prevention Of Errors In Transmission (AREA)
- Communication Control (AREA)
Abstract
The present invention discloses a method and apparatus for multi-layer integration for use in error recovery. An error is detected in a multimedia data based on a first layer protocol and the detected error in the multimedia data is concealed based on a second layer protocol. In one aspect, the error in a multimedia data is detected based on a communication layer protocol and controlled based on a transport layer protocol. An error distribution of the controlled error is then determined based on a sync layer protocol and the detected error in the multimedia data is concealed based on an application layer protocol. In another aspect, a method and apparatus for multimedia data processing comprises error recovery as well as scalability. Finally, a method and apparatus as disclosed allows processing of multimedia stream by receiving multiple streams of encoded multimedia data, performing error recovery on an erroneous portion of a stream, and reconstructing the multimedia data from the multiple streams.
Description
Advocate priority according to 35 U.S.C. § 119
Present application for patent advocate to have precedence on March 10th, 2005 filed an application and name be called " Method andApparatus for Error Recovery in Video Communications " the 60/660th, No. 681 provisional application cases, filed an application on March 10th, 2005 and the name be called " A Method And Apparatus For Video Decoding " the 60/660th, file an application in No. 923 provisional application cases and on March 10th, 2005 and the name be called " Method Of ErrorRecovery For A Decoder " the 60/660th, No. 867 provisional application cases, all described provisional application cases are all assigned in assignee of the present invention and are incorporated herein clearly by reference.
Technical field
The method and apparatus of the real-time streaming medium on the handheld apparatus the present invention relates to be used to decode.
Background technology
Because the explosive increase and the immense success of internet and radio communication, and to the demand that increases day by day of multimedia service, the streaming video on internet and moving/wireless channel has caused that people pay close attention to greatly.In heterogeneous the Internet protocol (IP) network, video is to be provided and can be by one or more client fluidisations by server.Wired connection comprises: dialing, integrated service digital network (ISDN), cable, digital subscriber line agreement (being referred to as xDSL), optical fiber, LAN (LAN), Wide Area Network (WAN) and other network.Transmission mode is unicast or multicast.
What be similar to heterogeneous IP network is mobile/radio communication.Have challenge by mobile/wireless channel transportation content of multimedia, this is because these channels often are subjected to grievous injury because of multipath decline, concealment, intersymbol interference and noise disturbance.Wait some other reason also can cause bandwidth change and loss such as mobility and competition business.Interchannel noise and the number of users of being served are determined the time-varying characteristics of channel circumstance.
In heterogeneous IP network and mobile communication system in the two, to high data rate more and more the demand of high quality-of-service increase rapidly.Yet factors such as for example limited time of delay, limited transmitting power, limited bandwidth and multipath decline are still continuing the handled message transmission rate of restriction real system.In multimedia communication, especially in the environment of easily makeing mistakes, the error resilience of institute's transmission medium is providing most important aspect the required service quality, this be because or even the mistake in the single decode value also may cause decoding illusion in the space and the time upload and broadcast.People have used various code means when keeping the data necessary transmission rate mistake being minimized, yet all these technology all exist mistake can arrive the problem of decoder-side.
By using source encoder, logarithm exert pressure factually contract-transmit the information of maximum by consuming minimum position, follow by channel encoder, described channel encoder tends to make the maximum capacity of transmission channel under the situation that receives these with set error probability.
People use chnnel coding (for example, Reed-SoLomon coding) to improve the robustness through the data of source code.People have used joint source-channel coding method to come for having error protection in various degree being provided or making the rate adaptation of coding video frequency data in available network bandwidth by cutting apart and abandoning grouping through source coded data of different significance levels.This is because transport protocol commonly used is not delivered to source encoder with corrupt data.
When having used for example reversible variable length code source code technology such as (for example, in MPEG-4) to divide into groups by receiving error actually, people come implementation mistake to recover packet decoding by reversed sequence.The source code technology can be sacrificed aspect code efficiency to some extent, and for set bit rate, this can be transferred to the quality of decoded video.
The hybrid coding standard, for example MPEG-1, MPEG-2, MPEG-4 (being referred to as MPEG-x), H.261, H.262, H.263 reach H.264 (being referred to as H.26x) and use synchronous points again in the bit stream as manage wrong main method everywhere at decoder.
The another kind of reason that can cause data degradation to surpass initial corruption is because incorrect codeword emulation.If to the identification of initial bit mistake position is not to be inappreciable task and not have usually a kind ofly to support that in MAC layer or physical layer component the specialized designs of the identification of contraposition errors present can't realize.Thereby, after detecting bitstream corruption, decoder may must stop decoding and in bit stream reach and in described process, must skip the data that may not have error of quite big quantity finding down synchronous points again and again.Although for above-mentioned sequence of events, imitate different code words-itself and original (promptly believable) code word to have equal length-may seem not to be too big problem, however in fact really not so.There are many approach to make this type of mistake can cause decoder decipher bit stream correctly.For example, in the existing coding decoder of great majority, in bit stream, exist its value can influence the object (parameter relevant) of the grammer of described bit stream further part with compression.Thereby the improper value of this kind object may cause bit stream by decipher mistakenly.
Because transport protocol commonly used is not delivered to corrupt data decoder (for example, video or audio decoder application), thereby the decoder processes bit-errors is limited in one's ability, wherein abandons grouping and is modal solution more synchronously.Need provide a kind of can processing meeting because of forfeiture for example synchronously and problems such as wrong codeword emulation make the mistake propagation and loss of data bit-errors through improving one's methods.
Summary of the invention
In one aspect, be used for following method or device a kind of comprising for the multi-layer integration of using in mistake is recovered and equipment: according to the mistake of ground floor protocol detection multi-medium data; And hide detected mistake in the described multi-medium data according to second layer agreement.In another aspect, a kind of equipment of integrating for the multilayer of using in mistake is recovered comprises: detector, and it is according to the mistake in the ground floor protocol detection multi-medium data; And concealer, it hides detected mistake in the described multi-medium data according to second layer agreement.In described multi-layer integration and equipment, described ground floor can comprise communication layers.Described communication layers can comprise one or combination in physical layer, MAC layer and the transport layer.In addition, described method and apparatus can further comprise method or the device that is used for controlling according to transition layer protocol detected mistake.Control detected mistake and can comprise the detected mistake in location.Described method and apparatus also can further comprise method or the device that is used for determining according to the layer, sync agreement Fault Distribution of detected mistake.The described second layer can comprise application layer.
In another aspect, a kind of method and apparatus of integrating for the multilayer of using in mistake is recovered comprises and is used for following method or device: the mistake that detects multi-medium data according to the communication layers agreement; Control detected mistake according to transition layer protocol; Determine the Fault Distribution of described controlled mistake according to the layer, sync agreement; And hide detected mistake in the described multi-medium data according to application layer protocol.In aspect another, be used for following method or device a kind of comprising for the method and apparatus that uses in multi-medium data is handled: implement the mistake recovery of encoded multimedia data; And the scalability of the described data of encoded multimedia of support.More on the one hand in, a kind ofly comprise for the equipment that in multi-medium data is handled, uses: error recovery component, it is used to implement the mistake of encoded multimedia data and recovers; And scalability component, it supports the scalability of the described data of encoded multimedia.Described for the method and apparatus that in multimedia is handled, uses in, scalability can comprise one in space and the time scalability or the two.Mistake recovers to comprise one or combination in timing error concealment, spatial error concealment and the frame rate up conversion.
In aspect further, be used for following method or device a kind of comprising for the method and apparatus that uses in media stream is handled: receive a plurality of encoded multimedia data streams; The error section implementation mistake of convection current recovers; And by described a plurality of stream reconstruct multi-medium datas.More further aspect in, a kind ofly comprise for the equipment that in media stream is handled, uses: receiver, it receives a plurality of encoded multimedia data streams; Error recovery component, the error section implementation mistake of its convection current recovers; And reconstructor, it is by described a plurality of stream reconstruct multi-medium datas.In described method and apparatus for use in media stream is handled, mistake recovers to comprise one or combination in timing error concealment, spatial error concealment and the frame rate up conversion.
It should be noted that above method and apparatus can be made up by the computer-readable media and/or the processor of the operation that is configured to implement described method or described equipment.
Description of drawings
Figure 1A is the block diagram of example that is used to send the communication system of streamed multimedia.
Figure 1B is the block diagram of example that is used to send the multiple field communication system of streamed multimedia.
Fig. 1 C is the block diagram of another example that is used to send the multiple field communication system of streamed multimedia.
Fig. 2 A is used to decode the block diagram of example of the multimedia decoder device architecture of streaming.
Fig. 2 B is another view that comprises the decoder device architecture of the protocol stack diagram of multilayer integrated management system of transmitter and Fig. 2 A.
Fig. 3 describes to be organized into and is used for that cascade Reed Solomon wipes and the example of the multimedia symbol of Turbo coding.
Fig. 4 is used to decode the flow chart of example of the multimedia method of streaming.
Fig. 5 describes the Turbo information packet structure of video data.
Fig. 6 shows the block diagram of exemplary components system, but the part of multi-media receiver 24 shown in described exemplary components Fig. 1 of system.
Fig. 7 shows the flow chart of error recovery procedure (ERP).
Embodiment
This paper describes a kind of method and apparatus that is used for providing at multimedia decoder improved error recovery capabilities.Integrated error recovery capabilities is provided, and for example, (for example, communication layers) located to detect wrong in the multimedia data stream and located detected wrong implementation mistake is recovered in application layer (for example, video or audio decoder) on the upper strata.In an example, provide a kind of decoder architecture, it provides corrupt bits flag information to be used for making sensible decision in the various types of error recovery technique of enforcement to application component.Described error recovery technique is used for replacing corrupt symbols with the estimate symbol that is obtained from the available information of application component (for example, previous decoded video, audio frequency, text and graphical information).In hereinafter illustrating again, for realizing providing detail to the thorough of described embodiment.Yet, it will be understood by one of ordinary skill in the art that described embodiment can carry under the situation that does not possess these specific detail to implement.For example, electric assembly can be shown by the block diagram form, in order to avoid make described embodiment indigestion with unnecessary details.In other illustration, can show in detail that this class component, other structure and technology come the described embodiment of further explain.It should also be apparent to those skilled in the art that and the electric assembly that is shown as the independent blocks form can be rearranged and/or is combined into an assembly.
It shall yet further be noted that and some embodiment can be described as process, described process is depicted as flow process, flow chart, structure chart or block diagram.Although flow process can be described as operation the succession process, yet also can implement the many operations in the described operation parallel or simultaneously and can repeat described process.In addition, can rearrange the order of described operation.When its operation was finished, process promptly stopped.Process can be corresponding to method, function, program, subroutine, subprogram etc.When process during corresponding to function, the termination of described process is back to the calling function or the function of tonic chord corresponding to described function.
Figure 1A is the block diagram of example that is used to send the communication system of streamed multimedia.System 20 comprises transmitter 22 and receiver multimedia decoder 24.Transmitter 22 contains various forms of compressing multimedia data, includes but not limited to video, audio frequency, figure, text and picture.But described data MPEG-x and compressed video and the audio frequency in the standard H.26x, MPEG-4 AAC, MP3, AMR and the G.723 compressed audio in audio frequency or the voice compression or the numerical data of any other type.
Figure 1B is the block diagram that is used for dividing at transmitter 22 and decoder 24 the multiple field protocol stack of task.The upper component 205 and 210 that lays respectively in transmitter 22 and the decoder 24 can comprise a plurality of application, for example, and video or audio coder and/or decoder.Some embodiment can comprise the information flow that a plurality of plans are decoded simultaneously.In these cases, also can in upper component 205 and 210, implement to make the synchronous task of described a plurality of stream.Upper component 205 can provide the timing information of having encoded in the bit stream by wireless network 26 and/or cable network 28 transmission.Upper component 210 can analyze described a plurality of information flow so that the application that is associated approximately side by side with its decoding.
Fig. 1 C is used at the block diagram of transmitter 22 with the more detailed example of the multi-layer protocol storehouse of decoder 24 division tasks.The upper component 205 of transmitter 22 is distributed in in application layer 206 and the layer, sync 207 one or more.215 of the lower layer components of transmitter 22 are distributed in in transport layer 216, stream or medium access control (MAC) layer 217 and the physical layer 218 one or more.Similarly, the upper component 210 of decoder 24 is distributed in in application layer 211 and the layer, sync 207 one or more.220 of the lower layer components of decoder 24 are distributed in in transport layer 221, stream or medium access control (MAC) layer 222 and the physical layer 223 one or more.The those skilled in the art will be familiar with these described layers and be familiar with the distribution therebetween of various tasks.To discuss the example of the framework of various layers in the integration decoder device 24 as indicated above now, so that the error resilience that is provided in the transmitter 22 to be provided.One or more elements can be added, rearranged or combined to transmitter 22 shown in Fig. 1 C or decoder 24.
Fig. 2 A is used to decode the block diagram of example of the multimedia decoder device architecture of streaming.Fig. 2 B is the protocol stack view that comprises the decoder device architecture of the protocol stack diagram of example of multilayer integrated management system of transmitter 22 and Fig. 2 A.With reference to Fig. 2 A and 2B, multimedia decoder 30 comprises physical layer component 32, MAC layer assembly 34, transportation and sync. parser (TSP) 39, reaches application layer component 50.Multimedia decoder 30 receives incoming bit stream (IN), and described incoming bit stream comprises the cascade error correction schemes such as the Turbo/Reed-Solomon concatenated schemes.Physical layer component 32 can be implemented demodulation tasks, includes but not limited to reception, mistake control decoding (for example, turbo code decoding) and be situated between connect MAC (medium access control) layer.But MAC layer assembly 34 implementation mistakes control decoding, for example, Reed-Solomon error detection, error correction reach the corrupt data (for example, one or more group) that can't revise are added flag.MAC layer assembly 34 is situated between with transportation and sync. parser component (TSP) 39 and connects.
Sync. (synchronously) layer parser component 38 implemented further analysis according to the sub-bit stream that is relative to each other to described bit stream on the time synchronized basis.The bit stream that the multimedia video bit stream can be parsed into video bit stream, audio bit stream and be associated with the text of closed caption.Sync. layer parser component 38 will be forwarded to related decoder application with time synchronization information together through the bit stream of analysis.This makes it possible in time show and play relevant audio frequency, video and text.
Except that the analysis of above argumentation, transport layer demultiplexer assembly 36 can be analyzed error and (for example add flag information, CRC failure information 33 and Reed-Solomon failure information 35) and (for example forward it to sync. layer parser component 38 and/or suitable application layer processes, Video Decoder or audio decoder), described error adds flag information and receives from MAC layer assembly 34 and physical layer 32.Sync. layer parser component 38 can add error flag information and transmit with the form of error distribution information 37.Sync. layer parser component 38 also can be transmitted the wrong control strategy information 41 of recommendation.Then, described application layer processes can be used error to add flag information 37 and control strategy information 41 to come implementation mistake to handle.
Core application decode component 46 can comprise Video Decoder, audio decoder and text and graphical application.By before with core application assembly 46 decoding or during the recovery of various application bit stream implementation mistake, scalability and FRUC are handled, can strengthen with improve the low quality bit stream (with low quality implement the bit stream of coding or because of the bit stream that mistake receives with low quality occurring) quality.For example, assembly 40,42 and 44 can provide deferring to H.264 standard baseline video bit stream (the baseline configuration file is the very simple configuration file at the low-power device design) and some characteristic from other configuration file in H.264 being provided, for example for the scalability that satisfies stream-type video, obtain and error resilience requires required B-frame and slice of data to cut apart.The details of the process of utilizing multimedia decoder 30 assemblies hereinafter will be provided.One or more elements can be added, rearranged or combined to transmitter 22 shown in Fig. 2 A and the 2B or decoder 30.
To give brief discussion to error detection and error correction now.(channel) code also comprised the cascade code of outer (channel) code in an example of error detection and amendment scheme used and both comprised.The cascaded channel code is by (interior) code of the Turbo in the physical layer and be positioned Reed-Solomon (outward) in the transportation/MAC layer and wipe and revise code and form.Fig. 3 describes to be organized into and is used for that cascade Reed Solomon wipes and the example of the multimedia symbol of Turbo decoding.With regard to the coding side, the symbol in self information source output in the future, change into byte 102 from the binary code word IOB of encoder.Each byte 102 all is regarded as being called the symbol in the limited field of " Galois field (256) ", so that be positioned at outer (N, K) Reed-Solomon (RS) code on " GF (256) ".N and K represent whole RS code word 104 with the symbol quantity form respectively and accommodate the size of the source data 106 of components of system as directed.Therefore, N subtracts K and can obtain the quantity that is contained in the parity character 108 in each code word 104.(N, K) RS code can be revised N and subtract K and wipe.
The K at a top row 106 holds in fact can be by being scanned into K row earlier or to classify first mode as with behavior from the symbol of information source output and these symbols.When error appears in set Turbo information packet row 112, by corrupt groups of bits obviously being shortened to classify the first staggered meeting that scanning was obtained as.Owing to exist the Turbo information block to wipe, described corrupt groups of bits is respectively 1 byte in the length when classifying scanning earlier as, and this is different from the length that is respectively (L-1) individual byte with behavior scanning earlier the time.At the decoder place, such as hereinafter argumentation, when these corrupt groups of bits are added flag, may need to discern the size of these groups and position (in bit stream).After this initial step that source data is placed, in described L (for K byte) row 104 each is listed as RS by adding N-K parity bytes and is encoded into N byte, and produce thus among Fig. 3 each go K+1 ..., N 108.K the row at the top of being made up of source data 106 is called RS information block, and is called through the block of RS coding or abbreviated as code block 110 by the whole set that N row formed.
Each row 112 is all additional have CRC (cyclic redundancy check (CRC)) verification and and some be to make the required tail bit of the correct running of turbo encoder.By to the 112 additional verifications of each row and, can announce those after the turbo decoding, fail to satisfy its each self checking and row be wiped free of.All every next row 112 ground of each code block 110 input in the turbo encoder, and therefore each row all is called the Turbo information block.
The Turbo decode procedure provides the hard decision data for the Reed-Solomon decode procedure, and this can further reduce remaining error rate.Success is revised the ability wipe and is depended on the total number of erasures in the code block and the quantity (N subtracts K) of the used parity character of each RS code word.
In chnnel coding design at the multimedia decoder 30 of Fig. 2, if code block had Reed-Solomon (RS) wipes and surpass the correction capacity, can and unite then that notice is passed to transport layer, layer, sync with the RS block of information (Fig. 3) of correspondence and be assembly in the application layer at last, the described notice of uniting can be to which error occurs and adds flag in described K the Turbo information block 112 (Fig. 3).The system configuration that described outer (N, K) RS code is positioned on the GF (256) makes it possible to the turbo information block that directly utilization correctly receives (no error).
Fig. 4 is used to decode the flow chart of example of the multimedia method of streaming.The inputing to decode procedure 60 and modulated 62 of receiving belt noise through modulating data (IN) and with it.Can go up at wired or wireless network (for example, wireless network among Fig. 1 26 and cable network 28) and receive described data.The physical layer component 32 of Fig. 2 is implemented the demodulation action of institute's receiving belt noise data at step 62 place.Transmit then through demodulated data, wherein can detect and correct mistakes 64.The physical layer component 32 of Fig. 2 can be implemented the Turbo decoding action in 64, and the MAC layer assembly 34 of Fig. 2 can be implemented the Reed-Solomon error correction action in 64.
After Turbo and Reed-Solomon decoding has detected and revised 64 all amendable mistakes, discern 66Turbo information block and/or corrupt bytes by for example adding flag.With bit stream to front transfer for the analysis 68 before, abandon in each Turbo information block CRC check and and tail bit and parity rows 108 (Fig. 3).Correctly receive, improperly receive but through correction, and the corrupt data of band flag, together be used to discern corrupt data add flag information all to front transfer for carrying out bit stream analysis 68.The physical layer component 32 of Fig. 2 and/or MAC layer assembly 34 can be implemented the action of identification (for example, by adding flag) 66 corrupt data.
As mentioned above, transportation demultiplexer assembly 36 sync. layer parser component 38 are used 68 one-tenth a plurality of bit streams of bit stream analysis for a plurality of application layer processes.In the example shown in Fig. 4, with bit stream analysis 68 one-tenth video bit streams 70, audio bit stream 72 and texts and/or graphics bitstream 74.Each bit stream 70,72 and 74 can contain the data that are useful on the corrupt groups of bits in the identification individual bitstreams.In addition, when individual bitstreams was wanted time synchronized, described bit stream can contain timing information.
After the analysis individual bitstreams, carry out fault processing at the application layer place, wherein utilize a kind of technology in several wrong recoveries or the concealment techniques to come corrupt bits is implemented to replace 78,82 and 86.The error recovery component 40 of the application layer component 50 among Fig. 2, scalability component 42 and FRUC assembly 44 can be implemented replacement operation 78,82 and 86.After replacing corrupt symbols, can be respectively with individual video, audio frequency and text/graphics bitstream decoding 80,84 and 88.The core application assembly 46 of the application layer component 50 among Fig. 2 can implement to decode 80,84 and 88.The core application assembly 46 of the application layer component 50 among Fig. 2 also can to may be by lower layer components in addition the corrupt bits of flag replace 78,82 and/or 86.One or more elements can be added, rearranged or combined to the process 60 shown in Fig. 4.
To discuss in more detail now as shown in Figure 4 pass through add the example that flag is discerned 66 corrupt bits to corrupt data.Fig. 5 describes the Turbo information packet structure of video data.Block 140 set of expression Turbo information block, for example K of the RS block of information of being described in the pie graph 3 capable Turbo information block 106.Frame of video the data that may need to encode more than a Turbo information block.For example, the first frame F1 starts among the interior capable 142A of block of information 148A.Remainder data among the frame F1 then is arranged in the first of capable 142B, 142C, 142D and row 142E.Row 142A also contains synchronous header (SH) 146A, and header (SH) 146A contains for example with inferior information synchronously: stream identification, lock in time, frame identification (if frame number and possible layer number-have basal layer and enhancement layer) and other information.Synchronously header 146A is used to sync. layer parser (referring to 38 among Fig. 2) identification will be contained in data in the follow-up piece of expression frame F1 and where is sent to (which is used).The CRC check that is as mentioned to be discussed in end of each row and, it is used for identification in combination with the Reed-Solomon decoding and wipes (error) and divide into groups.
Section start in each Turbo information packet row is a transport layer header (TH) 144.Each TH 144 all contains Last_Flag (the most last flag) and Offset_Pointer (offset pointer).Implement at state 66 places of Fig. 4 to error Turbo information block in addition flag can self finish by the Offset_Pointer value.Offset_Pointer is set to illegal value can indicate described grouping error to occur.Another is chosen as, and can use Error_Flag (mistake flag) field in TH.Can add flag by the set point among the corresponding TH (Error_Flag=1).For example, if error appears in row 142C, then the TH of row 142C can be set for and equals 1.It is the footline (for example, becoming to equal 1 by being set) of frame that Last_Flag is used to indicate current line, and if its be footline, then use Offset_Pointer to indicate next frame to originate in where (in byte quantity) in the Turbo information packet row.For example, among the 142E that is expert at, TH 144B can have (Last_Flag=1) and Offset_Pointer and can equal to be contained in byte quantity in the described row before at TH 144C initial (frame F3 is initial).Synchronously data and the bit stream of the header 146B indication Frame_ID (frame ID) that can contain as mentioned to be discussed.Block of information 148C contains the video data of expression frame F3.Error occurs if determine row 142E, then decoder can not determine that frame F1 finishes wherein and frame F3 begins wherein.
Except that (or substitute) as mentioned argumentation in transport header, comprise the Error_Flag, can make up tables of data and forward it to application layer, wherein in table 1, list the information of each frame of video.
| Integer numbering in the limited frame set of frame number-be in (for example 30 frames) is wherein restarted into column number at 1 place reaching the back of maximum numbering.The basal layer of the frame of B/E-indication under multilayer scalability situation or enhancement layer numbering.The length of frame of video in the Frame_Length-byte.Whether RAP_Flag-indication frame is the flag of random access point (for example, the I frame of the complete intraframe coding of warp). |
| When running into mistake, this flag also can be used as synchronous points again.Presentative time stabs the time of (PTS)-when wanting display frame in frame sequence.Which Turbo information block the Turbo information block quantity Error_Bit_Pattern-of frame/second (FPS) frame occupied (whole and partly the two) indicates the variable (five positions ' 00100 ' will indicate the 3rd Turbo information block in five groupings error to occur) of error occurs.Error_Ratio-indication error Turbo information block is to the variable of the ratio of correct grouping (1/5 will indicate have a grouping error to occur in five groupings). |
Table 1. is contained in the sample information in the video bit stream error list.
Can construct and similar other bitstream error tables of table 1.(for example be contained in transport header and/or error list, table 1) mistake in add flag information can by application layer component (for example, the assembly 40,42 and 44 of Fig. 2) be used for Fig. 4 discern respectively through analysis in the bit stream 70,72 and 74 corrupt symbols and at step 78,82 and 86 places these are replaced through the analysis bit stream.Discuss as mentioned, be not abandoned but be submitted to application layer from the grouping of wiping of RS coding, for example, Video Decoder.This whole group that can prevent loss-for video packets, may be 122 byte lengths or longer.
The operation that step 78 in Fig. 4,82 and 86 places replace corrupt symbols can present two kinds of principal modes: error correction and error concealment.To wiping and burst error enforcement bit stream staging error correction and concealment operations.Use MAP (maximum empirical probability) criterion to revise decoding error (because of byte error causes) as much as possible.The mistake that can't revise then use from the space of adjacent macroblocks and/or temporal information hide (macro block is 16 * 16 pixel regions of usually handling with video compression standard, but for example also can use 8 * 8,8 * 16 and other littler sub-macro block).For example, when image is static state in more than a frame, the up time concealment.If the corrupt macroblock of being hidden is in the zone that has kept constant relatively in the previous frame, then its may be present in the described corrupt frame and can use the frame zone of process as estimated value to described corrupt region.Edge or the object that continues existence on the adjacent macroblocks that can utilize in same frame hidden in the space.
Discuss as mentioned, can receive the coded data of a plurality of layers the same bit stream of expression.Described layer can comprise basal layer and one or more enhancement layers, and wherein said enhancement layer can be provided at the extra frame (for example, through bi-directional predicted B frame) that can't obtain in the basal layer, and perhaps it can provide the more high-quality difference of basal layer pixel coefficient is improved.Under providing the improved situation of the difference of base layer coefficient, if error appears in basal layer, then enhancement layer may be utterly useless, and this is to derive according to base layer pixel values because of it.Therefore, if error appears in base layer data, then enhancement layer can be omitted from decoding.This process is called the selectivity decoding and also can uses under the low-power situation.For example, when the device that holds decoder just with low-power mode operation or when under accumulator electric-quantity situation on the low side, moving, can be only with the basal layer decoding and omit enhancement layer, therefore save computation cycles and from and save power consumption.But usage space and/or time concealment techniques are replaced the base layer plus enhancement layer data of combination.
The power that can be lower than basal layer (or more priority layer) transmits enhancement layer (being regarded as the low priority layer sometimes).This can make the error probability in the enhancement layer increase with respect to basal layer.Thereby, when the basal layer that is received contains high percentage of errors, can potentially enhancement layer be abandoned.
Through bi-directional predicted frame (B frame) be according to previous frame and subsequent frame the two, use motion compensated prediction to be predicted.The B frame can provide high replacement ratio and be very beneficial for conserve bandwidth.The B frame is also satisfactory because of its time scalability features.Standard compliant B frame is not used in the prediction of any other frame, and for this reason, it can be dropped and can influence other frame sharply.Because the mistake in the frame that it relied on, the B frame is also more responsive to error propagation.For those reasons, the B frame carries in strengthening (or low priority) layer probably.If with the low-power transmission enhancement layer, then the B frame is responsive more to mistake.
If B frame (or frame of any other type) is complete error, make time or spatial error concealment infeasible thereby perhaps have the percentage of errors that is higher than threshold value, then can utilize frame rate up conversion (referring to the FRUC assembly 44 of Fig. 2).FRUC is used for producing again the frame of being lost.FRUC also can be used for producing again for former thereby frames that skipped in decoding such as for example saving powers.The FRUC technology is used to come from the motion vector of previous frame and subsequent frame and Pixel Information the value of interpolation motion vector and pixel.FRUC itself does not belong to the category of this explanation.The FRUC technology comprises form that decoder is only arranged and by the form of encoder-assisted.Only there is the FRUC of decoder under the situation of no any out of Memory, to implement the pseudo-interpolation of frame data.Then utilize the incidental information that in user_data (user data) message of H.264 supplemental enhancement information (SEI) message or MPEG-2, is transmitted by the FRUC of encoder-assisted.
The block diagram of the example of Fig. 6 display module system, but the part of described component system multi-media receiver (for example, receiver shown in Fig. 1 24).System 600 is the systems that integrate for the multilayer of using in mistake is recovered.Fig. 7 shows the flow chart of the error recovery procedure (ERP) that can be used by system 600.With reference to Fig. 6 and 7, process 700 receives 705 one or more multimedia data streams.Data flow can comprise video, audio frequency and/or closed caption text and other information.One or more streams that received are coded stream.Coded data is through data converted, the data through quantizing, compressed data or its combination.Receiving system (for example, the receiver among Figure 1A 24) can receive 705 described one or more streams.Except that receiving a plurality of media streams, the stream that each received all can comprise a plurality of layers, for example, and basal layer and enhancement layer.Receiver 24 wireless receiver or wire receiver or its combinations.
Detect 710 mistakes that remain in the bit stream.Detected mistake can comprise that those have revised the remaining mistake in wrong back that some is introduced by channel (for example, the wireless channel 26 of Figure 1A or wire message way 28) transmission in lower floor's error correction and detection protocol.As mentioned above, be not that all mistakes all can be revised and lower-layer protocols can mark corrupt data and/or contain wrong data group.The communication layers that the lower-layer protocols that is used for detecting 710 mistakes can be implemented in as mentioned to be discussed.But one or combination in communication layers physical layer, MAC layer (or fluid layer) or transport layer or its combination.Detector means (for example, the error detector among Fig. 6 605) but implementation mistake detects 710.Error detector 605 can use any in the various detection schemes that the those skilled in the art knows, for example, and above Reed-Solomon that is discussed and/or Turbo code scheme.Error detection is attributable to the CRC failure in the Turbo decoder.Error detection is attributable to the failure in the Reed-Solomon decoder.
Can control 715 detected mistakes by several methods.Control 715 can comprise as mentioned being discussed to corrupt data mark (or adding flag) in addition.Control 715 can comprise by identifying the data group (for example grouping, block, section, frame, macro block, sub-macro block) that contains detected mistake locatees mistake.Control 715 can be based on transition layer protocol.This kind agreement can by in transport layer header the coding one or more the position for the upper strata make be used for the mark residual error (referring to, Figure 1B and 1C).The upper strata can be used described transport layer header error indicator further to discern and/or be located the upper-layer packet that is made of one or more error transport layer packets, uses the mistake in the bit stream of upper strata, further location.Control device (for example, the wrong controller 610 among Fig. 6) but the implementation mistake control task.
Determine 720 Fault Distribution.In one aspect, determine Fault Distribution according to the layer, sync agreement.Can receive one or more bit streams of 705 in the analysis of layer, sync place.If layer, sync receives the information of mark corrupt data from a lower floor (for example, communication layers), then its can identify which bit stream which error partly appears.Having this kind information can make the layer, sync agreement can plan upper strata (for example, application layer) error concealment and/or wrong recovery policy.Decide to use different strategies by the size of corrupt data.The transport layer packets that may be marked as mistake can be incorporated in the synchronization layer packets and decide to be forwarded to different application layer component for a part that belongs to which data flow on it.But transport layer packets regular length, but and synchronization layer packets variable-length.Layer, sync can come identification error to distribute so that where part contains the error transport layer packets in the identification variable-length synchronized packets by inserting data.Except that using wrong control 715 information, the layer, sync agreement also can comprise other error-detecting method.These error-detecting methods can comprise the CRC that checks synchronization layer packets.Can be by fixed Fault Distribution being passed to application layer component in the packet of analysis, inserting error flag.Determine that device (for example, the Fault Distribution determiner 615) can be determined 720 to Fault Distribution.
Can be to described one or more each several part enforcement 725 wrong recoveries of encoded multimedia data streams.Mistake is recovered can be based on application layer protocol.Application layer component can be implemented described wrong the recovery.Described application layer component can be discussed as mentioned according to the information that receives from layer, sync and determine to use the mistake of which kind of type to recover.A kind of or combination and other method in timing error concealment, spatial error concealment and the frame rate up conversion (FRUC) that mistake is recovered to comprise above and discussed.The detected mistake of hiding in described one or more multimedia data streams can be based on application layer protocol.Mistake recovers also can support the scalability of described one or more data flow.Discuss as mentioned, scalability can comprise space or time scalability one or the two.Error recovery device (for example, the error recovery component 40 of Fig. 2 A) but implementation mistake recovers 725.Error concealment device (for example, error concealer 620) but implementation mistake concealment.Scalability device (for example, 620 of the scalability component 42 of Fig. 2 A and Fig. 6) can recover to support in 725 scalability at implementation mistake.
Process 700 can comprise reconstruct 730 described one or more bit streams.Reconstruct 730 can comprise that data and the concealed data that will correctly receive are combined.If reconstruct can comprise amount of error in the frame and surpass threshold value then reduce frame rate (a kind of time scalability form).Reconstruct can comprise that selection is to enhancement layer enforcement decoding or concealment or correction (a kind of SNR scalability form).Application layer protocol can be used as and is reconstructed 730 basis.Reconfiguration device (for example, data stream reconstructor 630) can be implemented reconstruct 600.One or more elements can be added, rearranged or combined to system 600.One or more elements can be added, rearranged or combined to process 700.
The example of said method and equipment comprises following example.
A kind of method that is used for the decoding multimedia data, it comprises: receive bit stream, to the bit stream implementation mistake control decoding that is received, wherein wrong control decoding comprises the corrupt bits of discerning unmodified, the bit stream through mistake control decoding that will comprise the corrupt bits of discerning is passed to decoder, replace the position that at least one is discerned, and will comprising through replacing the bitstream decoding of position through mistake control decoding.In one aspect, described method further comprises the information that will be parsed into one or more symbols through the bit stream of mistake control decoding and will discern corrupt bits, and to be parsed into any Symbol recognition that contains corrupt bits be error.In another aspect, described method further comprises: structure contains the error list through profile information that is useful on the identification corrupt bits, and wherein said error list contains the information that identify position is mapped to the position in the sequence of frames of video that is useful on.In aspect another, described method further comprise with the bit stream through error control coding be parsed into first through the analysis bit stream and second through the analysis bit stream, wherein said first through the analysis bit stream be base layer bitstream and described second through the analysis bit stream be enhanced layer bit.
A kind of method that is used for the decoding multimedia data, it comprises: receive bit stream, to the bit stream implementation mistake control decoding that is received, wherein wrong control decoding comprises the corrupt bits of discerning unmodified, the bit stream through mistake control decoding that will comprise the corrupt bits of discerning is passed to decoder, replace at least one institute's discrimination bit by between first frame and second frame, implementing frame rate up conversion, and will be through the bitstream decoding of mistake control decoding.
It will be understood by one of ordinary skill in the art that any technology that can use in numerous kinds of different technologies and the skill and technique comes expression information and signal.For example, data, instruction, order, information, signal, position, symbol and the chip that may mention in the whole above-mentioned explanation can be represented by voltage, electric current, electromagnetic wave, magnetic field or particle, light field or particle or its combination.
The those skilled in the art should be further appreciated that various illustrative logical blocks in conjunction with case description disclosed herein, module, and algorithm steps can be configured to electronic hardware, firmware, computer software, intermediate, microcode or its combination.Be the interchangeability of graphic extension hardware and software clearly, above be based on functional various exemplary components, block, module, circuit, and the step described.It still is that software depends on application-specific and puts on the design constraints of whole system that the functional reality of this kind is configured to hardware.The those skilled in the art can make up described functional at each application-specific in a different manner, but this type of embodiment decision should not be construed as the category that deviates from institute's revealing method.
Various illustrative logical blocks, assembly, module and the circuit of describing in conjunction with example disclosed herein can use general processor, digital signal processor (DSP), application-specific integrated circuit (ASIC) (ASIC), field-programmable gate array capable (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware component or be designed for its arbitrary combination of implementing function described herein and make up or implement.General processor can be microprocessor, but another is chosen as, and processor also can be any traditional processor, controller, microcontroller or state machine.Processor also can be configured to the combination of calculation element, for example, and the associating of the combination of DSP and microprocessor, the combination of a plurality of microprocessors or one or more microprocessors and DSP core, or arbitrary other this type of configuration.
In the software module that the method for describing in conjunction with example disclosed herein or the step of algorithm can be directly contained in the hardware, carried out by processor or in both combinations.Software module can reside in RAM memory, flash memory, ROM memory, eprom memory, eeprom memory, register, hard disc, can substitute in the medium of disk, CD-ROM or arbitrary other form well known in the prior art.Exemplary storage medium is coupled to processor, so that processor can be from read information and to the medium writing information.Another is chosen as, but the part of medium processor.Processor and medium can reside in the application-specific integrated circuit (ASIC) (ASIC).ASIC can reside in the radio modem.Another is chosen as, and processor and medium can be used as discrete component and reside in the radio modem.
Above the explanation to revealed instance is intended to make the those skilled in the art can make or use institute's revealing method and equipment.The those skilled in the art will easily know the various modification of these examples, and the principle that this paper defined also can be applicable to other example and also can add extra element.
Therefore, described in decoder application and to utilize a position error to add flag information and corrupt data is decoded the real-time streaming multimedia corrupt data is implemented the method and apparatus of intelligent error concealment and error correction.
Claims (43)
1. method of integrating for the multilayer of in mistake is recovered, using, it comprises:
Detect mistake in the multi-medium data according to the ground floor agreement; And
Hide described detected mistake in the described multi-medium data according to second layer agreement.
2. the method for claim 1, wherein said ground floor comprises communication layers.
3. method as claimed in claim 2, wherein said communication layers comprise one or combination in physical layer, MAC layer and the transport layer.
4. the method for claim 1, it further comprises:
Control described detected mistake according to transition layer protocol.
5. method as claimed in claim 4 is wherein controlled described detected mistake and is comprised: locatees described detected mistake.
6. the method for claim 1, it further comprises:
Determine the Fault Distribution of described detected mistake according to the layer, sync agreement.
7. the method for claim 1, the wherein said second layer comprises application layer.
8. equipment of integrating for the multilayer of in mistake is recovered, using, it comprises:
Checkout gear, it is used for detecting according to the ground floor agreement mistake of multi-medium data; And
Hiding apparatus, it is used for hiding according to second layer agreement the described detected mistake of described multi-medium data.
9. equipment as claimed in claim 8, wherein said ground floor comprises communication layers.
10. equipment as claimed in claim 9, wherein said communication layers comprise one or combination in physical layer, MAC layer and the transport layer.
11. equipment as claimed in claim 8, it further comprises:
Control device, it is used for controlling described detected mistake according to transition layer protocol.
12. equipment as claimed in claim 11, wherein said control device comprises the device that is used to locate described detected mistake.
13. equipment as claimed in claim 8, it further comprises:
Determine device, it is used for determining according to the layer, sync agreement Fault Distribution of described detected mistake.
14. equipment as claimed in claim 8, the wherein said second layer comprises application layer.
15. the equipment for the multilayer integration of using in mistake is recovered, it comprises:
Detector, it detects mistake in the multi-medium data according to the ground floor agreement; And
Concealer, it hides described detected mistake in the described multi-medium data according to second layer agreement.
16. equipment as claimed in claim 15, wherein said ground floor comprises communication layers.
17. equipment as claimed in claim 16, wherein said communication layers comprise one or combination in physical layer, MAC layer and the transport layer.
18. equipment as claimed in claim 15, it further comprises:
Controller, it controls described detected mistake according to transition layer protocol.
19. equipment as claimed in claim 18, wherein said controller are located described detected mistake.
20. equipment as claimed in claim 15, it further comprises:
Determiner, it determines the Fault Distribution of described detected mistake according to the layer, sync agreement.
21. equipment as claimed in claim 15, the wherein said second layer comprises application layer.
22. the processor for the multilayer integration of using in mistake is recovered, described processor is configured to:
Detect mistake in the multi-medium data according to the ground floor agreement; And
Hide described detected mistake in the described multi-medium data according to second layer agreement.
23. processor as claimed in claim 22, wherein said ground floor comprises communication layers.
24. processor as claimed in claim 23, wherein said communication layers comprise one or combination in physical layer, MAC layer and the transport layer.
25. processor as claimed in claim 22, it further is configured to:
Control described detected mistake according to transition layer protocol.
26. processor as claimed in claim 25, wherein said controller are located described detected mistake.
27. processor as claimed in claim 22, it further is configured to:
Determine the Fault Distribution of described detected mistake according to the layer, sync agreement.
28. processor as claimed in claim 22, the wherein said second layer comprises application layer.
29. a computer-readable media, it embodies the method for integrating for the multilayer of using in mistake is recovered, and described method comprises:
Detect mistake in the multi-medium data according to the ground floor agreement; And
Hide described detected mistake in the described multi-medium data according to second layer agreement.
30. computer-readable media as claimed in claim 29, wherein said ground floor comprises communication layers.
31. computer-readable media as claimed in claim 30, wherein said communication layers comprise one or combination in physical layer, MAC layer and the transport layer.
32. computer-readable media as claimed in claim 29, wherein said method further comprises:
Control described detected mistake according to transition layer protocol.
33. computer-readable media as claimed in claim 32 is wherein controlled described detected mistake and is comprised: locatees described detected mistake.
34. computer-readable media as claimed in claim 29, wherein said method further comprises:
Determine the Fault Distribution of described detected mistake according to the layer, sync agreement.
35. computer-readable media as claimed in claim 29, the wherein said second layer comprises application layer.
36. the method for the multilayer integration of using in mistake is recovered, it comprises:
Detect mistake in the multi-medium data according to the communication layers agreement;
Control described detected mistake according to transition layer protocol;
Determine the Fault Distribution of described controlled mistake according to the layer, sync agreement; And
Hide described detected mistake in the described multi-medium data according to application layer protocol.
37. the equipment for the multilayer integration of using in mistake is recovered, it comprises:
Checkout gear, it detects mistake in the multi-medium data according to the communication layers agreement;
Control device, it controls described detected mistake according to transition layer protocol;
Determine device, it determines the Fault Distribution of described controlled mistake according to the layer, sync agreement; And
Hiding apparatus, it hides described detected mistake in the described multi-medium data according to application layer protocol.
38. the equipment for the multilayer integration of using in mistake is recovered, it comprises:
Detector, it detects mistake in the multi-medium data according to the communication layers agreement;
Controller, it controls described detected mistake according to transition layer protocol;
Determiner, it determines the Fault Distribution of described controlled mistake according to the layer, sync agreement; And
Concealer, it hides described detected mistake in the described multi-medium data according to application layer protocol.
39. the processor for the multilayer integration of using in mistake is recovered, described processor is configured to:
Detect mistake in the multi-medium data according to the communication layers agreement;
Control described detected mistake according to transition layer protocol;
Determine the Fault Distribution of described controlled mistake according to the layer, sync agreement; And
Hide described detected mistake in the described multi-medium data according to application layer protocol.
40. a computer-readable media, it embodies the method for integrating for the multilayer of using in mistake is recovered, and described method comprises:
Detect mistake in the multi-medium data according to the communication layers agreement;
Control described detected mistake according to transition layer protocol;
Determine the Fault Distribution of described controlled mistake according to the layer, sync agreement; And
Hide described detected mistake in the described multi-medium data according to application layer protocol.
41. the equipment for use in multi-medium data is handled, it comprises:
Error recovery component, its mistake of implementing encoded multi-medium data is recovered; And
Scalability component, it supports the scalability of described encoded multi-medium data.
42. equipment as claimed in claim 41, wherein said scalability comprise one in space and the time scalability or the two.
43. equipment as claimed in claim 42, wherein said wrong one or the combination that recovers to comprise in timing error concealment, spatial error concealment and the frame rate up conversion.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201010146123.8A CN101790098B (en) | 2005-03-10 | 2006-03-10 | For optimizing the decoder architecture of mismanage in streamed multimedia |
Applications Claiming Priority (7)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US66086705P | 2005-03-10 | 2005-03-10 | |
| US66092305P | 2005-03-10 | 2005-03-10 | |
| US66068105P | 2005-03-10 | 2005-03-10 | |
| US60/660,923 | 2005-03-10 | ||
| US60/660,681 | 2005-03-10 | ||
| US60/660,867 | 2005-03-10 | ||
| PCT/US2006/008763 WO2006099223A2 (en) | 2005-03-10 | 2006-03-10 | A decoder architecture for optimized error management in streaming multimedia |
Related Child Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201010146123.8A Division CN101790098B (en) | 2005-03-10 | 2006-03-10 | For optimizing the decoder architecture of mismanage in streamed multimedia |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101176353A CN101176353A (en) | 2008-05-07 |
| CN101176353B true CN101176353B (en) | 2010-09-15 |
Family
ID=37908442
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN200680014271.6A Expired - Fee Related CN101176353B (en) | 2005-03-10 | 2006-03-10 | Decoder architecture for optimized error management in streaming multimedia |
Country Status (3)
| Country | Link |
|---|---|
| CN (1) | CN101176353B (en) |
| AR (1) | AR052602A1 (en) |
| UA (1) | UA92004C2 (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100262708A1 (en) * | 2009-04-08 | 2010-10-14 | Nokia Corporation | Method and apparatus for delivery of scalable media data |
| US9219925B2 (en) | 2010-01-14 | 2015-12-22 | Sumitomo Electric Industries, Ltd. | Video image coded data display method, device, and communications system |
-
2006
- 2006-03-10 AR ARP060100926 patent/AR052602A1/en active IP Right Grant
- 2006-03-10 CN CN200680014271.6A patent/CN101176353B/en not_active Expired - Fee Related
- 2006-03-10 UA UAA200711132A patent/UA92004C2/en unknown
Also Published As
| Publication number | Publication date |
|---|---|
| CN101176353A (en) | 2008-05-07 |
| UA92004C2 (en) | 2010-09-27 |
| AR052602A1 (en) | 2007-03-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101790098B (en) | For optimizing the decoder architecture of mismanage in streamed multimedia | |
| EP1882343B1 (en) | Improving error resilience using out of band directory information | |
| KR100939951B1 (en) | Error recovery using in band error patterns | |
| US20070150784A1 (en) | Method and system for data and video coding | |
| Cai et al. | FEC-based video streaming over packet loss networks with pre-interleaving | |
| CN101176353B (en) | Decoder architecture for optimized error management in streaming multimedia | |
| RU2374787C2 (en) | Structure of decoder for optimised control of error processing in multimedia data streaming | |
| Gharavi et al. | Partitioning of MPEG coded video bitstreams for wireless transmission | |
| Cai et al. | Use of pre-interleaving for video streaming over wireless access networks | |
| Rhaiem et al. | New robust decoding scheme-aware channel condition for video streaming transmission | |
| Nemethova et al. | Cross-layer Error Resilience Mechanisms | |
| Agueh et al. | Optimal switching for smart and robust JPEG 2000 images and video transmission over wireless networks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 1114276 Country of ref document: HK |
|
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: WD Ref document number: 1114276 Country of ref document: HK |
|
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20100915 |