EP2192577B1 - Optimierung von MP3-Kodierung mit vollständiger Dekodiererkompatibilität - Google Patents

Optimierung von MP3-Kodierung mit vollständiger Dekodiererkompatibilität Download PDF

Info

Publication number
EP2192577B1
EP2192577B1 EP08170396A EP08170396A EP2192577B1 EP 2192577 B1 EP2192577 B1 EP 2192577B1 EP 08170396 A EP08170396 A EP 08170396A EP 08170396 A EP08170396 A EP 08170396A EP 2192577 B1 EP2192577 B1 EP 2192577B1
Authority
EP
European Patent Office
Prior art keywords
scale
cost function
encoding
factors
quantization
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP08170396A
Other languages
English (en)
French (fr)
Other versions
EP2192577A1 (de
Inventor
Guixing Wu
En-Hui Yang
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
BlackBerry Ltd
Original Assignee
Research in Motion Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Research in Motion Ltd filed Critical Research in Motion Ltd
Priority to EP08170396A priority Critical patent/EP2192577B1/de
Priority to AT08170396T priority patent/ATE532175T1/de
Priority to CA2686264A priority patent/CA2686264C/en
Publication of EP2192577A1 publication Critical patent/EP2192577A1/de
Application granted granted Critical
Publication of EP2192577B1 publication Critical patent/EP2192577B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
    • G10L19/02—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders
    • G10L19/032—Quantisation or dequantisation of spectral components
    • G10L19/035—Scalar quantisation

Definitions

  • Example embodiments herein relate to audio signal encoding, and in particular to rate-distortion optimization for MP3 encoding.
  • MP3 MPEG I/II Layer-3
  • An example MP3 encoder is LAME, which refers to "LAME Ain't an Mp3 Encoder", as is known in the art.
  • Another MP3 encoder is ISO reference codec, which is based on the ISO standard.
  • MP3 encoders include use of two nested loop search (TNLS) algorithms, which are computationally complex and may not be guaranteed to converge. These encoders may be configured or operated to provide for additional functionality and customization.
  • TNLS nested loop search
  • the encoding algorithm is not standardized in MP3, the basic structure and syntax-related tools are fixed so that the MP3 encoded/compressed bitstreams can be correctly decoded by any standard compatible decoder. However, there may be opportunities to manipulate the encoding algorithm while maintaining full decoder compatibility.
  • Figure 1 shows an MP3 encoding process to which example embodiments may be applied
  • Figure 2 shows a flow diagram of an optimization process in accordance with an example embodiment
  • Figure 3 shows a graph of an optimal path search algorithm for use in the process of Figure 2 ;
  • Figure 4 shows the graph of Figure 3 , illustrating an optimal path
  • Figure 5 shows a flow diagram of a process to be used in the optimization process of Figure 2 ;
  • Figure 6 shows a graph of performance characteristics of an example embodiment, for encoding of audio file waltz.wav as compared to ISO reference codec;
  • Figure 7 shows a graph of performance characteristics of an example embodiment, for encoding of audio file waltz.wav as compared to LAME;
  • Figure 8 shows a graph of performance characteristics of an example embodiment, for encoding of audio file vioin.wav as compared to ISO reference codec;
  • Figure 9 shows a graph of performance characteristics of an example embodiment, for encoding of audio file violin.wav as compared to LAME.
  • Figure 10 shows an encoder for optimizing encoding performance of MP3 in accordance with an example embodiment.
  • the present application provides a method for optimizing audio encoding of a source sequence, the encoding being dependent on quantization factors, the quantization factors including a global quantization step size and scale factors.
  • the method includes defining a cost function of the encoding of the source sequence, the cost function being dependent on the quantization factors.
  • the method includes initializing fixed values of the scale factors; and determining values of the quantization factors which minimize the cost function by iteratively performing:
  • PE Perceptual Entropy of an encoded frame
  • R is an encoding bit rate
  • M is the number of audio samples to be encoded
  • c 1 , c 2 and c 3 are constants
  • calculating the cost function using ⁇ is Perceptual Entropy of an encoded frame
  • the present application provides an encoder for optimizing audio encoding of a source sequence, the audio encoding being dependent on quantization factors, the quantization factors including a global quantization step size and scale factors.
  • the encoder includes a controller, a memory accessible by the controller, a cost function of an encoding of the source sequence stored in memory, the cost function being dependent on the quantization factors; and a predetermined threshold of the cost function stored in the memory.
  • the controller is configured to access the cost function and predetermined threshold from memory, initialize fixed values of the scale factors, and determine values of the quantization factors which minimize the cost function by iteratively performing:
  • FIG. 1 shows an MP3 encoding process 20 to which example embodiments may be applied.
  • the MP3 encoding process 20 receives digital audio input 22 and produces a compressed or encoded output 32 in the form of a bitstream for storage and transmission.
  • the encoding process 20 may for example be implemented by an encoder such as a suitably configured computing device.
  • continuous lines denote the time or spectral domain signal flow, and dash lines denote the control information flow.
  • the encoding process 20 includes audio input 22 for input to a time/frequency (T/F) mapping module 24 and a psychoacoustic model module 26.
  • a quantization and entropy coding module 28 and a frame packing module 30 are also shown.
  • the encoding process 20 results in an encoded output 32 of the audio input 22, for example for sending to a decoder for subsequent decoding.
  • the audio input 22 (in time domain) are first input into the T/F mapping module 24, which converts the audio input 22 into spectral coefficients.
  • the T/F mapping module 24 is composed of three steps: pseudo-quadrature mirror filter (PQMF), windowing and modified discrete cosine transform (MDCT), and aliasing reduction.
  • PQMF pseudo-quadrature mirror filter
  • MDCT modified discrete cosine transform
  • aliasing reduction aliasing reduction.
  • the PQMF filterbank splits a so-called granule (in MPEG I and II layer 3 each audio frame contains 2 and 1 granules respectively) of 576 input audio samples into 32 equally spaced subbands, where each subband has 18 time domain audio samples.
  • the 18 time domain audio samples in each subband are then combined with their counterpart of the next frame, and processed by a sine-type window based on psychoacoustic modeling decisions.
  • a long window which covers a whole length of 36, addresses stationary audio parts.
  • Long windowing with MDCT afterwards ensures a high frequency resolution, but also causes quantization errors spreading over the 1152 time-samples in the process of quantization.
  • a short window is used to reduce the temporal noise to spread for the signals containing transients/attacks.
  • audio signals with a length of 36 are divided into 3 equal sub-blocks.
  • two transition windows, long-short (start) and short-long (stop), which have the same size as a long window are employed.
  • the psychoacoustic model module 26 is generally used to generate control information for the T/F mapping module 24, and for the quantization and entropy coding module 28. Based on the control information from the psychoacoustic model module 26, the spectral coefficients which are output from the T/F mapping module 24 are received by the quantization and entropy coding module 28, and are quantized and entropy coded. Finally these compressed bits streams are packed up along with format information, control information and other auxiliary data in MP3 frames, and output as the encoded output 32.
  • the MP3 syntax leaves the selection of quantization step sizes and Huffman codebooks to each encoder or encoding algorithm, which provides opportunity to apply rate-distortion consideration.
  • a conventional MP3 encoding algorithm is now be described as follows, which employs a "hard decision quantization", a two nested loop search (TNLS) algorithm, and fixed or static Huffman codebooks.
  • the MP3 quantization and entropy coding module 28 first subdivides an entire frame of 576 spectral coefficients into 21 or 12 scale factor bands for a long window block (including long-short window and short-long window) or a short window block respectively.
  • each of the parameters listed in (2.2) may be referred to as a "scale factor”, and all of which may be collectively referred to herein as “scale factors”, as appropriate.
  • global_gain and the scale factors may collectively be referred to herein as “quantization factors”.
  • sub_block is only used for short windows, and it refers to one of the 3 sub-blocks for a short window.
  • scalefac[sub_block][sb] is a scale factor parameter for scale factor band sb to color the quantization noise.
  • scalefac[sub_block][sb] are variable length transmitted according to scalefac_compress which occupies 4 bits (MPEG-1) or 9 bits (MPEG-2) in the side information of MP3 encoded frames.
  • preflag is a shortcut for additional high frequency amplification of the quantized values. If preflag is set, the values of a fixed table pretab[sb] are added to the scale factors. preflag is never used in short windows (for the purposes of the standard).
  • subblock_gain[sub_block] is the gain offset for the short window.
  • scalefac_scale is a one-bit parameter used to control the quantization step size.
  • the quantized spectral coefficients are then encoded by static Huffman coding, which utilizes 34 fixed Huffman codebooks.
  • static Huffman coding utilizes 34 fixed Huffman codebooks.
  • MP3 subdivides the entire quantized spectrum into three regions. Each region is coded with a different set of Huffman codebooks that best match the statistics of that region. Specifically, at high frequencies, MP3 identifies a region of "all zeros". The size of this region can be deduced from the sizes of the other two regions, and the coefficients in this region don't need to be coded. The only restriction is that it must contain an even number of zeros since the other two regions group their values in 2- or 4-tuples.
  • the second region contains a series of contiguous values consisting only of -1, 0, +1 just before the "zero” region, and is encoded in 4-tuples by Huffman codebook 32 or 33.
  • the low frequency region covers the remaining coefficients which are encoded in pairs. This region is further subdivided into 3 (for long window) or 2 (for short, long-short and short-long window) parts with each covered by a distinct Huffman codebook.
  • a noise shaping method may be applied to find the proper global quantization step size global_gain and scale factors before the actual quantization.
  • Some conventional algorithms use the TNLS algorithm to jointly control the bit rate and distortion.
  • the TNLS algorithm consists of an inner (rate control) loop and an outer (noise control) loop.
  • the task of the inner loop is to change the global quantization step size global_gain such that the given spectral data can just be encoded with the number of bits available. If the number of bits resulting from Huffman coding exceeds this number, the global_gain can be increased to result in a larger quantization step size, leading to smaller quantized values. This operation is repeated until the resulting bit demand for Huffman coding is small enough.
  • the TNLS algorithm may require quantization step sizes so small to obtain the best perceptual quality. On the other hand, it has to increase to the quantization step sizes to enable coding at the required bit rate. These two requirements are conflicting. Therefore, this conventional algorithm does not guarantee to converge.
  • soft decision quantization instead of the hard decision quantization, is applied, and the corresponding purpose of quantization and entropy coding in MP3 encoding is to achieve the minimum perceptual distortion for a given encoding bit rate by solving, mathematically, the following minimization problem: ⁇ min y , q , p , h ⁇ D w xr rxr , subject to R q + R y P H ⁇ R 1
  • xr is the original spectral signal
  • rxr is the reconstructed signal obtained from the quantized spectral coefficients y
  • P and H represent Huffman codebook region partition and Huffman codebooks selection respectively
  • q denotes the quantization factors including global_gain and scale factors
  • R ( q ) and R ( y , P, H ) are the bit rates to encode q and the quantized spectral coefficients y respectively
  • R 1 is the rate constraint
  • y is not calculated according to (2.1) any more; instead, it is treated as a variable in a cost function involving the distortion and rates, and has to be determined jointly along with q, P, and H.
  • Average noise-to mask ratio (ANMR) is used as the distortion measure.
  • the noise-to mask ratio (NMR), the ratio of the quantization noise to the masking threshold, is a widely used objective measure for the evaluation of an audio signal.
  • N is the number of scale factor bands
  • w [sb] is the inverse of the masking threshold for scale factor band sb
  • d[sb] is the quantization distortion, mean squared quantization error for scale factor band sb.
  • FIG. 2 shows a flow diagram of an optimization process 50 in accordance with an example embodiment.
  • the exact order of steps may vary from those shown in Figure 2 in different applications and embodiments. It can also be appreciated that more or less steps may be required in some example embodiments, as appropriate.
  • the parameters y , q, P and H are jointly optimized.
  • the general framework for the process 50 has been outlined previously in Xu and E.-h. Yang, "Rate-distortion optimization for MP3 audio coding with complete decoder compatibility," in Proc. 2005 IEEE Workshop on Multimedia Signal Processing, Oct. 2005 .
  • the process 50 selects the quantized spectral coefficients y and Huffman codebook region division P, quantization factors q and Huffman codebook region selection H alternatively to minimize the Lagrangian cost J.
  • the iterative searching for the parameters may be referred to as "soft-decision quantization” (rather than the formulaic "hard-decision quantization” of (2.1), described above).
  • the iterative algorithm of the process 50 can be described as follows.
  • step 52 specify a tolerance ⁇ as the convergence criterion for the Lagrangian cost J.
  • q t and H t are fixed or given for any t ⁇ 0 .
  • y t and P t achieve the minimum min y
  • P J ⁇ D w ⁇ xr , Q - 1 q y + ⁇ ⁇ R q t + R y P H t
  • Q -1 Q -1 ( q , y ) is used to generate the reconstructed signal rxr.
  • J ⁇ ( y t , q t , P t , H t ) by J ' ⁇ .
  • step 60 given y t , P t and q t+1 , update H t to H t+1 so that H t+1 achieves the minimum min H R ⁇ y t P t + 1 H t .
  • the final y , q, P and H may thereafter be provided for MP3 coding of xr .
  • Figure 3 shows a graph 80 of an optimal path search algorithm for use in the process of Figure 2 ; while Figure 4 shows an optimal path of the graph 80.
  • the graph 80 is defined with 4 layers (shown as I, II, III, and IV) and 288 nodes in each layer as shown in Figure 3 .
  • the 4 layers correspond to the three divisions of the big_value region and the count_1 region.
  • Two special states, frame_begin and frame_end denote the start and end of the frame respectively.
  • a cost which is defined as the minimum incremental Lagrangian cost of quantizing and Huffman encoding the coefficients of state S L,i (or states S L,i-1 and S L,i if L IV ) by using the Huffman codebook selected for layer L.
  • q j is the corresponding scale factor for y j
  • r L ( ⁇ ) denotes the codeword length by using the Huffman codebook selected for layer L.
  • every sequence of connections from the frame_begin state to the frame_end state corresponds to a Huffman codebook region division of the entire frame with a Lagrangian cost.
  • the sequence of connection in Figure 4 assigns scale factor band 0 and 1 to the fist two subdivisions of the big_value region respectively, the next 4 coefficients to the count_1 region, and the rest to the zero region.
  • any Huffman codebook region division of the entire frame that is compatible with the standard can be represented by a sequence of connections from the frame_begin to the frame_end state in the graph 80.
  • the algorithm preselects and stores the best quantized coefficients based on minimizing the Lagrangian cost of (3.7) for each legitimate state S L,i , and sets their associated cost as the cost of each connection to that state.
  • the algorithm also recursively precalculates, for each state, the distortion/cost resulting from ending the frame at that state, i.e., the cost of its connection to the state frame_end.
  • the algorithm begins with the state frame_begin by storing the cost of dropping the entire frame in J frame_begin .
  • the cost of each state is set to the cost of corresponding incoming connection, and added with the cost of dropping the remaining coefficients to get J I,0 and J IV,0 , respectively.
  • only states S L,1 has an incoming connection from states S I,0 .
  • Set its cost to the sum of the costs of state S I ,0 and the connection between S I,0 and S I,1 , and add it with the cost of dropping the remaining coefficients to get J I,1 .
  • a three-layer graph could be constructed for other three window cases.
  • Step 58 generally determines the quantization factors q (i.e., scale factors and global_gain ) that minimize the combined cost of weighted distortion and bit rate for encoding or transmittal. Given the nonuniform quantizer and nonlinear bit rate for quantization factors in the standard, there is no direct formula to calculate the optimal quantization factors. Direct search through all combinations of global_gain, scalefac_compress, scalefac, scalfac_scale, and subblock_gain (for short windows) or preflag (for other windows) may be computationally complex.
  • the method 100 includes the following alternating minimization procedure to minimize the combined cost.
  • global_gain is determined while the scale factors are fixed, and at step 104 the scale factors are determined while global_gain is fixed. This is repeated iteratively until the calculated rate-distortion cost is within a predetermined threshold.
  • step 102 update global_gain when scalefac, scalfac_sca / e and subblock_gain (for short windows) or preflag (for other windows) are fixed.
  • the bit rate for the transmission of scale factors is fixed. Therefore, at this stage only the encoding distortion is minimized, while rate is not considered.
  • s[sb] global_gain-210-scale_factor[sb]
  • l[sb] and l[sb+1]-1 are the start and end positions for scale factor band sb respectively
  • w[sb] is the inverse of the masking threshold for scale factor band sb.
  • step 104 fix global_gain. Update the scale factors scalefac, scalfac_scale and subblock_gain (for short windows) or preflag (for other windows) to minimize the combined cost of weighted distortion and bit rate for transmitting the scale factors.
  • preflag is equal to 0 or 1.
  • the value of pretab[sb] is typically fixed and is of the form as shown in Table 1.
  • scalefac_scale is equal to 0 or 1.
  • scalefac_compress determines the number of bits used for the transmission of the scalefactors according to Table 2.
  • bit length may be a first bit length for a first group of scale factor bands and the bit length may be a second bit length for a second group of scale factor bands.
  • slen1 is the bit length of scalefac for each of scalefactor bands 0 to 10
  • slen2 is the bit length of scalefac for each of scalefactor bands 11 to 20.
  • the maximum length for slen1 is 4 while the maximum length for slen2 is 3 (as based on the MP3 standard).
  • slen1 and s len2 are given, in some example embodiments, one can find the minimum encoding distortion for each scalefactor band and the corresponding scalefac[sb] which generates the minimum encoding distortion.
  • preflag and scalfac_scale are fixed, there only needs to be calculated 5 (the first 11 bands) or 4 (the last 10 bands) different cases of encoding distortion for each scale factor band, rather than calculate the encoding distortion 16 times for different scalefac_compress.
  • the pre-calculated encoding distortion is minimized with a certain value for scalefac[sb] given the length slen1 or slen2.
  • sf[sb][slen] the value for scalefac[sb] such that the weighted distortion is minimized for scale factor band sb when the bit length used for transmitting scalefac[sb] is slen.
  • s[sb] in equation (3.9) can be freely chosen. That is, s[sb] is not restricted by the value of scalefac[sb] to be one of the 16 integer numbers (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15). Apply the minimum mean square error criterion to find the minimum weighted distortion for (3.9).
  • scalefac[sb] cannot be freely chosen in reality (as defined by the standard), that is, it must be constrained to one of the 16 integer numbers (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15).
  • the value of scalefac[sb] can be determined using the following algorithm. Generally, it is determined whether T exceeds encoding within slen, and if so constraining T to within slen :
  • each scale factor band there exists one redundant case for each scale factor band if scalefac[sb] is equal to 0 (i.e., (3.16) may be calculated once). As a result, in some example embodiments, there are 9 (the first 11 scale factor bands) or 15 (the last 10 scale factor bands) different cases of encoding distortion for each scale factor band.
  • the total Lagrangian cost is the addition of the encoding distortion and the bit rate. Therefore, what remains is the addition of bit rate to calculate the combined cost.
  • the distortion based on bit rate for the transmission of all scale factors can also be looked up from a pre-generated table, as is known in the art. Similarly, for other window cases, a similar approach could be applied to reduce the computational complexity.
  • step 106 repeat steps 102 and 104 until the decrease of the combined cost is below a prescribed threshold. If the predetermined threshold is reached, at step 110 output the final global_gain and scale factors ( scalefac, scalfac_scale, preflag / subblock_gain ), and then ends at step 112 (or proceed to the next step in method 50 ( Figure 2 )).
  • the iterative method 100 generally converges after two rounds of iteration, the number of different cases to be computed for each scale factor band of an MPEG-1 encoded, long-window frame has been reduced from 16384 to 18 (the first 11 bands) or 30 (the last 10 bands).
  • the particular quantization factors or scale factors to be determined may depend on the particular application or coding scheme, and may not be limited to the parameters global_gain, scalefac, scalfac_scale, and preflag / subblock_gain.
  • step 60 Given Huffman coding region division P, the quantization factors q and quantized spectral coefficients y , determining the Huffman codebook H ay be performed as follows: for each region, every Huffman codebook that has encodable value limit larger than or equal to the greatest coefficient amplitude of that region is considered, and the one with the minimum codeword length is selected.
  • c 1 , c 2 and c 3 are determined from the experimental data using the least square criterion. This is for example generally described in C.
  • X represents independent variables PE and R.
  • y represents the dependent variable ⁇ final R .
  • fun represents the formula (4.1).
  • beta0 is a vector containing initial values for the coefficients for c 1 , c 2 and c 3 .
  • the average number of iterations was tested over the Lagrangian multiplier if the formula (4.1) with the above estimated coefficient is used as the initial point for the bisection search.
  • the average number of iterations over the Lagrangian multiplier is 1.5.
  • the average number of iterations over the Lagrangian multiplier ranges from 4 to 8 if an arbitrary number is used as the initial point. Therefore, on the average, using (4.1) as the initial point can run 4 times as fast as the method in which an arbitrary initial point is used.
  • Figure 6 shows a graph 140 of performance characteristics of an example embodiment, showing a comparison of the method 50 (Figure 20) for encoding of audio file waltz.wav as compared to ISO reference codec.
  • Figure 7 shows a graph 150 of performance characteristics of an example embodiment, for encoding of audio file waltz.wav as compared to LAME.
  • Figure 8 shows a graph 160 of performance characteristics of an example embodiment, for encoding of audio file vioin.wav as compared to ISO reference codec.
  • Figure 9 shows a graph 170 of performance characteristics of an example embodiment, for encoding of audio file violin.wav as compared to LAME.
  • the LAME MP3 encoder features a psychoacoustic model, joint stereo encoding and variable bit-rate encoding.
  • LAME still uses the basic structure of typical TNLS.
  • a refining TNLS is used to minimize the total noise to masking ratio for an entire frame after the successful termination of search process given its typical TNLS. Specifically, during each outer loop, the band with maximum noise to masking ratio is amplified and the best result based on total noise to mask ratio is stored.
  • the method 50 ( Figure 2 ) is implemented as described above. For each case, the perceptual model, joint stereo encoding mode and window switching decision are kept intact.
  • Figure 6 shows the rate-distortion performance of the method 50 ( Figure 2 ) (denoted as "RD optimization" in the graph 140) applied to ISO reference encoder, when compared to a conventional or normal ISO reference encoder implementing TNLS, in constant bit-rate mode for waltz.wav.
  • the test file may for example be encoded at 48khz, 2 channel, 16 bits/sample, 30 seconds.
  • ISO-HO represents the optimal Huffman tables used for Huffman coding
  • ISO-NH means that the first Huffman table satisfying the coding limit is selected for each Huffman coding region.
  • the vertical axes denote the average noise to mask ratio over all audio frames.
  • Figure 7 depicts the rate-distortion performance of the method 50 ( Figure 2 ) (also denoted as "RD optimization) applied to LAME when compared to the LAME reference encoder (implementing conventional TNLS) in constant bit-rate mode for waltz.wav. It is shown separately from ISO reference encoder because ISO reference encoder and LAME adopt different perceptual models. For an unbiased comparison, in some example embodiments the LAME encoder disables the functions of amplitude scaling and low pass filter. In Figure 7 , “LAME” means that the audio file is compressed using LAME's normal compression mode. As shown, the method 50 ( Figure 2 ) outperforms LAME in terms of compression performance. At 96kbps, the proposed optimization algorithm achieves about 1.34dB ANMR gain over LAME.
  • Figures 8 and 9 compare the compression performance of the method 50 ( Figure 2 ) for the music file violin.wav (MPEG lossless audio coding test file, 48khz, 2 channel, 16 bits/sample, 30 seconds) in constant bit-rate mode.
  • Figure 8 shows results from ISO reference encoder
  • Figure 9 shows results from LAME. It may be observed that "RD optimization" has improved rate-distortion over the conventional reference encoders. Similar results may be observed for other test music files.
  • y j is determined from (2.1)) and a is a fixed integer.
  • a is a fixed integer.
  • the average number of iterations over the Lagrangian multiplier is 1.5 if the formula (4.1) is used as the initial point.
  • the average number of iterations over the Lagrangian multiplier ranges from 4 to 8 if an arbitrary number is used as the initial point.
  • Table 3 lists the computation time (in seconds) on a Pentium PC, 2.16GHZ, 1G by tes of RAM to encode violin.wav and waltz.wav at different transmission rates for the method 50 based on LAME reference codec.
  • the proposed optimization algorithm generally reaches real time throughput, which suggests that the method 50 is computationally efficient.
  • the computation time is generally less than 30 seconds.
  • the computation time for ISO-based encoders is not listed, but are generally less-efficient than LAME-based encoders in both the computation time and compression performance.
  • the encoder 300 may for example be implemented on a suitable configured computer device.
  • the encoder 300 includes a controller such as a microprocessor 304 that controls the overall operation of the encoder 300.
  • the microprocessor 304 may also interact with other subsystems (not shown) such as a communications subsystem, display, and one or more auxiliary input/output (I/O) subsystems or devices.
  • the encoder 300 includes a memory 304 accessible by the microprocessor 304. Operating system software 306 and various software applications 308 used by the microprocessor 302 are, in some example embodiments, stored in memory 304 or similar storage element.
  • MP3 software application 310 such as the ISO-based encoder or LAME-based encoder described above, may be installed as one of the various software applications 308.
  • the microprocessor 302 in addition to its operating system functions, in example embodiments enables execution of software applications 308 on the device.
  • the encoder 300 may be used for optimizing performance of MP3 encoding of a source sequence. Specifically, the encoder 300 may enable the microprocessor 304 to determine quantization factors (for example including a global quantization step size and scale factors) for the source sequence.
  • the memory 304 may contain a cost function of an encoding of the source sequence, wherein the cost function is dependent on the quantization factors.
  • the memory 304 may also contain a predetermined tolerance of the cost function stored in the memory 304. Instructions residing in memory 304 enable the microprocessor 203 to access the cost function and predetermined tolerance from memory 304, determine the quantization factors which minimize the cost function within the predetermined tolerance, and store the determined quantization factors in memory 304 for MP3 encoding of the source sequence.
  • an iterative method is performed such that global_gain is determined while the scale factors are fixed, and the scale factors are determined while global_gain is fixed. This is repeated until a calculated rate-distortion cost is within a predetermined threshold.
  • the MP3 software application 310 may be used to perform MP3 encoding using the determined quantization factors.
  • the encoder 300 may be configured for optimizing of parameters including quantization factors, in a manner similar to the example methods described above.
  • the encoder 300 may be configured to perform the method 50 ( Figure 2 ).
  • example embodiments may be adapted to or implemented by other forms of signal encoding or audio signal encoding, for example Advanced Audio Coding.

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Spectroscopy & Molecular Physics (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Human Computer Interaction (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)
  • Compression Of Band Width Or Redundancy In Fax (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Claims (15)

  1. Verfahren zum Optimieren von Audiocodierung einer Quellsequenz, wobei das Codieren von Quantisierungsfaktoren abhängig ist, wobei die Quantisierungsfaktoren eine globale Quantisierungsschrittgröße und Skalierungsfaktoren umfassen, wobei das Verfahren aufweist:
    Definieren einer Kostenfunktion des Codierens der Quellsequenz, wobei die Kostenfunktion von den Quantisierungsfaktoren abhängig ist;
    Initialisieren von festen Werten der Skalierungsfaktoren; und
    Bestimmen von Werten der Quantisierungsfaktoren, die die Kostenfunktion minimieren, durch iteratives Durchführen:
    Bestimmen, für die festen Werte der Skalierungsfaktoren, eines Werts der globalen Quantisierungsschrittgröße, der die Kostenfunktion minimiert,
    Festlegen des bestimmten Werts der globalen Quantisierungsschrittgröße und Bestimmen von Werten von Skalierungsfaktoren, die die Kostenfunktion minimieren, und
    Festlegen der bestimmten Werte der Skalierungsfaktoren, und Bestimmen, ob die Kostenfunktion unter einer vorgegebenen Schwelle ist, und wenn dem so ist, Beenden des iterativen Durchführens.
  2. Verfahren gemäß Anspruch 1, wobei die Kostenfunktion auf einer Verzerrung der Codierung der Quellsequenz basiert und weiter auf einer Kompromiss-Funktion basiert, die einen Kompromiss einer Rate der Verzerrung repräsentiert, wobei die Rate eine Übertragungsbitrate der Codierung der Quellsequenz ist.
  3. Verfahren gemäß Anspruch 2, wobei, in dem Schritt des Festlegens des bestimmten Werts der globalen Quantisierungsschrittgröße und des Bestimmens von Werten von Skalierungsfaktoren, die die Kostenfunktion minimieren, die Verzerrung aus einer vorher erzeugten Tabelle erlangt wird.
  4. Verfahren gemäß Anspruch 2, wobei die Kompromiss-Funktion λ umfasst, wobei das Verfahren weiter aufweist:
    Berechnen von λ als: λ final R = c 1 ln 10 10 ⁢ M × 10 c 2 ⁢ PE - c 3 ⁢ R / M
    Figure imgb0036
     wobei PE eine Wahrnehmungsentropie (Perceptual Entropy) eines codierten Rahmens ist, R die Rate ist, M eine Anzahl von zu codierenden Audioabtastwerten ist, und c1, c2 und c3 Konstanten sind; und Berechnen der Kostenfunktion unter Verwendung von λ.
  5. Verfahren gemäß Anspruch 1, wobei das Bestimmen des Werts der globalen Quantisierungsschrittgröße ein Berechnen umfasst: 4 log 10 ⁢ 2 ⁢ log 10 ⁢ ∑ sb = 1 N b sb ∑ sb = 1 N a sb + 210
    Figure imgb0037
     wobei b sb = 2 - scale_factor sb / 4 ⋅ w sb ⋅ ∑ i = l sb l ⁢ sb + 1 - 1 x ⁢ r i ⋅ y i 4 / 3
    Figure imgb0038
     und a sb = 2 - scale_factor sb / 2 ⋅ w sb ⋅ ∑ i = l sb l ⁢ sb + 1 - 1 y i 4 / 3
    Figure imgb0039
     wobei xri die Quellsequenz ist, scale_factor[sb] eine Quantisierungsschrittgröße für das Skalierungsfaktorband sb ist, l[sb] und l[sb+1]-1 Start- beziehungsweise Endpositionen für das Skalierungsfaktorband sb sind, w[sb] ein Inverses der Maskierungsschwelle für das Skalierungsfaktorband sb ist, und yi ein quantisierter Spektralkoeffizient der Quellsequenz ist.
  6. Verfahren gemäß Anspruch 1, das weiter aufweist ein Begrenzen der Skalierungsfaktoren auf eine Bitlänge.
  7. Verfahren gemäß Anspruch 6, wobei die Bitlänge eine erste Bitlänge für eine erste Gruppe von Skalierungsfaktorbändern ist und die Bitlänge eine zweite Bitlänge für eine zweite Gruppe von Skalierungsfaktorbändern ist.
  8. Verfahren gemäß Anspruch 1, wobei die Skalierungsfaktoren einen Parameter scalefac umfassen, der ein Skalierungsfaktor für ein bestimmtes Skalierungsfaktorband ist, wobei das Verfahren weiter aufweist ein Begrenzen von scalefac auf eine Bitlänge.
  9. Verfahren gemäß Anspruch 8, wobei die Bitlänge eine erste Bitlänge für eine erste Gruppe von Skalierungsfaktorbändern ist und die Bitlänge eine zweite Bitlänge für eine zweite Gruppe von Skalierungsfaktorbändern ist.
  10. Verfahren gemäß Anspruch 1, wobei die Skalierungsfaktoren einen Parameter scalefac umfassen, der ein Skalierungsfaktor für ein bestimmtes Skalierungsfaktorband ist, wobei das Verfahren weiter aufweist:
    Berechnen eines Werts von scalefac, der die Kostenfunktion minimiert; und
    Bestimmen, ob der berechnete Wert von scalefac eine Codierung innerhalb einer Bitlänge übersteigt, und wenn dem so ist, Begrenzen von scalefac auf eine Bitlänge.
  11. Verfahren gemäß Anspruch 10, wobei der Schritt des Berechnens des Werts von scalefac ein Berechnen umfasst: 4 log 10 ⁢ 2 ⁢ log 10 ⁢ ∑ i = l sb l ⁢ sb + 1 - 1 x ⁢ r i ⋅ y i 4 / 3 ∑ i = l sb l ⁢ sb + 1 - 1 y i 8 / 3
    Figure imgb0040
    wobei xri die Quellsequenz ist, l[sb] und l[sb+1]-1 Start- beziehungsweise Endpositionen für das Skalierungsfaktorband sb sind, und yi ein quantisierter Spektralkoeffizient der Quellsequenz ist.
  12. Verfahren gemäß Anspruch 10, das weiter aufweist:
    Berechnen der Kostenfunktion für scalefac = 0 für ein Skalierungsfaktorband; und
    Speichern in einem Speicher der berechneten Kostenfunktion für scalefac = 0 für ein nachfolgendes Abrufen.
  13. Verfahren gemäß Anspruch 1, wobei die Audiocodierung eine "MPEG I/II Schicht-3"-Codierung ist.
  14. Verfahren gemäß Anspruch 1, wobei die Codierung weiter abhängig ist von quantisierten Spektralkoeffizienten, Huffman-Codebüchern und Huffman-Codierbereich-Teilung, wobei das Verfahren weiter umfasst ein Minimieren der Kostenfunktion hinsichtlich der quantisierten Spektralkoeffizienten, der Huffman-Codebücher und der Huffman-Codierbereich-Teilung.
  15. Codierer zum Optimieren einer Audiocodierung einer Quellsequenz, wobei das Audiocodieren von Quantisierungsfaktoren abhängig ist, wobei die Quantisierungsfaktoren eine globale Quantisierungsschrittgröße und Skalierungsfaktoren umfassen, wobei der Codierer aufweist:
    eine Steuervorrichtung;
    einen Speicher, auf den die Steuervorrichtung zugreifen kann, wobei eine Kostenfunktion der Codierung der Quellsequenz in dem Speicher gespeichert ist, wobei die Kostenfunktion von den Quantisierungsfaktoren abhängig ist; und
    eine vorgegebene Schwelle der Kostenfunktion, die in dem Speicher gespeichert ist,
    wobei die Steuervorrichtung konfiguriert ist zum:
    Zugreifen auf die Kostenfunktion und die vorgegebene Schwelle von dem Speicher,
    Initialisieren von festen Werten der Skalierungsfaktoren, und
    Bestimmen von Werten der Quantisierungsfaktoren, die die Kostenfunktion minimieren, durch iteratives Durchführen:
    Bestimmen, für die festen Werte der Skalierungsfaktoren, eines Werts der globalen Quantisierungsschrittgröße, der die Kostenfunktion minimiert,
    Festlegen des bestimmten Werts der globalen Quantisierungsschrittgröße und Bestimmen von Werten von Skalierungsfaktoren, die die Kostenfunktion minimieren, und
    Festlegen der bestimmten Werte der Skalierungsfaktoren, und Bestimmen, ob die Kostenfunktion unter der vorgegebenen Schwelle ist, und wenn dem so ist, Beenden des iterativen Durchführens.
EP08170396A 2008-12-01 2008-12-01 Optimierung von MP3-Kodierung mit vollständiger Dekodiererkompatibilität Active EP2192577B1 (de)

Priority Applications (3)

Application Number Priority Date Filing Date Title
EP08170396A EP2192577B1 (de) 2008-12-01 2008-12-01 Optimierung von MP3-Kodierung mit vollständiger Dekodiererkompatibilität
AT08170396T ATE532175T1 (de) 2008-12-01 2008-12-01 Optimierung von mp3-kodierung mit vollständiger dekodiererkompatibilität
CA2686264A CA2686264C (en) 2008-12-01 2009-11-24 Optimization of mp3 encoding with complete decoder compatibility

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
EP08170396A EP2192577B1 (de) 2008-12-01 2008-12-01 Optimierung von MP3-Kodierung mit vollständiger Dekodiererkompatibilität

Publications (2)

Publication Number Publication Date
EP2192577A1 EP2192577A1 (de) 2010-06-02
EP2192577B1 true EP2192577B1 (de) 2011-11-02

Family

ID=40568705

Family Applications (1)

Application Number Title Priority Date Filing Date
EP08170396A Active EP2192577B1 (de) 2008-12-01 2008-12-01 Optimierung von MP3-Kodierung mit vollständiger Dekodiererkompatibilität

Country Status (3)

Country Link
EP (1) EP2192577B1 (de)
AT (1) ATE532175T1 (de)
CA (1) CA2686264C (de)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2013146895A1 (ja) * 2012-03-28 2013-10-03 日本電信電話株式会社 符号化方法、符号化装置、復号方法、復号装置、プログラム及び記録媒体

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
SG136836A1 (en) * 2006-04-28 2007-11-29 St Microelectronics Asia Adaptive rate control algorithm for low complexity aac encoding

Also Published As

Publication number Publication date
CA2686264C (en) 2015-01-27
ATE532175T1 (de) 2011-11-15
CA2686264A1 (en) 2010-06-01
EP2192577A1 (de) 2010-06-02

Similar Documents

Publication Publication Date Title
US8457957B2 (en) Optimization of MP3 audio encoding by scale factors and global quantization step size
KR100852481B1 (ko) 양자화 스텝 사이즈 결정 장치 및 방법
US6064954A (en) Digital audio signal coding
US7392195B2 (en) Lossless multi-channel audio codec
US20060074693A1 (en) Audio coding device with fast algorithm for determining quantization step sizes based on psycho-acoustic model
US8032371B2 (en) Determining scale factor values in encoding audio data with AAC
JP7280306B2 (ja) 改良されたミッド／サイド決定を持つ包括的なｉｌｄを持つｍｄｃｔ ｍ／ｓステレオのための装置および方法
RU2505921C2 (ru) Способ и устройство кодирования и декодирования аудиосигналов (варианты)
KR101243412B1 (ko) 무손실의 다채널 오디오 코덱
US20090164223A1 (en) Lossless multi-channel audio codec
US8380524B2 (en) Rate-distortion optimization for advanced audio coding
US8239210B2 (en) Lossless multi-channel audio codec
WO2010016270A1 (ja) 量子化装置、符号化装置、量子化方法及び符号化方法
CN101868822A (zh) 基于整数变换的编码和解码的取整噪声整形
WO2005027096A1 (en) Method and apparatus for encoding audio
US7650277B2 (en) System, method, and apparatus for fast quantization in perceptual audio coders
EP2192577B1 (de) Optimierung von MP3-Kodierung mit vollständiger Dekodiererkompatibilität
US7181079B2 (en) Time signal analysis and derivation of scale factors
US7676360B2 (en) Method for scale-factor estimation in an audio encoder
EP2346031B1 (de) Raten-Verzerrungsoptimierung für eine erweiterte Audiokodierung
JP4721355B2 (ja) 符号化データの符号化則変換方法および装置
Yen et al. A low-complexity MP3 algorithm that uses a new rate control and a fast dequantization
Sugiura et al. Shape Control of Discrete Generalized Gaussian Distributions for Frequency-Domain Audio Coding
Movassagh et al. Scalable audio coding using trellis-based optimized joint entropy coding and quantization
Bauer et al. Multidimensional optimization of MPEG-4 AAC encoding

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20081201

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MT NL NO PL PT RO SE SI SK TR

AX Request for extension of the european patent

Extension state: AL BA MK RS

AKX Designation fees paid

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MT NL NO PL PT RO SE SI SK TR

AXX Extension fees paid

Extension state: AL

Payment date: 20081201

Extension state: BA

Payment date: 20081201

Extension state: MK

Payment date: 20081201

Extension state: RS

Payment date: 20081201

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MT NL NO PL PT RO SE SI SK TR

AX Request for extension of the european patent

Extension state: AL BA MK RS

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602008011041

Country of ref document: DE

Effective date: 20120105

REG Reference to a national code

Ref country code: NL

Ref legal event code: VDEP

Effective date: 20111102

LTIE Lt: invalidation of european patent or patent extension

Effective date: 20111102

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120302

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120202

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20111102

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20111102

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20111102

Ref country code: BE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20111102

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120302

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20111102

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20111102

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20111102

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120203

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20111102

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20111102

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20111102

Ref country code: MC

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20111231

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20111102

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20111102

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20111102

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120202

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20111102

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20111102

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 532175

Country of ref document: AT

Kind code of ref document: T

Effective date: 20111102

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

26N No opposition filed

Effective date: 20120803

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20111201

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602008011041

Country of ref document: DE

Effective date: 20120803

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20111102

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20111102

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120213

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20111201

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20111102

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20111102

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20121231

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20121231

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20111102

REG Reference to a national code

Ref country code: DE

Ref legal event code: R082

Ref document number: 602008011041

Country of ref document: DE

Representative=s name: MERH-IP MATIAS ERNY REICHL HOFFMANN, DE

REG Reference to a national code

Ref country code: DE

Ref legal event code: R082

Ref document number: 602008011041

Country of ref document: DE

Representative=s name: MERH-IP MATIAS ERNY REICHL HOFFMANN, DE

Effective date: 20140926

Ref country code: DE

Ref legal event code: R081

Ref document number: 602008011041

Country of ref document: DE

Owner name: BLACKBERRY LIMITED, WATERLOO, CA

Free format text: FORMER OWNER: RESEARCH IN MOTION LIMITED, WATERLOO, ONTARIO, CA

Effective date: 20140926

Ref country code: DE

Ref legal event code: R082

Ref document number: 602008011041

Country of ref document: DE

Representative=s name: MERH-IP MATIAS ERNY REICHL HOFFMANN PATENTANWA, DE

Effective date: 20140926

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 8

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 9

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 10

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20231227

Year of fee payment: 16

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20231227

Year of fee payment: 16

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20231229

Year of fee payment: 16

REG Reference to a national code

Ref country code: DE

Ref legal event code: R082

Ref document number: 602008011041

Country of ref document: DE

Ref country code: DE

Ref legal event code: R081

Ref document number: 602008011041

Country of ref document: DE

Owner name: MALIKIE INNOVATIONS LTD., IE

Free format text: FORMER OWNER: BLACKBERRY LIMITED, WATERLOO, ONTARIO, CA