WO2002045307A1

WO2002045307A1 - Method and apparatus for progressive transmission of time based signals

Info

Publication number: WO2002045307A1
Application number: PCT/US2000/032628
Authority: WO
Inventors: Halldor Gudjonsson
Original assignee: Oz.Com
Priority date: 2000-11-28
Filing date: 2000-11-28
Publication date: 2002-06-06
Also published as: AU2001219367A1

Abstract

A method and apparatus are provided for progressively transmitting time-based signals across an electronic network (204). A first computer (202) uses an encoding software application (208) to analyze and mathematically model the signal (206). The signal data is parsed according to when the information is actually required at the receiver to re-generate the signal and key values are identified. The key value data is sorted into data packets for transmission in temporal order, according to when it is needed, rather than according to when the data actually occurs in the signal. Any space that is not filled in earlier data packets is filled with data from later time slices in the signal. This data is stored in a buffer until needed to re-generate the signal. As a result, the data packets according to the present invention each contain approximately the same amount of data, but represent differing time slices. Data packet storage capacity is optimized. A second computer (212) receives the data packets and re-generates the original signal using a decoding software application (210).

Description

METHOD AND APPARATUS FOR PROGRESSIVE TRANSMISSION OF TIME BASED

SIGNALS

FIELD OF THE INVENTION

The present invention relates generally to signal transmission and, more specifically, to a computer-implemented method for optimizing the transmission of an animation across an electronic network.

BACKGROUND OF THE INVENTION

Animations were being produced soon after the development of the movie camera. These early animations were stop-motion films of two or three dimensional representations, such as watercolors or clay figures. With advances in the fields of computing and electronic networks, it has now become possible to computer generate animations, and to electronically transmit animations over networks such as the Internet.

One common technique for electronically transmitting signal data such as animations is known as streaming. In data streaming, a signal is progressively transmitted through a fixed rata data channel in time slices, known as packets, so that the signal can be regenerated on the receiving end during transmission. A fixed rate data channel can be characterized by a steady stream of data packets of uniform size.

To stream a signal, the signal must first be parsed into packets for transmission. One typical prior approach for parsing the signal is to sample the signal values over time at a fixed rate. A fixed time slice of the resulting information can then be packaged into a data packet. Compression methods can then be used to decrease the size of each packet before transmission. This produces data packets of fixed time length but, usually, variable size. During periods of low signal activity, higher compression can be achieved per time slice than during periods of high signal activity. Data packets describing periods of low signal activity can therefore be compressed to smaller sizes than other packets. This prior art method is subject to several limitations. First, sampling the signal values and transmitting the samples results in loss of some signal data. The sampling rate imposes a fixed frame rate on the signal curves and the receiver is not provided with precise information as to the behavior of the curves at the times between the samples. As a result of this limitation, the sampling rate is frequently used as a driving rate for the regeneration of the signal curve. Therefore, the receiver never receives the signal at a higher frame rate than the sampling rate.

Another limitation of the prior art method is that fixing the packet time span while varying the packet size is an inefficient use of bandwidth. Some packets may contain no data, while excess data may be truncated from other packets. This may result in significant signal degradation. Furthermore, the amount of time required to stream the entire signal can be significantly increased if many packets contain little or no data.

Figure 5a is a diagram of signal data packets according to the prior art. Each data packet 502, 504, 506 represents the same time slice, in this example, two seconds. However, the amount of data stored in each of these packets is different. For example, the first packet 502 is filled with data while the second packet 504 is only partially filled. The third packet 508 originally included more data than could be accommodated by the transmission bandwidth. As a result, the excess data 514 was truncated and not transmitted.

It would be an advantage to provide a method and apparatus for streaming signals that minimizes the loss of data resulting from bandwidth restrictions. It would be a further advantage if such method and apparatus maintained the resolution provided by the parametric representation of signal data. It would be yet another advantage if such method and apparatus were available to optimize the amount of data transmitted in streamed data packets.

SUMMARY AND OBJECTS OF THE INVENTION

The present invention is a computer-implemented method and apparatus for progressively transmitting time-based signals across an electronic network. In the invention, data signal parameters are analyzed and the signal is encoded and transmitted such that data packet storage capacity is optimized. The invention can be used for purposes including but not limited to transmitting animation, audio, and video signals across an electronic network and reconstructing the signal or signals at a receiver. The preferred embodiment of the present invention is directed to the streaming of animation signals. In the apparatus according to the present invention, a first computer is used to parse the signal into transmittable data packets using the method according to the present invention. In the presently preferred embodiment, an encoding software application is used to mathematically model the signal, to identify key values, and to sort the key data into data packets for transmission across the electronic network. A second computer receives the data packets and re-generates the original signal using a decoding software application.

The present invention can be used with signals that are reasonably well-behaved such that a mathematical model can be derived to approximate the signal to within an acceptable level of accuracy. This model must be localized such that relatively few points are needed to effectively use the model. In the invention, the amount of data in each transmitted data packet is optimized.

Data is not transmitted sequentially according to consecutive time slices. Rather, the signal data is parsed according to when the information is actually required at the receiver to regenerate the signal. This data is then transmitted in temporal order, according to when it is needed, rather than according to when the data actually occurs in the signal. Any space that is not filled in earlier data packets is filled with data from later time slices in the signal. This data is stored in a buffer until needed to re-generate the signal. As a result, the data packets according to the present invention each contain approximately the same amount of data, but represent differing time slices. BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is a block diagram of a computer network according to one embodiment of the present invention. Figure 2 is a block diagram illustrating an apparatus for accessing an electronic network, according one embodiment of the present invention.

Figure 3 is a graph of a general signal over time.

Figure 4 is a graph of an audio signal over time.

Figure 5 a is a diagram of signal data packets according to the prior art. Figure 5b is a diagram of signal data packets according to the present invention.

Figure 6 is a graph of the function/ft = rt.

Figure 7 is a graph illustrating the timing curve of Equation (5).

Figure 8 is a schematic depiction illustrating the progressive transmission of N signals from server to client according to the present invention. Figures 9a and b are bar graphs illustrating the bandwidth requirements for a packet stream according to Equation (23).

Figure 10 is a flow chart of the method according to a preferred embodiment of the present invention.

Figures 1 la-c are graphs of the timing curves in Equations (26) through (28). Figure l id depicts an example of the variance over time of an animated signal when different timing curves are combined into a piecewise curve.

Figure 12 is a series of graphs of concurrent real-time signals according to the present invention.

Figures 13 through 14 are graphs illustrating the encoding and temporal re-ordering of the real-time signals of Figure 12. DETAILED DESCRIPTION

A method and apparatus for progressively transmitting time-based signals is described. In the invention, data signal parameters are analyzed and the signal is encoded and transmitted such that data packet storage capacity is optimized.

In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be evident, however, to one skilled in the art that the present invention may be practiced without the specific details. In other instances, well-known structures and devices are shown in block diagram form to facilitate explanation. The description of preferred embodiments is not intended to limit the scope of the claims appended hereto.

The present invention can be used for purposes including but not limited to transmitting animation, audio, and video signals across an electronic network and reconstructing the signal or signals at a receiver. In the preferred embodiment, the present invention is implemented using a computer. Such computer can include but is not limited to a personal computer, network computer, network server computer, dummy terminal, local area network, wide area network, personal digital assistant, work station, minicomputer, and mainframe computer. One such computer is operable to parse the signal into transmittable data packets according to the present invention, and another such computer that receives the data packets is used to re-generate the original signal. In addition, the present invention can be used with any other hardware components that facilitate the creation, transmission, and regeneration of the signal. Such hardware components include but are not limited to video capture boards, sound cards, rendering devices, sound recording equipment, film editing equipment, video cameras, and microphones. The signal identification, parsing, sorting, transmission, buffering, and/or regeneration features of the present invention can be implemented as one or more software applications, software modules, firmware such as a programmable ROM or EEPROM, hardware such as an application-specific integrated circuit ("ASIC"), or any combination of the above. In addition, the present invention can be used with any appropriate third-party software applications, for example, animation rendering applications, music composition applications, sound mixing applications, and graphics applications.

Figure 1 is a block diagram of a computer network system 100 according to one embodiment of the present invention. In computer network system 100, a network server computer 104 is connected to a network client computer 102 through a network 110. The network interface between server computer 104 and client computer 102 can also include one or more routers, such as routers 106 and 108. The routers serve to buffer and route the data transmitted between the server and client computers. Network 110 may be the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), or any combination thereof. It should be noted that, although only one server and client computer each are illustrated in network system 100, a network that implements embodiments of the present invention may include a large number of interconnected client and server computers.

For example, one or more encoding software applications accessible to a transmitting network client computer can be used to mathematically model a signal and to identify and sort the key values of the mathematical model into data packets for transmission. Decoding software applications can also be used to re-generate the original signal at a receiving network client computer. Any or all of the software applications or hardware configurations of the present invention can be implemented by one skilled in the art using well known programming techniques and hardware components. The entire process and apparatus or any portion thereof can reside with one or more users or third parties. The present invention can be implemented as one or more software applications, software modules, plug-ins, firmware, and hardware that are provided to individual users for their utilization in transmitting or receiving time-based signals. For example, in one embodiment of the present invention, the encoding and decoding functions are performed by the same software application. However, in alternative embodiments, the encoding and decoding functions are performed by different applications.

The original signal data source for the original signal can be in any appropriate form, including but not limited to processed film (black-and-white, color, or negatives), video, CD, CD-ROM, photographs, optical disks, audio tape, phonographic recordings, Digital Video Disks, and computer images. Computer image data sources can be stored in any format including but not limited to MP3, and QuickTime animations.

The present invention is used to transmit signals across an electronic network, such as the Internet or a local area network ("LAN"). In the preferred embodiment of the invention, the transmitting and receiving computers are adapted for communication with the Internet. The signal is transmitted through any appropriate communication device or connection, including but not limited to modem, satellite transmission, cable transmission, Ethernet, ADSL, ISDN, or TI.

Figure 2 is a block diagram illustrating an apparatus 200 for accessing an electronic network, according to one embodiment of the present invention. In this embodiment, a computer 202 is adapted for communication with an electronic network 204 such as the Internet. Animation signal 206 can be stored on the computer, for example, in a database. An encoding software application according to the present invention 208, also resident on the computer, can be used for example, to mathematically model the invention, to identify key values, and to sort the key data into data packets for transmission across the electronic network. The re-generation of the received signal or signals can then be performed using a decoding software application 210 resident on the receiving computer 212.

The encoding and decoding software applications must be compatible with one another. As an example, there must be an agreement between these applications as to the types of parametric curves that are to be used in the streaming session. Generally, for animation streaming, the types of parametric curves are incorporated as a part of the software. For example, information that curve 1 is a constant curve while curve 2 is linear can be hardcoded into the encoder application that generates the streaming data and the decoder application that decodes the key values and timing curves into signal data. Both applications will therefore be configured to use the same types of parametric curves.

In an alternative embodiment, the transmitting computer and the receiving computer can initiate a "handshake" session to mutually establish an agreement as to the types of timing curves to be used in a streaming session. In yet another embodiment, the transmitting computer provides instructional information to the receiving computer so that the receiving computer can interpret different timing curves. The embodiment is advantageous because it is dynamic and provides flexibility.

While the preferred embodiment of the invention is directed to streaming a signal, such as an animation, in an alternative embodiment, the transmitted signal is stored for re- generation at a later time. The signal data can be stored on at least one database that is on the same computer as the signal re-generation software application, or on any storage device or other computer(s) accessible thereto. This embodiment is advantageous when it is desirable to optimize the data transmission process but there is no need for immediate re-generation of the transmitted signal. For example, this alternative embodiment can be used to reduce expenses when the cost of transmission is related to the length of the transmission across the electronic network.

The present invention can also be used with conversion methods that transform irregular signals that are not readily mathematically modeled into more regular signals. For example, when there is an error tolerance level for a re-generated signal such that an approximation of the original signal can be transmitted, the present invention can be adapted to approximate the signal and then transmit it according to the teachings herein.

A signal can be represented as a series of curves that describe the change of signal values over the time period of the signal. In the broad sense, a curve is the locus of a point moving in space with one degree of freedom. A common method for describing curves mathematically is by representing them with parametric equations. Parametric equations express the points on a curve explicitly in terms of other variables or parameters that are, in this context, regarded as independent variables.

Figure 3 is a graph of a general signal over time. The signal S can be any type of signal such as a sound, a video signal, and an animation. The term signal can refer to any particular quantity that varies with time. The graph of Figure 3 represents a model of 3,000 seconds of a signal. The present invention can be used with signals that are reasonably well- behaved such that a mathematical model can be derived to approximate the signal to within an acceptable level of accuracy. This model must be localized such that relatively few points are needed to effectively use the model. By contrast, in a non-localized signal, all of the points for the entire signal might be required to determine the signal parameters at, for example, point TI of Figure 3.

Figure 4 is a graph of an audio signal 400 over time. In the prior art, there is no pattern analysis of this signal. Rather, the signal is sampled at various points. However, it can be extremely difficult to sample a complex signal and might require the sampling of a very large number of points to achieve a reasonable level of accuracy. It is not practical or even possible to sample certain signals with any accuracy. By contrast, the present invention uses a mathematical model 420 of the audio signal with a set of key values 402, 404, 406,

408, 410, 412, 414 to represent the audio signal. The preferred embodiment of the present invention is directed to the streaming of animations. One way to transmit an animation over an electronic network is as a signal.

Animation signals are well-adapted to being represented by a mathematical model. The teachings of the invention can also be applied to any other type of signal that is capable of being represented by a mathematical model that is well localized. The invention can also be adapted to permit the streaming of signals that are not well-represented by a mathematical model, such as a high frequency signal. However, in this case, more data would be required to be processed to properly stream the signal.

In the prior art, a signal is sequentially parsed into data packets of fixed time slice.

Each data packet is then transmitted over an electronic network, regardless of how much data is actually included or excluded from a packet at a particular time slice. Thus, it is possible to send an empty packet, or to lose information that could not fit into the packet for the particular time slice.

By contrast, in the invention, the amount of data in each packet is optimized. Data is not transmitted sequentially according to consecutive time slices. Rather, the signal data is parsed according to when the information is actually required at the receiver to re-generate the signal. This data is then transmitted in temporal order, according to when it is needed, rather than according to when the data actually occurs in the signal. Any space that is not filled in earlier packets is filled with data from later time slices in the signal. This data is stored in a buffer until needed to re-generate the signal. Figure 5b is a diagram of signal data packets according to the present invention. Each data packet 508, 510, 512 is filled with data, unlike the prior art data packets 502, 504, 506 of Figure 5 a. In addition, the data packets according to the present invention each contain approximately the same amount of data, but represent differing time slices. Timing curves can be used to describe the change of a value or property over time. Thus, timing curves describe signals over time. Where time is the key parameter, it can be desirable to represent a timing curve by a parametric equation. For example, a timing curve, f, can be represented by an equation of the form:

f=f(t) (1)

where t = time.

For example, iff describes a signal that begins at zero but grows at a steady rate of r per second, then:

f(t) = rt. (2)

Figure 6 is a graph 600 of the function f(t) = rt. If the rate r of Equation (2) is a variable, then parametrization yields the equation:

f(r,t) = rt. (3)

Equation (3) is a parametric representation of the family of curves that begin at value zero and grow at some steady rate over a period of time. It is important to distinguish between the two parameters of time t and rate r. The rate r is a fixed attribute of a particular signal. The time t is a continuous variable that represents the time variations of the signal.

Equation (3) can be generalized to represent the family of linear timing curves as follows: f(a,b,t) = at + b. (4)

where f(a,b,t) = the function representing the linear timing curve with parameters a,b, at time t.

Equation (4) describes a signal that starts at a fixed value b and then changes at a steady rate a. However, Equation (4) does not represent a signal that does not change at a steady rate. Therefore, it is desirable to divide the time span of a signal into smaller intervals, each of which can have a different fixed value and can change at a different steady rate. Therefore:

( fi*ι, t) i£ tι < t < t₂ f(aι, h, t) i£ t₂ < t < t ^': (5)

where s(t) = the time-variable signal; b„ = the fixed value for interval n; a_n = the rate of change for interval n; and t„ = the time at the beginning of interval n.

Figure 7 illustrates the timing curve 700 described by Equation (5). As illustrated by the Figure, the formulation described by Equation (5) does not require the signal to change continuously with time. Figure 7 is a curve represented by individual line segments 702, 704, 706. It is possible to select individual values, or points on the curve of Figure 7 that, combined with Equation (4) can be used to describe a signal over time. These characteristic parameters of the signal will be referred to for the purposes of this discussion as key values. For example, the values a_, b_, and t_ are key values of the signal s(t) of Equation (5).

Equation (5) can also be written in a different, more compact notation. The linear timing curve can be defined as follows:

g(a, b_. tι, t2. t) f{a_ b, t) if i_L < t < t₂ ^{" ' ""}' -{ 1 00 .. i j££ tt << ttι aτ t ≥ t<_. (6) where g(a,b,t„t₂,t) = the function representing the linear timing curve at values a,b,t„t₂, and t.

On the interval [t,, t₂) the curve represented by Equation (6) is the same as that represented by Equation (4). Outside of the interval [t„ t₂), the value of the equation is zero. Equation (5) can therefore be re-written as:

=

₊3, £3i+6, i). (⁸)

where k = key value.

It is required that the key values of Equation (8) be ordered such that:

{***«, A_3i+2, i₊₃} - {α,.₊₁, _{bi+lt t}._{+l }} (9)

The entire set of key values K for a signal is sufficient to describe a signal completely for purposes of this discussion.

While the present invention is described herein with respect to linear timing curves, the invention is not in any way restricted to this class of curves. The method and apparatus taught herein can also readily be applied to any types of curves, or combinations thereof, that represent a signal where each curve has a limited contribution time to the signal value. For example, a combination of cubic splines, time harmonics, and constant curves can be used to describe a signal for purposes of the invention if these types of parametric curves have a limited time range and relatively few associated parameters. Therefore, the discussion of linear time curves is for descriptive purposes only and is not in any way intended to limit the scope and spirit of the present invention. Using the present invention, data from multiple parametric timing curves can be processed and transmitted as streamed data in a fixed rate data channel. Streaming is the process of progressively sending data in packets that are re-generated at a receiver during transmission. A data packet is a portion of a signal that is separated for transmission across an electronic network according to a selected time interval. A fixed rate data channel can be characterized by a steady stream of data packets of uniform size. In the invention, the key values in the set K are transmitted across the electronic network in data packets through such fixed rate channel.

A set S of signals described by parametric timing curves can be described by the notation:

S = {S_1>S2,...,_SN} (10)

where s, describes the time dependence of the signal.

Figure 8 is a schematic depiction of the signals in set S. The Figure illustrates the progressive transmission of N signals 802, 804, 806 from the server 810 to the client 820 across an electronic network 814. The signals are transmitted from the server as packet data 812 that is received 822 at the client. No correlation is assumed between the signals in S. For purposes of this description, these signals can be independent or interdependent. In

Figure 8, the time t_c is the current location of signal re-generation on the client side. The time t_s is the time length of server transmitted signal data. As has been discussed previously, each signal s_ can be completely described by a finite set of key values

^ir- ^e {* .*W..-. , J^n-,}. ^' (11)

where n_ - the number of key values for signal s,.

The number of key values is allowed to vary among signals. This number is specific to each signal. In general, signals exhibiting regularity or little variation require fewer key values than irregular signals. Because each signal is completely described by its key values, it is possible to define a general mapping, s(K_,t). Using this mapping, it is possible to derive the signal value at any time t given the key value set for a signal. Thus:

This mapping has strong locality, therefore, the signal value at each time can be deduced from only a few key values. The efficiency of the method increases with the strength of the locality. The locality requirement permits the definition of a subset of the key values, K¹ (f) that holds the smallest set sufficient to generate the signal value for s_ at time t. Thus:

*(K_it t) =* _a(K^<(t), t).

(13)

Two additional mappings, T_min(k_^ and T_max(k_iJ) can be introduced to properly define K (t). The mapping T_min(k^ gives the smallest time value at which a key value contributes to the signal value. The mapping T_max(k, gives- the largest time value at which a key value contributes to the signal value. These mappings essentially produce the time span over which a key k contributes to a signal s_t. The key values in the reduced set can therefore be described as:

Λjrø = {k Ki I l rø < t < T^ik)} . (14)

The present invention can be used to stream the key values in K by transmitting them in data packets through a fixed rate channel. As distinguished from the prior art approach of sampling the signals and transmitting the samples in fixed time slice data packets of varying size, in the invention, key values are transmitted in data packets of varying time slice and filled to approximately the same size.

The present invention uses a stream header to transmit certain information to the receiver. A stream header is a packet block that is sent before a streaming session is initiated. The stream header includes information that the decoder will need to re-generate the animation signal. This information can include but is not limited to the time length of the animation and mapping parameters. The stream header information is used by the decoder to determine the signal output. In the example of animation, the stream header information contains binding information regarding exactly what the signal animates. Such mapping parameters can include, for example, the information that a first signal is the height of a first element (property) in the animation and that a second signal is the color of a second element (property).

The efficiency of the present invention is increased by determining the temporal order of the key values. For the purposes of this disclosure, temporal order is defined as the order in which the key values should be sent to maximize the time length of valid signal data at the receiver. The union K of all key values can be represented as:

K = K_l \ K₂ U ... U K_N (15)

The elements in this set are ordered by requiring that

WW ≤r._fcft^. (16)

where k_m is the m-th element in K.

The order shown by Equation (16) is the temporal order of the key values. As has been previously discussed, T_mi„(Ki_j) is the time at which key / first contributes to the value of signal s. To properly re-generate this signal, the key y^' is needed at and after that time. Thus, the first key that should be transmitted is the key among all the signals that has the lowest T_min, i.e. the first key in K. This is because the time length of valid signal data at the receiver is only as long as the shortest valid signal. The order of T_min values can be determined using, for example, the Quick Sort function of the C programming language.

Once all keys from k, to k_m have been sent from K, it must then be determined which key should be transmitted next. This is the key that has the lowest T_min from its set of non- transmitted key values. According to the definition in Equation_(16) this is T_mi„(k_m+!). The present invention permits the transmission of the key values in optimal order. There is no fixed time slice associated with each packet of data.

Because the decoder software application at the receiving computer is programmed with the types of parametric timing curves, in other words the equations, that will be utilized by the encoder software application, the only information that the encoder requires to generate the signals according to the present invention is the key values. Each key value has a particular associated time. Therefore, even if packets are received in a non-sequential order, the decoder will be able to re-generate the signal from the timing information and the key values.

The timing information for a key value is transmitted to the receiving computer in the data packet containing that key value. For example, if a data packet contains ten different key values, the packet will also contain the timing information for each of the ten key values.

This is in contrast to the prior art method of time-stamping each packet. When multiple signals are being transmitted according to the present invention, the data packets also include information that identifies the particular signal to which each key value belongs.

Because the timing information is transmitted within the individual data packets, an effective data compression procedure is preferably used to compress the timing information.

Such compression is facilitated by the temporal ordering of key values, because adjacent key values have the smallest time difference possible. The timing information can therefore be encoded as time differences within the respective packets.

The time span of a packet is defined as the time interval that the packet data describes completely. It is assumed herein that all previous packets have been received. The time span of packet/? is therefore:

where k'_ denotes the first key in packet number i.

A common procedure used to facilitate streaming methods is buffering. Buffering is the act of transmitting signal data to the receiver before any signal re-generation is performed at the receiver. This signal data is temporarily stored or buffered in one or more buffers until it is required for the re-generation procedure. Buffering can be advantageous because it gives the transmitter a lead time over the receiver. This makes the streaming more tolerant of bandwidth fluctuations. Thus, if the channel data rate temporarily drops during the streaming process, the receiver will be unaffected for the duration of the buffered signal data. If the data rate accelerates before the buffer is cleared, then the receiver will not experience any signal loss.

Additionally, buffering can be used to lower the bandwidth required for the streaming process. This can be particularly effective for signal sets that require a large number of key values at the beginning, i.e. when E„,,„ approaches zero for a large number of keys in K.

The lead time provided to the transmitter is determined by the equation:

I

^buffer = Δ i = i fcf₊i). (18)

where / is the number of packets that are buffered before streaming commences; and where T_buffer (830) = the lead time.

All signals start at time zero. As a result, the first key in K must therefore have a T ______ equal to zero. Temporal ordering ensures that this is the first key in the first packet, i.e. T_min (k'_j) = 0. Equation (18) makes use of the fact that, as has been shown in Equation (17), the sum is telescoping. Equation (18) can also be viewed as an expression determining / for a given T_bφr Therefore, the equation determines the number of packets required to furnish a specified buffer time.

The bandwidth consumed by a stream of uniform size packets with variable time span must be estimated. Assuming that / packets have been buffered, there are T_buffer seconds of signal data at the receiver. At the same time as packet number / + 1 is received from the transmitter, signal generation commences at the receiving end. If the transmission of the next packet takes τ seconds, then the time span of valid data at the receiver reaches a minimum of

just before the new packet is received. To ensure that the time length of buffered data, T_receiver, never goes below T_buffer, then the number of seconds τ that it takes to transmit the next packet should be selected to be:

5

(20)

This requires a corresponding bandwidth of:

V

10 ^r Δr,_+l (21)

where is the uniform _.packet size, usually specified in bits; T_ are the time slices, usually .expressed in seconds; and r is the bandwidth (rate), usually expressed in bits per second. L5

Equation (21) expresses the bandwidth required by the first progressively transmitted packet. In the Example, this is packet number 1 + 2. Equation (21) can be used to determine a good location in the packet sequence to stop buffering and to start streaming. The number of buffered packets / is increased until the equation produces a reasonable bandwidth 20 requirement. This determination is especially useful if a specific buffer time is not mandated. To find the bandwidth required for a streaming session of n packets, Equation (21) must be maximized over the entire set of progressively transmitted packets:

According to Equation (17) the sum in the denominator is telescoping and thus:

The properties of Equation (23) are illustrated by Figure 9. Figures 9a and b are bar graphs illustrating the bandwidth requirements for a packet stream according to Equation (23). In the example shown in Figure 9a, the bandwidth, R, as determined by Equation (23) for packets 3 and 4 is sufficient to successfully stream packets 5 and 6 as well as parts of packets 7 and 8.

To lower the consumed bandwidth, a more aggressive approach can be adopted. Equation (23) is derived with the condition that at least a buffer of time length T_bφr is maintained. In an alternative, more aggressive embodiment of the present invention, this restriction can be loosened such that it is required only that the receiver never run out of valid signal data. As a result, in this embodiment, streaming can begin with packet number / + 1, instead of / + 2. The bandwidth required for this packet is:

^{* •}'-5GT ⁽²⁴⁾ A streaming session of n packets, where n - 1 are progressively transmitted, will require:

Equations (17) and (18) have both been used in the derivation of Equation (25).

Figure 10 is a flow chart of the method according to a preferred embodiment of the present invention. To convert signal data into a streamable file of key values, a new and empty streaming file must first be opened 1000 and signal data acquired 1005. A determination must then be made as to whether the signal data is in parametric timing curve format 1010. If not, the signals are converted to parametric format 1015.

Redundant key values are filtered out from the parametric signal data 1020. Curve fitting over two or more time intervals can be used to reduce the number of key values required to represent the signal data 1025. A user-specified error margin can be used to determine whether the curve fitting is within acceptable parameters. All key values are subsequently merged into one list 1030. The temporal order of the key values of the list are determination must then be made as to whether the signal data is in parametric timing curve format 1010. If not, the signals are converted to parametric format 1015.

Redundant key values are filtered out from the parametric signal data 1020. Curve fitting over two or more time intervals can be used to reduce the number of key values required to represent the signal data 1025. A user-specified error margin can be used to determine whether the curve fitting is within acceptable parameters. All key values are subsequently merged into one list 1030. The temporal order of the key values of the list are determined and the list is sorted accordingly 1035. If the key list is empty 1040, the packet time length table and Equations (23) and (25) are used to calculate the required streaming bandwidth 1045. If the bandwidth requirements are acceptable 1050, the streaming file is closed 1055. If the bandwidth requirements are not acceptable, the contents of the streaming file are erased 1065. New user-defined values for curve-fitting and quantization error are selected 1060 and the curve fitting step 1025 is once again performed.

If the key list is not empty, a new empty data packet of fixed size is created 1070. The first key is removed from the temporal order list and encoded into this empty data packet within a user-specified quantization error. A determination is made as to whether the data packet is now full 1080. If not, step 1075 is repeated with the next key in temporal order from the list. If the data packet is full, the time length of the valid curve data is stored in the packet in a table 1085. The packet is then written to the end of the streaming file 1090. As an example of present invention, the method disclosed herein will be applied to the streaming of three-dimensional ("3D") animation. Motion and any other temporal changes in a computer generated 3D environment are generally grouped under the term "3D animation" This type of animation is most commonly acquired by three means:

1. Digital recording of motion through use of a motion capture device;

2. Digital recording of animation through use of a connected puppeteering device; and

3. Direct authoring of computer animation by a human operator. Thus, for the purposes of this example, it is possible to use any of the three different types of curves: constant, linear, and cubic spline (also known as "Hermite spline"). The Hermite spline representation will be discussed herein because it has widespread use in the animation industry and is readily subject to parametrization. The three basic timing curves, constant, linear, and Hermite, are defined as follows, according to the previous discussion of the invention presented herein:

gι(P„Ltt * h *_t rt) = ^s J J Λ₀ * ^f*l≤*<*2 ^{a (26)}

₉₂(_Pi,_P2.h.t₂.t) = fcΛ-0+»{*r*ϋ)/« -eϋ **ι<*<fc (27)

[_ « else, z(_{P P}z.J__.d_.t _t) = {^ft«*)+aArø +*««+**»(*) «***<* ₍₂₈₎

The following Hermite basis functions have been introduced into Equation (28):

bo(t) - 2«t³(t) - Zu*{t) + 1 (29)

6ι(i) * -2³(t)+3²(i) (30)

bι(t) = («*(*) -2B»(*) +«(*)) (tβ-tϋ (31)

63(f) = (τx³(i}~ tfitήitz- ) (32)

where _u(_t\ _ *-*t (33)

and where b₂ and b₅are the basis functions for the curve tangents, d, and d₂ in Equation (28).

Figures 1 la-c are graphs of the timing curves in Equations (26) through (28). Figure lid depicts an example of the variance over time of an animated signal when different timing curves are combined into a piecewise curve. This is similar to the signal illustrated in Figure 7. However, in this example, curves of different degree can be used on each interval. Thus, the signal shape is potentially more varied that that shown in Figure 7.

In addition, the signal shown in Figure 11 changes continuously with time with the exception of the endpoints of constant curves g,. This is a common practice in animation authoring tools because animated values usually change continuously with time and any desired discontinuities can be introduced by using constant curves. This property allows adjacent signal intervals to share parameter values. For example, if a linear segment g₂ connects continuously with a Hermite segment g₃, the parameters^ and t₂ for the linear segment must be equal top, and t, for the Hermite segment. This property becomes evident upon examination of Equations (27) and (28). Thus, the key value vector k can be defined as follows:

where: k = (j, p, d^', d ⁺, f) (34)

j e {1,2,3} denotes the timing curve type; p is the value of the curve at time r; d ^" is the left tangent of the curve at time τ; and d ⁺ is the right tangent of the curve at time τ.

Tangents are only needed for parametrization of Hermite splines. Therefore, specifying tangents for other curve types is redundant. However, such redundancy is warranted in this example because it facilitates the unification of the notation and the simplification of the related algorithms. This redundant information is eliminated in the pre- process steps for progressive transmission.

For the purposes of this example, 1 and m will represent two successive key values. Therefore, it is required that the time rof 1 is less than that of m. The notation k_ will be used to denote the i-th component of vector k. The two key values are then successive if:

l_s < m₅ . (35) To simplify the notation, the general parametric timing curve is defined as:

Iteih, _a, k. m_5t i) if ϋ - 2 (36)

9a(h, m_2t ht n»_3l Z_β, m&. t) if ZL — ³ where the functions g, are defined in Equations (26) through (28). An animation signal can therefore be written as:

where K = {k„ ..., k„} is the set of key values for the signal.

The sharing of parameter values is demonstrated by the formulation of Equation (37). Curves on adjacent time intervals, for example,/ andy + 1, both require the key value k_j+1. The temporal order of the key values must also be determined. This is achieved using the operator T_min. T_mi„ gives the smallest time value at which a key contributes to a signal value. By examination of Equations (26) through (28) and (36) through (37), the following can be deduced:

A key value is needed at the time of the previous key ,.._; if the timing curve between the two is linear or Hermite. Both these curves interpolate between the points of two key values. Both keys are therefore required to generate a value on the curve. It is only in the case of the non-interpolating curve, g„ that the second key isn't needed.

Although the present invention has been described with reference to specific exemplary embodiments, it will be evident that various modifications and changes may be made to these embodiments without departing from the broader spirit and scope of the invention as set forth in the claims. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

For example, the preferred embodiment of the present invention is adapted for use with any 3D rendering engine for visualization, including but not limited to Microsoft's Direct3D and Criterion's RenderWare. The present invention can be implemented with any computer operating system including but not limited to Microsoft Windows, UNIX, LINUX, Apple's MacOS. It is also compatible with any networking applications including but not limited to those of Sybase, Novell, Apple, and Microsoft.

The present invention can be integrated with computer games or other applications through SDK. In addition, the application can be used as a complement to, or as a module for integration with other applications requiring 3D visualization. The preferred embodiment of the invention is compatible with high-end authoring tools such as Softimage 3D, Maya, and 3D Studio Max. The present invention can be used with any appropriate streaming protocol, including but not limited to RealNetwork's RTSP. The present invention can be applied to any type of signal transmitted over an electronic, satellite, or other type of data broadcast network. Such signals can also include analog, digital, and power signals. The method and apparatus described herein can readily be used to stream either compressed or uncompressed signals. Other streaming methods can also be used in conjunction with the present invention. The teachings of the present invention can be applied to the transmission of a plurality of concurrent real-time signals. Figure 12 is a series of graphs of concurrent real-time signals according to the present invention. One of skill in the art will readily recognize that the key values of signals s,(t) s₂(t) and s₃(t) can be determined, for example, using the process described herein with respect to Figure 10. Figures 13 and 14 are graphs illustrating the encoding and temporal re-ordering of the real-time signals of Figure 12. As shown in Figure 13, the bandwidth required to transmit the concurrent real-time signals of Figure 12 might, at times, exceed the available bandwidth for transmission and therefore not be transmitted. By encoding and temporally re-ordering the signal data according to the present invention, however, it is possible to transmit the signal data for the concurrent signals while minimizing, if not eliminating entirely, signal loss and degradation due to inadequate bandwidth.

Claims

CLAIMSWhat is claimed is:

1. A method for progressive transmission of time based signals through an electronic 5 network, comprising the steps of: selecting a set of key values that can be used to describe at least one data signal over a period of time; associating timing information with each key value; streaming the key values in the set by transmitting them through a fixed rate channel 10 in at least one data packet; and transmitting the timing information for a key value in the data packet containing the key value.

2. The method of claim 1 further comprising the step of temporally ordering the key values. L5

3. The method of claim 2, wherein the step of temporally ordering the key values comprising the steps of: determining the smallest time value at which a key value contributes to signal value; determining the largest time value at which a key value contributes to signal value; >0 and determining the time span over which a key value contributes to the at least one signal.

4. The method of claim 3 further comprising the step of streaming the key values in temporal £5 order.

5. The method of claim 1, further comprising the steps of: defining a subset of the key values that holds the smallest set sufficient to generate the 10 signal value during the time period; and streaming the key values in the subset.

6. The method of claim 5, wherein the step of defining a subset of the key values further comprises the steps of:

5 determining the smallest time value at which a key value contributes to the curve value; and determining the largest time value at which a key contributes to the signal value.

7. The method of claim 1 further comprising the step of regenerating the at least one signal [-0 from the streamed key values and associated timing information.

8. The method of claim 1 further comprising the step of using a stream header to transmit decoding information to a receiver of the streamed key values.

L5 9. The method of claim 4 wherein the decoding information comprises the time length of the signal and mapping parameters.

10. The method of claim 1 further comprising the steps of: transmitting key values and associated timing information for a plurality of signals; >0 and transmitting information identifying the particular signal to which a key value belongs in the data packet containing the key value.

11. The method of claim 1 further comprising the step of using a data compression ^>5 procedure to compress the timing information.

12. The method according to claim 1 further comprising the step of buffering transmitted signal data at the receiver.

13. A method for converting signal data into a streamable file of key values, comprising the steps of: opening a new and empty streaming file; acquiring signal data in the streaming file; determining whether the signal data is in parametric timing curve format; for signal data in parametric timing curve format, filtering out redundant key values from the parametric signal data; using curve fitting to reduce the number of key values required to represent the signal data; merging all key values into one list; determining the temporal order of the key values of the list; sorting the list according to temporal order; and determining whether the key list is empty.

14. The method of claim 13, further comprising the steps of: if the key list is empty, calculating the required streaming bandwidth; closing the streaming file if the bandwidth requirements are acceptable; if the bandwidth requirements are not acceptable, performing the steps of: erasing the contents of the streaming file; selecting new user-defined values for curve-fitting and quantization error; and using curve fitting to reduce the number of key values required to represent the signal data.

15. The method of claim 13, further comprising the steps of: if the key list is not empty, creating a new empty data packet of fixed size; removing the first key from the temporal order list and encoding the first key into the new empty data packet within a user-specified quantization error; determining whether the new data packet is full; if the data packet is not full, removing the next key in temporal order from the list and encoding the nest key into the new data packet within a user-specified quantization error; if the data packet is full, storing the time length of the valid curve data in the packet in a table; and writing the packet to the end of the streaming file.

16. The method of claim 15 further comprising the step of using a user-specified error margin to determine whether the curve fitting is within acceptable parameters.

17. The method of claim 13, further comprising the step of converting non-parametric format signals to parametric timing curve format.

18. A method for streaming a data signal over an electronic network, comprising the steps of: selecting a set of key values that can be used to describe at least one data signal over a period of time; associating timing information with each key value; using a data compression procedure to compress the timing information; temporally ordering the key values by determining the smallest time value at which a key value contributes to signal value, determining the largest time value at which a key value contributes to signal value, and determining the time span over which a key value contributes to the at least one signal; streaming the key values in the set in temporal order by transmitting them through a fixed rate channel in at least one data packet; transmitting the timing information for a key value in the data packet containing the key value; and transmitting decoding information to a receiver of the streamed key values using a stream header.

19. The method of claim 18, further comprising the steps of: defining a subset of the key values that holds the smallest set sufficient to generate the signal value during the time period; and streaming the key values in the subset.

5 20. The method of claim 18 further comprising the step of regenerating the at least one signal from the streamed key values and associated timing information.

21. The method of claim 18 further comprising the steps of: transmitting key values and associated timing information for a plurality of signals; .0 and transmitting information identifying the particular signal to which a key value belongs in the data packet containing the key value.

22. The method according to claim 18 further comprising the step of buffering transmitted .5 signal data at the receiver.

23. An article of manufacture embodying a program of instructions executable by a computer, the program of instructions including instructions for: selecting a set of key values that can be used to describe at least one data signal over a 0 period of time; associating timing information with each key value; using a data compression procedure to compress the timing information; temporally ordering the key values by determining the smallest time value at which a key value contributes to signal value, determining the largest time value at which a key value '5 contributes to signal value, and determining the time span over which a key value contributes to the at least one signal; streaming the key values in the set in temporal order by transmitting them through a fixed rate channel in at least one data packet; transmitting the timing information for a key value in the data packet containing the 0 key value; and transmitting decoding information to a receiver of the streamed key values using a stream header.

24. The article of manufacture of claim 23, the program of instructions further including instructions for: defining a subset of the key values that holds the smallest set sufficient to generate the signal value during the time period; and streaming the key values in the subset.

25. The article of manufacture of claim 23, the program of instructions further including instructions for regenerating the at least one signal from the streamed key values and associated timing information.

26. The article of manufacture of claim 23, the program of instructions further including instructions for: transmitting key values and associated timing information for a plurality of signals; and transmitting information identifying the particular signal to which a key value belongs in the data packet containing the key value.

27. The article of manufacture of claim 23, the program of instructions further including instructions for buffering transmitted signal data at the receiver.

28. A system for streaming data signals across an electronic network comprising: a first computer; at least one encoding software application accessible to the first computer for: selecting a set of key values that can be used to describe at least one data . signal over a period of time; associating timing information with each key value; using a data compression procedure to compress the timing information; temporally ordering the key values by determining the smallest time value at which a key value contributes to signal value, determining the largest time value at which a key value contributes to signal value, and determining the time span over which a key value contributes to the at least one signal; streaming the key values in the set in temporal order by transmitting them through a fixed rate channel in at least one data packet; transmitting the timing information for a key value in the data packet containing the key value; and transmitting decoding information to a receiver of the streamed key values using a stream header; a second computer; and at least one decoding software application accessible to the second computer for: receiving the data packets; and regenerating the data signal.

29. The system of claim 28, further comprising at least one database accessible to the second computer for buffering at least one received data packet.