AU676860B2 - Storyboard system for the simultaneous timing of multiple independent video animation clips - Google Patents

Description

~I -1- Storyboard System for the Simultaneous Timing of Multiple ndependntVde Animation Clips Field of the Invention The present invention relates to the display of video animation and in particular, the editing of video segments to produce a multilevel simultaneous display of multiple video clips.

Background Art Traditional film production techniques usually involve the shooting of multiple video or film 'takes' or 'clips'. Normally much more film is shot than is required for the final result. Ihe resultant film is normally edited by a film editor who uses scissor like splicers and adhesive tape to cut and paste together the desired shots. Artistic decisions about the selection of shots to enhance the performance are made and a final result produced.

If dissolves or wipes are required between two sources, then the processing becomes much more complicated, often involving the use of multiple video recorders and vision switchers.

Although editing and mixing can be implemented on a frame by frame basis, it is desirable to be able to format the editing/production process with what is known in the industry as a "storyboard". The storyboard represents a plan of the film video or animation that is to be created and assists in the visualisation of the production. Storyboards are traditionally created by hand, going through many interactions and revisions during production and is basically a chart of various sources of audio/video and how they are to be combined over time to produce the final film. Furthermore, as a director, or the like may wish to alter the production once an initial storyboard has been created, it is highly desirable that the storyboard itself be readily editable.

25 Recently, computer systems have been introduced which go part of the way to automate this process. Typically, these systems require animations to be created on a frame by frame basis, saving each frame to disk, with the image being played back at a later date by reading each frame off the disk and displaying it on a computer screen or the like.

Recently, in Australian Patent Application No. 3 823 8/93 (Attorney Ref: (RTO 1)(224857)), entitled 'An Audio and Video Processor System", claiming priority from Australian Provisional Patent No. PL2141 filed 29th April, 1992, the contents of which are hereby incorporated by cross-reference, a video animation system was proposed that was capable of producing real-time video animation through the use of real-time object (RTO) technology. That document, and other co-related documents describe apparatus that can perform audio and video mixing, editing, sampling, and the like at a substantially (RT022)(259582)

I

reduced cost than that of prior art arrangements. For a discussion of RTO technology, the reader is referred to Australian Patent Application No. 38244/93 (Attorney Ref: (RTO7)(202788)), entitled "A Real-Time Object Based Graphics System"claiming priority from Australian Provisional Patent No. PL2147, filed 29th April, 1992, the contents of which are hereby disclosed by cross-reference.

As a result of these developments, there is an ever increasing need for an automated storyboard system capable of editing, ordering and timing multiple clips for subsequent display.

It is an object of the present invention to provide an automated storyboard mechanism which substantially meets these needs.

Summary of the Invention In accordance with one aspect of the present invention there is provided a method for updating an automated storyboard system for the creation of information sequences, wherein said storyboard comprises an array of cells arranged into columns, each said cell having contents that indicates an information source having a certain active duration, said method comprising, for each column of the storyb ard which contains active cells, the computer-implemented steps f: determining which of said cells contains an information source having a shortest one of said active durations, 20 marking said column to have a column time duration of said cell of shortest active duration and terminating said information source of said cell of shortest active duration at the end of said column, and where other active cells exist for said colunm, adding a further column next to said column, said further column containing said other active. each having an updated 25 active duration of their previous active duration less said column time duration.

Preferably the method for updating an automated storyboard system further includes providing a means for terminating a multiplicity of cells at substantially the same time comprising the steps of: identifying in which cell of a storyboard that a second cell terminates; designating a first cell to stop in that cell, and adjusting the active duration of that cell to accord with the length of the second cell.

In accordance with other aspects of the present invention, there are provided various methods for automatically adding and moving clips within a storyboard.

F I- U))3 (RT022)(259582)(CFP0202AU) I =I r, II Lr L I -3- The columns can have any number of cells either active or inactive therein, thereby allowing different animations to be simultaneously displayed at different priority levels above one another.

Apparatus for implementing the methods is also disclosed.

Brief Description of the Drawings A preferred embodiment of the present invention will now be described with reference to the accompanying drawings in which: Fig. 1 illustrates an AVAPS system and its operational environment; Fig. 2 illustrates the AVAPS system of Fig. 1 in more detail; Fig. 3 illustrates a simple storyboard containing 4 columns, two rows, and therefore 8 cells.

.Fig. 4 illustrates a storyboard having two clips in a parallel; Fig. 5 illustrates a storyboard having two clips in a parallel; Fig. 6 illustrates a storyboard example that could be used for a presentation 15 describing the uses of a particular product in two countries.

Fig. 7 illustrates a storyboard directory entry.

Fig. 8 illustrates a ROM storyboard structure.

Fig. 9 illustrates a ROMCard Usage Table.

Fig. 10 illustrates a Cell structure.

Fig. 11 illustrates the structure of the flag entry in a cell.

Fig. 12 illustrates a clip reference pointed to by a cell in a storyboard.

Fig. 13 illustrates a clip address.

Fig. 14 illustrates a storyboard in RAM.

SFig. 15 illustrates an entry in the column table.

S 25 Fig. 16 illustrates a RowTable entry.

Fig. 17 illustrates the basic cell information copied in Get Cell Contents.

Detailed Description of the Preferred Embodiment The preferred embodiment of the present invention is designed to provide an automated storyboard mechanism capable of simultaneously editing, ordering and timifng multiple video animation clips for subsequent display on a audio and video animation processor system (AVAPS) such as that disclosed in the above mentioned patent application.

Referring now to Fig. 1 there is shown an AVAPS 1 and its operational environment.

Users purchase ROMcards 2, and operate the AVAPS 1 to sequence clips from the ROM (RT022)(259582) 9-~1 1 -4cards into storyboards. Storyboards are played on one or many AVAPSs. Users 4 can edit and save their sequenced clips (storyboards) onto RAMcards 3 for future use.

Referring now to Fig. 2 there is shown the A 'APS 1 in more detail. When playing a storyboard, the AVAPS generates graphics and selects shapes for miming based on audio input. These generated images are merged in a controlled manner with an incoming video signal, producing an output video signal. Sound effects are also generate 1, merged with the input audio signal and leave the AVAPS 1 in the form of audio out. Limited run-time animation control can be provided by the user, and a limited amount of synchronisation can occur with other the AVAPSs.

Those skilled will appreciate that in other embodiments, the AVAPS 1 can include laser-disc or another form of CD-ROM which can substitute or supplement the operation of the ROMcards 1 of Fig. 1 and Fig. 2, which are provided for a substantially portable Se AVAPS. Similar alternatives apply to the RAMcards 3 which can be supplemented by integral RAM or hard/floppydisc drives.

The overriding emphasis in the AVAPS 1 is on giving the user easy access to sophisticated but canned (preformatted and recorded) animations, which the user can modify in simple ways and combine to achieve the desired effect.

A clip is a predesigned 'canned' animation component that usually contains graphics and sound. However, those skilled in the art will appreciate that a clip can comprise only sound information, or visual information. The former would be used for preparing an audio recording or soundtrack, for example. The latter could be used to create a "silent film". A clip with visual information is capable of manipulating shapes, position, size, and colour and synchronising with the user by means of a keypress). The preferred embodiment to be described relates to clips with both sound and visual information.

25 Clips can contain user-editable components (such as editable text) and fixed (noneditable) components. The author of the clip can specify defaults for all of the user-editable components, so that a clip can be used without any customisation.

Each clip provided by the author has a duration type. The type is one of: fixed, where the clip can determine its duration without outside constraints by examining its own components). For example, a clip may show scrolling credits at a constant speed. The number of lines of text to scroll will determine the duration of the clip.

unknown, where the clip has no idea of its duration, nor can it easily work it out.

For example, if a clip waits for the user to press a key, the clip will not know ahead of play-time how long it will be before the user presses the key.

(RT022)(259582) 1N infinite, where the clip has no duration limit, and will display for as long as the user wishes it to. An example of this is a background image behind some sequenced animation. The background image should be displayed as long as necessary in the animation. Typically, a clip with infinite duration will need to be tied to the duration of another clip. For instance, a background image should be tied to last as long as the scrolling credit clip lasts.

The preferred embodiment of a storyboard system is configured to allow a user to: select a clip from a ROMcard; edit those components of a clip that are editable; play a clip; and sequence, layer and synchronize multiple clips.

For the purposes of the preferred embodiment, the user can decide to combine clips into a storyboard. The storyboard represents the display order and sequence of the clips. It indicates which clips are run together, what dependencies they have on their starting and finishing times, and what display order in terms of relative visual priority each clip has.

Consequently a storyboard represents: time order. This includes representing clips that run in parallel (clip A runs at the same time as clip and sequence (clip A follows clip B) display order. This includes representing the relative display order priorities of -lips that are active at the same time (clip A should be displayed above clip B) synchronization. This includes starting and stopping clips together (clip A should terminate when clip B terminates) as well as clip markers for random access during playback Fig. 3 illustrates that a storyboard is conceptually like a spreadsheet, with rows, S" 25 columns, and cells. Consecutive columns represent consecutive time segments, and rows represent display order within a time segment. Both dimensions are relative rather than absolute. Thus each column may represent a different length of time.

A storyboard is read from left to right, one column at a time. Each column represents one time segment, and the cells in each column represent relative display levels during-that time segment.

Each cell within a column may contain one of the following items: nothing. If a cell contains nothing, it means that there is nothing of consequence happening during that time segment on that relative display level.

a clip. If a cell contains a clip, it means that the clip starts at the beginning of the cell's time segment at that cell's relative display level.

(RT022)(259582) I _I I Lu~ I I, -6a continuation marker. If a cell contains a continuation marker, it means that the clip that began in an earlier time segment on this same row is continuing through this cell's time segment.

Each cell also has a stopped flag, which indicates if the clip is to be forced to stop at the end of that cell's time segment.

The length of time represented by a particular column is defined to be the minimum time of all cells in that column. All clips that are in a particular column are defined to start together. Their finishing times will vary depending on the length of each clip. The length of the column will therefore be the length of the shortest clip. All other clips will then continue into the next column (or time segment) via a continuation marker. The length of the next column will be the shortest of all remaining time for each clip and any new clips S:I" that have started in that column. For the purposes of calculating a minimum, an unknown Sduration is treated as zero time, and an infinite duration is treated as infinite.

oo oi The continuation markers are automatically added and removed by the AVAPS storyboard system, and therefore the number of columns that exist in a storyboard can change dynamically based on the number of time segments that are required to represent the clips.

A clip can be terminated prematurely by setting a stopped flag in a cell. This can be accomplished directly by changing the stopped flag explicitly), or indirectly by starting another clip where a continuation marker was previously). In either case, the clip will then terminate at the end of the column, regardless of how much time was remaining on that clip.

Storyboards also contain user-defined clip markers. Each column can have a clip 2 marker associated with it provided there is at least one clip in that column. The clip marker 25 refers to all clips that begin in the column.

Given the above representation of a storyboard, it is possible to interpret the meaning of a given storyboard.

Referring now to Fig. 4 there is shown a storyboard 6 having the same two clips in sequence, clip A and clip B. The first column is of length clip A, and the length of the, second column is of length clip B. Consequently clip A is run first, then clip B is run when clip A has finished. This is true regardless of the length of A or B. If clip A had an unknown duration, then clip B would be run only when clip A finished (however long that was). If clip A had an infinite duration, then clip B would never be run, as clip A would never terminate.

(RT022)(259582) I I -7- Referring now to Fig. 5 there is shown a storyboard 7 having the same two clips in a parallel. The first column has the two clips, so its length is that of the shortest of the two clips clip As clip A has finished by the end of the first time-segment, it does not continue to the next time-segment. Consequently, the second column has a continuation marker in the cell of the same row that clip B was in, to indicate that it continues to the next time-segment. The minimum time of the clips active in this second time-segment is the time (clip B clip The relative display order of the two clips is apparent from the storyboard. Clip A is displayed on a higher priority to clip B, therefore when rendering the storyboard, if an object from clip A is drawn overlapping an object from clip B, the viewer will see the object from clip A on-top-of the object from clip B.

The first column represents the starting time of the animation, the second column represents the time when clip A has finished (and B is continuing), and the third column represents the time when clip B has finished.

Referring now to Fig. 6 there is shown a storyboard example 8 that could be used for a presentation describing the uses of a particular product in two countries. Clips A, B, and C (the first column) may represent the animated company logo and title. Clip D may represent the text describing the product's use in the first country, while clip G may represent the text describing the product's use in the second country. Clip E represents the graphics of the product itself, while clips F and H represent the graphical maps of the two countries.

The first column contains clips A, B, and C. If clips A and B have their stopped flags set, they will both terminate when clip C terminates. i.e. the animated title and logo will be on the screen for the duration of the delay clip C.

The second column shows that clips D, E, and F are started simultaneously when 25 clips A, B, and C terminate. Clip F is terminated by the shortest clip in the column (clip D), and clip E continues after clips D and F are terminated. The third column shows that clips G and H start when D and F terminate, and finally all clips terminate when clip G finishes.

A storyboard is changed by performing actions on cells and columns in the storyboard. Each action affects the cell, and then the storyboard is re-evaluated.

Effectively, the minimum time for the cell's column is recalculated, and this calculation is propagated as far as required.

The following actions can be performed on cells in the storyboard: place a clip in a cell; delete a clip from a cell; change the stopped flag of a cell; (RT022)(259582) I~ -~ula~ -8edit a clip; and insert a clip between cells.

The effect on storyboard for each of these actions is described in the table below: Cell Place clip in cell Delete contents Change Edit cell contains of cell stopped flag nothing change cell to new clip, not meaningful, not not set time clip's duration, leave cell as meaning meaningful clear stopped flag nothing -ful leave flag clear clip preserve changed data change cell to change set time from ola clip to new clip, nothing, flag clip's change cell to new clip, set time 0, duration set time clip's duration, clear stopped flag preserve stopped flag

.C.

15 continuati on set stopped flag in previous cell in same row, change cell to new clip, set time clip's duration, preserve stopped flag not meaningful, leave cell as continuation change flag not meaningful

CCCCC.

The other action of inserting a clip between cells, is implemented by one of two actions: if inserting between two columns, add a column containing nothings and the clip (all clear stopped flags); or if inserting between rows, add a row containing nothings and the clip (all clear stopped flags).

Once an action has been performed on a particular cell within a column, the storyboard must be changed to reflect the changes to the cell. Effectively, the minimum time for the column needs to be recalculated, and the effects propagated as far as necessary throughout the storyboard.

The structure of a computer program in pseudo code implementing the process for recalculating is as follows: (RT022)(259582) II p L r d~CI -9while not finished recalculate column if column changed during recalculation advance to next column else finished The Pseudo-Code for recalculating a column is as follows: set minimum time of column to be infinity for each cell in column recalculate cell adjust minimum time ofcolumn for each cell of column set time time minimum time of column ifentire column is fidl of nothings delete column Recalculating a cell is based on the time remaining for the cell in the previous column on the same row. This time is called t. Eventually, a time will be calculated for the time remaining in the current column. This will be the t for use in calculating the next column.

The following table shows the actions to perform in order to recalculate a particular 0 Scell.

S Cell type t <0 t 0 nothing leave cell as nothing change cell to continuation marker with clear stopped flag, set time t, update minimum time for column (RT022)(259582) -I I e 9 0 090* 9* 9 9 o *0*0 9 *9 9 9 Cell type tO0 t>0 clip leave cell as clip with insert new colunm of nothings, original stopped flag, leave re-evaluate from new column time as clip's duration, update minimum time (m) for column continuati change cell to nothing, set leave cell as continuation marker with on marker time 0 original stopped flag, set time t, update minimum time for column The only way a cell can change (apart from a user action on a cell) is for a cell in the previous column to change. Therefore, if a column has not changed, then no subsequent columns will change.

15 A storyboard may be played. Playing involves rendering each of the clips in a column for the specified duration, and then stopping or continuing the clip based upon the next column. For a full discussion of the rendering of a storyboard, the reader is referred to Australian Patent Application No. PL6833 filed 15 January 1993 and entitled "Render System for rendering of Storyboard Structures on a Real-Time Animation System", the 20 disclosure of which is hereby incorporated by cross-reference.

Each storyboard incorporates a number of data structures and corresponding programs which are initially stored on ROM and RAM cards before being loaded into the AVAPS 1. The data structures are then manipulated by programs run on the AVAPS 1 under the command and control of the user. The various data structures utilized by the user will now be described with reference to Fig. 7 to Fig. 17.

Referring now to Fig. 7 there is shown a storyboard directory entry 9. Each storyboard has an entry in the Storyboard Directory either on a ROM card or a RAM card containing at least the following information: Icon is a pointer to the icon for a particular storyboard. Since storybcards normally may have no visible icons in the user-interface, this value can be NULL; Name is a pointer to the name for the storyboar, This is a textual description of the storyboard, and will have a maximum size associated with it (imposed implicitly by the AVAPS A fixed amount of space is allocated for RAM card storyboard name storage to enable renaming of storyboards without fragmentation of the RAMcard; and ROMStoryboard is a pointer to a ROMStoryboard structue.

(RT022)(259582) -11- Referring now to Fig. 8 there is shown a ROM storyboard structure 10. The story',oard structure is a means of accessing the various components that make up a storyboard. Each storyboard saved to ROMcard has this structure. Storyboards in RAM also have this structure, with some additional information added to increase access speed.

In particular: ROMCardUsage is a pointer to the first element in a linked list indicating status information about the ROM cards used by the storyboard. This linked list is known as a ROMCard Usage Table; Rows indicates the number of active rows in the storyboard; Columns indicates how many active columns are in the storyboard; and Cells is a pointer to the storyboard's cells, stored contiguously column at n time.

Referring now to Fig. 9 there is shown the ROMCard Usage Table. It is important to keep track of which ROMcards are used by a storyboard. The ROMCard usage table record keeps track of this information, as well as where the ROMcard is in the AVAPS memory.

The information stored in the ROMcard usage table is as follows: S- ROMcardID is the identification (ID) associated with a particular ROMcard that makes it unique; ROMcardname points to a text string defining the name of the ROMcard. This is useful when prompting the user for a particular card, where the name of the ROMcard has more meaning to the user rather than the ID; UsageCount indicates how many clips and editable local variable block (LVB) entries reference a particular ROMcard; and Next is the address of the next ROMcard Usage entry in this Storyboard. A NULL pointer indicates that there are no further ROMcards used.

25 Referring now to Fig. 10 there is shown a cell structure 12. A column consists of an array of cells. Each cell includes the following information: Type may be one of the following: Empty, ClipReference, and ContinuationMarker; Clip Ref Marker is NULL when the Type Empty. It is a pointer to a Clip Reference when the Type ClipReference or ContinuationMarker; TimeRemaining indicates how much time is remaining for an active clip after this cell has completed; Flags is a set of booleans. The structure of this record can be found in Fig. 11 and includes a Stopped flag and a Changed flag in addition to a number of reserved flags for future use; The Stopped flag, when set, signifies that the clip to terminate when the (RT022)(259582) ii -12cell's column terminates; The Changed flag is used internally by a Storybo:rd Manager to maintain what cells of a storyboard have changed since the last evaluation; The unused reserved bits should be set to 0.

Referring now to Fig. 12 there is shown a clip reference pointed to by a cell in a storyboard. A clip reference 13 contains the following information: ClipAddress is the address pointed to by the ClipRefI/Marker as shown in Fig. Component Types indicates how many editable component types a given clip has in its local variable block (LVB); LVB points to the user LVB structure for the clip. Note that the LVB pointed to by this reference is a different LVB to the actual clip. The LVB contains the user editable portions of each clip.

'00. Duration summarizes the duration of the clip in AVAPS time units; #ColoursUsed summarizes how many colour layers are required by the clip; #OpaqueEffects summarises how many opaque effects are used by the clip; o 15 #TranspEffects summarises how many transparent effects are used by the clip; #Externals indicates how many external variables are required by the clip; Externals points to an array of externals. Instantiated externals are non-NULL; Resolved indicates whether the clip reference is resolved or not. A resolved clip reference has all its components on loaded ROMcards, and can therefore be played; #Convertedhandles indicates how many non-primitive handles are in the clip; ConvertedHandles is a pointer to a list of pointers to converted handles a TextHandle would be converted to a GraphicHandle and stored via the ClipRef); #RTOObjects defines how many RTO objects are used by the clip; and MemForRenderClip defines how much memory is required for the clip to be 25 rendered. It includes two sets of MemForObjects.

Referring now to Fig. 13 there is shown a clip address 14. A clip address indicates where to find a particular clip. A clip address has the following structure: ROMcardID indicates the ID (identity) of the ROMcard to use to find the particular clip; Dir Index indicates which directory number within the particular ROMcard the clip can be found. This can also be used to determine which clip family the clip belongs to; Clip Index indicates which entry number in the directory on the particular ROMcard should be used to access the clip; and (RT022)(259582) j ~I -13- ClipDirEntry points to the resolved directory entry structure on the ROM card where the clip is kept.

The storyboards, which reside on the RAMCards, are initially copied to the internal RAM of the AVAPS 1. Storyboards in RAM are maintained by storyboard and associated data structures which are used in addition to the structures stored on the various memory cards.

Referring now to Fig. 14 there is shown a storyboard in RAM 15. When a storyboard is copied from ROM or RAMcard to RAM, additional information is kept to increase access speed to various components. The storyboard in RAM has the following structure: ROMStoryboard is the base storyboard structure as described in Fig. 8; #ClipsPlaying indicates how many of the storyboard's clips are currently being played. A storyboard can only be modified while no clips are being played; #UnResolvedClips indicates how many of the clips in the storyboard are still unresolved that is, they ieed a ROMcard to be inserted so that the slip can be played; 15 #UnresolvedExtemals indicates how many externals in the storyboard still require resolving before the storyboard can be played; S- MaxRows indicates the maximum number of rows allocated for storyboard growth; MaxColumns indicates the maximum number of columns allocated for storyboard 20 growth; CurrentColumn indicates the number of the current column either being played (if playing), or last played (if no longer playing); ColumnTable points to a table detailing information about the cells of the storyboard in RAM. This is used instead of the cells components of ROMStoryboard; 25 ClipMarkers indicates how many clip markers are currently in the storyboard; ClipMarkerTable is a pointer to the table containing clip marker information; and RowTable is a pointer to the table containing row information.

A storyboard consists of a spreadsheet of columns and rows. Since the storyboard reorganises itself in terms of columns (which are added and deleted dynamically as previously described), the storyboard itself is kept in column order. The column table indicates the current column order of the storyboard as an array of column pointers.

Shuffling the order of columns in a storyboard, and inserting and deleting columns is simply a matter of shuffling entries in this table.

(RT022)(259582) I I

I_

-14- Referring now to Fig. 15 there is shown an entry in the column table 16. The number of valid entries in the ColumnTable is given by the #Columns value in the ROMStoryboard structure. The information provided is as follows: Cells is a pointer to an array of cells for a particular column in the storyboard. The number of valid cells in a column is given by the #Rows value in the storyboard structure; and ClipMarkers indicates how many clip markers are present in the column.

Referring now to Fig. 16 there is shown a RowTable entry 17. While a storyboard is being edited, the number of clips in each row is maintained. When a row between two nonempty rows becomes empty, then the empty row is deleted from the storyboard, and the remainder of the storyboard is shifted up to take the place of the deleted row. The RowTable contains the row information required to achieve this functionality. The Row Table is a table of RowTable Entries with the following information: ComponentCount is the number of non-empty and non-continuation-marker cells 15 in the row.

In order to manipulate storyboards within the AVAPS 1, operations are performed on the aforementioned data structures. These operations are implemented in software codes and include providing means for the automatic manipulation of elements of a storyboard, including the manipulation of clips within a storyboard and the automatic placement of continuation markers to reflect the changes in timings within the storyboard. The functions Sprovided are as outlined in the following paragraphs: New Storyboard This operation creates a new, empty storyboard of a given size. The parameters for this operation are: 25 number of rows number of columns address of place to store the address of the new storyboard.

SA pseudo-code representation of this process is provided as follows: ColumnSize #Rows CellSize A SpreadsheetSize #Columns ColumnSize B ClipMarkerTableSize MaxClipMarkers sizeof(ClipMarkerTable entry) C ColumnTableSize #Columns sizeof(ColumnTable entry) D RowTableSize #Rows sizeof(RowTable entry) (RT022)(259582) I r_ Mem Required A B C D sizeof(Storyboard) Allocate the single portion of memory required for the storyboard If unable to allocate return NotEnoughMemory Set the maximum number of rows and columns Set the number of active rows and columns For each column: set the number of clip markers in the ColumnTable to 0 set the address of the column in the ColumnTable For each row: MakeEmpty(cell) For each row set the ComponentCount in the RowTable to be 0 15 Set ROMcardUsageTable to be NULL Store address ofnew storyboard in specified location #UnresolvedClips 0 #UnresolvedExternals 0 return OK S Clear Storyboard This operation clears all the cells of a given storyboard by deleting all clips and clip markers from it. The parameters for this operation are: the address of the storyboard to clear; 25 A pseudo-code representation of this process is provided as follows: If (Storyboard is playing) return StoryboardPlaying For each column from 0 to #Columns For each row in column Sub Delete Cell EvaluateCompleteStoryboard return OK (RT022)(259582)

I

-16- Delete Contents Of Cell This operation deletes the contents of a cell in the specified storyboard and frees any memory associated with any clip references that were present. The parameters for this operation are: the address of the storyboard being used; the row number of the cell; and the column number of the cell.

A pseudo-code representation of this process is provided as follows: If(Storyboard is playing) return StoryboardPlaying ErrorCode Get Cell Address(row, column, cell) If (ErrorCode OK) If(cell. type Empty or ContinuationMarker) S 15 return CellContainsNonDeletableComponent Sub Delete Cell ErrorCode EvaluateStotyboard(Column) return ErrorCode 20 AddClip This operation adds a clip to the specified cell. The parameters for this operation are: the address of the storyboard being used; a ClipAddress structure. If ClipDirEntry is NULL, it will be filled in; the row number of the cell; and 25 the column number of the cell.

A pseudo-code representation of this process is provided as follows: If (Storyboard is playing) return StoryboardPlaying ErrorCode Get Cell Address(row, column, cell) If (ErrorCode OK) ec column If (cell.type CONTINUATION_MARKER) ec-- SubSetStoppedFlag(row, ec, TRUE) (RT022)(259582) I I I I- -17- Sub_Add Clip(Row, column) EvaluateStoryboard(ec) return ErrorCode Add Clip Between Cells This operation adds a clip between the specified cells, adding a column or row or both as necessary. The parameters for this operation are: the address of the storyboard being used; a ClipAddress structure (see Fig. 13). If ClipDirEntry is NULL, it will be filled in; the row number of the first cell; the column number of the first cell; the row number of the second cell; the column number of the second cell; and A pseudo-code representation of this process is provided as follows: If (Storyboard is playing) return StoryboardPlaying ErrorCode InsertCell If(ErrorCode OK) 20 ec column If((GetCellAddress(row, ec+1, next_cell) OK) AND (next_cell. type CONTINUATION_MARKER)) ec-- Sub_Se StoppedFlag(row, ec, TRUE) Sub_Add Clip(row, column) EvaluateStoryboard(ec) return ErrorCode Move Cell This operation moves the contents of a particular cell to another. The parameters'for this operation are: the address of the storyboard being used; the source row number of the cell; the source column number of the cell; the destination row number for the cell contents; and (RT022)(259582) -e c I i -18the destination column number for the cell contents.

A pseudo-code representation of this process is provided as follows: If (Storyboard is playing) return StoryboardPlaying ErrorCode ValidateFromCell(fiom_row, from_column, from_cell) If(ErrorCode OK) ErrorCode Get Cell Address(to_row, to_column, to_cell) If((ErrorCode OK) AND (to_cell !=fiom_cell)) If(to_cell. type CONTINUATION MARKER) Sub_SctStoppedFlag(to_row, tocolumn-1, TRUE) Sub_Delete Cell (to_cell) *tocell *fiom_cell Make_Empty(from_cell) 15 IncrementRowUsage(torow) DecrementRowUsage(fi-omrow) if(to_cell.type CLIP_MARKER) Update_CMEntry(to_cell. cm_id, to_column) EvaluateCompleteStoryboard 20 return ErrorCode Move Cell Between Cells This operation moves the contents of a particular cell to another cell created between a two cells. The parameters for this operation are: the address of the storyboard being used; 25 the source row number of the cell; the source colunn number of the cell; the destination row number of the first cell; the destination column number of the first cell; the destination row number of the second cell; and the destination column number of the second cell.

A pseudo-code representation of this process is pro Aided as follows: If (Storyboard is playing) return StoryboardPlaying (RT022)(259582)

L

F -19- ErrorCode ValidateFromCell(f'm_row, from_column, from cell) If(ErrorCode OK) ErrorCode InsertCell If (ErrorCode OK) If((GetCelL4ddress(row, column-1, prev_cell) OK) AND (cell2.type CONTINUATION_MARKER)) Sub_SetStoppedFlag(row, column-1, TRUE) If(rowl row2) AND (rowl <fromrow) fromrow++ If(columnl column2) AND (columnl fomcolumn) from_column++ ErrorCode MoveCell(from_column, from_row, row, column) return ErrorCode Copy Cell 15 This operation copies the contents of a particular cell to another (only valid for clips).

S The parameters for this operation are: the address of the storyboard being used; the source row number of the cell; the source column number of the cell; the destination row number for the cell contents; and 20 the destination column number for the cell contents; and the destination column number for the cell contents, S. A pseudo-code representation of this process is provided as follows: 25 return Copy_Cell_To_Storyboard(this) Copy Cell Between Cells This operation copies the contents of a particular cell to another cell created between two cells. The parameters for this operation are: the address of the storyboard being used; the source r-w number of the cell; the source column number of the cell; the destination row number of the first cell; the destination column number of the first cell; the destination row number of the second cell; and (RT022)(259582) I, I id I

~I

the destination column number of the second cell.

A pseudo-code representation of this process is provided as follows: If (Storyboard is playing) return StoryboardPlaying ErrorCode ValidateFromCell(from_row, from_column, fiom_cell) If(ErrorCode OK) ErrorCode InsertCell Can copy the cell, we just must be careful about where to If (ErrorCode OK) ec column If((GetCellAddress(row, column+1, next_cell) OK) AND (next_cell.type CONTINUATION_MARKER)) 15 ec-- Sub_SetStoppedFlag(row, ec, TRUE) Fix up fiom address (may have changed due to an inserted column) If ((rowl row2) AND (row] <from_row)) fromrow++ 20 If((columnl column2) AND (column] fiomcolumn)) fromcolumn++ GetCellAddress(from row, fromcolumn, fromcell) Sub_Delete Cell (to_cell) ErrorCode Sub_Copy cell(from_cell, tocell) if (ErrorCode OK) EvaluateStoryboard(ec) return ErrorCode Get Cell Contents Referring now to Fig. 17 there is shown the basic cell information 18 copied in Get Cell Contents. Copies of the basic cell structure is made to a caller specified address. The information copied is: Type is one of Empty, Clip, ContinuationMarker as defined previously; Flags is the cell flags structure containing the stopped flag as defined previously; (RT022)(259582) '~llllllll'lr~r~E is -21- ClipAddress indicates where to find the clip (on which ROMcard) if Type Clip.

It is invalid otherwise. The structure of the record is as defined previously; Clip Marker. If Type ClipMarker, it holds the clip marker ID of the cell; Colour Usage summarises the colour usage for the particular clip if Type Clip. It is 0 otherwise; #OpaqueEffects summarises the opaque effects usage for the particular clip if Type Clip. It is 0 otherwise; #TranspEffects summarises the transparent effects usage for the particular clip if Type Clip. It is 0 otherwise; Duration summarises the duration of the clip if Type Clip. It is 0 otherwise; #Externals indicates how many externals the clip uses; and ResolvedFlag indicates if the clip can be run with the current ROMcard status.

"The parameters for this operation are: the address of the storyboard being used; 15 the row number of the cell; the column number of the cell; and address of location to copy the cell information.

A pseudo-code representation of this process is provided as follows: 20 ErrorCode Get Cell Address(row, column, cell) If (ErrorCode OK) Store base components of cell from cliprefat specified address return ErrorCode S 25 Get Column Level Usage This operation returns the colour and effect usage for a particular column. The parameters for this operation are: the address of the storyboard being used; the column number; address of location to copy the colour information; and address of location to copy the effect information.

A pseudo-code representation of this process is provided as follows: ErrorCode ValidateCell (Column, 0) If (ErrorCode OK) (RT022)(259582) L I -22- For each cell in column Add Effects and Colour usage to total Store total usage in desired location return ErrorCode Validate From Cell This operation gets the from cell's address and validates the cell type for move/copy.

A pseudo-code representation of this process is provided as follows: ErrorCode GetCellAddress(from_row, from_column, fiom_cell) If (ErrorCode OK) If (from_cell.type EMPTY OR CONTINUATION_MARKER) ErrorCode InvalidCellType return ErrorCode Column Number From Address A pseudo-code representation of this process is provided as follows: For each entry in ColumnTable up to #Columns S 20 If (ColumnTable[entry].cells column) Store column number (entry) in specified location return OK return NoSuchColumn 25 Get CipRef Address A pseudo-code representation of this process is provided as follows: ErrorCode GetCellAddress(row, column, cell) If (ErrorCode OK) If(cell->type Clip) store cell->clipref in specified location Else ErrorCode CellDoesNotContainClip return ErrorCode (RT022)(259582) L __I 23 t ellAddress A pseudo-code representation of this process is provided as follows: ErrorCode ValidateCell If(ErrorCode OK) store address of cell in specified location (cell address address of (columntable[column]. cells[row])) return ErrorCode Validate Cell A pseudo-code representation of this process is provided as follows: (**If(Column# Maxcolumns) return ColumnOutOjRange If (Row# Max rows) return RowOutOfRange return OK Make Fmpty 20 A pseudo-code representation of this process is provided as follows: cell->type Empty Scell->flags 0 cell->clipref= NULL 25 cell->timeremaining 0 Insert Cell This operation makes sure that the cell being inserted is valid, and then inserts it into the storyboard, inserting a row, column or both if necessary. A pseudo-code representation of this process is provided as follows: ErrorCode ValidateCell(rowl columnl#) IfErrorCode OK ErrorCode ValidateCell(rowl columnl IfErrorCode

OK

(RT022)(259582) I II I u 24 return ErrorCode row max('Row] il, Row2#) min row =min(Row]#, Row2#) column =max(Column]#f, Column2,) mnncolumn min(Column]#, Colunun2?#) Now validate that they are adacent if(rw- mmi row) 1) OR (('column min-column) 1)) return CelisNotAdjacent Do the work I*(Columnnl4 Column2#) InsertColumn (column) If (Row]#4!=Row2#) 15 InsertRow (row) :store row in specified location store column in specified location err =GetCellA ddress (row, column, cell) return OK( JnserL Row TH operation inserts a row at row N on all active columns within a storyboard. A pseudo-c_-xie representation of this process is provided as follows: If (grows+]-)) return RowOutOfRange If (4 rows max rows) return OutO,/i~emory for each active column for (r max -rows-i uip to N) copy data from cellfrJ to cell[r+1J set cellfr] to be Empty Fix up the row table. ThL w row inserted has no entries yet for (r =max-rows-i up to N,) (RT022)(259582) 25 copy row tab lefrj to row-tablefr+fl row table fri= 0 growvs++ return OK Delete Rows This operation deletes from Ri up to (but not including) R2 from all columns in SB.

A pseudo-code representation of this process is provided as follows: If (R2 >max-rows) return Row OutOiRange column =columnsfOi count max-columns) while (count) 15 %first copy the data in the column the cells) :to r] from r2 while (from max-rows) cellfto] =cell/ffrom] from++ to++ while (to max rows) to+ %now update the row table to =r] from =r2 while (fr-om max-rows) row-tab leftoj row tableirom] firom++ to+ while (to max rows) row-tablefto]. count =0 to++ (RT022)(259582) -26coluinn++ countreturn OK In,;i-tClu~mn This operation inserts a column at column N. A pseudo-code represcntation of this process is provided as follows: mn max columns I if (c >mi) return Column Out OTange row 0 cell colurnntable[c]. cells -while (row #rows) 15 if (cell. type !=Empty) return StoyboardTooSmall row++ cell++ 20 Now can insert the column tmp columntable[mn] ****while (in c) columntable[in] columntable fm-i] 25 r~olumntablefmJ=temp return OK Delete Columns This operation deletes columns from Cl up to (but not including) C2. A pseudo-;ode representation of this process is provided as follows: if c2 max-columns) return Column OutOJRange while (c2 ci) temp columns [c] (RT022)(2595 82) -27i cl+1 while (i max_columns) columns[i-l] columns[i] columns[i-l] temp c2return OK Increment Row Usage This operation updates the usage information of a particular row. A pseudo-code representation of this process is provided as follows: rowtable[row]++ 1Decrement Row Isacge 15 This operation updates the usage of a particular row. A pseudo-code representation of this process is provided a, follows: rowtable[row]-- Compress Rows This operation goes through all the rows deleting blank ones as necessary. At the end of processing, #rows will reflect the true number of active rows in the storyboard. A pseudo-code representation of this process is provided as follows: 25 last active row -1 rows to move 0 row 0 numrows max rows while (row numrows) if (RowTable[row]. count 0) rows tomove++ else if(rows tomove 0) last activerow++ DeleteRows(lastactive_row, row) (RT022)(259582) -28row last active row numrows rowstomove rows to move 0 else last active row row row++ #rows last active row 1 Evaluate Storyboard This operation evaluates all columns until the end of the storyboard is met column start column for evaluations fix up so is valid due to possible changes from inserting between cells. A pseudo-code representation of this process is provided as *follows: if(column 0) column 0 CompressRows See if the change was made outside the range of the active board.

Ifso, delete any columns from the active_board up to the new one if (column #columns) columns tomove column #columns #columns++ if (columms to move 0) 25 DeleteColumns(#columns, column) #columns++ return OK Otherwise re-evaluate the storyboard from the column of change to a %point where there is no change last active column column 1 columns to move 0 while (column #columns) column_status Evaluate_Column(column, isactive) switch (column status) (RT022)(259582)

I

-29case CANNOT INSERT.

return StoryboardTooSmall case INSERTED #columns++ if (#columns maxcolumns) #columns max columns return StoryboardTooSmall case CHANGED: case NOCHANGE: if is active) columns tomove++ if(is_active OR (column #columns 1)) S 15 if(columnstomove 0) last activecolumn++ DeleteColumns(last_active_column, column) column lastactivecolumn #columns columns to move columns to move 0 else last active column column if(column_status CHANGED) 25 column++ else column #columns column #columns return OK Evaluate Entire Storyboard This operation evaluates all columns until the end of the storyboard is met. A pseudocode representation of this process is provided as follows: CompressRows (RT022)(259582)

-E

column 0 last active column -1 numcols max columns while (column num_cols) switch (Evaluate_Column(column, is_active)) case CANNOT INSERT: return StoryboardTooSmall case INSERTED: num cols++ if(numcols max_columns) num cols max columns case CHANGED: 15 case NOCHANGE: if(is_active) if (columns to_move 0) lastactivecolumn DeleteColumns(lastactive_column, column) column last active column num_cols columns to move columns to move 0 last active column column else 25 columnstomove++ column++ #columns last active column I return OK Evaluate Column This operation evaluates a particular column and set is_active flag. It assumes that the column is inactive. A pseudo-code representation of this process is provided as follows: is active FALSE status UNCHANGED (RT022)(259582) I I -31 min time INFINITE Get the current cell andprevious column's cell cell columntable[column], cells if(column 0) prev_cell blank_cell else prev_cell columntable[column-1]. cells Now go through the cells one by one row 0 while (row #rows) t prev_cell-> time_remaining Update status if the cell had changed due to some external source 15 if(cell->flags. changed) status CHANGED cell->flags. changed FALSE Is a clip attempting to continue into this column? if(t> 0) switch (cell->type) case EMPTY: cell->type CONTINUATION_MARKER cell->flags.stopped FALSE 25 cell->timeremaining t min_time min(min_time, t) case CLIP.

case CLIP MARKER.

is active TRUE if(status CHANGED) cell->flags.changed TRUE err InsertColumn(column) if(err OK) return INSERTED (RT022)(259582) 32 else return CANNOTINSERT case CONTINUATION _4RKER: cell-> timneremaining =t cell-> clipref prey-cell-> clipref min-time min('min-time, t) else switch ('cell->type) case EMWPTY: do nothing case CLIP:' cell-> time-remaining cell->clipref->duration min time min(min-time, cell-> time remaining) case CLIP iVL4RKER: *is-active =TRUE case CONTINUATIONMA4RKER.

M1,ake Empty (cell) status CHANGED .ft Update to be at the next cell cell++ if (column 0) preyv-cell++ row++ Can now determine if the column is active (if not registered already) if (min time 0) is-active =TRUE (RT022)(259582) 33 If(is_active AND (mintime 0)) Finally go through and adjust the time of each cell by min time cell columntable[column], cells row 0 while (row #rows) t cell->time_remaining if(mnintime INFINITE) t=O else if (t INFINITE) min time if (cell->flags.stopped) t 0 cell->time remaining t S 15 cell++ row++ return status The forgoing can be performed on any suitably programmed computerised device such as that referred to previously by cross-reference. Accordingly, the preferred embodiment, provides a means for the automatic manipulation of elements of a storyboard, S. including the manipulation of clips within a storyboard and the automatic placement of continuation markers to reflect the changes in timings within the storyboard.

i'* The foregoing describes only one embodiment of the present invention, S: 25 modifications, including the implementation of the storyboard on different computer platforms in different computer programming languages, obvious to those skilled in the art, can be made thereto without departing from the scope of the present invention.

For example, although the preferred embodiment is configured for use with RTO technology, other graphics arrangements can be manipulated in corresponding ways. (RTO22)(259582) st

Claims (13)

1. A method for updating an automated storyboard system for the creation of information sequences, wherein said storyboard comprises an array of cells arranged into columns, each said cell having contents that indicates an information source having a certain active duration, said method comprising, for each column of the storyboard which contains active cells, the computer-implemented steps of: determining which of said cells contains an information source having a shortest one of said active durations, marking said column to have a column time duration of said cell of shortest active duration and terminating said information source of said cell of shortest active duration at the end of said column, and where other active cells exist for said column, adding a further column next to said column, said further column containing said other active cells each having an updated active duration of their previous active duration less said colunm time duration.

2. A method as claimed in claim 1 further comprising the step of deleting at S.least one cell from said storyboard and applying said steps to after said deletion.

3. A method as claimed in claim 1 or 2 further comprising the step of adding *o at least one cell having an information source, to said storyboard and applying said steps to after said addition. 20

4. A method as claimed in claim 3 further comprising the step of adding at least one new cell between two pre-existing columns and applying said method steps after said addition of the new cell to create a corresponding new column.

5. A method as claimed in any one of the preceding claims further comprising the step of altering the contents of at least one of said cells and applying said 25 steps to after altering said one cell.

6. A method as claimed in any one of the preceding claims further comprising the step of altering said active duration of one of said cells and then applying said steps to (c)

7. A method as claimed in any one of the preceding claims further comprising the step of terminating a multiplicity of selected ones of said cells substantially simultaneously with termination of a predetermined candidate cell, said terminating step comprising the sub-steps of: identifying a particular column of the storyboard thrt said information source of said predetermined candidate cell terminates; and (RT022)(259582)(CFPO2O2AU) I I I I designating said multiplicity of selected cells to terminate in said particular column, and adjusting the active duration of said selected cells to accord with the active duration of said candidate cell.

8. A method as claimed in any one of the preceding claims wherein said information source comprises a source of audio information.

9. A method as claimed in any one of the preceding claims wherein said information source comprises a source of video information.

A method as claimed in any one of the preceding claims wherein said information source comprises a source of audio and video information.

11. Apparatus configured to perform the method as claimed in any one of the preceding claims.

12. Apparatus for updating an automated storyboard system for the creation of information sequences, wherein said storyboard comprises an array of cells arranged into columns, each said cell having contents that indicates an information source having a certain active duration, said method comprising, for each column of the storyboard which contains active cells, said apparatus comprising: shrtmeans for determining which of said cells contains an information source having a shortest one of said active durations, means for marking said column to have a column time duration of said cell of 20 shortest active duration and terminating said information source of said cell of shortest active duration at the end of said column, and, where other active cells exist for said •column, means for adding a further column next to said column, said further column containing said other active cells each having an updated active duration of their previous 25 active duration less said column time duration.

13. A method for updating an automated storyboard system substantially as hereinbefore described with reference to the accompanying drawings. DATED this Eighth Day of January 1997 Canon Information Systems Research Australia Pty Ltd Canon Kabushiki Kaisha Patent Attorneys for the Applicant SPRUSON FERGUSON (RT022)(259582)(CFPO2O2AU) I II I