US20230281909A1 - Method for Editing Computer-Generated Images to Maintain Alignment Between Objects Specified in Frame Space and Objects Specified in Scene Space - Google Patents
Method for Editing Computer-Generated Images to Maintain Alignment Between Objects Specified in Frame Space and Objects Specified in Scene Space Download PDFInfo
- Publication number
- US20230281909A1 US20230281909A1 US17/856,344 US202217856344A US2023281909A1 US 20230281909 A1 US20230281909 A1 US 20230281909A1 US 202217856344 A US202217856344 A US 202217856344A US 2023281909 A1 US2023281909 A1 US 2023281909A1
- Authority
- US
- United States
- Prior art keywords
- frame
- image
- location
- image insertion
- computer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 238000000034 method Methods 0.000 title claims description 44
- 238000003780 insertion Methods 0.000 claims abstract description 68
- 230000037431 insertion Effects 0.000 claims abstract description 68
- 238000012545 processing Methods 0.000 claims abstract description 24
- 230000008859 change Effects 0.000 claims abstract description 5
- 239000013598 vector Substances 0.000 claims description 31
- 238000009877 rendering Methods 0.000 claims description 21
- 230000008569 process Effects 0.000 claims description 18
- 238000006073 displacement reaction Methods 0.000 claims 1
- 238000013507 mapping Methods 0.000 abstract description 4
- 230000033001 locomotion Effects 0.000 description 41
- 230000009471 action Effects 0.000 description 39
- 238000004891 communication Methods 0.000 description 15
- 230000000007 visual effect Effects 0.000 description 11
- 230000005540 biological transmission Effects 0.000 description 7
- 238000013459 approach Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 5
- 239000003973 paint Substances 0.000 description 4
- 238000003384 imaging method Methods 0.000 description 3
- 210000001503 joint Anatomy 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000003068 static effect Effects 0.000 description 3
- 238000004590 computer program Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 125000001475 halogen functional group Chemical group 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- RYGMFSIKBFXOCR-UHFFFAOYSA-N Copper Chemical compound [Cu] RYGMFSIKBFXOCR-UHFFFAOYSA-N 0.000 description 1
- 241001465754 Metazoa Species 0.000 description 1
- 206010042008 Stereotypy Diseases 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000005452 bending Methods 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 210000002310 elbow joint Anatomy 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000003278 mimic effect Effects 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 230000001151 other effect Effects 0.000 description 1
- 238000010422 painting Methods 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000010008 shearing Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/04—Texture mapping
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/60—Editing figures and text; Combining figures or text
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/10—Geometric effects
- G06T15/20—Perspective computation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
- G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20092—Interactive image processing based on input by user
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
Definitions
- the present disclosure generally relates to computer graphics processing, and more particularly to editing computer-generated images via editing in frame space and representations of image insertions in scene space.
- a computer imaging system might generate a computer-generated image by taking in an input description of objects positioned in a coordinate system of a virtual scene space, taking on a definition of a virtual camera position and a virtual camera view frame, and then generating one or more images of what the camera “sees” from the virtual camera position.
- the virtual scene space might be a three-dimensional (3D) space.
- a scene specification might be an input to the computer imaging system, wherein a scene specification specifies objects to appear in a virtual scene, object positions in the virtual scene space, lighting and lighting positions in the virtual scene space.
- the computer imaging system might use a renderer to render frame images of what is seen in the scene space.
- An output might be a two-dimensional (2D) pixel array, a deep pixel array (having pixel values for multiple depths), and/or two 2D images for stereoscopic imagery.
- the image or images depicted in the output would be in frame space, which is determined in part by the virtual camera location and the position in the scene space of the virtual camera view frame.
- an artist a user who edits images, animation, etc. or the like
- Rendering can be a compute-intensive operation, especially when multiple frames are needed, such as the rendering of a video sequence.
- some post-render edits might be composited in frame space onto a frame output and for consistency, might require edits to many frames of a video sequence. This can be a tedious process, in particular, if the number of images ranges in the hundreds or thousands. Therefore, improvements are needed to streamline the editing process for inserting an image object in computer-generated images.
- FIG. 1 illustrates a process of rendering from a virtual scene space description to form an image frame.
- FIG. 2 illustrates an example of editing in a frame space.
- FIG. 3 illustrates a system for editing in frame space and generating correspondence vectors for use in other frames.
- FIG. 4 illustrates a video sequence showing a series of generated images or frames that show various objects and a specific character.
- FIG. 5 illustrates a video sequence showing a series of generated images or frames that include various objects and a specific character having an inserted object image.
- FIG. 6 illustrates a flowchart for editing computer-generated images via correspondence vector approach, according to an embodiment.
- FIG. 7 illustrates an association between pixel of a rendered image and their origin points on objects of a scene.
- FIG. 8 illustrates point association that provides a mapping or connection between reference coordinates in the scene space of an object being attached to and the coordinates of a corresponding point on an image insertion, according to various embodiments.
- FIG. 9 illustrates a flowchart for processing computer-generated images via correspondence vector approach, according to an embodiment.
- FIG. 10 illustrates an example visual content generation system as might be used to generate imagery in the form of still images and/or video sequences of images, according to various embodiments.
- FIG. 11 is a block diagram illustrating an example computer system upon which computer systems of the systems illustrated in FIGS. 3 and 10 may be implemented.
- Computer-generated imagery might comprise images or sequences of images or frames having a time component that form a video sequence.
- An animation processing system might take in video frames captured from live action and alter them to produce a resulting video that is different than what was actually present in the live action scene.
- one or more images or frames in the video sequence are edited to include an object image, for example, attached to, or relative to, a specific character or object that appears in those images or frames.
- those images or frames are processed to identify the pixels and/or coordinates of the specific character, along with the character's movements in those images or frames.
- the object image to be inserted (e.g., attached to or put in relative to the specific character) can be placed so that the coordinates associated the object image and the coordinates of the specific character can be associated in the screen space.
- the coordinates of the object image can be tracked relative to the coordinates of the specific character. In doing so, as the specific character moves within the images or frames, the inserted object image can be tracked relative and be placed automatically.
- a computer graphics system might have a user interface that a user (such as an artist creating or modifying imagery) uses to view a frame of a video sequence, an input frame.
- the artist might specify edits to the contents of the input frame, such as placement of another object that covers an object present in the scene represented in the input frame.
- the artist can work in frame space and provide for an image insertion.
- the image insertion could appear as something added to a portion of the input frame but what the computer graphics system is to treat as something added to an object depicted in the input frame so that the image insertion in subsequent frames appears as having been attached to that object.
- the artist can paint an image insertion onto the input frame and the computer graphics system determines an attachment object set, of the one or more objects that the image insertion will remain connected to as those one or more objects move over a sequence of frames.
- a texture can be painted on an image frame of a person such that the texture is a pattern that is applied to an arm of the person and as the person and the arm move in the video, the pattern follows the arm, rotating, translating, deforming, etc. so as to approximately appear as being attached to the arm in the scene space even though it is not necessarily included in a rendering process.
- the computer graphics system maintains a point association for the image insertion, which can be stored as a data structure.
- the point association provides a mapping or connection between reference coordinates in the scene space of the object being attached to and the coordinates of the corresponding point on the image insertion.
- the reference coordinates of the object maintain a relative position to the object as the object moves around a scene (and/or a camera moves in the scene) and thus might represent motion vectors relative to a scene coordinate system.
- a correspondence vector which also can be stored as data, represents correspondence between a reference point position in the input frame and the same point in a second frame.
- the movements of the specific character and the inserted object image can be tracked from one point of view to another point of view, for example, from one perspective of camera position to another perspective position.
- the correspondence between the coordinates of the specific character and the inserted object image can be considered as a motion vector that relates relative positions of the specific character and the inserted object image.
- any other images or frames within the video sequence can be retroactively implemented to insert the object image onto or relative to, the specific character, object, etc.
- an object image can be inserted or placed relative to a specific character once in reference space, and all other instances of images or frames in which the specific character appears can be automatically traced, tracked and inserted, in both two-dimensional images or three-dimensional renderings in the video sequence.
- the following figures and disclosures are examples included to illustrate the method of editing computer-generated images via editing in screen space and representations of image insertions in reference space.
- FIG. 1 illustrates a process of rendering from a virtual scene space description to form an image frame.
- a scene 102 might be described by data stored as a scene description file 104 , which can be passed to a renderer 106 to render image 108 .
- the scene description file 104 might include a specification of a position of a virtual camera 120 and a position and extent of a camera view frame 122 having a border that would coincide with edges of the image 108 .
- Objects 130 , 132 , 136 , and 138 are shown having positions in scene space and are shown in image 108 in frame space.
- Some computer graphics editing systems might include an ability for a user, such as an artist, to manipulate objects in the scene space, but often there is a desire to edit in the frame space, as in painting onto image 108 , but have the image insertions that are painted on be “attached” to one or more objects in the scene so that in subsequent or prior frames in a video sequence, the image insertions can be composited or added onto each frame without requiring the artist to paint every frame in which the image insertion is to appear.
- FIG. 2 illustrates an example of editing in a frame space.
- a frame N 202 might be rendered by a renderer from a scene description file.
- the scene description file would define three objects—circle 204 , square 206 , and triangle 208 —as having locations in the scene space such that they are visible within the camera view frame and thus appear on frame N 202 .
- Frame N 202 might be one of several frames of a video sequence.
- the example objects in FIG. 2 appear as 2D objects, but it should be understood that the teachings here apply as well to 3D objects.
- an artist might add an image insertion—in this example, a star 222 —to frame N 202 to form an edited frame N 220 .
- the image insertion might be a tattoo, a label, a texture, etc. that is intended to remain in relative position to some object over the course of a video sequence.
- the video sequence might be a small part of a larger video sequence (e.g., part of a longer video might have the image insertion while another part does not, or has some other image insertion) and while in this example, the frame space location of the image insertion of star 222 (e.g., the pixels of a 2D pixel array occupied by the image insertion) overlaps with the frame space location of square 206 , this need not be the case.
- references 204 (N), 206 (N), and 208 (N) designate the depicted objects in positions they have in frame N.
- Frame M the objects might appear at different positions in the frame.
- Frame M might be a frame in the video sequence immediately subsequent to Frame N, immediately prior, or some number of frames distant from Frame N, either subsequent or prior. In some editing systems, the number of intervening frames might be limited for conserving computational efforts or other reasons.
- Frame M 230 depicts circle 204 (M), square 206 (M), and triangle 208 (M) in positions for Frame M. Circle 204 (M) has moved relative to circle 204 (N), as represented by a motion vector 232 , while square 206 (M) has moved relative to square 206 (M) as represented by a motion vector 234 , and triangle 208 (M) has not moved.
- Frame 240 depicts Frame M with the edits made to Frame N, including star 222 . This might be the desired result if star 222 is not to be connected to square 206 . However, it is often the case that the artist is designating star 222 as being applied to, attached to, connected to, or similarly associated with an attachment object set that comprises square 206 . In that case, the result would be frame 250 , wherein star 222 moves according to motion vector 234 .
- each object and the image insertion appear as rigid 2D objects that are simply translated.
- the computer graphics editing system can also handle more complex cases, such as rotation, shearing, occluding, bending, etc. of the objects. In part, this can be addressed using a point association data structure that associates multiple points on an object with corresponding points on an image insertion.
- FIG. 3 illustrates a system 300 for editing in frame space and generating correspondence vectors for use in other frames.
- a frame N 302 might be provided as an input frame and a frame M 304 might be provided as a second frame. These frames might be generated by a renderer 306 .
- An editing system 308 might take as input frame N 302 and an artist 310 might use the editing system 308 to edit frame N to create image insertions and specify objects in frame N to associate the image insertions with.
- Editing system 308 might output an image insertions dataset 310 that could include correspondence vectors.
- Image insertions dataset 310 could be an input to a compositor 314 .
- a motion detector 316 detects motion between frame N and frame M. Where the entirety of content, or relevant portions of content, in frames N and M are computer generated, motion detector 316 can determine motion by simply examining scene description data that indicates how objects are to move between frames. Motion detector 316 outputs motion vector data 320 that is provided to compositor 314 . Compositor 314 can then use these inputs to generate a frame M 330 that comprises objects from frame M and image insertions associated with one or more object in frame M where the image insertions track movements of points corresponding to those objects.
- FIG. 4 illustrates a video sequence 400 showing a series of generated images or frames (also referred to herein as “image frame”) that show various objects and a specific character.
- the video sequence 400 includes an image frame 410 showing a specific character 412 and objects 414 , 416 , and 418 in a captured scene.
- the video sequence 400 also includes another image frame 420 , which appears at a specific time or a specific number of image frames after the image frame 410 .
- the position of the specific character 412 has changed with respect to the original position captured in the image frame 410 , indicating a movement of the specific character 412 with respect to others, e.g., objects 414 , 416 , and 418 .
- the video sequence 400 can include many more captured scenes showing the specific character 412 in other image frames within the video sequence 400 .
- FIG. 5 illustrates a video sequence 500 showing a series of generated images or frames that include various objects and a specific character having an inserted object image.
- the video sequence 500 includes an image frame 510 showing the specific character 412 and objects 414 , 416 , and 418 in the captured scene of the image frame 410 with an object image 512 inserted or placed relative to the specific character 412 .
- object image 512 Prior to placing object image 512 on or relative to the specific character 412 , coordinates of the specific character 412 are determined for all the image frames in which the specific character 412 appears to obtain the reference coordinates of the specific character 412 .
- relative coordinates of the specific character 412 and object image 512 are determined to obtain a correspondence vector to track movements of the specific character 412 in order to place or insert object image 512 , as illustrated in image frame 520 of the video sequence 500 .
- the image frame 520 occurs at a specific time or a specific number of image frames after the image frame 510 .
- the position of the specific character 412 has changed with respect to the original position captured in the image frame 510 , indicating a movement of the specific character 412 with respect to others, e.g., objects 414 , 416 , and 418 .
- object image 512 can be placed onto or relative to the specific character 412 in the image frame 520 . That is, object image 512 is placed in the image frame 520 without manual insertion, and by simple using correspondence vector by tracking motion of the specific character 412 .
- object image 512 in the image frame 510 appears drastically different from object image 512 illustrated in the image frame 520 to indicate that the three-dimensional rendering that takes into account different perspectives depicted in the two different scenes, or camera positions.
- object image 512 appears on the upper left chest of the specific character 412 is maintained in both image frames 510 and 520 , although the positions and coordinates of object image 512 with respect to objects 414 , 416 , and 418 have changed in the two image frames.
- the video sequence 500 can include many more captured scenes showing the specific character 412 having object image 512 attached thereto, or placed relative to, in any other image frames within the video sequence 500 .
- FIG. 6 illustrates a flowchart for a computer-implemented method 600 for editing computer-generated images via correspondence vector approach.
- one or more computer systems is used for implementing the method of editing or processing image data.
- the method 600 includes, at 602 , obtaining generated imagery comprising a video sequence or a plurality of image frames representing objects of a geometric model positioned in a virtual scene space.
- the movement of the objects in the virtual scene space is tracked.
- the movements and positions of the generated imagery are tracked from a perspective of a camera position in the virtual scene space to other perspectives of camera position.
- the method 600 includes obtaining a selection of an input frame from among the plurality of frames (i.e., image frames), for example, in a video sequence, such as video sequences 400 or 500 as described with respect to FIGS. 4 and 5 .
- the method 600 includes, at 606 , obtaining an image insertion for inclusion onto, or relative to, the input frame, such as, for example, identifying the image frames where the specific character 412 , object, etc., appears in the video sequences 400 or 500 , as described with respect to FIGS. 4 and 5 .
- the method 600 further includes determining an attachment object set, wherein the attachment object set comprises one or more objects of the geometric model that the image insertion is to track.
- the method 600 includes determining a set of point associations, at 610 .
- a first point association in the set of point associations can be designated as a first data structure.
- the first data structure represents reference coordinates of a first point on a first object of the attachment object set.
- the reference coordinate is a set of coordinates independent of at least one position change, in the virtual scene space, of the first object.
- the first data structure also represents a second point on the image insertion, and a position, in the input frame, of the second point relative to the first point.
- the method 600 includes determining a second frame from among the plurality of frames (e.g., of the video sequences 400 or 500 ) distinct from the input frame.
- the first frame and second frame can be adjacent frames or can be separated by a specific amount of time or image frames, such as in the video sequences 400 or 500 .
- the method 600 further includes, at 614 , determining a correspondence vector, for the point association, as between the input frame and the second frame.
- the correspondence vector represents changes in position in frame space of the first point from the input frame to the second frame.
- the correspondence vector can be used to track the motion of the inserted image and adapted to any other frames within the plurality of frames.
- the method 600 can optionally include, at 616 , using the correspondence vector to define a region in the second frame for further editing.
- FIG. 7 illustrates an association between pixel of a rendered image and their origin points on objects of a scene.
- a renderer 708 might process scene geometry 702 that includes circle 704 ( 1 ), rectangle 704 ( 2 ), and triangle 704 ( 3 ), with rectangle 704 ( 2 ) having a feature 706 that has a location specified in a reference frame of rectangle 704 ( 2 ).
- An output of renderer 708 might be a rendered pixel array 712 , wherein pixels of rendered pixel array 712 are represented by pixel data structures.
- various pixels in rendered pixel array 712 are assigned values corresponding to objects from scene geometry 702 , such as circle 714 ( 1 ), rectangle 714 ( 2 ), and triangle 714 ( 3 ).
- pixel 720 data structure might comprise a pixel color value, an optional object ID that would reference rectangle 714 ( 2 ), a corresponding location, L(x,y), representing a location on the object that is rendered at that pixel (e.g., rectangle 714 ( 2 )), and possibly other pixel-specific data.
- the pixel position of a first pixel in a first image or frame that has a pixel data structure indicating that the first pixel is a render of the location on the scene geometry and the pixel position of a second pixel in a second image or frame that has a pixel data structure indicating that the second pixel is also a render of the location on the scene geometry can form a relative motion vector.
- an image processor can infer a motion vector from relative positions of those pixels.
- data elements of a neighborhood of pixels might be considered.
- FIG. 8 illustrates point association that provides a mapping or connection between reference coordinates (points) in the scene space of an object being attached to and the coordinates of a corresponding reference point on an image insertion, according to various embodiments.
- the object in frame 810 includes reference coordinates 812 , 814 , 816 , and 818 in the scene space and the image insertion in frame 820 includes corresponding reference points 822 , 824 , 826 , and 828 .
- an image insertion can be processed automatically to other frames by generating vectors representing movements of reference points, which can then be used to move additional objects in synchronization with other objects in the frame.
- the process to implement point association can include an artist first selecting an input frame, e.g., frame 810 , and inputting an image insertion, e.g., frame 820 onto, or in relative to the input frame 810 .
- the processor can then determine which object in the input frame 810 is the attach object and to which the image insertion is to be attached to.
- the processor determines a set of point association, wherein a point association is a data structure representing a first point on the attach object, by reference coordinates of the first point, and a second point on the image insertion.
- the process determines a motion vector in between two or more frames, e.g., frames 810 and 820 .
- the motion vector represents changes of the reference coordinates between the two frames, which the process applies to other frames. This is further illustrated by table 830 in FIG. 8 which shows one-to-one correspondence between the reference coordinates 812 , 814 , 816 , and 818 of the scene space and their corresponding reference points 822 , 824 , 826 , and 828 in the image insertion.
- FIG. 9 illustrates a flowchart 900 for processing computer-generated images via correspondence vector approach, according to an embodiment.
- an artist is presented with a first rendered image and in step 904 , and image editing system obtains artist input.
- the image editing system maps the artist input to pixels of the first rendered image. This might be done by noting which pixels of the first rendered image the artist selected for editing. Editing might be for the insertion of an overlay object, such as a tattoo, or other overlay, or for determining motion effects or the like.
- the image editing system reads location values from the data structures for those pixels, wherein the location values indicate locations on objects in the scene used to render the first rendered image with the locations being relative to an object reference frame.
- the image editing system might check a second rendered image for pixel having those same locations or nearby locations thus indicating pixels in the second rendered image that might be pixels in corresponding to an object rendered in the first rendered image. From those correspondences, the image editing system can define, at step 912 , motion vectors wherein a motion vector represents a matching of a pixel in the first rendered image and a pixel in the second rendered image based on the fact that those to pixels were rendered from the same or a neighboring location on an object in the scene geometry.
- the image editing system can store those motion vectors for subsequent operations.
- An example of subsequent operations might include adding motion effects, aligning an overlay from image to image or frame to frame, or other operations.
- FIG. 10 illustrates the example visual content generation system 1000 as might be used to generate imagery in the form of still images and/or video sequences of images.
- the visual content generation system 1000 might generate imagery of live action scenes, computer generated scenes, or a combination thereof.
- users are provided with tools that allow them to specify, at high levels and low levels where necessary, what is to go into that imagery.
- a user might be an animation artist and might use the visual content generation system 1000 to capture interaction between two human actors performing live on a sound stage and replace one of the human actors with a computer-generated anthropomorphic non-human being that behaves in ways that mimic the replaced human actor's movements and mannerisms, and then add in a third computer-generated character and background scene elements that are computer-generated, all in order to tell a desired story or generate desired imagery.
- Still images that are output by the visual content generation system 1000 might be represented in computer memory as pixel arrays, such as a two-dimensional array of pixel color values, each associated with a pixel having a position in a two-dimensional image array.
- Pixel color values might be represented by three or more (or fewer) color values per pixel, such as a red value, a green value, and a blue value (e.g., in RGB format).
- Dimensions of such a two-dimensional array of pixel color values might correspond to a preferred and/or standard display scheme, such as 1920-pixel columns by 1280-pixel rows.
- Images might or might not be stored in a compressed format, but either way, a desired image may be represented as a two-dimensional array of pixel color values.
- images are represented by a pair of stereo images for three-dimensional presentations and in other variations, some of the image output, or all of it, might represent three-dimensional imagery instead of just two-dimensional views.
- a stored video sequence might include a plurality of images such as the still images described above, but where each image of the plurality of images has a place in a timing sequence and the stored video sequence is arranged so that when each image is displayed in order, at a time indicated by the timing sequence, the display presents what appears to be moving and/or changing imagery.
- each image of the plurality of images is a video frame having a specified frame number that corresponds to an amount of time that would elapse from when a video sequence begins playing until that specified frame is displayed.
- a frame rate might be used to describe how many frames of the stored video sequence are displayed per unit time.
- Example video sequences might include 24 frames per second (24 FPS), 50 FPS, 140 FPS, or other frame rates.
- frames are interlaced or otherwise presented for display, but for clarity of description, in some examples, it is assumed that a video frame has one specified display time, but other variations might be contemplated.
- One method of creating a video sequence is to simply use a video camera to record a live action scene, i.e., events that physically occur and can be recorded by a video camera.
- the events being recorded can be events to be interpreted as viewed (such as seeing two human actors talk to each other) and/or can include events to be interpreted differently due to clever camera operations (such as moving actors about a stage to make one appear larger than the other despite the actors actually being of similar build, or using miniature objects with other miniature objects so as to be interpreted as a scene containing life-sized objects).
- Creating video sequences for story-telling or other purposes often calls for scenes that cannot be created with live actors, such as a talking tree, an anthropomorphic object, space battles, and the like. Such video sequences might be generated computationally rather than capturing light from live scenes. In some instances, an entirety of a video sequence might be generated computationally, as in the case of a computer-animated feature film. In some video sequences, it is desirable to have some computer-generated imagery and some live action, perhaps with some careful merging of the two.
- While computer-generated imagery might be creatable by manually specifying each color value for each pixel in each frame, this is likely too tedious to be practical.
- a creator uses various tools to specify the imagery at a higher level.
- an artist might specify the positions in a scene space, such as a three-dimensional coordinate system, of objects and/or lighting, as well as a camera viewpoint, and a camera view plane. From that, a rendering engine could take all of those as inputs, and compute each of the pixel color values in each of the frames.
- an artist specifies position and movement of an articulated object having some specified texture rather than specifying the color of each pixel representing that articulated object in each frame.
- a rendering engine performs ray tracing wherein a pixel color value is determined by computing which objects lie along a ray traced in the scene space from the camera viewpoint through a point or portion of the camera view plane that corresponds to that pixel.
- a camera view plane might be represented as a rectangle having a position in the scene space that is divided into a grid corresponding to the pixels of the ultimate image to be generated, and if a ray defined by the camera viewpoint in the scene space and a given pixel in that grid first intersects a solid, opaque, blue object, that given pixel is assigned the color blue.
- a live action capture system 1002 captures a live scene that plays out on a stage 1004 .
- the live action capture system 1002 is described herein in greater detail, but might include computer processing capabilities, image processing capabilities, one or more processors, program code storage for storing program instructions executable by the one or more processors, as well as user input devices and user output devices, not all of which are shown.
- cameras 1006 ( 1 ) and 1006 ( 2 ) capture the scene, while in some systems, there might be other sensor(s) 1008 that capture information from the live scene (e.g., infrared cameras, infrared sensors, motion capture (“mo-cap”) detectors, etc.).
- sensor(s) 1008 that capture information from the live scene (e.g., infrared cameras, infrared sensors, motion capture (“mo-cap”) detectors, etc.).
- sensors 1008 that capture information from the live scene
- sensors 1008 e.g., infrared cameras, infrared sensors, motion capture (“mo-cap”) detectors, etc.
- On the stage 1004 there might be human actors, animal actors, inanimate objects, background objects, and possibly an object such as a green screen 1010 that is designed to be captured in a live scene recording in such a way that it is easily overlaid with computer-generated imagery.
- the stage 1004 might also contain objects that serve as fiducials, such as fiducials 1012 ( 1 )-( 3 ), that might be used post-capture to determine where an object was during capture.
- a live action scene might be illuminated by one or more lights, such as an overhead light 1014 .
- the live action capture system 1002 might output live action footage to a live action footage storage 1020 .
- a live action processing system 1022 might process live action footage to generate data about that live action footage and store that data into a live action metadata storage 1024 .
- the live action processing system 1022 might include computer processing capabilities, image processing capabilities, one or more processors, program code storage for storing program instructions executable by the one or more processors, as well as user input devices and user output devices, not all of which are shown.
- the live action processing system 1022 might process live action footage to determine boundaries of objects in a frame or multiple frames, determine locations of objects in a live action scene, where a camera was relative to some action, distances between moving objects and fiducials, etc.
- the metadata might include location, color, and intensity of the overhead light 1014 , as that might be useful in post-processing to match computer-generated lighting on objects that are computer-generated and overlaid on the live action footage.
- the live action processing system 1022 might operate autonomously, perhaps based on predetermined program instructions, to generate and output the live action metadata upon receiving and inputting the live action footage.
- the live action footage can be camera-captured data as well as data from other sensors.
- An animation creation system 1030 is another part of the visual content generation system 1000 .
- the animation creation system 1030 might include computer processing capabilities, image processing capabilities, one or more processors, program code storage for storing program instructions executable by the one or more processors, as well as user input devices and user output devices, not all of which are shown.
- the animation creation system 1030 might be used by animation artists, managers, and others to specify details, perhaps programmatically and/or interactively, of imagery to be generated.
- the animation creation system 1030 might generate and output data representing objects (e.g., a horse, a human, a ball, a teapot, a cloud, a light source, a texture, etc.) to an object storage 1034 , generate and output data representing a scene into a scene description storage 1036 , and/or generate and output data representing animation sequences to an animation sequence storage 1038 .
- objects e.g., a horse, a human, a ball, a teapot, a cloud, a light source, a texture, etc.
- Scene data might indicate locations of objects and other visual elements, values of their parameters, lighting, camera location, camera view plane, and other details that a rendering engine 1050 might use to render CGI imagery.
- scene data might include the locations of several articulated characters, background objects, lighting, etc. specified in a two-dimensional space, three-dimensional space, or other dimensional space (such as a 2.5-dimensional space, three-quarter dimensions, pseudo-3D spaces, etc.) along with locations of a camera viewpoint and view place from which to render imagery.
- scene data might indicate that there is to be a red, fuzzy, talking dog in the right half of a video and a stationary tree in the left half of the video, all illuminated by a bright point light source that is above and behind the camera viewpoint.
- the camera viewpoint is not explicit, but can be determined from a viewing frustum.
- the frustum would be a truncated pyramid.
- Other shapes for a rendered view are possible and the camera view plane could be different for different shapes.
- the animation creation system 1030 might be interactive, allowing a user to read in animation sequences, scene descriptions, object details, etc. and edit those, possibly returning them to storage to update or replace existing data.
- an operator might read in objects from object storage into a baking processor that would transform those objects into simpler forms and return those to the object storage 1034 as new or different objects.
- an operator might read in an object that has dozens of specified parameters (movable joints, color options, textures, etc.), select some values for those parameters and then save a baked object that is a simplified object with now fixed values for those parameters.
- data from the data store 1032 might be used to drive object presentation. For example, if an artist is creating an animation of a spaceship passing over the surface of the Earth, instead of manually drawing or specifying a coastline, the artist might specify that the animation creation system 1030 is to read data from the data store 1032 in a file containing coordinates of Earth coastlines and generate background elements of a scene using that coastline data.
- Animation sequence data might be in the form of time series of data for control points of an object that has attributes that are controllable.
- an object might be a humanoid character with limbs and joints that are movable in manners similar to typical human movements.
- An artist can specify an animation sequence at a high level, such as “the left hand moves from location (X1, Y1, Z1) to (X2, Y2, Z2) over time T1 to T2”, at a lower level (e.g., “move the elbow joint 2.5 degrees per frame”) or even at a very high level (e.g., “character A should move, consistent with the laws of physics that are given for this scene, from point P1 to point P2 along a specified path”).
- Animation sequences in an animated scene might be specified by what happens in a live action scene.
- An animation driver generator 1044 might read in live action metadata, such as data representing movements and positions of body parts of a live actor during a live action scene.
- the animation driver generator 1044 might generate corresponding animation parameters to be stored in the animation sequence storage 1038 for use in animating a CGI object. This can be useful where a live action scene of a human actor is captured while wearing mo-cap fiducials (e.g., high-contrast markers outside actor clothing, high-visibility paint on actor skin, face, etc.) and the movement of those fiducials is determined by the live action processing system 1022 .
- the animation driver generator 1044 might convert that movement data into specifications of how joints of an articulated CGI character are to move over time.
- a rendering engine 1050 can read in animation sequences, scene descriptions, and object details, as well as rendering engine control inputs, such as a resolution selection and a set of rendering parameters. Resolution selection might be useful for an operator to control a trade-off between speed of rendering and clarity of detail, as speed might be more important than clarity for a movie maker to test some interaction or direction, while clarity might be more important than speed for a movie maker to generate data that will be used for final prints of feature films to be distributed.
- the rendering engine 1050 might include computer processing capabilities, image processing capabilities, one or more processors, program code storage for storing program instructions executable by the one or more processors, as well as user input devices and user output devices, not all of which are shown.
- the visual content generation system 1000 can also include a merging system 1060 that merges live footage with animated content.
- the live footage might be obtained and input by reading from the live action footage storage 1020 to obtain live action footage, by reading from the live action metadata storage 1024 to obtain details such as presumed segmentation in captured images segmenting objects in a live action scene from their background (perhaps aided by the fact that the green screen 1010 was part of the live action scene), and by obtaining CGI imagery from the rendering engine 1050 .
- a merging system 1060 might also read data from rulesets for merging/combining storage 1062 .
- a very simple example of a rule in a ruleset might be “obtain a full image including a two-dimensional pixel array from live footage, obtain a full image including a two-dimensional pixel array from the rendering engine 1050 , and output an image where each pixel is a corresponding pixel from the rendering engine 1050 when the corresponding pixel in the live footage is a specific color of green, otherwise output a pixel value from the corresponding pixel in the live footage.”
- the merging system 1060 might include computer processing capabilities, image processing capabilities, one or more processors, program code storage for storing program instructions executable by the one or more processors, as well as user input devices and user output devices, not all of which are shown.
- the merging system 1060 might operate autonomously, following programming instructions, or might have a user interface or programmatic interface over which an operator can control a merging process.
- an operator can specify parameter values to use in a merging process and/or might specify specific tweaks to be made to an output of the merging system 1060 , such as modifying boundaries of segmented objects, inserting blurs to smooth out imperfections, or adding other effects.
- the merging system 1060 can output an image to be stored in a static image storage 1070 and/or a sequence of images in the form of video to be stored in an animated/combined video storage 1072 .
- the visual content generation system 1000 can be used to generate video that combines live action with computer-generated animation using various components and tools, some of which are described in more detail herein. While the visual content generation system 1000 might be useful for such combinations, with suitable settings, it can be used for outputting entirely live action footage or entirely CGI sequences.
- the code may also be provided and/or carried by a transitory computer readable medium, e.g., a transmission medium such as in the form of a signal transmitted over a network.
- the techniques described herein are implemented by one or more generalized computing systems programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination.
- Special-purpose computing devices may be used, such as desktop computer systems, portable computer systems, handheld devices, networking devices or any other device that incorporates hard-wired and/or program logic to implement the techniques.
- FIG. 11 is a block diagram that illustrates a computer system 1100 upon which the computer systems of the systems described herein and/or the visual content generation system 1000 (see FIG. 10 ) may be implemented.
- the computer system 1100 includes a bus 1102 or other communication mechanism for communicating information, and a processor 1104 coupled with the bus 1102 for processing information.
- the processor 1104 may be, for example, a general-purpose microprocessor.
- the computer system 1100 also includes a main memory 1106 , such as a random-access memory (RAM) or other dynamic storage device, coupled to the bus 1102 for storing information and instructions to be executed by the processor 1104 .
- the main memory 1106 may also be used for storing temporary variables or other intermediate information during execution of instructions to be executed by the processor 1104 .
- Such instructions when stored in non-transitory storage media accessible to the processor 1104 , render the computer system 1100 into a special-purpose machine that is customized to perform the operations specified in the instructions.
- the computer system 1100 further includes a read only memory (ROM) 1108 or other static storage device coupled to the bus 1102 for storing static information and instructions for the processor 1104 .
- ROM read only memory
- a storage device 1110 such as a magnetic disk or optical disk, is provided and coupled to the bus 1102 for storing information and instructions.
- the computer system 1100 may be coupled via the bus 1102 to a display 1112 , such as a computer monitor, for displaying information to a computer user.
- a display 1112 such as a computer monitor
- An input device 1114 is coupled to the bus 1102 for communicating information and command selections to the processor 1104 .
- a cursor control 1116 is Another type of user input device, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to the processor 1104 and for controlling cursor movement on the display 1112 .
- This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane.
- the computer system 1100 may implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware and/or program logic which in combination with the computer system causes or programs the computer system 1100 to be a special-purpose machine. According to one embodiment, the techniques herein are performed by the computer system 1100 in response to the processor 1104 executing one or more sequences of one or more instructions contained in the main memory 1106 . Such instructions may be read into the main memory 1106 from another storage medium, such as the storage device 1110 . Execution of the sequences of instructions contained in the main memory 1106 causes the processor 1104 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.
- Non-volatile media includes, for example, optical or magnetic disks, such as the storage device 1110 .
- Volatile media includes dynamic memory, such as the main memory 1106 .
- Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, an EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge.
- Storage media is distinct from but may be used in conjunction with transmission media.
- Transmission media participates in transferring information between storage media.
- transmission media includes coaxial cables, copper wire, and fiber optics, including the wires that include the bus 1102 .
- Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.
- the instructions may initially be carried on a magnetic disk or solid-state drive of a remote computer.
- the remote computer can load the instructions into its dynamic memory and send the instructions over a network connection.
- a modem or network interface local to the computer system 1100 can receive the data.
- the bus 1102 carries the data to the main memory 1106 , from which the processor 1104 retrieves and executes the instructions.
- the instructions received by the main memory 1106 may optionally be stored on the storage device 1110 either before or after execution by the processor 1104 .
- the computer system 1100 also includes a communication interface 1118 coupled to the bus 1102 .
- the communication interface 1118 provides a two-way data communication coupling to a network link 1120 that is connected to a local network 1122 .
- the communication interface 1118 may be a network card, a modem, a cable modem, or a satellite modem to provide a data communication connection to a corresponding type of telephone line or communications line.
- Wireless links may also be implemented.
- the communication interface 1118 sends and receives electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information.
- the network link 1120 typically provides data communication through one or more networks to other data devices.
- the network link 1120 may provide a connection through the local network 1122 to a host computer 1124 or to data equipment operated by an Internet Service Provider (ISP) 1126 .
- the ISP 1126 in turn provides data communication services through the world-wide packet data communication network now commonly referred to as the “Internet” 1128 .
- the local network 1122 and Internet 1128 both use electrical, electromagnetic, or optical signals that carry digital data streams.
- the signals through the various networks and the signals on the network link 1120 and through the communication interface 1118 which carry the digital data to and from the computer system 1100 , are example forms of transmission media.
- the computer system 1100 can send messages and receive data, including program code, through the network(s), the network link 1120 , and communication interface 1118 .
- a server 1130 might transmit a requested code for an application program through the Internet 1128 , ISP 1126 , local network 1122 , and communication interface 1118 .
- the received code may be executed by the processor 1104 as it is received, and/or stored in the storage device 1110 , or other non-volatile storage for later execution.
- Processes described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context.
- Processes described herein may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs or one or more applications) executing collectively on one or more processors, by hardware or combinations thereof.
- the code may be stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors.
- the computer-readable storage medium may be non-transitory.
- the code may also be provided carried by a transitory computer readable medium e.g., a transmission medium such as in the form of a signal transmitted over a network.
- the conjunctive phrases “at least one of A, B, and C” and “at least one of A, B and C” refer to any of the following sets: ⁇ A ⁇ , ⁇ B ⁇ , ⁇ C ⁇ , ⁇ A, B ⁇ , ⁇ A, C ⁇ , ⁇ B, C ⁇ , ⁇ A, B, C ⁇ .
- conjunctive language is not generally intended to imply that certain embodiments require at least one of A, at least one of B and at least one of C each to be present.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Computer Graphics (AREA)
- Computing Systems (AREA)
- Geometry (AREA)
- Processing Or Creating Images (AREA)
Abstract
Description
- This application claims priority from and is a continuation of non-provisional U.S. patent application Ser. No. 17/498,668, filed Oct. 11, 2021, now U.S. Pat. No. 11,393,155, which is a continuation of non-provisional U.S. patent application Ser. No. 17/394,020, filed Aug. 4, 2021, and is also a continuation of non-provisional U.S. patent application Ser. No. 17/221,119, filed Apr. 2, 2021, now U.S. patent Ser. No. 11/145,109 and also claims the benefit of U.S. Provisional Application No. 63/087,832 filed on Oct. 5, 2020, the disclosures of which applications are herein incorporated by reference in their entireties.
- The present disclosure generally relates to computer graphics processing, and more particularly to editing computer-generated images via editing in frame space and representations of image insertions in scene space.
- A computer imaging system might generate a computer-generated image by taking in an input description of objects positioned in a coordinate system of a virtual scene space, taking on a definition of a virtual camera position and a virtual camera view frame, and then generating one or more images of what the camera “sees” from the virtual camera position. The virtual scene space might be a three-dimensional (3D) space. A scene specification might be an input to the computer imaging system, wherein a scene specification specifies objects to appear in a virtual scene, object positions in the virtual scene space, lighting and lighting positions in the virtual scene space. The computer imaging system might use a renderer to render frame images of what is seen in the scene space. An output might be a two-dimensional (2D) pixel array, a deep pixel array (having pixel values for multiple depths), and/or two 2D images for stereoscopic imagery. The image or images depicted in the output would be in frame space, which is determined in part by the virtual camera location and the position in the scene space of the virtual camera view frame.
- In some editing steps, an artist (a user who edits images, animation, etc. or the like) might modify a resulting image by editing a scene specification to move objects, add objects, etc., and have the scene rerendered.
- Rendering can be a compute-intensive operation, especially when multiple frames are needed, such as the rendering of a video sequence. As a result, some post-render edits might be composited in frame space onto a frame output and for consistency, might require edits to many frames of a video sequence. This can be a tedious process, in particular, if the number of images ranges in the hundreds or thousands. Therefore, improvements are needed to streamline the editing process for inserting an image object in computer-generated images.
- Various embodiments in accordance with the present disclosure will be described with reference to the drawings, in which:
-
FIG. 1 illustrates a process of rendering from a virtual scene space description to form an image frame. -
FIG. 2 illustrates an example of editing in a frame space. -
FIG. 3 illustrates a system for editing in frame space and generating correspondence vectors for use in other frames. -
FIG. 4 illustrates a video sequence showing a series of generated images or frames that show various objects and a specific character. -
FIG. 5 illustrates a video sequence showing a series of generated images or frames that include various objects and a specific character having an inserted object image. -
FIG. 6 illustrates a flowchart for editing computer-generated images via correspondence vector approach, according to an embodiment. -
FIG. 7 illustrates an association between pixel of a rendered image and their origin points on objects of a scene. -
FIG. 8 illustrates point association that provides a mapping or connection between reference coordinates in the scene space of an object being attached to and the coordinates of a corresponding point on an image insertion, according to various embodiments. -
FIG. 9 illustrates a flowchart for processing computer-generated images via correspondence vector approach, according to an embodiment. -
FIG. 10 illustrates an example visual content generation system as might be used to generate imagery in the form of still images and/or video sequences of images, according to various embodiments. -
FIG. 11 is a block diagram illustrating an example computer system upon which computer systems of the systems illustrated inFIGS. 3 and 10 may be implemented. - In the following description, various embodiments will be described. For purposes of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the embodiments. However, it will also be apparent to one skilled in the art that the embodiments may be practiced without the specific details. Furthermore, well-known features may be omitted or simplified in order not to obscure the embodiment being described.
- Computer-generated imagery might comprise images or sequences of images or frames having a time component that form a video sequence. An animation processing system might take in video frames captured from live action and alter them to produce a resulting video that is different than what was actually present in the live action scene. In one instance, one or more images or frames in the video sequence are edited to include an object image, for example, attached to, or relative to, a specific character or object that appears in those images or frames. To streamline and efficiently edit those images or frames, those images or frames are processed to identify the pixels and/or coordinates of the specific character, along with the character's movements in those images or frames. Once the pixels and/or coordinates of the specific character are identified, the object image to be inserted (e.g., attached to or put in relative to the specific character) can be placed so that the coordinates associated the object image and the coordinates of the specific character can be associated in the screen space. In such association, the coordinates of the object image can be tracked relative to the coordinates of the specific character. In doing so, as the specific character moves within the images or frames, the inserted object image can be tracked relative and be placed automatically.
- A computer graphics system might have a user interface that a user (such as an artist creating or modifying imagery) uses to view a frame of a video sequence, an input frame. The artist might specify edits to the contents of the input frame, such as placement of another object that covers an object present in the scene represented in the input frame. Thus, the artist can work in frame space and provide for an image insertion. In the frame space, the image insertion could appear as something added to a portion of the input frame but what the computer graphics system is to treat as something added to an object depicted in the input frame so that the image insertion in subsequent frames appears as having been attached to that object. More generally, the artist can paint an image insertion onto the input frame and the computer graphics system determines an attachment object set, of the one or more objects that the image insertion will remain connected to as those one or more objects move over a sequence of frames. In a specific example, a texture can be painted on an image frame of a person such that the texture is a pattern that is applied to an arm of the person and as the person and the arm move in the video, the pattern follows the arm, rotating, translating, deforming, etc. so as to approximately appear as being attached to the arm in the scene space even though it is not necessarily included in a rendering process.
- The computer graphics system maintains a point association for the image insertion, which can be stored as a data structure. The point association provides a mapping or connection between reference coordinates in the scene space of the object being attached to and the coordinates of the corresponding point on the image insertion. The reference coordinates of the object maintain a relative position to the object as the object moves around a scene (and/or a camera moves in the scene) and thus might represent motion vectors relative to a scene coordinate system. A correspondence vector, which also can be stored as data, represents correspondence between a reference point position in the input frame and the same point in a second frame.
- In some instances, the movements of the specific character and the inserted object image can be tracked from one point of view to another point of view, for example, from one perspective of camera position to another perspective position. The correspondence between the coordinates of the specific character and the inserted object image can be considered as a motion vector that relates relative positions of the specific character and the inserted object image. In some instances, once a reference image frame is determined to identify the coordinates of the specific character and the inserted object image, any other images or frames within the video sequence can be retroactively implemented to insert the object image onto or relative to, the specific character, object, etc. In doing so, instead of manually editing each image frame within the video sequence in the screen space, which can be repeated hundreds or thousands of times, an object image can be inserted or placed relative to a specific character once in reference space, and all other instances of images or frames in which the specific character appears can be automatically traced, tracked and inserted, in both two-dimensional images or three-dimensional renderings in the video sequence. The following figures and disclosures are examples included to illustrate the method of editing computer-generated images via editing in screen space and representations of image insertions in reference space.
-
FIG. 1 illustrates a process of rendering from a virtual scene space description to form an image frame. As shown there, ascene 102 might be described by data stored as ascene description file 104, which can be passed to arenderer 106 to renderimage 108. Thescene description file 104 might include a specification of a position of avirtual camera 120 and a position and extent of a camera view frame 122 having a border that would coincide with edges of theimage 108.Objects image 108 in frame space. Some computer graphics editing systems might include an ability for a user, such as an artist, to manipulate objects in the scene space, but often there is a desire to edit in the frame space, as in painting ontoimage 108, but have the image insertions that are painted on be “attached” to one or more objects in the scene so that in subsequent or prior frames in a video sequence, the image insertions can be composited or added onto each frame without requiring the artist to paint every frame in which the image insertion is to appear. -
FIG. 2 illustrates an example of editing in a frame space. As shown there, aframe N 202 might be rendered by a renderer from a scene description file. In this example, the scene description file would define three objects—circle 204, square 206, andtriangle 208—as having locations in the scene space such that they are visible within the camera view frame and thus appear onframe N 202.Frame N 202 might be one of several frames of a video sequence. For simplicity of depiction, the example objects inFIG. 2 appear as 2D objects, but it should be understood that the teachings here apply as well to 3D objects. - Using an image editing tool, an artist might add an image insertion—in this example, a
star 222—to frameN 202 to form an editedframe N 220. The image insertion might be a tattoo, a label, a texture, etc. that is intended to remain in relative position to some object over the course of a video sequence. The video sequence might be a small part of a larger video sequence (e.g., part of a longer video might have the image insertion while another part does not, or has some other image insertion) and while in this example, the frame space location of the image insertion of star 222 (e.g., the pixels of a 2D pixel array occupied by the image insertion) overlaps with the frame space location ofsquare 206, this need not be the case. For example, an artist might paint a halo above and to the side of a head of a character in a manner that the halo does not coincide in the frame with any part of the head. InFIG. 2 , the references 204(N), 206(N), and 208(N) designate the depicted objects in positions they have in frame N. - Where the objects are in motion in the video frame, in a second frame—here Frame M—the objects might appear at different positions in the frame. Frame M might be a frame in the video sequence immediately subsequent to Frame N, immediately prior, or some number of frames distant from Frame N, either subsequent or prior. In some editing systems, the number of intervening frames might be limited for conserving computational efforts or other reasons. As illustrated,
Frame M 230 depicts circle 204(M), square 206(M), and triangle 208(M) in positions for Frame M. Circle 204(M) has moved relative to circle 204(N), as represented by amotion vector 232, while square 206(M) has moved relative to square 206(M) as represented by amotion vector 234, and triangle 208(M) has not moved. -
Frame 240 depicts Frame M with the edits made to Frame N, includingstar 222. This might be the desired result ifstar 222 is not to be connected to square 206. However, it is often the case that the artist is designatingstar 222 as being applied to, attached to, connected to, or similarly associated with an attachment object set that comprises square 206. In that case, the result would beframe 250, whereinstar 222 moves according tomotion vector 234. - In the simplified examples of
FIG. 2 , each object and the image insertion appear as rigid 2D objects that are simply translated. As explained herein, the computer graphics editing system can also handle more complex cases, such as rotation, shearing, occluding, bending, etc. of the objects. In part, this can be addressed using a point association data structure that associates multiple points on an object with corresponding points on an image insertion. -
FIG. 3 illustrates asystem 300 for editing in frame space and generating correspondence vectors for use in other frames. As illustrated there, aframe N 302 might be provided as an input frame and aframe M 304 might be provided as a second frame. These frames might be generated by arenderer 306. Anediting system 308 might take asinput frame N 302 and anartist 310 might use theediting system 308 to edit frame N to create image insertions and specify objects in frame N to associate the image insertions with. Editingsystem 308 might output animage insertions dataset 310 that could include correspondence vectors. Image insertions dataset 310 could be an input to acompositor 314. - A
motion detector 316 detects motion between frame N and frame M. Where the entirety of content, or relevant portions of content, in frames N and M are computer generated,motion detector 316 can determine motion by simply examining scene description data that indicates how objects are to move between frames.Motion detector 316 outputsmotion vector data 320 that is provided tocompositor 314.Compositor 314 can then use these inputs to generate aframe M 330 that comprises objects from frame M and image insertions associated with one or more object in frame M where the image insertions track movements of points corresponding to those objects. -
FIG. 4 illustrates avideo sequence 400 showing a series of generated images or frames (also referred to herein as “image frame”) that show various objects and a specific character. Thevideo sequence 400 includes animage frame 410 showing a specific character 412 and objects 414, 416, and 418 in a captured scene. Thevideo sequence 400 also includes anotherimage frame 420, which appears at a specific time or a specific number of image frames after theimage frame 410. In theimage frame 420, the position of the specific character 412 has changed with respect to the original position captured in theimage frame 410, indicating a movement of the specific character 412 with respect to others, e.g., objects 414, 416, and 418. Although not explicitly illustrated inFIG. 4 , it can be understood that thevideo sequence 400 can include many more captured scenes showing the specific character 412 in other image frames within thevideo sequence 400. -
FIG. 5 illustrates avideo sequence 500 showing a series of generated images or frames that include various objects and a specific character having an inserted object image. Thevideo sequence 500 includes animage frame 510 showing the specific character 412 and objects 414, 416, and 418 in the captured scene of theimage frame 410 with anobject image 512 inserted or placed relative to the specific character 412. Prior to placingobject image 512 on or relative to the specific character 412, coordinates of the specific character 412 are determined for all the image frames in which the specific character 412 appears to obtain the reference coordinates of the specific character 412. When placing or insertingobject image 512, relative coordinates of the specific character 412 andobject image 512 are determined to obtain a correspondence vector to track movements of the specific character 412 in order to place or insertobject image 512, as illustrated inimage frame 520 of thevideo sequence 500. - As illustrated in
FIG. 5 , theimage frame 520 occurs at a specific time or a specific number of image frames after theimage frame 510. In theimage frame 520, the position of the specific character 412 has changed with respect to the original position captured in theimage frame 510, indicating a movement of the specific character 412 with respect to others, e.g., objects 414, 416, and 418. Using a correspondence vector approach to reference coordinates from the movement of the specific character 412,object image 512 can be placed onto or relative to the specific character 412 in theimage frame 520. That is,object image 512 is placed in theimage frame 520 without manual insertion, and by simple using correspondence vector by tracking motion of the specific character 412. In addition,object image 512 in theimage frame 510 appears drastically different fromobject image 512 illustrated in theimage frame 520 to indicate that the three-dimensional rendering that takes into account different perspectives depicted in the two different scenes, or camera positions. As illustrated,object image 512 appears on the upper left chest of the specific character 412 is maintained in both image frames 510 and 520, although the positions and coordinates ofobject image 512 with respect to objects 414, 416, and 418 have changed in the two image frames. Although not explicitly illustrated inFIG. 5 , it can be understood that thevideo sequence 500 can include many more captured scenes showing the specific character 412 havingobject image 512 attached thereto, or placed relative to, in any other image frames within thevideo sequence 500. -
FIG. 6 illustrates a flowchart for a computer-implementedmethod 600 for editing computer-generated images via correspondence vector approach. In accordance with the disclosure herein, one or more computer systems is used for implementing the method of editing or processing image data. - As illustrated in
FIG. 6 , themethod 600 includes, at 602, obtaining generated imagery comprising a video sequence or a plurality of image frames representing objects of a geometric model positioned in a virtual scene space. In some instances, the movement of the objects in the virtual scene space is tracked. In some instances, the movements and positions of the generated imagery are tracked from a perspective of a camera position in the virtual scene space to other perspectives of camera position. - At 604, the
method 600 includes obtaining a selection of an input frame from among the plurality of frames (i.e., image frames), for example, in a video sequence, such asvideo sequences FIGS. 4 and 5 . - The
method 600 includes, at 606, obtaining an image insertion for inclusion onto, or relative to, the input frame, such as, for example, identifying the image frames where the specific character 412, object, etc., appears in thevideo sequences FIGS. 4 and 5 . - At 608, the
method 600 further includes determining an attachment object set, wherein the attachment object set comprises one or more objects of the geometric model that the image insertion is to track. - The
method 600, then, includes determining a set of point associations, at 610. In determining the point associations, a first point association in the set of point associations can be designated as a first data structure. For example, the first data structure represents reference coordinates of a first point on a first object of the attachment object set. As described above, the reference coordinate is a set of coordinates independent of at least one position change, in the virtual scene space, of the first object. The first data structure also represents a second point on the image insertion, and a position, in the input frame, of the second point relative to the first point. - At 612, the
method 600 includes determining a second frame from among the plurality of frames (e.g., of thevideo sequences 400 or 500) distinct from the input frame. In some instances, the first frame and second frame can be adjacent frames or can be separated by a specific amount of time or image frames, such as in thevideo sequences - Once the first and second frames are determined, the
method 600 further includes, at 614, determining a correspondence vector, for the point association, as between the input frame and the second frame. In such instances, the correspondence vector represents changes in position in frame space of the first point from the input frame to the second frame. In some instances, the correspondence vector can be used to track the motion of the inserted image and adapted to any other frames within the plurality of frames. - Furthermore, the
method 600 can optionally include, at 616, using the correspondence vector to define a region in the second frame for further editing. -
FIG. 7 illustrates an association between pixel of a rendered image and their origin points on objects of a scene. As illustrated there, arenderer 708 might processscene geometry 702 that includes circle 704(1), rectangle 704(2), and triangle 704(3), with rectangle 704(2) having afeature 706 that has a location specified in a reference frame of rectangle 704(2). An output ofrenderer 708 might be a renderedpixel array 712, wherein pixels of renderedpixel array 712 are represented by pixel data structures. In the example shown, various pixels in renderedpixel array 712 are assigned values corresponding to objects fromscene geometry 702, such as circle 714(1), rectangle 714(2), and triangle 714(3). One such pixel data structure,pixel 720 data structure, is illustrated inFIG. 7 . As illustrated there,pixel 720 data structure might comprise a pixel color value, an optional object ID that would reference rectangle 714(2), a corresponding location, L(x,y), representing a location on the object that is rendered at that pixel (e.g., rectangle 714(2)), and possibly other pixel-specific data. By rendering in this manner and storing at least some indication of a location on an object in the object's reference frame in scene geometry, two images or frames can be processed to identify instances where some location that was on an object in the scene geometry is present in both images or frames. The pixel position of a first pixel in a first image or frame that has a pixel data structure indicating that the first pixel is a render of the location on the scene geometry and the pixel position of a second pixel in a second image or frame that has a pixel data structure indicating that the second pixel is also a render of the location on the scene geometry can form a relative motion vector. For example, if one image contains a pixel that has a data element indicating that the pixel was rendered, at least in part, based on what was at location L(x,y) onobject # 2 in a scene geometry and another image contains a pixel that has a data element indicating that that pixel was rendered, at least in part, based that same location L(x,y), an image processor can infer a motion vector from relative positions of those pixels. In some cases, where there is not a one-to-one correspondence between features of an object and pixels, data elements of a neighborhood of pixels might be considered. -
FIG. 8 illustrates point association that provides a mapping or connection between reference coordinates (points) in the scene space of an object being attached to and the coordinates of a corresponding reference point on an image insertion, according to various embodiments. As shown inFIG. 8 , the object inframe 810 includes reference coordinates 812, 814, 816, and 818 in the scene space and the image insertion inframe 820 includescorresponding reference points - According to one embodiment, the process to implement point association can include an artist first selecting an input frame, e.g.,
frame 810, and inputting an image insertion, e.g.,frame 820 onto, or in relative to theinput frame 810. The processor can then determine which object in theinput frame 810 is the attach object and to which the image insertion is to be attached to. Then, the processor determines a set of point association, wherein a point association is a data structure representing a first point on the attach object, by reference coordinates of the first point, and a second point on the image insertion. The process then determines a motion vector in between two or more frames, e.g., frames 810 and 820. The motion vector represents changes of the reference coordinates between the two frames, which the process applies to other frames. This is further illustrated by table 830 inFIG. 8 which shows one-to-one correspondence between the reference coordinates 812, 814, 816, and 818 of the scene space and theircorresponding reference points -
FIG. 9 illustrates aflowchart 900 for processing computer-generated images via correspondence vector approach, according to an embodiment. In afirst step 902, an artist is presented with a first rendered image and instep 904, and image editing system obtains artist input. Atstep 906, the image editing system maps the artist input to pixels of the first rendered image. This might be done by noting which pixels of the first rendered image the artist selected for editing. Editing might be for the insertion of an overlay object, such as a tattoo, or other overlay, or for determining motion effects or the like. - At
step 908, the image editing system reads location values from the data structures for those pixels, wherein the location values indicate locations on objects in the scene used to render the first rendered image with the locations being relative to an object reference frame. Atstep 910, the image editing system might check a second rendered image for pixel having those same locations or nearby locations thus indicating pixels in the second rendered image that might be pixels in corresponding to an object rendered in the first rendered image. From those correspondences, the image editing system can define, atstep 912, motion vectors wherein a motion vector represents a matching of a pixel in the first rendered image and a pixel in the second rendered image based on the fact that those to pixels were rendered from the same or a neighboring location on an object in the scene geometry. - At
step 914, the image editing system can store those motion vectors for subsequent operations. An example of subsequent operations might include adding motion effects, aligning an overlay from image to image or frame to frame, or other operations. -
FIG. 10 illustrates the example visualcontent generation system 1000 as might be used to generate imagery in the form of still images and/or video sequences of images. The visualcontent generation system 1000 might generate imagery of live action scenes, computer generated scenes, or a combination thereof. In a practical system, users are provided with tools that allow them to specify, at high levels and low levels where necessary, what is to go into that imagery. For example, a user might be an animation artist and might use the visualcontent generation system 1000 to capture interaction between two human actors performing live on a sound stage and replace one of the human actors with a computer-generated anthropomorphic non-human being that behaves in ways that mimic the replaced human actor's movements and mannerisms, and then add in a third computer-generated character and background scene elements that are computer-generated, all in order to tell a desired story or generate desired imagery. - Still images that are output by the visual
content generation system 1000 might be represented in computer memory as pixel arrays, such as a two-dimensional array of pixel color values, each associated with a pixel having a position in a two-dimensional image array. Pixel color values might be represented by three or more (or fewer) color values per pixel, such as a red value, a green value, and a blue value (e.g., in RGB format). Dimensions of such a two-dimensional array of pixel color values might correspond to a preferred and/or standard display scheme, such as 1920-pixel columns by 1280-pixel rows. Images might or might not be stored in a compressed format, but either way, a desired image may be represented as a two-dimensional array of pixel color values. In another variation, images are represented by a pair of stereo images for three-dimensional presentations and in other variations, some of the image output, or all of it, might represent three-dimensional imagery instead of just two-dimensional views. - A stored video sequence might include a plurality of images such as the still images described above, but where each image of the plurality of images has a place in a timing sequence and the stored video sequence is arranged so that when each image is displayed in order, at a time indicated by the timing sequence, the display presents what appears to be moving and/or changing imagery. In one representation, each image of the plurality of images is a video frame having a specified frame number that corresponds to an amount of time that would elapse from when a video sequence begins playing until that specified frame is displayed. A frame rate might be used to describe how many frames of the stored video sequence are displayed per unit time. Example video sequences might include 24 frames per second (24 FPS), 50 FPS, 140 FPS, or other frame rates. In some embodiments, frames are interlaced or otherwise presented for display, but for clarity of description, in some examples, it is assumed that a video frame has one specified display time, but other variations might be contemplated.
- One method of creating a video sequence is to simply use a video camera to record a live action scene, i.e., events that physically occur and can be recorded by a video camera. The events being recorded can be events to be interpreted as viewed (such as seeing two human actors talk to each other) and/or can include events to be interpreted differently due to clever camera operations (such as moving actors about a stage to make one appear larger than the other despite the actors actually being of similar build, or using miniature objects with other miniature objects so as to be interpreted as a scene containing life-sized objects).
- Creating video sequences for story-telling or other purposes often calls for scenes that cannot be created with live actors, such as a talking tree, an anthropomorphic object, space battles, and the like. Such video sequences might be generated computationally rather than capturing light from live scenes. In some instances, an entirety of a video sequence might be generated computationally, as in the case of a computer-animated feature film. In some video sequences, it is desirable to have some computer-generated imagery and some live action, perhaps with some careful merging of the two.
- While computer-generated imagery might be creatable by manually specifying each color value for each pixel in each frame, this is likely too tedious to be practical. As a result, a creator uses various tools to specify the imagery at a higher level. As an example, an artist might specify the positions in a scene space, such as a three-dimensional coordinate system, of objects and/or lighting, as well as a camera viewpoint, and a camera view plane. From that, a rendering engine could take all of those as inputs, and compute each of the pixel color values in each of the frames. In another example, an artist specifies position and movement of an articulated object having some specified texture rather than specifying the color of each pixel representing that articulated object in each frame.
- In a specific example, a rendering engine performs ray tracing wherein a pixel color value is determined by computing which objects lie along a ray traced in the scene space from the camera viewpoint through a point or portion of the camera view plane that corresponds to that pixel. For example, a camera view plane might be represented as a rectangle having a position in the scene space that is divided into a grid corresponding to the pixels of the ultimate image to be generated, and if a ray defined by the camera viewpoint in the scene space and a given pixel in that grid first intersects a solid, opaque, blue object, that given pixel is assigned the color blue. Of course, for modern computer-generated imagery, determining pixel colors—and thereby generating imagery—can be more complicated, as there are lighting issues, reflections, interpolations, and other considerations.
- As illustrated in
FIG. 10 , a liveaction capture system 1002 captures a live scene that plays out on astage 1004. The liveaction capture system 1002 is described herein in greater detail, but might include computer processing capabilities, image processing capabilities, one or more processors, program code storage for storing program instructions executable by the one or more processors, as well as user input devices and user output devices, not all of which are shown. - In a specific live action capture system, cameras 1006(1) and 1006(2) capture the scene, while in some systems, there might be other sensor(s) 1008 that capture information from the live scene (e.g., infrared cameras, infrared sensors, motion capture (“mo-cap”) detectors, etc.). On the
stage 1004, there might be human actors, animal actors, inanimate objects, background objects, and possibly an object such as agreen screen 1010 that is designed to be captured in a live scene recording in such a way that it is easily overlaid with computer-generated imagery. Thestage 1004 might also contain objects that serve as fiducials, such as fiducials 1012(1)-(3), that might be used post-capture to determine where an object was during capture. A live action scene might be illuminated by one or more lights, such as anoverhead light 1014. - During or following the capture of a live action scene, the live
action capture system 1002 might output live action footage to a liveaction footage storage 1020. A liveaction processing system 1022 might process live action footage to generate data about that live action footage and store that data into a liveaction metadata storage 1024. The liveaction processing system 1022 might include computer processing capabilities, image processing capabilities, one or more processors, program code storage for storing program instructions executable by the one or more processors, as well as user input devices and user output devices, not all of which are shown. The liveaction processing system 1022 might process live action footage to determine boundaries of objects in a frame or multiple frames, determine locations of objects in a live action scene, where a camera was relative to some action, distances between moving objects and fiducials, etc. Where elements have sensors attached to them or are detected, the metadata might include location, color, and intensity of theoverhead light 1014, as that might be useful in post-processing to match computer-generated lighting on objects that are computer-generated and overlaid on the live action footage. The liveaction processing system 1022 might operate autonomously, perhaps based on predetermined program instructions, to generate and output the live action metadata upon receiving and inputting the live action footage. The live action footage can be camera-captured data as well as data from other sensors. - An
animation creation system 1030 is another part of the visualcontent generation system 1000. Theanimation creation system 1030 might include computer processing capabilities, image processing capabilities, one or more processors, program code storage for storing program instructions executable by the one or more processors, as well as user input devices and user output devices, not all of which are shown. Theanimation creation system 1030 might be used by animation artists, managers, and others to specify details, perhaps programmatically and/or interactively, of imagery to be generated. From user input and data from a database or other data source, indicated as adata store 1032, theanimation creation system 1030 might generate and output data representing objects (e.g., a horse, a human, a ball, a teapot, a cloud, a light source, a texture, etc.) to anobject storage 1034, generate and output data representing a scene into ascene description storage 1036, and/or generate and output data representing animation sequences to ananimation sequence storage 1038. - Scene data might indicate locations of objects and other visual elements, values of their parameters, lighting, camera location, camera view plane, and other details that a
rendering engine 1050 might use to render CGI imagery. For example, scene data might include the locations of several articulated characters, background objects, lighting, etc. specified in a two-dimensional space, three-dimensional space, or other dimensional space (such as a 2.5-dimensional space, three-quarter dimensions, pseudo-3D spaces, etc.) along with locations of a camera viewpoint and view place from which to render imagery. For example, scene data might indicate that there is to be a red, fuzzy, talking dog in the right half of a video and a stationary tree in the left half of the video, all illuminated by a bright point light source that is above and behind the camera viewpoint. In some cases, the camera viewpoint is not explicit, but can be determined from a viewing frustum. In the case of imagery that is to be rendered to a rectangular view, the frustum would be a truncated pyramid. Other shapes for a rendered view are possible and the camera view plane could be different for different shapes. - The
animation creation system 1030 might be interactive, allowing a user to read in animation sequences, scene descriptions, object details, etc. and edit those, possibly returning them to storage to update or replace existing data. As an example, an operator might read in objects from object storage into a baking processor that would transform those objects into simpler forms and return those to theobject storage 1034 as new or different objects. For example, an operator might read in an object that has dozens of specified parameters (movable joints, color options, textures, etc.), select some values for those parameters and then save a baked object that is a simplified object with now fixed values for those parameters. - Rather than requiring user specification of each detail of a scene, data from the
data store 1032 might be used to drive object presentation. For example, if an artist is creating an animation of a spaceship passing over the surface of the Earth, instead of manually drawing or specifying a coastline, the artist might specify that theanimation creation system 1030 is to read data from thedata store 1032 in a file containing coordinates of Earth coastlines and generate background elements of a scene using that coastline data. - Animation sequence data might be in the form of time series of data for control points of an object that has attributes that are controllable. For example, an object might be a humanoid character with limbs and joints that are movable in manners similar to typical human movements. An artist can specify an animation sequence at a high level, such as “the left hand moves from location (X1, Y1, Z1) to (X2, Y2, Z2) over time T1 to T2”, at a lower level (e.g., “move the elbow joint 2.5 degrees per frame”) or even at a very high level (e.g., “character A should move, consistent with the laws of physics that are given for this scene, from point P1 to point P2 along a specified path”).
- Animation sequences in an animated scene might be specified by what happens in a live action scene. An
animation driver generator 1044 might read in live action metadata, such as data representing movements and positions of body parts of a live actor during a live action scene. Theanimation driver generator 1044 might generate corresponding animation parameters to be stored in theanimation sequence storage 1038 for use in animating a CGI object. This can be useful where a live action scene of a human actor is captured while wearing mo-cap fiducials (e.g., high-contrast markers outside actor clothing, high-visibility paint on actor skin, face, etc.) and the movement of those fiducials is determined by the liveaction processing system 1022. Theanimation driver generator 1044 might convert that movement data into specifications of how joints of an articulated CGI character are to move over time. - A
rendering engine 1050 can read in animation sequences, scene descriptions, and object details, as well as rendering engine control inputs, such as a resolution selection and a set of rendering parameters. Resolution selection might be useful for an operator to control a trade-off between speed of rendering and clarity of detail, as speed might be more important than clarity for a movie maker to test some interaction or direction, while clarity might be more important than speed for a movie maker to generate data that will be used for final prints of feature films to be distributed. Therendering engine 1050 might include computer processing capabilities, image processing capabilities, one or more processors, program code storage for storing program instructions executable by the one or more processors, as well as user input devices and user output devices, not all of which are shown. - The visual
content generation system 1000 can also include amerging system 1060 that merges live footage with animated content. The live footage might be obtained and input by reading from the liveaction footage storage 1020 to obtain live action footage, by reading from the liveaction metadata storage 1024 to obtain details such as presumed segmentation in captured images segmenting objects in a live action scene from their background (perhaps aided by the fact that thegreen screen 1010 was part of the live action scene), and by obtaining CGI imagery from therendering engine 1050. - A
merging system 1060 might also read data from rulesets for merging/combiningstorage 1062. A very simple example of a rule in a ruleset might be “obtain a full image including a two-dimensional pixel array from live footage, obtain a full image including a two-dimensional pixel array from therendering engine 1050, and output an image where each pixel is a corresponding pixel from therendering engine 1050 when the corresponding pixel in the live footage is a specific color of green, otherwise output a pixel value from the corresponding pixel in the live footage.” - The
merging system 1060 might include computer processing capabilities, image processing capabilities, one or more processors, program code storage for storing program instructions executable by the one or more processors, as well as user input devices and user output devices, not all of which are shown. Themerging system 1060 might operate autonomously, following programming instructions, or might have a user interface or programmatic interface over which an operator can control a merging process. In some embodiments, an operator can specify parameter values to use in a merging process and/or might specify specific tweaks to be made to an output of themerging system 1060, such as modifying boundaries of segmented objects, inserting blurs to smooth out imperfections, or adding other effects. Based on its inputs, themerging system 1060 can output an image to be stored in astatic image storage 1070 and/or a sequence of images in the form of video to be stored in an animated/combinedvideo storage 1072. - Thus, as described, the visual
content generation system 1000 can be used to generate video that combines live action with computer-generated animation using various components and tools, some of which are described in more detail herein. While the visualcontent generation system 1000 might be useful for such combinations, with suitable settings, it can be used for outputting entirely live action footage or entirely CGI sequences. The code may also be provided and/or carried by a transitory computer readable medium, e.g., a transmission medium such as in the form of a signal transmitted over a network. - According to one embodiment, the techniques described herein are implemented by one or more generalized computing systems programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination. Special-purpose computing devices may be used, such as desktop computer systems, portable computer systems, handheld devices, networking devices or any other device that incorporates hard-wired and/or program logic to implement the techniques.
- For example,
FIG. 11 is a block diagram that illustrates acomputer system 1100 upon which the computer systems of the systems described herein and/or the visual content generation system 1000 (seeFIG. 10 ) may be implemented. Thecomputer system 1100 includes abus 1102 or other communication mechanism for communicating information, and aprocessor 1104 coupled with thebus 1102 for processing information. Theprocessor 1104 may be, for example, a general-purpose microprocessor. - The
computer system 1100 also includes amain memory 1106, such as a random-access memory (RAM) or other dynamic storage device, coupled to thebus 1102 for storing information and instructions to be executed by theprocessor 1104. Themain memory 1106 may also be used for storing temporary variables or other intermediate information during execution of instructions to be executed by theprocessor 1104. Such instructions, when stored in non-transitory storage media accessible to theprocessor 1104, render thecomputer system 1100 into a special-purpose machine that is customized to perform the operations specified in the instructions. - The
computer system 1100 further includes a read only memory (ROM) 1108 or other static storage device coupled to thebus 1102 for storing static information and instructions for theprocessor 1104. Astorage device 1110, such as a magnetic disk or optical disk, is provided and coupled to thebus 1102 for storing information and instructions. - The
computer system 1100 may be coupled via thebus 1102 to adisplay 1112, such as a computer monitor, for displaying information to a computer user. Aninput device 1114, including alphanumeric and other keys, is coupled to thebus 1102 for communicating information and command selections to theprocessor 1104. Another type of user input device is acursor control 1116, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to theprocessor 1104 and for controlling cursor movement on thedisplay 1112. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane. - The
computer system 1100 may implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware and/or program logic which in combination with the computer system causes or programs thecomputer system 1100 to be a special-purpose machine. According to one embodiment, the techniques herein are performed by thecomputer system 1100 in response to theprocessor 1104 executing one or more sequences of one or more instructions contained in themain memory 1106. Such instructions may be read into themain memory 1106 from another storage medium, such as thestorage device 1110. Execution of the sequences of instructions contained in themain memory 1106 causes theprocessor 1104 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions. - The term “storage media” as used herein refers to any non-transitory media that store data and/or instructions that cause a machine to operation in a specific fashion. Such storage media may include non-volatile media and/or volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as the
storage device 1110. Volatile media includes dynamic memory, such as themain memory 1106. Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, an EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge. - Storage media is distinct from but may be used in conjunction with transmission media. Transmission media participates in transferring information between storage media. For example, transmission media includes coaxial cables, copper wire, and fiber optics, including the wires that include the
bus 1102. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications. - Various forms of media may be involved in carrying one or more sequences of one or more instructions to the
processor 1104 for execution. For example, the instructions may initially be carried on a magnetic disk or solid-state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a network connection. A modem or network interface local to thecomputer system 1100 can receive the data. Thebus 1102 carries the data to themain memory 1106, from which theprocessor 1104 retrieves and executes the instructions. The instructions received by themain memory 1106 may optionally be stored on thestorage device 1110 either before or after execution by theprocessor 1104. - The
computer system 1100 also includes acommunication interface 1118 coupled to thebus 1102. Thecommunication interface 1118 provides a two-way data communication coupling to anetwork link 1120 that is connected to alocal network 1122. For example, thecommunication interface 1118 may be a network card, a modem, a cable modem, or a satellite modem to provide a data communication connection to a corresponding type of telephone line or communications line. Wireless links may also be implemented. In any such implementation, thecommunication interface 1118 sends and receives electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information. - The
network link 1120 typically provides data communication through one or more networks to other data devices. For example, thenetwork link 1120 may provide a connection through thelocal network 1122 to ahost computer 1124 or to data equipment operated by an Internet Service Provider (ISP) 1126. TheISP 1126 in turn provides data communication services through the world-wide packet data communication network now commonly referred to as the “Internet” 1128. Thelocal network 1122 andInternet 1128 both use electrical, electromagnetic, or optical signals that carry digital data streams. The signals through the various networks and the signals on thenetwork link 1120 and through thecommunication interface 1118, which carry the digital data to and from thecomputer system 1100, are example forms of transmission media. - The
computer system 1100 can send messages and receive data, including program code, through the network(s), thenetwork link 1120, andcommunication interface 1118. In the Internet example, aserver 1130 might transmit a requested code for an application program through theInternet 1128,ISP 1126,local network 1122, andcommunication interface 1118. The received code may be executed by theprocessor 1104 as it is received, and/or stored in thestorage device 1110, or other non-volatile storage for later execution. - Operations of processes described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. Processes described herein (or variations and/or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs or one or more applications) executing collectively on one or more processors, by hardware or combinations thereof. The code may be stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable storage medium may be non-transitory. The code may also be provided carried by a transitory computer readable medium e.g., a transmission medium such as in the form of a signal transmitted over a network.
- Conjunctive language, such as phrases of the form “at least one of A, B, and C,” or “at least one of A, B and C,” unless specifically stated otherwise or otherwise clearly contradicted by context, is otherwise understood with the context as used in general to present that an item, term, etc., may be either A or B or C, or any nonempty subset of the set of A and B and C. For instance, in the illustrative example of a set having three members, the conjunctive phrases “at least one of A, B, and C” and “at least one of A, B and C” refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such conjunctive language is not generally intended to imply that certain embodiments require at least one of A, at least one of B and at least one of C each to be present.
- The use of examples, or exemplary language (e.g., “such as”) provided herein, is intended merely to better illuminate embodiments of the invention and does not pose a limitation on the scope of the invention unless otherwise claimed. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the invention.
- In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. The sole and exclusive indicator of the scope of the invention, and what is intended by the applicants to be the scope of the invention, is the literal and equivalent scope of the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction.
- Further embodiments can be envisioned to one of ordinary skill in the art after reading this disclosure. In other embodiments, combinations or sub-combinations of the above-disclosed invention can be advantageously made. The example arrangements of components are shown for purposes of illustration and combinations, additions, re-arrangements, and the like are contemplated in alternative embodiments of the present invention. Thus, while the invention has been described with respect to exemplary embodiments, one skilled in the art will recognize that numerous modifications are possible.
- For example, the processes described herein may be implemented using hardware components, software components, and/or any combination thereof. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. It will, however, be evident that various modifications and changes may be made thereunto without departing from the broader spirit and scope of the invention as set forth in the claims and that the invention is intended to cover all modifications and equivalents within the scope of the following claims.
- All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.
Claims (21)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/856,344 US20230281909A1 (en) | 2020-10-05 | 2022-07-01 | Method for Editing Computer-Generated Images to Maintain Alignment Between Objects Specified in Frame Space and Objects Specified in Scene Space |
Applications Claiming Priority (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US202063087832P | 2020-10-05 | 2020-10-05 | |
US17/221,119 US11145109B1 (en) | 2020-10-05 | 2021-04-02 | Method for editing computer-generated images to maintain alignment between objects specified in frame space and objects specified in scene space |
US17/394,020 US11417048B2 (en) | 2020-10-05 | 2021-08-04 | Computer graphics system user interface for obtaining artist inputs for objects specified in frame space and objects specified in scene space |
US17/498,668 US11393155B2 (en) | 2020-10-05 | 2021-10-11 | Method for editing computer-generated images to maintain alignment between objects specified in frame space and objects specified in scene space |
US17/856,344 US20230281909A1 (en) | 2020-10-05 | 2022-07-01 | Method for Editing Computer-Generated Images to Maintain Alignment Between Objects Specified in Frame Space and Objects Specified in Scene Space |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/394,020 Continuation US11417048B2 (en) | 2020-10-05 | 2021-08-04 | Computer graphics system user interface for obtaining artist inputs for objects specified in frame space and objects specified in scene space |
Publications (1)
Publication Number | Publication Date |
---|---|
US20230281909A1 true US20230281909A1 (en) | 2023-09-07 |
Family
ID=78008016
Family Applications (4)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/221,119 Active US11145109B1 (en) | 2020-10-05 | 2021-04-02 | Method for editing computer-generated images to maintain alignment between objects specified in frame space and objects specified in scene space |
US17/394,020 Active US11417048B2 (en) | 2020-10-05 | 2021-08-04 | Computer graphics system user interface for obtaining artist inputs for objects specified in frame space and objects specified in scene space |
US17/498,668 Active US11393155B2 (en) | 2020-10-05 | 2021-10-11 | Method for editing computer-generated images to maintain alignment between objects specified in frame space and objects specified in scene space |
US17/856,344 Abandoned US20230281909A1 (en) | 2020-10-05 | 2022-07-01 | Method for Editing Computer-Generated Images to Maintain Alignment Between Objects Specified in Frame Space and Objects Specified in Scene Space |
Family Applications Before (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/221,119 Active US11145109B1 (en) | 2020-10-05 | 2021-04-02 | Method for editing computer-generated images to maintain alignment between objects specified in frame space and objects specified in scene space |
US17/394,020 Active US11417048B2 (en) | 2020-10-05 | 2021-08-04 | Computer graphics system user interface for obtaining artist inputs for objects specified in frame space and objects specified in scene space |
US17/498,668 Active US11393155B2 (en) | 2020-10-05 | 2021-10-11 | Method for editing computer-generated images to maintain alignment between objects specified in frame space and objects specified in scene space |
Country Status (2)
Country | Link |
---|---|
US (4) | US11145109B1 (en) |
WO (2) | WO2022075860A1 (en) |
Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20170085867A1 (en) * | 2015-09-17 | 2017-03-23 | Lumii, Inc. | Multi-view displays and associated systems and methods |
Family Cites Families (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP1549041A3 (en) * | 1998-04-10 | 2005-09-21 | Fuji Photo Film Co., Ltd. | Electronic album producing and viewing system and method |
JP3825615B2 (en) * | 2000-08-11 | 2006-09-27 | 株式会社東芝 | Moving picture coding apparatus, moving picture coding method, and medium recording program |
KR20080090218A (en) * | 2007-04-04 | 2008-10-08 | 엔에이치엔(주) | Method for uploading an edited file automatically and apparatus thereof |
WO2010002921A1 (en) * | 2008-07-01 | 2010-01-07 | Yoostar Entertainment Group, Inc. | Interactive systems and methods for video compositing |
US20120018501A1 (en) * | 2009-05-12 | 2012-01-26 | Richard Wilen | Single Ply Multi-Purpose Form With Detachable Components |
US9041717B2 (en) * | 2011-09-12 | 2015-05-26 | Disney Enterprises, Inc. | Techniques for processing image data generated from three-dimensional graphic models |
US20130275120A1 (en) * | 2012-04-11 | 2013-10-17 | Lee Michael DeGross | Process for a Signified Correct Contextual Meaning Sometimes Interspersed with Complementary Related Trivia |
CN102903128B (en) * | 2012-09-07 | 2016-12-21 | 北京航空航天大学 | The video image content editor's transmission method kept based on Similarity of Local Characteristic Structure |
US10304084B2 (en) * | 2016-07-06 | 2019-05-28 | Hiro Media Ltd. | Real-time monitoring of ads inserted in real-time into a web page |
-
2021
- 2021-04-02 US US17/221,119 patent/US11145109B1/en active Active
- 2021-06-15 WO PCT/NZ2021/050093 patent/WO2022075860A1/en active Application Filing
- 2021-08-04 US US17/394,020 patent/US11417048B2/en active Active
- 2021-08-30 WO PCT/NZ2021/050157 patent/WO2022075861A1/en active Application Filing
- 2021-10-11 US US17/498,668 patent/US11393155B2/en active Active
-
2022
- 2022-07-01 US US17/856,344 patent/US20230281909A1/en not_active Abandoned
Patent Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20170085867A1 (en) * | 2015-09-17 | 2017-03-23 | Lumii, Inc. | Multi-view displays and associated systems and methods |
Also Published As
Publication number | Publication date |
---|---|
US20220108515A1 (en) | 2022-04-07 |
WO2022075861A1 (en) | 2022-04-14 |
US20220108516A1 (en) | 2022-04-07 |
US11393155B2 (en) | 2022-07-19 |
WO2022075860A1 (en) | 2022-04-14 |
US11145109B1 (en) | 2021-10-12 |
US11417048B2 (en) | 2022-08-16 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11227428B2 (en) | Modification of a live-action video recording using volumetric scene reconstruction to replace a designated region | |
US11354774B2 (en) | Facial model mapping with a neural network trained on varying levels of detail of facial scans | |
US11228707B1 (en) | Scene capture for reconstruction of obscured views | |
US11676252B2 (en) | Image processing for reducing artifacts caused by removal of scene elements from images | |
US11055911B1 (en) | Method of generating surface definitions usable in computer-generated imagery to include procedurally-generated microdetail | |
US11393155B2 (en) | Method for editing computer-generated images to maintain alignment between objects specified in frame space and objects specified in scene space | |
US11228706B2 (en) | Plate reconstruction of obscured views of a main imaging device using capture device inputs of the same scene | |
US11605171B1 (en) | Method and apparatus for processing reference inputs for video compositing with replacement | |
US11450054B2 (en) | Method for operating a character rig in an image-generation system using constraints on reference nodes | |
US11170553B1 (en) | Methods and systems for generating an animation control rig | |
US20220028147A1 (en) | Operating animation controls using evaluation logic |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: UNITY TECHNOLOGIES SF, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:UNITY SOFTWARE INC.;REEL/FRAME:062381/0488 Effective date: 20211201 Owner name: UNITY SOFTWARE INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:WETA DIGITAL LIMITED;REEL/FRAME:062381/0451 Effective date: 20211201 Owner name: WETA DIGITAL LIMITED, NEW ZEALAND Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:HILLMAN, PETER M;REEL/FRAME:062364/0984 Effective date: 20210929 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |