EP3956810B1 - Generating videos - Google Patents
Generating videos Download PDFInfo
- Publication number
- EP3956810B1 EP3956810B1 EP20740118.3A EP20740118A EP3956810B1 EP 3956810 B1 EP3956810 B1 EP 3956810B1 EP 20740118 A EP20740118 A EP 20740118A EP 3956810 B1 EP3956810 B1 EP 3956810B1
- Authority
- EP
- European Patent Office
- Prior art keywords
- video
- sub
- videos
- target object
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 claims description 47
- 238000012545 processing Methods 0.000 claims description 33
- 230000004044 response Effects 0.000 claims description 20
- 238000003860 storage Methods 0.000 claims description 11
- 238000013507 mapping Methods 0.000 claims description 9
- 230000000007 visual effect Effects 0.000 claims description 9
- 230000000699 topical effect Effects 0.000 description 49
- 238000009826 distribution Methods 0.000 description 40
- 230000008569 process Effects 0.000 description 20
- 230000015572 biosynthetic process Effects 0.000 description 18
- 238000003786 synthesis reaction Methods 0.000 description 18
- 238000004590 computer program Methods 0.000 description 14
- 238000013515 script Methods 0.000 description 8
- 230000005540 biological transmission Effects 0.000 description 7
- 238000004891 communication Methods 0.000 description 7
- 238000001514 detection method Methods 0.000 description 7
- 230000009471 action Effects 0.000 description 6
- 230000003993 interaction Effects 0.000 description 5
- 239000002131 composite material Substances 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 239000011521 glass Substances 0.000 description 4
- 238000010801 machine learning Methods 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 230000001934 delay Effects 0.000 description 2
- 238000012854 evaluation process Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000011524 similarity measure Methods 0.000 description 2
- 241000282326 Felis catus Species 0.000 description 1
- 241001465754 Metazoa Species 0.000 description 1
- 241000009334 Singa Species 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000003796 beauty Effects 0.000 description 1
- 235000013405 beer Nutrition 0.000 description 1
- 235000013361 beverage Nutrition 0.000 description 1
- 230000001149 cognitive effect Effects 0.000 description 1
- 230000009193 crawling Effects 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 235000013305 food Nutrition 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 230000003116 impacting effect Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000004091 panning Methods 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 235000013550 pizza Nutrition 0.000 description 1
- 230000000644 propagated effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 239000000758 substrate Substances 0.000 description 1
- 230000009182 swimming Effects 0.000 description 1
- 230000026676 system process Effects 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
- G11B27/02—Editing, e.g. varying the order of information signals recorded on, or reproduced from, record carriers
- G11B27/031—Electronic editing of digitised analogue information signals, e.g. audio or video signals
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
- G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
- G11B27/19—Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier
- G11B27/28—Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier by using information signals recorded by the same method as the main recording
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/44016—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving splicing one content stream with another content stream, e.g. for substituting a video clip
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/49—Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/435—Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
- H04N21/4355—Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream involving reformatting operations of additional data, e.g. HTML pages on a television screen
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
- H04N21/47205—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content for manipulating displayed content, e.g. interacting with MPEG-4 objects, editing locally
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
Definitions
- This specification relates to generating videos.
- the Internet facilitates the exchange of information between users across the globe.
- Content from multiple different providers, including videos can be integrated into a single electronic document to create a composite document.
- a portion of the content included in the electronic document may be selected (or specified) by a publisher of the electronic document.
- a different portion of content e.g., third party content, including videos
- can be provided by a third party e.g., an entity that is not a publisher of the electronic document
- EP 3249651 A1 describes a method for generating a summary video sequence from a source video sequence.
- a method is provided as defined in claim 1.
- a system is provided as defined in claim 8.
- a computer-readable storage medium is provided as defined in claim 9.
- This specification describes a system implemented as computer programs on one or more computers in one or more locations that can process an input video to extract multiple sub-videos from the input video that each show a respective object of a target object type.
- the system can combine the sub-videos showing the objects of the target object type to generate a "topical" video (e.g., that includes a target object type related to a given topic).
- the topical video may be integrated into a digital component that is transmitted for presentation with electronic documents at user devices, e.g., in blocks alongside search results or on third party websites.
- a method performed by one or more data processing apparatus comprising: receiving: (i) an input video comprising a sequence of video frames, and (ii) data indicating a target object type; processing the input video to generate tracking data that identifies and tracks visual locations of one or more instances of target objects of the target object type in the input video; generating a plurality of sub-videos based on the input video and the tracking data, including: for each sub-video, generating a respective sequence of sub-video frames that are each extracted from a respective video frame of the input video to include a respective instance of a given target object from among the identified target objects of the target object type, wherein at least one of the sub-video frames is cropped from a respective video frame of the input video to include less than all content of the respective video frame; and generating an output video that comprises the plurality of sub-videos.
- processing the input video to generate the tracking data comprises, for each instance of a target object of the target object type in the input video: determining, for each of multiple video frames of the input video, a respective bounding box that encloses the instance of the target object in the video frame.
- generating the respective sequence of sub-video frames comprises: cropping the respective bounding box that encloses the instance of the given target object corresponding to the sub-video from each of the multiple video frames of the input video.
- Generating the respective sequence of sub-video frames comprises: determining, based on the tracking data, a same bounding box that encloses the instance of the given target object corresponding to the sub-video in each of multiple video frames of the input video; and cropping the same bounding box from each of the multiple video frames of the input video.
- a sub-video frame of a first sub-video and a sub-video frame of a second sub-video are each cropped from a same video frame of the input video.
- generating the output video that comprises the plurality of sub-videos comprises: assigning each sub-video of the plurality of sub-videos to a respective slot in a video template that defines a format for combining the plurality of sub-videos.
- the method further comprises: receiving one or more additional data elements; and assigning each additional data element to a respective slot in the video template.
- each additional data element comprises image data, text data, or both.
- the output video concurrently shows at least two of the sub-videos of the plurality of sub-videos.
- the method further comprises: receiving a request for a digital component; determining that a digital component that includes the output video is responsive to the request; and providing the digital component that includes the output video in response to the request to be presented alongside search results or on a third-party webpage.
- receiving data indicating the target object type comprises: receiving data specifying a keyword; and mapping the keyword to the target object type in accordance with a mapping from keywords to a predefined set of possible target object types.
- a system comprising one or more data processing apparatus and one or more storage devices storing instructions that, when executed by the one or more data processing apparatus, cause the one or more data processing apparatus to perform the operations of the method described above.
- a computer-readable storage medium storing instructions that, when executed by one or more data processing apparatus, cause the one or more data processing apparatus to perform the operations of the method described above.
- This specification describes a video generation system that can process an input video to extract multiple sub-videos that each show a respective object of a target object type, and then combine the sub-videos to generate a topical video.
- the video generation system may combine the sub-videos, e.g., by slotting the sub-videos into a video template where multiple sub-videos may be concurrently shown.
- the topical video may effectively summarize the content of the input video that relates to the target object type, while potentially having a shorter duration than the input video, occupying less space in a memory than the input video, and/or requiring less bandwidth to transmit over a communications network (e.g., the Internet) than the input video.
- a communications network e.g., the Internet
- the topical video may occupy less space in a memory than the input video and/or require less bandwidth to transmit over a communications network than the input video, e.g., because each sub-video frame of each sub-video (e.g., that makes up the topical video) is cropped from a respective video frame of the input video to include less than all of the content of the video frame and/or because each sub-video comprises content from a subset of the video frames of the input video.
- each sub-video comprises a spatial and/or temporal subset of the content of the input video.
- this subset of the content is based on generated tracking data that identifies and tracks visual locations of one or more instances of target objects of a target object type in the input video.
- This provides more efficient generation of videos through filtering content of an input video to identify and select targeted portions of the input video. Instead of the entire input video being transmitted to distributed systems, a topical video comprising a targeted subset of the input video can be generated and transmitted to a particular system. This can reduce the bandwidth required for transmitting content of the input video when requests are received for content included in the input video. Therefore, the video generation system may enable more efficient use of computational resources, e.g., memory, bandwidth, and computing power. Additionally, by summarizing the content of the input video that relates to the target object type by combining the sub-videos, the information about the target object type from the input video can be presented in a more condensed form and without requiring a user to view the entirety of the input video. This reduces the amount of network bandwidth required to present the information about the target object type to users, and also reduces the amount of time required to present the information about the target object type.
- computational resources e.g., memory, bandwidth, and computing power.
- the video generation system described in this specification can rapidly generate visually pleasing topical videos (e.g., in a matter of minutes) while requiring little or no manual user input, e.g., beyond providing the input video and specifying the target object type.
- manually generating topical videos may be expensive and time consuming, e.g., requiring hours or days, and a significant amount of expertise. Therefore, the video generation system described in this specification enables more efficient allocation of time and resources by automating the task of generating topical videos.
- FIG. 1 shows an example video generation system 100.
- the video generation system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.
- the system 100 is configured to process: (i) an input video 102, and (ii) data identifying a "target" object type 104, to generate a topical video 106.
- the input video 102 may include a sequence of video frames, where each video frame may be represented as an array of numerical values and be associated with a respective time point in the input video 102.
- the target object type 104 may be a possible object type from a predefined set of possible object types.
- the set of possible object types may include any appropriate object types, e.g., food types (e.g., pizza, sandwich, etc.), beverage types (e.g., wine, beer, etc.), clothing types (e.g., dress, pants, shirt, blouse, etc.), accessory types (e.g., hat, handbag, glasses, etc.), electronics types (e.g., computer, smartphone, etc.), animal types (e.g., dog, cat, etc.), vehicle types (e.g., truck, sedan, sport utility vehicle (SUV), etc.), health/beauty types (e.g., lipstick, palette, nail polish, etc.), sporting goods types (e.g., football, soccer ball, tennis racket, swimming goggles, etc.), household appliance types (e.g., dishwasher, refrigerator, etc.), furniture types (e.g., chair, couch, etc.), and/or person types (e.
- the system 100 processes the input video 102 and the target object type 104 to generate a topical video 106 that includes multiple "sub-videos" that are extracted (e.g., cropped) from the input video 102, where each sub-video shows (e.g., depicts) a respective object of the target object type 104. That is, the system 100 generates a topical video 106 that includes portions of the input video 102 which show objects of the target object type 104.
- the system 100 can be used for any of a variety of applications.
- the system 100 may be used to generate topical videos that are integrated into digital components that are transmitted for presentation with electronic documents at user devices, e.g., in blocks alongside search results or on third party websites, as will be described in more detail with reference to FIG. 3 and FIG. 4 .
- the system 100 includes a tracking engine 108, a cropping engine 110, and a synthesis engine 112, which will each be described in more detail next.
- the tracking engine 108 is configured to process the input video 102 and the data identifying the target object type 104 to generate tracking data 114 that identifies, labels, and tracks visual locations of one or more instances of objects of the target object type 104 in the input video 102.
- an object of the target object type 104 may be referred to as a "target object”.
- the tracking data 114 may include data defining one or more "target object trajectories", where each target object trajectory corresponds to a respective instance of a target object and tracks the position of the target object through each video frame in a sequence of multiple video frames. More specifically, a target object trajectory corresponding to a target object may specify the position of the target object in each video frame in a sequence of video frames, e.g., by specifying a respective bounding box corresponding to each video frame that encloses the target object in the video frame.
- a bounding box corresponding to a video frame may be defined, e.g., by the coordinates of the vertices of the bounding box, where the coordinates are expressed in the frame of reference of the video frame.
- a "bounding box" in a video frame may specify a region of the video frame having any appropriate shape, e.g., a square shape, a rectangular shape, or a circular shape.
- a bounding box specifying the position of a target object in a video frame may enclose less than all of the video frame, e.g., such that the bounding box excludes at least part of the video frame. (However, in some cases, a bounding box in a video frame may enclose the entirety of the video frame).
- each target object trajectory may track the position of a corresponding target object through a sequence of video frames, which is a proper subset of the entire sequence of video frames of the input video.
- the tracking engine 108 may use any appropriate object detection and tracking technique to generate the target object trajectories of the tracking data 114.
- the tracking engine 108 may use the object detection and tracking techniques described in: J. Henriques, R. Caseiro, P. Martins, J. Batista, "High-speed tracking with kemelized correlation filters," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, no. 3, pp. 583-596 (2014 ).
- the tracking engine 108 may use the object detection and tracking techniques described in: N. Wojke, A. Bewley, D. Paulus, "Simple online and realtime tracking with a deep association metric," 2017 IEEE International Conference on Image Processing (ICIP), pp. 3645-3649 (2017 ).
- the tracking engine 108 may process each video frame of the input video 102 using an object detection neural network that is configured to process a video frame to generate an object detection output.
- the object detection output may include: (i) one or more bounding boxes that each enclose a respective object depicted in the video frame, and (ii) for each bounding box, a label identifying a type of the object that is enclosed by the bounding box.
- the tracking engine 108 may maintain the bounding boxes that are labeled as enclosing objects of the target object type 104, while discarding (e.g., refraining from further processing of) any other bounding boxes.
- the tracking engine 108 may determine that a first bounding box in a first frame and a second bounding box in a second (e.g., subsequent) frame enclose the same object by determining that a similarity measure between the first bounding box and the second bounding box exceeds a threshold value.
- the similarity measure may be based on: (i) a visual similarity between the contents of the first bounding box and the second bounding box, and (ii) an overlap between the first bounding box and the second bounding box.
- the tracking engine 108 may determine that the first bounding box and the second bounding box are included in the same target object trajectory, e.g., thereby tracking the position of the object from the first frame to the second frame.
- the cropping engine 110 is configured to process the input video 102 and the tracking data 114 to generate one or more sub-videos 116. More specifically, the cropping engine 110 may generate a respective sub-video 116 corresponding to each target object trajectory specified by the tracking data 114.
- a "sub-video” refers to a video having a sequence of video frames, referred to as "sub-video frames", such that each sub-video frame of the sub-video is cropped from a respective video frame of the video 102.
- a sub-video frame of a sub-video 116 that is cropped from a corresponding video frame of the video 102 may include less than all of the corresponding video frame, e.g., such that the sub-video frame excludes at least a portion of the corresponding video frame (however, in some cases, a sub-video frame may include the entirety of a corresponding video frame of the video 102).
- a video frame may have dimensions a ⁇ b, where a is the width of the video frame (e.g., the number of pixels along the width spatial dimension of the video frame) and b is the height of the video frame (e.g., the number of pixels along the height spatial dimension of the video frame).
- the sub-video frame cropped from the video frame may have dimensions c ⁇ d, where c is less than a and d is less than b .
- the "dimensions" of a video frame may refer to data specifying: (i) the width of the video frame (e.g., the number of pixels along the width spatial dimension of the video frame), and (ii) the height of the video frame (e.g., the number of pixels along the height spatial dimension of the video frame).
- the cropping engine 110 can generate a sub-video 116 corresponding to a target object trajectory of a target object in a variety of ways. A few example implementations of the cropping engine 110 are described in more detail next.
- the cropping engine 110 may extract a respective sub-video frame from each video frame corresponding to the target object trajectory by extracting a portion of the video frame based on the position of the target object in the video frame.
- the target object trajectory may specify a respective bounding box for each video frame that encloses the target object in the video frame, and the cropping engine 110 may extract the portion of each video frame that is enclosed by the bounding box for the video frame. That is, the cropping engine 110 may generate each sub-video frame by extracting the portion of the corresponding video frame that is enclosed by the bounding box specified by the target object trajectory as enclosing the target object in the video frame.
- the tracking engine 108 may be configured to generate target object trajectories that specify a bounding box having fixed dimensions (e.g., width and height) for each video frame, e.g., such that each of the extracted sub-video frames have the same dimensions.
- the cropping engine 110 determines the dimensions (e.g., width and height) and position (e.g., center) of a "super" bounding box. In particular, the cropping engine 110 determines the dimensions and position of the super bounding box such that the target object is enclosed by the super bounding box in each video frame corresponding to the target object trajectory.
- the target object trajectory may specify a respective bounding box for each video frame that encloses the target object in the video frame, and the cropping engine 110 may identify a super bounding box that encloses each of the bounding boxes specified by the target object trajectory.
- the cropping engine 110 may identify a super bounding box that encloses the union of the set of bounding boxes specified by the target object trajectory (while still enclosing less than all of each video frame). After determining the dimensions and position of the super bounding box, the cropping engine 110 may generate each sub-video frame by extracting the portion of a corresponding video frame that is enclosed by the super bounding box. Generating the sub-video 116 using a super bounding box that is the same for each video frame may improve the visual quality of the sub-video 116, e.g., by better preserving the cinematographic effects of the original video 102, e.g., panning and zooming.
- certain sub-videos 116 generated by the cropping engine 110 may be "temporally overlapping", e.g., such two or more of the sub-videos include respective sub-video frames that are extracted from the same video frame of the input video 102. This may occur, e.g., when the tracking engine 108 tracks two different instances of target objects through the same video frame in the video.
- the synthesis engine 112 is configured to process the sub-videos 116 and a video template 118 to generate the topical video 106.
- the video template 118 defines a format for combining multiple sub-videos 116 into a composite video (e.g., the topical video 106) that includes each of the sub-videos.
- the video template may define a partition of a two-dimensional (2-D) region (e.g., having a square or rectangular shape) into one or more sub-regions, referred to as "slots".
- the synthesis engine 112 generates the topical video 106 by assigning each sub-video 116 to a respective slot in the video template 118.
- the video template 118 may dynamically change over the duration of the topical video 106, e.g., slots may be removed, slots may be added, or the sizes of slots may be changed over the duration of the topical video.
- the video template 200 includes three slots - i.e., slot 202, slot 204, and slot 206.
- the synthesis engine 112 may generate the topical video 106 by assigning a respective sub-video 116 to each slot of the video template 200, e.g., such that the sub-videos in each slot are shown concurrently.
- the target object type may be "vehicle”
- each slot may be assigned a respective sub-video showing an instance of a vehicle.
- the video template 118 may be defined such that each sub-video 116 is shown consecutively, e.g., one after another, rather than any sub-videos being shown concurrently.
- the synthesis engine 112 may modify a sub-video 116 prior to assigning it to a slot in the video template 118.
- the synthesis engine 112 may modify (e.g., reduce) the dimensions (e.g., width and/or the height) of the sub-video frames of a sub-video, e.g., to cause the dimensions of the sub-video to match the dimensions of a corresponding slot in the video template 118.
- the synthesis engine 112 may reduce the duration of a sub-video (e.g., by removing one or more sub-video frames of the sub-video) to cause the duration of the sub-video to match the duration of a slot.
- the duration of a slot may refer to the length of time the slot is included in the template, e.g., before being removed from the template.
- the synthesis engine 112 may determine the assignment of the sub-videos 116 to the slots of the video template 118 in any of a variety of ways. In one example, the synthesis engine 112 may randomly assign sub-videos 116 to respective slots of the video template 118. In another example, the synthesis engine 112 may assign each sub-video 116 to a corresponding slot having dimensions that are most similar (e.g., from amongst all the slots) to the dimensions of the sub-video 116, e.g., to minimize the modifications applied to the sub-videos to cause them to fit their assigned slots.
- the synthesis engine 112 may determine a respective score for each sub-video, where the score for a sub-video characterizes a quality of the sub-video.
- the synthesis engine 112 may determine the score for each sub-video based on one or more criteria, e.g., the sharpness of the sub-video and the amount of motion in the sub-video.
- the sharpness of a sub-video may be computed using any appropriate sharpness measure, e.g., by computing the maximum pixel intensity value in the sub-video frames after convolving the sub-video frames with a Laplacian kernel.
- the amount of motion in the sub-video may be computed, e.g., by computing the average value of a set of optical flow frames corresponding to the sub-video frames.
- the synthesis engine 112 may generate the overall score for a sub-video based on, e.g., a linear combination of a sharpness score characterizing the sharpness of the sub-video and a motion score characterizing the amount of motion in the sub-video.
- the synthesis engine 112 may generate higher scores for sub-videos that are sharper and include more motion.
- the synthesis engine 112 may use the scores for the sub-videos 116 to determine which sub-videos 116 should be included in the topical video 106. For example, the synthesis engine 112 may determine that only a proper subset of the sub-videos 116 having the highest scores should be included in the topical video 106. As another example, the synthesis engine 112 may determine that any sub-video 116 having a score that fails to satisfy a predefined threshold value (e.g., by exceeding the threshold value) should not be included in the topical video 106.
- a predefined threshold value e.g., by exceeding the threshold value
- the system 100 may receive one or more data elements 120 to be included in the topical video 106, e.g., other than sub-videos 116 extracted from the input video 102.
- the data elements 120 may include, e.g., images or portions of text.
- the synthesis engine 112 may include the data elements 120 in the topical video 106 by assigning the data elements to respective slots of the video template 118.
- the topical videos 106 generated by the video generation system 100 can be used in any of a variety of applications. A few example applications are described in more detail next.
- topical videos 106 generated by the video generation system 100 may be included in digital components that are provided in response to digital component requests, e.g., as described in more detail with reference to FIG. 3 and FIG. 4 .
- the video generation system 100 may process: (i) input videos that are generated, e.g., by one or more surveillance cameras (security cameras), and (ii) target object types 104 that are specified by a user, to generate corresponding topical videos 106.
- the input videos generated by the surveillance cameras may be lengthy, e.g., having a duration of hours or days.
- the video generation system 100 may enable rapid processing of videos generated by surveillance cameras to generate topical videos showing objects of interest to a user, e.g., pets, people, or vehicles.
- a surveillance camera may refer to a camera that is placed at a given position (e.g., beside the front door of a home) and continuously captures video of a region in the vicinity of the given position.
- the video generation system 100 may process: (i) input videos that are generated by a user device, e.g., a video camera of a smartphone, and (ii) target object types 104 that are specified by a user, to generate corresponding topical videos 106.
- a user device e.g., a video camera of a smartphone
- target object types 104 that are specified by a user
- FIG. 3 shows an example digital component generation system 300
- the digital component generation system 300 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.
- the digital component generation system 300 is configured to receive one or more keywords 302 and an input video 102 from a content provider, and to generate one or more digital components 306 that include topical videos 106 relevant to respective keywords 302.
- the content provider may be, e.g., a publisher, advertiser, or other content source.
- digital component refers to a discrete unit of digital content or digital information that can include one or more of, e.g., images, video clips (e.g., topical videos), audio clips, multimedia clips, text segments, or uniform resource locators (URLs).
- a digital component can be electronically stored in a physical memory device as a single file or in a collection of files, and digital components can take the form of video files, audio files, multimedia files, image files, or text files and include streaming video, streaming audio, social network posts, blog posts, and/or advertising information, such that an advertisement is a type of digital component.
- a digital component can be enhanced with data from one or more sources (e.g., weather information, real time event information, or other information obtained from other sources).
- the system 300 includes a keyword mapping engine 308 and a video generation system 100 (as described with reference to FIG. 1 ).
- the system 300 may be associated with a user interface (e.g., a graphical user interface, or any other appropriate sort of user interface) that enables the content provider to interact with the system 300, as will be described in more detail below.
- a user interface e.g., a graphical user interface, or any other appropriate sort of user interface
- the keyword mapping engine 308 is configured to map each keyword 302 to a corresponding target object type 104 from a set of possible object types, e.g., in accordance with a predefined mapping from keywords to possible object types. For example, the keyword mapping engine 308 may map the each of the keywords “spectacles”, “specs”, “reading glasses”, “sunglasses", and “eyeglasses” to the possible object type of "glasses".
- the set of possible object types may be the set of object types that the tracking engine of the video generation system 100 is trained (or otherwise configured) to identify and track in videos.
- the system 300 may provide an indication of the target object types 104 corresponding to the keywords 302 to the content provider, and request that the content provider "approve” (e.g., perform an action to accept) each of the target object types 104.
- the system 300 may use the video generation system 100 to generate one or more respective topical videos 106 corresponding to each target object type 104, e.g., by processing the target object type 104 and the input video 102 using the video generation system 100.
- the video generation system 100 may generate multiple topical videos 106 corresponding to a single target object type 104, e.g., by generating topical videos 106 using different video templates from a set of possible video templates 310, by generating topical videos 106 using different combinations of sub-videos extracted from the input video, by generating topical videos 106 where the sub-videos of the input video 102 are assigned to different combinations of slots in a video template, or by overlaying different audio soundtracks over the topical videos 106.
- the system 300 may enable the content provider to specify (e.g., through the user interface) various factors controlling the generation of the topical videos 106, e.g., which video templates 310 and/or audio soundtracks should be used by the video generation system 100 to generate the topical videos 106.
- the content provider may also provide one or more other data elements (e.g., images or portions of text) to the system 300 to be included in the topical videos 106.
- the video generation system 100 may include the provided data elements in the topical videos, e.g., as described with reference to FIG. 1 .
- the system 300 may enable the content provider to select one or more of the topical videos 106 to be included in respective digital components 306.
- Each digital component 306 may include a topical video 106 and other data elements (e.g., videos, images, portions of text, URLs) that may be provided by the content provider.
- the content provider may provide the digital components 306 generated by the system 300 to a digital component distribution system, e.g., the digital component distribution system 410 described with reference to FIG. 4 .
- the digital component distribution system can transmit the digital components 306 in response to requests for digital components to be presented with electronic documents at user devices, e.g., in blocks alongside search results or on third party websites.
- a content provider may provide the system 300 with an input video 102 that includes several hours of footage of a fashion show, and the keyword 302 "handbag".
- the system 300 may process the input video 102 and the keyword 302 to generate a corresponding set of topical videos 106, each of which include multiple sub-videos of the input video that show instances of handbags, where the sub-videos are automatically edited into visually pleasing video templates, and omit portions of the input video, such that the populated video templates have a shorter duration (e.g., 30 second, 1 minute, 2 minutes, or another duration) than the duration of the input video 102 (e.g., the several hours of footage of the fashion show).
- the content provider may then select one or more of the topical videos 106 to be included in digital components that are provided to the digital component distribution system.
- FIG. 4 is a block diagram of an example environment 400 in which a digital component distribution system 410 transmits digital components from a digital component database 416 for presentation with electronic documents
- a content provider may generate digital components using the digital component generation system 300 (described with reference to FIG. 3 ) and provide the generated digital components to the digital component distribution system 410.
- the example environment 400 includes a network 402, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof.
- the network 402 connects electronic document servers 404, client devices 406, digital component servers 408, and the digital component distribution system 410 (also referred to as the "distribution system" 410).
- the example environment 400 may include many different electronic document servers 404, client devices 406, and digital component servers 408.
- a client device 406 is an electronic device that is capable of requesting and receiving resources over the network 402.
- Example client devices 406 include personal computers, mobile communication devices (e.g., mobile phones), and other devices that can send and receive data over the network 402.
- a client device 406 typically includes a user application, such as a web browser, to facilitate the sending and receiving of data over the network 402, but native applications executed by the client device 406 can also facilitate the sending and receiving of data over the network 402.
- An electronic document is data that presents a set of content at a client device 406.
- Examples of electronic documents include webpages, word processing documents, portable document format (PDF) documents, images, videos, search results pages, and feed sources.
- Native applications e.g., "apps"
- the electronic document servers 404 can include servers that host publisher websites.
- the client device 406 can initiate a request for a given publisher webpage, and the electronic server 404 that hosts the given publisher webpage can respond to the request by sending machine executable instructions that initiate presentation of the given webpage at the client device 406.
- the electronic document servers 404 can include app servers from which client devices 406 can download apps.
- the client device 406 can download files required to install an app at the client device 406, and then execute the downloaded app locally.
- Electronic documents can include a variety of content.
- an electronic document can include static content (e.g., text or other specified content) that is within the electronic document itself and/or does not change over time.
- Electronic documents can also include dynamic content that may change over time or on a per-request basis
- a publisher of a given electronic document can maintain a data source that is used to populate portions of the electronic document.
- the given electronic document can include one or more tags or scripts that cause the client device 406 to request content from the data source when the given electronic document is processed (e.g., rendered or executed) by a client device 406.
- the client device 406 integrates the content obtained from the data source into the given electronic document to create a composite electronic document including the content obtained from the data source.
- a given electronic document can include one or more digital component tags or digital component scripts that reference the digital component distribution system 410.
- the digital component tags or digital component scripts are executed by the client device 406 when the given electronic document is processed by the client device 406. Execution of the digital component tags or digital component scripts configures the client device 406 to generate a request for one or more digital components 412 (referred to as a "component request"), which is transmitted over the network 402 to the digital component distribution system 410.
- a digital component tag or digital component script can enable the client device 406 to generate a packetized data request including a header and payload data.
- the component request 412 can include event data specifying features such as a name (or network location) of a server from which the digital component is being requested, a name (or network location) of the requesting device (e.g., the client device 406), and/or information that the digital component distribution system 410 can use to select one or more digital components provided in response to the request.
- the component request 412 is transmitted, by the client device 406, over the network 402 (e.g., a telecommunications network) to a server of the digital component distribution system 410.
- the component request 412 can include event data specifying other event features, such as the electronic document being requested and characteristics of locations of the electronic document at which digital component can be presented.
- event data specifying a reference (e.g., URL) to an electronic document (e.g., webpage) in which the digital component will be presented, available locations of the electronic documents that are available to present digital components, sizes of the available locations, and/or media types that are eligible for presentation in the locations can be provided to the digital component distribution system 410.
- event data specifying keywords associated with the electronic document (“document keywords") or entities (e.g., people, places, or things) that are referenced by the electronic document can also be included in the component request 412 (e.g., as payload data) and provided to the digital component distribution system 410 to facilitate identification of digital components that are eligible for presentation with the electronic document.
- the event data can also include a search query that was submitted from the client device 406 to obtain a search results page, and/or data specifying search results and/or textual, audible, or other visual content that is included in the search results.
- Component requests 412 can also include event data related to other information, such as information that a user of the client device has provided, geographic information indicating a state or region from which the component request was submitted, or other information that provides context for the environment in which the digital component will be displayed (e.g., a time of day of the component request, a day of the week of the component request, a type of device at which the digital component will be displayed, such as a mobile device or tablet device).
- Component requests 412 can be transmitted, for example, over a packetized network, and the component requests 412 themselves can be formatted as packetized data having a header and payload data.
- the header can specify a destination of the packet and the payload data can include any of the information discussed above.
- the component distribution system 410 chooses digital components that will be presented with the given electronic document in response to receiving the component request 412 and/or using information included in the component request 412.
- a digital component is selected (using the techniques described herein) in less than a second to avoid errors that could be caused by delayed selection of the digital component. For example, delays in providing digital components in response to a component request 412 can result in page load errors at the client device 406 or cause portions of the electronic document to remain unpopulated even after other portions of the electronic document are presented at the client device 406.
- delays in providing the digital component can result in a failed delivery of the digital component, for example, if the electronic document is no longer presented at the client device 406 when the digital component is provided.
- the digital component distribution system 410 is implemented in a distributed computing system that includes, for example, a server and a set of multiple computing devices 414 that are interconnected and identify and distribute digital components in response to requests 412.
- the set of multiple computing devices 414 operate together to identify a set of digital components that are eligible to be presented in the electronic document from a corpus of millions of available digital components (DC1-x).
- the millions of available digital components can be indexed, for example, in a digital component database 416.
- Each digital component index entry can reference the corresponding digital component and/or include distribution parameters (DP1-DPx) that contribute to (e.g., condition or limit) the distribution/transmission of the corresponding digital component.
- DP1-DPx distribution parameters
- the distribution parameters can contribute to the transmission of a digital component by requiring that a component request include at least one criterion that matches (e.g., either exactly or with some pre-specified level of similarity) one of the distribution parameters of the digital component.
- the distribution parameters for a particular digital component can include distribution keywords that must be matched (e.g., by electronic documents, document keywords, or terms specified in the component request 412) in order for the digital component to be eligible for presentation.
- the distribution parameters are used to trigger distribution (e.g., transmission) of the digital components over the network 402.
- the distribution parameters can also require that the component request 412 include information specifying a particular geographic region (e.g., country or state) and/or information specifying that the component request 412 originated at a particular type of client device (e.g., mobile device or tablet device) in order for the digital component to be eligible for presentation.
- the distribution parameters can also specify an eligibility value (e.g., ranking score, bid, or some other specified value) that is used for evaluating the eligibility of the digital component for distribution/transmission (e.g., among other available digital components), for example, by the component evaluation process.
- the eligibility value can specify a maximum amount of compensation that a provider of the digital component is willing to submit in response to the transmission of the digital component (e.g., for each instance of specific events attributed to the presentation of the digital component, such as user interaction with the digital component).
- the identification of the eligible digital component can be segmented into multiple tasks 417a-417c that are then assigned among computing devices within the set of multiple computing devices 414.
- different computing devices in the set 414 can each analyze a different portion of the digital component database 416 to identify various digital components having distribution parameters that match information included in the component request 412.
- each given computing device in the set 414 can analyze a different data dimension (or set of dimensions) and pass (e.g., transmit) results (Res 1-Res 3) 418a-418c of the analysis back to the digital component distribution system 410.
- the results 418a-418c provided by each of the computing devices in the set 414 may identify a subset of digital components that are eligible for distribution in response to the component request and/or a subset of the digital components that have certain distribution parameters.
- the identification of the subset of digital components can include, for example, comparing the event data to the distribution parameters, and identifying the subset of digital components having distribution parameters that match at least some features of the event data.
- the digital component distribution system 410 aggregates the results 418a-418c received from the set of multiple computing devices 414 and uses information associated with the aggregated results to: (i) select one or more digital components that will be provided in response to the request 412, and (ii) determine transmission requirements for the one or more digital components. For example, the digital component distribution system 410 can select a set of winning digital components (one or more digital components) based on the outcome of one or more component evaluation processes.
- the digital component distribution system 410 can generate and transmit, over the network 402, reply data 420 (e.g., digital data representing a reply) that enables the client device 406 to integrate the set of winning digital components into the given electronic document, such that the set of winning digital components and the content of the electronic document are presented together at a display of the client device 406.
- reply data 420 e.g., digital data representing a reply
- the client device 406 executes instructions included in the reply data 420, which configures and enables the client device 406 to obtain the set of winning digital components from one or more digital component servers.
- the instructions in the reply data 420 can include a network location (e.g., a Uniform Resource Locator (URL)) and a script that causes the client device 406 to transmit a server request (SR) 421 to the digital component server 408 to obtain a given winning digital component from the digital component server 408.
- SR server request
- the digital component server 408 will identify the given winning digital component specified in the server request 421 (e.g., within a database storing multiple digital components) and transmit, to the client device 406, digital component data (DC Data) 422 that presents the given winning digital component in the electronic document at the client device 406.
- DC Data digital component data
- the environment 400 can include a search system 450 that identifies the electronic documents by crawling and indexing the electronic documents (e.g., indexed based on the crawled content of the electronic documents). Data about the electronic documents can be indexed based on the electronic document with which the data are associated. The indexed and, optionally, cached copies of the electronic documents are stored in a search index 452 (e.g., hardware memory device(s)). Data that are associated with an electronic document is data that represents content included in the electronic document and/or metadata for the electronic document.
- a search index 452 e.g., hardware memory device(s)
- Client devices 406 can submit search queries to the search system 450 over the network 402.
- the search system 450 accesses the search index 452 to identify electronic documents that are relevant to the search query.
- the search system 450 identifies the electronic documents in the form of search results and returns the search results to the client device 406 in a search results page.
- a search result is data generated by the search system 450 that identifies an electronic document that is responsive (e.g., relevant) to a particular search query, and includes an active link (e.g., hypertext link) that causes a client device to request data from a specified network location (e.g., URL) in response to user interaction with the search result.
- an active link e.g., hypertext link
- An example search result can include a web page title, a snippet of text or a portion of an image extracted from the web page, and the URL of the web page.
- Another example search result can include a title of a downloadable application, a snippet of text describing the downloadable application, an image depicting a user interface of the downloadable application, and/or a URL to a location from which the application can be downloaded to the client device 406.
- the search system 450 can be part of, or interact with, an application store (or an online portal) from which applications can be downloaded for install at a client device 406 in order to present information about downloadable applications that are relevant to a submitted search query.
- search results pages can include one or more slots in which digital components (e.g., advertisements, video clips, audio clips, images, or other digital components) can be presented.
- the distribution system 410 may identify a set of digital components that are eligible to be transmitted in response to the component request. The distribution system 410 may then select one or more of the eligible digital components to be transmitted through, e.g., an auction procedure. In some implementations, the distribution system 410 performs an auction procedure by ranking the eligible digital components in accordance with their respective eligibility values, and selecting one or more highest-ranked digital components to be transmitted in response to the component request.
- FIG. 5 is a flow diagram of an example process 500 for generating a topical video.
- the process 500 will be described as being performed by a system of one or more computers located in one or more locations.
- a video generation system e.g., the video generation system 100 of FIG. 1 , appropriately programmed in accordance with this specification, can perform the process 500.
- the system receives: (i) an input video including a sequence of video frames, and (ii) data indicating a target object type (502).
- the system may receive data specifying a keyword (e.g., "sunglasses"), and then map the received keyword to a corresponding target object type (e.g., "glasses") in accordance with a mapping from keywords to a predefined set of possible target object types.
- the system may also receive one or more additional data elements, e.g., images, portions of text, or both.
- the system processes the input video to generate tracking data that identifies and tracks visual locations of one or more instances of target objects of the target object type in the input video (504). For example, for each instance of a target object of the target object type in the input video, the system may determine a respective bounding box that encloses the instance of the target object in each of multiple video frames of the input video. The system may use any appropriate object detection and tracking technique to generate the tracking data, e.g., as described with reference to FIG. 1 .
- the system generates multiple sub-videos based on the input video and the tracking data (506). For each sub-video, the system may generate a respective sequence of sub-video frames that are each cropped from a respective video frame of the input video to include (i) less than all content of the respective video frame and (ii) a respective instance of a given target object from among the identified target objects of the target object type. In some implementations, instead of each of the sub-video frames being cropped to include less than all content of the respective video frame, at least one of the sub-video frames is cropped from at least one respective video frame of the input video to include less than all content of the at least one respective video frame.
- one or more of the sub-video frames may comprise all content of the respective video frames, for example because a target object occupies all of the respective video frames or most of the respective video frames in those respective video frames.
- a sub-video frame may comprise a subset of all content of the respective video frame, with the subset being less than or equal to all content of the respective video frame.
- the system may generate the sequence of sub-video frames of a sub-video corresponding to an instance of a given target object by cropping a respective bounding box that encloses the instance of the given target object from each of multiple video frames of the input video.
- the bounding boxes enclosing instance of the given target object may be specified by the tracking data.
- the system may determine, based on the tracking data, a same bounding box that encloses the instance of the given target object in each of multiple video frames of the input video. The system may then generate the sequence of sub-video frames of the sub-video by cropping the same bounding box from each of the multiple video frames of the input video.
- certain sub-videos may be "temporally overlapping", e.g., such two or more of the sub-videos include respective sub-video frames that are extracted from the same video frame of the input video. This may occur, e.g., when the system tracks two different instances of target objects through the same video frame in the input video.
- the system generates an output video that includes the sub-videos (508). For example, to generate the output video, the system may assign each sub-video to a respective slot in a video template that defines a format for combining the sub-videos. As part of generating the output video, the system may determine a respective score for each sub video (e.g., based on the sharpness of the sub-video), and select a proper subset of the sub-videos for inclusion in the output video (e.g., a predefined number of sub-videos with the highest scores). In some cases, the output video may concurrently show at least two of the sub-videos, while in other cases, the output video may show each of the sub-videos consecutively, e.g., one after another. As part of generating the output video, the system may assign each of one or more additional data elements (e.g., textual or image data elements, as described with reference to 502) to a respective slot in the video template.
- additional data elements e.g., textual or image
- the output video generated by the system may be included in a digital component that is provided in response to a digital component request and subsequently presented alongside search results or on a third-party webpage.
- Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.
- Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, e.g., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus.
- the computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.
- the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.
- data processing apparatus refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers.
- the apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).
- the apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
- a computer program which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
- a program may, but need not, correspond to a file in a file system.
- a program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code.
- a computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.
- engine is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions.
- an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.
- the processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output.
- the processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.
- Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit.
- a central processing unit will receive instructions and data from a read-only memory or a random access memory or both.
- the essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data.
- the central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
- a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks.
- a computer need not have such devices.
- a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.
- PDA personal digital assistant
- GPS Global Positioning System
- USB universal serial bus
- Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.
- semiconductor memory devices e.g., EPROM, EEPROM, and flash memory devices
- magnetic disks e.g., internal hard disks or removable disks
- magneto-optical disks e.g., CD-ROM and DVD-ROM disks.
- embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer.
- a display device e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor
- keyboard and a pointing device e.g., a mouse or a trackball
- Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.
- a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's device in response to requests received from the web browser.
- a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.
- Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, e.g., inference, workloads.
- Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework, a Microsoft Cognitive Toolkit framework, an Apache Singa framework, or an Apache MXNet framework.
- a machine learning framework e.g., a TensorFlow framework, a Microsoft Cognitive Toolkit framework, an Apache Singa framework, or an Apache MXNet framework.
- Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components.
- the components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.
- LAN local area network
- WAN wide area network
- the computing system can include clients and servers.
- a client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
- a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client.
- Data generated at the user device e.g., a result of the user interaction, can be received at the server from the device.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Signal Processing (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Databases & Information Systems (AREA)
- Human Computer Interaction (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Description
- This specification relates to generating videos.
- The Internet facilitates the exchange of information between users across the globe. Content from multiple different providers, including videos, can be integrated into a single electronic document to create a composite document. For example, a portion of the content included in the electronic document may be selected (or specified) by a publisher of the electronic document. A different portion of content (e.g., third party content, including videos) can be provided by a third party (e.g., an entity that is not a publisher of the electronic document), and integrated into the electronic document to form a composite document that includes content from multiple different sources.
- Reference is made to Luo Chu: "Video Summarization for Object Tracking in the Internet of Things", 2014 Eight International Conference on Next Generation Mobile Apps, Services and Technologies, IEEE, 10 September 2014, pages 288-293.
EP 3249651 A1 describes a method for generating a summary video sequence from a source video sequence. - The document
EP 3 249 651 A1 (AXIS AB [SE]) 29 November 2017 discloses to generate sub-videos of target objects by cropping. - In an aspect, a method is provided as defined in
claim 1. In another aspect, a system is provided as defined in claim 8. In another aspect, a computer-readable storage medium is provided as defined in claim 9. - This specification describes a system implemented as computer programs on one or more computers in one or more locations that can process an input video to extract multiple sub-videos from the input video that each show a respective object of a target object type. The system can combine the sub-videos showing the objects of the target object type to generate a "topical" video (e.g., that includes a target object type related to a given topic). The topical video may be integrated into a digital component that is transmitted for presentation with electronic documents at user devices, e.g., in blocks alongside search results or on third party websites.
- According to a first implementation there is provided a method performed by one or more data processing apparatus, the method comprising: receiving: (i) an input video comprising a sequence of video frames, and (ii) data indicating a target object type; processing the input video to generate tracking data that identifies and tracks visual locations of one or more instances of target objects of the target object type in the input video; generating a plurality of sub-videos based on the input video and the tracking data, including: for each sub-video, generating a respective sequence of sub-video frames that are each extracted from a respective video frame of the input video to include a respective instance of a given target object from among the identified target objects of the target object type, wherein at least one of the sub-video frames is cropped from a respective video frame of the input video to include less than all content of the respective video frame; and generating an output video that comprises the plurality of sub-videos.
- In some implementations, processing the input video to generate the tracking data comprises, for each instance of a target object of the target object type in the input video: determining, for each of multiple video frames of the input video, a respective bounding box that encloses the instance of the target object in the video frame.
- In some implementations, for each sub-video, generating the respective sequence of sub-video frames comprises: cropping the respective bounding box that encloses the instance of the given target object corresponding to the sub-video from each of the multiple video frames of the input video.
- Generating the respective sequence of sub-video frames comprises: determining, based on the tracking data, a same bounding box that encloses the instance of the given target object corresponding to the sub-video in each of multiple video frames of the input video; and cropping the same bounding box from each of the multiple video frames of the input video.
- In some implementations, a sub-video frame of a first sub-video and a sub-video frame of a second sub-video are each cropped from a same video frame of the input video.
- In some implementations, generating the output video that comprises the plurality of sub-videos comprises: assigning each sub-video of the plurality of sub-videos to a respective slot in a video template that defines a format for combining the plurality of sub-videos.
- In some implementations, the method further comprises: receiving one or more additional data elements; and assigning each additional data element to a respective slot in the video template.
- In some implementations, each additional data element comprises image data, text data, or both.
- In some implementations, the output video concurrently shows at least two of the sub-videos of the plurality of sub-videos.
- In some implementations, the method further comprises: receiving a request for a digital component; determining that a digital component that includes the output video is responsive to the request; and providing the digital component that includes the output video in response to the request to be presented alongside search results or on a third-party webpage.
- In some implementations, receiving data indicating the target object type comprises: receiving data specifying a keyword; and mapping the keyword to the target object type in accordance with a mapping from keywords to a predefined set of possible target object types.
- According to another implementation, there is provided a system comprising one or more data processing apparatus and one or more storage devices storing instructions that, when executed by the one or more data processing apparatus, cause the one or more data processing apparatus to perform the operations of the method described above.
- According to another implementation, there is provided a computer-readable storage medium storing instructions that, when executed by one or more data processing apparatus, cause the one or more data processing apparatus to perform the operations of the method described above.
- Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.
- This specification describes a video generation system that can process an input video to extract multiple sub-videos that each show a respective object of a target object type, and then combine the sub-videos to generate a topical video. The video generation system may combine the sub-videos, e.g., by slotting the sub-videos into a video template where multiple sub-videos may be concurrently shown. The topical video may effectively summarize the content of the input video that relates to the target object type, while potentially having a shorter duration than the input video, occupying less space in a memory than the input video, and/or requiring less bandwidth to transmit over a communications network (e.g., the Internet) than the input video. The topical video may occupy less space in a memory than the input video and/or require less bandwidth to transmit over a communications network than the input video, e.g., because each sub-video frame of each sub-video (e.g., that makes up the topical video) is cropped from a respective video frame of the input video to include less than all of the content of the video frame and/or because each sub-video comprises content from a subset of the video frames of the input video. In other words, each sub-video comprises a spatial and/or temporal subset of the content of the input video. Moreover, this subset of the content is based on generated tracking data that identifies and tracks visual locations of one or more instances of target objects of a target object type in the input video. This provides more efficient generation of videos through filtering content of an input video to identify and select targeted portions of the input video. Instead of the entire input video being transmitted to distributed systems, a topical video comprising a targeted subset of the input video can be generated and transmitted to a particular system. This can reduce the bandwidth required for transmitting content of the input video when requests are received for content included in the input video. Therefore, the video generation system may enable more efficient use of computational resources, e.g., memory, bandwidth, and computing power. Additionally, by summarizing the content of the input video that relates to the target object type by combining the sub-videos, the information about the target object type from the input video can be presented in a more condensed form and without requiring a user to view the entirety of the input video. This reduces the amount of network bandwidth required to present the information about the target object type to users, and also reduces the amount of time required to present the information about the target object type.
- The video generation system described in this specification can rapidly generate visually pleasing topical videos (e.g., in a matter of minutes) while requiring little or no manual user input, e.g., beyond providing the input video and specifying the target object type. In contrast, manually generating topical videos may be expensive and time consuming, e.g., requiring hours or days, and a significant amount of expertise. Therefore, the video generation system described in this specification enables more efficient allocation of time and resources by automating the task of generating topical videos.
- The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.
-
-
FIG. 1 shows an example video generation system. -
FIG. 2 shows an illustration of an example video template. -
FIG. 3 shows an example digital component generation system. -
FIG. 4 is a block diagram of an example environment in which a digital component distribution system transmits digital components from a digital component database for presentation with electronic documents. -
FIG. 5 is a flow diagram of an example process for generating a topical video. - Like reference numbers and designations in the various drawings indicate like elements.
-
FIG. 1 shows an examplevideo generation system 100. Thevideo generation system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented. - The
system 100 is configured to process: (i) aninput video 102, and (ii) data identifying a "target"object type 104, to generate atopical video 106. - The
input video 102 may include a sequence of video frames, where each video frame may be represented as an array of numerical values and be associated with a respective time point in theinput video 102. - The
target object type 104 may be a possible object type from a predefined set of possible object types. The set of possible object types may include any appropriate object types, e.g., food types (e.g., pizza, sandwich, etc.), beverage types (e.g., wine, beer, etc.), clothing types (e.g., dress, pants, shirt, blouse, etc.), accessory types (e.g., hat, handbag, glasses, etc.), electronics types (e.g., computer, smartphone, etc.), animal types (e.g., dog, cat, etc.), vehicle types (e.g., truck, sedan, sport utility vehicle (SUV), etc.), health/beauty types (e.g., lipstick, palette, nail polish, etc.), sporting goods types (e.g., football, soccer ball, tennis racket, swimming goggles, etc.), household appliance types (e.g., dishwasher, refrigerator, etc.), furniture types (e.g., chair, couch, etc.), and/or person types (e.g., man, woman, child, etc.). In some implementations, the set of possible object types may include fine-grained object types, e.g., that identify particular products, e.g., a vehicle of a particular make and model. - The
system 100 processes theinput video 102 and thetarget object type 104 to generate atopical video 106 that includes multiple "sub-videos" that are extracted (e.g., cropped) from theinput video 102, where each sub-video shows (e.g., depicts) a respective object of thetarget object type 104. That is, thesystem 100 generates atopical video 106 that includes portions of theinput video 102 which show objects of thetarget object type 104. - The
system 100 can be used for any of a variety of applications. For example, thesystem 100 may be used to generate topical videos that are integrated into digital components that are transmitted for presentation with electronic documents at user devices, e.g., in blocks alongside search results or on third party websites, as will be described in more detail with reference toFIG. 3 andFIG. 4 . - The
system 100 includes atracking engine 108, acropping engine 110, and asynthesis engine 112, which will each be described in more detail next. - The
tracking engine 108 is configured to process theinput video 102 and the data identifying thetarget object type 104 to generate trackingdata 114 that identifies, labels, and tracks visual locations of one or more instances of objects of thetarget object type 104 in theinput video 102. For convenience, throughout this specification an object of thetarget object type 104 may be referred to as a "target object". - The tracking
data 114 may include data defining one or more "target object trajectories", where each target object trajectory corresponds to a respective instance of a target object and tracks the position of the target object through each video frame in a sequence of multiple video frames. More specifically, a target object trajectory corresponding to a target object may specify the position of the target object in each video frame in a sequence of video frames, e.g., by specifying a respective bounding box corresponding to each video frame that encloses the target object in the video frame. A bounding box corresponding to a video frame may be defined, e.g., by the coordinates of the vertices of the bounding box, where the coordinates are expressed in the frame of reference of the video frame. As used throughout this specification, a "bounding box" in a video frame may specify a region of the video frame having any appropriate shape, e.g., a square shape, a rectangular shape, or a circular shape. - A bounding box specifying the position of a target object in a video frame may enclose less than all of the video frame, e.g., such that the bounding box excludes at least part of the video frame. (However, in some cases, a bounding box in a video frame may enclose the entirety of the video frame). Moreover, each target object trajectory may track the position of a corresponding target object through a sequence of video frames, which is a proper subset of the entire sequence of video frames of the input video.
- The
tracking engine 108 may use any appropriate object detection and tracking technique to generate the target object trajectories of the trackingdata 114. As an example, thetracking engine 108 may use the object detection and tracking techniques described in: J. Henriques, R. Caseiro, P. Martins, J. Batista, "High-speed tracking with kemelized correlation filters," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, no. 3, pp. 583-596 (2014). As another example, thetracking engine 108 may use the object detection and tracking techniques described in: N. Wojke, A. Bewley, D. Paulus, "Simple online and realtime tracking with a deep association metric," 2017 IEEE International Conference on Image Processing (ICIP), pp. 3645-3649 (2017). - In a particular example, to generate the tracking
data 114, thetracking engine 108 may process each video frame of theinput video 102 using an object detection neural network that is configured to process a video frame to generate an object detection output. The object detection output may include: (i) one or more bounding boxes that each enclose a respective object depicted in the video frame, and (ii) for each bounding box, a label identifying a type of the object that is enclosed by the bounding box. Thetracking engine 108 may maintain the bounding boxes that are labeled as enclosing objects of thetarget object type 104, while discarding (e.g., refraining from further processing of) any other bounding boxes. Thetracking engine 108 may determine that a first bounding box in a first frame and a second bounding box in a second (e.g., subsequent) frame enclose the same object by determining that a similarity measure between the first bounding box and the second bounding box exceeds a threshold value. The similarity measure may be based on: (i) a visual similarity between the contents of the first bounding box and the second bounding box, and (ii) an overlap between the first bounding box and the second bounding box. In response to determining that the first bounding box and the second bounding box enclose the same object, thetracking engine 108 may determine that the first bounding box and the second bounding box are included in the same target object trajectory, e.g., thereby tracking the position of the object from the first frame to the second frame. - The cropping
engine 110 is configured to process theinput video 102 and the trackingdata 114 to generate one or more sub-videos 116. More specifically, the croppingengine 110 may generate arespective sub-video 116 corresponding to each target object trajectory specified by the trackingdata 114. As used throughout this specification, a "sub-video" refers to a video having a sequence of video frames, referred to as "sub-video frames", such that each sub-video frame of the sub-video is cropped from a respective video frame of thevideo 102. A sub-video frame of a sub-video 116 that is cropped from a corresponding video frame of thevideo 102 may include less than all of the corresponding video frame, e.g., such that the sub-video frame excludes at least a portion of the corresponding video frame (however, in some cases, a sub-video frame may include the entirety of a corresponding video frame of the video 102). In a particular example, a video frame may have dimensions a × b, where a is the width of the video frame (e.g., the number of pixels along the width spatial dimension of the video frame) and b is the height of the video frame (e.g., the number of pixels along the height spatial dimension of the video frame). In this example, the sub-video frame cropped from the video frame may have dimensions c × d, where c is less than a and d is less than b. - As used throughout this specification, the "dimensions" of a video frame (or a sub-video frame, or a bounding box in a video frame) may refer to data specifying: (i) the width of the video frame (e.g., the number of pixels along the width spatial dimension of the video frame), and (ii) the height of the video frame (e.g., the number of pixels along the height spatial dimension of the video frame).
- The cropping
engine 110 can generate a sub-video 116 corresponding to a target object trajectory of a target object in a variety of ways. A few example implementations of thecropping engine 110 are described in more detail next. - In one implementation, the cropping
engine 110 may extract a respective sub-video frame from each video frame corresponding to the target object trajectory by extracting a portion of the video frame based on the position of the target object in the video frame. For example, the target object trajectory may specify a respective bounding box for each video frame that encloses the target object in the video frame, and thecropping engine 110 may extract the portion of each video frame that is enclosed by the bounding box for the video frame. That is, the croppingengine 110 may generate each sub-video frame by extracting the portion of the corresponding video frame that is enclosed by the bounding box specified by the target object trajectory as enclosing the target object in the video frame. In this example, not according to the claims, thetracking engine 108 may be configured to generate target object trajectories that specify a bounding box having fixed dimensions (e.g., width and height) for each video frame, e.g., such that each of the extracted sub-video frames have the same dimensions. - In the implementation according to the claimed invention to generate the sub-video 116 corresponding to the target object trajectory, the cropping
engine 110 determines the dimensions (e.g., width and height) and position (e.g., center) of a "super" bounding box. In particular, the croppingengine 110 determines the dimensions and position of the super bounding box such that the target object is enclosed by the super bounding box in each video frame corresponding to the target object trajectory. For example, the target object trajectory may specify a respective bounding box for each video frame that encloses the target object in the video frame, and thecropping engine 110 may identify a super bounding box that encloses each of the bounding boxes specified by the target object trajectory. That is, the croppingengine 110 may identify a super bounding box that encloses the union of the set of bounding boxes specified by the target object trajectory (while still enclosing less than all of each video frame). After determining the dimensions and position of the super bounding box, the croppingengine 110 may generate each sub-video frame by extracting the portion of a corresponding video frame that is enclosed by the super bounding box. Generating the sub-video 116 using a super bounding box that is the same for each video frame may improve the visual quality of the sub-video 116, e.g., by better preserving the cinematographic effects of theoriginal video 102, e.g., panning and zooming. - In some cases,
certain sub-videos 116 generated by the croppingengine 110 may be "temporally overlapping", e.g., such two or more of the sub-videos include respective sub-video frames that are extracted from the same video frame of theinput video 102. This may occur, e.g., when thetracking engine 108 tracks two different instances of target objects through the same video frame in the video. - The
synthesis engine 112 is configured to process thesub-videos 116 and avideo template 118 to generate thetopical video 106. Thevideo template 118 defines a format for combiningmultiple sub-videos 116 into a composite video (e.g., the topical video 106) that includes each of the sub-videos. For example, the video template may define a partition of a two-dimensional (2-D) region (e.g., having a square or rectangular shape) into one or more sub-regions, referred to as "slots". Thesynthesis engine 112 generates thetopical video 106 by assigning each sub-video 116 to a respective slot in thevideo template 118. Thevideo template 118 may dynamically change over the duration of thetopical video 106, e.g., slots may be removed, slots may be added, or the sizes of slots may be changed over the duration of the topical video. - An
example video template 200 is illustrated with reference toFIG. 2 . Thevideo template 200 includes three slots - i.e.,slot 202,slot 204, andslot 206. Thesynthesis engine 112 may generate thetopical video 106 by assigning arespective sub-video 116 to each slot of thevideo template 200, e.g., such that the sub-videos in each slot are shown concurrently. In the example illustrated with reference toFIG. 2 , the target object type may be "vehicle", and each slot may be assigned a respective sub-video showing an instance of a vehicle. - In another example, the
video template 118 may be defined such that each sub-video 116 is shown consecutively, e.g., one after another, rather than any sub-videos being shown concurrently. - As part of generating the
topical video 106, thesynthesis engine 112 may modify a sub-video 116 prior to assigning it to a slot in thevideo template 118. For example, thesynthesis engine 112 may modify (e.g., reduce) the dimensions (e.g., width and/or the height) of the sub-video frames of a sub-video, e.g., to cause the dimensions of the sub-video to match the dimensions of a corresponding slot in thevideo template 118. As another example, thesynthesis engine 112 may reduce the duration of a sub-video (e.g., by removing one or more sub-video frames of the sub-video) to cause the duration of the sub-video to match the duration of a slot. The duration of a slot may refer to the length of time the slot is included in the template, e.g., before being removed from the template. - The
synthesis engine 112 may determine the assignment of thesub-videos 116 to the slots of thevideo template 118 in any of a variety of ways. In one example, thesynthesis engine 112 may randomly assignsub-videos 116 to respective slots of thevideo template 118. In another example, thesynthesis engine 112 may assign each sub-video 116 to a corresponding slot having dimensions that are most similar (e.g., from amongst all the slots) to the dimensions of the sub-video 116, e.g., to minimize the modifications applied to the sub-videos to cause them to fit their assigned slots. - In some cases, prior to assigning the
sub-videos 116 to slots in thevideo template 118, thesynthesis engine 112 may determine a respective score for each sub-video, where the score for a sub-video characterizes a quality of the sub-video. Thesynthesis engine 112 may determine the score for each sub-video based on one or more criteria, e.g., the sharpness of the sub-video and the amount of motion in the sub-video. The sharpness of a sub-video may be computed using any appropriate sharpness measure, e.g., by computing the maximum pixel intensity value in the sub-video frames after convolving the sub-video frames with a Laplacian kernel. The amount of motion in the sub-video may be computed, e.g., by computing the average value of a set of optical flow frames corresponding to the sub-video frames. Thesynthesis engine 112 may generate the overall score for a sub-video based on, e.g., a linear combination of a sharpness score characterizing the sharpness of the sub-video and a motion score characterizing the amount of motion in the sub-video. Thesynthesis engine 112 may generate higher scores for sub-videos that are sharper and include more motion. - The
synthesis engine 112 may use the scores for thesub-videos 116 to determine which sub-videos 116 should be included in thetopical video 106. For example, thesynthesis engine 112 may determine that only a proper subset of thesub-videos 116 having the highest scores should be included in thetopical video 106. As another example, thesynthesis engine 112 may determine that any sub-video 116 having a score that fails to satisfy a predefined threshold value (e.g., by exceeding the threshold value) should not be included in thetopical video 106. - In some implementations, the
system 100 may receive one ormore data elements 120 to be included in thetopical video 106, e.g., other thansub-videos 116 extracted from theinput video 102. Thedata elements 120 may include, e.g., images or portions of text. Thesynthesis engine 112 may include thedata elements 120 in thetopical video 106 by assigning the data elements to respective slots of thevideo template 118. - The
topical videos 106 generated by thevideo generation system 100 can be used in any of a variety of applications. A few example applications are described in more detail next. - In one example,
topical videos 106 generated by thevideo generation system 100 may be included in digital components that are provided in response to digital component requests, e.g., as described in more detail with reference toFIG. 3 andFIG. 4 . - In another example, the
video generation system 100 may process: (i) input videos that are generated, e.g., by one or more surveillance cameras (security cameras), and (ii)target object types 104 that are specified by a user, to generate correspondingtopical videos 106. The input videos generated by the surveillance cameras may be lengthy, e.g., having a duration of hours or days. Thevideo generation system 100 may enable rapid processing of videos generated by surveillance cameras to generate topical videos showing objects of interest to a user, e.g., pets, people, or vehicles. A surveillance camera may refer to a camera that is placed at a given position (e.g., beside the front door of a home) and continuously captures video of a region in the vicinity of the given position. - In another example, the
video generation system 100 may process: (i) input videos that are generated by a user device, e.g., a video camera of a smartphone, and (ii)target object types 104 that are specified by a user, to generate correspondingtopical videos 106. -
FIG. 3 shows an example digitalcomponent generation system 300 The digitalcomponent generation system 300 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented. - The digital
component generation system 300 is configured to receive one ormore keywords 302 and aninput video 102 from a content provider, and to generate one or moredigital components 306 that includetopical videos 106 relevant torespective keywords 302. The content provider may be, e.g., a publisher, advertiser, or other content source. - As used throughout this specification, the phrase digital component refers to a discrete unit of digital content or digital information that can include one or more of, e.g., images, video clips (e.g., topical videos), audio clips, multimedia clips, text segments, or uniform resource locators (URLs). A digital component can be electronically stored in a physical memory device as a single file or in a collection of files, and digital components can take the form of video files, audio files, multimedia files, image files, or text files and include streaming video, streaming audio, social network posts, blog posts, and/or advertising information, such that an advertisement is a type of digital component. A digital component can be enhanced with data from one or more sources (e.g., weather information, real time event information, or other information obtained from other sources).
- The
system 300 includes akeyword mapping engine 308 and a video generation system 100 (as described with reference toFIG. 1 ). In addition, thesystem 300 may be associated with a user interface (e.g., a graphical user interface, or any other appropriate sort of user interface) that enables the content provider to interact with thesystem 300, as will be described in more detail below. - The
keyword mapping engine 308 is configured to map eachkeyword 302 to a correspondingtarget object type 104 from a set of possible object types, e.g., in accordance with a predefined mapping from keywords to possible object types. For example, thekeyword mapping engine 308 may map the each of the keywords "spectacles", "specs", "reading glasses", "sunglasses", and "eyeglasses" to the possible object type of "glasses". The set of possible object types may be the set of object types that the tracking engine of thevideo generation system 100 is trained (or otherwise configured) to identify and track in videos. In some implementations, thesystem 300 may provide an indication of thetarget object types 104 corresponding to thekeywords 302 to the content provider, and request that the content provider "approve" (e.g., perform an action to accept) each of the target object types 104. - The
system 300 may use thevideo generation system 100 to generate one or more respectivetopical videos 106 corresponding to eachtarget object type 104, e.g., by processing thetarget object type 104 and theinput video 102 using thevideo generation system 100. Thevideo generation system 100 may generate multipletopical videos 106 corresponding to a singletarget object type 104, e.g., by generatingtopical videos 106 using different video templates from a set of possible video templates 310, by generatingtopical videos 106 using different combinations of sub-videos extracted from the input video, by generatingtopical videos 106 where the sub-videos of theinput video 102 are assigned to different combinations of slots in a video template, or by overlaying different audio soundtracks over thetopical videos 106. Thesystem 300 may enable the content provider to specify (e.g., through the user interface) various factors controlling the generation of thetopical videos 106, e.g., which video templates 310 and/or audio soundtracks should be used by thevideo generation system 100 to generate thetopical videos 106. - In addition to the
keywords 302 and theinput video 102, the content provider may also provide one or more other data elements (e.g., images or portions of text) to thesystem 300 to be included in thetopical videos 106. Thevideo generation system 100 may include the provided data elements in the topical videos, e.g., as described with reference toFIG. 1 . - The
system 300 may enable the content provider to select one or more of thetopical videos 106 to be included in respectivedigital components 306. Eachdigital component 306 may include atopical video 106 and other data elements (e.g., videos, images, portions of text, URLs) that may be provided by the content provider. - The content provider may provide the
digital components 306 generated by thesystem 300 to a digital component distribution system, e.g., the digitalcomponent distribution system 410 described with reference toFIG. 4 . The digital component distribution system can transmit thedigital components 306 in response to requests for digital components to be presented with electronic documents at user devices, e.g., in blocks alongside search results or on third party websites. - In a particular example illustrating the utility of the
system 300, a content provider may provide thesystem 300 with aninput video 102 that includes several hours of footage of a fashion show, and thekeyword 302 "handbag". Thesystem 300 may process theinput video 102 and thekeyword 302 to generate a corresponding set oftopical videos 106, each of which include multiple sub-videos of the input video that show instances of handbags, where the sub-videos are automatically edited into visually pleasing video templates, and omit portions of the input video, such that the populated video templates have a shorter duration (e.g., 30 second, 1 minute, 2 minutes, or another duration) than the duration of the input video 102 (e.g., the several hours of footage of the fashion show). The content provider may then select one or more of thetopical videos 106 to be included in digital components that are provided to the digital component distribution system. -
FIG. 4 is a block diagram of an example environment 400 in which a digitalcomponent distribution system 410 transmits digital components from adigital component database 416 for presentation with electronic documents A content provider may generate digital components using the digital component generation system 300 (described with reference toFIG. 3 ) and provide the generated digital components to the digitalcomponent distribution system 410. - The example environment 400 includes a
network 402, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. Thenetwork 402 connectselectronic document servers 404,client devices 406,digital component servers 408, and the digital component distribution system 410 (also referred to as the "distribution system" 410). The example environment 400 may include many differentelectronic document servers 404,client devices 406, anddigital component servers 408. - A
client device 406 is an electronic device that is capable of requesting and receiving resources over thenetwork 402.Example client devices 406 include personal computers, mobile communication devices (e.g., mobile phones), and other devices that can send and receive data over thenetwork 402. Aclient device 406 typically includes a user application, such as a web browser, to facilitate the sending and receiving of data over thenetwork 402, but native applications executed by theclient device 406 can also facilitate the sending and receiving of data over thenetwork 402. - An electronic document is data that presents a set of content at a
client device 406. Examples of electronic documents include webpages, word processing documents, portable document format (PDF) documents, images, videos, search results pages, and feed sources. Native applications (e.g., "apps"), such as applications installed on mobile, tablet, or desktop computing devices are also examples of electronic documents. Electronic documents can be provided toclient devices 406 by electronic document servers 404 ("Electronic Doc Servers"). For example, theelectronic document servers 404 can include servers that host publisher websites. In this example, theclient device 406 can initiate a request for a given publisher webpage, and theelectronic server 404 that hosts the given publisher webpage can respond to the request by sending machine executable instructions that initiate presentation of the given webpage at theclient device 406. - In another example, the
electronic document servers 404 can include app servers from whichclient devices 406 can download apps. In this example, theclient device 406 can download files required to install an app at theclient device 406, and then execute the downloaded app locally. - Electronic documents can include a variety of content. For example, an electronic document can include static content (e.g., text or other specified content) that is within the electronic document itself and/or does not change over time. Electronic documents can also include dynamic content that may change over time or on a per-request basis For example, a publisher of a given electronic document can maintain a data source that is used to populate portions of the electronic document. In this example, the given electronic document can include one or more tags or scripts that cause the
client device 406 to request content from the data source when the given electronic document is processed (e.g., rendered or executed) by aclient device 406. Theclient device 406 integrates the content obtained from the data source into the given electronic document to create a composite electronic document including the content obtained from the data source. - In some situations, a given electronic document can include one or more digital component tags or digital component scripts that reference the digital
component distribution system 410. In these situations, the digital component tags or digital component scripts are executed by theclient device 406 when the given electronic document is processed by theclient device 406. Execution of the digital component tags or digital component scripts configures theclient device 406 to generate a request for one or more digital components 412 (referred to as a "component request"), which is transmitted over thenetwork 402 to the digitalcomponent distribution system 410. For example, a digital component tag or digital component script can enable theclient device 406 to generate a packetized data request including a header and payload data. Thecomponent request 412 can include event data specifying features such as a name (or network location) of a server from which the digital component is being requested, a name (or network location) of the requesting device (e.g., the client device 406), and/or information that the digitalcomponent distribution system 410 can use to select one or more digital components provided in response to the request. Thecomponent request 412 is transmitted, by theclient device 406, over the network 402 (e.g., a telecommunications network) to a server of the digitalcomponent distribution system 410. - The
component request 412 can include event data specifying other event features, such as the electronic document being requested and characteristics of locations of the electronic document at which digital component can be presented. For example, event data specifying a reference (e.g., URL) to an electronic document (e.g., webpage) in which the digital component will be presented, available locations of the electronic documents that are available to present digital components, sizes of the available locations, and/or media types that are eligible for presentation in the locations can be provided to the digitalcomponent distribution system 410. Similarly, event data specifying keywords associated with the electronic document ("document keywords") or entities (e.g., people, places, or things) that are referenced by the electronic document can also be included in the component request 412 (e.g., as payload data) and provided to the digitalcomponent distribution system 410 to facilitate identification of digital components that are eligible for presentation with the electronic document. The event data can also include a search query that was submitted from theclient device 406 to obtain a search results page, and/or data specifying search results and/or textual, audible, or other visual content that is included in the search results. - Component requests 412 can also include event data related to other information, such as information that a user of the client device has provided, geographic information indicating a state or region from which the component request was submitted, or other information that provides context for the environment in which the digital component will be displayed (e.g., a time of day of the component request, a day of the week of the component request, a type of device at which the digital component will be displayed, such as a mobile device or tablet device). Component requests 412 can be transmitted, for example, over a packetized network, and the component requests 412 themselves can be formatted as packetized data having a header and payload data. The header can specify a destination of the packet and the payload data can include any of the information discussed above.
- The
component distribution system 410 chooses digital components that will be presented with the given electronic document in response to receiving thecomponent request 412 and/or using information included in thecomponent request 412. In some implementations, a digital component is selected (using the techniques described herein) in less than a second to avoid errors that could be caused by delayed selection of the digital component. For example, delays in providing digital components in response to acomponent request 412 can result in page load errors at theclient device 406 or cause portions of the electronic document to remain unpopulated even after other portions of the electronic document are presented at theclient device 406. Also, as the delay in providing the digital component to theclient device 406 increases, it is more likely that the electronic document will no longer be presented at theclient device 406 when the digital component is delivered to theclient device 406, thereby negatively impacting a user's experience with the electronic document. Further, delays in providing the digital component can result in a failed delivery of the digital component, for example, if the electronic document is no longer presented at theclient device 406 when the digital component is provided. - In some implementations, the digital
component distribution system 410 is implemented in a distributed computing system that includes, for example, a server and a set ofmultiple computing devices 414 that are interconnected and identify and distribute digital components in response torequests 412. The set ofmultiple computing devices 414 operate together to identify a set of digital components that are eligible to be presented in the electronic document from a corpus of millions of available digital components (DC1-x). The millions of available digital components can be indexed, for example, in adigital component database 416. Each digital component index entry can reference the corresponding digital component and/or include distribution parameters (DP1-DPx) that contribute to (e.g., condition or limit) the distribution/transmission of the corresponding digital component. For example, the distribution parameters can contribute to the transmission of a digital component by requiring that a component request include at least one criterion that matches (e.g., either exactly or with some pre-specified level of similarity) one of the distribution parameters of the digital component. - In some implementations, the distribution parameters for a particular digital component can include distribution keywords that must be matched (e.g., by electronic documents, document keywords, or terms specified in the component request 412) in order for the digital component to be eligible for presentation. In other words, the distribution parameters are used to trigger distribution (e.g., transmission) of the digital components over the
network 402. The distribution parameters can also require that thecomponent request 412 include information specifying a particular geographic region (e.g., country or state) and/or information specifying that thecomponent request 412 originated at a particular type of client device (e.g., mobile device or tablet device) in order for the digital component to be eligible for presentation. - The distribution parameters can also specify an eligibility value (e.g., ranking score, bid, or some other specified value) that is used for evaluating the eligibility of the digital component for distribution/transmission (e.g., among other available digital components), for example, by the component evaluation process. In some situations, the eligibility value can specify a maximum amount of compensation that a provider of the digital component is willing to submit in response to the transmission of the digital component (e.g., for each instance of specific events attributed to the presentation of the digital component, such as user interaction with the digital component).
- The identification of the eligible digital component can be segmented into
multiple tasks 417a-417c that are then assigned among computing devices within the set ofmultiple computing devices 414. For example, different computing devices in theset 414 can each analyze a different portion of thedigital component database 416 to identify various digital components having distribution parameters that match information included in thecomponent request 412. In some implementations, each given computing device in theset 414 can analyze a different data dimension (or set of dimensions) and pass (e.g., transmit) results (Res 1-Res 3) 418a-418c of the analysis back to the digitalcomponent distribution system 410. For example, theresults 418a-418c provided by each of the computing devices in theset 414 may identify a subset of digital components that are eligible for distribution in response to the component request and/or a subset of the digital components that have certain distribution parameters. The identification of the subset of digital components can include, for example, comparing the event data to the distribution parameters, and identifying the subset of digital components having distribution parameters that match at least some features of the event data. - The digital
component distribution system 410 aggregates theresults 418a-418c received from the set ofmultiple computing devices 414 and uses information associated with the aggregated results to: (i) select one or more digital components that will be provided in response to therequest 412, and (ii) determine transmission requirements for the one or more digital components. For example, the digitalcomponent distribution system 410 can select a set of winning digital components (one or more digital components) based on the outcome of one or more component evaluation processes. In turn, the digitalcomponent distribution system 410 can generate and transmit, over thenetwork 402, reply data 420 (e.g., digital data representing a reply) that enables theclient device 406 to integrate the set of winning digital components into the given electronic document, such that the set of winning digital components and the content of the electronic document are presented together at a display of theclient device 406. - In some implementations, the
client device 406 executes instructions included in thereply data 420, which configures and enables theclient device 406 to obtain the set of winning digital components from one or more digital component servers. For example, the instructions in thereply data 420 can include a network location (e.g., a Uniform Resource Locator (URL)) and a script that causes theclient device 406 to transmit a server request (SR) 421 to thedigital component server 408 to obtain a given winning digital component from thedigital component server 408. In response to the request, thedigital component server 408 will identify the given winning digital component specified in the server request 421 (e.g., within a database storing multiple digital components) and transmit, to theclient device 406, digital component data (DC Data) 422 that presents the given winning digital component in the electronic document at theclient device 406. - To facilitate searching of electronic documents, the environment 400 can include a
search system 450 that identifies the electronic documents by crawling and indexing the electronic documents (e.g., indexed based on the crawled content of the electronic documents). Data about the electronic documents can be indexed based on the electronic document with which the data are associated. The indexed and, optionally, cached copies of the electronic documents are stored in a search index 452 (e.g., hardware memory device(s)). Data that are associated with an electronic document is data that represents content included in the electronic document and/or metadata for the electronic document. -
Client devices 406 can submit search queries to thesearch system 450 over thenetwork 402. In response, thesearch system 450 accesses thesearch index 452 to identify electronic documents that are relevant to the search query. Thesearch system 450 identifies the electronic documents in the form of search results and returns the search results to theclient device 406 in a search results page. A search result is data generated by thesearch system 450 that identifies an electronic document that is responsive (e.g., relevant) to a particular search query, and includes an active link (e.g., hypertext link) that causes a client device to request data from a specified network location (e.g., URL) in response to user interaction with the search result. An example search result can include a web page title, a snippet of text or a portion of an image extracted from the web page, and the URL of the web page. Another example search result can include a title of a downloadable application, a snippet of text describing the downloadable application, an image depicting a user interface of the downloadable application, and/or a URL to a location from which the application can be downloaded to theclient device 406. In some situations, thesearch system 450 can be part of, or interact with, an application store (or an online portal) from which applications can be downloaded for install at aclient device 406 in order to present information about downloadable applications that are relevant to a submitted search query. Like other electronic documents, search results pages can include one or more slots in which digital components (e.g., advertisements, video clips, audio clips, images, or other digital components) can be presented. - To select a digital component to be transmitted in response to a component request, the
distribution system 410 may identify a set of digital components that are eligible to be transmitted in response to the component request. Thedistribution system 410 may then select one or more of the eligible digital components to be transmitted through, e.g., an auction procedure. In some implementations, thedistribution system 410 performs an auction procedure by ranking the eligible digital components in accordance with their respective eligibility values, and selecting one or more highest-ranked digital components to be transmitted in response to the component request. -
FIG. 5 is a flow diagram of anexample process 500 for generating a topical video. For convenience, theprocess 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a video generation system, e.g., thevideo generation system 100 ofFIG. 1 , appropriately programmed in accordance with this specification, can perform theprocess 500. - The system receives: (i) an input video including a sequence of video frames, and (ii) data indicating a target object type (502). For example, the system may receive data specifying a keyword (e.g., "sunglasses"), and then map the received keyword to a corresponding target object type (e.g., "glasses") in accordance with a mapping from keywords to a predefined set of possible target object types. Optionally, the system may also receive one or more additional data elements, e.g., images, portions of text, or both.
- The system processes the input video to generate tracking data that identifies and tracks visual locations of one or more instances of target objects of the target object type in the input video (504). For example, for each instance of a target object of the target object type in the input video, the system may determine a respective bounding box that encloses the instance of the target object in each of multiple video frames of the input video. The system may use any appropriate object detection and tracking technique to generate the tracking data, e.g., as described with reference to
FIG. 1 . - The system generates multiple sub-videos based on the input video and the tracking data (506). For each sub-video, the system may generate a respective sequence of sub-video frames that are each cropped from a respective video frame of the input video to include (i) less than all content of the respective video frame and (ii) a respective instance of a given target object from among the identified target objects of the target object type. In some implementations, instead of each of the sub-video frames being cropped to include less than all content of the respective video frame, at least one of the sub-video frames is cropped from at least one respective video frame of the input video to include less than all content of the at least one respective video frame. In such implementations, one or more of the sub-video frames may comprise all content of the respective video frames, for example because a target object occupies all of the respective video frames or most of the respective video frames in those respective video frames. A sub-video frame may comprise a subset of all content of the respective video frame, with the subset being less than or equal to all content of the respective video frame. In some implementations, the system may generate the sequence of sub-video frames of a sub-video corresponding to an instance of a given target object by cropping a respective bounding box that encloses the instance of the given target object from each of multiple video frames of the input video. The bounding boxes enclosing instance of the given target object may be specified by the tracking data. In some implementations, to generate the sequence of sub-video frames of a sub-video corresponding to an instance of a given target object, the system may determine, based on the tracking data, a same bounding box that encloses the instance of the given target object in each of multiple video frames of the input video. The system may then generate the sequence of sub-video frames of the sub-video by cropping the same bounding box from each of the multiple video frames of the input video.
- In some cases, certain sub-videos may be "temporally overlapping", e.g., such two or more of the sub-videos include respective sub-video frames that are extracted from the same video frame of the input video. This may occur, e.g., when the system tracks two different instances of target objects through the same video frame in the input video.
- The system generates an output video that includes the sub-videos (508). For example, to generate the output video, the system may assign each sub-video to a respective slot in a video template that defines a format for combining the sub-videos. As part of generating the output video, the system may determine a respective score for each sub video (e.g., based on the sharpness of the sub-video), and select a proper subset of the sub-videos for inclusion in the output video (e.g., a predefined number of sub-videos with the highest scores). In some cases, the output video may concurrently show at least two of the sub-videos, while in other cases, the output video may show each of the sub-videos consecutively, e.g., one after another. As part of generating the output video, the system may assign each of one or more additional data elements (e.g., textual or image data elements, as described with reference to 502) to a respective slot in the video template.
- The output video generated by the system may be included in a digital component that is provided in response to a digital component request and subsequently presented alongside search results or on a third-party webpage.
- This specification uses the term "configured" in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.
- Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, e.g., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.
- The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
- A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.
- In this specification the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.
- The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.
- Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.
- Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.
- To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.
- Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, e.g., inference, workloads.
- Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework, a Microsoft Cognitive Toolkit framework, an Apache Singa framework, or an Apache MXNet framework.
- Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.
- The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.
- Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.
Claims (9)
- A method performed by one or more data processing apparatus, the method comprising:receiving: (i) an input video (102) comprising a sequence of video frames, and (ii) data indicating a target object type (104);processing the input video to generate tracking data (114) that identifies and tracks visual locations of one or more instances of target objects of the target object type in the input video;generating a plurality of sub-videos (116) based on the input video and the tracking data, including:
for each sub-video, generating a respective sequence of sub-video frames that are each extracted from a respective video frame of the input video to include a respective instance of a given target object from among the identified target objects of the target object type, wherein at least one of the sub-video frames is cropped from a respective video frame of the input video to include less than all content of the respective video frame; andgenerating an output video (106) that comprises the plurality of sub-videos,wherein for each sub-video, generating the respective sequence of sub-video frames comprises:determining, based on the tracking data, a same bounding box that encloses the instance of the given target object corresponding to the sub-video in each of multiple video frames of the input video, comprising determining the dimensions and position of the same bounding box such that the given target object is enclosed by the same bounding box in each of the multiple video frames; andcropping the same bounding box from each of the multiple video frames of the input video. - The method of any preceding claim, wherein a sub-video frame of a first sub-video (116) and a sub-video frame of a second sub-video (116) are each cropped from a same video frame of the input video (102).
- The method of any preceding claim, wherein generating the output video (106) that comprises the plurality of sub-videos (116) comprises:
assigning each sub-video of the plurality of sub-videos to a respective slot (202, 204, 206) in a video template (118, 200, 310) that defines a format for combining the plurality of sub-videos. - The method of claim 3, further comprising:receiving one or more additional data elements (120) other than the sub videos (116) extracted from the input video (102); andassigning each additional data element to a respective slot (202, 204, 206) in the video template (118, 200, 310),wherein each additional data element comprises image data, text data, or both.
- The method of any preceding claim, wherein the output video (106) concurrently shows at least two of the sub-videos (116) of the plurality of sub-videos.
- The method of any preceding claim, further comprising:receiving a request (412) for a digital component (306);determining that a digital component that includes the output video (106) is responsive to the request; andproviding the digital component that includes the output video in response to the request to be presented alongside search results or on a third-party webpage.
- The method of any preceding claim, wherein receiving data indicating the target object type (104) comprises:receiving data specifying a keyword (302); andmapping the keyword to the target object type in accordance with a mapping from keywords to a predefined set of possible target object types.
- A system comprising one or more data processing apparatus and one or more storage devices storing instructions that, when executed by the one or more data processing apparatus, cause the one or more data processing apparatus to perform operations comprising the method of any preceding claim.
- A computer-readable storage medium storing instructions that, when executed by one or more data processing apparatus, cause the one or more data processing apparatus to perform operations comprising the method of any of claims 1-7.
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/US2020/038963 WO2021262137A1 (en) | 2020-06-22 | 2020-06-22 | Generating videos |
Publications (2)
Publication Number | Publication Date |
---|---|
EP3956810A1 EP3956810A1 (en) | 2022-02-23 |
EP3956810B1 true EP3956810B1 (en) | 2023-12-27 |
Family
ID=71608058
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP20740118.3A Active EP3956810B1 (en) | 2020-06-22 | 2020-06-22 | Generating videos |
Country Status (5)
Country | Link |
---|---|
US (2) | US11915724B2 (en) |
EP (1) | EP3956810B1 (en) |
JP (1) | JP7299327B2 (en) |
CN (1) | CN114080818B (en) |
WO (1) | WO2021262137A1 (en) |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP4246465B1 (en) * | 2022-03-16 | 2024-01-03 | Axis AB | Improved masking of objects in an image stream |
EP4276772A1 (en) * | 2022-05-12 | 2023-11-15 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Method, computer program and system for analysing one or more moving objects in a video |
CN115278338B (en) * | 2022-07-28 | 2023-09-19 | 广州市百果园信息技术有限公司 | Video wallpaper processing method and device, equipment and medium thereof |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2005094799A (en) | 2004-11-15 | 2005-04-07 | Chuo Electronics Co Ltd | Integrated video display device |
US10218954B2 (en) | 2013-08-15 | 2019-02-26 | Cellular South, Inc. | Video to data |
US20160034956A1 (en) * | 2014-07-29 | 2016-02-04 | Facebook, Inc. | Presenting targeting criteria options for inclusion in targeting criteria associated with content items |
EP3249651B1 (en) * | 2016-05-23 | 2018-08-29 | Axis AB | Generating a summary video sequence from a source video sequence |
JP2019140526A (en) | 2018-02-09 | 2019-08-22 | キヤノン株式会社 | Information processing apparatus, information processing method, program, and storage medium |
CN110188719B (en) * | 2019-06-04 | 2022-03-29 | 北京字节跳动网络技术有限公司 | Target tracking method and device |
CN110139158B (en) * | 2019-06-21 | 2021-04-02 | 上海摩象网络科技有限公司 | Video and sub-video generation method and device, and electronic equipment |
-
2020
- 2020-06-22 US US17/423,623 patent/US11915724B2/en active Active
- 2020-06-22 WO PCT/US2020/038963 patent/WO2021262137A1/en unknown
- 2020-06-22 JP JP2021544221A patent/JP7299327B2/en active Active
- 2020-06-22 CN CN202080011238.8A patent/CN114080818B/en active Active
- 2020-06-22 EP EP20740118.3A patent/EP3956810B1/en active Active
-
2024
- 2024-01-23 US US18/420,509 patent/US20240161783A1/en active Pending
Also Published As
Publication number | Publication date |
---|---|
US11915724B2 (en) | 2024-02-27 |
CN114080818B (en) | 2024-07-16 |
US20240161783A1 (en) | 2024-05-16 |
US20230095856A1 (en) | 2023-03-30 |
JP7299327B2 (en) | 2023-06-27 |
JP2022541361A (en) | 2022-09-26 |
CN114080818A (en) | 2022-02-22 |
EP3956810A1 (en) | 2022-02-23 |
WO2021262137A1 (en) | 2021-12-30 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20240161783A1 (en) | Generating videos | |
JP6334696B2 (en) | Hashtag and content presentation | |
US9013553B2 (en) | Virtual advertising platform | |
US9830522B2 (en) | Image processing including object selection | |
CN111178970B (en) | Advertisement putting method and device, electronic equipment and computer readable storage medium | |
US10721330B2 (en) | Content delivery acceleration system | |
US20230186348A1 (en) | Image Recognition Based Content Item Selection | |
US9665595B2 (en) | Image processing client | |
US10445753B1 (en) | Determining popular and trending content characteristics | |
JP6514314B2 (en) | Method, machine readable storage and system for determining whether to send a callout to a bidder in an online content auction. | |
Mei et al. | ImageSense: Towards contextual image advertising | |
US11455299B1 (en) | Providing content in response to user actions | |
US20220148299A1 (en) | Image extension neural networks | |
EP2751782A1 (en) | Virtual advertising platform | |
US10140631B2 (en) | Image processing server | |
US11336737B2 (en) | Opt-out compliance | |
US9053129B1 (en) | Content item relevance based on presentation data | |
US9466029B1 (en) | Demographic inference calibration | |
US20150006300A1 (en) | Content distribution through an exchange | |
US20160260125A1 (en) | Systems and Methods for Cold-start and Continuous-learning via Evolutionary Explorations | |
US10346519B1 (en) | Selecting content based on entities | |
CN110622159A (en) | Improving opt-out compliance |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: UNKNOWN |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
17P | Request for examination filed |
Effective date: 20210630 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R079 Ref document number: 602020023416 Country of ref document: DE Free format text: PREVIOUS MAIN CLASS: G06K0009000000 Ipc: G06V0020400000 Ref country code: DE Free format text: PREVIOUS MAIN CLASS: G06K0009000000 |
|
DAV | Request for validation of the european patent (deleted) | ||
DAX | Request for extension of the european patent (deleted) | ||
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G06V 20/52 20220101ALI20230822BHEP Ipc: G06V 20/40 20220101AFI20230822BHEP |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
INTG | Intention to grant announced |
Effective date: 20230928 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE PATENT HAS BEEN GRANTED |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602020023416 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
P01 | Opt-out of the competence of the unified patent court (upc) registered |
Effective date: 20240116 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20240328 |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG9D |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231227 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231227 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231227 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20240328 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231227 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231227 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20240327 |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20231227 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 1645258 Country of ref document: AT Kind code of ref document: T Effective date: 20231227 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231227 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231227 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231227 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20240327 Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231227 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231227 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231227 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20240427 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: GB Payment date: 20240627 Year of fee payment: 5 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: DE Payment date: 20240627 Year of fee payment: 5 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231227 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231227 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231227 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231227 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231227 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231227 Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231227 Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20240427 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231227 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231227 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231227 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: FR Payment date: 20240625 Year of fee payment: 5 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231227 Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20240429 |