WO2018124794A1 - Système et procédé de synthèse d'images basés sur un travail à la caméra - Google Patents

Système et procédé de synthèse d'images basés sur un travail à la caméra Download PDF

Info

Publication number
WO2018124794A1
WO2018124794A1 PCT/KR2017/015698 KR2017015698W WO2018124794A1 WO 2018124794 A1 WO2018124794 A1 WO 2018124794A1 KR 2017015698 W KR2017015698 W KR 2017015698W WO 2018124794 A1 WO2018124794 A1 WO 2018124794A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
video clip
source
synthesis
dynamic
Prior art date
Application number
PCT/KR2017/015698
Other languages
English (en)
Korean (ko)
Inventor
천솔지
서일경
Original Assignee
(주)잼투고
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by (주)잼투고 filed Critical (주)잼투고
Publication of WO2018124794A1 publication Critical patent/WO2018124794A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/272—Means for inserting a foreground image in a background image, i.e. inlay, outlay
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/62—Control of parameters via user interfaces
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/265—Mixing

Definitions

  • the present invention relates to a camerawalk-based video synthesis system and a video synthesis method, and more particularly, to a video synthesis system and a video synthesis method for automatically synthesizing a dynamic video clip of video content having a similar camerawork through a data camerawork. It is about.
  • the object synthesis in the image synthesis method is to synthesize the object of the other video content to the original video content, and can exemplify the synthesized image of the object image of the weather caster whose background image is separated, such as the weather forecast news, into the virtual studio background image.
  • the CG background image and the object image of the weather caster with the real world camera are combined.
  • the weather caster object image is separated from the original background image by a chroma key technique and synthesized into the CG background image.
  • a method of compositing objects of video content in which different people appear may include synthesizing a second person object image of the second video content into the first video content, or composing the first person object image and the second video content of the first video content. 2
  • Combining a person object image into a new background image does not require any technology compared to the background synthesis described above.
  • the first person object image and the second person object image are naturally synthesized from the synthesized video content.
  • the first video content is photographed while the camera is panning and the horizontal axis moves around the first person, for example, while the second video content is photographed by the fixed camera.
  • the first person video content and the second person video content are synthesized into one composite video content, an unnatural composite image is generated because different camera walks are photographed.
  • UGCs are being produced and distributed through video content providing services such as YouTube.
  • Some UGC creators are generating high advertising revenue through high quality UGC, and a variety of industries are developing that leverage the power of high quality UGC.
  • UGC production surges the life cycle of UGC's attention to content consumers is shortening.
  • UGC creators have to upload more UGC more often.
  • One way to cope with this problem is to extend the vitality of existing UGCs by increasing the value and novelty of existing UGCs by reproduction of the already produced UGC.
  • UGC reproduction is not enough to meet the high standards of content consumers because it merely combines or displays parallel video clips of multiple UGCs in time.
  • Japanese Patent No. 4140006 of FUJINON CORP is a shooting system for synthesizing a background image and an object image by synchronizing the photographing camera of the background image and the photographing camera of the object image. Start the system. According to such a shooting system, there is an advantage of generating a high quality composite image to which dynamic camerawork is applied through synchronization control.
  • a photographing system has taken a background image and an object image according to a photographing plan on the premise of a synthesized image, and there is a problem that it cannot be applied to generating a dynamic composite image between UGCs without prescribing the image.
  • two sets of automatic control cameras are required for background and portrait shooting, the cost of generating a composite image increases, which is suitable for producing a high-cost broadcasting program but not for producing a low-cost UGC composite image. have.
  • United States Patent No. 7,843,510 entitled “Method and System for Combining Video Sequences with Spatio-temproal Alignment” by autoimmune Polytechnique Federale de Lausanne is one of the leading and trailing athletes in the field of broadcast broadcasts such as skiing.
  • a video synthesis system for displaying on a screen is disclosed. According to this video synthesizing system, viewers can easily visually check the record difference between the two players by providing a composite image of the object video of the player who played the game first and the object video of the player who is currently playing in the recording game. As a result, the camera tracked object images are combined into a single composite image without discomfort.
  • a video synthesizing system for synthesizing the vocal moving images of the same piece of music in a tile form.
  • the video synthesis system automatically searches for a plurality of videos that share the same characteristics (songs of the same music) among numerous videos, and automatically synthesizes them into a single composite video, thereby reducing the cost and time required for video synthesis. Provide effect.
  • the camerawalk based image synthesis system and image synthesis method according to the present invention automatically search for another composite target dynamic video clip having a dynamic camera walk similar to the source dynamic video clip of the source auto-shoot video content through the similarity of camera control parameters.
  • a composite video clip that synthesizes the source object image and the target object image a high quality composite video clip having dynamic camerawork can be generated without discomfort and can be similar among a number of video contents such as UGC.
  • Content consumers can eliminate the time and effort required to search for video content with camerawalks, and recreate it as new composite video content by adding new performers to the video content that has already been produced and published to the public, in a method of object synthesis. Continuous attention It aims to extend the vitality of pre-produced video content.
  • the synthesis target in a plurality of composite target content each comprising a target video object video composition belonging to the category of dynamic camera walk similar to the source dynamic video clip Another object is to generate a composite image of a dynamic camera walk in which a plurality of performers appear by, for example, synthesizing the video contents of a single performer by extracting an object and generating a single composite image.
  • Camera walk-based image synthesis system and image synthesis method according to an embodiment of the present invention, according to the user's synthesis request for the source auto-shooting video content, extracts a plurality of dynamic camera walk shooting section as a source dynamic video clip and each section
  • the object synthesis is automatically performed for all dynamic shooting sections of the source auto-shooting video content by searching and synthesizing the synthesized video content including all the dynamic video clips in the range similar to the camera walk of the camera.
  • Camera walk-based image synthesis system and image synthesis method by receiving a new background image selected by the user to synthesize the source object image and the target object image to generate a composite video clip to compare with the source video content Another goal is to create a reconstructed composite image with a large change and to continuously supply video content that is not eaten by content consumers.
  • Camera walk-based image synthesis system and image synthesis method by projecting a planar background to a three-dimensional background projection surface having a curvature, by synthesizing the source object image and the target object image, dynamic camera walk Another object is to create a high-quality composite image with a background image having an appropriate perspective with the object images.
  • Camera walk-based image synthesis system and image synthesis method according to an embodiment of the present invention, by first searching and synthesizing the auto-shoot video content from the performer of the source auto-shoot video content and other performers with a social relationship online Another goal is to automatically create custom composite video content.
  • Camera walk-based image synthesis system and image synthesis method according to another embodiment of the present invention, and further stores the content genre information associated with the auto-shoot video content in the storage module, and has the same content genre information as the source auto-shoot video content
  • Another objective is to automate the compositing of video content that shares the same genre attributes without the user having to directly view and verify the properties of the auto-shoot video content by first searching the dynamic video clip of the auto-shoot video content as a composite dynamic video clip.
  • the background music information is further stored in the storage module associated with the auto-shoot video content, and has the same background music information as the source auto-shoot video content
  • a composite video clip is created by chorusing or collaborating by naturally synthesizing video content that performs a song or instrumental performance, for example, using the same music as background music. For other purposes.
  • the storage module further stores the upload time information in association with the auto-shoot video content, and at a time close to the upload time of the source auto-shoot video content Synthesizing video content uploaded at similar times or synthesizing the most recent video content by first retrieving a dynamic video clip of auto-shoot video content with upload time information or with the most recent upload time information as a composite dynamic video clip. For other purposes.
  • the storage module further stores the online social relationship between upload users or video performers, uploading user or video performer of the source auto-shoot video content
  • Another object of the present invention is to generate a composite video clip based on a user relationship by first searching a dynamic video clip of an auto-shooting video content of an uploading user or a video performer having a social relationship online with a composite target dynamic video clip.
  • the camera-walk-based image synthesis system is a dynamic video which is a section captured by an automatic control camera controlled by at least one camera control parameter of pan control, tilt control or moving control.
  • a storage module for storing a plurality of auto-shoot video contents including a clip in association with the camera control parameter;
  • a content providing module for providing one source auto-shooting video content of the plurality of auto-shooting video contents to a user terminal through a communication network;
  • a retrieval module for retrieving a synthesis target dynamic video clip of the synthesis target auto captured video content having the matching control parameter as the camera control parameter;
  • a synthesis module for generating a composite video clip to include at least a source object image of the source dynamic video clip and a synthesis object image of the
  • the search module further searches for synthesizing dynamic video clips in the plurality of auto-captured video contents having the matching control parameter, and the synthesizing module. And generating a composite video clip such that the source object image of the source dynamic video clip and the synthesis object image of the plurality of synthesis target video clips are included.
  • the search module receives an object synthesis request for the source auto-captured video content from the user terminal, and the first section of the first interval source dynamic video clip.
  • a first interval reference control parameter of the interval reference control parameter to the nth interval (n is an integer of 2 or more) source dynamic video clip, the first interval reference control parameter to the nth interval reference control parameter respectively corresponding to the first interval reference control parameter;
  • the first interval synthesis video clip including the first interval synthesis object image of the first interval synthesis target video clip to the nth interval source object image and the nth interval synthesis target dynamic video of the nth interval source dynamic video clip Respectively generating an n-th section composite video clip including an n-th section synthesis target object image of the clip, and the content providing module is configured to generate the synthesized video content including the first to n-th section composite video clip. It is characterized in that provided to the user terminal.
  • the synthesis module is to generate the composite video clip by synthesizing the source object image and the target object image to the background image of the dynamic video clip. It features.
  • Camera-based image synthesis system further comprising a background image input module for receiving a user selected background image from the user terminal, wherein the composition module, the user selected background image And synthesizing a source object image and the synthesis target object image to generate the composite video clip.
  • the synthesizing module forms a background projection surface in a virtual space having a predetermined height and a concave center portion in a virtual space, and at the virtual projection point on the virtual space. Projecting at least a portion of the user-selected background image on a background projection surface, arranging the source object image and the composite object image between the virtual projection point on the virtual space and the background projection surface, and being disposed in the virtual space And generating at least a portion of the user-selected background image, at least a portion of the source object image, and at least a portion of the synthesis target object image in an angle of view of the camera to generate the composite video clip.
  • the search module preferentially searches for the auto-shoot video content played by the performer of the source auto-shoot video content and other performers who have a social relationship online. Characterized in that.
  • the video synthesis system includes a dynamic video clip that is a section captured by an automatic control camera controlled by at least one of the camera control parameters of pan control, tilt control or moving control. Storing a plurality of auto-shoot video content in association with the camera control parameter; Providing, by the image synthesizing system, a source auto-captured video content of any one of the plurality of auto-captured video contents to a user terminal through a communication network; Receiving, by the image synthesis system, an object synthesis request for a source dynamic video clip of the source auto-capture video content from the user terminal; Acquiring, by the image synthesis system, the camera control parameter of the source dynamic video clip as a reference control parameter; Retrieving, by the image synthesis system, a synthesis target dynamic video clip of the synthesis target auto-photographed video content having a registration control parameter which is the camera control parameter which is the same as or similar to the reference control parameter; Generating a composite video
  • the step of retrieving the dynamic video clip to be synthesized, the image synthesis system synthesizes each of a plurality of the auto-shoot video content having the matching control parameter; Retrieving target dynamic video clips and generating the composite video clip, wherein the image synthesis system includes the source object image of the source dynamic video clip and the composite object images of the plurality of composite object dynamic video clips. Generate a composite video clip for inclusion.
  • the acquiring of the reference control parameter may include: controlling, by the image synthesis system, a first interval reference control parameter of a first interval source dynamic video clip to an nth interval (n is an integer of 2 or more).
  • the image synthesis system includes first to nth interval control parameters corresponding to the first to nth interval reference control parameters, respectively; Automated Shooting of Composite Objects with Matched Control Parameters Retrieving the first to n-th composite video clip of the video content and generating the composite video clip, and the synthesizing system may include generating a first video of the first interval source dynamic video clip.
  • the user terminal to synthesize video content including a through n-th section composite video clip It characterized in that it gives.
  • the step of generating the composite video clip, the image synthesis system in the background image of the source dynamic video clip, the source object image and the synthesis target object image To synthesize the synthesized video clip.
  • Camerawork-based image synthesis method further comprises the step of receiving a user selected background image from the user terminal from the image synthesis system, the step of generating the composite video clip, the The image synthesis system generates the composite video clip by synthesizing the source object image and the synthesis target object image to the user-selected background image.
  • the step of generating the composite video clip, the image synthesis system to form a background projection surface formed in a curved surface having a predetermined height and a concave central portion in the virtual space First substep; A second sub-step of the image synthesis system projecting at least a portion of the user-selected background image on the background projection surface at a virtual projection point on the virtual space; A third sub-step of the image synthesizing system disposing the source object image and the synthesis target object image between the virtual projection point and the background projection surface on the virtual space; And at least a portion of the user-selected background image, at least a portion of the source object image, and at least a portion of the synthesis target object image in an angle of view of the virtual camera in which the image synthesis system is disposed in the virtual space.
  • the fourth sub-step of generating a characterized in that comprises a.
  • the step of searching for the dynamic video clip to be synthesized, the image synthesis system is another performer having an online social relationship with the performer of the source auto-shoot video content Search for the auto-captured video content which appeared in the first place.
  • the camerawalk based image synthesis system and image synthesis method according to the present invention automatically search for another composite target dynamic video clip having a dynamic camera walk similar to the source dynamic video clip of the source auto-shoot video content through the similarity of camera control parameters.
  • a composite video clip that synthesizes the source object image and the target object image a high quality composite video clip having dynamic camerawork can be generated without discomfort and can be similar among a number of video contents such as UGC.
  • Content consumers can eliminate the time and effort required to search for video content with camerawalks, and recreate it as new composite video content by adding new performers to the video content that has already been produced and published to the public, in a method of object synthesis. Continuous attention To prolong the vitality of the prebuilt video content.
  • the synthesis target in a plurality of composite target content each comprising a target video object video composition belonging to the category of dynamic camera walk similar to the source dynamic video clip By extracting an object and generating it as a single composite image, for example, a single performer synthesizes video contents each appeared, thereby providing an effect of generating a composite image of a dynamic camera walk in which a plurality of performers appear.
  • Camera walk-based image synthesis system and image synthesis method according to an embodiment of the present invention, according to the user's synthesis request for the source auto-shooting video content, extracts a plurality of dynamic camera walk shooting section as a source dynamic video clip and each section
  • the object synthesis is automatically performed for all dynamic shooting sections of the source auto-shooting video content by searching and compositing the synthesis-target video content including all the dynamic video clips in the range similar to the camera walk of the camera. to provide.
  • Camera walk-based image synthesis system and image synthesis method by receiving a new background image selected by the user to synthesize the source object image and the target object image to generate a composite video clip to compare with the source video content
  • a new background image selected by the user to synthesize the source object image and the target object image to generate a composite video clip to compare with the source video content
  • Camera walk-based image synthesis system and image synthesis method by projecting a planar background to a three-dimensional background projection surface having a curvature, by synthesizing the source object image and the target object image, dynamic camera walk As a result, the background image has an appropriate perspective with the object images, thereby creating a high quality composite image that changes.
  • Camera walk-based image synthesis system and image synthesis method according to an embodiment of the present invention, by first searching and synthesizing the auto-shoot video content from the performer of the source auto-shoot video content and other performers with a social relationship online Provides the effect of automatically creating custom composite video content.
  • Camera walk-based image synthesis system and image synthesis method according to another embodiment of the present invention, and further stores the content genre information associated with the auto-shoot video content in the storage module, and has the same content genre information as the source auto-shoot video content
  • the background music information is further stored in the storage module associated with the auto-shoot video content, and has the same background music information as the source auto-shoot video content
  • a composite video clip is created by chorusing or collaborating by naturally synthesizing video content that performs a song or instrumental performance, for example, using the same music as background music.
  • the storage module further stores the upload time information in association with the auto-shoot video content, and at a time close to the upload time of the source auto-shoot video content Synthesizing video content uploaded at similar times or synthesizing the most recent video content by prioritizing dynamic video clips of auto-shoot video content with upload time information or with the most recent upload time information as composite dynamic video clips. Can be provided.
  • the storage module further stores the online social relationship between upload users or video performers, uploading user or video performer of the source auto-shoot video content
  • the user may provide an effect of generating a composite video clip based on the user relationship by first searching the dynamic video clip of the auto-shooting video content of the uploading user or the video performer having a social relationship online with the composite target dynamic video clip.
  • FIG. 1 is a block diagram showing an image synthesis system according to the present invention.
  • FIG. 2 is a block diagram showing an automatic control camera according to an embodiment of the present invention.
  • FIG. 3 is a data structure diagram of a camera control parameter according to an embodiment of the present invention.
  • FIG. 4 is a video clip structure diagram of auto-shoot video content according to an embodiment of the present invention.
  • FIG. 5 is an exemplary diagram of an object composition request input interface in a user terminal according to an embodiment of the present invention.
  • FIG. 6 is a block diagram showing image synthesis according to an embodiment of the present invention.
  • FIG. 7 is a flowchart illustrating an image synthesis method according to the present invention.
  • the description that a part “includes” an element means that the element may further include other elements, except for the absence of a special objection thereto.
  • the terms “.. module”, “.. unit” and “.. system” described in the specification mean a unit that processes at least one function or operation, which is hardware or software or a combination of hardware and software. It may be implemented, and may be included in one device or each other device.
  • Camera walk based image synthesis system 100 is configured to include a storage module 110, the content providing module 120, the search module 130 and the synthesis module 140 as shown in FIG. .
  • the storage module 110 includes a plurality of auto-captured video contents including a dynamic video clip which is a section captured by the auto control camera 301 controlled by at least one camera control parameter of pan control, tilt control or moving control. And store them in association with the camera control parameters.
  • the storage module 110 may be a conventional storage device for storing data, and may include an optical storage device, a magnetic storage device, a semiconductor storage device, and the like.
  • the storage module 110 may be a storage device installed in the image synthesis system 100 or may be a storage device connected through a network.
  • the camera control parameter refers to a control command for changing the position of the automatic control camera 301 to be controlled in a mechanical manner or changing the direction of the camera in a mechanical manner.
  • the automatic control camera 301 moves horizontally or vertically along a rail according to camera movement control in a specific time interval of the camera control parameter.
  • the automatic control camera 301 rotates in the left and right directions so that the camera orientation angle changes.
  • the automatic control camera 301 rotates in the vertical direction to change the camera orientation angle.
  • the zoom of the automatic control camera 301 may be adjusted through the zoom control.
  • the dynamic video clip of the present invention does not include a video clip to which only the zoom control is applied.
  • the dynamic video clip may be a pan controlled autonomous camera 301, a tilt controlled autonomous camera 301, a moving controlled autonomous camera 301, or a pan and tilt controlled autonomous camera 301.
  • the portion that is not the dynamic video clip is a static video clip and does not involve changes in camerawork with time, a composite image can be easily produced by a conventional image synthesis method.
  • the auto-captured video content in the present invention refers to video content photographed by the automatic control camera 301 in which the photographing position or the photographing angle is controlled by a control command.
  • video captured by the camera held in the hands of the photographer The content does not correspond to the auto-shoot video content of the present invention.
  • FIG. 2 illustrates a camera automatic control system.
  • Utility Model Application No. 2016-0004490, “Stage and Studio Control System,” filed by the applicant of the present patent application discloses an embodiment of such a camera automatic control system.
  • the automatic control camera 301 of the present invention is not limited to the embodiment of FIG. 2 but may be a camera system mounted on a robot arm, a camera system mounted on a ground mobile device, a camera system mounted on an aerial drone, or the like. have.
  • the automatic control camera 301 uses camera control parameters, which are digitized data, for camera control for each time zone. 3 shows a configuration example of such a camera control parameter.
  • the camera control parameter includes at least a timestamp and location information of the camera or orientation angle information of the camera at the corresponding timestamp.
  • the image synthesis system 100 determines whether a specific section of the auto-capture video content corresponds to a dynamic video clip or a static video clip through camera control parameters. For example, when the positional information of the camera or the orientation angle information of the camera changes in comparison with the previous preceding timestamp at a specific timestamp, it may be determined that the corresponding timestamp is included in the dynamic video clip.
  • the time period may be determined to be included in the dynamic video clip. This means that if the camera's location information continuously changes over a certain time period, for example, when the camera stops for a short time of about 1 second, and the camera's location information changes again, two dynamic video clips are displayed based on the brief stop movement of the camera. This is because it is more efficient for the image synthesis process to judge a single dynamic video clip by ignoring the momentary stop of camera movement rather than distinguishing.
  • the storage module 110 stores a plurality of auto-shoot video contents, for example, a plurality of users access the web server to upload the auto-shoot video contents produced or produced by the user.
  • the camera control parameters used to capture the auto-shoot video content together with the auto-shoot video content are stored in association with each other.
  • a corresponding camera control parameter may be retrieved through identification information of the auto-shoot video content, or the computer file of the auto-shoot video content may be configured to include a camera control parameter.
  • the auto-shoot video content does not necessarily need to be captured by the same auto-shoot camera system.
  • Auto-shoot video content shot through the same auto-shoot camera system is easy to determine similarity with each other because of the same type of camera control parameter.
  • Auto-shoot video content shot through different auto-shoot camera systems may use different types of camera control parameters.
  • the image synthesis system 100 may change and store different types of camera control parameters into a standardized form.
  • the content providing module 120 performs a function of providing one of the plurality of auto-captured video contents to the user terminal 200 through the communication network.
  • the content providing module 120 may exemplify a web service server that provides a user's desired video content by a method such as content search or content recommendation, such as a video sharing web server such as YouTube and Vimeo. Can be.
  • Source auto-capture video content refers to auto-capture video content that is provided to a user from among a plurality of auto-capture video contents stored by the storage module 110 and used for the production of the composite video content (contents_composite).
  • the search module 130 receives an object synthesis request for the source dynamic video clip clip_dynamic_source of the source auto-capture video content content_source from the user terminal 200 and references the camera control parameter of the source dynamic video clip clip_dynamic_source.
  • the composite target dynamic video clip (clip_dynamic_add) of the composite target auto-shoot video content (contents_add) obtained as a control parameter (parameter_ref) and having a matching control parameter (parameter_match) that is the same or similar to the reference control parameter (parameter_ref). Perform a search function.
  • the user terminal 200 may search for the object synthesis request icon 201 by clicking on the object synthesis request icon 201 while playing the source auto-capture video content on a web browser or a video viewer application. You can send an object composition request.
  • Such an object synthesis request may be a synthesis request for a source dynamic video clip (clip_dynamic_source), or in some embodiments, may be a synthesis request for source auto-photographed video contents (contents_source).
  • the retrieval module 130 includes an auto-shooting video including a dynamic video clip having a camera control parameter of the same or similar category as the reference control parameter (parameter_ref) among a plurality of auto-shooting video contents except the source auto-shooting video contents (contents_source). Search for content.
  • dynamic video clips of the plurality of autophotographed video contents may be searched, or a plurality of dynamic video clips may be searched in one autophotographed video content.
  • the search module 130 may select one or more synthesis video clips clip_dynamic_add from among a plurality of searched dynamic video clips according to a user setting or a system setting. Various methods for selecting the final synthesis target video clip clip_dynamic_add may be implemented.
  • the user terminal 200 may provide a recommendation list of a plurality of dynamic video clips or a plurality of auto-captured video contents to select a synthesis target video clip clip_dynamic_add according to a user's selection.
  • attributes of the composite dynamic video clip clip_dynamic_add such as the genre of the video content, the background music of the video content, the time of shooting or uploading the video content, or online social relations between upload users, for example, the video performer Uploading a video clip such as an online social relationship between the users or performers in consideration of the properties individually or in combination
  • the search module 130 may first search for the synthesized dynamic video clip (clip_dynamic_add).
  • the storage module 110 further stores the content genre information in association with the auto-shoot video content, and stores the content genre information of the auto-shoot video content having the same content genre information as the source auto-shoot video content (contents_source).
  • content_source content genre information of the auto-shoot video content having the same content genre information as the source auto-shoot video content.
  • the storage module 110 further stores the background music information in association with the auto recording video content, and stores the background music information of the auto recording video content having the same background music information as the source auto recording video content (contents_source).
  • a composite dynamic video clip for example, a composite video clip (clip_dynamic_composite) that naturally synthesizes video content performing a song or instrumental performance by using the same music as background music, and chorus or concert is performed. It can provide the effect to create.
  • the storage module 110 further stores the upload time information in association with the auto-shoot video content, and has upload time information of a time close to the upload time of the source auto-shoot video content contents_source.
  • Dynamic video clips of auto-shoot video content with the latest or most recent upload time information to be synthesized with dynamic video clips (clip_dynamic_add). can do.
  • the storage module 110 further stores online social relations between uploading users or video performers, and online online with uploading users or video performers of source auto-capture video content (contents_source). It is possible to provide an effect of generating a composite video clip (clip_dynamic_composite) based on a user relationship by first searching a dynamic video clip of an auto-shooting video content of a uploading user or a video performer having a social relationship with the composite target dynamic video clip (clip_dynamic_add).
  • the online social relationship may be, for example, social relationship information established through making friends or fans between users on a web service server providing the auto-shooting video content of the present invention.
  • Social relationship information provided from a social network service independent of a video content providing service such as
  • the reference control parameter is a camera control parameter of the source dynamic video clip (clip_dynamic_source) that is the target of the synthesis request
  • the match control parameter is the composition target dynamic video clip searched by the search module 130 as a synthesis target.
  • the criteria for selecting matching control parameters (parameter_match) that are the same as or similar to the reference control parameters of the search module 130 do not require complete identity but require the same degree of incongruity that does not occur with each other in the composite image to which camerawork is applied.
  • the search module 130 may select a matching control parameter (parameter_match) in a range of 10% difference of each parameter such as time interval, camera position information, and camera angle information based on time stamp information of the reference control parameter parameter_ref.
  • the synthesis module 140 generates a composite video clip clip_dynamic_composite to include at least a source object image 431 of the source dynamic video clip clip_dynamic_source and a synthesis object image 432 of the synthesis target dynamic video clip clip_dynamic_add. Perform the function.
  • the camera control parameter is composed of a combination of various parameters including time stamps, camera position information, and camera orientation angle information, and the combination rarely coincide in different video contents. Therefore, the relaxed equality condition of equality within a certain margin of error should be applied, not exact match. For example, if the time required for the camera to move 1m to the right is 30 seconds for the source dynamic video clip (clip_dynamic_source) and 33 seconds for the target video clip (dynamic clip), the synthesis module 140 may be within an error range of 10%.
  • a 30-second composite video clip (clip_dynamic_composite) with moving camerawork may be generated. If the time required for the camera to move 1m to the right is 30 seconds for the source dynamic video clip (clip_dynamic_source) and 27 seconds for the target dynamic video clip (clip_dynamic_add), it is within an error range of 10% and the synthesis module 140 moves. Create a 27-second composite video clip (clip_dynamic_composite) with camerawork or a 30-second composite video clip (clip_dynamic_composite) with moving camerawork by slowly playing the source dynamic video clip (clip_dynamic_source) at 90% speed. Can be.
  • the angle of the camera rotated to the right for 10 seconds is 10 degrees for the source dynamic video clip (clip_dynamic_source) and 11 degrees for the target dynamic video clip (clip_dynamic_add), it is within the error range of 10%.
  • a camera when a camera shoots a source dynamic video clip (clip_dynamic_source) for 10 seconds, the camera rotates 10 degrees to the right, and a camera shoots a composite dynamic video clip (clip_dynamic_add) 10 degrees to the left.
  • 140 may generate a 10-second composite video clip clip_dynamic_composite to which panning camerawork is applied by inverting the synthesis target object image 432 of the synthesis target video clip clip_dynamic_add.
  • the synthesis module 140 may generate a composite video clip clip_dynamic_composite in various ways. For example, the synthesis module 140 may generate a composite video clip clip_dynamic_composite by combining the source object image 431 and the synthesis target object image 432 with the background image of the source dynamic video clip clip_dynamic_source. In this case, the composing module 140 synthesizes the object video separated from the composing target dynamic video clip clip_dynamic_add without separating the source object video 431 of the source dynamic video clip clip_dynamic_source.
  • the two object images can be rearranged to generate a composite video clip (clip_dynamic_composite). Separation of the object image may be acquired from a chroma key-photographed captured image or by detecting an outline of an object in a dynamic video clip.
  • the image synthesis system 100 may further include a background image input module for receiving a user selected background image from the user terminal 200, and the synthesis module 140 may include a source object image in the user selected background image. 431) and the target object image 432 may be synthesized to generate a composite video clip clip_dynamic_composite.
  • the background image input module provides an interface for allowing a user to input a user input background image including a still image, a video image, or 3D modeling data, for example, through the user terminal 200 and inputs a user from the user terminal 200.
  • the background image may be received through a communication network.
  • the composite video clip (clip_dynamic_composite) is compared with the source dynamic video clip (clip_dynamic_source), so that not only the composite object image 432 such as the performer is added but also the background thereof is changed, thereby changing the degree of change through content reproduction.
  • the height is effective.
  • Korean Patent Application No. 2016-0171469 “Image Synthesis System and Image Synthesis Method,” filed by the applicant of the present patent application has a background image and an object in a virtual space so that the background image and the foreground image are naturally synthesized according to the change of camera viewpoint.
  • An image synthesis system 100 for synthesizing an image is disclosed.
  • the synthesizing module 140 preferably generates a composite video clip (clip_dynamic_composite) through the same method so that the object image and the background image are naturally synthesized according to the camera walk. To this end, as shown in FIG.
  • the synthesizing module 140 forms a background projection surface 410 having a predetermined height and a concave central surface in a virtual space, and a background projection surface 410 at a virtual projection point on the virtual space. Projecting at least a portion of a user-selected background image on the screen, arranging a source object image 431 and a composition target object image 432 between the virtual projection point on the virtual space and the background projection surface 410, and placing the virtual object in the virtual space It is preferable to generate a composite image by including at least a portion of the user-selected background image, at least a portion of the source object image 431, and at least a portion of the synthesis target object image 432 in the angle of view of the camera 420.
  • the virtual projection point is set to a point corresponding to the position of the real world camera where the object image is taken, and the projection of the user-selected background image onto the background projection surface 410 is deformed so that there is no distortion when viewed from the virtual projection point.
  • the user-selected background image may be formed by texturing the background projection surface 410.
  • two or more synthetic object images 432 may be synthesized instead of one synthetic object image 432 in the source video clip.
  • the search module 130 further searches for each of the composite target dynamic video clips (clip_dynamic_add) in the plurality of auto-shooting video contents having a match control parameter, and the synthesis module 140 performs a source of the source dynamic video clip (clip_dynamic_source).
  • the composite video clip clip_dynamic_composite may be generated to include the composite image object 432 and the composite target image 432 of the plurality of composite target video clips.
  • a composite image of the entire source auto-captured video content as well as the dynamic video clip section to which the dynamic camera work is applied may be generated.
  • One moving picture is composed of a plurality of dynamic video clip sections and a plurality of static video clip sections.
  • Static video clip sections without camera change can be easily synthesized according to a conventional image synthesis method, and a plurality of dynamic video clip sections can generate composite video contents (contents_composite) by applying the synthesis method according to the present invention, respectively.
  • the search module 130 receives the object synthesis request for the source auto-capture video content contents_source from the user terminal 200, and receives the first interval reference control parameter parameter_ref of the first interval source dynamic video clip clip_dynamic_source.
  • n is an integer of 2 or more.
  • the n th interval reference control parameter parameter_ref of the source dynamic video clip clip_dynamic_source is obtained, and the first interval reference control parameter parameter_ref to the n th interval reference control parameter the first segment synthesis target video clip clip_dynamic_add to the first segment matching control parameter (parameter_match) to the nth segment matching control parameter (parameter_match) respectively corresponding to parameter_ref) n Configure to retrieve the composite video clips (clip_dynamic_add) There.
  • the synthesizing module 140 includes a first segment source object image 431 of the first segment source dynamic video clip clip_dynamic_source and a first segment composition object image 432 of the first segment compositing dynamic video clip clip_dynamic_add.
  • Each of the n-th section composite video clips clip_dynamic_composite including the object image 432 may be generated.
  • the content providing module 120 may configure the user terminal 200 to provide the composite video contents (contents_composite) including the first interval composite video clip clip_dynamic_composite to the nth interval composite video clip clip_dynamic_composite.
  • the image synthesis method of the present invention is performed by the image synthesis system 100, the image synthesis system 100 is an application installed on the server side of the server-client network connected to the communication network, the operation processing apparatus of the server for driving the application, It may be configured as a storage device of the server, or an application installed in a client such as a storage device of the server and an information terminal such as a smartphone or a PC, an operation processing device of a client for driving the application, and the like.
  • a plurality of automatic images including a dynamic video clip in which the image synthesis system 100 is a section photographed by the automatic control camera 301 controlled by at least one camera control parameter of pan control, tilt control or moving control.
  • the captured video contents are stored in association with a camera control parameter.
  • Auto-shoot video content means video content produced by an auto-control camera 301 in which at least one of pan control, tilt control, or moving control is controlled by a predefined camera control parameter.
  • the video content photographed by the above is not included in the auto-shoot video content of the present invention.
  • the image synthesis system 100 provides the user terminal 200 with one of the plurality of pieces of auto-captured video contents to the user terminal 200 through a communication network.
  • the user may access the web server that provides the auto-shoot video content through an application or a web browser of the user terminal 200 to provide the auto-shoot video content.
  • the image synthesis system 100 receives an object synthesis request for the source dynamic video clip clip_dynamic_source of the source auto-capture video content content_source from the user terminal 200 (S30).
  • the source auto-shoot video content (contents_source) is the auto-shoot video content to be synthesized by the user
  • the source dynamic video clip (clip_dynamic_source) is one of pan motion, tilt motion, and moving motion among the source auto-shoot video content (contents_source). It means the time interval that dynamic camera walk is applied.
  • the image synthesis system 100 provides a user with dynamic video clip segments among source auto-captured video contents (contents_source) through camera control parameters, and the user selects one of the video clip segments.
  • the synthesis system 100 may receive an object synthesis request for the source dynamic video clip clip_dynamic_source.
  • the user requests object composition for the source video content (contents_source), and the image synthesis system 100 includes the source dynamic video clip included in the source request video content (contents_source). object composition for clip_dynamic_source).
  • the image synthesis system 100 obtains a camera control parameter of the source dynamic video clip clip_dynamic_source as a reference control parameter parameter_ref.
  • the image synthesis system 100 may acquire, as a reference control parameter (parameter_ref), a camera control parameter of a time interval corresponding to a source dynamic video clip (clip_dynamic_source) among camera control meters stored in association with the source auto-capture video content (contents_source). have.
  • Searching for (s50) is performed.
  • the image synthesis system 100 compares camera control parameters of other auto-shooting video contents with a reference control parameter (parameter_ref), and if a time interval of the same or similar camera control parameter is found, recognizes it as a matching control parameter (parameter_match) and time.
  • the video clip of the section is searched for as the synthesis target video clip (clip_dynamic_add).
  • the image synthesis system 100 searches for the synthesis target video clip clip_dynamic_add in step S50, for example, the genre of the video content, the background music of the video content, the shooting time or the uploading time of the video content, and the like.
  • the search module 130 synthesizes the attributes of the video clip (clip_dynamic_add) or the properties of the uploading user or performer of the video clip, such as online social relations between uploading users and online social relations between video performers, individually or in combination.
  • the target dynamic video clip clip_dynamic_add may be searched first.
  • the composite video clip clip_dynamic_composite includes at least a source object image 431 of the source dynamic video clip clip_dynamic_source and a synthesis target object image 432 of the target dynamic video clip clip_dynamic_add. Step (s60) is generated. Since the source object image 431 and the composite object image 432 share the same or similar camerawork, the composite video clip clip_dynamic_composite may provide a natural composite image to the user despite the dynamic camerawork.
  • the image synthesis system 100 may apply various synthesis methods. For example, in operation S60 of generating the composite video clip clip_dynamic_composite, the image synthesis system 100 synthesizes the source object image 431 and the synthesis target object image 432 on the background image of the source dynamic video clip clip_dynamic_source. A composite video clip (clip_dynamic_composite) may be generated.
  • the image synthesizing method further includes a step in which the image synthesis system 100 receives a user-selected background image from the user terminal 200 and generates a composite video clip clip_dynamic_composite (s60). ), The image synthesis system 100 may generate a composite video clip clip_dynamic_composite by synthesizing the source object image 431 and the synthesis target object image 432 with the user-selected background image.
  • the first part of the image synthesis system 100 forming a background projection surface 410 having a predetermined height and having a central concave curved surface in a virtual space.
  • the steps can be performed.
  • the image synthesis system 100 may perform a second sub-step of projecting at least a portion of the user-selected background image to the background projection surface 410 at the virtual projection point in the virtual space.
  • the image synthesis system 100 may perform a third sub-step of placing the source object image 431 and the synthesis target object image 432 between the virtual projection point and the background projection surface 410 in the virtual space. . Finally, at least a portion of the user-selected background image, at least a portion of the source object image 431, and at least a portion of the synthesis target object image 432 are included in an angle of view of the virtual camera 420 in which the image synthesis system 100 is disposed in the virtual space.
  • the fourth sub-step of generating a composite video clip clip_dynamic_composite may be performed, including a portion thereof.
  • the image synthesis system 100 performs a step S70 of providing the synthesized video clip clip_dynamic_composite to the user terminal 200.
  • the step (s50) of searching for the synthesis target dynamic video clip (clip_dynamic_add) may be performed by an image synthesis system ( 100 may further search for synthesis dynamic video clips clip_dynamic_adds in the plurality of auto-shoot video contents having the match control parameter, respectively.
  • the composite video clip clip_dynamic_composite may be generated by the image synthesis system 100, which is a composite object of the source object image 431 of the source dynamic video clip clip_dynamic_source and the plurality of composite target dynamic video clips clip_dynamic_add.
  • the composite video clip_dynamic_composite may be generated to include the images 432.
  • a composite video content content_composite is generated by synthesizing the synthesis target object video 432 with the entire source auto-capture video content content_source without synthesizing the synthesis target object video 432 only with the source dynamic video clip clip_dynamic_source.
  • the image synthesis system 100 may receive an object composition request for the source auto-capture video content contents_source from the user terminal 200.
  • the image synthesis system 100 is the first interval reference control parameter (parameter_ref) to the nth interval (n is 2) of the first interval source dynamic video clip (clip_dynamic_source) Integer)
  • the n-th section reference control parameter parameter_ref of the source dynamic video clip clip_dynamic_source may be acquired.
  • the image synthesis system 100 may include a first section corresponding to each of the first section reference control parameter parameter_ref to the nth section reference control parameter parameter_ref.
  • the composite video clip clip_dynamic_composite may be generated by the image synthesis system 100.
  • the first segment source object image of the first segment source dynamic video clip clip_dynamic_source and the first segment composition target dynamic video clip_dynamic_add The nth interval source object image of the first interval synthesis video clip clip_dynamic_composite to the nth interval source dynamic video clip clip_dynamic_source and the nth interval synthesis target dynamic video clip clip_dynamic_add including the first interval synthesis target object image N-th composite video clips (clip_dynamic_composite) including the n-th composite image object may be generated.
  • the image synthesis system 100 includes the composite video content (contents_composite) including the first interval composite video clip (clip_dynamic_composite) to the nth interval composite video clip (clip_dynamic_composite) It may be provided to the user terminal 200.
  • the composite video content including the first interval composite video clip (clip_dynamic_composite) to the nth interval composite video clip (clip_dynamic_composite) It may be provided to the user terminal 200.
  • a time section that does not correspond to a source dynamic video clip (clip_dynamic_source) that is, a time section that corresponds to a static video clip photographed using a static camera walk, it is synthesized using a conventional image object synthesis technique.
  • Video content (contents_composite) may be generated.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Human Computer Interaction (AREA)
  • Studio Circuits (AREA)
  • Studio Devices (AREA)

Abstract

La présente invention concerne un système et un procédé de synthèse d'images basés sur un travail à la caméra, qui sont capables de générer, sans incongruité, un clip vidéo synthétisé de haute qualité présentant un travail dynamique à la caméra et, plus spécifiquement, un système et un procédé de synthèse d'images destinés à synthétiser automatiquement des clips vidéo dynamiques de contenu video présentant un travail similaire à la caméra, par l'intermédiaire d'une conversion en données du travail à la caméra.
PCT/KR2017/015698 2016-12-30 2017-12-29 Système et procédé de synthèse d'images basés sur un travail à la caméra WO2018124794A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR10-2016-0183636 2016-12-30
KR1020160183636A KR101843025B1 (ko) 2016-12-30 2016-12-30 카메라워크 기반 영상합성 시스템 및 영상합성방법

Publications (1)

Publication Number Publication Date
WO2018124794A1 true WO2018124794A1 (fr) 2018-07-05

Family

ID=61901640

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2017/015698 WO2018124794A1 (fr) 2016-12-30 2017-12-29 Système et procédé de synthèse d'images basés sur un travail à la caméra

Country Status (2)

Country Link
KR (1) KR101843025B1 (fr)
WO (1) WO2018124794A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021056353A1 (fr) * 2019-09-26 2021-04-01 深圳市大疆创新科技有限公司 Procédé de montage vidéo et appareil de terminal
CN113259754A (zh) * 2020-02-12 2021-08-13 北京达佳互联信息技术有限公司 视频生成方法、装置、电子设备及存储介质

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102259401B1 (ko) 2019-11-28 2021-06-01 (주)테슬라시스템 드론 동영상에 증강현실 부가정보를 합성하는 방법

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20040005111A (ko) * 2002-07-08 2004-01-16 파인드테크 주식회사 지능형 ｐｖｒ 시스템 및 그 운영방법
JP2004531113A (ja) * 2001-02-09 2004-10-07 リー，クジン 注釈記入による全方位性３次元イメージデータ獲得装置、その方法及び感光範囲の拡大方法
KR20050077144A (ko) * 2004-01-27 2005-08-01 엘지전자 주식회사 비디오 디코딩 장치 및 그 방법
JP2009055332A (ja) * 2007-08-27 2009-03-12 Mitsubishi Electric Corp 映像合成表示装置
KR20110123907A (ko) * 2010-05-10 2011-11-16 에스케이플래닛 주식회사 Ｉｐｔｖ 시스템 및 그의 컨텐츠 제공 방법

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004531113A (ja) * 2001-02-09 2004-10-07 リー，クジン 注釈記入による全方位性３次元イメージデータ獲得装置、その方法及び感光範囲の拡大方法
KR20040005111A (ko) * 2002-07-08 2004-01-16 파인드테크 주식회사 지능형 ｐｖｒ 시스템 및 그 운영방법
KR20050077144A (ko) * 2004-01-27 2005-08-01 엘지전자 주식회사 비디오 디코딩 장치 및 그 방법
JP2009055332A (ja) * 2007-08-27 2009-03-12 Mitsubishi Electric Corp 映像合成表示装置
KR20110123907A (ko) * 2010-05-10 2011-11-16 에스케이플래닛 주식회사 Ｉｐｔｖ 시스템 및 그의 컨텐츠 제공 방법

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021056353A1 (fr) * 2019-09-26 2021-04-01 深圳市大疆创新科技有限公司 Procédé de montage vidéo et appareil de terminal
CN113259754A (zh) * 2020-02-12 2021-08-13 北京达佳互联信息技术有限公司 视频生成方法、装置、电子设备及存储介质
CN113259754B (zh) * 2020-02-12 2023-09-19 北京达佳互联信息技术有限公司 视频生成方法、装置、电子设备及存储介质

Also Published As

Publication number Publication date
KR101843025B1 (ko) 2018-03-28

Similar Documents

Publication Publication Date Title
US11862198B2 (en) Synthesizing a presentation from multiple media clips
US11381739B2 (en) Panoramic virtual reality framework providing a dynamic user experience
EP3942437B1 (fr) Systèmes et procédés destinés à des essaims multimédias
US20200388068A1 (en) System and apparatus for user controlled virtual camera for volumetric video
CN106713942B (zh) 视频处理方法和装置
CN106060526A (zh) 一种基于双摄像头的直播方法及装置
CN113115110B (zh) 视频合成方法、装置、存储介质与电子设备
WO2018124794A1 (fr) Système et procédé de synthèse d'images basés sur un travail à la caméra
US20140294366A1 (en) Capture, Processing, And Assembly Of Immersive Experience
KR20160021706A (ko) 360도 공간영상 재생방법 및 그 시스템
KR20180013391A (ko) 스크립트 생성 장치, 동영상 재생 장치 및 360도 동영상에 대한 화면 제어 방법
CN112543344A (zh) 直播控制方法、装置、计算机可读介质及电子设备
US10764655B2 (en) Main and immersive video coordination system and method
WO2017209468A1 (fr) Système et procédé de synthèse d'incrustation couleur permettant de fournir des effets stéréoscopiques tridimensionnels
WO2017065503A1 (fr) Système d'édition multimédia répartie et procédé d'édition
WO2020138541A1 (fr) Procédé et appareil de génération d'une vidéo multicanaux à l'aide d'un terminal mobile
WO2015034123A1 (fr) Procédé de gestion de tampon pour la reproduction d'images provenant de plusieurs caméras d'action et appareil de reproduction associé
US20170287521A1 (en) Methods, circuits, devices, systems and associated computer executable code for composing composite content
WO2017217789A1 (fr) Système et procédé de génération de contenu vidéo à l'aide d'une coopération d'une pluralité d'objets de performance
WO2014058404A1 (fr) Système d'enregistrement, d'affichage et de diffusion d'une vidéo dans un format tridimensionnel
Jiang et al. Multiple HD Screen‐Based Virtual Studio System with Learned Mask‐Free Portrait Harmonization
TWI246324B (en) Method and system for media production in virtual studio
WO2023042935A1 (fr) Procédé de partage d'image, et dispositif et système utilisant ledit procédé
Jung et al. Focused on multi viewed virtual reality short film'sweeney TODD-the story of a barber'production study
JP2023166836A (ja) 多視点画像管理装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17886139

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17886139

Country of ref document: EP

Kind code of ref document: A1