WO2021199715A1 - Information processing device, information processing method, and program - Google Patents

Information processing device, information processing method, and program Download PDF

Info

Publication number
WO2021199715A1
WO2021199715A1 PCT/JP2021/005289 JP2021005289W WO2021199715A1 WO 2021199715 A1 WO2021199715 A1 WO 2021199715A1 JP 2021005289 W JP2021005289 W JP 2021005289W WO 2021199715 A1 WO2021199715 A1 WO 2021199715A1
Authority
WO
WIPO (PCT)
Prior art keywords
viewpoint
image
camera
information
camera work
Prior art date
Application number
PCT/JP2021/005289
Other languages
French (fr)
Japanese (ja)
Inventor
翔 小倉
Original Assignee
ソニーグループ株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by ソニーグループ株式会社 filed Critical ソニーグループ株式会社
Publication of WO2021199715A1 publication Critical patent/WO2021199715A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics

Definitions

  • This technology relates to an information processing device, its method, and a program, and particularly to a processing technology related to a free-viewpoint image capable of observing an imaged subject from an arbitrary viewpoint in a three-dimensional space.
  • a free viewpoint image also called a free viewpoint image, a virtual viewpoint image (video), etc.
  • the technology to generate is known.
  • Patent Document 1 discloses a technique relating to the generation of camera work, which can be said to be a movement locus of a viewpoint.
  • the free viewpoint image may be used as a replay image in broadcast contents such as sports broadcasts.
  • camera work corresponding to each scene assumed as the target scene of the free viewpoint image such as the shooting scene of team A and the shooting scene of team B in soccer, is created in advance. Can be considered.
  • the camera work corresponding to the scene is selected from the plurality of camera works created in advance in this way, and the free viewpoint image is generated.
  • the camera work creation work takes a form in which a user as a broadcasting staff inputs an operation to a creation screen provided by a computer device such as a PC (personal computer).
  • a creation work it is desirable that the user can confirm how the camera work has changed due to his / her own operation input. This confirmation can be realized, for example, in the form of previewing the free viewpoint image generated according to the created camera work information to the user.
  • the user can determine whether or not the created camera work is as intended, and if it is not as intended, the user can recreate the camera work.
  • This technology was made in view of the above circumstances, and aims to improve the efficiency of camera work creation work.
  • the information processing device accepts a designated operation reception that accepts an operation input for designating at least a part of the camera work information as a camera work information creation operation screen which is information indicating at least the movement trajectory of the viewpoint in the free viewpoint image. It is provided with a display processing unit that performs screen display processing including an area and a camera work display area that visualizes and displays the movement locus of the viewpoint based on camera work information that reflects the specified contents by the operation input. be. As a result, the user can perform the camera work creation operation while visually recognizing the movement locus of the visualized viewpoint on the camera work creation operation screen.
  • the designated operation receiving area can be configured to be capable of receiving designated operations of the start point and the end point of the movement locus. As a result, it is possible to set an arbitrary start point and end point for the movement locus of the viewpoint instead of being fixed.
  • the designated operation receiving area can be configured to be capable of receiving the designated operation of the waypoint of the viewpoint.
  • the designated operation receiving area can be configured to be capable of receiving the designated operation of the waypoint of the viewpoint.
  • the designated operation receiving area can be configured to be capable of receiving a designated operation at a timing when the viewpoint passes through the waypoint. This makes it possible to set not only the waypoint of the viewpoint but also the timing of the viewpoint passing through the waypoint.
  • the designated operation receiving area can be configured to be capable of receiving designated operations of the shape type of the movement locus. This makes it possible to change the shape type of the movement locus of the viewpoint instead of fixing it.
  • the designated operation receiving area can be configured to be capable of receiving the designated operation of the moving speed of the viewpoint. This makes it possible to change the moving speed of the viewpoint instead of fixing it.
  • the designated operation receiving area can be configured to be capable of receiving designated operations of a section in which the moving speed is changed in the moving locus. This makes it possible to dynamically change the moving speed of the viewpoint in the moving locus.
  • the designated operation receiving area has a configuration capable of receiving the operation input to the timeline indicating the period from the movement start time to the movement end time of the viewpoint. Is possible.
  • the timeline By accepting input operations on the timeline, for example, by dragging and dropping the camera icon etc. onto the timeline, it is possible to specify the waypoint and the waypoint at the same time.
  • Facilitating the operation of specifying various information related to camera work, such as the section where the curve interpolation of the movement trajectory should be performed and the section where a predetermined effect should be given can be specified by specifying the range by dragging on the timeline. It becomes possible to plan.
  • the display processing unit can be configured to perform processing for displaying information that visualizes the visual field range from the viewpoint in the camera work display area. .. By visually showing the field of view, it is easy for the user to grasp the camera work.
  • the display processing unit can be configured to perform processing of displaying information representing a visual field range from the viewpoint in a graphic shape in the camera work display area. be. By displaying the field of view in a graphic form, the user can easily grasp the camera work.
  • the display processing unit can be configured to perform a process of displaying an image of observing a three-dimensional space from the viewpoint on the creation operation screen. As a result, it is possible to preview and display an image similar to the free viewpoint image generated based on the camera work information to the user, and it is possible to facilitate the grasp of the camera work.
  • the designated operation receiving area can be configured to be capable of receiving a designated operation of a target position that determines a line-of-sight direction from the viewpoint. This makes it possible to generate an image that follows the target as a free viewpoint image.
  • the image that follows the target means an image in which the target is continuously positioned at a predetermined position (for example, the center position) in the image frame.
  • the designated operation receiving area can be configured to be capable of receiving designated operations during the period facing the target.
  • the period of facing the target means a period of keeping the target positioned at a predetermined position in the image frame of the free viewpoint image.
  • the designated operation receiving area can be configured to be capable of accepting a plurality of target position designation operations as the target position designation operation. be.
  • the designated operation receiving area can be configured to be capable of accepting a plurality of target position designation operations as the target position designation operation. be.
  • the information processing apparatus specifies at least a part of the camerawork information as an operation screen for creating camerawork information which is information indicating at least the movement trajectory of the viewpoint in the free viewpoint image.
  • Information processing that performs screen display processing including a designated operation reception area that receives The method Even with such an information processing method according to the present technology, the same operation as that of the above-mentioned information processing apparatus according to the present technology can be obtained.
  • the program according to the present technology is a program that can be read by a computer device, and at least a part of the camera work information is displayed as an operation screen for creating camera work information which is information indicating at least the movement trajectory of the viewpoint in the free viewpoint image.
  • Screen display processing including a designated operation receiving area that accepts a designated operation input and a camerawork display area that visualizes and displays the movement locus of the viewpoint based on the camerawork information that reflects the specified contents of the operation input. It is a program that makes the computer device realize the function of performing the above. By such a program, the information processing apparatus according to the present technology described above is realized.
  • FIG. 1 shows a configuration example of an image processing system according to an embodiment of the present technology.
  • the image processing system includes an image creation controller 1, a free viewpoint image server 2, a video server 3, a plurality of (for example, 4) video servers 4A, 4B, 4C, 4D, NAS (Network Attached Storage) 5, a switcher 6, and image conversion. It has a unit 7, a utility server 8, and a plurality of (for example, 16) imaging devices 10.
  • the term “camera” refers to the imaging device 10.
  • “camera arrangement” means arrangement of a plurality of imaging devices 10.
  • video server 4 when the video servers 4A, 4B, 4C, and 4D are generically referred to without any particular distinction, they are referred to as "video server 4".
  • this image processing system based on the captured images (for example, image data V1 to V16) acquired from a plurality of imaging devices 10, a free viewpoint image corresponding to an observation image from an arbitrary viewpoint in a three-dimensional space is generated and freely selected. You can create an output clip that includes a viewpoint image.
  • connection state of each part is shown by a solid line, a broken line, and a double line.
  • the solid line indicates the connection of SDI (Serial Digital Interface), which is an interface standard for connecting broadcasting devices such as cameras and switchers, and is compatible with 4K, for example.
  • Image data is mainly transmitted and received between each device by SDI wiring.
  • the double line indicates the connection of the communication standard for constructing the computer network, for example, 10 Gigabit Ethernet.
  • the broken line between the video servers 3 and 4 indicates a state in which the video servers 3 and 4 equipped with the inter-server file sharing function are connected by, for example, a 10G network.
  • each video server can preview and send the material in the other video server. That is, a system using a plurality of video servers has been constructed so that efficient highlight editing and transmission can be realized.
  • Each image pickup device 10 is configured as a digital camera device having an image pickup element such as a CCD (Charge Coupled Devices) sensor or a CMOS (Complementary Metal-Oxide-Semiconductor) sensor, and is an image captured as digital data (from image data V1). V16) is obtained. In this example, each image pickup device 10 obtains a captured image as a moving image.
  • an image pickup element such as a CCD (Charge Coupled Devices) sensor or a CMOS (Complementary Metal-Oxide-Semiconductor) sensor
  • each image pickup device 10 is supposed to capture a state in which a game such as basketball or soccer is being performed, and each of them is arranged in a predetermined direction at a predetermined position in a competition venue where the competition is held.
  • the number of image pickup devices 10 is 16, but the number of image pickup devices 10 may be at least 2 or more in order to enable the generation of a free viewpoint image.
  • FIG. 2 shows an example of arrangement of the image pickup device 10 around the basketball court. It is assumed that ⁇ is the imaging device 10. For example, this is an example of camera arrangement when it is desired to focus on the vicinity of the goal on the left side in the drawing. Of course, the camera arrangement and number are examples, and should be set according to the content and purpose of shooting and broadcasting.
  • the image creation controller 1 is composed of an information processing device.
  • the image creation controller 1 can be realized by using, for example, a dedicated workstation, a general-purpose personal computer, a mobile terminal device, or the like.
  • the image creation controller 1 performs control / operation management of the video servers 3 and 4 and processing for creating a clip.
  • the image creation controller 1 is a device that can be operated by the operator OP1.
  • the operator OP1 gives instructions such as selection of clip contents and creation.
  • the free-viewpoint image server 2 is configured as an information processing device that actually creates a free-viewpoint image (FV (Free View) clip, which will be described later) in response to an instruction from the image creation controller 1.
  • This free-viewpoint image server 2 can also be realized by using, for example, a dedicated workstation, a general-purpose personal computer, a mobile terminal device, or the like.
  • the free viewpoint image server 2 is a device that can be operated by the operator OP2.
  • the operator OP2 performs work related to, for example, creating an FV clip as a free viewpoint image. Specifically, the operator OP2 performs a camera work designation operation (selection operation) for generating a free viewpoint image. Further, in this example, the operator OP2 also performs the work of creating the camera work.
  • the configuration and processing of the image creation controller 1 and the free viewpoint image server 2 will be described in detail later. Further, it is assumed that the operators OP1 and OP2 perform the operation.
  • the image creation controller 1 and the free viewpoint image server 2 may be arranged side by side and operated by one operator.
  • the video servers 3 and 4 are image recording devices, for example, a data recording unit such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive), and a control unit that controls data recording / playback of the data recording unit. To be equipped.
  • a data recording unit such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive)
  • HDD Hard Disk Drive
  • Each of the video servers 4A, 4B, 4C, and 4D is capable of inputting, for example, four systems, and simultaneously records images captured by four image pickup devices 10.
  • the video server 4A records the image data V1, V2, V3, V4.
  • the video server 4B records the image data V5, V6, V7, V8.
  • the video server 4C records the image data V9, V10, V11, and V12.
  • the video server 4D records the image data V13, V14, V15, V16.
  • the video servers 4A, 4B, 4C, and 4D are supposed to constantly record, for example, during a sports match to be broadcast.
  • the video server 3 is directly connected to, for example, the image creation controller 1, and can, for example, input two systems and output two systems.
  • Image data Vp and Vq are shown as two inputs.
  • the image data Vp and Vq it is possible to select an image captured by any of the two imaging devices 10 (any two of the image data V1 to V16). Of course, it may be an image captured by another imaging device.
  • the image data Vp and Vq can be displayed on the display by the image creation controller 1 as monitor images.
  • the operator OP1 can confirm the status of a scene shot / recorded for broadcasting, for example, by using the image data Vp and Vq input to the video server 3. Further, since the video servers 3 and 4 are connected to the file sharing state, the image creation controller 1 also monitors and displays the captured images of the imaging devices 10 recorded on the video servers 4A, 4B, 4C, and 4D. This allows the operator OP1 to check sequentially.
  • a time code is attached to the image captured by each imaging device 10, and it is possible to synchronize frames in the processing of the video servers 3, 4A, 4B, 4C, and 4D.
  • NAS5 is a storage device arranged on a network, and is composed of, for example, an SSD or an HDD. In the case of this example, NAS5 is when some frames of the image data V1, V2 ... V16 recorded on the video servers 4A, 4B, 4C, 4D are transferred for the generation of the free viewpoint image. , A device that stores a free-viewpoint image for processing in the free-viewpoint image server 2 or stores a created free-viewpoint image.
  • the switcher 6 is a device that inputs an image output via the video server 3 and finally selects and broadcasts the main line image PGMout.
  • a broadcasting director or the like performs necessary operations.
  • the image conversion unit 7 performs resolution conversion and composition of image data by, for example, the image pickup device 10, generates a monitoring image of the camera arrangement, and supplies the monitoring image to the utility server 8.
  • 16 systems of image data V1 to V16
  • V1 to V16 are converted into 4K images in resolution and then arranged in tiles to form 4 systems of images, which are supplied to the utility server 8.
  • the utility server 8 is a computer device capable of performing various related processes.
  • the utility server 8 is a device that detects camera movement for calibration.
  • the utility server 8 monitors the image data from the image conversion unit 7 and detects the movement of the camera.
  • the camera movement is, for example, the movement of one of the arrangement positions of the image pickup apparatus 10 arranged as shown in FIG.
  • the information on the arrangement position of the image pickup apparatus 10 is an important element for generating a free-viewpoint image, and if the arrangement position changes, it is necessary to redo the parameter setting. Therefore, the movement of the camera is monitored.
  • the image creation controller 1, the free-viewpoint image server 2, the video servers 3, 4, and the utility server 8 in the above configuration can be realized as an information processing device 70 having the configuration shown in FIG. 3, for example.
  • the CPU 71 of the information processing apparatus 70 executes various processes according to a program stored in the ROM 72 or a program loaded from the storage unit 79 into the RAM 73.
  • the RAM 73 also appropriately stores data and the like necessary for the CPU 71 to execute various processes.
  • the CPU 71, ROM 72, and RAM 73 are connected to each other via a bus 74.
  • An input / output interface 75 is also connected to the bus 74.
  • An input unit 76 including an operator and an operation device is connected to the input / output interface 75.
  • various controls and operation devices such as a keyboard, mouse, keys, dial, touch panel, touch pad, and remote controller are assumed.
  • the user's operation is detected by the input unit 76, and the signal corresponding to the input operation is interpreted by the CPU 71.
  • a display unit 77 made of an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel and an audio output unit 78 made of a speaker or the like are connected to the input / output interface 75 as one or a separate body.
  • the display unit 77 is a display unit that performs various displays, and is composed of, for example, a display device provided in the housing of the information processing device 70, a separate display device connected to the information processing device 70, and the like.
  • the display unit 77 executes the display of various images for image processing, moving images to be processed, and the like on the display screen based on the instruction of the CPU 71. Further, the display unit 77 displays various operation menus, icons, messages, etc., that is, as a GUI (Graphical User Interface) based on the instruction of the CPU 71.
  • GUI Graphic User Interface
  • a storage unit 79 composed of a hard disk, a solid-state memory, or the like, or a communication unit 80 composed of a modem or the like may be connected to the input / output interface 75.
  • the communication unit 80 performs communication processing via a transmission line such as the Internet, wire / wireless communication with various devices, bus communication, and the like.
  • a drive 82 is also connected to the input / output interface 75, if necessary, and a removable recording medium 81 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted.
  • the drive 82 can read data files such as an image file MF and various computer programs from the removable recording medium 81.
  • the read data file is stored in the storage unit 79, and the image and sound included in the data file are output by the display unit 77 and the sound output unit 78. Further, the computer program or the like read from the removable recording medium 81 is installed in the storage unit 79 as needed.
  • software can be installed via network communication by the communication unit 80 or a removable recording medium 81.
  • the software may be stored in the ROM 72, the storage unit 79, or the like in advance.
  • FIG. 4 shows a section identification processing unit 21, a target image transmission control unit 22, and an output image generation unit 23 as functions formed in the CPU 71 of the information processing device 70 that serves as the image creation controller 1.
  • the section specifying processing unit 21 performs a process of specifying a generation target image section to be generated as a free viewpoint image for a plurality of captured images (image data V1 to V16) simultaneously captured by the plurality of imaging devices 10. For example, when the operator OP1 performs an operation of selecting a scene to be replayed in an image, the time code for that scene, particularly the section of the scene to be a free viewpoint image (image section to be generated) can be specified. , Performs processing such as notifying the free viewpoint image server 2 of the time code.
  • the image section to be generated is a frame section that is actually a free viewpoint image.
  • a free viewpoint image is generated for a certain frame in a moving image
  • that one frame is a generation target image section.
  • the in / out points for the free viewpoint image have the same time code.
  • the plurality of frames are the generation target image sections.
  • the in / out points for the free viewpoint image have different time codes.
  • the target image transmission control unit 22 uses the image data of the generation target image section in each of the plurality of captured images 10, that is, one or more frames for the image data V1 to V16, to generate the free viewpoint image in the free viewpoint image server 2. Controls transmission as image data to be used. Specifically, control is performed to transfer the image data as the image section to be generated from the video servers 4A, 4B, 4C, 4D to NAS5.
  • the output image generation unit 23 performs a process of generating an output image (output clip) including the received free viewpoint image (FV clip) generated by the free viewpoint image server 2.
  • the image creation controller 1 adds an FV clip, which is a virtual image generated by the free-viewpoint image server 2 by the processing of the output image generation unit 23, a front clip, which is an actual moving image at a time before that, and a later clip.
  • the rear clips which are the actual moving images at the time, are combined on the time axis to form an output clip. That is, the front clip + FV clip + rear clip is regarded as one output clip.
  • the front clip + FV clip may be used as one output clip.
  • the FV clip + rear clip may be used as one output clip.
  • the output clip of only the FV clip may be generated without combining the front clip and the rear clip.
  • the image creation controller 1 generates an output clip including the FV clip and outputs it to the switcher 6 so that it can be used for broadcasting.
  • FIG. 5 shows a target image acquisition unit 31, an image generation processing unit 32, a transmission control unit 33, and a camera work generation processing unit 34 as functions formed in the CPU 71 of the information processing device 70 that serves as the free viewpoint image server 2. ing.
  • the target image acquisition unit 31 acquires the image data of the generation target image section that is the generation target of the free viewpoint image in each of the plurality of captured images (image data V1 to V16) simultaneously captured by the plurality of imaging devices 10. Perform the processing to be performed. That is, the image data of one frame or a plurality of frames specified by the in / out points of the generation target image section specified by the image creation controller 1 by the function of the section specifying processing unit 21 is output from the video servers 4A, 4B, 4C, 4D to NAS5. It can be acquired via and used for the generation of a free viewpoint image.
  • the target image acquisition unit 31 acquires image data of one frame or a plurality of frames of the generation target image section for all of the image data V1 to V16.
  • the reason for acquiring the image data of the image section to be generated for all of the image data V1 to V16 is to generate a high-quality free-viewpoint image.
  • the image generation processing unit 32 is a function of generating a free viewpoint image, that is, an FV clip in the case of this example, using the image data acquired by the target image acquisition unit 31.
  • the image generation processing unit 32 performs processing such as modeling processing including 3D model generation and subject analysis, and rendering to generate a free viewpoint image which is a two-dimensional image from the 3D model.
  • the 3D model generation represents a subject in a three-dimensional space (that is, from a two-dimensional image) based on an image captured by each imaging device 10 and a camera parameter for each imaging device 10 input from, for example, a utility server 8.
  • This is a process for generating 3D model data (which restores the three-dimensional structure of the subject).
  • the 3D model data includes data in which the subject is represented by a (X, Y, Z) three-dimensional coordinate system.
  • Subject analysis analyzes the position, orientation, and posture of a subject as a person (player) based on 3D model data. Specifically, the position of the subject is estimated, a simple model of the subject is generated, and the orientation of the subject is estimated. Then, a free viewpoint image is generated based on the 3D model data and the subject analysis information. For example, a free viewpoint image that moves the viewpoint is generated for a 3D model in which the player who is the subject is stationary.
  • FIG. 6A shows an image of a free viewpoint image in which a subject is captured from a required viewpoint set in a three-dimensional space.
  • the subject S1 is viewed substantially in front and the subject S2 is viewed substantially in the rear.
  • FIG. 6B shows an image of a virtual viewpoint image when a viewpoint is set in which the position of the viewpoint is changed in the direction of the arrow C in FIG. 6A and the subject S1 is viewed substantially backward.
  • the subject S2 is viewed substantially in front, and the subject S3 and the basket goal, which were not projected in FIG. 6A, are projected.
  • the viewpoint is gradually moved in the direction of arrow C, and an image of about 1 to 2 seconds that reaches the state of FIG. 6B is generated as a free viewpoint image (FV clip). ..
  • FV clip free viewpoint image
  • the time length of the FV clip as a free viewpoint image and the trajectory of the viewpoint movement can be considered in various ways.
  • the free-viewpoint image server 2 (CPU71) of this example has a function as a display processing unit 32a as a part of the function of the image generation processing unit 32.
  • the display processing unit 32a performs display processing of the camera work designation screen Gs that accepts the camera work information designation operation used for generating the free viewpoint image. The details of the camera work related to the free viewpoint image and the camera work designation screen Gs will be described later.
  • the free viewpoint image server 2 in this example also has a function as a camera work editing processing unit 32b as a part of the function of the image generation processing unit 32, but the function as the camera work editing processing unit 32b will be revisited later. explain.
  • the transmission control unit 33 controls to transmit the free viewpoint image (FV clip) generated by the image generation processing unit 32 to the image creation controller 1 via the NAS 5.
  • the transmission control unit 33 controls to transmit the accompanying information for generating the output image to the image creation controller 1.
  • the accompanying information is assumed to be information that specifies the image of the front clip or the rear clip. That is, it is information for designating which image of the image data V1 to V16 is used to create (cut out) the front clip or the rear clip.
  • information that specifies the time length of the front clip and the rear clip is also assumed as incidental information.
  • the camera work generation processing unit 34 performs processing related to generation of camera work information used for generating a free viewpoint image.
  • a software program for creating camera work is installed in the free-viewpoint image server 2 of this example.
  • the camera work generation processing unit 34 is a function realized by this software program, and performs camera work generation processing based on a user's operation input.
  • the camera work generation processing unit 34 has a function as a display processing unit 34a.
  • the display processing unit 34a performs display processing of the creation operation screen Gg so that the user (operator OP2 in this example) can accept various operation inputs for creating the camera work.
  • GUI Overview> With reference to FIGS. 7 and 8, the outline of the camera work designation screen Gs used for creating the free viewpoint image and the creation operation screen Gg used for creating the camera work will be described.
  • the camera work designation screen Gs and the creation operation screen Gg are displayed on the display unit 77 of the free viewpoint image server 2, for example, and can be confirmed and operated by the operator OP2.
  • a scene window 41 On the camera work designation screen Gs shown in FIG. 7, a scene window 41, a scene list display unit 42, a camera work window 43, a camera work list display unit 44, a parameter display unit 45, and a transmission window 46 are arranged.
  • the scene window 41 for example, the image of the image section to be generated is displayed on the monitor so that the operator OP2 can confirm the content of the scene for generating the free viewpoint image.
  • the scene list display unit 42 displays, for example, a list of scenes designated for the image section to be generated. The operator OP2 can select the scene to be displayed in the scene window 41 on the scene list display unit 42.
  • the position of the arranged image pickup device 10 the selected camera work, a plurality of selectable camera works, and the like are displayed.
  • the camera work information is at least information indicating the movement locus of the viewpoint in the free viewpoint image.
  • the camera work window 43 displays at least information that visualizes the movement locus of the viewpoint as a display of the camera work.
  • the camera work list display unit 44 displays a list of various camera work information created and stored in advance.
  • the operator OP2 can select and specify the camera work to be used for FV clip generation from the camera works displayed on the camera work list display unit 44.
  • Various parameters related to the selected camera work are displayed on the parameter display unit 45.
  • Information about transmitting the created FV clip to the image creation controller 1 is displayed in the transmission window 46.
  • a preset list display unit 51, a camera work list display unit 52, a camera work window 53, an operation panel unit 54, and a preview window 55 are arranged on the creation operation screen Gg.
  • the preset list display unit 51 can selectively display a camera preset list, a target preset list, and a 3D model preset list.
  • the camera preset list is list information of position information (position information in three-dimensional space) for each camera preset by the user regarding the camera arrangement position in the field. As will be described later, when a camera preset list is selected, information indicating the position of each camera identification information (for example, camera1, camera2, ..., camera16) is displayed in a list on the preset list display unit 51.
  • NS the position information in three-dimensional space
  • the target means a target position that determines the direction of the line of sight from the viewpoint in the free viewpoint image.
  • the line-of-sight direction from the viewpoint is determined to face the target.
  • the preset list display unit 51 displays a list of identification information about the target preset by the user and information indicating the position thereof.
  • target Tg the target that determines the line-of-sight direction from the viewpoint in the free-viewpoint image as described above.
  • the 3D model preset list is a 3D model preset list to be displayed as the background of the camera work window 43, and when the 3D model preset list is selected, the preset list display unit 51 identifies the preset 3D model. The information is listed.
  • the camera work list display unit 52 can display a list of camera work information created through the creation operation screen Gg and camera work information (entry described later) to be newly created through the creation operation screen Gg. ..
  • the camera work window 53 displays at least information that visualizes the movement locus of the viewpoint as a display of the camera work.
  • the operation panel unit 54 is an area for receiving various operation inputs in creating camera work.
  • An observation image from the viewpoint is displayed in the preview window 55.
  • the preview window 55 sequentially displays the observation images from each viewpoint position on the movement locus. Further, as will be described later, when the preset list of the camera is displayed on the preset list display unit 51 in the preview window 55 of this example, the operation of designating the camera from the preset list of the camera is performed. Displays an observation image from the arrangement position of the camera.
  • FIG. 10 shows a state in which a front clip, an FV clip, and a rear clip are connected to each other as an example of an output clip.
  • the front clip is an actual moving image of the section of the time code TC1 to TC2 in a certain image data Vx of the image data V1 to the image data V16.
  • the rear clip is an actual moving image of the section of the time code TC5 to TC6 in a certain image data Vy of the image data V1 to the image data V16. It is usually assumed that the image data Vx is the image data of the image pickup device 10 before the start of the viewpoint movement by the FV clip, and the image data Vy is the image data of the image pickup device 10 at the end of the viewpoint movement by the FV clip.
  • the front clip is a video with a time length of t1
  • the FV clip is a free viewpoint image with a time length of t2
  • the rear clip is a video with a time length of t3.
  • the playback time length of the entire output clip is t1 + t2 + t3.
  • a configuration such as a 1.5-second moving image, a 2-second free-viewpoint image, a 1.5-second moving image, or the like can be considered.
  • the FV clip for moving the viewpoint while the video time is stopped is called “still image FV clip”
  • the FV clip for moving the viewpoint without stopping the video time is called "video FV clip". I will do it.
  • FIG. 10 shows the still image FV clip with reference to the frame of the moving image.
  • the time codes TC5 and TC6 of the rear clip become the time codes of the frames F83 and F166. That is, it is a case of generating a free viewpoint image in which the viewpoint moves with respect to the still image of one frame of the frame F82.
  • the moving image FV clip is as shown in FIG.
  • the time codes TC1 and TC2 of the front clip are the time codes of the frames F1 and F101
  • the time codes of the frames F102 and F302 are the time codes TC3 and TC4 of FIG.
  • the time codes TC5 and TC6 of the rear clip become the time codes of the frames F303 and F503. That is, it is a case where a free viewpoint image in which the viewpoint moves is generated for a moving image in a section of a plurality of frames from the frame F102 to the frame 302.
  • the generation target image section determined by the image creation controller 1 is one frame section of the frame F82 when the still image FV clip of FIG. 10 is created, and the frame when the moving image FV clip of FIG. 11 is created. It is a section of a plurality of frames from F102 to frame 302.
  • FIG. 12 an example of the image content of the output clip is shown in FIG.
  • the front clip is an actual moving image from frame F1 to frame F81.
  • the FV clip it becomes a virtual image in which the viewpoint is moved in the scene of the frame F81.
  • the rear clip is an actual moving image from frame F83 to frame F166.
  • an output clip including an FV clip is generated in this way and used as an image to be broadcast.
  • FIG. 13 will explain the flow of processing including the operations of operators OP1 and OP2. Note that the processing of the operator OP1 in FIG. 13 shows the GUI processing of the image creation controller 1 and the operator operation collectively. Further, the processing of the operator OP2 collectively shows the GUI processing and the operator operation of the free viewpoint image server 2.
  • Step S1 Scene selection
  • the operator OP1 first selects a scene to be used as an FV clip. For example, the operator OP1 searches for a scene to be used as an FV clip while monitoring the captured image displayed on the display unit 77 on the image creation controller 1 side. Then, the generation target image section of one frame or a plurality of frames is selected.
  • the information of the image section to be generated is transmitted to the free viewpoint image server 2, and the operator OP2 can be recognized by the GUI on the display unit 77 on the free viewpoint image server 2 side.
  • Step S2 Scene image transfer instruction
  • the operator OP2 operates the image transfer instruction of the corresponding scene according to the designation of the generation target image section.
  • the free-viewpoint image server 2 transmits a transfer request for image data in the sections of the time codes TC3 and TC4 to the image creation controller 1.
  • Step S3 Synchronous cutout
  • the image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D in response to the image data transfer request, and for each of the 16 systems of image data from the image data V1 to the image data V16. , The section of the time codes TC3 and TC4 is cut out.
  • Step S4 NAS transfer Then, the image creation controller 1 transfers the data of all the time codes TC3 and TC4 sections of the image data V1 to the image data V16 to the NAS5.
  • Step S5 Thumbnail display
  • the free-viewpoint image server 2 displays thumbnails of image data V16 from image data V1 in the sections of time codes TC3 and TC4 transferred to NAS5.
  • Step S6 Scene check
  • the operator OP2 confirms the scene contents of the section indicated by the time codes TC3 and TC4 on the camera work designation screen Gs by the free viewpoint image server 2.
  • Step S7 Camera work selection
  • the operator OP2 selects (designates) the camera work that is considered appropriate on the camera work designation screen Gs according to the scene content.
  • Step S9 Modeling
  • the free-viewpoint image server 2 uses parameters such as frame data in the sections of time codes TC3 and TC4 in each of the image data V1 to V16, and the arrangement position of each image pickup device 10 input in advance. Then, a 3D model of the subject is generated, the subject is analyzed, and the like.
  • the free viewpoint image server 2 generates a free viewpoint image based on the 3D model data and the subject analysis information. At this time, a free viewpoint image is generated so that the viewpoint is moved based on the camera work selected in step S7.
  • Step S11 Transfer The free viewpoint image server 2 transfers the generated FV clip to the image creation controller 1. At this time, not only the FV clip but also the specification information of the front clip and the rear clip and the specification information of the time length of the front clip and the rear clip can be transmitted as incidental information.
  • Step S13 Playlist generation
  • the image creation controller 1 generates an output clip using the transmitted FV clip.
  • one or both of the front clip and the rear clip are combined with the FV clip on the time axis to generate an output clip.
  • This output clip may be generated as stream data in which each frame as a front clip, each virtually generated frame as an FV clip, and each frame as a rear clip are actually connected in chronological order.
  • the output clip can be played back without generating the stream data.
  • Step S14 Quality confirmation
  • the GUI on the image creation controller 1 side performs playback based on the playlist, and the operator OP1 confirms the contents of the output clip.
  • Step S15 Reproduction instruction
  • the operator OP1 gives a reproduction instruction by a predetermined operation according to the quality confirmation.
  • the image creation controller 1 recognizes the input of the playback instruction.
  • Step S16 Reproduction The image creation controller 1 supplies the output clip to the switcher 6 in response to the reproduction / reproduction instruction. This makes it possible to broadcast the output clip.
  • a 3D model is generated using the image data V1, V2 ... V16, so that parameters including the position information of each imaging device 10 are important. For example, if the position of a certain imaging device 10 is moved during broadcasting or the imaging direction is changed in the pan direction, tilt direction, or the like, it is necessary to calibrate the parameters accordingly. Therefore, in the image processing system of FIG. 1, the utility server 8 detects fluctuations in the camera.
  • the fluctuation of the camera referred to here means that at least one of the position of the camera and the imaging direction changes.
  • FIG. 14 describes the processing procedure of the image creation controller 1 and the utility server 8 when detecting fluctuations in the camera. Note that FIG. 14 shows the processing procedure in the same format as in FIG. 13, but the operator OP2 also operates the utility server 8.
  • Step S30 HD output
  • the image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D to output image data to the image conversion unit 7 for camera movement detection.
  • the images from the video servers 4A, 4B, 4C, and 4D, that is, the images of the 16 image pickup devices 10 are converted in resolution by the image conversion unit 7 and supplied to the utility server 8.
  • Step S31 The background generation utility server 8 generates a background image based on the supplied image. Since the background image is an image that does not change unless the camera fluctuates, for example, a background image excluding a subject such as a player is generated for 16 systems of image data (V1 to V16). Step S32: Difference confirmation The background image is displayed in GUI, so that the operator OP2 can confirm the change in the image. Step S33: Automatic fluctuation detection By comparing the background images at each time point, the fluctuation of the camera can be automatically detected.
  • Step S34 Camera fluctuation detection As a result of the above steps S33 or S32, fluctuations in a certain imaging device 10 are detected.
  • Step S35 Image acquisition Calibration is required according to the detection of fluctuations in the image pickup apparatus 10. Therefore, the utility server 8 requests the image creation controller 1 for the image data in the changed state.
  • Step S36 Clip cutting The image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D in response to the request for image acquisition from the utility server 8 to execute clip cutting for the image data V1 to V16. ..
  • Step S37 NAS transfer The image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D to transfer the image data cut out as a clip to the NAS 5.
  • Step S38 Feature point correction
  • the utility server 8 can refer to and display the image of the state after the camera change.
  • the operator OP2 performs operations necessary for calibration such as correction of feature points.
  • Step S39 The recalibration utility server 8 re-executes the calibration for creating the 3D model using the image data (V1 to V16) in the state after the camera change.
  • Step S40 Background reacquisition
  • the utility server 8 requests reacquisition of image data for the background image in response to the operation of the operator OP2 after the calibration.
  • Step S41 Clip cutting
  • the image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D in response to the request for image acquisition from the utility server 8 to execute clip cutting for the image data V1 to V16. ..
  • Step S42 NAS transfer
  • the image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D to transfer the image data cut out as a clip to the NAS 5.
  • Step S43 The background generation utility server 8 generates a background image using the image data transferred to the NAS 5. This is, for example, a background image that serves as a reference for subsequent camera fluctuation detection.
  • FIG. 15 is a diagram illustrating an initial screen of the creation operation screen Gg.
  • the preset list display unit 51, the camera work list display unit 52, the camera work window 53, the operation panel unit 54, and the preview window 55 are arranged on the creation operation screen Gg.
  • the preset list display unit 51 is provided with a camera button B1, a target button B2, and a 3D model button B3.
  • the camera button B1 is a button for instructing the preset list display unit 51 to display the preset list of the camera described above, and the target buttons B2 and the 3D model button B3 are described above on the preset list display unit 51, respectively. It is a button for instructing to display the preset list of the target and the preset list of the 3D model as the background.
  • the camera button B1 is underlined, which means that the preset list display of the camera is selected.
  • the preset list display unit 51 is provided with a folder reference button B4.
  • the folder reference button B4 By operating the folder reference button B4, the user can refer to the folder in which the data to be displayed in the list is stored in the preset list display unit 51.
  • a new creation button B5 is provided for the camera work list display unit 52.
  • the new creation button B5 By operating the new creation button B5, the user can give an instruction to add a new entry of the camera work. The added camera work entry is displayed on the camera work list display unit 52.
  • the camera work window 53 is provided with an X viewpoint button B6, a Y viewpoint button B7, a Z viewpoint button B8, a Ca viewpoint button B9, and a Pe viewpoint button B10.
  • Each of these viewpoint buttons is a button for instructing an observation viewpoint for an object to be displayed in the camera work window 53.
  • the X-viewpoint button B6, the Y-viewpoint button B7, and the Z-viewpoint button B8 are the X-axis upper viewpoint and the Y-axis upper viewpoint as viewpoints for observing the visualization information of the camera work information displayed on the camera work window 53, respectively.
  • the button for instructing the viewpoint on the Z-axis, and the Pe viewpoint button B10 is a button for instructing the transition to the mode in which the observation viewpoint of the visualization information of the camera work information is changed to an arbitrary position.
  • the Ca viewpoint button B9 is a button that gives an instruction to display an image of observing the target three-dimensional space from the viewpoint movement locus defined as camera work information.
  • the display image in the camera work window 53 or the preview window 55 can be enlarged or reduced according to a predetermined operation such as a mouse wheel operation. Further, in the camera work window 53 and the preview window 55, it is possible to scroll the display image in response to a predetermined operation such as a drag operation. It should be noted that the enlargement / reduction and scrolling of the displayed image can be performed according to the operation of the buttons provided on the screen.
  • the operation panel unit 54 is provided with a play button B11, a pause button B12, a stop button B13, a timeline operation unit 54a, a speed adjustment operation unit 56, and a locus shape adjustment operation unit 57.
  • the play button B11, the pause button B12, and the stop button B13 play, pause, and stop the visualized image of the camera work information displayed in the camera work window 53 and the observed image from the viewpoint displayed in the preview window 55, respectively. It is a button to instruct.
  • the play button B11, the pause button B12, and the stop button B13 are enabled at least when the information on the movement locus of the viewpoint is determined as the camera work information.
  • the timeline operation unit 54a is an area that accepts operations related to the creation of camera work on the timeline that represents the movement period of the viewpoint of the free viewpoint image.
  • the operation for the timeline operation unit 54a for example, one of the cameras listed in the preset list display unit 51 is dragged and & dropped to an arbitrary position on the timeline (that is, an arbitrary time point within the viewpoint movement period).
  • the dropping operation can be mentioned (see FIGS. 27 to 29). As will be described later, this operation functions as an operation for designating the timing at which the viewpoint passes the position of the dragged and dropped camera within the movement period of the viewpoint.
  • the speed adjustment operation unit 56 is arranged with various operation buttons for adjusting the moving speed of the viewpoint. Further, the locus shape adjustment operation unit 57 is arranged with various operation buttons for adjusting the shape of the movement locus of the viewpoint. The speed adjustment operation unit 56 and the locus shape adjustment operation unit 57 will be described later.
  • the user performs an operation for acquiring the preset list of the camera as shown in FIG.
  • This operation is an operation for acquiring a preset list of cameras indicating the positions of the cameras actually installed in the field.
  • the preset list of the camera is acquired by operating the folder reference button B4 and specifying the corresponding folder.
  • the display processing unit 34a When the folder is designated, the display processing unit 34a performs a process of displaying the preset list of the camera according to the data contents of the designated folder on the preset list display unit 51. At the same time, the display processing unit 34a performs a process of displaying information visually indicating the arrangement of each camera on the camera work window 53 based on the acquired position information of the cameras. Specifically, a process of displaying a camera position mark Mc indicating the position of each camera is performed. Regarding the display of the camera position mark Mc, each camera may be identified by color coding. For example, each camera can be color-coded in the camera preset list, and each camera position mark Mc can be displayed in the same color-coded manner in the camera work window 43. It is also conceivable to display camera identification information (for example, camera1, camera2, ..., cameraX, etc.) on the camera position mark Mc over which the mouse is over in the camera work window 53.
  • camera identification information for example, camera1, camera2, ..., cameraX, etc.
  • the 3D model displayed as the background can be changed in the camera work window 53.
  • the user operates the 3D model button B3 to change the display state of the preset list display unit 51 to the preset of the 3D model.
  • the list is displayed.
  • the default designation button B14, the grid designation button B15, and the N / A designation button B16 as shown in the figure are displayed in the preset list display unit 51, and the user can switch the background 3D model by operating these buttons. It is possible to do.
  • the default designation button B14 is a button for instructing switching to a background 3D model (for example, a 3D model indicating a stage, a ground, etc.) prepared in advance as an initial setting, and the grid designation button B15 can visually recognize the distance and angle. It is a button for instructing switching to a background 3D model (for example, grid lines, squares, etc.).
  • the N / A designation button B16 is a button for instructing the display off of the background 3D model.
  • FIG. 17 shows an example of a background 3D model of the camera work window 53 when the Grid designation button B15 is operated.
  • FIG. 18 shows an example of a background 3D model of the camera work window 53 when the default designation button B14 is operated.
  • the display processing unit 34a displays a new entry of the camera work on the camera work list display unit 52.
  • an operation unit for designating an In camera as a start point of the viewpoint movement and an Out camera as the end point of the viewpoint movement is displayed.
  • the movement locus of the viewpoint is created as much as possible via the camera position. Is desirable.
  • the start point and the end point of the viewpoint movement in the free viewpoint image are the switching points of the image with the front and rear clips, the start point and the end point of the viewpoint movement should be the same as the camera position. Therefore, the start point and the end point of the viewpoint movement are designated as the camera positions as the In camera and the Out camera, respectively.
  • the movement start point and movement end point of the viewpoint are not necessarily limited to the camera position, and can be any position other than the camera position.
  • the operation unit for designating the In camera and Out camera is, for example, an operation unit for designating the camera in a pull-down format.
  • a pull-down is instructed by a user operation, information indicating a camera that can be specified, that is, each camera listed in the preset list of the camera specified by the user (camera number information in this example) is displayed (in this example, camera number information). See FIGS. 22 and 24, which will be described later).
  • a new entry of the camera work is displayed on the camera work list display unit 52 as described above, and the target Tg set by the user in the camera work window 53 is displayed.
  • a mark indicating the position (hereinafter referred to as "target mark Mt") is displayed.
  • the position of the target Tg is, for example, a position near the goal in the target three-dimensional space (for example, soccer ground in the case of soccer) when it is desired to generate an image of a shooting scene as a free viewpoint image. It is set to the appropriate position assumed for the target scene, such as being set to.
  • the position of the target Tg can be set in advance on the free viewpoint image server 2 by the user.
  • the free viewpoint image can be generated so that the line-of-sight direction from the viewpoint faces the target Tg.
  • the free viewpoint image of this example can be generated so that the target Tg continues to be positioned at a predetermined position (for example, the center position) in the image frame during at least a part of the movement period of the viewpoint. It is possible.
  • continuing to position the target Tg at a predetermined position in the image frame as described above is expressed as "following the target Tg". This "following the target Tg" is synonymous with keeping the line-of-sight direction from the viewpoint continuing to face the target Tg while the viewpoint is moving.
  • the field of view range when the viewpoint is set at the designated camera position in response to the camera designation operation from the camera preset list is displayed in the camera work window 53 and the preview window 55, respectively.
  • FIGS. 20 and 21 show an example of the display contents of the creation operation screen Gg when a camera as camera1 and a camera as camera2 are designated from the preset list of cameras.
  • the camera work window 53 displays the field of view information Fv that visualizes the field of view from the camera for the camera specified from the preset list of the cameras.
  • information representing the visual field range is displayed as the visual field range information Fv.
  • the camera position mark Mc for the specified camera is highlighted more than the camera position mark Mc for other cameras (an example of increasing the size is shown in the figure). This makes it possible for the user to easily grasp the position of the designated camera.
  • the preview window 55 an image of observing the three-dimensional space from the designated camera is displayed.
  • the camera work creation work is performed prior to the generation of the free viewpoint image. That is, it is premised that the captured image used for generating the free viewpoint image is performed in an unacquired state. Therefore, the image obtained by observing the three-dimensional space from the viewpoint referred to here is a 3D model generated by detecting the subject from the image captured by each camera that captures the target real space (hereinafter, "real three-dimensional" in the explanation.
  • 22 to 25 are explanatory views of a method for designating an In camera and an Out camera. As shown in FIG. 22, for the designation of the In camera, the camera designation operation is performed from the pull-down list of the In camera in the entry added to the camera work list display unit 52.
  • FIG. 23 shows the state of the creation operation screen Gg when camera1 is designated as the In camera.
  • camera1 is designated as the In camera
  • "1" is displayed in the In camera item in the entry added to the camera work list display unit 52 as shown in the figure.
  • the camera work window 53 the camera position mark Mc for camera1 is highlighted, and the field of view range information Fv for camera1 is displayed.
  • the display mode of the camera position mark Mc and the field of view range information Fv in the camera work window 53 depends on whether the camera is specified from the preset list of the camera or the In camera. It may have a different display mode.
  • the preview window 55 an image of observing the three-dimensional space from camera1 is displayed.
  • FIG. 25 shows the state of the creation operation screen Gg when camera9 is designated as the Out camera.
  • “9” is added to the Out camera item in the entry added to the camera work list display unit 52. Is displayed.
  • the movement trajectory of the viewpoint is determined. Therefore, in the camera work window 53 in this case, information indicating the movement locus of the viewpoint connecting the positions of the In camera and the Out camera, which is shown as the movement locus information Mm in the figure, is displayed.
  • the movement locus information Mm is information that visualizes the movement locus of the viewpoint.
  • the camera position mark Mc for the camera9 designated as the Out camera is highlighted, and the linear movement trajectory information connecting the positions of the In camera and the Out camera is displayed. Mm is additionally displayed from the case of FIG. 23.
  • the preview display can be instructed to start by operating the play button B11 on the operation panel unit 54.
  • the preview display of the camera work an image in which the visual field range information Fv changes momentarily as the viewpoint moves is displayed.
  • the preview window 55 displays an observation image (observation image from the viewpoint) of the three-dimensional space that changes every moment as the viewpoint moves. ..
  • FIG. 26 shows how the seek bar B17 is positioned at a desired position on the timeline by the drag operation of the seek bar B17 in the timeline operation unit 54a. While the seek bar B17 is being dragged, the position of the seek bar B17 changes every moment on the timeline, that is, on the time axis from the start timing to the end timing of the free viewpoint image.
  • the field of view range information Fv corresponding to the viewpoint position at the timing indicated by the seek bar B17 is sequentially displayed in the camera work window 53, and the user can move the viewpoint.
  • the visual field range information Fv is visually recognized as an image that changes from moment to moment.
  • the preview window 55 an observation image of the three-dimensional space from a viewpoint that changes from moment to moment is displayed according to the movement of the seek bar B17.
  • the waypoints of the viewpoint will be described.
  • a waypoint of the viewpoint and a timing at which the viewpoint passes through the waypoint are explanatory views for designating a waypoint and a way timing of the viewpoint.
  • the waypoint and the timing at which the viewpoint passes through the waypoint can be specified by dragging and dropping the camera to be designated as the waypoint on the timeline in the timeline operation unit 54a.
  • FIG. 27 to 29 are explanatory views of an operation example when camera 6 is designated as a waypoint.
  • a camera to be designated as a waypoint is selected from the preset list of the cameras on the preset list display unit 51.
  • the operation for this selection is the pressing operation of the left mouse button.
  • the camera selected in this way is dragged on the screen as shown in FIG. 28, and dropped at a desired position on the timeline of the timeline operation unit 54a as shown in FIG. 29 (left in this example). Release the pressed state of the click).
  • FIG. 29 the operation example when camera 6 is designated as a waypoint.
  • the waypoint mark Mv is displayed on the timeline of the timeline operation unit 54a.
  • This waypoint mark Mv is displayed at a position designated on the timeline by the above drop operation. That is, it is displayed at a position indicating a designated timing within the period from the start timing to the end timing of the free viewpoint image (within the movement period of the viewpoint).
  • the waypoint mark Mv is displayed by a square mark as shown in the initial state in the initial state.
  • the camera position mark Mc for the camera (here, camera6) designated as the waypoint is highlighted in the camera work window 53, and the waypoint is highlighted.
  • the field of view information Fv indicating the field of view from the camera is displayed.
  • an image of observing the three-dimensional space from the viewpoint of the camera position designated as the waypoint is displayed.
  • FIG. 30 illustrates the state of the creation operation screen Gg when a waypoint and a waying timing are specified for two cameras in the same manner as described above.
  • an example of designating the camera position as a waypoint of the viewpoint is given here, it is also possible to specify an arbitrary position other than the camera position as the waypoint.
  • FIG. 31 on the timeline of the timeline operation unit 54a, a target range for which the type of the movement locus is to be specified is specified.
  • a target range for which the type of the movement locus is to be specified is specified.
  • FIG. 30 when three waypoints of the viewpoint are set as shown in FIG. 30, an example in which the range from the first waypoint to the third waypoint is specified is shown.
  • the operation button provided on the locus shape adjustment operation unit 57 For example, as illustrated in FIG. 32, the curve interpolation button B18 provided in the locus shape adjustment operation unit 57 is operated.
  • the camera work generation processing unit 34 performs curve interpolation of the movement locus for a part of the specified range of the viewpoint movement locus designated in FIG. 31. Then, as illustrated in FIG. 33, the display processing unit 34a performs a process of displaying the movement locus information Mm generated by this curve interpolation on the camera work window 53.
  • the process of changing the shape of the waypoint mark Mv displayed on the timeline in the timeline operation unit 54a to a shape corresponding to the curve interpolation is performed. conduct. Specifically, in this example, as illustrated in the figure, the shape of the waypoint mark Mv is changed from the square mark to the round mark. As a result, it is possible to notify the user on the timeline that curve interpolation is being performed for the movement locus of the viewpoint connecting the waypoints.
  • an operation button for instructing the locus shape adjustment operation unit 57 to make the movement locus shape linear is arranged, and when the operation button is operated, the shape of the corresponding waypoint mark Mv. Is changed to a square mark.
  • the movement locus shape a shape other than a curved line or a straight line can be set.
  • the shape may be a mixture of curved lines and straight lines.
  • the movement locus by the curve is not limited to the constant curvature, and it is also possible to set the curvature to be different in some sections.
  • the waypoint mark Mv is not limited to the two types of display forms of straight lines and curves as illustrated above, but also has different display forms corresponding to each variation. May be possible to display.
  • the target range is specified on the timeline in the timeline operation unit 54a.
  • the operation buttons provided on the speed adjustment operation unit 56 are operated.
  • the speed adjustment button B19 provided on the speed adjustment operation unit 56 is operated.
  • the camera work generation processing unit 34 adjusts the speed of the viewpoint according to the operated button for a part range of the designated viewpoint movement locus.
  • the display processing unit 34a performs the shape of the corresponding waypoint mark Mv on the timeline as illustrated in FIG. 35. A process of changing the shape according to the mode is performed.
  • the indicated display mode is merely an example. By making such a shape change, it is possible to notify the user on the timeline that the speed adjustment has been performed in the corresponding range in the viewpoint movement locus.
  • the target Tg is used to determine the direction of the line of sight from the viewpoint in the free viewpoint image.
  • the visual field range Rf Rf1, Rf3, Rf6, Rf9 for each camera position
  • Dg Dg1, Dg3, Dg6, Dg9
  • the tracking of the target Tg is performed so that the target Tg is continuously positioned at the center position in the image frame as illustrated in FIG. 37, for example.
  • FIG. 36 illustrates the line-of-sight direction Dg and the field-of-view range Rf when following the target Tg within the movement period of the viewpoints from camera1 to camera9.
  • the field of view Rf is set so as to capture the target Tg at the center position in the image frame.
  • the position of the target Tg can be adjusted on the creation operation screen Gg.
  • the adjustment operation of the target Tg for example, an operation of adjusting the position of the target mark Mt displayed on the camera work window 53 can be considered.
  • the camera work generation processing unit 34 increases the line-of-sight direction Dg and the visual field range Rf at each viewpoint position so as to keep the changed target Tg at a predetermined position in the image frame. To set.
  • a new point Ptn of the target Tg is designated.
  • an operation of setting the target position designation mark Mtn for designating a new point Ptn at a desired position is performed.
  • the target position designation mark Mtn is displayed superimposed on the target mark Mt in the initial state, and the user drags and displays this target position designation mark Mtn from the position of the target mark Mt to a desired position. Drop it.
  • a new point Ptn of the target Tg is designated.
  • the user operates the target button B2 provided on the preset list display unit 51 to put the preset list display unit 51 in the display state of the list of the target Tg.
  • the addition button B20 of the target Tg is displayed on the preset list display unit 51, and the user operates the addition button B20 to perform the position specified by the target position designation mark Mtn. It is possible to give an instruction to add a new target Tg to the target.
  • the preset list display unit 51 displays the identification information (“Target0” in the figure) and the position information (position information indicating the new point Ptn) of the added target Tg as shown in the figure. Is displayed. Further, in the camera work window 53, an additional target mark Mtt as a mark representing the added target Tg is displayed at the position of the target position designation mark Mtn.
  • the user then performs an additional operation on the timeline of the new target as shown as the transition from FIGS. 40 to 42. Specifically, the operation of dragging and dropping the target newly displayed on the preset list display unit 51 (here, “Target0”) to a desired position on the timeline of the timeline operation unit 54a is performed. At the position on the timeline designated by the drop operation, the arrival target timing mark Mem as shown in FIG. 42 is displayed.
  • This arrival target timing mark Mem is a target when moving the position of the target Tg of the viewpoint from the position indicated by the target mark Mt (that is, the initial position of the target Tg) to the position indicated by the additional target mark Mtt (that is, the new point Ptn).
  • the operation of adding a new target on the timeline as described above is an operation of designating the target timing for reaching the position of the target Tg to the new point Ptn with respect to the movement of the target Tg.
  • the user After performing the operation of adding a new target on the timeline as described above, the user performs the operation of designating the period facing the target Tg as shown in FIGS. 43 to 45.
  • designating the period for facing the target Tg first, as shown in FIG. 43, the LookAt button B21 provided on the timeline operation unit 54a is operated. Then, as shown in the figure, the period designation bar B22 for specifying the period is displayed on the timeline.
  • the LookAt button B21 When the LookAt button B21 is operated, the period designation bar B22 is displayed in a manner of designating a period from the start time of movement of the viewpoint to the time point indicated by the arrival target timing mark Mem as shown in the figure.
  • the user When the user wants to change the period for facing the target Tg, the user performs an operation of extending or reducing the period designation bar B22.
  • the period designation bar B22 is extended and the period facing the target Tg is designated as the period from the start to the end of the movement of the viewpoint (the period from the start to the end of the movement of the viewpoint). It shall be assumed that it was done.
  • the movement of the target Tg is performed so as to reach the new point Ptn at the timing indicated by the arrival target timing mark Mem on the timeline within the period from the start to the end of the movement of the viewpoint. Therefore, as shown in FIGS. 46 and 47, the target mark Mt is the target position designation mark Mtn (in the camera work window 53, from the start of movement of the viewpoint to the timing indicated by the arrival target timing mark Mem). It will gradually approach the additional target mark Mtt).
  • the target initial position mark Mst is shown in each of the camera work window 53 and the preview window 55, and the target initial position mark Mst is the target at the start of movement of the viewpoint. It is a mark indicating the position of Tg.
  • the period of facing the target Tg is specified as the entire period from the start to the end of the movement of the viewpoint. That is, as the period for facing the target Tg, a period exceeding the period until the timing indicated by the arrival target timing mark Mem is specified.
  • the period of facing the target Tg is specified as a period exceeding the period up to the arrival target timing mark Mem in this way, in this example, as the movement of the target Tg in the period exceeding the period up to the arrival target timing mark Mem. , The position of the target Tg is gradually returned from the new point Ptn to the movement start position. Therefore, as shown in FIGS. 48 and 49, in the period in which the target mark Mt faces the target Tg and exceeds the period up to the arrival target timing mark Mem, the target mark Mt indicates the movement start position with the passage of time. It will gradually approach the position mark Mst.
  • the period exceeding the period up to the arrival target timing mark Mem is specified as the period for facing the target Tg is illustrated, but is shown in FIG.
  • the period for facing the target Tg the period from the start point of viewpoint movement to the arrival target timing mark Mem can also be specified.
  • a free viewpoint image that does not follow the designated target Tg is generated in the viewpoint movement period after the period up to the arrival target timing mark Mem.
  • the arrival target timing marks Mem-1 and Mem-2 are individually displayed on the timeline. Is shown.
  • the period from the start of movement of the viewpoint to the arrival target timing mark Mem-1 is specified as the period for facing the target Tg. do.
  • the arrival target timing starts from the time after a predetermined time elapses from the time indicated by the arrival target timing mark Mem-1 as the period for facing the target Tg. It is assumed that the period until the timing indicated by Mark Mem-2 is specified.
  • the position of the target Tg gradually changes from the initial position (the position of the target Tg at the start of the viewpoint movement) to the position of the target Tg-1.
  • the period indicated by the period designation bar B22-2 for example, a free viewpoint image in which the position of the target Tg gradually moves from the initial position to the position of the target Tg-2 is generated.
  • the creation operation screen Gg can accept the operation of designating the positions of a plurality of target Tg.
  • the creation operation screen Gg can accept the operation of designating the positions of a plurality of target Tg.
  • the position of the target Tg an example of designating the position of the target Tg as the movement destination point when the position of the target Tg is moved during the movement period of the viewpoint is given, but the position of the target Tg is specified. Can also be specified as the position of the target Tg that does not move during the viewpoint movement period.
  • FIGS. 52 and 53 processing related to generation and display of a movement locus according to an operation input on the creation operation screen Gg will be described.
  • the processes shown in FIGS. 52 and 53 are executed by the CPU 71 of the free-viewpoint image server 2. This process is a process for realizing a part of the functions of the camera work generation processing unit 34 described above.
  • FIG. 52 shows the processing related to the generation / display of the viewpoint movement locus according to the designation of the In camera and the Out camera.
  • the CPU 71 waits for the designated operation of the In camera.
  • this designation operation is performed as an operation of designating the camera number listed in the pull-down list of the In camera in the entry of the camera work displayed on the camera work list display unit 52 as illustrated in FIG.
  • the CPU 71 When there is an In-camera designation operation, the CPU 71 performs various display processes related to the In-camera as described in FIG. 23 as the In-camera display process in step S102. For example, in the camera work window 53, processing for highlighting the camera position mark Mc for the camera designated as the In camera, displaying the field of view range information Fv, and the like are performed.
  • step S103 the CPU 71 waits for an Out camera designation operation (see the description of FIG. 24), and if there is an Out camera designation operation, proceeds to step S104.
  • step S104 the CPU 71 performs a process of generating a viewpoint movement locus connecting the In camera and the Out camera. Then, in the following step S105, the CPU 71 executes the display process of the Out camera and the viewpoint movement locus. That is, various display processes related to the Out camera and display processing of the movement locus information Mm of the viewpoint as described with reference to FIG. 25 are performed. The CPU 71 completes a series of processes shown in FIG. 52 in response to executing the display process in step S105.
  • FIG. 53 shows a process related to the generation / display of the viewpoint movement locus according to the designation of the waypoint.
  • the CPU 71 waits for the operation of designating the waypoint.
  • this designation operation is a series of operations including an operation on the timeline as described with reference to FIGS. 26 to 28.
  • the CPU 71 When the waypoint is designated, the CPU 71 generates a viewpoint movement locus passing through the designated point in step S111. That is, the movement locus of the viewpoint connecting the In camera, the designated point, and the Out camera is generated. Then, in the following step S112, the CPU 71 performs display processing of the waypoint and the viewpoint movement locus. That is, for the designated waypoint, processing for highlighting the camera position mark in the camera work window 53, displaying the field of view range information Fv, and displaying the waypoint mark Mv in the timeline as described with reference to FIG. 29. conduct. The CPU 71 completes a series of processes shown in FIG. 53 in response to the execution of the display process in step S112.
  • FIG. 54 is a diagram illustrating an initial screen of the camera work designation screen Gs.
  • the processing related to the display of various information on the screen of the camera work designation screen Gs described below is performed by the display processing unit 32a (see FIG. 5) described above.
  • the camera work designation screen Gs is provided with a scene window 41, a scene list display unit 42, a camera work window 43, a camera work list display unit 44, a parameter display unit 45, and a transmission window 46. Further, the camera work designation screen Gs is provided with a camera designation operation unit 47, a still image import button B31, and a video import button B32 for the scene window 41, and a play button B33, a pause button B34, and a pause button B34 at the bottom of the screen. A stop button B35 is provided.
  • the X-axis viewpoint button B36, the Y-axis viewpoint button B37, the Z-axis viewpoint button B38, the Ca viewpoint button B39, the Pe viewpoint button B40, and the display path restriction button B41 are displayed on the camera work window 43.
  • the restriction release button B42 are provided, and a filtering operation unit 48 is provided for the camera work list display unit 44.
  • the filtering operation unit 48 is provided with a pull-down button B43 and a reset button B44.
  • the user In the generation of the free viewpoint image on the camera work designation screen Gs, the user first sets the image of the free viewpoint image generation target section as the above-mentioned image data V1 to V16, in other words, the generation target of the free viewpoint image. Perform the operation to import the image of the scene to be performed. In performing this import, the user operates either the still image import button B31 or the moving image import button B32 in the figure.
  • the still image import button B31 is a button for instructing to import the image data V1 to V16 of the still image for generating the still image FV clip described above as a free viewpoint image
  • the moving image import button B32 is a free viewpoint. It is a button for giving an instruction to import V16 from the moving image data V1 for generating the moving image FV clip described above as an image.
  • the pop-up window W1 as shown in FIG. 55 is displayed on the camera work designation screen Gs, and the user is provided in the pop-up window W1.
  • the "GET IN / OUT TC" button as shown in FIG. 56, information indicating V1 to V16 of the image data to be imported can be displayed in the pop-up window W1.
  • the user operates the OK button provided in the pop-up window W1.
  • the information of the imported scene is added in the scene list display unit 42, and the image of the imported scene is displayed in the scene window 41. Is displayed.
  • the scene information added to the list by import includes thumbnail images of the scene, time information indicated by the time code (in this example, both the start time and end time of the relevant scene are displayed), information indicating the period of the scene, and the like. Is displayed.
  • the illustrated example is a case where V1 to V16 as still images are imported, and the value indicating the period of the scene is set to "0".
  • the camera designation operation unit 47 is provided with a camera selection button for selecting which camera to display the image of each camera for the imported scene, that is, the image data V1 to V16. ..
  • the X-axis viewpoint button B36, the Y-axis viewpoint button B37, the Z-axis viewpoint button B38, the Ca viewpoint button B39, and the Pe viewpoint button B40 are used to switch the observation viewpoint of the camera work in the three-dimensional space. It is possible to do it.
  • the X-axis viewpoint button B36, the Y-axis viewpoint button B37, and the Z-axis viewpoint button B38 are buttons for switching the observation viewpoint in the three-dimensional space between the viewpoint on the X-axis, the viewpoint on the Y-axis, and the viewpoint on the Z-axis, respectively.
  • the X-axis, the Y-axis, and the Z-axis are three axes that define a three-dimensional space.
  • the X-axis is in the horizontal direction
  • the Y-axis is in the vertical direction
  • the Z-axis is in both the X-axis and the Y-axis. It is an axis that defines the directions that are orthogonal to each other.
  • the Pe viewpoint button B40 is a button for switching the observation viewpoint in the three-dimensional space to an arbitrary viewpoint specified by the user.
  • the Ca viewpoint button B39 is a button for switching the observation viewpoint in the three-dimensional space to the viewpoint (point on the viewpoint movement locus) in the camera work.
  • 58, 59, 60, 61, and 62 show the case where the X-axis viewpoint button B36, the Y-axis viewpoint button B37, the Z-axis viewpoint button B38, the Pe viewpoint button B40, and the Ca viewpoint button B39 are operated, respectively.
  • the display image in the camera work window 43 is illustrated.
  • the camera work displayed on the camera work list display unit 44 is the camera work created through the above-mentioned creation operation screen Gg, and is a candidate for the camera work used for generating the free viewpoint image. In other words, it is a camera work that can be specified as a camera work used for generating a free viewpoint image.
  • the camera work window 43 As the information indicating the camera work, the above-mentioned movement locus information Mm, the camera position mark Mc, and the field of view range information Fv (also referred to as information indicating the field of view by a figure) are displayed. Further, as shown in FIG. 61, the camera position mark Mc indicating the positions of the In camera and the Out camera is displayed larger than the other camera position marks Mc, and the positions of the In camera and the Out camera are shown.
  • the camera position mark Mc of the In camera and the Out camera that is, the information indicating each position of the camera that is the movement start point of the viewpoint and the camera that is the movement end point (start point arrangement position information, end point arrangement position information) is described as In.
  • the target mark Mt indicating the position of the target Tg described above is displayed as the information indicating the camera work.
  • the movement locus information Mm, the camera position mark Mc, and the field of view range information Fv are not displayed because the observation image is obtained from the viewpoint movement locus. , Target mark Mt is displayed.
  • two target marks Mt are displayed as target marks Mt (see particularly FIGS. 60 and 61), but this is a plurality of candidates displayed on the camera work list display unit 44.
  • the camera work it means that the camera work in which different target Tg is set is mixed. That is, for the camera work as a candidate in this case, for example, the target Tg whose position is indicated by the target mark Mt on the left side in FIG. 61 is set, and the target mark Mt on the right side in FIG. The target Tg whose position is indicated is mixed with the set target Tg.
  • Such dynamic preview playback can be instructed by operating the playback button B33.
  • the play button B33 When the play button B33 is operated, in each of the X-axis upper viewpoint, Y-axis upper viewpoint, Z-axis upper viewpoint, and arbitrary viewpoint shown in FIGS. 58 to 61, the visual field range is accompanied by the movement of the viewpoint on the movement locus. An image in which the position and shape of the information Fv changes from moment to moment is displayed. Further, in the case of FIG. 62, the images of observing the target three-dimensional space from each point from the viewpoint movement start point to the viewpoint movement end point on the viewpoint movement locus are sequentially switched and displayed. That is, the observation image of the three-dimensional space that changes every moment as the viewpoint moves is displayed.
  • the pause button B12 and the stop button B13 are buttons for instructing the pause and stop of the dynamic preview playback as described above, respectively.
  • the free viewpoint image as an FV clip generated by using the selected camera work can be displayed as the display image. .. That is, the above-mentioned real three-dimensional model is generated from the imported image data V1 to V16, and a two-dimensional image rendered by pasting a texture on the real three-dimensional model is displayed as a preview image. ..
  • the generation of the free-viewpoint image as an FV clip requires a considerable processing load and processing time, it takes a long time for the user to wait until the start of the preview playback, which hinders the rapid creation of the free-viewpoint image. There is a risk of coming. Therefore, in this example, when the observation image from the viewpoint is preview-reproduced as in the case of FIG. 62, the actual three-dimensional image generated based on the image data V1 to V16 (that is, the captured image in the real space) is displayed as the display image. Instead of the model, the image obtained by rendering the above-mentioned virtual three-dimensional model (virtual 3D model in real space) is displayed. As a result, the processing time required for previewing the observation image from the viewpoint can be shortened, and the work of creating the free viewpoint image can be executed quickly.
  • the camera work window 43 it is possible to display the selected camera work or a plurality of selectable camera works.
  • FIGS. 58 to 62 the case where the camera work information about only the selected camera work is displayed is illustrated, but in the camera work window 43, all the plurality of camera works displayed on the camera work list display unit 44 are displayed. It is also possible to display the camera work information of. In this way, switching of the number of camera works to be displayed in the camera work window 43 can be instructed by the display path restriction button B41 and the restriction release button B42.
  • the display path restriction button B41 is a button for instructing the display of only the selected camera work
  • the restriction release button B42 is for releasing the state restricted to the display of only the selected camera work. Button, and functions as an instruction button for displaying the camera work information of all the plurality of camera works displayed on the camera work list display unit 44.
  • the camera work list display unit 44 displays camera work as a candidate that can be used to generate a free-viewpoint image (see, for example, FIG. 63).
  • the camera work information displayed on the camera work list display unit 44 includes a camera work ID, identification information of an In camera or an Out camera, tag information, and the like.
  • the camera work list display unit 44 displays a thumbnail image of the movement locus information Mm for each camera work. By displaying such a thumbnail image, the user can be made to confirm what kind of viewpoint movement locus each camera work has on the camera work list.
  • the above tag information is information that can be attached to each created camera work when creating a camera work through the above-mentioned creation operation screen Gg, and in this example, it is information in text.
  • the tag information for the camera work is set, for example, by inputting information to the "Tag" field (see, for example, FIG. 19 and the like) provided in the entry of the camera work on the camera work list display unit 52 of the creation operation screen Gg. It can be carried out.
  • this tag information will be referred to as "tag information I1".
  • the camera work list display unit 44 is provided with a filtering operation unit 48 for filtering the camera work to be displayed in the list, that is, the camera work displayed on the camera work list display unit 44.
  • the function related to the filtering of the camera work using the filtering operation unit 48 will be described with reference to FIGS. 63 to 65.
  • the user operates the pull-down button B43 in the filtering operation unit 48.
  • the pull-down list 48a is displayed.
  • a list of tag information I1 is displayed in the pull-down list 48a.
  • the tag information I1 displayed in the pull-down list 48a is the tag information I1 set for each camera work as a candidate. That is, when there is a camera work as a candidate in which tag information I1 such as "CW, Cam9" or "CW, Right” is set as illustrated in the figure, these "CW,” are displayed in the pull-down list 48a.
  • Tag information I1 such as "Cam9" and "CW, Right” is displayed.
  • each tag information I1 in the pull-down list 48a corresponds to the filtering condition information indicating the filtering conditions for filtering and displaying the camera work information.
  • FIG. 64 illustrates the state of the camera work designation screen Gs when "CW, Right” is designated as the tag information I1.
  • the camera work in which "CW, Right” is set as the tag information I1 is displayed on the camera work list display unit 44.
  • the information of the camera work in which "CW, Right” was set is displayed in the camera work window 43.
  • the camera work in which "CW, Right” is set is positioned at the target mark Mt on the right side of the target Tg whose position is indicated by the two target marks Mt shown in the camera work window 43 of FIG. 61 above. Is the set camera work of the target Tg shown in.
  • the camera work window 43 in this case, only one target mark Mt is displayed as the camera work information.
  • the camera work information displayed at a predetermined position such as the head position on the list is displayed in the camera work window 43. Can be considered.
  • the camera work By filtering the camera work based on the tag information I1 as described above, it is possible to realize filtering according to an arbitrary standard depending on the information content set as the tag information I1. For example, if team information (for example, team A, team B, etc.) is set as tag information I1, the camera work targets the shooting scene of team A, or the shooting scene of team B is targeted. It is possible to realize filtering based on criteria such as whether or not it is a product. Alternatively, by setting information indicating the moving direction of the viewpoint (for example, clockwise, counterclockwise, etc.) as the tag information I1, filtering based on the moving direction of the viewpoint can be realized. Further, by setting the camera closest to the field of view of interest, such as the field of view closest to the subject as the target Tg, as tag information I1, it is possible to realize filtering of camera work based on the field of view of interest. ..
  • the reset button B44 is a button for instructing the reset of the filtering.
  • the reset button B44 is operated, as illustrated as a screen transition from FIG. 64 to FIG. 65, the filtering display state of the camera work in the camera work list display unit 44 is canceled, and a candidate used for generating a free viewpoint image is released.
  • the filtering based on the tag information I1 has been illustrated, but the filtering of the camera work can also be performed based on the information of the In camera and the Out camera included in the information of the camera work. be.
  • the information indicating the filtering condition such as the tag information I1 is displayed in the pull-down list 48a, but the information indicating the filtering condition (filtering condition information) is displayed as a button as illustrated in FIG. You can also do it.
  • the information to be displayed as a button can be determined based on the history information of the camera work used for generating the free viewpoint image in the past.
  • the tag information I1 of the upper predetermined camera work that has been used frequently in the past can be displayed as a button.
  • FIG. 66 exemplifies the button arrangement corresponding to the case where the upper predetermined camera work that has been used frequently is the camera work with the tag information I1 such as “goal mouth”, “Left”, “Right”, and the like. doing.
  • the button displaying the filtering condition information may be customized by the user.
  • FIG. 67 shows an example of button display in that case.
  • the user can set arbitrary information about the display information of each button.
  • the image generation processing unit 32 manages the set information as information indicating the filtering condition of the camera work. For example, the information of "Team A”, “Left”, and “Right” illustrated in the figure is managed as information indicating the filtering condition of the camera work.
  • the image generation processing unit 32 (specifically, the display processing unit 32a) sets the camera work in which the tag information I1 that matches the information managed corresponding to the button is set. The process of displaying on the list display unit 44 is performed.
  • the information indicating the filtering condition can be accepted as the input keyword information by the user.
  • the keyword input unit 48b as illustrated in FIGS. 66 and 67 is provided in the filtering operation unit 48.
  • the display processing unit 32a displays the camera work in which the tag information I1 matching the input keyword information is set on the camera work list display unit 44 in response to the keyword input to the keyword input unit 48b. Perform processing.
  • the camera work used to generate the free viewpoint image is specified as the camera work displayed on the camera work list display unit 44, but the camera work is specified in the camera work window. It is also possible to make it as a designation of the camera work displayed on 43.
  • the information of the camera work to be displayed in the camera work window 43 is displayed by visualizing the moving speed of the viewpoint.
  • 68 and 69 show an example of displaying visualization information of the moving speed of the viewpoint.
  • FIGS. 68 and 69 show an example in which the observation image of the camera work information from the above-mentioned Y-axis viewpoint is displayed in the camera work window 43.
  • information that visualizes the moving speed of the viewpoint information indicating the period during which the moving speed of the viewpoint decreases is displayed.
  • FIG. 68 shows an example in which the camera position mark Mc indicating the camera located in the section where the moving speed of the viewpoint decreases is displayed in a display mode different from that of other camera position mark Mc.
  • the corresponding camera position mark Mc is displayed in a color and size different from those of other camera position marks Mc.
  • FIG. 69 shows an example in which the display mode of the corresponding section in the movement locus information Mm is different from that of other sections for the period during which the movement speed of the viewpoint decreases.
  • the movement locus of the corresponding section as a dotted line and the movement locus of another section as a solid line.
  • the moving speed of the viewpoint can be expressed by the density of points when the moving locus is indicated by the dotted line.
  • the density of points increases as the moving speed increases.
  • FIGS. 68 and 69 Although the observation images from the Y-axis viewpoint are illustrated in FIGS. 68 and 69, the same display is performed when the X-axis viewpoint, the Z-axis viewpoint, and the Pe viewpoint (arbitrary viewpoint) are used.
  • a process of updating the information of the position of the target Tg in the camera work information is performed in response to the operation of changing the position of the target mark Mt.
  • This process is the process of the camera work editing processing unit 32b shown in FIG.
  • FIG. 70 illustrates the camera work information displayed in the camera work window 43. It is assumed that the operation of changing the position of the target mark Mt is performed in the camera work window 43 as shown in the figure.
  • the operation of changing the position of the target mark Mt may be, for example, a drag-and-drop operation of the target mark Mt.
  • position Pta the position of the target Tg after the change by such a change operation
  • Ptb the position of the target Tg before the change
  • the camera work editing processing unit 32b positions the position information of the target Tg with respect to the camera work information displayed in the camera work window 43. The process of updating from Ptb to the position Pta is performed.
  • FIG. 71 shows an image of a change in the visual field range Rf when the line-of-sight direction Dg from each position on the viewpoint movement locus is directed to the updated position Pta.
  • the display processing unit 32a displays the camera position mark Mc on the camera work window 43 based on the result of the fluctuation detection of each camera (for example, the automatic fluctuation detection in step S33) by the utility server 8 described with reference to FIG. Among the cameras displayed with, it is determined whether or not there is a camera in which fluctuation is detected. When there is a camera in which fluctuation is detected, the display processing unit 32a performs display processing of information for notifying the corresponding camera (that is, the camera in which fluctuation is detected) in the camera work window 43.
  • the camera in which the fluctuation is detected can be rephrased as the camera in which the change in the visual field range is detected.
  • FIG. 72 shows a display example of the notification information of the camera in which the fluctuation is detected.
  • the camera position mark Mc of the corresponding camera is displayed in a display mode different from that of the other camera position mark Mc (also in this case, the color, size, shape, etc. are different). Can be considered). It is also conceivable to display information for calling attention in the vicinity of the corresponding camera position mark Mc, such as the exclamation mark illustrated in the figure.
  • FIGS. 73 and 74 The processing related to the filtering of the camera work described above will be described with reference to the flowcharts of FIGS. 73 and 74.
  • the processes shown in FIGS. 73 and 74 are executed by the CPU 71 of the free-viewpoint image server 2 as the processing of the display processing unit 32a.
  • FIG. 73 shows a process corresponding to the case where the camera work is filtered based on the tag information I1 displayed on the camera work designation screen Gs as in FIG. 63 above.
  • the CPU 71 performs a process of acquiring tag information I1 in each camera work information as a candidate. That is, each camera work information as a candidate that can be used for generating a free viewpoint image is acquired.
  • the camera work information as a candidate is stored in a readable storage device inside or outside the free-viewpoint image server 2.
  • the camera work information as the candidate stored in this way is acquired.
  • step S202 following step S201 the CPU 71 performs display processing of tag information I1. That is, when displaying on the pull-down list 48a as shown in FIG. 63, the tag information I1 included in the camera work information acquired in step S201 is displayed in response to the operation of the pull-down button B43.
  • step S203 the CPU 71 waits for the tag information I1 designation operation, and if there is a tag information I1 designation operation, proceeds to step S204 to perform camera work with the designated tag information I1.
  • Performs filtering and display processing That is, among the candidate camera work information, the camera work information including the designated tag information I1 (where the designated tag information I1 is set) is displayed on the camera work list display unit 44.
  • the camera work information to be displayed is, for example, camera work identification information, In camera, Out camera information, tag information I1 and the like.
  • the CPU 71 ends a series of processes shown in FIG. 73 in response to executing the process of step S204.
  • FIG. 74 shows the processing related to the filter rig of the camera work according to the input keyword.
  • the CPU 71 waits for the keyword input from the user in step S210, and when there is a keyword input, selects the camera work including the input keyword in step S211. That is, among the camera work information as candidates, the camera work information including the input keyword in the tag information I1 is selected. Then, in the following step S212, the CPU 71 executes a process of displaying the selected camera work. That is, the process of displaying the selected camera work information on the camera work list display unit 44 is performed. In response to the execution of the process of step S212, the CPU 71 ends a series of processes shown in FIG. 74.
  • FIG. 75 is a flowchart of a process related to notification of a camera requiring calibration illustrated in FIG. 72.
  • the CPU 71 of the free viewpoint image server 2 executes the processing as the processing of the display processing unit 32a, as in the processing of FIGS. 73 and 74.
  • step S301 the CPU 71 waits for a camera fluctuation notification. That is, the utility server 8 waits for the fluctuation notification to be transmitted when the fluctuation of the camera is detected by the above-mentioned automatic fluctuation detection (step S33 in FIG. 14).
  • the fluctuation notification includes information for identifying the camera in which the fluctuation is detected.
  • the CPU 71 determines in step S302 whether or not the camera is being displayed. That is, it is determined whether or not the camera notified by the fluctuation notification is the camera displaying the camera position mark Mc in the camera work window 43. If it is not the camera being displayed, the CPU 71 ends a series of processes shown in FIG. 75.
  • step S303 the CPU 71 proceeds to step S303 to execute the fluctuation notification process. That is, for the corresponding camera position mark Mc displayed in the camera work window 43, information for notifying the change is displayed in the display mode as illustrated in FIG. 72, for example.
  • the CPU 71 ends a series of processes shown in FIG. 75.
  • the filtering display of the camera work on the camera work designation screen Gs an example of filtering according to the operated portion such as a button showing the filtering condition is given, but it is displayed in the camera work window 43, for example. It is also possible to perform filtering display of camera work according to the designation of the target Tg such as the designation of the target mark Mt. Specifically, among the camera works as candidates, only the camera works for which the designated target Tg is set are filtered and displayed.
  • the creation operation screen Gg and the camera work designation screen Gs there is a range in which the image quality cannot be guaranteed (for example, a range in which the resolution is equal to or less than a predetermined value) on the movement locus of the viewpoint, for example, because the actual camera is too far from the subject.
  • a range in which the image quality cannot be guaranteed for example, a range in which the resolution is equal to or less than a predetermined value
  • information notifying the range can also be displayed.
  • the first information processing apparatus of the embodiment has at least a part of the camera work information as a camera work information creation operation screen (Gg) which is information indicating at least the movement trajectory of the viewpoint in the free viewpoint image.
  • Gg camera work information creation operation screen
  • It is provided with a display processing unit (34a) that performs screen display processing including a camera work display area (camera work window 53) to be displayed.
  • the user can perform the camera work creation operation while visually recognizing the movement locus of the visualized viewpoint on the camera work creation operation screen. Therefore, the efficiency of the camera work creation work can be improved.
  • the designated operation receiving area can receive the designated operation of the start point and the end point of the movement locus (see FIGS. 22 to 25 and 52). As a result, it is possible to set an arbitrary start point and end point for the movement locus of the viewpoint instead of being fixed. Therefore, it is possible to improve the degree of freedom in creating a free viewpoint image.
  • the designated operation receiving area can receive the designated operation of the waypoint of the viewpoint (see FIGS. 27 to 30 and 53).
  • the designated operation receiving area can receive the designated operation of the waypoint of the viewpoint (see FIGS. 27 to 30 and 53).
  • the designated operation receiving area is capable of receiving the designated operation of the timing at which the viewpoint passes through the waypoint (see FIGS. 27 to 30). This makes it possible to set not only the waypoint of the viewpoint but also the timing of the viewpoint passing through the waypoint. Therefore, it is possible to improve the degree of freedom in setting the timing at which the viewpoint passes through the waypoint as well as the degree of freedom in setting the position through which the viewpoint passes, and it is possible to improve the degree of freedom in creating the free viewpoint image.
  • the designated operation receiving area can receive the designated operation of the shape type of the movement locus (see FIGS. 32 and 33).
  • This makes it possible to change the shape type of the movement locus of the viewpoint instead of fixing it. Therefore, it is possible to improve the degree of freedom in creating a free viewpoint image.
  • the shape type of the movement locus is a curved shape, it is possible to prevent the distance from the target subject to the viewpoint from changing significantly even if the viewpoint moves. In other words, it is possible to prevent the size of the target subject in the free-viewpoint image from changing significantly.
  • the designated operation receiving area can receive the designated operation of the moving speed of the viewpoint (see FIGS. 34 and 35). This makes it possible to change the moving speed of the viewpoint instead of fixing it. Therefore, it is possible to improve the degree of freedom in creating a free viewpoint image.
  • the designated operation receiving area can receive the designated operation of the section in which the moving speed is changed in the moving locus (see FIGS. 34 and 35). .. This makes it possible to dynamically change the moving speed of the viewpoint in the moving locus. Therefore, it is possible to improve the degree of freedom in creating a free viewpoint image.
  • the designated operation receiving area can accept the operation input for the timeline indicating the period from the movement start time to the movement end time of the viewpoint (timeline operation). See part 54a).
  • timeline operation By accepting input operations on the timeline, for example, by dragging and dropping the camera icon on the timeline, it is possible to specify the waypoint and the waypoint timing at the same time, or the time.
  • the range by dragging on the line it is possible to specify the section that should give a predetermined effect, such as the section where the curve interpolation of the movement trajectory should be performed, and facilitate the operation of specifying various information related to camera work. It becomes possible to plan. Therefore, the efficiency of the camera work creation work can be improved.
  • the display processing unit performs a process of displaying information (visual field range information Fv) that visualizes the visual field range from the viewpoint in the camera work display area (the visual field range information Fv). (See FIG. 20 and the like).
  • the display processing unit performs a process of displaying information representing the visual field range from the viewpoint in a graphic shape in the camera work display area.
  • the display processing unit By displaying the field of view in a graphic form, the user can easily grasp the camera work. Therefore, it is possible for the user to easily grasp how the camera work is changed by the operation input, and the efficiency of the camera work creation work can be improved.
  • the display processing unit performs a process of displaying an image of observing the three-dimensional space from the viewpoint on the creation operation screen (see the preview window 55).
  • the display processing unit performs a process of displaying an image of observing the three-dimensional space from the viewpoint on the creation operation screen (see the preview window 55).
  • the designated operation receiving area can receive the designated operation of the position of the target that determines the line-of-sight direction from the viewpoint (see FIGS. 38, 39, etc.). ..
  • the image that follows the target means an image in which the target is continuously positioned at a predetermined position (for example, the center position) in the image frame. Therefore, it is possible to improve the degree of freedom in creating a free viewpoint image.
  • the designated operation receiving area is capable of receiving designated operations during the period facing the target (see FIGS. 41 to 52).
  • the period of facing the target means a period of keeping the target positioned at a predetermined position in the image frame of the free viewpoint image.
  • the designated operation receiving area can accept a plurality of target position designation operations as target position designation operations (see FIG. 51).
  • target position designation operations see FIG. 51.
  • the information processing apparatus provides at least a part of the camerawork information as an operation screen for creating camerawork information which is information indicating at least the movement locus of the viewpoint in the free viewpoint image.
  • the second information processing apparatus of the embodiment has a plurality of camera work information as a camera work designation screen (Gs) that accepts a camera work information designation operation which is information indicating at least the movement locus of the viewpoint in the free viewpoint image.
  • a display processing unit (32a) for performing display processing of a screen showing by filtering camera work information according to user input information is provided.
  • the display processing unit performs a process of filtering and displaying the camera work information according to the keyword as the input information on the camera work designation screen (FIG. 66). , FIG. 67, FIG. 74, etc.).
  • This makes it possible to perform appropriate filtering of camera work information that reflects the user's intention. Therefore, it becomes possible to make it easier for the user to find the desired camera work information, and it is possible to further shorten the time required to specify the camera work information.
  • an operated unit indicating filtering conditions for camera work information is arranged on the camera work designation screen, and the display processing unit responds to the operation of the operated unit.
  • the camera work information is filtered and displayed according to the filtering conditions indicated by the operated unit (see FIGS. 63, 64, 66, 67, and 73).
  • the operation required for the filtering display of the camera work information can be limited to the selection operation of the filtering condition information. Therefore, it is possible to reduce the operational burden on the user required for the filtering display of the camera work information.
  • the display processing unit performs a process of displaying information that visualizes the movement locus of the viewpoint on the camera work designation screen (see FIG. 61 and the like).
  • the display processing unit performs a process of displaying information that visualizes the movement trajectory of the viewpoint on the camera work designation screen (see FIG. 61 and the like).
  • the display processing unit displays camera placement position information indicating the placement positions of a plurality of cameras that perform imaging for generating a free-viewpoint image on the camera work designation screen. (See FIG. 61 and the like).
  • camera placement position information indicating the placement positions of a plurality of cameras that perform imaging for generating a free-viewpoint image on the camera work designation screen.
  • the display processing unit indicates, on the camera work designation screen, the positions of the camera that is the movement start point of the viewpoint and the camera that is the movement end point of the plurality of cameras. Processing is performed to display the placement position information and the end point placement position information (see FIG. 61 and the like). This makes it possible for the user to grasp from which camera position the movement of the viewpoint starts and ends at which camera position. Therefore, when specifying the camera work information used for creating the free viewpoint image, it is possible to make it easier for the user to find the desired camera work information.
  • a camera that serves as a viewpoint movement starting point is used so that the connection between the clips becomes natural. It is desirable to match the image camera of the front clip and the camera that is the end point of movement of the viewpoint with the image camera of the rear clip, but as described above, the positions of the camera that is the start point of movement and the camera that is the end point of movement are displayed. By doing so, it is possible to easily specify appropriate camera work according to the image pickup camera of the front clip and the image pickup camera of the rear clip.
  • the display processing unit has the start point arrangement position information and the end point arrangement position information, and the arrangement positions other than the camera that is the movement start point and the camera that is the movement end point among the plurality of cameras. Processing is performed to display information in a different manner. This makes it possible for the user to intuitively grasp from which camera position the movement of the viewpoint starts and ends at which camera position. Therefore, when specifying the camera work information used for creating the free viewpoint image, it is possible to make it easier for the user to find the desired camera work information.
  • the display processing unit performs a process of displaying information that visualizes the moving speed of the viewpoint on the camera work designation screen (see FIGS. 68 and 69). ).
  • the period in which the viewpoint is moved the period in which the movement speed of the viewpoint is changed is an important factor in creating a free viewpoint image. Therefore, by displaying the visualization information of the moving speed of the viewpoint as described above, it is possible to make it easier for the user to find the desired camera work information, and to shorten the time required to specify the camera work information. Can be done.
  • the display processing unit performs a process of displaying information indicating a period during which the moving speed decreases as information that visualizes the moving speed of the viewpoint.
  • the period in which the movement speed of the viewpoint is reduced is an important factor in creating a free viewpoint image. Therefore, by displaying the information indicating the period during which the moving speed of the viewpoint decreases as described above, it is possible to make it easier for the user to find the desired camera work information, and the time required for specifying the camera work information can be shortened. Can be planned.
  • the display processing unit performs a process of displaying information that visualizes the visual field range from the viewpoint on the camera work designation screen (see FIG. 61 and the like). ).
  • the display processing unit By visually showing the field of view, it is easy for the user to grasp the camera work. Therefore, it is possible to make it easier for the user to find the desired camera work information, and it is possible to shorten the time required to specify the camera work information.
  • the display processing unit performs a process of displaying a target that determines the line-of-sight direction from the viewpoint on the camera work designation screen (see FIG. 61 and the like). This makes it possible for the user to easily grasp which position of the subject in the three-dimensional space the camera work is intended for. Therefore, it is possible to make it easier for the user to find the desired camera work information, and it is possible to shorten the time required to specify the camera work information.
  • a camera work editing processing unit (32b) that updates the target position information in the camera work information in response to a change in the target position on the camera work designation screen is provided. (See FIGS. 70 and 71).
  • the display processing unit performs a process of displaying an image of observing the three-dimensional space from the viewpoint on the camera work designation screen (see FIG. 62).
  • the display processing unit performs a process of displaying an image of observing the three-dimensional space from the viewpoint on the camera work designation screen (see FIG. 62).
  • the display processing unit performs a process of displaying an image obtained by rendering a virtual three-dimensional model of the real space as an image of observing the three-dimensional space from the viewpoint (). See FIG. 62).
  • the processing time required for the preview display of the observation image from the viewpoint can be shortened, and the work of creating the free viewpoint image can be executed quickly.
  • the display processing unit performs display processing of information for notifying the camera in which the change in the visual field range is detected among the plurality of cameras (see FIG. 72). ).
  • the display processing unit performs display processing of information for notifying the camera in which the change in the visual field range is detected among the plurality of cameras (see FIG. 72). ).
  • the display processing unit performs display processing of information for notifying the camera in which the change in the visual field range is detected among the plurality of cameras (see FIG. 72).
  • a plurality of camera works are used as a camera work designation screen in which the information processing apparatus accepts a camera work information designation operation which is information indicating at least the movement locus of the viewpoint in the free viewpoint image.
  • a camera work information designation operation which is information indicating at least the movement locus of the viewpoint in the free viewpoint image.
  • This is an information processing method that performs screen display processing that filters and shows camera work information according to user input information. According to such a second information processing method, the same operation and effect as the above-mentioned second information processing apparatus can be obtained.
  • a program can be considered in which the processing by the display processing unit 34a described with reference to FIGS. 52 and 53 is executed by, for example, a CPU, a DSP (Digital Signal Processor), or a device including these. .. That is, the first program of the embodiment is a program that can be read by a computer device, and at least as a camera work information creation operation screen which is information indicating at least the movement locus of the viewpoint in the free viewpoint image.
  • a screen that includes a designated operation reception area that accepts operation input that specifies some information, and a camerawork display area that visualizes and displays the movement trajectory of the viewpoint based on the camerawork information that reflects the specified content of the operation input.
  • It is a program that realizes a function to perform display processing on a computer device. With such a program, the above-mentioned display processing unit 34a can be realized in the device as the information processing device 70.
  • a program can be considered in which the processing by the display processing unit 32a described with reference to FIGS. 73 and 74 is executed by, for example, a CPU, a DSP, or a device including these.
  • the second program of the embodiment is a program that can be read by a computer device, and serves as a camera work designation screen that accepts a camera work information designation operation that is information indicating at least the movement trajectory of the viewpoint in the free viewpoint image.
  • This is a program that causes a computer device to execute a function of filtering and displaying camera work information according to user input information among a plurality of camera work information.
  • the above-mentioned display processing unit 32a can be realized in the device as the information processing device 70.
  • These programs can be recorded in advance in an HDD as a recording medium built in a device such as a computer device, a ROM in a microcomputer having a CPU, or the like.
  • a recording medium built in a device such as a computer device, a ROM in a microcomputer having a CPU, or the like.
  • flexible discs CD-ROMs (Compact Disc Read Only Memory), MO (Magneto Optical) discs, DVDs (Digital Versatile Discs), Blu-ray discs (Blu-ray Discs (registered trademarks)), magnetic discs, semiconductor memories, It can be temporarily or permanently stored (recorded) on a removable recording medium such as a memory card.
  • a removable recording medium can be provided as so-called package software.
  • it can also be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.
  • LAN Local Area Network
  • the display processing unit 34a and the display processing unit 32a of the embodiment it is suitable for a wide range of provision of the display processing unit 34a and the display processing unit 32a of the embodiment.
  • a program for example, by downloading a program to a personal computer, a portable information processing device, a mobile phone, a game device, a video device, a PDA (Personal Digital Assistant), or the like, the personal computer or the like can be displayed by the display processing unit 34a or display processing of the present disclosure. It can function as a device that realizes the processing as the unit 32a.
  • the present technology can also adopt the following configurations.
  • (1) As an operation screen for creating camera work information, which is information indicating at least the movement trajectory of the viewpoint in the free viewpoint image, a designated operation reception area for accepting an operation input for specifying at least a part of the camera work information, and a designated content specified by the operation input.
  • An information processing device including a display processing unit that performs screen display processing including a camera work display area that visualizes and displays the movement locus of the viewpoint based on camera work information that reflects the above.
  • (2) The information processing device according to (1) above, wherein the designated operation receiving area is capable of receiving designated operations of the start point and the end point of the movement locus.
  • the designated operation receiving area is capable of receiving a designated operation of a waypoint of the viewpoint.
  • the designated operation receiving area can receive a designated operation at a timing when the viewpoint passes through the waypoint.
  • the designated operation receiving area can receive a designated operation of the shape type of the movement locus.
  • the designated operation receiving area is capable of receiving a designated operation of a moving speed of the viewpoint.
  • the information processing device can receive a designated operation of a section for changing the moving speed in the moving locus.
  • the display processing unit The information processing device according to any one of (1) to (8) above, which performs a process of displaying information that visualizes a visual field range from the viewpoint in the camera work display area.
  • the display processing unit The information processing apparatus according to (9) above, wherein in the camera work display area, a process of displaying information representing a visual field range from the viewpoint in a graphic shape is performed.
  • the display processing unit The information processing apparatus according to any one of (1) to (10) above, which performs a process of displaying an image of a three-dimensional space observed from the viewpoint on the creation operation screen.
  • the designated operation receiving area can receive a designated operation of a target position that determines a line-of-sight direction from the viewpoint.
  • the designated operation receiving area is capable of receiving designated operations during a period facing the target.
  • the information processing apparatus according to (12) or (13), wherein the designated operation receiving area can accept a plurality of target position designation operations as target position designation operations.
  • Information processing device As an operation screen for creating camera work information, which is information indicating at least the movement trajectory of the viewpoint in the free viewpoint image, a designated operation reception area for accepting an operation input for specifying at least a part of the camera work information, and a designated content specified by the operation input.
  • An information processing method that performs screen display processing including a camera work display area that visualizes and displays the movement locus of the viewpoint based on camera work information that reflects the above.
  • a program that can be read by a computer device As an operation screen for creating camera work information, which is information indicating at least the movement trajectory of the viewpoint in the free viewpoint image, a designated operation reception area for accepting an operation input for specifying at least a part of the camera work information, and a designated content specified by the operation input.
  • a program that enables the computer device to perform a screen display process including a camera work display area that visualizes and displays the movement locus of the viewpoint based on the camera work information that reflects the above.

Abstract

An information processing device according to the present technology comprises a display processing unit which performs a display process for a screen including: a designated operation reception area which is a camera walk information creation operation screen, the camera walk information indicating a movement track of at least a viewpoint in a free viewpoint image, and in which an operation input for designating at least a portion of the camera walk information is received; and a camera walk display area which visualizes and displays the movement track of the viewpoint based on the camera walk information to which the designation content by the operation input is reflected.

Description

情報処理装置、情報処理方法、プログラムInformation processing equipment, information processing methods, programs
　本技術は、情報処理装置とその方法、及びプログラムに関し、特に撮像された被写体を三次元空間上の任意の視点から観察できる自由視点画像に係る処理の技術に関する。 This technology relates to an information processing device, its method, and a program, and particularly to a processing technology related to a free-viewpoint image capable of observing an imaged subject from an arbitrary viewpoint in a three-dimensional space.
　撮像された被写体を三次元空間上で表した三次元情報に基づき、三次元空間上の任意視点からの観察画像に相当する自由視点画像（自由視点映像、仮想視点画像（映像）などとも呼ばれる）を生成する技術が知られている。
　関連する従来技術については下記特許文献１を挙げることができる。特許文献１には視点の移動軌跡といえるカメラワークの生成に関する技術が開示されている。 A free viewpoint image (also called a free viewpoint image, a virtual viewpoint image (video), etc.) corresponding to an observation image from an arbitrary viewpoint in the three-dimensional space based on three-dimensional information representing the captured subject in the three-dimensional space. The technology to generate is known.
The following Patent Document 1 can be mentioned as a related prior art. Patent Document 1 discloses a technique relating to the generation of camera work, which can be said to be a movement locus of a viewpoint.
ＷＯ２０１８／０３０２０６号公報WO2018 / 030206
　ここで、自由視点画像は、例えばスポーツ中継等の放送コンテンツにおいて、リプレイ画像として用いられることもある。その場合、放送の現場では、例えばサッカーにおけるＡチームのシュートシーン、Ｂチームのシュートシーン等、自由視点画像の対象シーンとして想定される各シーンについてそれぞれ対応するカメラワークを事前に作成しておくということが考えられる。実際の放送時には、このように事前に作成しておいた複数のカメラワークのうちからシーンに対応したカメラワークを選択し、自由視点画像の生成を行う。 Here, the free viewpoint image may be used as a replay image in broadcast contents such as sports broadcasts. In that case, at the broadcasting site, camera work corresponding to each scene assumed as the target scene of the free viewpoint image, such as the shooting scene of team A and the shooting scene of team B in soccer, is created in advance. Can be considered. At the time of actual broadcasting, the camera work corresponding to the scene is selected from the plurality of camera works created in advance in this way, and the free viewpoint image is generated.
　このとき、カメラワークの作成作業は、放送スタッフとしてのユーザが例えばＰＣ（パーソナルコンピュータ）等のコンピュータ装置により提供される作成画面に対し操作入力を行うという形態を採ることが想定されるが、このような作成作業においては、ユーザが自身の操作入力によってカメラワークがどのように変化したかを確認できることが望ましい。
　この確認については、例えば、作成されたカメラワークの情報に従って生成した自由視点画像をユーザにプレビュー表示するという形態で実現することが考えられる。ユーザは、表示された自由視点画像に基づき、作成したカメラワークが意図通りのものであるか否かを判断し、意図通りのものでなければカメラワークの作成をやり直すことができる。 At this time, it is assumed that the camera work creation work takes a form in which a user as a broadcasting staff inputs an operation to a creation screen provided by a computer device such as a PC (personal computer). In such a creation work, it is desirable that the user can confirm how the camera work has changed due to his / her own operation input.
This confirmation can be realized, for example, in the form of previewing the free viewpoint image generated according to the created camera work information to the user. Based on the displayed free-viewpoint image, the user can determine whether or not the created camera work is as intended, and if it is not as intended, the user can recreate the camera work.
　しかしながら、自由視点画像を生成するには、三次元情報から二次元画像へのレンダリング処理が必要となるため、上記のように自由視点画像をプレビュー表示する手法ではカメラワークの確認に時間を要するものとなってしまい、結果、カメラワーク作成作業の効率化を図ることが困難となる。 However, in order to generate a free-viewpoint image, rendering processing from 3D information to a 2-dimensional image is required. Therefore, it takes time to confirm the camera work in the method of previewing the free-viewpoint image as described above. As a result, it becomes difficult to improve the efficiency of the camera work creation work.
　本技術は上記事情に鑑み為されたものであり、カメラワーク作成作業の効率化を図ることを目的とする。 This technology was made in view of the above circumstances, and aims to improve the efficiency of camera work creation work.
　本技術に係る情報処理装置は、自由視点画像における少なくとも視点の移動軌跡を示す情報であるカメラワーク情報の作成操作画面として、カメラワーク情報の少なくとも一部情報を指定する操作入力を受け付ける指定操作受付領域と、前記操作入力による指定内容を反映させたカメラワーク情報に基づいた前記視点の移動軌跡を視覚化して表示するカメラワーク表示領域とを含む画面の表示処理を行う表示処理部を備えるものである。
　これにより、ユーザは、カメラワークの作成操作画面において、視覚化された視点の移動軌跡を視認しながらカメラワークの作成操作を行うことが可能となる。 The information processing device according to the present technology accepts a designated operation reception that accepts an operation input for designating at least a part of the camera work information as a camera work information creation operation screen which is information indicating at least the movement trajectory of the viewpoint in the free viewpoint image. It is provided with a display processing unit that performs screen display processing including an area and a camera work display area that visualizes and displays the movement locus of the viewpoint based on camera work information that reflects the specified contents by the operation input. be.
As a result, the user can perform the camera work creation operation while visually recognizing the movement locus of the visualized viewpoint on the camera work creation operation screen.
　上記した本技術に係る情報処理装置においては、前記指定操作受付領域は、前記移動軌跡の始点と終点の指定操作を受け付け可能とされた構成とすることが可能である。
　これにより、視点の移動軌跡について、固定ではなく任意の始点及び終点を設定可能となる。 In the information processing apparatus according to the present technology described above, the designated operation receiving area can be configured to be capable of receiving designated operations of the start point and the end point of the movement locus.
As a result, it is possible to set an arbitrary start point and end point for the movement locus of the viewpoint instead of being fixed.
　上記した本技術に係る情報処理装置においては、前記指定操作受付領域は、前記視点の経由地点の指定操作を受け付け可能とされた構成とすることが可能である。
　これにより、視点の移動軌跡として、始点と終点の２点間を結ぶ直線状の軌跡ではなく、指定地点を経由する軌跡を設定することが可能となる。 In the information processing apparatus according to the present technology described above, the designated operation receiving area can be configured to be capable of receiving the designated operation of the waypoint of the viewpoint.
As a result, it is possible to set a locus that passes through a designated point instead of a linear locus that connects the two points of the start point and the end point as the movement locus of the viewpoint.
　上記した本技術に係る情報処理装置においては、前記指定操作受付領域は、前記経由地点を前記視点が経由するタイミングの指定操作を受け付け可能とされた構成とすることが可能である。
　これにより、視点の経由地点のみでなく、視点が経由地点を経由するタイミングを設定することが可能となる。 In the information processing apparatus according to the present technology described above, the designated operation receiving area can be configured to be capable of receiving a designated operation at a timing when the viewpoint passes through the waypoint.
This makes it possible to set not only the waypoint of the viewpoint but also the timing of the viewpoint passing through the waypoint.
　上記した本技術に係る情報処理装置においては、前記指定操作受付領域は、前記移動軌跡の形状の種類の指定操作を受け付け可能とされた構成とすることが可能である。
　これにより、視点の移動軌跡の形状種類を固定ではなく可変とすることが可能となる。 In the information processing apparatus according to the present technology described above, the designated operation receiving area can be configured to be capable of receiving designated operations of the shape type of the movement locus.
This makes it possible to change the shape type of the movement locus of the viewpoint instead of fixing it.
　上記した本技術に係る情報処理装置においては、前記指定操作受付領域は、前記視点の移動速度の指定操作を受け付け可能とされた構成とすることが可能である。
　これにより、視点の移動速度を固定ではなく可変とすることが可能となる。 In the information processing apparatus according to the present technology described above, the designated operation receiving area can be configured to be capable of receiving the designated operation of the moving speed of the viewpoint.
This makes it possible to change the moving speed of the viewpoint instead of fixing it.
　上記した本技術に係る情報処理装置においては、前記指定操作受付領域は、前記移動軌跡中において前記移動速度を変化させる区間の指定操作を受け付け可能とされた構成とすることが可能である。
　これにより、移動軌跡中における視点の移動速度を動的に変化させることが可能となる。 In the information processing apparatus according to the present technology described above, the designated operation receiving area can be configured to be capable of receiving designated operations of a section in which the moving speed is changed in the moving locus.
This makes it possible to dynamically change the moving speed of the viewpoint in the moving locus.
　上記した本技術に係る情報処理装置においては、前記指定操作受付領域は、前記視点の移動開始時点から移動終了時点までの期間を示すタイムラインに対する前記操作入力を受け付け可能とされた構成とすることが可能である。
　タイムライン上での入力操作受け付けとすることで、例えばカメラのアイコン等をタイムライン上にドラッグ&ドロップすることで経由地点の指定とその経由タイミングの指定とを同時に行うことが可能となったり、タイムライン上でのドラッグ操作による範囲指定により移動軌跡の曲線補間を行うべき区間等、所定効果を与えるべき区間の指定が可能となったりする等、カメラワークに係る各種情報の指定操作の容易化を図ることが可能となる。 In the information processing device according to the present technology described above, the designated operation receiving area has a configuration capable of receiving the operation input to the timeline indicating the period from the movement start time to the movement end time of the viewpoint. Is possible.
By accepting input operations on the timeline, for example, by dragging and dropping the camera icon etc. onto the timeline, it is possible to specify the waypoint and the waypoint at the same time. Facilitating the operation of specifying various information related to camera work, such as the section where the curve interpolation of the movement trajectory should be performed and the section where a predetermined effect should be given can be specified by specifying the range by dragging on the timeline. It becomes possible to plan.
　上記した本技術に係る情報処理装置においては、前記表示処理部は、前記カメラワーク表示領域において、前記視点からの視野範囲を視覚化した情報を表示する処理を行う構成とすることが可能である。
　視野範囲が視覚的に示されることで、ユーザによるカメラワークの把握の容易化が図られる。 In the information processing apparatus according to the present technology described above, the display processing unit can be configured to perform processing for displaying information that visualizes the visual field range from the viewpoint in the camera work display area. ..
By visually showing the field of view, it is easy for the user to grasp the camera work.
　上記した本技術に係る情報処理装置においては、前記表示処理部は、前記カメラワーク表示領域において、前記視点からの視野範囲を図形により表した情報を表示する処理を行う構成とすることが可能である。
　視野範囲が図形化して示されることで、ユーザがカメラワークの把握を容易に行うことが可能となる。 In the information processing device according to the present technology described above, the display processing unit can be configured to perform processing of displaying information representing a visual field range from the viewpoint in a graphic shape in the camera work display area. be.
By displaying the field of view in a graphic form, the user can easily grasp the camera work.
　上記した本技術に係る情報処理装置においては、前記表示処理部は、前記視点から三次元空間を観察した画像を前記作成操作画面において表示する処理を行う構成とすることが可能である。
　これにより、カメラワーク情報に基づき生成される自由視点画像と同様の画像をユーザにプレビュー表示することが可能となり、カメラワークの把握の容易化を図ることが可能となる。 In the information processing apparatus according to the present technology described above, the display processing unit can be configured to perform a process of displaying an image of observing a three-dimensional space from the viewpoint on the creation operation screen.
As a result, it is possible to preview and display an image similar to the free viewpoint image generated based on the camera work information to the user, and it is possible to facilitate the grasp of the camera work.
　上記した本技術に係る情報処理装置においては、前記指定操作受付領域は、前記視点からの視線方向を定めるターゲットの位置の指定操作を受け付け可能とされた構成とすることが可能である。
　これにより、自由視点画像として、ターゲットに追従する画像を生成可能となる。ターゲットに追従する画像とは、画枠内の所定位置（例えば中央位置等）にターゲットが位置され続けるようにした画像を意味する。 In the information processing apparatus according to the present technology described above, the designated operation receiving area can be configured to be capable of receiving a designated operation of a target position that determines a line-of-sight direction from the viewpoint.
This makes it possible to generate an image that follows the target as a free viewpoint image. The image that follows the target means an image in which the target is continuously positioned at a predetermined position (for example, the center position) in the image frame.
　上記した本技術に係る情報処理装置においては、前記指定操作受付領域は、前記ターゲットを向く期間の指定操作を受け付け可能とされた構成とすることが可能である。
　ターゲットを向く期間とは、自由視点画像の画枠内の所定位置にターゲットを位置させ続ける期間を意味する。上記のようにターゲットを向く期間の指定操作が可能とされることで、自由視点画像として、視点移動期間内における或る期間ではターゲット位置に追従し他の期間ではターゲット位置に追従しない画像を生成する等といったことが可能となり、ターゲット位置に追従する期間についての設定自由向上が図られる。 In the information processing apparatus according to the present technology described above, the designated operation receiving area can be configured to be capable of receiving designated operations during the period facing the target.
The period of facing the target means a period of keeping the target positioned at a predetermined position in the image frame of the free viewpoint image. By making it possible to specify the period for facing the target as described above, an image that follows the target position during a certain period within the viewpoint movement period and does not follow the target position during another period is generated as a free viewpoint image. It is possible to improve the freedom of setting for the period of following the target position.
　上記した本技術に係る情報処理装置においては、前記指定操作受付領域は、前記ターゲットの位置の指定操作として、複数の前記ターゲットの位置の指定操作を受け付け可能とされた構成とすることが可能である。
　これにより、視点移動期間内における或る期間ではターゲットＡに追従し、他の期間ではターゲットＢに追従する自由視点画像を生成する等といったことが可能となり、追従対象とするターゲットについての設定自由向上が図られる。 In the information processing apparatus according to the present technology described above, the designated operation receiving area can be configured to be capable of accepting a plurality of target position designation operations as the target position designation operation. be.
As a result, it is possible to generate a free viewpoint image that follows the target A in a certain period within the viewpoint movement period and follows the target B in another period, and the setting freedom of the target to be followed is improved. Is planned.
　本技術に係る情報処理方法は、情報処理装置が、自由視点画像における少なくとも視点の移動軌跡を示す情報であるカメラワーク情報の作成操作画面として、カメラワーク情報の少なくとも一部情報を指定する操作入力を受け付ける指定操作受付領域と、前記操作入力による指定内容を反映させたカメラワーク情報に基づいた前記視点の移動軌跡を視覚化して表示するカメラワーク表示領域とを含む画面の表示処理を行う情報処理方法である。
　このような本技術に係る情報処理方法によっても、上記した本技術に係る情報処理装置と同様の作用が得られる。 In the information processing method according to the present technology, the information processing apparatus specifies at least a part of the camerawork information as an operation screen for creating camerawork information which is information indicating at least the movement trajectory of the viewpoint in the free viewpoint image. Information processing that performs screen display processing including a designated operation reception area that receives The method.
Even with such an information processing method according to the present technology, the same operation as that of the above-mentioned information processing apparatus according to the present technology can be obtained.
　本技術に係るプログラムは、コンピュータ装置が読み取り可能なプログラムであって、自由視点画像における少なくとも視点の移動軌跡を示す情報であるカメラワーク情報の作成操作画面として、カメラワーク情報の少なくとも一部情報を指定する操作入力を受け付ける指定操作受付領域と、前記操作入力による指定内容を反映させたカメラワーク情報に基づいた前記視点の移動軌跡を視覚化して表示するカメラワーク表示領域とを含む画面の表示処理を行う機能を前記コンピュータ装置に実現させるプログラムである。
　このようなプログラムにより、上記した本技術に係る情報処理装置が実現される。 The program according to the present technology is a program that can be read by a computer device, and at least a part of the camera work information is displayed as an operation screen for creating camera work information which is information indicating at least the movement trajectory of the viewpoint in the free viewpoint image. Screen display processing including a designated operation receiving area that accepts a designated operation input and a camerawork display area that visualizes and displays the movement locus of the viewpoint based on the camerawork information that reflects the specified contents of the operation input. It is a program that makes the computer device realize the function of performing the above.
By such a program, the information processing apparatus according to the present technology described above is realized.
本技術の実施形態のシステム構成のブロック図である。It is a block diagram of the system configuration of embodiment of this technology. 実施形態の自由視点画像生成のためのカメラ配置例の説明図である。It is explanatory drawing of the camera arrangement example for the free viewpoint image generation of embodiment. 実施形態の情報処理装置のハードウエア構成のブロック図である。It is a block diagram of the hardware structure of the information processing apparatus of embodiment. 実施形態の画像作成コントローラの機能の説明図である。It is explanatory drawing of the function of the image creation controller of embodiment. 実施形態の自由視点画像サーバの機能の説明図である。It is explanatory drawing of the function of the free viewpoint image server of embodiment. 実施形態の自由視点画像における視点の説明図である。It is explanatory drawing of the viewpoint in the free viewpoint image of embodiment. 実施形態におけるカメラワーク指定画面の概要の説明図である。It is explanatory drawing of the outline of the camera work designation screen in an embodiment. 実施形態における作成操作画面の概要の説明図である。It is explanatory drawing of the outline of the creation operation screen in an embodiment. 実施形態の出力クリップの説明図である。It is explanatory drawing of the output clip of an embodiment. 実施形態の静止画ＦＶクリップを含む出力クリップの説明図である。It is explanatory drawing of the output clip including the still image FV clip of an embodiment. 実施形態の動画ＦＶクリップを含む出力クリップの説明図である。It is explanatory drawing of the output clip including the moving image FV clip of embodiment. 実施形態の出力クリップの画像例の説明図である。It is explanatory drawing of the image example of the output clip of an embodiment. 実施形態のクリップ作成の作業手順の説明図である。It is explanatory drawing of the work procedure of the clip making of an embodiment. 実施形態のカメラ変動検出の作業手順の説明図である。It is explanatory drawing of the work procedure of the camera variation detection of embodiment. 実施形態の作成操作画面の初期画面を例示した図である。It is a figure which illustrated the initial screen of the creation operation screen of an embodiment. カメラのプリセットリストを取得するための操作の例を説明するための図である。It is a figure for demonstrating the example of the operation for acquiring the preset list of a camera. 背景３Ｄモデルの変更についての説明図である。It is explanatory drawing about the change of the background 3D model. 同じく、背景３Ｄモデルの変更についての説明図である。Similarly, it is explanatory drawing about the change of the background 3D model. カメラワークのエントリの追加についての説明図である。It is explanatory drawing about addition of the entry of the camera work. 指定されたカメラの視野範囲や観察画像を表示する例の説明図である。It is explanatory drawing of the example which displays the field of view range of a designated camera and the observation image. 同じく、指定されたカメラの視野範囲や観察画像を表示する例の説明図である。Similarly, it is explanatory drawing of the example which displays the field of view range of a designated camera and the observation image. Ｉｎカメラの指定手法の例についての説明図である。It is explanatory drawing about the example of the designation method of an In camera. Ｉｎカメラが指定された場合の画面表示例を示した図である。It is a figure which showed the screen display example when an In camera is specified. Ｏｕｔカメラの指定手法の例についての説明図である。It is explanatory drawing of the example of the designation method of an Out camera. Ｏｕｔカメラが指定された場合の画面表示例を示した図である。It is a figure which showed the screen display example when the Out camera is specified. タイムライン操作部におけるシークバーの操作に応じた画面表示変化についての説明図である。It is explanatory drawing about the screen display change corresponding to the operation of the seek bar in the timeline operation part. 視点の経由地点の指定手法の例についての説明図である。It is explanatory drawing of the example of the method of designating a waypoint of a viewpoint. 同じく、視点の経由地点の指定手法の例についての説明図である。Similarly, it is an explanatory diagram of an example of a method of designating a waypoint of a viewpoint. 視点の経由地点の指定が完了した場合の画面表示例の説明図である。It is explanatory drawing of the screen display example when the designation of the waypoint of a viewpoint is completed. 複数の経由地点が指定された場合の画面表示例の説明図である。It is explanatory drawing of the screen display example when a plurality of waypoints are specified. 視点の移動軌跡の形状種類の指定手法例を説明するための図である。It is a figure for demonstrating an example of the specification method of the shape type of the movement locus of a viewpoint. 同じく、視点の移動軌跡の形状種類の指定手法例を説明するための図である。Similarly, it is a figure for demonstrating an example of the method of specifying the shape type of the movement locus of a viewpoint. 視点の移動軌跡の形状種類が指定された場合の画面表示例の説明図である。It is explanatory drawing of the screen display example when the shape type of the movement locus of a viewpoint is specified. 視点の移動速度の指定手法例について説明するための図である。It is a figure for demonstrating an example of the method of specifying the moving speed of a viewpoint. 視点の移動速度が指定された場合の画面表示例の説明図である。It is explanatory drawing of the screen display example when the moving speed of a viewpoint is specified. 実施形態におけるターゲットの意義についての説明図である。It is explanatory drawing about the significance of the target in an embodiment. 同じく、実施形態におけるターゲットの意義についての説明図である。Similarly, it is explanatory drawing about the significance of the target in an embodiment. 実施形態におけるターゲットの移動指定についての説明図である。It is explanatory drawing about the movement designation of the target in an embodiment. 同じく、実施形態におけるターゲットの移動指定についての説明図である。Similarly, it is explanatory drawing about the movement designation of the target in embodiment. 同じく、実施形態におけるターゲットの移動指定についての説明図である。Similarly, it is explanatory drawing about the movement designation of the target in embodiment. 同じく、実施形態におけるターゲットの移動指定についての説明図である。Similarly, it is explanatory drawing about the movement designation of the target in embodiment. 同じく、実施形態におけるターゲットの移動指定についての説明図である。Similarly, it is explanatory drawing about the movement designation of the target in embodiment. ターゲットを向く期間の指定操作例の説明図である。It is explanatory drawing of the designation operation example of the period facing a target. 同じく、ターゲットを向く期間の指定操作例の説明図である。Similarly, it is explanatory drawing of the designation operation example of the period which faces a target. 同じく、ターゲットを向く期間の指定操作例の説明図である。Similarly, it is explanatory drawing of the designation operation example of the period which faces a target. ターゲットの移動についての指定が行われた場合のプレビュー画像の表示例を示した図である。It is a figure which showed the display example of the preview image when the movement of a target is specified. 同じく、ターゲットの移動についての指定が行われた場合のプレビュー画像の表示例を示した図である。Similarly, it is a figure which showed the display example of the preview image when the movement of a target is specified. 同じく、ターゲットの移動についての指定が行われた場合のプレビュー画像の表示例を示した図である。Similarly, it is a figure which showed the display example of the preview image when the movement of a target is specified. 同じく、ターゲットの移動についての指定が行われた場合のプレビュー画像の表示例を示した図である。Similarly, it is a figure which showed the display example of the preview image when the movement of a target is specified. ターゲットを向く期間についての別の設定例を示した図である。It is a figure which showed another setting example about the period which faces a target. 複数追加したターゲットについて、ターゲットを向く期間を個別に指定する例の説明図である。It is explanatory drawing of the example which specifies the period which faces the target individually about the added target. Ｉｎカメラ、Ｏｕｔカメラの指定に応じた視点移動軌跡の生成・表示に係る処理のフローチャートである。It is a flowchart of the process related to the generation / display of the viewpoint movement locus according to the designation of the In camera and the Out camera. 経由地点の指定に応じた視点移動軌跡の生成・表示に係る処理を示したフローチャートである。It is a flowchart which showed the process to generate and display the viewpoint movement locus according to the designation of a waypoint. 実施形態のカメラワーク指定画面の初期画面を例示した図である。It is a figure which illustrated the initial screen of the camera work designation screen of an embodiment. 自由視点画像の生成に用いる画像のインポート手法の例の説明図である。It is explanatory drawing of the example of the image import method used for generating a free viewpoint image. 同じく、自由視点画像の生成に用いる画像のインポート手法の例の説明図である。Similarly, it is explanatory drawing of the example of the image import method used for generating a free viewpoint image. 画像インポート後の画面表示例を示した図である。It is a figure which showed the screen display example after image import. Ｘ軸視点の画像の表示例を示した図である。It is a figure which showed the display example of the image of the X-axis viewpoint. Ｙ軸視点の画像の表示例を示した図である。It is a figure which showed the display example of the image of the Y-axis viewpoint. Ｚ軸視点の画像の表示例を示した図である。It is a figure which showed the display example of the image of the Z-axis viewpoint. Ｐｅ視点の画像の表示例を示した図である。It is a figure which showed the display example of the image of a Pe viewpoint. Ｃａ視点の画像の表示例を示した図である。It is a figure which showed the display example of the image of the Ca viewpoint. カメラワークのフィルタリング表示のための操作手順例を説明するための図である。It is a figure for demonstrating the operation procedure example for the filtering display of a camera work. カメラワークのフィルタリング表示の例を示した図である。It is a figure which showed the example of the filtering display of a camera work. 実施形態におけるフィルタリング操作部のリセットボタンについての説明図である。It is explanatory drawing of the reset button of the filtering operation part in an embodiment. フィルタリング操作部の変形例の説明図である。It is explanatory drawing of the modification of the filtering operation part. フィルタリング操作部の他の変形例の説明図である。It is explanatory drawing of another modification of a filtering operation part. 視点の移動速度の視覚化情報の表示例を示した図である。It is a figure which showed the display example of the visualization information of the moving speed of a viewpoint. 視点の移動速度の視覚化情報の他の表示例を示した図である。It is a figure which showed the other display example of the visualization information of the moving speed of a viewpoint. 実施形態のカメラワーク指定画面におけるターゲット位置の編集についての説明図である。It is explanatory drawing about the editing of the target position in the camera work designation screen of an embodiment. 同じく、実施形態のカメラワーク指定画面におけるターゲット位置の編集についての説明図である。Similarly, it is explanatory drawing about the editing of the target position in the camera work designation screen of embodiment. 変動が検出されたカメラの通知情報の表示例を示した図である。It is a figure which showed the display example of the notification information of the camera which detected the fluctuation. 画面上に表示したタグ情報に基づくカメラワークのフィルタリングに係る処理を例示したフローチャートである。It is a flowchart which exemplifies the process related to the filtering of the camera work based on the tag information displayed on the screen. 入力キーワードに応じたカメラワークのフィルタリグに係る処理を例示したフローチャートである。It is a flowchart which exemplifies the process which concerns on the filter rig of the camera work corresponding to the input keyword. キャリブレーションを要するカメラの通知に係る処理を例示したフローチャートである。It is a flowchart which exemplifies the process which concerns on the notification of the camera which requires calibration.
　以下、実施の形態を次の順序で説明する。
<１．システム構成>
<２．画像作成コントローラ及び自由視点画像サーバの構成>
<３．ＧＵＩの概要>
<４．自由視点画像を含むクリップ>
<５．クリップ作成処理>
<６．カメラ変動検出>
<７．カメラワーク作成のためのＧＵＩ>
<８．自由視点画像作成のためのＧＵＩ>
<９．変形例>
<１０．実施形態のまとめ>
<１１．本技術>

Hereinafter, embodiments will be described in the following order.
<1. System configuration>
<2. Configuration of image creation controller and free-viewpoint image server>
<3. GUI Overview>
<4. Clips containing free-viewpoint images>
<5. Clip creation process>
<6. Camera fluctuation detection>
<7. GUI for creating camera work>
<8. GUI for creating free-viewpoint images>
<9. Modification example>
<10. Summary of embodiments>
<11. This technology>
<１．システム構成>

　図１に、本技術に係る実施の形態の画像処理システムの構成例を示す。
　画像処理システムは、画像作成コントローラ１、自由視点画像サーバ２、ビデオサーバ３、複数（例えば４台）のビデオサーバ４Ａ，４Ｂ，４Ｃ，４Ｄ、ＮＡＳ（Network Attached Storage）５、スイッチャー６、画像変換部７、ユーティリティサーバ８、複数（例えば１６台）の撮像装置１０を有する。
　なお以降、「カメラ」という用語は撮像装置１０を指す。例えば「カメラ配置」とは複数の撮像装置１０の配置を意味する。
　また、ビデオサーバ４Ａ，４Ｂ，４Ｃ，４Ｄを特に区別せずに総称するときは「ビデオサーバ４」と表記する。
　この画像処理システムでは、複数の撮像装置１０から取得される撮像画像（例えば画像データＶ１からＶ１６）に基づき、三次元空間上の任意視点からの観察画像に相当する自由視点画像を生成し、自由視点画像を含む出力クリップを作成することができる。 <1. System configuration>

FIG. 1 shows a configuration example of an image processing system according to an embodiment of the present technology.
The image processing system includes an image creation controller 1, a free viewpoint image server 2, a video server 3, a plurality of (for example, 4) video servers 4A, 4B, 4C, 4D, NAS (Network Attached Storage) 5, a switcher 6, and image conversion. It has a unit 7, a utility server 8, and a plurality of (for example, 16) imaging devices 10.
Hereinafter, the term "camera" refers to the imaging device 10. For example, "camera arrangement" means arrangement of a plurality of imaging devices 10.
Further, when the video servers 4A, 4B, 4C, and 4D are generically referred to without any particular distinction, they are referred to as "video server 4".
In this image processing system, based on the captured images (for example, image data V1 to V16) acquired from a plurality of imaging devices 10, a free viewpoint image corresponding to an observation image from an arbitrary viewpoint in a three-dimensional space is generated and freely selected. You can create an output clip that includes a viewpoint image.
　図１においては、各部の接続状態を実線、破線、二重線で示している。
　実線は、カメラやスイッチャーなどの放送機器間を接続するインタフェース規格であるＳＤＩ（Serial Digital Interface）の接続を示し、例えば４Ｋ対応としている。各機器間はＳＤＩ配線により主に画像データの送受信が行われる。 In FIG. 1, the connection state of each part is shown by a solid line, a broken line, and a double line.
The solid line indicates the connection of SDI (Serial Digital Interface), which is an interface standard for connecting broadcasting devices such as cameras and switchers, and is compatible with 4K, for example. Image data is mainly transmitted and received between each device by SDI wiring.
　二重線は、例えば１０ギガビット・イーサネットなどの、コンピュータネットワークを構築する通信規格の接続を示している。画像作成コントローラ１、自由視点画像サーバ２、ビデオサーバ３、４Ａ，４Ｂ，４Ｃ，４Ｄ、ＮＡＳ５、ユーティリティサーバ８はコンピュータネットワークで接続されることで、互いに画像データや各種制御信号の送受信が可能とされる。 The double line indicates the connection of the communication standard for constructing the computer network, for example, 10 Gigabit Ethernet. By connecting the image creation controller 1, the free viewpoint image server 2, the video server 3, 4A, 4B, 4C, 4D, NAS5, and the utility server 8 via a computer network, it is possible to send and receive image data and various control signals to each other. Will be done.
　ビデオサーバ３、４間の破線は、サーバ間ファイル共有機能を搭載したビデオサーバ３、４を例えば１０Ｇネットワークで接続した状態を示している。これによりビデオサーバ３、及びビデオサーバ４Ａ，４Ｂ，４Ｃ，４Ｄの間では、各ビデオサーバが他のビデオサーバ内の素材のプレビューや送出が可能となる。即ち複数のビデオサーバを使用したシステムが構築され、効率的なハイライト編集・送出を実現できるようにされている。 The broken line between the video servers 3 and 4 indicates a state in which the video servers 3 and 4 equipped with the inter-server file sharing function are connected by, for example, a 10G network. As a result, between the video server 3 and the video servers 4A, 4B, 4C, and 4D, each video server can preview and send the material in the other video server. That is, a system using a plurality of video servers has been constructed so that efficient highlight editing and transmission can be realized.
　各撮像装置１０は、例えばＣＣＤ（Charge Coupled Devices）センサやＣＭＯＳ（Complementary Metal-Oxide-Semiconductor）センサ等による撮像素子を有したデジタルカメラ装置として構成され、デジタルデータとしての撮像画像（画像データＶ１からＶ１６）を得る。本例では、各撮像装置１０は動画としての撮像画像を得る。 Each image pickup device 10 is configured as a digital camera device having an image pickup element such as a CCD (Charge Coupled Devices) sensor or a CMOS (Complementary Metal-Oxide-Semiconductor) sensor, and is an image captured as digital data (from image data V1). V16) is obtained. In this example, each image pickup device 10 obtains a captured image as a moving image.
　各撮像装置１０は、本例ではバスケットボールやサッカー等の競技が行われている様子を撮像するものとされ、それぞれが競技の開催される競技会場における所定位置において所定の向きに配置されている。本例では、撮像装置１０の数は１６台としているが、自由視点画像の生成を可能とする上では撮像装置１０の数は少なくとも２以上あればよい。撮像装置１０の台数を多くし、対象とする被写体をより多くの角度から撮像することで、被写体の三次元復元の精度向上が図られ、仮想視点画像の画質向上を図ることができる。 In this example, each image pickup device 10 is supposed to capture a state in which a game such as basketball or soccer is being performed, and each of them is arranged in a predetermined direction at a predetermined position in a competition venue where the competition is held. In this example, the number of image pickup devices 10 is 16, but the number of image pickup devices 10 may be at least 2 or more in order to enable the generation of a free viewpoint image. By increasing the number of image pickup devices 10 and imaging the target subject from a larger angle, the accuracy of three-dimensional restoration of the subject can be improved, and the image quality of the virtual viewpoint image can be improved.
　図２に、バスケットボールのコートの周囲における撮像装置１０の配置例を示している。○が撮像装置１０であるとする。例えば図面で左側のゴール近傍を重点的に撮りたい場合のカメラ配置例である。もちろんカメラ配置や数は一例であり、撮影や放送の内容、目的に応じて設定されるべきものである。 FIG. 2 shows an example of arrangement of the image pickup device 10 around the basketball court. It is assumed that ◯ is the imaging device 10. For example, this is an example of camera arrangement when it is desired to focus on the vicinity of the goal on the left side in the drawing. Of course, the camera arrangement and number are examples, and should be set according to the content and purpose of shooting and broadcasting.
　画像作成コントローラ１は、情報処理装置により構成される。この画像作成コントローラ１は、例えば専用のワークステーションや、汎用のパーソナルコンピュータ、モバイル端末装置等を利用して実現することができる。
　画像作成コントローラ１は、ビデオサーバ３、４の制御／動作管理や、クリップ作成のための処理を行う。
　一例として、画像作成コントローラ１はオペレータＯＰ１が操作可能な装置とする。オペレータＯＰ１は、例えばクリップ内容の選択や作成の指示等を行う。 The image creation controller 1 is composed of an information processing device. The image creation controller 1 can be realized by using, for example, a dedicated workstation, a general-purpose personal computer, a mobile terminal device, or the like.
The image creation controller 1 performs control / operation management of the video servers 3 and 4 and processing for creating a clip.
As an example, the image creation controller 1 is a device that can be operated by the operator OP1. The operator OP1 gives instructions such as selection of clip contents and creation.
　自由視点画像サーバ２は、画像作成コントローラ１の指示等に応じて、実際に自由視点画像（後述するＦＶ（Free View）クリップ）を作成する処理を行う情報処理装置として構成される。この自由視点画像サーバ２も、例えば専用のワークステーションや、汎用のパーソナルコンピュータ、モバイル端末装置等を利用して実現することができる。
　一例として、自由視点画像サーバ２はオペレータＯＰ２が操作可能な装置とする。オペレータＯＰ２は、例えば自由視点画像としてのＦＶクリップの作成に係る作業を行う。具体的に、オペレータＯＰ２は、自由視点画像の生成のためのカメラワークの指定操作（選択操作）などを行う。また、本例においてオペレータＯＰ２は、カメラワークの作成作業も行う。 The free-viewpoint image server 2 is configured as an information processing device that actually creates a free-viewpoint image (FV (Free View) clip, which will be described later) in response to an instruction from the image creation controller 1. This free-viewpoint image server 2 can also be realized by using, for example, a dedicated workstation, a general-purpose personal computer, a mobile terminal device, or the like.
As an example, the free viewpoint image server 2 is a device that can be operated by the operator OP2. The operator OP2 performs work related to, for example, creating an FV clip as a free viewpoint image. Specifically, the operator OP2 performs a camera work designation operation (selection operation) for generating a free viewpoint image. Further, in this example, the operator OP2 also performs the work of creating the camera work.
　画像作成コントローラ１と自由視点画像サーバ２の構成や処理について詳しくは後述する。また、オペレータＯＰ１，ＯＰ２が操作を行うものとするが、例えば画像作成コントローラ１と自由視点画像サーバ２が並べて配置され、一人のオペレータによって操作されるようにしてもよい。 The configuration and processing of the image creation controller 1 and the free viewpoint image server 2 will be described in detail later. Further, it is assumed that the operators OP1 and OP2 perform the operation. For example, the image creation controller 1 and the free viewpoint image server 2 may be arranged side by side and operated by one operator.
　ビデオサーバ３、４は、それぞれ画像記録装置とされ、例えばＳＳＤ（Solid State Drive）やＨＤＤ（Hard Disk Drive）等のデータ記録部と、該データ記録部についてデータの記録再生制御を行う制御部とを備える。 The video servers 3 and 4 are image recording devices, for example, a data recording unit such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive), and a control unit that controls data recording / playback of the data recording unit. To be equipped.
　ビデオサーバ４Ａ，４Ｂ，４Ｃ，４Ｄは、それぞれ例えば４系統の入力が可能とされて、それぞれ４台の撮像装置１０の撮像画像を同時に記録する。
　例えばビデオサーバ４Ａは、画像データＶ１，Ｖ２，Ｖ３，Ｖ４の記録を行う。ビデオサーバ４Ｂは、画像データＶ５，Ｖ６，Ｖ７，Ｖ８の記録を行う。ビデオサーバ４Ｃは、画像データＶ９，Ｖ１０，Ｖ１１，Ｖ１２の記録を行う。ビデオサーバ４Ｄは、画像データＶ１３，Ｖ１４，Ｖ１５，Ｖ１６の記録を行う。
　これにより、１６台の撮像装置１０の撮像画像が全て同時に記録される状態となる。
　ビデオサーバ４Ａ，４Ｂ，４Ｃ，４Ｄは、例えば放送対象のスポーツの試合中などに、常時録画を行うものとされる。 Each of the video servers 4A, 4B, 4C, and 4D is capable of inputting, for example, four systems, and simultaneously records images captured by four image pickup devices 10.
For example, the video server 4A records the image data V1, V2, V3, V4. The video server 4B records the image data V5, V6, V7, V8. The video server 4C records the image data V9, V10, V11, and V12. The video server 4D records the image data V13, V14, V15, V16.
As a result, all the captured images of the 16 imaging devices 10 are recorded at the same time.
The video servers 4A, 4B, 4C, and 4D are supposed to constantly record, for example, during a sports match to be broadcast.
　ビデオサーバ３は、例えば画像作成コントローラ１に直接接続され、例えば２系統の入力と２系統の出力が可能とされる。２系統の入力として画像データＶｐ，Ｖｑを示している。画像データＶｐ，Ｖｑとしては、いずれかの２台の撮像装置１０の撮像画像（画像データＶ１からＶ１６の内のいずれか２つ）を選択することが可能である。もちろん他の撮像装置の撮像画像であってもよい。 The video server 3 is directly connected to, for example, the image creation controller 1, and can, for example, input two systems and output two systems. Image data Vp and Vq are shown as two inputs. As the image data Vp and Vq, it is possible to select an image captured by any of the two imaging devices 10 (any two of the image data V1 to V16). Of course, it may be an image captured by another imaging device.
　画像データＶｐ，Ｖｑについては、モニタ画像として画像作成コントローラ１がディスプレイに表示させることができる。オペレータＯＰ１は、ビデオサーバ３に入力された画像データＶｐ，Ｖｑにより、例えば放送のために撮影・収録しているシーンの状況を確認することができる。
　また、ビデオサーバ３、４はファイル共有状態に接続されているため、画像作成コントローラ１は、ビデオサーバ４Ａ，４Ｂ，４Ｃ，４Ｄに記録している各撮像装置１０の撮像画像についてもモニタ表示させることができ、オペレータＯＰ１が逐次確認できるようにされる。 The image data Vp and Vq can be displayed on the display by the image creation controller 1 as monitor images. The operator OP1 can confirm the status of a scene shot / recorded for broadcasting, for example, by using the image data Vp and Vq input to the video server 3.
Further, since the video servers 3 and 4 are connected to the file sharing state, the image creation controller 1 also monitors and displays the captured images of the imaging devices 10 recorded on the video servers 4A, 4B, 4C, and 4D. This allows the operator OP1 to check sequentially.
　なお本例において、各撮像装置１０による撮像画像にはタイムコードが付され、ビデオサーバ３，４Ａ，４Ｂ，４Ｃ，４Ｄにおける処理においてフレーム同期をとることが可能とされている。 In this example, a time code is attached to the image captured by each imaging device 10, and it is possible to synchronize frames in the processing of the video servers 3, 4A, 4B, 4C, and 4D.
　ＮＡＳ５はネットワーク上に配置されたストレージデバイスであり、例えばＳＳＤやＨＤＤ等で構成される。本例の場合、ＮＡＳ５は、ビデオサーバ４Ａ，４Ｂ，４Ｃ，４Ｄに録画された画像データＶ１、Ｖ２・・・Ｖ１６について一部のフレームが自由視点画像の生成のために転送されてきたときに、自由視点画像サーバ２における処理のために記憶したり、作成された自由視点画像を記憶したりするデバイスとされる。 NAS5 is a storage device arranged on a network, and is composed of, for example, an SSD or an HDD. In the case of this example, NAS5 is when some frames of the image data V1, V2 ... V16 recorded on the video servers 4A, 4B, 4C, 4D are transferred for the generation of the free viewpoint image. , A device that stores a free-viewpoint image for processing in the free-viewpoint image server 2 or stores a created free-viewpoint image.
　スイッチャー６は、ビデオサーバ３を介して出力される画像を入力し、最終的に選択して放送する本線画像ＰＧＭｏｕｔを選択する機器である。例えば放送のディレクター等が必要な操作を行う。 The switcher 6 is a device that inputs an image output via the video server 3 and finally selects and broadcasts the main line image PGMout. For example, a broadcasting director or the like performs necessary operations.
　画像変換部７は、例えば撮像装置１０による画像データの解像度変換及び合成を行い、カメラ配置のモニタリング画像を生成してユーティリティサーバ８に供給する。例えば８Ｋ画像とされる１６系統の画像データ（Ｖ１からＶ１６）を、４Ｋ画像に解像度変換した上でタイル状に配置した４系統の画像とし、ユーティリティサーバ８に供給する。 The image conversion unit 7 performs resolution conversion and composition of image data by, for example, the image pickup device 10, generates a monitoring image of the camera arrangement, and supplies the monitoring image to the utility server 8. For example, 16 systems of image data (V1 to V16), which are 8K images, are converted into 4K images in resolution and then arranged in tiles to form 4 systems of images, which are supplied to the utility server 8.
　ユーティリティサーバ８は、各種の関連処理が可能なコンピュータ装置であるが、本例の場合、特にキャリブレーション用のカメラ移動の検出処理を行う装置としている。例えばユーティリティサーバ８は、画像変換部７からの画像データを監視してカメラ移動を検出する。カメラ移動とは、例えば図２のように配置された撮像装置１０のいずれかの配置位置の移動のことである。撮像装置１０の配置位置の情報は自由視点画像の生成に重要な要素であり、配置位置が変化したらパラメータ設定のやり直しが必要になる。そのためカメラ移動の監視が行われる。
The utility server 8 is a computer device capable of performing various related processes. In the case of this example, the utility server 8 is a device that detects camera movement for calibration. For example, the utility server 8 monitors the image data from the image conversion unit 7 and detects the movement of the camera. The camera movement is, for example, the movement of one of the arrangement positions of the image pickup apparatus 10 arranged as shown in FIG. The information on the arrangement position of the image pickup apparatus 10 is an important element for generating a free-viewpoint image, and if the arrangement position changes, it is necessary to redo the parameter setting. Therefore, the movement of the camera is monitored.
<２．画像作成コントローラ及び自由視点画像サーバの構成>

　以上の構成における画像作成コントローラ１、自由視点画像サーバ２、ビデオサーバ３、４、ユーティリティサーバ８は、例えば図３に示す構成を備えた情報処理装置７０として実現できる。 <2. Configuration of image creation controller and free-viewpoint image server>

The image creation controller 1, the free-viewpoint image server 2, the video servers 3, 4, and the utility server 8 in the above configuration can be realized as an information processing device 70 having the configuration shown in FIG. 3, for example.
　図３において、情報処理装置７０のＣＰＵ７１は、ＲＯＭ７２に記憶されているプログラム、または記憶部７９からＲＡＭ７３にロードされたプログラムに従って各種の処理を実行する。ＲＡＭ７３にはまた、ＣＰＵ７１が各種の処理を実行する上において必要なデータなども適宜記憶される。
　ＣＰＵ７１、ＲＯＭ７２、およびＲＡＭ７３は、バス７４を介して相互に接続されている。このバス７４にはまた、入出力インタフェース７５も接続されている。 In FIG. 3, the CPU 71 of the information processing apparatus 70 executes various processes according to a program stored in the ROM 72 or a program loaded from the storage unit 79 into the RAM 73. The RAM 73 also appropriately stores data and the like necessary for the CPU 71 to execute various processes.
The CPU 71, ROM 72, and RAM 73 are connected to each other via a bus 74. An input / output interface 75 is also connected to the bus 74.
　入出力インタフェース７５には、操作子や操作デバイスよりなる入力部７６が接続される。
　例えば入力部７６としては、キーボード、マウス、キー、ダイヤル、タッチパネル、タッチパッド、リモートコントローラ等の各種の操作子や操作デバイスが想定される。
　入力部７６によりユーザの操作が検知され、入力された操作に応じた信号はＣＰＵ７１によって解釈される。 An input unit 76 including an operator and an operation device is connected to the input / output interface 75.
For example, as the input unit 76, various controls and operation devices such as a keyboard, mouse, keys, dial, touch panel, touch pad, and remote controller are assumed.
The user's operation is detected by the input unit 76, and the signal corresponding to the input operation is interpreted by the CPU 71.
　また入出力インタフェース７５には、ＬＣＤ（Liquid Crystal Display）或いは有機ＥＬ（Electro-Luminescence）パネルなどよりなる表示部７７や、スピーカなどよりなる音声出力部７８が一体又は別体として接続される。
　表示部７７は各種表示を行う表示部であり、例えば情報処理装置７０の筐体に設けられるディスプレイデバイスや、情報処理装置７０に接続される別体のディスプレイデバイス等により構成される。
　表示部７７は、ＣＰＵ７１の指示に基づいて表示画面上に各種の画像処理のための画像や処理対象の動画等の表示を実行する。また表示部７７はＣＰＵ７１の指示に基づいて、各種操作メニュー、アイコン、メッセージ等、即ちＧＵＩ（Graphical User Interface）としての表示を行う。 Further, a display unit 77 made of an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel and an audio output unit 78 made of a speaker or the like are connected to the input / output interface 75 as one or a separate body.
The display unit 77 is a display unit that performs various displays, and is composed of, for example, a display device provided in the housing of the information processing device 70, a separate display device connected to the information processing device 70, and the like.
The display unit 77 executes the display of various images for image processing, moving images to be processed, and the like on the display screen based on the instruction of the CPU 71. Further, the display unit 77 displays various operation menus, icons, messages, etc., that is, as a GUI (Graphical User Interface) based on the instruction of the CPU 71.
　入出力インタフェース７５には、ハードディスクや固体メモリなどより構成される記憶部７９や、モデムなどより構成される通信部８０が接続される場合もある。
　通信部８０は、インターネット等の伝送路を介しての通信処理や、各種機器との有線／無線通信、バス通信などによる通信を行う。 A storage unit 79 composed of a hard disk, a solid-state memory, or the like, or a communication unit 80 composed of a modem or the like may be connected to the input / output interface 75.
The communication unit 80 performs communication processing via a transmission line such as the Internet, wire / wireless communication with various devices, bus communication, and the like.
　入出力インタフェース７５にはまた、必要に応じてドライブ８２が接続され、磁気ディスク、光ディスク、光磁気ディスク、或いは半導体メモリなどのリムーバブル記録媒体８１が適宜装着される。
　ドライブ８２により、リムーバブル記録媒体８１からは画像ファイルＭＦ等のデータファイルや、各種のコンピュータプログラムなどを読み出すことができる。読み出されたデータファイルは記憶部７９に記憶されたり、データファイルに含まれる画像や音声が表示部７７や音声出力部７８で出力されたりする。またリムーバブル記録媒体８１から読み出されたコンピュータプログラム等は必要に応じて記憶部７９にインストールされる。 A drive 82 is also connected to the input / output interface 75, if necessary, and a removable recording medium 81 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted.
The drive 82 can read data files such as an image file MF and various computer programs from the removable recording medium 81. The read data file is stored in the storage unit 79, and the image and sound included in the data file are output by the display unit 77 and the sound output unit 78. Further, the computer program or the like read from the removable recording medium 81 is installed in the storage unit 79 as needed.
　この情報処理装置７０では、ソフトウエアを、通信部８０によるネットワーク通信やリムーバブル記録媒体８１を介してインストールすることができる。或いは当該ソフトウエアは予めＲＯＭ７２や記憶部７９等に記憶されていてもよい。 In this information processing device 70, software can be installed via network communication by the communication unit 80 or a removable recording medium 81. Alternatively, the software may be stored in the ROM 72, the storage unit 79, or the like in advance.
　このような情報処理装置７０を用いて画像作成コントローラ１や自由視点画像サーバ２を実現する場合、例えばソフトウエアにより、図４，図５のような処理機能がＣＰＵ７１において実現されるようにする。 When the image creation controller 1 and the free-viewpoint image server 2 are realized by using such an information processing device 70, for example, the processing functions shown in FIGS. 4 and 5 are realized in the CPU 71 by software.
　図４は、画像作成コントローラ１となる情報処理装置７０のＣＰＵ７１において形成される機能として、区間特定処理部２１、対象画像送信制御部２２、出力画像生成部２３を示している。 FIG. 4 shows a section identification processing unit 21, a target image transmission control unit 22, and an output image generation unit 23 as functions formed in the CPU 71 of the information processing device 70 that serves as the image creation controller 1.
　区間特定処理部２１は、複数の撮像装置１０により同時に撮像された複数の撮像画像（画像データＶ１からＶ１６）について、自由視点画像の生成対象とする生成対象画像区間を特定する処理を行う。例えばオペレータＯＰ１が画像内でリプレイ再生させたいシーンを選択する操作を行うことに応じて、そのシーン、特には自由視点画像とするシーンの区間（生成対象画像区間）についてのタイムコードを特定したり、当該タイムコードを自由視点画像サーバ２に通知したりする処理を行う。 The section specifying processing unit 21 performs a process of specifying a generation target image section to be generated as a free viewpoint image for a plurality of captured images (image data V1 to V16) simultaneously captured by the plurality of imaging devices 10. For example, when the operator OP1 performs an operation of selecting a scene to be replayed in an image, the time code for that scene, particularly the section of the scene to be a free viewpoint image (image section to be generated) can be specified. , Performs processing such as notifying the free viewpoint image server 2 of the time code.
　ここで生成対象画像区間とは、実際に自由視点画像とするフレーム区間をという。動画内のある１フレームについて自由視点画像を生成する場合は、その１フレームが生成対象画像区間となる。この場合、自由視点画像のためのイン点／アウト点は同じタイムコードとなる。
  また動画内の複数フレームの区間について自由視点画像を生成する場合は、その複数フレームが生成対象画像区間となる。この場合、自由視点画像のためのイン点／アウト点は異なるタイムコードとなる。
　なお、クリップの構造については後述するが、生成対象画像区間のイン点／アウト点は、最終的に生成する出力クリップとしてのイン点／アウト点とは異なることが想定される。後述する前クリップや後クリップが結合されるためである。 Here, the image section to be generated is a frame section that is actually a free viewpoint image. When a free viewpoint image is generated for a certain frame in a moving image, that one frame is a generation target image section. In this case, the in / out points for the free viewpoint image have the same time code.
When a free viewpoint image is generated for a section of a plurality of frames in a moving image, the plurality of frames are the generation target image sections. In this case, the in / out points for the free viewpoint image have different time codes.
Although the structure of the clip will be described later, it is assumed that the in / out points of the image section to be generated are different from the in / out points of the output clip to be finally generated. This is because the front clip and the rear clip, which will be described later, are combined.
　対象画像送信制御部２２は、複数の撮像画像１０のそれぞれにおける生成対象画像区間の画像データ、即ち画像データＶ１からＶ１６についての１又は複数フレームを、自由視点画像サーバ２における自由視点画像の生成に用いる画像データとして送信させる制御を行う。具体的には生成対象画像区間としての画像データを、ビデオサーバ４Ａ，４Ｂ，４Ｃ，４ＤからＮＡＳ５に転送させる制御を行う。 The target image transmission control unit 22 uses the image data of the generation target image section in each of the plurality of captured images 10, that is, one or more frames for the image data V1 to V16, to generate the free viewpoint image in the free viewpoint image server 2. Controls transmission as image data to be used. Specifically, control is performed to transfer the image data as the image section to be generated from the video servers 4A, 4B, 4C, 4D to NAS5.
　出力画像生成部２３は、自由視点画像サーバ２が生成し、受信した自由視点画像（ＦＶクリップ）を含む出力画像（出力クリップ）を生成する処理を行う。
　例えば画像作成コントローラ１は、出力画像生成部２３の処理により、自由視点画像サーバ２が生成した仮想的な画像であるＦＶクリップに、その前の時点の実際の動画である前クリップと、後の時点の実際の動画である後クリップを時間軸上で結合させて出力クリップとする。即ち、前クリップ＋ＦＶクリップ＋後クリップを１つの出力クリップとする。
　もちろん、前クリップ＋ＦＶクリップを１つの出力クリップとしてもよい。
　或いは、ＦＶクリップ＋後クリップを１つの出力クリップとしてもよい。
　さらには、前クリップや後クリップを結合せずにＦＶクリップのみの出力クリップを生成してもよい。
　いずれにしても画像作成コントローラ１は、ＦＶクリップを含む出力クリップを生成してスイッチャー６に出力し、放送に用いることができるようにする。 The output image generation unit 23 performs a process of generating an output image (output clip) including the received free viewpoint image (FV clip) generated by the free viewpoint image server 2.
For example, the image creation controller 1 adds an FV clip, which is a virtual image generated by the free-viewpoint image server 2 by the processing of the output image generation unit 23, a front clip, which is an actual moving image at a time before that, and a later clip. The rear clips, which are the actual moving images at the time, are combined on the time axis to form an output clip. That is, the front clip + FV clip + rear clip is regarded as one output clip.
Of course, the front clip + FV clip may be used as one output clip.
Alternatively, the FV clip + rear clip may be used as one output clip.
Further, the output clip of only the FV clip may be generated without combining the front clip and the rear clip.
In any case, the image creation controller 1 generates an output clip including the FV clip and outputs it to the switcher 6 so that it can be used for broadcasting.
　図５は、自由視点画像サーバ２となる情報処理装置７０のＣＰＵ７１において形成される機能として、対象画像取得部３１、画像生成処理部３２、送信制御部３３、及びカメラワーク生成処理部３４を示している。 FIG. 5 shows a target image acquisition unit 31, an image generation processing unit 32, a transmission control unit 33, and a camera work generation processing unit 34 as functions formed in the CPU 71 of the information processing device 70 that serves as the free viewpoint image server 2. ing.
　対象画像取得部３１は、複数の撮像装置１０により同時に撮像された複数の撮像画像（画像データＶ１からＶ１６）のそれぞれにおける、自由視点画像の生成対象とされた生成対象画像区間の画像データを取得する処理を行う。即ち画像作成コントローラ１が区間特定処理部２１の機能により特定した生成対象画像区間のイン点／アウト点で指定される１フレーム又は複数フレームの画像データをビデオサーバ４Ａ，４Ｂ，４Ｃ，４ＤからＮＡＳ５を介して取得して、自由視点画像の生成に用いることができるようにする。 The target image acquisition unit 31 acquires the image data of the generation target image section that is the generation target of the free viewpoint image in each of the plurality of captured images (image data V1 to V16) simultaneously captured by the plurality of imaging devices 10. Perform the processing to be performed. That is, the image data of one frame or a plurality of frames specified by the in / out points of the generation target image section specified by the image creation controller 1 by the function of the section specifying processing unit 21 is output from the video servers 4A, 4B, 4C, 4D to NAS5. It can be acquired via and used for the generation of a free viewpoint image.
　例えば対象画像取得部３１は、画像データＶ１からＶ１６の全てについて、生成対象画像区間の１フレーム又は複数フレームの画像データを取得する。画像データＶ１からＶ１６の全てについて生成対象画像区間の画像データを取得するのは、高品質な自由視点画像の生成のためである。上述のように少なくとも２以上の撮像装置１０の撮像画像を用いれば自由視点画像の生成は可能であるが、撮像装置１０の数（即ち視点の数）を多くすることにより、より精細な３Ｄモデルを生成して高品質な自由視点画像の生成が可能になる。そのため、例えば１６台の撮像装置１０を配置した場合は、１６台の撮像装置１０の画像データ（Ｖ１からＶ１６）の全てについて、生成対象画像区間の画像データを取得することが行われる。 For example, the target image acquisition unit 31 acquires image data of one frame or a plurality of frames of the generation target image section for all of the image data V1 to V16. The reason for acquiring the image data of the image section to be generated for all of the image data V1 to V16 is to generate a high-quality free-viewpoint image. As described above, it is possible to generate a free viewpoint image by using the images captured by at least two or more imaging devices 10, but by increasing the number of imaging devices 10 (that is, the number of viewpoints), a finer 3D model can be generated. Can be generated to generate high-quality free-viewpoint images. Therefore, for example, when 16 image pickup devices 10 are arranged, the image data of the generation target image section is acquired for all the image data (V1 to V16) of the 16 image pickup devices 10.
　画像生成処理部３２は、対象画像取得部３１が取得した画像データを用いて自由視点画像、即ち本例の場合のＦＶクリップを生成する機能である。
　例えば画像生成処理部３２は、３Ｄモデル生成、被写体解析を含むモデリング処理や、３Ｄモデルから２次元画像である自由視点画像を生成するレンダリング等の処理を行う。 The image generation processing unit 32 is a function of generating a free viewpoint image, that is, an FV clip in the case of this example, using the image data acquired by the target image acquisition unit 31.
For example, the image generation processing unit 32 performs processing such as modeling processing including 3D model generation and subject analysis, and rendering to generate a free viewpoint image which is a two-dimensional image from the 3D model.
　３Ｄモデル生成とは、各撮像装置１０による撮像画像と、例えばユーティリティサーバ８等から入力した撮像装置１０ごとのカメラパラメータとに基づいて、被写体を三次元空間上で表した（つまり二次元画像から被写体の三次元構造を復元した）３Ｄモデルデータを生成する処理である。具体的に、３Ｄモデルデータは、被写体を（Ｘ，Ｙ，Ｚ）による三次元座標系で表したデータを含む。
　被写体解析は、３Ｄモデルデータに基づき、人物（選手）としての被写体について位置や向き、姿勢についての解析を行う。具体的には、被写体の位置の推定、被写体の簡易モデルの生成、被写体の向きの推定などを行う。
　そして３Ｄモデルデータと被写体解析情報とに基づき自由視点画像を生成する。例えば被写体である選手が静止した状態の３Ｄモデルに対して、視点を動かしていくような自由視点画像の生成を行う。 The 3D model generation represents a subject in a three-dimensional space (that is, from a two-dimensional image) based on an image captured by each imaging device 10 and a camera parameter for each imaging device 10 input from, for example, a utility server 8. This is a process for generating 3D model data (which restores the three-dimensional structure of the subject). Specifically, the 3D model data includes data in which the subject is represented by a (X, Y, Z) three-dimensional coordinate system.
Subject analysis analyzes the position, orientation, and posture of a subject as a person (player) based on 3D model data. Specifically, the position of the subject is estimated, a simple model of the subject is generated, and the orientation of the subject is estimated.
Then, a free viewpoint image is generated based on the 3D model data and the subject analysis information. For example, a free viewpoint image that moves the viewpoint is generated for a 3D model in which the player who is the subject is stationary.
　図６を参照して自由視点画像の視点について述べておく。
　図６Ａでは、三次元空間上に設定した所要の視点から被写体を捉えた自由視点画像のイメージを示している。この場合の自由視点画像では、被写体Ｓ１が略正面視され、被写体Ｓ２が略背面視されている。
　図６Ｂでは、視点の位置を図６Ａの矢印Ｃ方向に変化させ、被写体Ｓ１を略背面視する視点が設定された場合の仮想視点画像のイメージを示している。この図６Ｂの自由視点画像では、被写体Ｓ２が略正面視され、また図６Ａでは映し出されていなかった被写体Ｓ３やバスケットゴールが映し出されている。
　例えば図６Ａの状態から、矢印Ｃの方向に徐々に視点を移動させ、図６Ｂの状態に至るような１秒から２秒程度の画像が自由視点画像（ＦＶクリップ）として生成されることになる。もちろん自由視点画像としてのＦＶクリップの時間長や、視点移動の軌跡は多様に考えられる。 The viewpoint of the free viewpoint image will be described with reference to FIG.
FIG. 6A shows an image of a free viewpoint image in which a subject is captured from a required viewpoint set in a three-dimensional space. In the free viewpoint image in this case, the subject S1 is viewed substantially in front and the subject S2 is viewed substantially in the rear.
FIG. 6B shows an image of a virtual viewpoint image when a viewpoint is set in which the position of the viewpoint is changed in the direction of the arrow C in FIG. 6A and the subject S1 is viewed substantially backward. In the free viewpoint image of FIG. 6B, the subject S2 is viewed substantially in front, and the subject S3 and the basket goal, which were not projected in FIG. 6A, are projected.
For example, from the state of FIG. 6A, the viewpoint is gradually moved in the direction of arrow C, and an image of about 1 to 2 seconds that reaches the state of FIG. 6B is generated as a free viewpoint image (FV clip). .. Of course, the time length of the FV clip as a free viewpoint image and the trajectory of the viewpoint movement can be considered in various ways.
　ここで、本例の自由視点画像サーバ２（ＣＰＵ７１）は、画像生成処理部３２の一部機能として、表示処理部３２ａとしての機能を有している。
　表示処理部３２ａは、自由視点画像の生成に用いるカメラワーク情報の指定操作を受け付けるカメラワーク指定画面Ｇｓの表示処理を行う。なお、自由視点画像に係るカメラワークや、カメラワーク指定画面Ｇｓの詳細については後に改めて説明する。 Here, the free-viewpoint image server 2 (CPU71) of this example has a function as a display processing unit 32a as a part of the function of the image generation processing unit 32.
The display processing unit 32a performs display processing of the camera work designation screen Gs that accepts the camera work information designation operation used for generating the free viewpoint image. The details of the camera work related to the free viewpoint image and the camera work designation screen Gs will be described later.
　また、本例における自由視点画像サーバ２は、画像生成処理部３２の一部機能として、カメラワーク編集処理部３２ｂとしての機能も有するが、このカメラワーク編集処理部３２ｂとしての機能についても後に改めて説明する。 Further, the free viewpoint image server 2 in this example also has a function as a camera work editing processing unit 32b as a part of the function of the image generation processing unit 32, but the function as the camera work editing processing unit 32b will be revisited later. explain.
　送信制御部３３は、画像生成処理部３２で生成した自由視点画像（ＦＶクリップ）を、ＮＡＳ５を介して画像作成コントローラ１に送信する制御を行う。この場合、送信制御部３３は、出力画像生成のための付随情報も画像作成コントローラ１に送信するように制御する。付随情報とは、前クリップや後クリップの画像を指定する情報が想定される。即ち、画像データＶ１からＶ１６のいずれの画像を用いて前クリップや後クリップを作成（切り出し）するかを指定する情報である。また付随情報として前クリップや後クリップの時間長を指定する情報も想定される。 The transmission control unit 33 controls to transmit the free viewpoint image (FV clip) generated by the image generation processing unit 32 to the image creation controller 1 via the NAS 5. In this case, the transmission control unit 33 controls to transmit the accompanying information for generating the output image to the image creation controller 1. The accompanying information is assumed to be information that specifies the image of the front clip or the rear clip. That is, it is information for designating which image of the image data V1 to V16 is used to create (cut out) the front clip or the rear clip. In addition, information that specifies the time length of the front clip and the rear clip is also assumed as incidental information.
　カメラワーク生成処理部３４は、自由視点画像の生成に用いるカメラワーク情報の生成に係る処理を行う。自由視点画像の作成にあたっては、様々なシーンに対応するために複数の候補となるカメラワークを事前に作成しておくことになる。このようなカメラワークの事前作成を可能とするために、本例の自由視点画像サーバ２には、カメラワーク作成用のソフトウエアプログラムがインストールされている。カメラワーク生成処理部３４は、このソフトウエアプログラムにより実現される機能であり、ユーザの操作入力に基づいてカメラワークの生成処理を行う。
　カメラワーク生成処理部３４は、表示処理部３４ａとしての機能を有する。表示処理部３４ａは、ユーザ（本例ではオペレータＯＰ２）によるカメラワーク作成のための各種操作入力の受け付けを可能とするべく、作成操作画面Ｇｇの表示処理を行う。
The camera work generation processing unit 34 performs processing related to generation of camera work information used for generating a free viewpoint image. When creating a free-viewpoint image, it is necessary to create a plurality of candidate camera works in advance in order to correspond to various scenes. In order to enable such pre-creation of camera work, a software program for creating camera work is installed in the free-viewpoint image server 2 of this example. The camera work generation processing unit 34 is a function realized by this software program, and performs camera work generation processing based on a user's operation input.
The camera work generation processing unit 34 has a function as a display processing unit 34a. The display processing unit 34a performs display processing of the creation operation screen Gg so that the user (operator OP2 in this example) can accept various operation inputs for creating the camera work.
<３．ＧＵＩの概要>

　図７及び図８を参照し、自由視点画像の作成に用いられるカメラワーク指定画面Ｇｓ、及びカメラワークの作成に用いられる作成操作画面Ｇｇの概要について説明しておく。本例において、これらカメラワーク指定画面Ｇｓ、作成操作画面Ｇｇは、例えば自由視点画像サーバ２における表示部７７に表示され、オペレータＯＰ２による確認や操作が可能とされている。 <3. GUI Overview>

With reference to FIGS. 7 and 8, the outline of the camera work designation screen Gs used for creating the free viewpoint image and the creation operation screen Gg used for creating the camera work will be described. In this example, the camera work designation screen Gs and the creation operation screen Gg are displayed on the display unit 77 of the free viewpoint image server 2, for example, and can be confirmed and operated by the operator OP2.
　図７に示すカメラワーク指定画面Ｇｓには、シーンウインドウ４１、シーンリスト表示部４２、カメラワークウインドウ４３、カメラワークリスト表示部４４、パラメータ表示部４５、及び送信ウインドウ４６が配置される。
　シーンウインドウ４１において、例えば生成対象画像区間の画像のモニタ表示が行われ、オペレータＯＰ２が自由視点画像を生成するシーンの内容を確認できるようにされる。
　シーンリスト表示部４２には、例えば生成対象画像区間に指定されたシーンのリストが表示される。オペレータＯＰ２はシーンウインドウ４１に表示させるシーンをシーンリスト表示部４２で選択できる。 On the camera work designation screen Gs shown in FIG. 7, a scene window 41, a scene list display unit 42, a camera work window 43, a camera work list display unit 44, a parameter display unit 45, and a transmission window 46 are arranged.
In the scene window 41, for example, the image of the image section to be generated is displayed on the monitor so that the operator OP2 can confirm the content of the scene for generating the free viewpoint image.
The scene list display unit 42 displays, for example, a list of scenes designated for the image section to be generated. The operator OP2 can select the scene to be displayed in the scene window 41 on the scene list display unit 42.
　カメラワークウインドウ４３には、配置されている撮像装置１０の位置や、選択されているカメラワーク、或いは選択可能な複数のカメラワークなどが表示される。 In the camera work window 43, the position of the arranged image pickup device 10, the selected camera work, a plurality of selectable camera works, and the like are displayed.
　ここで、カメラワークの情報とは、少なくとも自由視点画像における視点の移動軌跡を示す情報である。例えば、３Ｄモデルを生成した被写体に対して、視点の位置や視線方向、及び画角（焦点距離）を変化させていくようなＦＶクリップを作成する場合に、その視点の移動軌跡や視線方向の変化態様、画角の変化態様を定めるのに必要なパラメータが、カメラワークの情報とされる。
　カメラワークウインドウ４３には、カメラワークの表示として、少なくとも視点の移動軌跡を視覚化して示す情報が表示される。 Here, the camera work information is at least information indicating the movement locus of the viewpoint in the free viewpoint image. For example, when creating an FV clip that changes the position of the viewpoint, the direction of the line of sight, and the angle of view (focal length) of the subject for which the 3D model is generated, the movement trajectory and the direction of the line of sight of the viewpoint are changed. The parameters necessary to determine the change mode and the change mode of the angle of view are used as camera work information.
The camera work window 43 displays at least information that visualizes the movement locus of the viewpoint as a display of the camera work.
　カメラワークリスト表示部４４には、予め作成されて記憶されている各種のカメラワークの情報が一覧表示される。オペレータＯＰ２は、カメラワークリスト表示部４４に表示されているカメラワークのうちで、ＦＶクリップ生成に用いるカメラワークを選択し指定することができる。
　パラメータ表示部４５には、選択されているカメラワークに関する各種のパラメータが表示される。 The camera work list display unit 44 displays a list of various camera work information created and stored in advance. The operator OP2 can select and specify the camera work to be used for FV clip generation from the camera works displayed on the camera work list display unit 44.
Various parameters related to the selected camera work are displayed on the parameter display unit 45.
　送信ウインドウ４６には、作成したＦＶクリップを画像作成コントローラ１に送信することに関する情報が表示される。 Information about transmitting the created FV clip to the image creation controller 1 is displayed in the transmission window 46.
　続いて、図８の作成操作画面Ｇｇについて説明する。
　作成操作画面Ｇｇには、プリセットリスト表示部５１、カメラワークリスト表示部５２、カメラワークウインドウ５３、操作パネル部５４、及びプレビューウインドウ５５が配置される。 Subsequently, the creation operation screen Gg of FIG. 8 will be described.
A preset list display unit 51, a camera work list display unit 52, a camera work window 53, an operation panel unit 54, and a preview window 55 are arranged on the creation operation screen Gg.
　プリセットリスト表示部５１には、カメラのプリセットリスト、ターゲットのプリセットリスト、３Ｄモデルのプリセットリストを選択的に表示可能とされる。
　カメラのプリセットリストは、現場でのカメラ配置位置について、ユーザがプリセットしたカメラごとの位置情報（三次元空間上の位置情報）のリスト情報である。後述するように、カメラのプリセットリストが選択された場合、プリセットリスト表示部５１にはカメラの識別情報（例えば、camera1、camera2、・・・，camera16）ごとにその位置を示す情報が一覧表示される。 The preset list display unit 51 can selectively display a camera preset list, a target preset list, and a 3D model preset list.
The camera preset list is list information of position information (position information in three-dimensional space) for each camera preset by the user regarding the camera arrangement position in the field. As will be described later, when a camera preset list is selected, information indicating the position of each camera identification information (for example, camera1, camera2, ..., camera16) is displayed in a list on the preset list display unit 51. NS.
　また、ターゲットのプリセットリストについて、ターゲットとは、自由視点画像における視点からの視線方向を定める目標位置を意味する。自由視点画像の生成においては、視点からの視線方向はターゲットを向くように定められる。
　ターゲットのプリセットリストが選択された場合、プリセットリスト表示部５１にはユーザがプリセットしたターゲットについての識別情報とその位置を示す情報がリスト表示される。
　ここで、以下、上記のように自由視点画像における視点からの視線方向を定めるターゲットについては「ターゲットＴｇ」と表記する。 Further, in the preset list of targets, the target means a target position that determines the direction of the line of sight from the viewpoint in the free viewpoint image. In the generation of the free viewpoint image, the line-of-sight direction from the viewpoint is determined to face the target.
When the preset list of the target is selected, the preset list display unit 51 displays a list of identification information about the target preset by the user and information indicating the position thereof.
Here, hereinafter, the target that determines the line-of-sight direction from the viewpoint in the free-viewpoint image as described above is referred to as "target Tg".
　３Ｄモデルのプリセットリストは、カメラワークウインドウ４３の背景として表示する３Ｄモデルのプリセットリストであり、３Ｄモデルのプリセットリストが選択された場合、プリセットリスト表示部５１にはプリセットされた該３Ｄモデルの識別情報がリスト表示される。 The 3D model preset list is a 3D model preset list to be displayed as the background of the camera work window 43, and when the 3D model preset list is selected, the preset list display unit 51 identifies the preset 3D model. The information is listed.
　カメラワークリスト表示部５２には、作成操作画面Ｇｇを通じて作成されたカメラワークの情報や、作成操作画面Ｇｇを通じて新たに作成しようとするカメラワークの情報（後述するエントリ）を一覧表示可能とされる。 The camera work list display unit 52 can display a list of camera work information created through the creation operation screen Gg and camera work information (entry described later) to be newly created through the creation operation screen Gg. ..
　カメラワークウインドウ５３には、カメラワークの表示として、少なくとも視点の移動軌跡を視覚化して示す情報が表示される。
　操作パネル部５４は、カメラワーク作成における各種の操作入力を受け付ける領域とされる。
　プレビューウインドウ５５には、視点からの観察画像が表示される。プレビューウインドウ５５には、移動軌跡上で視点を移動させる操作が行われた場合に、該移動軌跡上の各視点位置からの観察画像が逐次表示される。また、後述するように、本例のプレビューウインドウ５５には、プリセットリスト表示部５１にカメラのプリセットリストが表示されている状態において、該カメラのプリセットリストからカメラを指定する操作が行われた場合は、該カメラの配置位置からの観察画像が表示される。 The camera work window 53 displays at least information that visualizes the movement locus of the viewpoint as a display of the camera work.
The operation panel unit 54 is an area for receiving various operation inputs in creating camera work.
An observation image from the viewpoint is displayed in the preview window 55. When the operation of moving the viewpoint on the movement locus is performed, the preview window 55 sequentially displays the observation images from each viewpoint position on the movement locus. Further, as will be described later, when the preset list of the camera is displayed on the preset list display unit 51 in the preview window 55 of this example, the operation of designating the camera from the preset list of the camera is performed. Displays an observation image from the arrangement position of the camera.
　なお、図７に示したカメラワーク指定画面Ｇｓの詳細やカメラワーク指定の具体的な手順、及び図８に示した作成操作画面Ｇｇの詳細やカメラワーク作成の具体的な手順については後に改めて説明する。
The details of the camera work designation screen Gs shown in FIG. 7, the specific procedure for specifying the camera work, the details of the creation operation screen Gg shown in FIG. 8, and the specific procedure for creating the camera work will be described later. do.
<４．自由視点画像を含むクリップ>

　続いて、自由視点画像としてのＦＶクリップを含む出力クリップについて説明する。
　図１０は出力クリップの一例として、前クリップ、ＦＶクリップ、後クリップを連結して構成されている状態を示している。 <4. Clips containing free-viewpoint images>

Next, an output clip including an FV clip as a free viewpoint image will be described.
FIG. 10 shows a state in which a front clip, an FV clip, and a rear clip are connected to each other as an example of an output clip.
　例えば前クリップは、画像データＶ１から画像データＶ１６のうちの或る画像データＶｘにおけるタイムコードＴＣ１からＴＣ２の区間の実際の動画である。
　また後クリップは、画像データＶ１から画像データＶ１６のうちの或る画像データＶｙにおけるタイムコードＴＣ５からＴＣ６の区間の実際の動画である。
　画像データＶｘは、ＦＶクリップによる視点移動開始前の撮像装置１０の画像データで、画像データＶｙは、ＦＶクリップによる視点移動終了時点の撮像装置１０の画像データであることが通常想定される。 For example, the front clip is an actual moving image of the section of the time code TC1 to TC2 in a certain image data Vx of the image data V1 to the image data V16.
The rear clip is an actual moving image of the section of the time code TC5 to TC6 in a certain image data Vy of the image data V1 to the image data V16.
It is usually assumed that the image data Vx is the image data of the image pickup device 10 before the start of the viewpoint movement by the FV clip, and the image data Vy is the image data of the image pickup device 10 at the end of the viewpoint movement by the FV clip.
　そしてこの例では、前クリップは、時間長ｔ１の動画、ＦＶクリップは時間長ｔ２の自由視点画像、後クリップは時間長ｔ３の動画としている。出力クリップ全体の再生時間長はｔ１＋ｔ２＋ｔ３となる。例えば５秒間の出力クリップとして、１．５秒の動画、２秒の自由視点画像、１．５秒の動画、などというような構成が考えられる。 And in this example, the front clip is a video with a time length of t1, the FV clip is a free viewpoint image with a time length of t2, and the rear clip is a video with a time length of t3. The playback time length of the entire output clip is t1 + t2 + t3. For example, as a 5-second output clip, a configuration such as a 1.5-second moving image, a 2-second free-viewpoint image, a 1.5-second moving image, or the like can be considered.
　ここで、ＦＶクリップについては、タイムコードＴＣ３からＴＣ４の区間として示しているが、これは実際の動画のフレーム数に相当することもあれば、相当しないこともある。
　即ちＦＶクリップとしては、動画の時刻を止めた状態で視点を移動させる場合（ＴＣ３＝ＴＣ４となる場合）と、動画の時刻を止めずに視点を移動させる場合（ＴＣ３≠ＴＣ４となる場合）があるためである。
  説明上、動画の時刻を止めた状態で視点を移動させる場合のＦＶクリップを「静止画ＦＶクリップ」、動画の時刻を止めずに視点を移動させる場合のＦＶクリップを「動画ＦＶクリップ」と呼ぶこととする。 Here, the FV clip is shown as a section from the time code TC3 to TC4, but this may or may not correspond to the actual number of frames of the moving image.
That is, as an FV clip, there are cases where the viewpoint is moved while the time of the moving image is stopped (when TC3 = TC4) and cases where the viewpoint is moved without stopping the time of the moving image (when TC3 ≠ TC4). Because there is.
For the sake of explanation, the FV clip for moving the viewpoint while the video time is stopped is called "still image FV clip", and the FV clip for moving the viewpoint without stopping the video time is called "video FV clip". I will do it.
　静止画ＦＶクリップを動画のフレームを基準にして示すと図１０のようになる。この例の場合、前クリップのタイムコードＴＣ１、ＴＣ２は、フレームＦ１、Ｆ８１のタイムコードとなり、続くフレームＦ８２のタイムコードが、図９のタイムコードＴＣ３＝ＴＣ４となる。そして後クリップのタイムコードＴＣ５、ＴＣ６は、フレームＦ８３、Ｆ１６６のタイムコードとなる。
　つまり、フレームＦ８２の１フレームの静止画に対して、視点が移動するような自由視点画像を生成する場合である。 FIG. 10 shows the still image FV clip with reference to the frame of the moving image. In the case of this example, the time codes TC1 and TC2 of the previous clip become the time codes of the frames F1 and F81, and the time code of the subsequent frame F82 becomes the time code TC3 = TC4 of FIG. Then, the time codes TC5 and TC6 of the rear clip become the time codes of the frames F83 and F166.
That is, it is a case of generating a free viewpoint image in which the viewpoint moves with respect to the still image of one frame of the frame F82.
　一方、動画ＦＶクリップについては図１１のようになる。この例の場合、前クリップのタイムコードＴＣ１、ＴＣ２は、フレームＦ１、Ｆ１０１のタイムコードとなり、フレームＦ１０２、Ｆ３０２のタイムコードが、図９のタイムコードＴＣ３、ＴＣ４となる。そして後クリップのタイムコードＴＣ５、ＴＣ６は、フレームＦ３０３、Ｆ５０３のタイムコードとなる。
　つまり、フレームＦ１０２からフレーム３０２までの複数フレームの区間の動画に対して、視点が移動するような自由視点画像を生成する場合である。 On the other hand, the moving image FV clip is as shown in FIG. In the case of this example, the time codes TC1 and TC2 of the front clip are the time codes of the frames F1 and F101, and the time codes of the frames F102 and F302 are the time codes TC3 and TC4 of FIG. Then, the time codes TC5 and TC6 of the rear clip become the time codes of the frames F303 and F503.
That is, it is a case where a free viewpoint image in which the viewpoint moves is generated for a moving image in a section of a plurality of frames from the frame F102 to the frame 302.
　従って画像作成コントローラ１が決定する生成対象画像区間とは、図１０の静止画ＦＶクリップを作成する場合は、フレームＦ８２の１フレームの区間となり、図１１の動画ＦＶクリップを作成する場合は、フレームＦ１０２からフレーム３０２までの複数フレームの区間となる。 Therefore, the generation target image section determined by the image creation controller 1 is one frame section of the frame F82 when the still image FV clip of FIG. 10 is created, and the frame when the moving image FV clip of FIG. 11 is created. It is a section of a plurality of frames from F102 to frame 302.
　図１０の静止画ＦＶクリップの例で、出力クリップの画像内容の例を図１２に示す。
　図１２において、前クリップはフレームＦ１からフレームＦ８１までの実際の動画である。ＦＶクリップではフレームＦ８１の場面において視点を移動させた仮想的な画像となる。後クリップはフレームＦ８３からフレームＦ１６６までの実際の動画である。
　例えばこのようにＦＶクリップを含む出力クリップが生成され、放送する画像として使用される。
In the example of the still image FV clip of FIG. 10, an example of the image content of the output clip is shown in FIG.
In FIG. 12, the front clip is an actual moving image from frame F1 to frame F81. In the FV clip, it becomes a virtual image in which the viewpoint is moved in the scene of the frame F81. The rear clip is an actual moving image from frame F83 to frame F166.
For example, an output clip including an FV clip is generated in this way and used as an image to be broadcast.
<５．クリップ作成処理>

　以下、図１の画像処理システムにおいて行われる出力クリップ作成の処理例を説明する。主に画像作成コントローラ１と自由視点画像サーバ２の処理に注目して説明する。
　まず図１３でオペレータＯＰ１、ＯＰ２の操作を含めた処理の流れを説明する。なお図１３におけるオペレータＯＰ１の処理は、画像作成コントローラ１のＧＵＩ処理とオペレータ操作をまとめて示している。またオペレータＯＰ２の処理は、自由視点画像サーバ２のＧＵＩ処理とオペレータ操作をまとめて示している。 <5. Clip creation process>

Hereinafter, an example of processing for creating an output clip performed in the image processing system of FIG. 1 will be described. The processing of the image creation controller 1 and the free-viewpoint image server 2 will be mainly focused on.
First, FIG. 13 will explain the flow of processing including the operations of operators OP1 and OP2. Note that the processing of the operator OP1 in FIG. 13 shows the GUI processing of the image creation controller 1 and the operator operation collectively. Further, the processing of the operator OP2 collectively shows the GUI processing and the operator operation of the free viewpoint image server 2.
・ステップＳ１：シーン選択
　出力クリップを作成する際は、まずオペレータＯＰ１がＦＶクリップとするシーンの選択を行うことになる。例えばオペレータＯＰ１は、画像作成コントローラ１側の表示部７７に表示される撮像画像をモニタリングしながら、ＦＶクリップとしたい場面を探す。そして１フレーム又は複数フレームの生成対象画像区間を選択する。
　この生成対象画像区間の情報は自由視点画像サーバ２に伝えられ、自由視点画像サーバ２側の表示部７７でのＧＵＩによりオペレータＯＰ２が認識できるようにされる。
　生成対象画像区間の情報とは、具体的には図９のタイムコードＴＣ３，ＴＣ４の情報となる。上述のように静止画ＦＶクリップの場合はタイムコードＴＣ３＝ＴＣ４となる。 Step S1: Scene selection When creating an output clip, the operator OP1 first selects a scene to be used as an FV clip. For example, the operator OP1 searches for a scene to be used as an FV clip while monitoring the captured image displayed on the display unit 77 on the image creation controller 1 side. Then, the generation target image section of one frame or a plurality of frames is selected.
The information of the image section to be generated is transmitted to the free viewpoint image server 2, and the operator OP2 can be recognized by the GUI on the display unit 77 on the free viewpoint image server 2 side.
The information of the image section to be generated is specifically the information of the time codes TC3 and TC4 of FIG. As described above, in the case of a still image FV clip, the time code TC3 = TC4.
・ステップＳ２：シーン画像転送指示
　オペレータＯＰ２は、生成対象画像区間の指定に応じて、該当のシーンの画像の転送指示の操作を行う。この操作に応じて自由視点画像サーバ２が、画像作成コントローラ１に対してタイムコードＴＣ３、ＴＣ４の区間の画像データの転送要求を送信する。 Step S2: Scene image transfer instruction The operator OP2 operates the image transfer instruction of the corresponding scene according to the designation of the generation target image section. In response to this operation, the free-viewpoint image server 2 transmits a transfer request for image data in the sections of the time codes TC3 and TC4 to the image creation controller 1.
・ステップＳ３：同期切り出し
　画像データの転送要求に応じて画像作成コントローラ１は、ビデオサーバ４Ａ，４Ｂ，４Ｃ，４Ｄを制御し、画像データＶ１から画像データＶ１６までの１６系統の画像データのそれぞれについて、タイムコードＴＣ３、ＴＣ４の区間の切り出しを実行させる。
・ステップＳ４：ＮＡＳ転送
　そして画像作成コントローラ１は画像データＶ１から画像データＶ１６の全てのタイムコードＴＣ３、ＴＣ４の区間のデータをＮＡＳ５に転送させる。 Step S3: Synchronous cutout The image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D in response to the image data transfer request, and for each of the 16 systems of image data from the image data V1 to the image data V16. , The section of the time codes TC3 and TC4 is cut out.
Step S4: NAS transfer Then, the image creation controller 1 transfers the data of all the time codes TC3 and TC4 sections of the image data V1 to the image data V16 to the NAS5.
・ステップＳ５：サムネイル表示
　自由視点画像サーバ２ではＮＡＳ５に転送されたタイムコードＴＣ３、ＴＣ４の区間の画像データＶ１から画像データＶ１６についてのサムネイルを表示させる。
・ステップＳ６：シーンチェック
　オペレータＯＰ２は、自由視点画像サーバ２によるカメラワーク指定画面ＧｓによりタイムコードＴＣ３，ＴＣ４で示される区間のシーン内容を確認する。
・ステップＳ７：カメラワーク選択
　オペレータＯＰ２は、シーン内容に応じて、カメラワーク指定画面Ｇｓで適切と考えるカメラワークを選択（指定）する。
・ステップＳ８：生成実行
　オペレータＯＰ２は、カメラワーク選択を行った後、ＦＶクリップの生成実行の操作を行う。 Step S5: Thumbnail display The free-viewpoint image server 2 displays thumbnails of image data V16 from image data V1 in the sections of time codes TC3 and TC4 transferred to NAS5.
Step S6: Scene check The operator OP2 confirms the scene contents of the section indicated by the time codes TC3 and TC4 on the camera work designation screen Gs by the free viewpoint image server 2.
Step S7: Camera work selection The operator OP2 selects (designates) the camera work that is considered appropriate on the camera work designation screen Gs according to the scene content.
Step S8: Generation execution The operator OP2 performs an operation of generating and executing an FV clip after selecting the camera work.
・ステップＳ９：モデリング
　自由視点画像サーバ２は、画像データＶ１からＶ１６のそれぞれにおけるタイムコードＴＣ３、ＴＣ４の区間のフレームのデータ、及び予め入力されていた各撮像装置１０の配置位置等のパラメータを用いて、被写体の３Ｄモデルの生成や、被写体解析等を行う。
・ステップＳ１０：レンダリング
　自由視点画像サーバ２は、３Ｄモデルデータと被写体解析情報とに基づき自由視点画像を生成する。このとき、ステップＳ７で選択されたカメラワークに基づく視点移動が行われるように自由視点画像を生成する。 Step S9: Modeling The free-viewpoint image server 2 uses parameters such as frame data in the sections of time codes TC3 and TC4 in each of the image data V1 to V16, and the arrangement position of each image pickup device 10 input in advance. Then, a 3D model of the subject is generated, the subject is analyzed, and the like.
Step S10: Rendering The free viewpoint image server 2 generates a free viewpoint image based on the 3D model data and the subject analysis information. At this time, a free viewpoint image is generated so that the viewpoint is moved based on the camera work selected in step S7.
・ステップＳ１１：転送
　自由視点画像サーバ２は、生成したＦＶクリップを画像作成コントローラ１に転送する。このとき、ＦＶクリップだけでなく、付随情報として前クリップ、後クリップの指定情報や、前クリップ、後クリップの時間長の指定情報も送信できる。
・ステップＳ１２：クオリティ確認
　なお自由視点画像サーバ２側では、ステップＳ１１の転送に先立って、或いは転送後に、オペレータＯＰ２によるクオリティ確認を行うことができる。即ち自由視点画像サーバ２は、生成したＦＶクリップをカメラワーク指定画面Ｇｓで再生表示させオペレータＯＰ２が確認できるようにする。場合によっては、オペレータＯＰ２が転送を実行させずに、ＦＶクリップの生成をやり直すと行ったことも可能とすることができる。 Step S11: Transfer The free viewpoint image server 2 transfers the generated FV clip to the image creation controller 1. At this time, not only the FV clip but also the specification information of the front clip and the rear clip and the specification information of the time length of the front clip and the rear clip can be transmitted as incidental information.
Step S12: Quality Confirmation On the free viewpoint image server 2 side, the quality can be confirmed by the operator OP2 before or after the transfer in step S11. That is, the free viewpoint image server 2 reproduces and displays the generated FV clip on the camera work designation screen Gs so that the operator OP2 can confirm it. In some cases, it is possible that the operator OP2 does not execute the transfer and regenerates the FV clip.
　ステップＳ１３：プレイリスト生成
　画像作成コントローラ１は、送信されてきたＦＶクリップを用いて出力クリップを生成する。この場合、ＦＶクリップに前クリップ、後クリップの一方又は両方を時間軸上で結合させて出力クリップを生成する。
　この出力クリップは、前クリップとしての各フレームと、ＦＶクリップとしての仮想的に生成した各フレームと、後クリップとしての各フレームを実際に時系列に連結したストリームデータとして生成してもよいが、この処理例では、プレイリストとして仮想的に連結することとしている。
　即ち前クリップとしてのフレーム区間の再生に続いて、ＦＶクリップが再生され、そのあとで後クリップとしてのフレーム区間が再生されるように、プレイリストを生成することで、出力クリップとしての実際に連結したストリームデータを生成しなくとも、出力クリップの再生が可能となるようにする。 Step S13: Playlist generation The image creation controller 1 generates an output clip using the transmitted FV clip. In this case, one or both of the front clip and the rear clip are combined with the FV clip on the time axis to generate an output clip.
This output clip may be generated as stream data in which each frame as a front clip, each virtually generated frame as an FV clip, and each frame as a rear clip are actually connected in chronological order. In this processing example, it is assumed that they are virtually linked as a playlist.
That is, by generating a playlist so that the FV clip is played after the frame section as the front clip is played, and then the frame section as the back clip is played, the actual connection as the output clip is performed. The output clip can be played back without generating the stream data.
　ステップＳ１４：クオリティ確認
　画像作成コントローラ１側のＧＵＩにより、プレイリストに基づく再生を行い、オペレータＯＰ１が出力クリップの内容を確認する。
　ステップＳ１５：再生指示
　オペレータＯＰ１は、クオリティ確認に応じて、所定の操作により再生指示を行う。画像作成コントローラ１は再生指示の入力を認識する。
　ステップＳ１６：再生
　再生指示に応じて画像作成コントローラ１は、出力クリップをスイッチャー６に供給する。これにより出力クリップの放送が実行可能となる。
Step S14: Quality confirmation The GUI on the image creation controller 1 side performs playback based on the playlist, and the operator OP1 confirms the contents of the output clip.
Step S15: Reproduction instruction The operator OP1 gives a reproduction instruction by a predetermined operation according to the quality confirmation. The image creation controller 1 recognizes the input of the playback instruction.
Step S16: Reproduction The image creation controller 1 supplies the output clip to the switcher 6 in response to the reproduction / reproduction instruction. This makes it possible to broadcast the output clip.
<６．カメラ変動検出>

　自由視点画像の生成のためには、画像データＶ１、Ｖ２・・・Ｖ１６を用いて３Ｄモデルを生成することから、各撮像装置１０の位置情報を含むパラメータが重要となる。
　例えば、放送の途中で或る撮像装置１０の位置が移動されたり、パン方向やチルト方向等に撮像方向が変化されたりした場合には、それに応じたパラメータのキャリブレーションが必要になる。そのため、図１の画像処理システムでは、ユーティリティサーバ８によりカメラの変動検出が行われるようにしている。ここで言うカメラの変動とは、カメラの位置、撮像方向の少なくとも何れかが変化することを意味する。 <6. Camera fluctuation detection>

In order to generate a free-viewpoint image, a 3D model is generated using the image data V1, V2 ... V16, so that parameters including the position information of each imaging device 10 are important.
For example, if the position of a certain imaging device 10 is moved during broadcasting or the imaging direction is changed in the pan direction, tilt direction, or the like, it is necessary to calibrate the parameters accordingly. Therefore, in the image processing system of FIG. 1, the utility server 8 detects fluctuations in the camera. The fluctuation of the camera referred to here means that at least one of the position of the camera and the imaging direction changes.
　図１４により、カメラの変動検出の際の画像作成コントローラ１とユーティリティサーバ８の処理手順を説明する。なお図１４は図１３と同様の形式で処理手順を示しているが、ユーティリティサーバ８についてもオペレータＯＰ２が操作を行う例としている。 FIG. 14 describes the processing procedure of the image creation controller 1 and the utility server 8 when detecting fluctuations in the camera. Note that FIG. 14 shows the processing procedure in the same format as in FIG. 13, but the operator OP2 also operates the utility server 8.
・ステップＳ３０：ＨＤ出力
　画像作成コントローラ１は、カメラ移動検出のため、ビデオサーバ４Ａ，４Ｂ，４Ｃ，４Ｄから画像データを画像変換部７に出力させるように制御する。ビデオサーバ４Ａ，４Ｂ，４Ｃ，４Ｄからの画像、即ち１６台の撮像装置１０の画像は、画像変換部７で解像度変換されてユーティリティサーバ８に供給される。 Step S30: HD output The image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D to output image data to the image conversion unit 7 for camera movement detection. The images from the video servers 4A, 4B, 4C, and 4D, that is, the images of the 16 image pickup devices 10 are converted in resolution by the image conversion unit 7 and supplied to the utility server 8.
・ステップＳ３１：背景生成
　ユーティリティサーバ８では、供給された画像に基づいて背景画像を生成する。背景画像は、カメラに変動がなければ変化しない画像であるため、例えば選手等の被写体を除いた背景画像を、１６系統の画像データ（Ｖ１からＶ１６）について生成する。
・ステップＳ３２：差分確認
　背景画像はＧＵＩ表示されることで、オペレータＯＰ２は画像の変化を確認できる。
・ステップＳ３３：変動自動検出
　各時点の背景画像を比較処理することで、カメラの変動を自動検出することもできる。 Step S31: The background generation utility server 8 generates a background image based on the supplied image. Since the background image is an image that does not change unless the camera fluctuates, for example, a background image excluding a subject such as a player is generated for 16 systems of image data (V1 to V16).
Step S32: Difference confirmation The background image is displayed in GUI, so that the operator OP2 can confirm the change in the image.
Step S33: Automatic fluctuation detection By comparing the background images at each time point, the fluctuation of the camera can be automatically detected.
・ステップＳ３４：カメラ変動検出
　上記のステップＳ３３又はステップＳ３２の結果として、或る撮像装置１０の変動が検出される。
・ステップＳ３５：画像取得
　撮像装置１０の変動が検出されたことに応じてキャリブレーションが必要になる。そこでユーティリティサーバ８は、変動後の状態の画像データを画像作成コントローラ１に要求する。
・ステップＳ３６：クリップ切り出し
　画像作成コントローラ１は、ユーティリティサーバ８からの画像取得の要求に応じて、ビデオサーバ４Ａ，４Ｂ，４Ｃ，４Ｄを制御し、画像データＶ１からＶ１６についてのクリップ切り出しを実行させる。
・ステップＳ３７：ＮＡＳ転送
　画像作成コントローラ１は、ビデオサーバ４Ａ，４Ｂ，４Ｃ，４Ｄに対してクリップとして切り出した画像データをＮＡＳ５に転送させる制御を行う。 Step S34: Camera fluctuation detection As a result of the above steps S33 or S32, fluctuations in a certain imaging device 10 are detected.
Step S35: Image acquisition Calibration is required according to the detection of fluctuations in the image pickup apparatus 10. Therefore, the utility server 8 requests the image creation controller 1 for the image data in the changed state.
Step S36: Clip cutting The image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D in response to the request for image acquisition from the utility server 8 to execute clip cutting for the image data V1 to V16. ..
Step S37: NAS transfer The image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D to transfer the image data cut out as a clip to the NAS 5.
・ステップＳ３８：特徴点修正
　ＮＡＳ５への転送により、ユーティリティサーバ８は、カメラ変動後の状態の画像を参照し、また表示させることができる。オペレータＯＰ２は特徴点修正などのキャリブレーションに必要な操作を行う。
・ステップＳ３９：再キャリブレーション
　ユーティリティサーバ８は、カメラ変動後の状態の画像データ（Ｖ１からＶ１６）を用いて、３Ｄモデル作成のためのキャリブレーションを再実行する。 -Step S38: Feature point correction By transferring to NAS5, the utility server 8 can refer to and display the image of the state after the camera change. The operator OP2 performs operations necessary for calibration such as correction of feature points.
Step S39: The recalibration utility server 8 re-executes the calibration for creating the 3D model using the image data (V1 to V16) in the state after the camera change.
・ステップＳ４０：背景再取得
　キャリブレーション後にオペレータＯＰ２の操作に応じて、ユーティリティサーバ８は背景画像のための画像データの再取得要求を行う。
・ステップＳ４１：クリップ切り出し
　画像作成コントローラ１は、ユーティリティサーバ８からの画像取得の要求に応じて、ビデオサーバ４Ａ，４Ｂ，４Ｃ，４Ｄを制御し、画像データＶ１からＶ１６についてのクリップ切り出しを実行させる。
・ステップＳ４２：ＮＡＳ転送
　画像作成コントローラ１は、ビデオサーバ４Ａ，４Ｂ，４Ｃ，４Ｄに対してクリップとして切り出した画像データをＮＡＳ５に転送させる制御を行う。
・ステップＳ４３：背景生成
　ユーティリティサーバ８はＮＡＳ５に転送された画像データを用いて背景画像を生成する。これは、例えば以降のカメラ変動検出の基準となる背景画像とされる。 Step S40: Background reacquisition The utility server 8 requests reacquisition of image data for the background image in response to the operation of the operator OP2 after the calibration.
Step S41: Clip cutting The image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D in response to the request for image acquisition from the utility server 8 to execute clip cutting for the image data V1 to V16. ..
Step S42: NAS transfer The image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D to transfer the image data cut out as a clip to the NAS 5.
Step S43: The background generation utility server 8 generates a background image using the image data transferred to the NAS 5. This is, for example, a background image that serves as a reference for subsequent camera fluctuation detection.
　例えば以上の手順のようにカメラ変動検出やキャリブレーションが行われることで、例えば放送中に撮像装置１０の位置や撮像方向が変化されたような場合にも、それに対応してパラメータが修正されるため、精度のよいＦＶクリップを継続して生成することができる。
For example, by performing camera fluctuation detection and calibration as in the above procedure, even if the position or imaging direction of the imaging device 10 is changed during broadcasting, the parameters are corrected accordingly. Therefore, it is possible to continuously generate an accurate FV clip.
<７．カメラワーク作成のためのＧＵＩ>

　以下、図８に示した作成操作画面Ｇｇの詳細、カメラワークの作成手順例、及びカメラワーク作成に係る各種の機能について図１５から図５１を参照して説明する。
　図１５は、作成操作画面Ｇｇの初期画面を例示した図である。
　前述もしたように作成操作画面Ｇｇには、プリセットリスト表示部５１、カメラワークリスト表示部５２、カメラワークウインドウ５３、操作パネル部５４、及びプレビューウインドウ５５が配置される。 <7. GUI for creating camera work>

Hereinafter, details of the creation operation screen Gg shown in FIG. 8, an example of a camera work creation procedure, and various functions related to camera work creation will be described with reference to FIGS. 15 to 51.
FIG. 15 is a diagram illustrating an initial screen of the creation operation screen Gg.
As described above, the preset list display unit 51, the camera work list display unit 52, the camera work window 53, the operation panel unit 54, and the preview window 55 are arranged on the creation operation screen Gg.
　図示のようにプリセットリスト表示部５１に対しては、カメラボタンＢ１、ターゲットボタンＢ２、及び３ＤモデルボタンＢ３が設けられている。カメラボタンＢ１は、プリセットリスト表示部５１に前述したカメラのプリセットリストを表示することの指示を行うためのボタンとされ、ターゲットボタンＢ２、３ＤモデルボタンＢ３は、それぞれプリセットリスト表示部５１に前述したターゲットのプリセットリスト、背景としての３Ｄモデルのプリセットリストを表示することの指示を行うためのボタンとされる。
　図中では、カメラボタンＢ１に下線マークが示されているが、これは、カメラのプリセットリスト表示が選択されている状態を意味する。 As shown in the figure, the preset list display unit 51 is provided with a camera button B1, a target button B2, and a 3D model button B3. The camera button B1 is a button for instructing the preset list display unit 51 to display the preset list of the camera described above, and the target buttons B2 and the 3D model button B3 are described above on the preset list display unit 51, respectively. It is a button for instructing to display the preset list of the target and the preset list of the 3D model as the background.
In the figure, the camera button B1 is underlined, which means that the preset list display of the camera is selected.
　また、プリセットリスト表示部５１には、フォルダ参照ボタンＢ４が設けられている。このフォルダ参照ボタンＢ４を操作することで、ユーザは、プリセットリスト表示部５１にリスト表示させたいデータを格納したフォルダを参照することができる。 Further, the preset list display unit 51 is provided with a folder reference button B4. By operating the folder reference button B4, the user can refer to the folder in which the data to be displayed in the list is stored in the preset list display unit 51.
　カメラワークリスト表示部５２に対しては、新規作成ボタンＢ５が設けられている。ユーザはこの新規作成ボタンＢ５を操作することで、カメラワークの新たなエントリの追加指示を行うことができる。追加されたカメラワークのエントリは、カメラワークリスト表示部５２に表示される。 A new creation button B5 is provided for the camera work list display unit 52. By operating the new creation button B5, the user can give an instruction to add a new entry of the camera work. The added camera work entry is displayed on the camera work list display unit 52.
　カメラワークウインドウ５３に対しては、Ｘ視点ボタンＢ６、Ｙ視点ボタンＢ７、Ｚ視点ボタンＢ８、Ｃａ視点ボタンＢ９、及びＰｅ視点ボタンＢ１０が設けられている。これらの各視点ボタンは、カメラワークウインドウ５３に表示する対象についての観察視点を指示するためのボタンとされる。具体的に、Ｘ視点ボタンＢ６、Ｙ視点ボタンＢ７、Ｚ視点ボタンＢ８は、それぞれカメラワークウインドウ５３に表示するカメラワーク情報の視覚化情報を観察する視点として、Ｘ軸上視点、Ｙ軸上視点、Ｚ軸上視点をそれぞれ指示するボタンとされ、Ｐｅ視点ボタンＢ１０は、該カメラワーク情報の視覚化情報の観察視点を任意位置に変更なモードへの移行指示を行うボタンとされる。また、Ｃａ視点ボタンＢ９は、カメラワーク情報として定められた視点移動軌跡上から対象の三次元空間を観察した画像の表示指示を行うボタンとされる。なお、Ｘ軸視点、Ｙ軸視点、Ｚ軸視点、Ｐｅ視点、Ｃａ視点のイメージについては後述する図５８から図６２を参照されたい。 The camera work window 53 is provided with an X viewpoint button B6, a Y viewpoint button B7, a Z viewpoint button B8, a Ca viewpoint button B9, and a Pe viewpoint button B10. Each of these viewpoint buttons is a button for instructing an observation viewpoint for an object to be displayed in the camera work window 53. Specifically, the X-viewpoint button B6, the Y-viewpoint button B7, and the Z-viewpoint button B8 are the X-axis upper viewpoint and the Y-axis upper viewpoint as viewpoints for observing the visualization information of the camera work information displayed on the camera work window 53, respectively. , The button for instructing the viewpoint on the Z-axis, and the Pe viewpoint button B10 is a button for instructing the transition to the mode in which the observation viewpoint of the visualization information of the camera work information is changed to an arbitrary position. Further, the Ca viewpoint button B9 is a button that gives an instruction to display an image of observing the target three-dimensional space from the viewpoint movement locus defined as camera work information. For the images of the X-axis viewpoint, the Y-axis viewpoint, the Z-axis viewpoint, the Pe viewpoint, and the Ca viewpoint, refer to FIGS. 58 to 62 described later.
　ここで、作成操作画面Ｇｇにおいて、カメラワークウインドウ５３やプレビューウインドウ５５における表示画像については、例えばマウスのホイール操作等の所定操作に応じて拡大や縮小することが可能とされる。また、これらカメラワークウインドウ５３、プレビューウインドウ５５では、例えばドラッグ操作等の所定の操作に応じて表示画像のスクロールを行うことが可能とされる。なお、表示画像の拡大や縮小、スクロールは、画面上に設けられたボタンの操作に応じて行われるようにすることもできる。 Here, on the creation operation screen Gg, the display image in the camera work window 53 or the preview window 55 can be enlarged or reduced according to a predetermined operation such as a mouse wheel operation. Further, in the camera work window 53 and the preview window 55, it is possible to scroll the display image in response to a predetermined operation such as a drag operation. It should be noted that the enlargement / reduction and scrolling of the displayed image can be performed according to the operation of the buttons provided on the screen.
　操作パネル部５４には、再生ボタンＢ１１、ポーズボタンＢ１２、停止ボタンＢ１３、タイムライン操作部５４ａ、速度調整操作部５６、及び軌跡形状調整操作部５７が設けられている。
　再生ボタンＢ１１、ポーズボタンＢ１２、停止ボタンＢ１３は、カメラワークウインドウ５３に表示するカメラワーク情報の視覚化画像、及びプレビューウインドウ５５に表示する視点からの観察画像についての再生、一時停止、停止をそれぞれ指示するボタンとされる。これら再生ボタンＢ１１、ポーズボタンＢ１２、停止ボタンＢ１３については、カメラワーク情報として少なくとも視点の移動軌跡の情報が定まった段階で有効化される。 The operation panel unit 54 is provided with a play button B11, a pause button B12, a stop button B13, a timeline operation unit 54a, a speed adjustment operation unit 56, and a locus shape adjustment operation unit 57.
The play button B11, the pause button B12, and the stop button B13 play, pause, and stop the visualized image of the camera work information displayed in the camera work window 53 and the observed image from the viewpoint displayed in the preview window 55, respectively. It is a button to instruct. The play button B11, the pause button B12, and the stop button B13 are enabled at least when the information on the movement locus of the viewpoint is determined as the camera work information.
　タイムライン操作部５４ａは、自由視点画像の視点の移動期間を表すタイムライン上において、カメラワークの作成に係る操作を受け付ける領域とされる。タイムライン操作部５４ａに対する操作の一例としては、例えばプリセットリスト表示部５１に一覧表示されたカメラのうちの一つを、タイムライン上の任意位置（つまり視点移動期間内の任意時点）にドラッグ&ドロップする操作を挙げることができる（図２７から図２９を参照）。後述するように、この操作は、視点の移動期間内において、ドラッグ&ドロップしたカメラの位置を視点が通過するタイミングの指定操作として機能する。 The timeline operation unit 54a is an area that accepts operations related to the creation of camera work on the timeline that represents the movement period of the viewpoint of the free viewpoint image. As an example of the operation for the timeline operation unit 54a, for example, one of the cameras listed in the preset list display unit 51 is dragged and & dropped to an arbitrary position on the timeline (that is, an arbitrary time point within the viewpoint movement period). The dropping operation can be mentioned (see FIGS. 27 to 29). As will be described later, this operation functions as an operation for designating the timing at which the viewpoint passes the position of the dragged and dropped camera within the movement period of the viewpoint.
　速度調整操作部５６は、視点の移動速度を調整するための各種の操作ボタンが配置される。また、軌跡形状調整操作部５７は、視点の移動軌跡の形状を調整するための各種の操作ボタンが配置される。
　これら速度調整操作部５６、軌跡形状調整操作部５７については後述する。 The speed adjustment operation unit 56 is arranged with various operation buttons for adjusting the moving speed of the viewpoint. Further, the locus shape adjustment operation unit 57 is arranged with various operation buttons for adjusting the shape of the movement locus of the viewpoint.
The speed adjustment operation unit 56 and the locus shape adjustment operation unit 57 will be described later.
　カメラワークの作成手順、及びカメラワーク作成に係る各種機能について説明する。
　先ず、ユーザ（本例ではオペレータＯＰ２）は、図１６に示すように、カメラのプリセットリストを取得するための操作を行う。この操作は、現場に実際に設置された各カメラの位置を示すカメラのプリセットリストを取得するための操作である。
　カメラのプリセットリストの取得は、フォルダ参照ボタンＢ４を操作し、該当するフォルダを指定することで行う。 The procedure for creating the camera work and various functions related to the creation of the camera work will be described.
First, the user (operator OP2 in this example) performs an operation for acquiring the preset list of the camera as shown in FIG. This operation is an operation for acquiring a preset list of cameras indicating the positions of the cameras actually installed in the field.
The preset list of the camera is acquired by operating the folder reference button B4 and specifying the corresponding folder.
　フォルダが指定されると、表示処理部３４ａは、指定されたフォルダのデータ内容に従ったカメラのプリセットリストをプリセットリスト表示部５１に表示する処理を行う。
　また、同時に、表示処理部３４ａは、カメラワークウインドウ５３に対し、取得したカメラの位置情報に基づき、各カメラの配置を視覚的に示す情報を表示する処理を行う。具体的には、各カメラの位置を示すカメラ位置マークＭｃを表示する処理を行う。
　なお、カメラ位置マークＭｃの表示に関しては、各カメラを色分けにより識別する表示を行ってもよい。例えば、カメラのプリセットリストにおいて各カメラを色分けして示し、カメラワークウインドウ４３においても同様の色分けにより各カメラ位置マークＭｃを表示するなどといったことが可能である。
　また、カメラワークウインドウ５３において、マウスオーバーされたカメラ位置マークＭｃに対し、カメラの識別情報（例えば、camera1、camera2、・・・、cameraXなど）を表示することも考えられる。 When the folder is designated, the display processing unit 34a performs a process of displaying the preset list of the camera according to the data contents of the designated folder on the preset list display unit 51.
At the same time, the display processing unit 34a performs a process of displaying information visually indicating the arrangement of each camera on the camera work window 53 based on the acquired position information of the cameras. Specifically, a process of displaying a camera position mark Mc indicating the position of each camera is performed.
Regarding the display of the camera position mark Mc, each camera may be identified by color coding. For example, each camera can be color-coded in the camera preset list, and each camera position mark Mc can be displayed in the same color-coded manner in the camera work window 43.
It is also conceivable to display camera identification information (for example, camera1, camera2, ..., cameraX, etc.) on the camera position mark Mc over which the mouse is over in the camera work window 53.
　本例では、カメラワークウインドウ５３において、背景として表示する３Ｄモデルの変更が可能とされる。
　図１７、図１８により、背景３Ｄモデルの変更について説明すると、背景３Ｄモデルの変更を希望する場合、ユーザは、３ＤモデルボタンＢ３を操作してプリセットリスト表示部５１の表示状態を３Ｄモデルのプリセットリストの表示状態とする。この表示状態では、プリセットリスト表示部５１内に図示のようなデフォルト指定ボタンＢ１４、Ｇｒｉｄ指定ボタンＢ１５、及びＮ／Ａ指定ボタンＢ１６が表示され、ユーザはこれらのボタンの操作によって背景３Ｄモデルの切り替えを行うことが可能とされる。ここで、デフォルト指定ボタンＢ１４は、初期設定として予め用意された背景３Ｄモデル（例えば、ステージやグラウンド等を示す３Ｄモデル）への切り替え指示を行うボタン、Ｇｒｉｄ指定ボタンＢ１５は距離や角度が視認できる背景３Ｄモデル（例えばグリッド線やマス目等）への切り替えを指示するボタンとされる。また、Ｎ／Ａ指定ボタンＢ１６は、背景３Ｄモデルの表示オフを指示するボタンとされる。
　図１７では一例として、Ｇｒｉｄ指定ボタンＢ１５が操作された場合におけるカメラワークウインドウ５３の背景３Ｄモデルの例を示している。また、図１８では、デフォルト指定ボタンＢ１４が操作された場合におけるカメラワークウインドウ５３の背景３Ｄモデルの例を示している。 In this example, the 3D model displayed as the background can be changed in the camera work window 53.
Explaining the change of the background 3D model with reference to FIGS. 17 and 18, when the user wishes to change the background 3D model, the user operates the 3D model button B3 to change the display state of the preset list display unit 51 to the preset of the 3D model. The list is displayed. In this display state, the default designation button B14, the grid designation button B15, and the N / A designation button B16 as shown in the figure are displayed in the preset list display unit 51, and the user can switch the background 3D model by operating these buttons. It is possible to do. Here, the default designation button B14 is a button for instructing switching to a background 3D model (for example, a 3D model indicating a stage, a ground, etc.) prepared in advance as an initial setting, and the grid designation button B15 can visually recognize the distance and angle. It is a button for instructing switching to a background 3D model (for example, grid lines, squares, etc.). Further, the N / A designation button B16 is a button for instructing the display off of the background 3D model.
As an example, FIG. 17 shows an example of a background 3D model of the camera work window 53 when the Grid designation button B15 is operated. Further, FIG. 18 shows an example of a background 3D model of the camera work window 53 when the default designation button B14 is operated.
　ユーザは、カメラワークの作成を開始するにあたっては、図１９に示すように、新規作成ボタンＢ５を操作する。
　新規作成ボタンＢ５の操作に応じ、表示処理部３４ａは、カメラワークリスト表示部５２にカメラワークの新たなエントリを表示する。このエントリには、視点移動の始点となるＩｎカメラ、及び視点移動の終点となるＯｕｔカメラを指定するための操作部が表示される。
　本例では、自由視点画像の生成において、被写体の３Ｄモデルに対しカメラの撮像画像に基づくテクスチャの貼り付けを行うため、視点の移動軌跡としては、できる限りカメラ位置を経由する移動軌跡を作成することが望ましい。特に、自由視点画像における視点移動の始点と終点は前後クリップとの画像の切り替わり点となるため、視点移動の始点、終点は、カメラ位置と一致させたい。このため、視点移動の始点、終点については、それぞれＩｎカメラ、Ｏｕｔカメラとしての、カメラ位置として指定させる。
　なお、視点の移動始点、移動終点については必ずしもカメラ位置に限定されるものではなく、カメラ位置以外の任意の位置とすることも可能である。 When starting the creation of the camera work, the user operates the new creation button B5 as shown in FIG.
In response to the operation of the new creation button B5, the display processing unit 34a displays a new entry of the camera work on the camera work list display unit 52. In this entry, an operation unit for designating an In camera as a start point of the viewpoint movement and an Out camera as the end point of the viewpoint movement is displayed.
In this example, in the generation of the free viewpoint image, since the texture based on the image captured by the camera is pasted on the 3D model of the subject, the movement locus of the viewpoint is created as much as possible via the camera position. Is desirable. In particular, since the start point and the end point of the viewpoint movement in the free viewpoint image are the switching points of the image with the front and rear clips, the start point and the end point of the viewpoint movement should be the same as the camera position. Therefore, the start point and the end point of the viewpoint movement are designated as the camera positions as the In camera and the Out camera, respectively.
The movement start point and movement end point of the viewpoint are not necessarily limited to the camera position, and can be any position other than the camera position.
　図示のようにＩｎカメラ、Ｏｕｔカメラを指定するための操作部については、例えばプルダウン形式でカメラを指定させる操作部とされる。ユーザ操作によりプルダウンが指示されると、指定可能なカメラ、すなわちユーザにより指定されたカメラのプリセットリストに掲載されている各カメラを示す情報（本例では、カメラの番号情報）が表示される（後述する図２２や図２４を参照）。 As shown in the figure, the operation unit for designating the In camera and Out camera is, for example, an operation unit for designating the camera in a pull-down format. When a pull-down is instructed by a user operation, information indicating a camera that can be specified, that is, each camera listed in the preset list of the camera specified by the user (camera number information in this example) is displayed (in this example, camera number information). See FIGS. 22 and 24, which will be described later).
　また、新規作成ボタンＢ５の操作に応じては、上記のようにカメラワークリスト表示部５２にカメラワークの新たなエントリが表示されると共に、カメラワークウインドウ５３において、ユーザにより設定されたターゲットＴｇの位置を示すマーク（以下「ターゲットマークＭｔ」と表記する）が表示される。
　ここで、カメラワークの作成作業において、ターゲットＴｇの位置は、例えば自由視点画像としてシュートシーンの画像を生成したい場合には対象の三次元空間（例えばサッカーであればサッカーグラウンド）におけるゴール付近の位置に設定される等、対象とするシーンに想定した適切とされる位置に設定される。ここでは、ターゲットＴｇの位置が、ユーザにより事前に自由視点画像サーバ２に設定可能とされている前提とする。 Further, in response to the operation of the new creation button B5, a new entry of the camera work is displayed on the camera work list display unit 52 as described above, and the target Tg set by the user in the camera work window 53 is displayed. A mark indicating the position (hereinafter referred to as "target mark Mt") is displayed.
Here, in the work of creating the camera work, the position of the target Tg is, for example, a position near the goal in the target three-dimensional space (for example, soccer ground in the case of soccer) when it is desired to generate an image of a shooting scene as a free viewpoint image. It is set to the appropriate position assumed for the target scene, such as being set to. Here, it is assumed that the position of the target Tg can be set in advance on the free viewpoint image server 2 by the user.
　先に触れたように、自由視点画像は、視点からの視線方向がターゲットＴｇを向く方向となるように生成することが可能とされる。具体的に、本例の自由視点画像は、視点の移動期間中における少なくとも一部期間内において、画枠内の所定位置（例えば、中央位置）にターゲットＴｇが位置し続けるように生成することが可能とされる。
　なお、自由視点画像の生成において、上記のように画枠内の所定位置にターゲットＴｇを位置させ続けることを「ターゲットＴｇに追従する」と表現する。この「ターゲットＴｇに追従する」とは、視点が移動している間に、視点からの視線方向がターゲットＴｇを向き続けることと同義である。 As mentioned earlier, the free viewpoint image can be generated so that the line-of-sight direction from the viewpoint faces the target Tg. Specifically, the free viewpoint image of this example can be generated so that the target Tg continues to be positioned at a predetermined position (for example, the center position) in the image frame during at least a part of the movement period of the viewpoint. It is possible.
In the generation of the free viewpoint image, continuing to position the target Tg at a predetermined position in the image frame as described above is expressed as "following the target Tg". This "following the target Tg" is synonymous with keeping the line-of-sight direction from the viewpoint continuing to face the target Tg while the viewpoint is moving.
　ここで、カメラワークウインドウ５３においてカメラのプリセットリストが表示されている状態では、該カメラのプリセットリストからのカメラの指定操作に応じて、指定されたカメラの位置に視点を設定した場合における視野範囲や視点からの観察画像（視点から三次元空間を観察した画像）がカメラワークウインドウ５３、プレビューウインドウ５５においてそれぞれ表示される。 Here, when the camera preset list is displayed in the camera work window 53, the field of view range when the viewpoint is set at the designated camera position in response to the camera designation operation from the camera preset list. And the observation image from the viewpoint (the image of observing the three-dimensional space from the viewpoint) is displayed in the camera work window 53 and the preview window 55, respectively.
　具体的に、図２０、図２１では、カメラのプリセットリストからcamera1としてのカメラ、camera2としてのカメラがそれぞれ指定された場合における作成操作画面Ｇｇの表示内容例を示している。
　この場合、カメラワークウインドウ５３には、カメラのプリセットリストから指定されたカメラについて、該カメラからの視野範囲を視覚化して示す視野範囲情報Ｆｖが表示される。図示のように本例では、視野範囲情報Ｆｖとして、視野範囲を図形により表した情報の表示が行われる。
　また、この場合のカメラワークウインドウ５３では、指定されたカメラについてのカメラ位置マークＭｃを他のカメラについてのカメラ位置マークＭｃよりも強調表示する（図中ではサイズを大きくする例を示している）ことで、ユーザが、指定したカメラが何れの位置にあるカメラであるかを容易に把握できるようにする。 Specifically, FIGS. 20 and 21 show an example of the display contents of the creation operation screen Gg when a camera as camera1 and a camera as camera2 are designated from the preset list of cameras.
In this case, the camera work window 53 displays the field of view information Fv that visualizes the field of view from the camera for the camera specified from the preset list of the cameras. As shown in the figure, in this example, information representing the visual field range is displayed as the visual field range information Fv.
Further, in the camera work window 53 in this case, the camera position mark Mc for the specified camera is highlighted more than the camera position mark Mc for other cameras (an example of increasing the size is shown in the figure). This makes it possible for the user to easily grasp the position of the designated camera.
　プレビューウインドウ５５では、指定されたカメラから三次元空間を観察した画像が表示される。
　ここで、本例では、カメラワークの作成作業は、自由視点画像の生成に先立って行われる前提としている。すなわち、自由視点画像の生成に用いる撮像画像が未取得の状態で行われる前提としている。このため、ここで言う視点から三次元空間を観察した画像は、対象とする実空間を撮像する各カメラの撮像画像から被写体検出等を行って生成した３Ｄモデル（以下、説明上「実三次元モデル」と表記する）を二次元画像としてレンダリングした画像ではなく、該対象とする実空間を模した仮想的な３Ｄモデル（「仮想三次元モデル」と表記する）を二次元画像としてレンダリングした画像となる。この場合における視点からの観察画像を生成する処理では、撮像画像が未取得の状態であるため、３Ｄモデルに対して撮像画像から生成したテクスチャを貼り付ける処理は行われないものとなる。 In the preview window 55, an image of observing the three-dimensional space from the designated camera is displayed.
Here, in this example, it is assumed that the camera work creation work is performed prior to the generation of the free viewpoint image. That is, it is premised that the captured image used for generating the free viewpoint image is performed in an unacquired state. Therefore, the image obtained by observing the three-dimensional space from the viewpoint referred to here is a 3D model generated by detecting the subject from the image captured by each camera that captures the target real space (hereinafter, "real three-dimensional" in the explanation. An image obtained by rendering a virtual 3D model (referred to as "virtual 3D model") that imitates the target real space as a 2D image, not an image obtained by rendering "model") as a 2D image. It becomes. In the process of generating the observation image from the viewpoint in this case, since the captured image is in an unacquired state, the process of pasting the texture generated from the captured image on the 3D model is not performed.
　図２２から図２５は、Ｉｎカメラ、Ｏｕｔカメラの指定手法についての説明図である。
　図２２に示すように、Ｉｎカメラの指定については、カメラワークリスト表示部５２に追加されたエントリにおけるＩｎカメラのプルダウンリストから、カメラの指定操作を行う。 22 to 25 are explanatory views of a method for designating an In camera and an Out camera.
As shown in FIG. 22, for the designation of the In camera, the camera designation operation is performed from the pull-down list of the In camera in the entry added to the camera work list display unit 52.
　図２３は、camera1がＩｎカメラとして指定された場合の作成操作画面Ｇｇの様子を示している。camera1がＩｎカメラとして指定された場合は、図示のようにカメラワークリスト表示部５２に追加されたエントリにおけるＩｎカメラの項目に「１」が表示される。
　また、カメラワークウインドウ５３では、camera1についてのカメラ位置マークＭｃが強調表示されると共に、camera1についての視野範囲情報Ｆｖが表示される。
　なお、先の図２０との対比として、カメラワークウインドウ５３におけるカメラ位置マークＭｃや視野範囲情報Ｆｖの表示態様については、カメラのプリセットリストから指定された場合とＩｎカメラに指定された場合とで異なる表示態様としてもよい。
　プレビューウインドウ５５には、camera1から三次元空間を観察した画像が表示される。 FIG. 23 shows the state of the creation operation screen Gg when camera1 is designated as the In camera. When camera1 is designated as the In camera, "1" is displayed in the In camera item in the entry added to the camera work list display unit 52 as shown in the figure.
Further, in the camera work window 53, the camera position mark Mc for camera1 is highlighted, and the field of view range information Fv for camera1 is displayed.
In comparison with FIG. 20, the display mode of the camera position mark Mc and the field of view range information Fv in the camera work window 53 depends on whether the camera is specified from the preset list of the camera or the In camera. It may have a different display mode.
In the preview window 55, an image of observing the three-dimensional space from camera1 is displayed.
　Ｏｕｔカメラを指定する場合には、図２４に示すように、カメラワークリスト表示部５２に追加されたエントリにおけるＯｕｔカメラのプルダウンリストから、カメラの指定操作を行う。
　図２５は、camera9がＯｕｔカメラとして指定された場合の作成操作画面Ｇｇの様子を示しているが、この場合、カメラワークリスト表示部５２に追加されたエントリにおけるＯｕｔカメラの項目に「９」が表示される。
　ここで、ＩｎカメラとＯｕｔカメラが指定されたことで、視点の移動軌跡が定まる。そのため、この場合のカメラワークウインドウ５３では、図中の移動軌跡情報Ｍｍとして示す、ＩｎカメラとＯｕｔカメラの位置間を結ぶ視点の移動軌跡を示す情報が表示される。ここで、移動軌跡情報Ｍｍは、視点の移動軌跡を視覚化した情報である。
　具体的に、この場合のカメラワークウインドウ５３では、Ｏｕｔカメラとして指定されたcamera9についてのカメラ位置マークＭｃが強調表示されると共に、ＩｎカメラとＯｕｔカメラとの位置間を結ぶ直線状の移動軌跡情報Ｍｍが図２３の場合から追加表示される。 When the Out camera is specified, as shown in FIG. 24, the camera designation operation is performed from the pull-down list of the Out camera in the entry added to the camera work list display unit 52.
FIG. 25 shows the state of the creation operation screen Gg when camera9 is designated as the Out camera. In this case, “9” is added to the Out camera item in the entry added to the camera work list display unit 52. Is displayed.
Here, by designating the In camera and the Out camera, the movement trajectory of the viewpoint is determined. Therefore, in the camera work window 53 in this case, information indicating the movement locus of the viewpoint connecting the positions of the In camera and the Out camera, which is shown as the movement locus information Mm in the figure, is displayed. Here, the movement locus information Mm is information that visualizes the movement locus of the viewpoint.
Specifically, in the camera work window 53 in this case, the camera position mark Mc for the camera9 designated as the Out camera is highlighted, and the linear movement trajectory information connecting the positions of the In camera and the Out camera is displayed. Mm is additionally displayed from the case of FIG. 23.
　図示による説明は省略するが、作成操作画面Ｇｇにおいては、上記のように少なくともＩｎカメラとＯｕｔカメラが指定されて視点の移動軌跡が形成された以降において、カメラワークのプレビュー表示を行うことが可能とされる。
　具体的に、このプレビュー表示は、操作パネル部５４における再生ボタンＢ１１の操作によりその開始の指示を行うことができる。カメラワークウインドウ５３において、カメラワークのプレビュー表示としては、視点の移動に伴い視野範囲情報Ｆｖが刻々と変化していく画像の表示が行われる。また、このようなカメラワークのプレビュー表示と連動して、プレビューウインドウ５５においては、視点の移動に伴い刻々と変化していく三次元空間の観察画像（視点からの観察画像）の表示が行われる。 Although the description by illustration is omitted, on the creation operation screen Gg, it is possible to preview the camera work after at least the In camera and the Out camera are specified and the movement trajectory of the viewpoint is formed as described above. It is said that.
Specifically, the preview display can be instructed to start by operating the play button B11 on the operation panel unit 54. In the camera work window 53, as the preview display of the camera work, an image in which the visual field range information Fv changes momentarily as the viewpoint moves is displayed. Further, in conjunction with the preview display of the camera work, the preview window 55 displays an observation image (observation image from the viewpoint) of the three-dimensional space that changes every moment as the viewpoint moves. ..
　また、本例において、このようなカメラワークのプレビュー表示や三次元空間の観察画像のプレビュー表示は、再生ボタンＢ１１の操作のみでなく、タイムライン操作部５４ａにおけるシークバーＢ１７のドラッグ操作によっても可能とされる。
　図２６では、タイムライン操作部５４ａにおけるシークバーＢ１７のドラッグ操作により、シークバーＢ１７をタイムライン上の所望の位置に位置させた様子を示している。
　シークバーＢ１７のドラッグ操作が行われている間は、タイムライン上、すなわち自由視点画像の開始タイミングから終了タイミングまでの時間軸上でのシークバーＢ１７の位置が刻々と変化していく。このため、ドラッグ操作が行われている間は、カメラワークウインドウ５３には、シークバーＢ１７が示すタイミングでの視点位置に対応した視野範囲情報Ｆｖが逐次表示されていき、ユーザには、視点の移動に伴い視野範囲情報Ｆｖが刻々と変化していく画像として視認される。同様にプレビューウインドウ５５においては、シークバーＢ１７の移動に応じて、刻々と変化していく視点からの三次元空間の観察画像が表示される。 Further, in this example, such a preview display of the camera work and a preview display of the observed image in the three-dimensional space can be performed not only by the operation of the play button B11 but also by the drag operation of the seek bar B17 in the timeline operation unit 54a. Will be done.
FIG. 26 shows how the seek bar B17 is positioned at a desired position on the timeline by the drag operation of the seek bar B17 in the timeline operation unit 54a.
While the seek bar B17 is being dragged, the position of the seek bar B17 changes every moment on the timeline, that is, on the time axis from the start timing to the end timing of the free viewpoint image. Therefore, while the drag operation is being performed, the field of view range information Fv corresponding to the viewpoint position at the timing indicated by the seek bar B17 is sequentially displayed in the camera work window 53, and the user can move the viewpoint. As a result, the visual field range information Fv is visually recognized as an image that changes from moment to moment. Similarly, in the preview window 55, an observation image of the three-dimensional space from a viewpoint that changes from moment to moment is displayed according to the movement of the seek bar B17.
　続いて、視点の経由地点について説明する。
　作成操作画面Ｇｇにおいては、視点の経由地点、及び該経由地点を視点が経由するタイミングの指定を行うことが可能とされる。
　図２７から図３０は、視点の経由地点、及び経由タイミングの指定についての説明図である。
　本例では、経由地点、及び該経由地点を視点が経由するタイミングの指定は、タイムライン操作部５４ａにおけるタイムライン上に経由地点として指定したいカメラをドラッグ&ドロップする操作により行うことができる。 Next, the waypoints of the viewpoint will be described.
On the creation operation screen Gg, it is possible to specify a waypoint of the viewpoint and a timing at which the viewpoint passes through the waypoint.
27 to 30 are explanatory views for designating a waypoint and a way timing of the viewpoint.
In this example, the waypoint and the timing at which the viewpoint passes through the waypoint can be specified by dragging and dropping the camera to be designated as the waypoint on the timeline in the timeline operation unit 54a.
　図２７から図２９は、camera6を経由地点として指定する場合の操作例の説明図である。
　先ず、図２７に示すように、プリセットリスト表示部５１におけるカメラのプリセットリストから経由地点として指定したいカメラを選択する。この選択のための操作は、本例ではマウスの左ボタンの押圧操作とされる。
　このように選択したカメラを、図２８に示すように画面上でドラッグしていき、図２９に示すように、タイムライン操作部５４ａのタイムライン上の所望の位置にドロップする（本例では左クリックの押圧状態を解除する）。これにより、経由地点とするカメラと、視点が該経由地点を経由するタイミングとの指定が完了する。
　この指定の完了に応じては、図２９に示すように、タイムライン操作部５４ａのタイムライン上に経由地点マークＭｖが表示される。この経由地点マークＭｖは、上記のドロップ操作によってタイムライン上において指定された位置に表示される。すなわち、自由視点画像の開始タイミングから終了タイミングまでの期間内（視点の移動期間内）における指定のタイミングを示す位置に表示される。
　本例において、この経由地点マークＭｖは、初期状態では図示のように四角マークにより表示される。 27 to 29 are explanatory views of an operation example when camera 6 is designated as a waypoint.
First, as shown in FIG. 27, a camera to be designated as a waypoint is selected from the preset list of the cameras on the preset list display unit 51. In this example, the operation for this selection is the pressing operation of the left mouse button.
The camera selected in this way is dragged on the screen as shown in FIG. 28, and dropped at a desired position on the timeline of the timeline operation unit 54a as shown in FIG. 29 (left in this example). Release the pressed state of the click). This completes the designation of the camera as the waypoint and the timing at which the viewpoint passes through the waypoint.
Upon completion of this designation, as shown in FIG. 29, the waypoint mark Mv is displayed on the timeline of the timeline operation unit 54a. This waypoint mark Mv is displayed at a position designated on the timeline by the above drop operation. That is, it is displayed at a position indicating a designated timing within the period from the start timing to the end timing of the free viewpoint image (within the movement period of the viewpoint).
In this example, the waypoint mark Mv is displayed by a square mark as shown in the initial state in the initial state.
　また、経由地点と経由のタイミングの指定が完了した状態では、カメラワークウインドウ５３には、該経由地点として指定されたカメラ（ここではcamera6）についてのカメラ位置マークＭｃが強調表示されると共に、該カメラからの視野範囲を示す視野範囲情報Ｆｖが表示される。また、プレビューウインドウ５５には、経由地点として指定されたカメラ位置の視点から三次元空間を観察した画像が表示される。 In addition, when the specification of the waypoint and the waypoint is completed, the camera position mark Mc for the camera (here, camera6) designated as the waypoint is highlighted in the camera work window 53, and the waypoint is highlighted. The field of view information Fv indicating the field of view from the camera is displayed. Further, in the preview window 55, an image of observing the three-dimensional space from the viewpoint of the camera position designated as the waypoint is displayed.
　上記のようなプリセットリストからのカメラの選択、及び選択したカメラのドラッグ&ドロップ操作により、視点の経由地点や該経由地点を視点が経由するタイミングの指定を行うことができる。
　図３０では、上記と同様の要領でさらに二つのカメラについて経由地点と経由のタイミングが指定された場合の作成操作画面Ｇｇの様子を例示している。
　なお、ここでは視点の経由地点としてカメラ位置を指定する例を挙げたが、該経由地点としてはカメラ位置以外の任意位置を指定することも可能である。 By selecting a camera from the preset list as described above and dragging and dropping the selected camera, it is possible to specify the waypoint of the viewpoint and the timing at which the viewpoint passes through the waypoint.
FIG. 30 illustrates the state of the creation operation screen Gg when a waypoint and a waying timing are specified for two cameras in the same manner as described above.
Although an example of designating the camera position as a waypoint of the viewpoint is given here, it is also possible to specify an arbitrary position other than the camera position as the waypoint.
　また、作成操作画面Ｇｇでは、視点の移動軌跡の形状の種類を指定することが可能とされる。
　図３１から図３３を参照して、具体的な操作手順及び操作に応じた画面遷移について説明する。
　先ず、図３１に例示するように、タイムライン操作部５４ａにおけるタイムライン上において、移動軌跡の種類の指定を行いたい対象範囲を指定する。図中では、図３０のように視点の経由地点が三つ設定された場合において、第一経由地点から第三経由地点までの範囲が指定された例を示している。 Further, on the creation operation screen Gg, it is possible to specify the type of shape of the movement locus of the viewpoint.
A specific operation procedure and screen transitions according to the operation will be described with reference to FIGS. 31 to 33.
First, as illustrated in FIG. 31, on the timeline of the timeline operation unit 54a, a target range for which the type of the movement locus is to be specified is specified. In the figure, when three waypoints of the viewpoint are set as shown in FIG. 30, an example in which the range from the first waypoint to the third waypoint is specified is shown.
　移動軌跡の形状の種類の指定には、軌跡形状調整操作部５７に設けられた操作ボタンを操作する。例えば、図３２に例示するように、軌跡形状調整操作部５７に設けられた曲線補間ボタンＢ１８を操作する。 To specify the type of shape of the movement locus, operate the operation button provided on the locus shape adjustment operation unit 57. For example, as illustrated in FIG. 32, the curve interpolation button B18 provided in the locus shape adjustment operation unit 57 is operated.
　この曲線補間ボタンＢ１８の操作に応じ、カメラワーク生成処理部３４は、図３１で指定された視点移動軌跡の一部指定範囲について、移動軌跡の曲線補間を行う。そして、表示処理部３４ａは、図３３に例示するように、この曲線補間によって生成された移動軌跡情報Ｍｍをカメラワークウインドウ５３に表示する処理を行う。 In response to the operation of the curve interpolation button B18, the camera work generation processing unit 34 performs curve interpolation of the movement locus for a part of the specified range of the viewpoint movement locus designated in FIG. 31. Then, as illustrated in FIG. 33, the display processing unit 34a performs a process of displaying the movement locus information Mm generated by this curve interpolation on the camera work window 53.
　このように視点の移動軌跡を曲線形状とすることで、視点が移動しても対象被写体から視点までの距離が大きく変化しないように図ることができる。換言すれば、自由視点画像内における対象被写体の大きさが大きく変化しないように図ることができる。 By making the movement locus of the viewpoint curved in this way, it is possible to prevent the distance from the target subject to the viewpoint from changing significantly even if the viewpoint moves. In other words, it is possible to prevent the size of the target subject in the free-viewpoint image from changing significantly.
　本例では、上記のような曲線補間が行われた場合には、タイムライン操作部５４ａにおけるタイムライン上に表示される経由地点マークＭｖの形状を、曲線補間に対応した形状に変更する処理を行う。具体的に本例では、図中に例示するように、経由地点マークＭｖの形状を四角マークから丸型マークに変更する。これにより、経由地点を結ぶ視点の移動軌跡について曲線補間が行われている旨をタイムライン上においてもユーザに通知することができる。
　なお、本例では、軌跡形状調整操作部５７において移動軌跡形状を直線状とすることを指示するための操作ボタンが配置され、該操作ボタンが操作された場合、該当する経由地点マークＭｖの形状は四角マークに変更される。 In this example, when the curve interpolation as described above is performed, the process of changing the shape of the waypoint mark Mv displayed on the timeline in the timeline operation unit 54a to a shape corresponding to the curve interpolation is performed. conduct. Specifically, in this example, as illustrated in the figure, the shape of the waypoint mark Mv is changed from the square mark to the round mark. As a result, it is possible to notify the user on the timeline that curve interpolation is being performed for the movement locus of the viewpoint connecting the waypoints.
In this example, an operation button for instructing the locus shape adjustment operation unit 57 to make the movement locus shape linear is arranged, and when the operation button is operated, the shape of the corresponding waypoint mark Mv. Is changed to a square mark.
　ここで、移動軌跡形状については、曲線、直線以外の形状の設定が可能である。例えば、図３３の移動軌跡情報Ｍｍで例示しているように、曲線と直線とが混在する形状などとすることができる。また、曲線による移動軌跡については、曲率を一定とすることに限定されず、一部区間で曲率を異ならせるような設定を行うことも可能である。
　また、このように移動軌跡形状にバリエーションを持たせる場合、経由地点マークＭｖについては、上記で例示したような直線、曲線の二種の表示形態のみでなく、各バリエーションに対応した異なる表示形態での表示を可能としてもよい。 Here, as for the movement locus shape, a shape other than a curved line or a straight line can be set. For example, as illustrated in the movement locus information Mm of FIG. 33, the shape may be a mixture of curved lines and straight lines. Further, the movement locus by the curve is not limited to the constant curvature, and it is also possible to set the curvature to be different in some sections.
Further, when the movement locus shape is provided with variations in this way, the waypoint mark Mv is not limited to the two types of display forms of straight lines and curves as illustrated above, but also has different display forms corresponding to each variation. May be possible to display.
　また、本例の作成操作画面Ｇｇにおいては、視点の移動速度についての指定を行うことが可能とされる。
　移動速度の指定においては、先ず、図３１で例示したように、タイムライン操作部５４ａにおけるタイムライン上において対象範囲の指定を行う。ここでは、図３４に例示するように、第二経由地点のみを跨がる範囲が指定されたとする。その上で、速度調整操作部５６に設けられた操作ボタンを操作する。例えば、図３４に例示するように、速度調整操作部５６に設けられた速度調整ボタンＢ１９を操作する。 Further, on the creation operation screen Gg of this example, it is possible to specify the moving speed of the viewpoint.
In specifying the moving speed, first, as illustrated in FIG. 31, the target range is specified on the timeline in the timeline operation unit 54a. Here, as illustrated in FIG. 34, it is assumed that a range straddling only the second waypoint is specified. Then, the operation buttons provided on the speed adjustment operation unit 56 are operated. For example, as illustrated in FIG. 34, the speed adjustment button B19 provided on the speed adjustment operation unit 56 is operated.
　この速度調整ボタンＢ１９の操作に応じ、カメラワーク生成処理部３４は、指定された視点移動軌跡の一部範囲について、操作されたボタンに応じた視点の速度調整を行う。
　本例では、このような速度調整が行われたことに応じ、表示処理部３４ａは、図３５に例示するように、タイムライン上における該当する経由地点マークＭｖの形状を、実行した速度調整の態様に応じた形状に変更する処理を行う。なお、図示の表示態様はあくまで一例である。このような形状変更を行うことで、視点移動軌跡中の該当範囲において速度調整が行われた旨をタイムライン上においてユーザに通知することができる。 In response to the operation of the speed adjustment button B19, the camera work generation processing unit 34 adjusts the speed of the viewpoint according to the operated button for a part range of the designated viewpoint movement locus.
In this example, in response to such speed adjustment, the display processing unit 34a performs the shape of the corresponding waypoint mark Mv on the timeline as illustrated in FIG. 35. A process of changing the shape according to the mode is performed. The indicated display mode is merely an example. By making such a shape change, it is possible to notify the user on the timeline that the speed adjustment has been performed in the corresponding range in the viewpoint movement locus.
　続いて、ターゲット位置の調整について説明する。
　先ず、ターゲットＴｇの意義について、図３６及び図３７を参照して説明しておく。
　前述のようにターゲットＴｇは、自由視点画像における視点からの視線方向を定めるために用いられる。図３６では、視点の移動軌跡における各カメラ位置（ここではcamera1,3,6,9としている）からの視野範囲Ｒｆ（カメラ位置ごとにＲｆ１、Ｒｆ３、Ｒｆ６、Ｒｆ９）、及び視線方向Ｄｇ（同じくカメラ位置ごとにＤｇ１、Ｄｇ３、Ｄｇ６、Ｄｇ９）を示している。 Subsequently, the adjustment of the target position will be described.
First, the significance of the target Tg will be described with reference to FIGS. 36 and 37.
As described above, the target Tg is used to determine the direction of the line of sight from the viewpoint in the free viewpoint image. In FIG. 36, the visual field range Rf (Rf1, Rf3, Rf6, Rf9 for each camera position) from each camera position (here, cameras1,3,6,9) in the movement locus of the viewpoint, and the line-of-sight direction Dg (same as above). Dg1, Dg3, Dg6, Dg9) are shown for each camera position.
　前述のように、本例における自由視点画像の生成においては、視点の移動期間内において、ターゲットＴｇを向く期間の指定を行うことが可能とされる。換言すれば、ターゲットＴｇに追従する期間としての、画枠内の所定位置にターゲットＴｇを位置させ続ける期間の指定を行うことが可能とされる。本例では、ターゲットＴｇの追従は、例えば図３７に例示するように画枠内の中央位置にターゲットＴｇを位置させ続けるようにして行う。
　図３６では、camera1からcamera9までの視点の移動期間内においてターゲットＴｇに追従する場合の視線方向Ｄｇ、視野範囲Ｒｆを例示しているが、図示のように本例では、それぞれの視線方向Ｄｇ、視野範囲Ｒｆは、画枠内の中央位置にターゲットＴｇを捉えるように設定される。 As described above, in the generation of the free viewpoint image in this example, it is possible to specify the period for facing the target Tg within the movement period of the viewpoint. In other words, it is possible to specify a period for keeping the target Tg positioned at a predetermined position in the image frame as a period for following the target Tg. In this example, the tracking of the target Tg is performed so that the target Tg is continuously positioned at the center position in the image frame as illustrated in FIG. 37, for example.
FIG. 36 illustrates the line-of-sight direction Dg and the field-of-view range Rf when following the target Tg within the movement period of the viewpoints from camera1 to camera9. The field of view Rf is set so as to capture the target Tg at the center position in the image frame.
　本例において、作成操作画面Ｇｇでは、ターゲットＴｇの位置を調整することが可能とされる。ターゲットＴｇの調整操作としては、例えばカメラワークウインドウ５３に表示されたターゲットマークＭｔの位置を調整する操作とすることが考えられる。
　ターゲットＴｇの位置が調整により変更された場合、カメラワーク生成処理部３４は、変更後のターゲットＴｇを画枠内の所定位置に位置させ続けるように各視点位置での視線方向Ｄｇや視野範囲Ｒｆを設定する。 In this example, the position of the target Tg can be adjusted on the creation operation screen Gg. As the adjustment operation of the target Tg, for example, an operation of adjusting the position of the target mark Mt displayed on the camera work window 53 can be considered.
When the position of the target Tg is changed by the adjustment, the camera work generation processing unit 34 increases the line-of-sight direction Dg and the visual field range Rf at each viewpoint position so as to keep the changed target Tg at a predetermined position in the image frame. To set.
　ここで、本例の作成操作画面Ｇｇでは、視点の移動期間中において、時間経過に伴ってターゲットＴｇの位置を移動させる指定を行うことが可能とされる。
　このようなターゲットＴｇの移動指定について、具体的な操作手順を図３８から図４５を参照して説明する。
　先ず、図３８に例示するように、ターゲットＴｇの新規地点Ｐｔｎの指定操作を行う。具体的には、カメラワークウインドウ５３において、新規地点Ｐｔｎを指定するためのターゲット位置指定マークＭｔｎを所望の位置にセットする操作を行う。図示は省略したが、ターゲット位置指定マークＭｔｎは、初期状態ではターゲットマークＭｔに重畳して表示されており、ユーザは、このターゲット位置指定マークＭｔｎをターゲットマークＭｔの位置から所望の位置にドラッグ&ドロップする。
　このような操作により、ターゲットＴｇの新規地点Ｐｔｎが指定されたことになる。 Here, on the creation operation screen Gg of this example, it is possible to specify that the position of the target Tg is moved with the passage of time during the moving period of the viewpoint.
A specific operation procedure for designating the movement of the target Tg will be described with reference to FIGS. 38 to 45.
First, as illustrated in FIG. 38, a new point Ptn of the target Tg is designated. Specifically, in the camera work window 53, an operation of setting the target position designation mark Mtn for designating a new point Ptn at a desired position is performed. Although not shown, the target position designation mark Mtn is displayed superimposed on the target mark Mt in the initial state, and the user drags and displays this target position designation mark Mtn from the position of the target mark Mt to a desired position. Drop it.
By such an operation, a new point Ptn of the target Tg is designated.
　次いでユーザは、プリセットリスト表示部５１に対して設けられたターゲットボタンＢ２を操作して、プリセットリスト表示部５１をターゲットＴｇのリストの表示状態とする。当該表示状態では、図３９に例示するように、プリセットリスト表示部５１にターゲットＴｇの追加ボタンＢ２０が表示され、ユーザはこの追加ボタンＢ２０を操作することで、ターゲット位置指定マークＭｔｎで指定した位置に対する新規ターゲットＴｇの追加指示を行うことができる。
　追加ボタンＢ２０の操作に応じては、図示のようにプリセットリスト表示部５１において、追加されたターゲットＴｇについての識別情報（図中「Target0」）と位置情報（新規地点Ｐｔｎを示す位置情報）とが表示される。また、カメラワークウインドウ５３においては、ターゲット位置指定マークＭｔｎの位置に、追加されたターゲットＴｇを表すマークとしての追加ターゲットマークＭｔｔが表示される。 Next, the user operates the target button B2 provided on the preset list display unit 51 to put the preset list display unit 51 in the display state of the list of the target Tg. In this display state, as illustrated in FIG. 39, the addition button B20 of the target Tg is displayed on the preset list display unit 51, and the user operates the addition button B20 to perform the position specified by the target position designation mark Mtn. It is possible to give an instruction to add a new target Tg to the target.
Depending on the operation of the add button B20, the preset list display unit 51 displays the identification information (“Target0” in the figure) and the position information (position information indicating the new point Ptn) of the added target Tg as shown in the figure. Is displayed. Further, in the camera work window 53, an additional target mark Mtt as a mark representing the added target Tg is displayed at the position of the target position designation mark Mtn.
　次いで、ユーザは、図４０から図４２の遷移として示すような新規ターゲットのタイムライン上への追加操作を行う。具体的には、プリセットリスト表示部５１に新たに表示されたターゲット（ここでは「Target0」）を、タイムライン操作部５４ａのタイムライン上の所望の位置にドラッグ&ドロップする操作を行う。
　ドロップ操作により指定されたタイムライン上の位置には、図４２に示すような到達目標タイミングマークＭｅｍが表示される。この到達目標タイミングマークＭｅｍは、視点のターゲットＴｇの位置をターゲットマークＭｔが示す位置（つまりターゲットＴｇの初期位置）から追加ターゲットマークＭｔｔが示す位置（つまり新規地点Ｐｔｎ）まで移動させる際における、ターゲットＴｇの新規地点Ｐｔｎまでの到達目標タイミングを示すマークとなる。換言すれば、上記のようなタイムライン上に対する新規ターゲットの追加操作は、ターゲットＴｇの移動に関して、ターゲットＴｇの位置を新規地点Ｐｔｎまで到達させる目標タイミングを指定する操作となる。 The user then performs an additional operation on the timeline of the new target as shown as the transition from FIGS. 40 to 42. Specifically, the operation of dragging and dropping the target newly displayed on the preset list display unit 51 (here, “Target0”) to a desired position on the timeline of the timeline operation unit 54a is performed.
At the position on the timeline designated by the drop operation, the arrival target timing mark Mem as shown in FIG. 42 is displayed. This arrival target timing mark Mem is a target when moving the position of the target Tg of the viewpoint from the position indicated by the target mark Mt (that is, the initial position of the target Tg) to the position indicated by the additional target mark Mtt (that is, the new point Ptn). It is a mark indicating the arrival target timing to the new point Ptn of Tg. In other words, the operation of adding a new target on the timeline as described above is an operation of designating the target timing for reaching the position of the target Tg to the new point Ptn with respect to the movement of the target Tg.
　ユーザは、上記のようなタイムライン上に対する新規ターゲットの追加操作を行った上で、図４３から図４５に示すようなターゲットＴｇを向く期間の指定操作を行う。
　ターゲットＴｇを向く期間の指定にあたっては、先ず、図４３に示すように、タイムライン操作部５４ａに設けられたLookAtボタンＢ２１を操作する。すると、図示のようにタイムライン上に期間指定のための期間指定バーＢ２２が表示される。この期間指定バーＢ２２は、LookAtボタンＢ２１が操作された段階では、図示のように視点の移動開始時点から到達目標タイミングマークＭｅｍが示す時点までの期間を指定する態様で表示される。ユーザは、ターゲットＴｇを向く期間を変更したい場合には、期間指定バーＢ２２を延伸又は縮小する操作を行う。ここでは、図４４及び図４５に例示するように、期間指定バーＢ２２を延伸して、ターゲットＴｇを向く期間が視点の移動終了時点までの期間（視点の移動開始から終了までの期間）として指定されたものとする。 After performing the operation of adding a new target on the timeline as described above, the user performs the operation of designating the period facing the target Tg as shown in FIGS. 43 to 45.
In designating the period for facing the target Tg, first, as shown in FIG. 43, the LookAt button B21 provided on the timeline operation unit 54a is operated. Then, as shown in the figure, the period designation bar B22 for specifying the period is displayed on the timeline. When the LookAt button B21 is operated, the period designation bar B22 is displayed in a manner of designating a period from the start time of movement of the viewpoint to the time point indicated by the arrival target timing mark Mem as shown in the figure. When the user wants to change the period for facing the target Tg, the user performs an operation of extending or reducing the period designation bar B22. Here, as illustrated in FIGS. 44 and 45, the period designation bar B22 is extended and the period facing the target Tg is designated as the period from the start to the end of the movement of the viewpoint (the period from the start to the end of the movement of the viewpoint). It shall be assumed that it was done.
　図４６から図４９は、図３８から図４５により説明したターゲットＴｇの移動についての各種指定操作が行われた場合のカメラワークのプレビュー再生画像、及び視点からの観察画像のプレビュー再生画像について説明するための図である。具体的には、図３８から図４５により説明したターゲットＴｇの移動についての各種指定操作が行われた後に、再生ボタンＢ１１が操作されたことに応じてカメラワークウインドウ５３、プレビューウインドウ５５にそれぞれ表示される画像の遷移を例示したものである。 46 to 49 show a preview reproduction image of the camera work and a preview reproduction image of the observation image from the viewpoint when various designation operations for the movement of the target Tg described with reference to FIGS. 38 to 45 are performed. It is a figure for. Specifically, after various designation operations for moving the target Tg described with reference to FIGS. 38 to 45 are performed, they are displayed in the camera work window 53 and the preview window 55 according to the operation of the play button B11, respectively. This is an example of the transition of the image to be performed.
　この場合、ターゲットＴｇの移動は、視点の移動開始から終了までの期間内において、タイムライン上の到達目標タイミングマークＭｅｍが示すタイミングで新規地点Ｐｔｎに到達するように行われる。このため、視点の移動開始時点から到達目標タイミングマークＭｅｍが示すタイミングに至るまでの間は、図４６及び図４７に示すように、ターゲットマークＭｔがターゲット位置指定マークＭｔｎ（カメラワークウインドウ５３においては追加ターゲットマークＭｔｔ）に徐々に近づいていくことになる。ここで、これら図４６及び図４７では、カメラワークウインドウ５３、プレビューウインドウ５５のそれぞれにターゲット初期位置マークＭｓｔが示されているが、このターゲット初期位置マークＭｓｔは、視点の移動開始時点でのターゲットＴｇの位置を示すマークである。 In this case, the movement of the target Tg is performed so as to reach the new point Ptn at the timing indicated by the arrival target timing mark Mem on the timeline within the period from the start to the end of the movement of the viewpoint. Therefore, as shown in FIGS. 46 and 47, the target mark Mt is the target position designation mark Mtn (in the camera work window 53, from the start of movement of the viewpoint to the timing indicated by the arrival target timing mark Mem). It will gradually approach the additional target mark Mtt). Here, in FIGS. 46 and 47, the target initial position mark Mst is shown in each of the camera work window 53 and the preview window 55, and the target initial position mark Mst is the target at the start of movement of the viewpoint. It is a mark indicating the position of Tg.
　本例では、ターゲットＴｇを向く期間が視点の移動開始から終了までの全期間として指定されている。すなわち、ターゲットＴｇを向く期間として、到達目標タイミングマークＭｅｍが示すタイミングまでの期間を超える期間が指定されている。
　このようにターゲットＴｇを向く期間が到達目標タイミングマークＭｅｍまでの期間を超える期間として指定されている場合、本例では、該到達目標タイミングマークＭｅｍまでの期間を超える期間において、ターゲットＴｇの移動として、ターゲットＴｇの位置を新規地点Ｐｔｎから移動開始位置まで徐々に戻すということを行う。
　このため、ターゲットＴｇを向く期間のうち到達目標タイミングマークＭｅｍまでの期間を超える期間では、図４８及び図４９に示すように、ターゲットマークＭｔが、時間経過に伴い、移動開始位置を示すターゲット初期位置マークＭｓｔに徐々に近づいていくことになる。 In this example, the period of facing the target Tg is specified as the entire period from the start to the end of the movement of the viewpoint. That is, as the period for facing the target Tg, a period exceeding the period until the timing indicated by the arrival target timing mark Mem is specified.
When the period of facing the target Tg is specified as a period exceeding the period up to the arrival target timing mark Mem in this way, in this example, as the movement of the target Tg in the period exceeding the period up to the arrival target timing mark Mem. , The position of the target Tg is gradually returned from the new point Ptn to the movement start position.
Therefore, as shown in FIGS. 48 and 49, in the period in which the target mark Mt faces the target Tg and exceeds the period up to the arrival target timing mark Mem, the target mark Mt indicates the movement start position with the passage of time. It will gradually approach the position mark Mst.
　上記のようなターゲットＴｇの移動を可能としていることで、ターゲットＴｇの位置が固定とされる場合よりも自由視点画像の作成自由度の向上を図ることができる。 By enabling the movement of the target Tg as described above, it is possible to improve the degree of freedom in creating a free viewpoint image as compared with the case where the position of the target Tg is fixed.
　上記では、ターゲットＴｇの位置を移動させる指定が行われた場合において、ターゲットＴｇを向く期間として、到達目標タイミングマークＭｅｍまでの期間を超える期間が指定された場合を例示したが、図５０に示すように、ターゲットＴｇを向く期間としては、視点移動開始時点から到達目標タイミングマークＭｅｍまでの期間を指定することもできる。この場合、到達目標タイミングマークＭｅｍまでの期間よりも以降の視点移動期間では、指定されたターゲットＴｇに対して追従しない自由視点画像が生成される。 In the above, when the position of the target Tg is specified to be moved, the case where the period exceeding the period up to the arrival target timing mark Mem is specified as the period for facing the target Tg is illustrated, but is shown in FIG. As described above, as the period for facing the target Tg, the period from the start point of viewpoint movement to the arrival target timing mark Mem can also be specified. In this case, a free viewpoint image that does not follow the designated target Tg is generated in the viewpoint movement period after the period up to the arrival target timing mark Mem.
　また、上記では、ターゲット位置指定マークＭｔｎを用いた新規のターゲットＴｇの追加として、一つのターゲットＴｇを追加する例を挙げたが、複数のターゲットＴｇを追加することもできる。
　そして、その場合、ターゲットＴｇを向く期間については、例えば図５１に例示するように、追加した各ターゲットＴｇについて個別に指定することもできる。図５１では、ターゲット位置指定マークＭｔｎを用いた操作（図３８及び図３９参照）により新規のターゲットＴｇとして二つのターゲットＴｇ（以下、ターゲットＴｇ－１、ターゲットＴｇ－２と表記する）が追加された場合において、それらのターゲットＴｇ－１、Ｔｇ－２をタイムライン上の異なる位置にそれぞれ追加することで、タイムライン上に到達目標タイミングマークＭｅｍ－１、Ｍｅｍ－２が個別に表示された状態を示している。この場合、ターゲットＴｇ－１については、図中の期間指定バーＢ２２－１が示すように、ターゲットＴｇを向く期間として視点の移動開始時点から到達目標タイミングマークＭｅｍ－１までの期間が指定されたとする。また、ターゲットＴｇ－２については、図中の期間指定バーＢ２２－２が示すように、ターゲットＴｇを向く期間として到達目標タイミングマークＭｅｍ－１が示す時点から所定時間経過後の時点から到達目標タイミングマークＭｅｍ－２が示すタイミングまでの期間が指定されたとする。 Further, in the above, as an example of adding one target Tg as the addition of a new target Tg using the target position designation mark Mtn, an example of adding one target Tg has been given, but a plurality of target Tg can also be added.
Then, in that case, the period for facing the target Tg can be individually specified for each added target Tg, for example, as illustrated in FIG. 51. In FIG. 51, two target Tg (hereinafter referred to as target Tg-1 and target Tg-2) are added as new target Tg by the operation using the target position designation mark Mtn (see FIGS. 38 and 39). In this case, by adding those targets Tg-1 and Tg-2 to different positions on the timeline, the arrival target timing marks Mem-1 and Mem-2 are individually displayed on the timeline. Is shown. In this case, for the target Tg-1, as shown by the period designation bar B22-1 in the figure, the period from the start of movement of the viewpoint to the arrival target timing mark Mem-1 is specified as the period for facing the target Tg. do. As for the target Tg-2, as shown by the period designation bar B22-2 in the figure, the arrival target timing starts from the time after a predetermined time elapses from the time indicated by the arrival target timing mark Mem-1 as the period for facing the target Tg. It is assumed that the period until the timing indicated by Mark Mem-2 is specified.
　この場合には、視点の移動期間内における期間指定バーＢ２２－１が示す期間では、ターゲットＴｇの位置が初期位置（視点移動開始時点でのターゲットＴｇの位置）からターゲットＴｇ－１の位置まで徐々に移動する自由視点画像が生成され、期間指定バーＢ２２－２が示す期間では、例えばターゲットＴｇの位置が初期位置からターゲットＴｇ－２の位置まで徐々に移動する自由視点画像が生成される。 In this case, in the period indicated by the period designation bar B22-1 within the viewpoint movement period, the position of the target Tg gradually changes from the initial position (the position of the target Tg at the start of the viewpoint movement) to the position of the target Tg-1. In the period indicated by the period designation bar B22-2, for example, a free viewpoint image in which the position of the target Tg gradually moves from the initial position to the position of the target Tg-2 is generated.
　ここで、図５１の説明から理解されるように、作成操作画面Ｇｇにおいては、複数のターゲットＴｇの位置の指定操作を受け付け可能とされている。
　これにより、視点移動期間内における或る期間ではターゲットＡに追従し、他の期間ではターゲットＢに追従する自由視点画像を生成する等といったことが可能となり、追従対象とするターゲットについての設定自由向上が図られる。
　従って、自由視点画像の作成自由度の向上を図ることができる。 Here, as understood from the explanation of FIG. 51, the creation operation screen Gg can accept the operation of designating the positions of a plurality of target Tg.
As a result, it is possible to generate a free viewpoint image that follows the target A in a certain period within the viewpoint movement period and follows the target B in another period, and the setting freedom of the target to be followed is improved. Is planned.
Therefore, it is possible to improve the degree of freedom in creating a free viewpoint image.
　なお、上記では、ターゲットＴｇの位置の指定として、視点の移動期間中にターゲットＴｇの位置を移動させる場合における移動先地点としてのターゲットＴｇの位置を指定する例を挙げたが、ターゲットＴｇの位置の指定は、視点移動期間中に移動を行わないターゲットＴｇの位置についての指定として行うこともできる。 In the above, as the designation of the position of the target Tg, an example of designating the position of the target Tg as the movement destination point when the position of the target Tg is moved during the movement period of the viewpoint is given, but the position of the target Tg is specified. Can also be specified as the position of the target Tg that does not move during the viewpoint movement period.
　図５２及び図５３のフローチャートを参照し、作成操作画面Ｇｇ上での操作入力に応じた移動軌跡の生成や表示に係る処理について説明する。
　なお、これら図５２及び図５３に示す処理は、自由視点画像サーバ２のＣＰＵ７１が実行する。この処理は、前述したカメラワーク生成処理部３４の一部機能を実現するための処理となる。 With reference to the flowcharts of FIGS. 52 and 53, processing related to generation and display of a movement locus according to an operation input on the creation operation screen Gg will be described.
The processes shown in FIGS. 52 and 53 are executed by the CPU 71 of the free-viewpoint image server 2. This process is a process for realizing a part of the functions of the camera work generation processing unit 34 described above.
　図５２は、Ｉｎカメラ、Ｏｕｔカメラの指定に応じた視点移動軌跡の生成・表示に係る処理を示している。
　先ず、ＣＰＵ７１はステップＳ１０１で、Ｉｎカメラの指定操作を待機する。本例において、この指定操作は、図２２で例示したようにカメラワークリスト表示部５２に表示されたカメラワークのエントリにおけるＩｎカメラのプルダウンリストに掲載されたカメラ番号を指定する操作として行われる。 FIG. 52 shows the processing related to the generation / display of the viewpoint movement locus according to the designation of the In camera and the Out camera.
First, in step S101, the CPU 71 waits for the designated operation of the In camera. In this example, this designation operation is performed as an operation of designating the camera number listed in the pull-down list of the In camera in the entry of the camera work displayed on the camera work list display unit 52 as illustrated in FIG.
　Ｉｎカメラの指定操作があった場合、ＣＰＵ７１はステップＳ１０２のＩｎカメラ表示処理として、図２３で説明したようなＩｎカメラに係る各種の表示処理を行う。例えば、カメラワークウインドウ５３における、Ｉｎカメラとして指定されたカメラについてのカメラ位置マークＭｃの強調表示や視野範囲情報Ｆｖの表示のための処理等を行う。 When there is an In-camera designation operation, the CPU 71 performs various display processes related to the In-camera as described in FIG. 23 as the In-camera display process in step S102. For example, in the camera work window 53, processing for highlighting the camera position mark Mc for the camera designated as the In camera, displaying the field of view range information Fv, and the like are performed.
　ステップＳ１０２に続くステップＳ１０３でＣＰＵ７１は、Ｏｕｔカメラの指定操作（図２４の説明を参照）を待機し、Ｏｕｔカメラの指定操作があった場合は、ステップＳ１０４に処理を進める。 In step S103 following step S102, the CPU 71 waits for an Out camera designation operation (see the description of FIG. 24), and if there is an Out camera designation operation, proceeds to step S104.
　ステップＳ１０４でＣＰＵ７１は、ＩｎカメラとＯｕｔカメラを結ぶ視点移動軌跡を生成する処理を行う。
　そして、続くステップＳ１０５でＣＰＵ７１は、Ｏｕｔカメラ及び視点移動軌跡の表示処理を実行する。すなわち、図２５で説明したようなＯｕｔカメラに係る各種の表示処理や視点の移動軌跡情報Ｍｍの表示処理を行う。
　ＣＰＵ７１は、ステップＳ１０５の表示処理を実行したことに応じて図５２に示す一連の処理を終える。 In step S104, the CPU 71 performs a process of generating a viewpoint movement locus connecting the In camera and the Out camera.
Then, in the following step S105, the CPU 71 executes the display process of the Out camera and the viewpoint movement locus. That is, various display processes related to the Out camera and display processing of the movement locus information Mm of the viewpoint as described with reference to FIG. 25 are performed.
The CPU 71 completes a series of processes shown in FIG. 52 in response to executing the display process in step S105.
　図５３は、経由地点の指定に応じた視点移動軌跡の生成・表示に係る処理を示している。
　ＣＰＵ７１はステップＳ１１０で、経由地点の指定操作を待機する。この指定操作は、本例では、図２６から図２８で説明したようなタイムラインに対する操作を含む一連の操作となる。 FIG. 53 shows a process related to the generation / display of the viewpoint movement locus according to the designation of the waypoint.
In step S110, the CPU 71 waits for the operation of designating the waypoint. In this example, this designation operation is a series of operations including an operation on the timeline as described with reference to FIGS. 26 to 28.
　経由地点が指定された場合、ＣＰＵ７１はステップＳ１１１で指定地点を経由する視点移動軌跡を生成する。すなわち、Ｉｎカメラと指定地点とＯｕｔカメラとを結ぶ視点の移動軌跡を生成する。
　そして、続くステップＳ１１２でＣＰＵ７１は、経由地点及び視点移動軌跡の表示処理を行う。すなわち、指定された経由地点について、図２９で説明したようなカメラワークウインドウ５３におけるカメラ位置マークの強調表示や視野範囲情報Ｆｖの表示、及びタイムラインにおける経由地点マークＭｖの表示のための処理を行う。
　ＣＰＵ７１は、ステップＳ１１２の表示処理を実行したことに応じて図５３に示す一連の処理を終える。
When the waypoint is designated, the CPU 71 generates a viewpoint movement locus passing through the designated point in step S111. That is, the movement locus of the viewpoint connecting the In camera, the designated point, and the Out camera is generated.
Then, in the following step S112, the CPU 71 performs display processing of the waypoint and the viewpoint movement locus. That is, for the designated waypoint, processing for highlighting the camera position mark in the camera work window 53, displaying the field of view range information Fv, and displaying the waypoint mark Mv in the timeline as described with reference to FIG. 29. conduct.
The CPU 71 completes a series of processes shown in FIG. 53 in response to the execution of the display process in step S112.
<８．自由視点画像作成のためのＧＵＩ>

　続いて、図７に示したカメラワーク指定画面Ｇｓの詳細、自由視点画像の生成に用いるカメラワークの指定手順例、及びカメラワーク指定に係る各種の機能について、図５４から図７２を参照して説明する。
　図５４は、カメラワーク指定画面Ｇｓの初期画面を例示した図である。ここで、以下で説明するカメラワーク指定画面Ｇｓの画面上における各種情報の表示に係る処理は、前述した表示処理部３２ａ（図５を参照）が行う。 <8. GUI for creating free-viewpoint images>

Subsequently, with reference to FIGS. 54 to 72, the details of the camera work designation screen Gs shown in FIG. 7, the example of the camera work designation procedure used for generating the free viewpoint image, and various functions related to the camera work designation are described. explain.
FIG. 54 is a diagram illustrating an initial screen of the camera work designation screen Gs. Here, the processing related to the display of various information on the screen of the camera work designation screen Gs described below is performed by the display processing unit 32a (see FIG. 5) described above.
　前述したようにカメラワーク指定画面Ｇｓには、シーンウインドウ４１、シーンリスト表示部４２、カメラワークウインドウ４３、カメラワークリスト表示部４４、パラメータ表示部４５、及び送信ウインドウ４６が設けられる。
　さらに、カメラワーク指定画面Ｇｓには、シーンウインドウ４１に対してカメラ指定操作部４７、静止画インポートボタンＢ３１、及び動画インポートボタンＢ３２が設けられると共に、画面下部に再生ボタンＢ３３、ポーズボタンＢ３４、及び停止ボタンＢ３５が設けられる。
　また、カメラワーク指定画面Ｇｓには、カメラワークウインドウ４３に対してＸ軸視点ボタンＢ３６、Ｙ軸視点ボタンＢ３７、Ｚ軸視点ボタンＢ３８、Ｃａ視点ボタンＢ３９、Ｐｅ視点ボタンＢ４０、表示パス制限ボタンＢ４１、及び制限解除ボタンＢ４２が設けられると共に、カメラワークリスト表示部４４に対してフィルタリング操作部４８が設けられる。フィルタリング操作部４８には、プルダウンボタンＢ４３とリセットボタンＢ４４が設けられる。 As described above, the camera work designation screen Gs is provided with a scene window 41, a scene list display unit 42, a camera work window 43, a camera work list display unit 44, a parameter display unit 45, and a transmission window 46.
Further, the camera work designation screen Gs is provided with a camera designation operation unit 47, a still image import button B31, and a video import button B32 for the scene window 41, and a play button B33, a pause button B34, and a pause button B34 at the bottom of the screen. A stop button B35 is provided.
Further, on the camera work designation screen Gs, the X-axis viewpoint button B36, the Y-axis viewpoint button B37, the Z-axis viewpoint button B38, the Ca viewpoint button B39, the Pe viewpoint button B40, and the display path restriction button B41 are displayed on the camera work window 43. , And the restriction release button B42 are provided, and a filtering operation unit 48 is provided for the camera work list display unit 44. The filtering operation unit 48 is provided with a pull-down button B43 and a reset button B44.
　カメラワーク指定画面Ｇｓにおいて、自由視点画像の生成にあたっては、先ずユーザは、前述した画像データＶ１からＶ１６としての、自由視点画像の生成対象区間の画像、換言すれば、自由視点画像の生成対象とするシーンの画像をインポートするための操作を行う。このインポートを行うにあたり、ユーザは、図中の静止画インポートボタンＢ３１、動画インポートボタンＢ３２の何れかを操作する。静止画インポートボタンＢ３１は、自由視点画像として前述した静止画ＦＶクリップを生成するための静止画による画像データＶ１からＶ１６をインポートする指示を行うためのボタンであり、動画インポートボタンＢ３２は、自由視点画像として前述した動画ＦＶクリップを生成するための動画による画像データＶ１からＶ１６をインポートする指示を行うためのボタンである。 In the generation of the free viewpoint image on the camera work designation screen Gs, the user first sets the image of the free viewpoint image generation target section as the above-mentioned image data V1 to V16, in other words, the generation target of the free viewpoint image. Perform the operation to import the image of the scene to be performed. In performing this import, the user operates either the still image import button B31 or the moving image import button B32 in the figure. The still image import button B31 is a button for instructing to import the image data V1 to V16 of the still image for generating the still image FV clip described above as a free viewpoint image, and the moving image import button B32 is a free viewpoint. It is a button for giving an instruction to import V16 from the moving image data V1 for generating the moving image FV clip described above as an image.
　これら静止画インポートボタンＢ３１、動画インポートボタンＢ３２の何れかが操作されたことに応じ、カメラワーク指定画面Ｇｓには図５５に示すようなポップアップウインドウＷ１が表示され、ユーザはポップアップウインドウＷ１に設けられた「GET IN/OUT TC」ボタンを操作することで、図５６に示すように、ポップアップウインドウＷ１内にインポート対象となる画像データＶ１からＶ１６を示す情報を表示させることができる。表示されたデータをインポートする場合、ユーザは、ポップアップウインドウＷ１内に設けられたＯＫボタンを操作する。 In response to the operation of either the still image import button B31 or the moving image import button B32, the pop-up window W1 as shown in FIG. 55 is displayed on the camera work designation screen Gs, and the user is provided in the pop-up window W1. By operating the "GET IN / OUT TC" button, as shown in FIG. 56, information indicating V1 to V16 of the image data to be imported can be displayed in the pop-up window W1. When importing the displayed data, the user operates the OK button provided in the pop-up window W1.
　ＯＫボタンが操作された場合、カメラワーク指定画面Ｇｓでは、図５７に示すように、シーンリスト表示部４２においてインポートしたシーンの情報が追加され、また、シーンウインドウ４１にはインポートしたシーンの画像が表示される。インポートによりリストに追加されたシーンの情報としては、シーンのサムネイル画像、タイムコードが示す時刻情報（本例では該当シーンの開始時刻、終了時刻の双方を表示）、シーンの期間を示す情報等が表示される。なお、図示の例は静止画としての画像データＶ１からＶ１６がインポートされた場合であり、シーンの期間を示す値が「０」とされている。 When the OK button is operated, on the camera work designation screen Gs, as shown in FIG. 57, the information of the imported scene is added in the scene list display unit 42, and the image of the imported scene is displayed in the scene window 41. Is displayed. The scene information added to the list by import includes thumbnail images of the scene, time information indicated by the time code (in this example, both the start time and end time of the relevant scene are displayed), information indicating the period of the scene, and the like. Is displayed. The illustrated example is a case where V1 to V16 as still images are imported, and the value indicating the period of the scene is set to "0".
　カメラ指定操作部４７には、インポートされたシーンについてのカメラごとの画像、すなわち画像データＶ１からＶ１６について、何れのカメラによる画像を表示するかの選択を行うためのカメラ選択ボタンが設けられている。 The camera designation operation unit 47 is provided with a camera selection button for selecting which camera to display the image of each camera for the imported scene, that is, the image data V1 to V16. ..
　カメラワーク指定画面Ｇｓでは、カメラワークウインドウ４３において、カメラワークのプレビューを行うことが可能とされる。
　カメラワークウインドウ４３においては、Ｘ軸視点ボタンＢ３６、Ｙ軸視点ボタンＢ３７、Ｚ軸視点ボタンＢ３８、Ｃａ視点ボタンＢ３９、及びＰｅ視点ボタンＢ４０により三次元空間上でのカメラワークの観察視点の切り替えを行うことが可能とされている。 On the camera work designation screen Gs, it is possible to preview the camera work in the camera work window 43.
In the camera work window 43, the X-axis viewpoint button B36, the Y-axis viewpoint button B37, the Z-axis viewpoint button B38, the Ca viewpoint button B39, and the Pe viewpoint button B40 are used to switch the observation viewpoint of the camera work in the three-dimensional space. It is possible to do it.
　Ｘ軸視点ボタンＢ３６、Ｙ軸視点ボタンＢ３７、Ｚ軸視点ボタンＢ３８は、三次元空間の観察視点をそれぞれＸ軸上の視点、Ｙ軸上の視点、Ｚ軸上の視点に切り替えるボタンとなる。ここで、Ｘ軸、Ｙ軸、Ｚ軸は、三次元空間を定める三つの軸であり、本例においてＸ軸は水平方向、Ｙ軸は垂直方向、Ｚ軸はＸ軸とＹ軸の双方に直交する方向を定義する軸とされている。
　Ｐｅ視点ボタンＢ４０は、三次元空間の観察視点をユーザが指定した任意視点に切り替えるボタンとなる。
　Ｃａ視点ボタンＢ３９は、三次元空間の観察視点をカメラワークにおける視点（視点移動軌跡上の点）に切り替えるボタンとなる。 The X-axis viewpoint button B36, the Y-axis viewpoint button B37, and the Z-axis viewpoint button B38 are buttons for switching the observation viewpoint in the three-dimensional space between the viewpoint on the X-axis, the viewpoint on the Y-axis, and the viewpoint on the Z-axis, respectively. Here, the X-axis, the Y-axis, and the Z-axis are three axes that define a three-dimensional space. In this example, the X-axis is in the horizontal direction, the Y-axis is in the vertical direction, and the Z-axis is in both the X-axis and the Y-axis. It is an axis that defines the directions that are orthogonal to each other.
The Pe viewpoint button B40 is a button for switching the observation viewpoint in the three-dimensional space to an arbitrary viewpoint specified by the user.
The Ca viewpoint button B39 is a button for switching the observation viewpoint in the three-dimensional space to the viewpoint (point on the viewpoint movement locus) in the camera work.
　図５８、図５９、図６０、図６１、図６２は、それぞれＸ軸視点ボタンＢ３６、Ｙ軸視点ボタンＢ３７、Ｚ軸視点ボタンＢ３８、Ｐｅ視点ボタンＢ４０、Ｃａ視点ボタンＢ３９が操作された場合のカメラワークウインドウ４３における表示画像を例示している。 58, 59, 60, 61, and 62 show the case where the X-axis viewpoint button B36, the Y-axis viewpoint button B37, the Z-axis viewpoint button B38, the Pe viewpoint button B40, and the Ca viewpoint button B39 are operated, respectively. The display image in the camera work window 43 is illustrated.
　ここで、カメラワークウインドウ４３には、カメラワークリスト表示部４４に表示されたカメラワークを示す情報が表示される。カメラワークリスト表示部４４に表示されるカメラワークは、前述した作成操作画面Ｇｇを通じて作成されたカメラワークであり、自由視点画像の生成に用いるカメラワークの候補となる。換言すれば、自由視点画像の生成に用いるカメラワークとして指定可能なカメラワークである。 Here, in the camera work window 43, information indicating the camera work displayed on the camera work list display unit 44 is displayed. The camera work displayed on the camera work list display unit 44 is the camera work created through the above-mentioned creation operation screen Gg, and is a candidate for the camera work used for generating the free viewpoint image. In other words, it is a camera work that can be specified as a camera work used for generating a free viewpoint image.
　カメラワークウインドウ４３において、カメラワークを示す情報としては、前述した移動軌跡情報Ｍｍ、カメラ位置マークＭｃ、視野範囲情報Ｆｖ（この場合も視野範囲を図形により示す情報とされる）が表示される。また、特に図６１に示すように、Ｉｎカメラ、Ｏｕｔカメラの位置を示すカメラ位置マークＭｃについては、他のカメラ位置マークＭｃよりも大きく表示され、Ｉｎカメラ、Ｏｕｔカメラの位置が示される。ここで、Ｉｎカメラ、Ｏｕｔカメラの各カメラ位置マークＭｃ、すなわち視点の移動始点となるカメラと移動終点となるカメラの各位置を示す情報（始点配置位置情報、終点配置位置情報）については、Ｉｎカメラ、Ｏｕｔカメラ以外の他のカメラのカメラ位置マークＭｃとは異なる態様で示されればよく、表示の大きさを異ならせることに限定されるものではない。
　さらに、カメラワークウインドウ４３においては、カメラワークを示す情報として、前述したターゲットＴｇの位置を示すターゲットマークＭｔが表示される。
　なお、図６２の場合、すなわちＣａ視点ボタンＢ３９が操作された場合は、視点移動軌跡上からの観察画像となるため、移動軌跡情報Ｍｍ、カメラ位置マークＭｃ、視野範囲情報Ｆｖの表示は行われず、ターゲットマークＭｔが表示される。 In the camera work window 43, as the information indicating the camera work, the above-mentioned movement locus information Mm, the camera position mark Mc, and the field of view range information Fv (also referred to as information indicating the field of view by a figure) are displayed. Further, as shown in FIG. 61, the camera position mark Mc indicating the positions of the In camera and the Out camera is displayed larger than the other camera position marks Mc, and the positions of the In camera and the Out camera are shown. Here, the camera position mark Mc of the In camera and the Out camera, that is, the information indicating each position of the camera that is the movement start point of the viewpoint and the camera that is the movement end point (start point arrangement position information, end point arrangement position information) is described as In. It suffices if it is shown in a manner different from the camera position mark Mc of other cameras other than the camera and the Out camera, and is not limited to different display sizes.
Further, in the camera work window 43, the target mark Mt indicating the position of the target Tg described above is displayed as the information indicating the camera work.
In the case of FIG. 62, that is, when the Ca viewpoint button B39 is operated, the movement locus information Mm, the camera position mark Mc, and the field of view range information Fv are not displayed because the observation image is obtained from the viewpoint movement locus. , Target mark Mt is displayed.
　図中では、ターゲットマークＭｔとして二つのターゲットマークＭｔが表示された例としている（特に図６０や図６１を参照）が、これは、カメラワークリスト表示部４４に表示された候補としての複数のカメラワークとして、それぞれ別のターゲットＴｇが設定されたカメラワークが混在していることを意味している。すなわち、この場合の候補としてのカメラワークには、例えば図６１中における左側のターゲットマークＭｔでその位置が示されたターゲットＴｇを設定されたものと、同図中における右側のターゲットマークＭｔでその位置が示されたターゲットＴｇを設定されたものとが混在している。 In the figure, two target marks Mt are displayed as target marks Mt (see particularly FIGS. 60 and 61), but this is a plurality of candidates displayed on the camera work list display unit 44. As the camera work, it means that the camera work in which different target Tg is set is mixed. That is, for the camera work as a candidate in this case, for example, the target Tg whose position is indicated by the target mark Mt on the left side in FIG. 61 is set, and the target mark Mt on the right side in FIG. The target Tg whose position is indicated is mixed with the set target Tg.
　カメラワークウインドウ４３においては、カメラワークの動的なプレビュー再生を行うことが可能とされている。このような動的なプレビュー再生は、再生ボタンＢ３３の操作により指示することができる。
　再生ボタンＢ３３が操作されると、図５８から図６１に示すＸ軸上視点、Ｙ軸上視点、Ｚ軸上視点、任意視点の各場合には、移動軌跡上における視点の移動に伴い視野範囲情報Ｆｖの位置や形状が刻々と変化していく画像が表示される。また、図６２の場合は、視点移動軌跡上の視点移動開始地点から終了地点にかけての各点から対象の三次元空間を観察した画像が逐次切り替えられて表示される。すなわち、視点の移動に伴い刻々と変化していく三次元空間の観察画像の表示が行われる。
　なお、ポーズボタンＢ１２、停止ボタンＢ１３は、それぞれ上記のような動的なプレビュー再生の一時停止、停止をそれぞれ指示するボタンである。 In the camera work window 43, it is possible to perform a dynamic preview reproduction of the camera work. Such dynamic preview playback can be instructed by operating the playback button B33.
When the play button B33 is operated, in each of the X-axis upper viewpoint, Y-axis upper viewpoint, Z-axis upper viewpoint, and arbitrary viewpoint shown in FIGS. 58 to 61, the visual field range is accompanied by the movement of the viewpoint on the movement locus. An image in which the position and shape of the information Fv changes from moment to moment is displayed. Further, in the case of FIG. 62, the images of observing the target three-dimensional space from each point from the viewpoint movement start point to the viewpoint movement end point on the viewpoint movement locus are sequentially switched and displayed. That is, the observation image of the three-dimensional space that changes every moment as the viewpoint moves is displayed.
The pause button B12 and the stop button B13 are buttons for instructing the pause and stop of the dynamic preview playback as described above, respectively.
　ここで、図６２の場合のように視点からの観察画像をプレビュー再生する場合、表示画像としては、選択されているカメラワークを用いて生成したＦＶクリップとしての自由視点画像を表示することもできる。すなわち、インポートした画像データＶ１からＶ１６に基づいて前述した実三次元モデルを生成し、実三次元モデルに対しテクスチャの貼り付け等を行ってレンダリングした二次元画像をプレビュー画像として表示するものである。 Here, when the observation image from the viewpoint is previewed and reproduced as in the case of FIG. 62, the free viewpoint image as an FV clip generated by using the selected camera work can be displayed as the display image. .. That is, the above-mentioned real three-dimensional model is generated from the imported image data V1 to V16, and a two-dimensional image rendered by pasting a texture on the real three-dimensional model is displayed as a preview image. ..
　しかしながら、ＦＶクリップとしての自由視点画像の生成には相応の処理負担や処理時間を要するものとなるため、プレビュー再生の開始までユーザを待たせる時間が長くなり、迅速な自由視点画像の作成に支障を来す虞がある。
　そのため、本例では、図６２の場合のように視点からの観察画像をプレビュー再生する場合に、その表示画像として、画像データＶ１からＶ１６（つまり実空間の撮像画像）に基づき生成した実三次元モデルではなく、前述した仮想三次元モデル（実空間の仮想３Ｄモデル）をレンダリングした画像を表示する。
　これにより、視点からの観察画像のプレビュー表示に要する処理時間の短縮化を図ることができ、自由視点画像の作成作業を迅速に実行することができる。 However, since the generation of the free-viewpoint image as an FV clip requires a considerable processing load and processing time, it takes a long time for the user to wait until the start of the preview playback, which hinders the rapid creation of the free-viewpoint image. There is a risk of coming.
Therefore, in this example, when the observation image from the viewpoint is preview-reproduced as in the case of FIG. 62, the actual three-dimensional image generated based on the image data V1 to V16 (that is, the captured image in the real space) is displayed as the display image. Instead of the model, the image obtained by rendering the above-mentioned virtual three-dimensional model (virtual 3D model in real space) is displayed.
As a result, the processing time required for previewing the observation image from the viewpoint can be shortened, and the work of creating the free viewpoint image can be executed quickly.
　ここで、先に触れたように、カメラワークウインドウ４３では、選択されているカメラワーク、又は選択可能な複数のカメラワークの表示を行うことが可能とされる。
　図５８から図６２では、選択されているカメラワークのみについてのカメラワーク情報が表示された場合を例示したが、カメラワークウインドウ４３では、カメラワークリスト表示部４４に表示された複数のカメラワーク全てのカメラワーク情報を表示することも可能とされる。
　このようにカメラワークウインドウ４３に表示するカメラワーク数の切り替えは、表示パス制限ボタンＢ４１、制限解除ボタンＢ４２により指示することができる。表示パス制限ボタンＢ４１は、選択されているカメラワークのみの表示を指示するボタンであり、制限解除ボタンＢ４２は、このように選択されているカメラワークのみの表示に制限された状態を解除するためのボタンであり、カメラワークリスト表示部４４に表示された複数のカメラワーク全てのカメラワーク情報を表示することの指示ボタンとして機能する。 Here, as mentioned above, in the camera work window 43, it is possible to display the selected camera work or a plurality of selectable camera works.
In FIGS. 58 to 62, the case where the camera work information about only the selected camera work is displayed is illustrated, but in the camera work window 43, all the plurality of camera works displayed on the camera work list display unit 44 are displayed. It is also possible to display the camera work information of.
In this way, switching of the number of camera works to be displayed in the camera work window 43 can be instructed by the display path restriction button B41 and the restriction release button B42. The display path restriction button B41 is a button for instructing the display of only the selected camera work, and the restriction release button B42 is for releasing the state restricted to the display of only the selected camera work. Button, and functions as an instruction button for displaying the camera work information of all the plurality of camera works displayed on the camera work list display unit 44.
　続いて、カメラワークリスト表示部４４について説明する。
　カメラワークリスト表示部４４には、自由視点画像の生成に用いることが可能な候補としてのカメラワークが表示される（例えば図６３等を参照）。カメラワークリスト表示部４４に表示されるカメラワークの情報としては、カメラワークのＩＤ、ＩｎカメラやＯｕｔカメラの識別情報、及びタグ（Ｔａｇ）情報等がある。
　また、本例において、カメラワークリスト表示部４４では、カメラワークごとに移動軌跡情報Ｍｍのサムネイル画像が表示される。このようなサムネイル画像を表示することで、カメラワークリスト上においても各カメラワークがどのような視点移動軌跡を有するものであるかをユーザに確認させることができる。 Subsequently, the camera work list display unit 44 will be described.
The camera work list display unit 44 displays camera work as a candidate that can be used to generate a free-viewpoint image (see, for example, FIG. 63). The camera work information displayed on the camera work list display unit 44 includes a camera work ID, identification information of an In camera or an Out camera, tag information, and the like.
Further, in this example, the camera work list display unit 44 displays a thumbnail image of the movement locus information Mm for each camera work. By displaying such a thumbnail image, the user can be made to confirm what kind of viewpoint movement locus each camera work has on the camera work list.
　ここで、上記のタグ情報は、前述した作成操作画面Ｇｇを通じてカメラワークを作成する際に、作成したカメラワークごとに付すことが可能とされた情報であり、本例では、テキストによる情報とされる。カメラワークに対するタグ情報の設定は、例えば、作成操作画面Ｇｇのカメラワークリスト表示部５２において、カメラワークのエントリに設けられた「Ｔａｇ」の欄（例えば、図１９等を参照）に対する情報入力により行うことができる。
　なお、以下、このタグ情報については「タグ情報Ｉ１」と表記する。 Here, the above tag information is information that can be attached to each created camera work when creating a camera work through the above-mentioned creation operation screen Gg, and in this example, it is information in text. NS. The tag information for the camera work is set, for example, by inputting information to the "Tag" field (see, for example, FIG. 19 and the like) provided in the entry of the camera work on the camera work list display unit 52 of the creation operation screen Gg. It can be carried out.
Hereinafter, this tag information will be referred to as "tag information I1".
　カメラワークリスト表示部４４に対しては、リスト表示の対象とするカメラワーク、つまりは、カメラワークリスト表示部４４に表示するカメラワークをフィルタリングするためのフィルタリング操作部４８が設けられている。
　このフィルタリング操作部４８を用いたカメラワークのフィルタリングに係る機能について、図６３から図６５を参照して説明する。 The camera work list display unit 44 is provided with a filtering operation unit 48 for filtering the camera work to be displayed in the list, that is, the camera work displayed on the camera work list display unit 44.
The function related to the filtering of the camera work using the filtering operation unit 48 will be described with reference to FIGS. 63 to 65.
　先ず、フィルタリングを行うにあたっては、図６３に示すように、ユーザはフィルタリング操作部４８におけるプルダウンボタンＢ４３を操作する。すると、図中に示すように、プルダウンリスト４８ａが表示される。このプルダウンリスト４８ａには、タグ情報Ｉ１の一覧が表示される。プルダウンリスト４８ａに表示されるタグ情報Ｉ１は、候補としての各カメラワークに設定されたタグ情報Ｉ１である。すなわち、図中で例示するように候補としてのカメラワークとして「CW,Cam9」や「CW,Right」等のタグ情報Ｉ１が設定されたものがある場合、プルダウンリスト４８ａには、これら「CW,Cam9」や「CW,Right」等のタグ情報Ｉ１が表示される。 First, in performing filtering, as shown in FIG. 63, the user operates the pull-down button B43 in the filtering operation unit 48. Then, as shown in the figure, the pull-down list 48a is displayed. A list of tag information I1 is displayed in the pull-down list 48a. The tag information I1 displayed in the pull-down list 48a is the tag information I1 set for each camera work as a candidate. That is, when there is a camera work as a candidate in which tag information I1 such as "CW, Cam9" or "CW, Right" is set as illustrated in the figure, these "CW," are displayed in the pull-down list 48a. Tag information I1 such as "Cam9" and "CW, Right" is displayed.
　ユーザは、プルダウンリスト４８ａに表示されたタグ情報Ｉ１を指定する操作（例えばクリック操作等）を行うことで、該タグ情報Ｉ１が設定されたカメラワークのみをカメラワークリスト表示部４４に表示することの指示を行うことができる。
　なお、この点からも理解されるように、プルダウンリスト４８ａにおける各タグ情報Ｉ１の表示部分は、カメラワーク情報をフィルタリング表示する上でのフィルタリング条件を示したフィルタリング条件情報に該当する。 By performing an operation (for example, a click operation) for designating the tag information I1 displayed in the pull-down list 48a, the user displays only the camera work in which the tag information I1 is set on the camera work list display unit 44. Can be instructed.
As can be understood from this point as well, the display portion of each tag information I1 in the pull-down list 48a corresponds to the filtering condition information indicating the filtering conditions for filtering and displaying the camera work information.
　図６４では、タグ情報Ｉ１として「CW,Right」が指定された場合のカメラワーク指定画面Ｇｓの様子を例示している。この場合、カメラワークリスト表示部４４には、タグ情報Ｉ１として「CW,Right」が設定されたカメラワークのみが表示される。
　この場合、「CW,Right」が設定されたカメラワークは一つのみであったため、カメラワークウインドウ４３では、この「CW,Right」が設定されたカメラワークの情報が表示される。なお、「CW,Right」が設定されたカメラワークは、先の図６１のカメラワークウインドウ４３において示した二つのターゲットマークＭｔにより位置が示されたターゲットＴｇのうち、右側のターゲットマークＭｔで位置が示されたターゲットＴｇの設定されたカメラワークとなる。そのため、この場合のカメラワークウインドウ４３では、カメラワークの情報として一つのターゲットマークＭｔのみが表示されている。
　なお、指定されたタグ情報Ｉ１が設定されたカメラワークが複数であった場合には、例えばリスト上で先頭位置等の所定位置に表示されたカメラワークの情報をカメラワークウインドウ４３に表示することが考えられる。 FIG. 64 illustrates the state of the camera work designation screen Gs when "CW, Right" is designated as the tag information I1. In this case, only the camera work in which "CW, Right" is set as the tag information I1 is displayed on the camera work list display unit 44.
In this case, since there was only one camera work in which "CW, Right" was set, the information of the camera work in which "CW, Right" was set is displayed in the camera work window 43. The camera work in which "CW, Right" is set is positioned at the target mark Mt on the right side of the target Tg whose position is indicated by the two target marks Mt shown in the camera work window 43 of FIG. 61 above. Is the set camera work of the target Tg shown in. Therefore, in the camera work window 43 in this case, only one target mark Mt is displayed as the camera work information.
When there are a plurality of camera works for which the specified tag information I1 is set, the camera work information displayed at a predetermined position such as the head position on the list is displayed in the camera work window 43. Can be considered.
　上記のようなタグ情報Ｉ１に基づくカメラワークのフィルタリングとすることで、タグ情報Ｉ１として設定する情報内容次第で任意の基準によるフィルタリングを実現することができる。例えば、タグ情報Ｉ１としてチームの情報（例えばチームＡやチームＢ等）を設定しておけば、カメラワークがチームＡのシュートシーンを対象としたものであるか、チームＢのシュートシーンを対象としたものであるかといった基準によるフィルタリングを実現できる。或いは、タグ情報Ｉ１として視点の移動方向を示す情報（例えば、右回り、左回り等）を設定しておくことで、視点の移動方向を基準としたフィルタリングを実現できる。
　また、ターゲットＴｇとなる被写体に最も近い視野等、注目の視野に一番近いカメラをタグ情報Ｉ１として設定しておくことで、注目の視野を基準としたカメラワークのフィルタリングを実現することができる。 By filtering the camera work based on the tag information I1 as described above, it is possible to realize filtering according to an arbitrary standard depending on the information content set as the tag information I1. For example, if team information (for example, team A, team B, etc.) is set as tag information I1, the camera work targets the shooting scene of team A, or the shooting scene of team B is targeted. It is possible to realize filtering based on criteria such as whether or not it is a product. Alternatively, by setting information indicating the moving direction of the viewpoint (for example, clockwise, counterclockwise, etc.) as the tag information I1, filtering based on the moving direction of the viewpoint can be realized.
Further, by setting the camera closest to the field of view of interest, such as the field of view closest to the subject as the target Tg, as tag information I1, it is possible to realize filtering of camera work based on the field of view of interest. ..
　フィルタリング操作部４８において、リセットボタンＢ４４は、フィルタリングのリセットを指示するボタンとなる。リセットボタンＢ４４が操作された場合は、図６４から図６５への画面遷移として例示するように、カメラワークリスト表示部４４におけるカメラワークのフィルタリング表示状態が解除され、自由視点画像の生成に用いる候補としてのカメラワークが一覧表示された状態となる。 In the filtering operation unit 48, the reset button B44 is a button for instructing the reset of the filtering. When the reset button B44 is operated, as illustrated as a screen transition from FIG. 64 to FIG. 65, the filtering display state of the camera work in the camera work list display unit 44 is canceled, and a candidate used for generating a free viewpoint image is released. The camera work as is displayed in a list.
　ここで、上記ではカメラワークのフィルタリングに関して、タグ情報Ｉ１に基づくフィルタリングを例示したが、カメラワークのフィルタリングは、カメラワークの情報に含まれるＩｎカメラやＯｕｔカメラの情報に基づいて行うことも可能である。 Here, regarding the filtering of the camera work, the filtering based on the tag information I1 has been illustrated, but the filtering of the camera work can also be performed based on the information of the In camera and the Out camera included in the information of the camera work. be.
　また、上記ではタグ情報Ｉ１等のフィルタリング条件を示す情報をプルダウンリスト４８ａに表示する例を挙げたが、フィルタリング条件を示す情報（フィルタリング条件情報）は、図６６に例示するようにボタンとして表示することもできる。
　このとき、ボタンとして表示する情報については、過去に自由視点画像の生成に用いられたカメラワークの履歴情報に基づいて定めることができる。例えば、過去における使用回数の多かった上位所定位のカメラワークのタグ情報Ｉ１をボタンとして表示するができる。
　図６６では、使用回数の多かった上位所定位のカメラワークが、「goal mouth」や「Left」「Right」等といったタグ情報Ｉ１の付されたカメラワークであった場合に対応したボタン配置を例示している。 Further, in the above example, the information indicating the filtering condition such as the tag information I1 is displayed in the pull-down list 48a, but the information indicating the filtering condition (filtering condition information) is displayed as a button as illustrated in FIG. You can also do it.
At this time, the information to be displayed as a button can be determined based on the history information of the camera work used for generating the free viewpoint image in the past. For example, the tag information I1 of the upper predetermined camera work that has been used frequently in the past can be displayed as a button.
FIG. 66 exemplifies the button arrangement corresponding to the case where the upper predetermined camera work that has been used frequently is the camera work with the tag information I1 such as “goal mouth”, “Left”, “Right”, and the like. doing.
　また、フィルタリング条件情報を表示したボタンについては、ユーザによるカスタマイズを可能としてもよい。
　図６７は、その場合のボタン表示例を示している。この場合、各ボタンの表示情報については、ユーザが任意の情報を設定可能とされる。画像生成処理部３２は、ユーザによりボタンの表示情報が設定された場合は、該設定された情報をカメラワークのフィルタリング条件を示す情報として管理する。例えば、図中で例示する「TeamA」「Left」「Right」の情報をカメラワークのフィルタリング条件を示す情報として管理する。ボタンが操作された場合、画像生成処理部３２（具体的には表示処理部３２ａ）は、そのボタンに対応して管理している情報と一致するタグ情報Ｉ１が設定されたカメラワークをカメラワークリスト表示部４４に表示する処理を行う。 Further, the button displaying the filtering condition information may be customized by the user.
FIG. 67 shows an example of button display in that case. In this case, the user can set arbitrary information about the display information of each button. When the display information of the button is set by the user, the image generation processing unit 32 manages the set information as information indicating the filtering condition of the camera work. For example, the information of "Team A", "Left", and "Right" illustrated in the figure is managed as information indicating the filtering condition of the camera work. When the button is operated, the image generation processing unit 32 (specifically, the display processing unit 32a) sets the camera work in which the tag information I1 that matches the information managed corresponding to the button is set. The process of displaying on the list display unit 44 is performed.
　さらに、フィルタリング条件を示す情報については、ユーザによる入力キーワード情報として受け付けるようにすることもできる。
　その場合には、例えば図６６や図６７で例示するようなキーワード入力部４８ｂをフィルタリング操作部４８に設ける。この場合、表示処理部３２ａは、キーワード入力部４８ｂに対するキーワード入力が行われたことに応じて、入力キーワード情報と一致するタグ情報Ｉ１が設定されたカメラワークをカメラワークリスト表示部４４に表示する処理を行う。 Further, the information indicating the filtering condition can be accepted as the input keyword information by the user.
In that case, for example, the keyword input unit 48b as illustrated in FIGS. 66 and 67 is provided in the filtering operation unit 48. In this case, the display processing unit 32a displays the camera work in which the tag information I1 matching the input keyword information is set on the camera work list display unit 44 in response to the keyword input to the keyword input unit 48b. Perform processing.
　なお、上記では、自由視点画像の生成に用いるカメラワークの指定が、カメラワークリスト表示部４４に表示されたカメラワークの指定として行われる例としたが、該カメラワークの指定は、カメラワークウインドウ４３に表示されたカメラワークの指定として行われるようにすることもできる。 In the above, the camera work used to generate the free viewpoint image is specified as the camera work displayed on the camera work list display unit 44, but the camera work is specified in the camera work window. It is also possible to make it as a designation of the camera work displayed on 43.
　ここで、本実施形態では、カメラワークウインドウ４３に表示するカメラワークの情報について、視点の移動速度を視覚化した情報の表示を行う。
　図６８、図６９は、視点の移動速度の視覚化情報の表示例を示している。なお、これら図６８、図６９では、カメラワークウインドウ４３において前述したＹ軸上視点でのカメラワーク情報の観察画像を表示した例を示している。
　本例では、視点の移動速度を視覚化した情報として、視点の移動速度が低下する期間を示す情報を表示する。図６８では、視点の移動速度が低下する区間に位置されたカメラを示すカメラ位置マークＭｃを、他のカメラ位置マークＭｃとは異なる表示態様で表示する例を示している。例えば、該当するカメラ位置マークＭｃについて、他のカメラ位置マークＭｃとは異なる色や大きさで表示することが考えられる。 Here, in the present embodiment, the information of the camera work to be displayed in the camera work window 43 is displayed by visualizing the moving speed of the viewpoint.
68 and 69 show an example of displaying visualization information of the moving speed of the viewpoint. Note that FIGS. 68 and 69 show an example in which the observation image of the camera work information from the above-mentioned Y-axis viewpoint is displayed in the camera work window 43.
In this example, as information that visualizes the moving speed of the viewpoint, information indicating the period during which the moving speed of the viewpoint decreases is displayed. FIG. 68 shows an example in which the camera position mark Mc indicating the camera located in the section where the moving speed of the viewpoint decreases is displayed in a display mode different from that of other camera position mark Mc. For example, it is conceivable that the corresponding camera position mark Mc is displayed in a color and size different from those of other camera position marks Mc.
　図６９では、視点の移動速度が低下する期間について、移動軌跡情報Ｍｍにおける該当区間の表示態様を他区間とは異なるようにした例を示している。例えば、図中に例示するように、該当区間の移動軌跡を点線とし、他区間の移動軌跡を実線により表示することが考えられる。或いは、該当区間と他区間とで移動軌跡の色、太さ、線形状（例えば直線と波線等）等を異ならせて表示することも考えられる。 FIG. 69 shows an example in which the display mode of the corresponding section in the movement locus information Mm is different from that of other sections for the period during which the movement speed of the viewpoint decreases. For example, as illustrated in the figure, it is conceivable to display the movement locus of the corresponding section as a dotted line and the movement locus of another section as a solid line. Alternatively, it is conceivable to display the movement locus with different colors, thicknesses, line shapes (for example, straight lines and wavy lines, etc.) between the relevant section and other sections.
　また、図示による説明は省略するが、視点の移動速度については、移動軌跡を点線で示す場合において、点の密度により表現することもできる。例えば、移動速度が速いほど点の密度を高くする表示を行う等が考えられる。 Although the explanation by illustration is omitted, the moving speed of the viewpoint can be expressed by the density of points when the moving locus is indicated by the dotted line. For example, it is conceivable to display a display in which the density of points increases as the moving speed increases.
　なお、図６８、図６９ではＹ軸上視点からの観察画像を例示したが、Ｘ軸上視点やＺ軸上視点、Ｐｅ視点（任意視点）とした場合も同様の表示が行われる。 Although the observation images from the Y-axis viewpoint are illustrated in FIGS. 68 and 69, the same display is performed when the X-axis viewpoint, the Z-axis viewpoint, and the Pe viewpoint (arbitrary viewpoint) are used.
　また、本実施形態では、カメラワークウインドウ４３に表示されたカメラワークの情報に関して、ターゲットマークＭｔの位置の変更操作に応じてカメラワーク情報におけるターゲットＴｇの位置の情報を更新する処理を行う。
　この処理は、図５に示したカメラワーク編集処理部３２ｂの処理となる。 Further, in the present embodiment, regarding the camera work information displayed in the camera work window 43, a process of updating the information of the position of the target Tg in the camera work information is performed in response to the operation of changing the position of the target mark Mt.
This process is the process of the camera work editing processing unit 32b shown in FIG.
　図７０、図７１を参照して、このようなカメラワーク編集処理部３２ｂによるターゲット位置の編集処理について説明する。
　図７０は、カメラワークウインドウ４３に表示されたカメラワーク情報を例示している。
　カメラワークウインドウ４３において、図示のようにターゲットマークＭｔの位置を変更する操作が行われたとする。ターゲットマークＭｔの位置の変更操作は、例えばターゲットマークＭｔのドラッグ&ドロップ操作とすることが考えられる。
　ここで、このような変更操作による変更後のターゲットＴｇの位置を「位置Ｐｔａ」とし、変更前のターゲットＴｇの位置を「Ｐｔｂ」と表記する。 With reference to FIGS. 70 and 71, the editing process of the target position by the camera work editing processing unit 32b will be described.
FIG. 70 illustrates the camera work information displayed in the camera work window 43.
It is assumed that the operation of changing the position of the target mark Mt is performed in the camera work window 43 as shown in the figure. The operation of changing the position of the target mark Mt may be, for example, a drag-and-drop operation of the target mark Mt.
Here, the position of the target Tg after the change by such a change operation is referred to as "position Pta", and the position of the target Tg before the change is referred to as "Ptb".
　上記のようなターゲットマークＭｔの位置の変更操作が行われたことに応じ、カメラワーク編集処理部３２ｂは、カメラワークウインドウ４３に表示されているカメラワーク情報について、ターゲットＴｇの位置の情報を位置Ｐｔｂから位置Ｐｔａに更新する処理を行う。 In response to the operation of changing the position of the target mark Mt as described above, the camera work editing processing unit 32b positions the position information of the target Tg with respect to the camera work information displayed in the camera work window 43. The process of updating from Ptb to the position Pta is performed.
　このようにカメラワーク情報におけるターゲットＴｇの位置情報を更新することで、当該カメラワーク情報を用いた自由視点画像の生成処理においては、視点移動軌跡上の各位置からの視線方向Ｄｇが更新後のターゲットＴｇの位置を向くように、自由視点画像の生成が行われる。図７１では、更新後の位置である位置Ｐｔａに視点移動軌跡上の各位置からの視線方向Ｄｇを向けた場合における視野範囲Ｒｆの変化のイメージを示している。 By updating the position information of the target Tg in the camera work information in this way, in the process of generating the free viewpoint image using the camera work information, the line-of-sight direction Dg from each position on the viewpoint movement locus is updated. The free viewpoint image is generated so as to face the position of the target Tg. FIG. 71 shows an image of a change in the visual field range Rf when the line-of-sight direction Dg from each position on the viewpoint movement locus is directed to the updated position Pta.
　上記のようにカメラワーク指定画面Ｇｓ上での操作に応じてカメラワーク情報の編集を可能としたことで、自由視点画像の生成に用いるカメラワークを指定する段階でカメラワーク情報の編集を行いたい場合に、カメラワーク情報を生成するためのソフトウエアを立ち上げる必要がなくなる。
　従って、カメラワーク情報の編集を要する場合であっても自由視点画像の作成作業を迅速に実行することができる。 By making it possible to edit the camera work information according to the operation on the camera work specification screen Gs as described above, we want to edit the camera work information at the stage of specifying the camera work to be used for generating the free viewpoint image. In that case, it is not necessary to launch the software for generating the camera work information.
Therefore, even when it is necessary to edit the camera work information, the work of creating the free viewpoint image can be executed quickly.
　また、本実施形態では、カメラワーク指定画面Ｇｓにおいて、前述したキャリブレーションが必要なカメラを通知する情報の表示処理を行う。
　具体的に、表示処理部３２ａは、先の図１４で説明したユーティリティサーバ８による各カメラの変動検出（例えば、ステップＳ３３の変動自動検出）の結果に基づき、カメラワークウインドウ４３にカメラ位置マークＭｃが表示されているカメラのうちで、変動が検出されたカメラがあるか否かを判定する。変動が検出されたカメラがある場合、表示処理部３２ａは、カメラワークウインドウ４３において該当するカメラ（つまり変動が検出されたカメラ）を通知する情報の表示処理を行う。
　なお、変動が検出されたカメラは、視野範囲の変化が検出されたカメラと換言できるものである。 Further, in the present embodiment, on the camera work designation screen Gs, the above-mentioned display processing of information for notifying the camera requiring calibration is performed.
Specifically, the display processing unit 32a displays the camera position mark Mc on the camera work window 43 based on the result of the fluctuation detection of each camera (for example, the automatic fluctuation detection in step S33) by the utility server 8 described with reference to FIG. Among the cameras displayed with, it is determined whether or not there is a camera in which fluctuation is detected. When there is a camera in which fluctuation is detected, the display processing unit 32a performs display processing of information for notifying the corresponding camera (that is, the camera in which fluctuation is detected) in the camera work window 43.
The camera in which the fluctuation is detected can be rephrased as the camera in which the change in the visual field range is detected.
　図７２は、変動が検出されたカメラの通知情報の表示例を示している。
　該通知情報の表示については、図示のように該当するカメラのカメラ位置マークＭｃを他のカメラ位置マークＭｃとは異なる表示態様で表示する（この場合も色、大きさ、形状等を異ならせることが考えられる）。また、図中に例示するエクスクラメーションマークのように、該当するカメラ位置マークＭｃの近傍に注意喚起を促すための情報を表示することも考えられる。 FIG. 72 shows a display example of the notification information of the camera in which the fluctuation is detected.
Regarding the display of the notification information, as shown in the figure, the camera position mark Mc of the corresponding camera is displayed in a display mode different from that of the other camera position mark Mc (also in this case, the color, size, shape, etc. are different). Can be considered). It is also conceivable to display information for calling attention in the vicinity of the corresponding camera position mark Mc, such as the exclamation mark illustrated in the figure.
　自由視点画像の生成にあたり、複数のカメラによる撮像画像から三次元情報を正確に生成するためには、各カメラが予め想定した位置や向きを維持していることを要し、何れかのカメラに位置や向きの変化が生じた場合には、三次元情報の生成に用いるパラメータについてのキャリブレーションが必要となる。上記のように視野範囲の変化が検知されたカメラを通知することで、キャリブレーションが必要であるカメラをユーザに通知することが可能となる。
　従って、正確な三次元情報に基づく自由視点画像の生成が行われるように図ることができ、自由視点画像の画質向上を図ることができる。 In generating free-viewpoint images, in order to accurately generate 3D information from images captured by multiple cameras, it is necessary for each camera to maintain the position and orientation assumed in advance, and one of the cameras must be used. When the position or orientation changes, it is necessary to calibrate the parameters used to generate the three-dimensional information. By notifying the camera in which the change in the field of view is detected as described above, it is possible to notify the user of the camera that needs to be calibrated.
Therefore, it is possible to generate a free-viewpoint image based on accurate three-dimensional information, and it is possible to improve the image quality of the free-viewpoint image.
　図７３及び図７４のフローチャートを参照し、上述したカメラワークのフィルタリングに係る処理について説明する。なお、これら図７３及び図７４に示す処理は、表示処理部３２ａの処理として、自由視点画像サーバ２のＣＰＵ７１が実行するものである。 The processing related to the filtering of the camera work described above will be described with reference to the flowcharts of FIGS. 73 and 74. The processes shown in FIGS. 73 and 74 are executed by the CPU 71 of the free-viewpoint image server 2 as the processing of the display processing unit 32a.
　図７３は、先の図６３のようにカメラワーク指定画面Ｇｓ上に表示したタグ情報Ｉ１に基づいてカメラワークのフィルタリングを行う場合に対応した処理を示している。
　先ず、ＣＰＵ７１はステップＳ２０１で、候補としての各カメラワーク情報におけるタグ情報Ｉ１を取得する処理を行う。すなわち、自由視点画像の生成に用いることが可能とされた候補としての各カメラワーク情報を取得する。ここで、候補としてのカメラワーク情報は、自由視点画像サーバ２の内部又は外部の読み出し可能な記憶装置に記憶されている。ステップＳ２０１の処理では、このように記憶された候補としてのカメラワーク情報を取得する。 FIG. 73 shows a process corresponding to the case where the camera work is filtered based on the tag information I1 displayed on the camera work designation screen Gs as in FIG. 63 above.
First, in step S201, the CPU 71 performs a process of acquiring tag information I1 in each camera work information as a candidate. That is, each camera work information as a candidate that can be used for generating a free viewpoint image is acquired. Here, the camera work information as a candidate is stored in a readable storage device inside or outside the free-viewpoint image server 2. In the process of step S201, the camera work information as the candidate stored in this way is acquired.
　ステップＳ２０１に続くステップＳ２０２でＣＰＵ７１は、タグ情報Ｉ１の表示処理を行う。すなわち、図６３のようにプルダウンリスト４８ａに表示する場合には、プルダウンボタンＢ４３の操作に応じて、ステップＳ２０１で取得したカメラワーク情報に含まれるタグ情報Ｉ１を表示する。 In step S202 following step S201, the CPU 71 performs display processing of tag information I1. That is, when displaying on the pull-down list 48a as shown in FIG. 63, the tag information I1 included in the camera work information acquired in step S201 is displayed in response to the operation of the pull-down button B43.
　ステップＳ２０２に続くステップＳ２０３でＣＰＵ７１は、タグ情報Ｉ１の指定操作を待機し、タグ情報Ｉ１の指定操作があった場合は、ステップＳ２０４に進んで指定されたタグ情報Ｉ１が付されたカメラワークをフィルタリングして表示する処理を行う。すなわち、候補としてのカメラワーク情報のうち、指定されたタグ情報Ｉ１を含む（指定されたタグ情報Ｉ１が設定された）カメラワーク情報をカメラワークリスト表示部４４に表示する処理を行う。図６４で例示したように、本例では、表示するカメラワーク情報は、例えばカメラワークの識別情報やＩｎカメラ、Ｏｕｔカメラの情報、タグ情報Ｉ１等とされる。
　ＣＰＵ７１は、ステップＳ２０４の処理を実行したことに応じて図７３に示す一連の処理を終える。 In step S203 following step S202, the CPU 71 waits for the tag information I1 designation operation, and if there is a tag information I1 designation operation, proceeds to step S204 to perform camera work with the designated tag information I1. Performs filtering and display processing. That is, among the candidate camera work information, the camera work information including the designated tag information I1 (where the designated tag information I1 is set) is displayed on the camera work list display unit 44. As illustrated in FIG. 64, in this example, the camera work information to be displayed is, for example, camera work identification information, In camera, Out camera information, tag information I1 and the like.
The CPU 71 ends a series of processes shown in FIG. 73 in response to executing the process of step S204.
　図７４は、入力キーワードに応じたカメラワークのフィルタリグに係る処理を示している。
　図７４において、ＣＰＵ７１はステップＳ２１０で、ユーザからのキーワード入力を待機し、キーワード入力があった場合は、ステップＳ２１１で入力キーワードを含むカメラワークを選択する。すなわち、候補としてのカメラワーク情報のうち、タグ情報Ｉ１に入力キーワードを含むカメラワーク情報を選択する。
　そして、ＣＰＵ７１は続くステップＳ２１２で、選択したカメラワークを表示する処理を実行する。すなわち、選択したカメラワーク情報をカメラワークリスト表示部４４に表示する処理を行う。
　このステップＳ２１２の処理を実行したことに応じ、ＣＰＵ７１は図７４に示す一連の処理を終える。 FIG. 74 shows the processing related to the filter rig of the camera work according to the input keyword.
In FIG. 74, the CPU 71 waits for the keyword input from the user in step S210, and when there is a keyword input, selects the camera work including the input keyword in step S211. That is, among the camera work information as candidates, the camera work information including the input keyword in the tag information I1 is selected.
Then, in the following step S212, the CPU 71 executes a process of displaying the selected camera work. That is, the process of displaying the selected camera work information on the camera work list display unit 44 is performed.
In response to the execution of the process of step S212, the CPU 71 ends a series of processes shown in FIG. 74.
　図７５は、図７２で例示したキャリブレーションを要するカメラの通知に係る処理のフローチャートである。なお、この図７５に示す処理としても、図７３及び図７４の処理と同様、自由視点画像サーバ２のＣＰＵ７１が表示処理部３２ａの処理として実行するものである。 FIG. 75 is a flowchart of a process related to notification of a camera requiring calibration illustrated in FIG. 72. As for the processing shown in FIG. 75, the CPU 71 of the free viewpoint image server 2 executes the processing as the processing of the display processing unit 32a, as in the processing of FIGS. 73 and 74.
　ステップＳ３０１でＣＰＵ７１は、カメラの変動通知を待機する。すなわち、ユーティリティサーバ８が前述した変動自動検出（図１４のステップＳ３３）によりカメラの変動を検出した場合に送信する変動通知を待機する。この変動通知には、変動が検出されたカメラを特定するための情報が含まれる。 In step S301, the CPU 71 waits for a camera fluctuation notification. That is, the utility server 8 waits for the fluctuation notification to be transmitted when the fluctuation of the camera is detected by the above-mentioned automatic fluctuation detection (step S33 in FIG. 14). The fluctuation notification includes information for identifying the camera in which the fluctuation is detected.
　カメラの変動通知があった場合、ＣＰＵ７１はステップＳ３０２において、表示中のカメラか否かを判定する。すなわち、変動通知により通知されたカメラが、カメラワークウインドウ４３においてカメラ位置マークＭｃを表示中のカメラであるか否かを判定する。　表示中のカメラでなければ、ＣＰＵ７１は図７５に示す一連の処理を終える。 When there is a camera fluctuation notification, the CPU 71 determines in step S302 whether or not the camera is being displayed. That is, it is determined whether or not the camera notified by the fluctuation notification is the camera displaying the camera position mark Mc in the camera work window 43. If it is not the camera being displayed, the CPU 71 ends a series of processes shown in FIG. 75.
　一方、表示中のカメラであれば、ＣＰＵ７１はステップＳ３０３に進み、変動通知処理を実行する。すなわち、カメラワークウインドウ４３に表示中の該当するカメラ位置マークＭｃについて、例えば図７２で例示したような表示態様により変動を通知する情報を表示する処理を行う。
　ステップＳ３０３の処理を実行したことに応じ、ＣＰＵ７１は図７５に示す一連の処理を終える。
On the other hand, if the camera is being displayed, the CPU 71 proceeds to step S303 to execute the fluctuation notification process. That is, for the corresponding camera position mark Mc displayed in the camera work window 43, information for notifying the change is displayed in the display mode as illustrated in FIG. 72, for example.
In response to the execution of the process of step S303, the CPU 71 ends a series of processes shown in FIG. 75.
<９．変形例>

　なお、実施形態としては上記により説明した具体例に限定されるものではなく、多様な変形例としての構成を採り得る。
　例えば、上記では、作成操作画面Ｇｇの表示処理やカメラワーク生成のための操作入力受け付けを行う装置と、カメラワーク指定画面Ｇｓの表示処理や自由視点画像生成のための操作入力受け付けを行う装置とが、自由視点画像サーバ２としての共通装置とされる例を挙げたが、これらの装置が別装置とされる形態を採ることもできる。 <9. Modification example>

It should be noted that the embodiment is not limited to the specific examples described above, and configurations as various modified examples can be adopted.
For example, in the above, a device that performs operation input reception for display processing of the creation operation screen Gg and camera work generation, and a device that performs operation input reception for display processing of the camera work designation screen Gs and free viewpoint image generation. However, although the example in which the free viewpoint image server 2 is used as a common device is given, a form in which these devices are used as separate devices can also be adopted.
　また、上記では、カメラワーク指定画面Ｇｓにおけるカメラワークのフィルタリング表示に関して、フィルタリング条件を示したボタン等の***作部に応じてフィルタリングを行う例を挙げたが、例えば、カメラワークウインドウ４３において表示されたターゲットマークＭｔの指定等、ターゲットＴｇの指定に応じてカメラワークのフィルタリング表示を行うこともできる。具体的には、候補としてのカメラワークのうち、指定されたターゲットＴｇが設定されたカメラワークのみがフィルタリング表示されるようにするものである。 Further, in the above, regarding the filtering display of the camera work on the camera work designation screen Gs, an example of filtering according to the operated portion such as a button showing the filtering condition is given, but it is displayed in the camera work window 43, for example. It is also possible to perform filtering display of camera work according to the designation of the target Tg such as the designation of the target mark Mt. Specifically, among the camera works as candidates, only the camera works for which the designated target Tg is set are filtered and displayed.
　また、作成操作画面Ｇｇやカメラワーク指定画面Ｇｓにおいては、視点の移動軌跡上において、例えば実カメラが被写体から遠すぎる等により画質を担保できない範囲（例えば、解像度が所定値以下となる範囲）がある場合には、該範囲を通知する情報を表示することもできる。
Further, on the creation operation screen Gg and the camera work designation screen Gs, there is a range in which the image quality cannot be guaranteed (for example, a range in which the resolution is equal to or less than a predetermined value) on the movement locus of the viewpoint, for example, because the actual camera is too far from the subject. In some cases, information notifying the range can also be displayed.
<１０．実施形態のまとめ>

　上記のように実施形態の第一の情報処理装置は、自由視点画像における少なくとも視点の移動軌跡を示す情報であるカメラワーク情報の作成操作画面（同Ｇｇ）として、カメラワーク情報の少なくとも一部情報を指定する操作入力を受け付ける指定操作受付領域（カメラワークリスト表示部４４、操作パネル部５４等）と、操作入力による指定内容を反映させたカメラワーク情報に基づいた視点の移動軌跡を視覚化して表示するカメラワーク表示領域（カメラワークウインドウ５３）とを含む画面の表示処理を行う表示処理部（同３４ａ）を備えるものである。
　これにより、ユーザは、カメラワークの作成操作画面において、視覚化された視点の移動軌跡を視認しながらカメラワークの作成操作を行うことが可能となる。
　従って、カメラワーク作成作業の効率化を図ることができる。 <10. Summary of embodiments>

As described above, the first information processing apparatus of the embodiment has at least a part of the camera work information as a camera work information creation operation screen (Gg) which is information indicating at least the movement trajectory of the viewpoint in the free viewpoint image. Visualize the designated operation reception area (camera work list display unit 44, operation panel unit 54, etc.) that accepts the operation input to specify, and the movement trajectory of the viewpoint based on the camera work information that reflects the specified content by the operation input. It is provided with a display processing unit (34a) that performs screen display processing including a camera work display area (camera work window 53) to be displayed.
As a result, the user can perform the camera work creation operation while visually recognizing the movement locus of the visualized viewpoint on the camera work creation operation screen.
Therefore, the efficiency of the camera work creation work can be improved.
　また、実施形態の第一の情報処理装置においては、指定操作受付領域は、移動軌跡の始点と終点の指定操作を受け付け可能とされている（図２２から図２５、図５２を参照）。
　これにより、視点の移動軌跡について、固定ではなく任意の始点及び終点を設定可能となる。
　従って、自由視点画像の作成自由度の向上を図ることができる。 Further, in the first information processing apparatus of the embodiment, the designated operation receiving area can receive the designated operation of the start point and the end point of the movement locus (see FIGS. 22 to 25 and 52).
As a result, it is possible to set an arbitrary start point and end point for the movement locus of the viewpoint instead of being fixed.
Therefore, it is possible to improve the degree of freedom in creating a free viewpoint image.
　さらに、実施形態の第一の情報処理装置においては、指定操作受付領域は、視点の経由地点の指定操作を受け付け可能とされている（図２７から図３０、図５３を参照）。
　これにより、視点の移動軌跡として、始点と終点の２点間を結ぶ直線状の軌跡ではなく、指定地点を経由する軌跡を設定することが可能となる。
　従って、自由視点画像の作成自由度の向上を図ることができる。 Further, in the first information processing apparatus of the embodiment, the designated operation receiving area can receive the designated operation of the waypoint of the viewpoint (see FIGS. 27 to 30 and 53).
As a result, it is possible to set a locus that passes through a designated point instead of a linear locus that connects the two points of the start point and the end point as the movement locus of the viewpoint.
Therefore, it is possible to improve the degree of freedom in creating a free viewpoint image.
　さらにまた、実施形態の第一の情報処理装置においては、指定操作受付領域は、経由地点を視点が経由するタイミングの指定操作を受け付け可能とされている（図２７から図３０を参照）。
　これにより、視点の経由地点のみでなく、視点が経由地点を経由するタイミングを設定することが可能となる。
　従って、視点を経由させる位置の設定自由度と共に、視点が経由地点を経由するタイミングについての設定自由度の向上を図ることができ、自由視点画像の作成自由度の向上を図ることができる。 Furthermore, in the first information processing apparatus of the embodiment, the designated operation receiving area is capable of receiving the designated operation of the timing at which the viewpoint passes through the waypoint (see FIGS. 27 to 30).
This makes it possible to set not only the waypoint of the viewpoint but also the timing of the viewpoint passing through the waypoint.
Therefore, it is possible to improve the degree of freedom in setting the timing at which the viewpoint passes through the waypoint as well as the degree of freedom in setting the position through which the viewpoint passes, and it is possible to improve the degree of freedom in creating the free viewpoint image.
　また、実施形態の第一の情報処理装置においては、指定操作受付領域は、移動軌跡の形状の種類の指定操作を受け付け可能とされている（図３２、図３３を参照）。
　これにより、視点の移動軌跡の形状種類を固定ではなく可変とすることが可能となる。
　従って、自由視点画像の作成自由度の向上を図ることができる。
　例えば、移動軌跡の形状種類を曲線形状とすれば、視点が移動しても対象被写体から視点までの距離が大きく変化しないように図ることができる。換言すれば、自由視点画像内における対象被写体の大きさが大きく変化しないように図ることができる。 Further, in the first information processing apparatus of the embodiment, the designated operation receiving area can receive the designated operation of the shape type of the movement locus (see FIGS. 32 and 33).
This makes it possible to change the shape type of the movement locus of the viewpoint instead of fixing it.
Therefore, it is possible to improve the degree of freedom in creating a free viewpoint image.
For example, if the shape type of the movement locus is a curved shape, it is possible to prevent the distance from the target subject to the viewpoint from changing significantly even if the viewpoint moves. In other words, it is possible to prevent the size of the target subject in the free-viewpoint image from changing significantly.
　さらに、実施形態の第一の情報処理装置においては、指定操作受付領域は、視点の移動速度の指定操作を受け付け可能とされている（図３４、図３５を参照）。
　これにより、視点の移動速度を固定ではなく可変とすることが可能となる。
　従って、自由視点画像の作成自由度の向上を図ることができる。 Further, in the first information processing apparatus of the embodiment, the designated operation receiving area can receive the designated operation of the moving speed of the viewpoint (see FIGS. 34 and 35).
This makes it possible to change the moving speed of the viewpoint instead of fixing it.
Therefore, it is possible to improve the degree of freedom in creating a free viewpoint image.
　さらにまた、実施形態の第一の情報処理装置においては、指定操作受付領域は、移動軌跡中において移動速度を変化させる区間の指定操作を受け付け可能とされている（図３４、図３５を参照）。
　これにより、移動軌跡中における視点の移動速度を動的に変化させることが可能となる。
　従って、自由視点画像の作成自由度の向上を図ることができる。 Furthermore, in the first information processing apparatus of the embodiment, the designated operation receiving area can receive the designated operation of the section in which the moving speed is changed in the moving locus (see FIGS. 34 and 35). ..
This makes it possible to dynamically change the moving speed of the viewpoint in the moving locus.
Therefore, it is possible to improve the degree of freedom in creating a free viewpoint image.
　また、実施形態の第一の情報処理装置においては、指定操作受付領域は、視点の移動開始時点から移動終了時点までの期間を示すタイムラインに対する操作入力を受け付け可能とされている（タイムライン操作部５４ａを参照）。
　タイムライン上での入力操作受け付けとすることで、例えばカメラのアイコンをタイムライン上にドラッグ&ドロップすることで経由地点の指定とその経由タイミングの指定とを同時に行うことが可能となったり、タイムライン上でのドラッグ操作による範囲指定により移動軌跡の曲線補間を行うべき区間等、所定効果を与えるべき区間の指定が可能となったりする等、カメラワークに係る各種情報の指定操作の容易化を図ることが可能となる。
　従って、カメラワーク作成作業の効率化を図ることができる。 Further, in the first information processing apparatus of the embodiment, the designated operation receiving area can accept the operation input for the timeline indicating the period from the movement start time to the movement end time of the viewpoint (timeline operation). See part 54a).
By accepting input operations on the timeline, for example, by dragging and dropping the camera icon on the timeline, it is possible to specify the waypoint and the waypoint timing at the same time, or the time. By specifying the range by dragging on the line, it is possible to specify the section that should give a predetermined effect, such as the section where the curve interpolation of the movement trajectory should be performed, and facilitate the operation of specifying various information related to camera work. It becomes possible to plan.
Therefore, the efficiency of the camera work creation work can be improved.
　さらに、実施形態の第一の情報処理装置においては、表示処理部は、カメラワーク表示領域において、視点からの視野範囲を視覚化した情報（視野範囲情報Ｆｖ）を表示する処理を行っている（図２０等を参照）。
　視野範囲が視覚的に示されることで、ユーザによるカメラワークの把握の容易化が図られる。
　従って、操作入力によりカメラワークがどのように変化するかをユーザに容易に把握させることが可能となり、カメラワーク作成作業の効率化を図ることができる。 Further, in the first information processing apparatus of the embodiment, the display processing unit performs a process of displaying information (visual field range information Fv) that visualizes the visual field range from the viewpoint in the camera work display area (the visual field range information Fv). (See FIG. 20 and the like).
By visually showing the field of view, it is easy for the user to grasp the camera work.
Therefore, it is possible for the user to easily grasp how the camera work is changed by the operation input, and the efficiency of the camera work creation work can be improved.
　さらにまた、実施形態の第一の情報処理装置においては、表示処理部は、カメラワーク表示領域において、視点からの視野範囲を図形により表した情報を表示する処理を行っている。
　視野範囲が図形化して示されることで、ユーザがカメラワークの把握を容易に行うことが可能となる。
　従って、操作入力によりカメラワークがどのように変化するかをユーザに容易に把握させることが可能となり、カメラワーク作成作業の効率化を図ることができる。 Furthermore, in the first information processing apparatus of the embodiment, the display processing unit performs a process of displaying information representing the visual field range from the viewpoint in a graphic shape in the camera work display area.
By displaying the field of view in a graphic form, the user can easily grasp the camera work.
Therefore, it is possible for the user to easily grasp how the camera work is changed by the operation input, and the efficiency of the camera work creation work can be improved.
　また、実施形態の第一の情報処理装置においては、表示処理部は、視点から三次元空間を観察した画像を作成操作画面において表示する処理を行っている（プレビューウインドウ５５を参照）。
　これにより、カメラワーク情報に基づき生成される自由視点画像と同様の画像をユーザにプレビュー表示することが可能となり、カメラワークの把握の容易化を図ることが可能となる。
　従って、カメラワーク作成作業の効率化を図ることができる。 Further, in the first information processing apparatus of the embodiment, the display processing unit performs a process of displaying an image of observing the three-dimensional space from the viewpoint on the creation operation screen (see the preview window 55).
As a result, it is possible to preview and display an image similar to the free viewpoint image generated based on the camera work information to the user, and it is possible to facilitate the grasp of the camera work.
Therefore, the efficiency of the camera work creation work can be improved.
　さらに、実施形態の第一の情報処理装置においては、指定操作受付領域は、視点からの視線方向を定めるターゲットの位置の指定操作を受け付け可能とされている（図３８、図３９等を参照）。
　これにより、自由視点画像として、ターゲットに追従する画像を生成可能となる。ターゲットに追従する画像とは、画枠内の所定位置（例えば中央位置等）にターゲットが位置され続けるようにした画像を意味する。
　従って、自由視点画像の作成自由度の向上を図ることができる。 Further, in the first information processing apparatus of the embodiment, the designated operation receiving area can receive the designated operation of the position of the target that determines the line-of-sight direction from the viewpoint (see FIGS. 38, 39, etc.). ..
This makes it possible to generate an image that follows the target as a free viewpoint image. The image that follows the target means an image in which the target is continuously positioned at a predetermined position (for example, the center position) in the image frame.
Therefore, it is possible to improve the degree of freedom in creating a free viewpoint image.
　さらにまた、実施形態の第一の情報処理装置においては、指定操作受付領域は、ターゲットを向く期間の指定操作を受け付け可能とされている（図４１から図５２を参照）。
　ターゲットを向く期間とは、自由視点画像の画枠内の所定位置にターゲットを位置させ続ける期間を意味する。上記のようにターゲットを向く期間の指定操作が可能とされることで、自由視点画像として、視点移動期間内における或る期間ではターゲット位置に追従し他の期間ではターゲット位置に追従しない画像を生成する等といったことが可能となり、ターゲット位置に追従する期間についての設定自由向上が図られる。
　従って、自由視点画像の作成自由度の向上を図ることができる。 Furthermore, in the first information processing apparatus of the embodiment, the designated operation receiving area is capable of receiving designated operations during the period facing the target (see FIGS. 41 to 52).
The period of facing the target means a period of keeping the target positioned at a predetermined position in the image frame of the free viewpoint image. By making it possible to specify the period for facing the target as described above, an image that follows the target position during a certain period within the viewpoint movement period and does not follow the target position during another period is generated as a free viewpoint image. It is possible to improve the freedom of setting for the period of following the target position.
Therefore, it is possible to improve the degree of freedom in creating a free viewpoint image.
　また、実施形態の第一の情報処理装置においては、指定操作受付領域は、ターゲットの位置の指定操作として、複数のターゲットの位置の指定操作を受け付け可能とされている（図５１を参照）。
　これにより、視点移動期間内における或る期間ではターゲットＡに追従し、他の期間ではターゲットＢに追従する自由視点画像を生成する等といったことが可能となり、追従対象とするターゲットについての設定自由向上が図られる。
　従って、自由視点画像の作成自由度の向上を図ることができる。 Further, in the first information processing apparatus of the embodiment, the designated operation receiving area can accept a plurality of target position designation operations as target position designation operations (see FIG. 51).
As a result, it is possible to generate a free viewpoint image that follows the target A in a certain period within the viewpoint movement period and follows the target B in another period, and the setting freedom of the target to be followed is improved. Is planned.
Therefore, it is possible to improve the degree of freedom in creating a free viewpoint image.
　また、実施形態の第一の情報処理方法は、情報処理装置が、自由視点画像における少なくとも視点の移動軌跡を示す情報であるカメラワーク情報の作成操作画面として、カメラワーク情報の少なくとも一部情報を指定する操作入力を受け付ける指定操作受付領域と、操作入力による指定内容を反映させたカメラワーク情報に基づいた視点の移動軌跡を視覚化して表示するカメラワーク表示領域とを含む画面の表示処理を行う情報処理方法である。
　このような第一の情報処理方法によれば、上記した第一の情報処理装置と同様の作用及び効果を得ることができる。 Further, in the first information processing method of the embodiment, the information processing apparatus provides at least a part of the camerawork information as an operation screen for creating camerawork information which is information indicating at least the movement locus of the viewpoint in the free viewpoint image. Performs screen display processing including a designated operation receiving area that accepts specified operation input and a camerawork display area that visualizes and displays the movement trajectory of the viewpoint based on camerawork information that reflects the specified content of the operation input. It is an information processing method.
According to such a first information processing method, the same operation and effect as the above-mentioned first information processing apparatus can be obtained.
　また、実施形態の第二の情報処理装置は、自由視点画像における少なくとも視点の移動軌跡を示す情報であるカメラワーク情報の指定操作を受け付けるカメラワーク指定画面（同Ｇｓ）として、複数のカメラワーク情報のうちユーザの入力情報に応じたカメラワーク情報をフィルタリングして示す画面の表示処理を行う表示処理部（同３２ａ）を備えたものである。
　カメラワーク情報をユーザの入力情報に応じてフィルタリングして表示することで、ユーザが所望するカメラワーク情報を見つけ易くすることが可能となり、カメラワーク情報の指定に要する時間の短縮化を図ることが可能となる。
　従って、自由視点画像の作成作業を迅速に実行することができる。 Further, the second information processing apparatus of the embodiment has a plurality of camera work information as a camera work designation screen (Gs) that accepts a camera work information designation operation which is information indicating at least the movement locus of the viewpoint in the free viewpoint image. Among them, a display processing unit (32a) for performing display processing of a screen showing by filtering camera work information according to user input information is provided.
By filtering and displaying the camera work information according to the input information of the user, it becomes possible to easily find the camera work information desired by the user, and it is possible to shorten the time required to specify the camera work information. It will be possible.
Therefore, the work of creating a free-viewpoint image can be executed quickly.
　さらに、実施形態の第二の情報処理装置においては、表示処理部は、カメラワーク指定画面において、前記入力情報としてキーワードに応じたカメラワーク情報をフィルタリングして表示する処理を行っている（図６６、図６７、図７４等を参照）。
　これにより、ユーザの意思を反映した適切なカメラワーク情報のフィルタリングを行うことが可能となる。
　従って、ユーザが所望するカメラワーク情報をより見つけ易くすることが可能となり、カメラワーク情報の指定に要する時間のさらなる短縮化を図ることができる。 Further, in the second information processing apparatus of the embodiment, the display processing unit performs a process of filtering and displaying the camera work information according to the keyword as the input information on the camera work designation screen (FIG. 66). , FIG. 67, FIG. 74, etc.).
This makes it possible to perform appropriate filtering of camera work information that reflects the user's intention.
Therefore, it becomes possible to make it easier for the user to find the desired camera work information, and it is possible to further shorten the time required to specify the camera work information.
　さらにまた、実施形態の第二の情報処理装置においては、カメラワーク指定画面には、カメラワーク情報のフィルタリング条件を示した***作部が配置され、表示処理部は、***作部の操作に応じて、操作された***作部が示すフィルタリング条件によりカメラワーク情報をフィルタリングして表示する処理を行っている（図６３、図６４、図６６、図６７、図７３を参照）。
　これにより、カメラワーク情報のフィルタリング表示に要する操作をフィルタリング条件情報の選択操作のみとすることが可能となる。
　従って、カメラワーク情報のフィルタリング表示に要するユーザの操作負担軽減を図ることができる。 Furthermore, in the second information processing apparatus of the embodiment, an operated unit indicating filtering conditions for camera work information is arranged on the camera work designation screen, and the display processing unit responds to the operation of the operated unit. The camera work information is filtered and displayed according to the filtering conditions indicated by the operated unit (see FIGS. 63, 64, 66, 67, and 73).
As a result, the operation required for the filtering display of the camera work information can be limited to the selection operation of the filtering condition information.
Therefore, it is possible to reduce the operational burden on the user required for the filtering display of the camera work information.
　また、実施形態の第二の情報処理装置においては、表示処理部は、カメラワーク指定画面において、視点の移動軌跡を視覚化した情報を表示する処理を行っている（図６１等を参照）。
　視点の移動軌跡を視覚化した情報を表示することで、ユーザがどのようなカメラワークであるかをイメージし易くなる。
　従って、自由視点画像の作成に用いるカメラワーク情報を指定するにあたり、ユーザが所望するカメラワーク情報を見つけ易くすることが可能となり、カメラワーク情報の指定に要する時間の短縮化を図ることができる。 Further, in the second information processing apparatus of the embodiment, the display processing unit performs a process of displaying information that visualizes the movement locus of the viewpoint on the camera work designation screen (see FIG. 61 and the like).
By displaying information that visualizes the movement trajectory of the viewpoint, it becomes easier for the user to imagine what kind of camera work it is.
Therefore, when the camera work information used for creating the free-viewpoint image is specified, it is possible to make it easier for the user to find the desired camera work information, and it is possible to shorten the time required for specifying the camera work information.
　さらに、実施形態の第二の情報処理装置においては、表示処理部は、カメラワーク指定画面において、自由視点画像の生成のための撮像を行う複数のカメラの配置位置を示すカメラ配置位置情報を表示する処理を行っている（図６１等を参照）。
　各カメラの配置位置を示す情報が表示されることで、自由視点画像としてどのような画像を生成すべきかをユーザがイメージし易くなる。
　従って、自由視点画像の作成作業を迅速に実行することができる。 Further, in the second information processing apparatus of the embodiment, the display processing unit displays camera placement position information indicating the placement positions of a plurality of cameras that perform imaging for generating a free-viewpoint image on the camera work designation screen. (See FIG. 61 and the like).
By displaying the information indicating the arrangement position of each camera, it becomes easy for the user to imagine what kind of image should be generated as the free viewpoint image.
Therefore, the work of creating a free-viewpoint image can be executed quickly.
　さらにまた、実施形態の第二の情報処理装置においては、表示処理部は、カメラワーク指定画面において、複数のカメラのうち視点の移動始点となるカメラと移動終点となるカメラの各位置を示す始点配置位置情報と終点配置位置情報を表示する処理を行っている（図６１等を参照）。
　これにより、視点の移動がどのカメラ位置から開始しどのカメラ位置で終了するカメラワークであるのかをユーザに把握させることが可能となる。
　従って、自由視点画像の作成に用いるカメラワーク情報を指定するにあたり、ユーザが所望するカメラワーク情報をより見つけ易くすることができる。特に、前述したように自由視点画像に対して前クリップと後クリップとを連結した画像を生成する場合においては、クリップ間の繋がりが自然なものとなるように、視点の移動始点となるカメラを前クリップの撮像カメラと一致させ、視点の移動終点となるカメラを後クリップの撮像カメラと一致させることが望ましいが、上記のように移動始点となるカメラと移動終点となるカメラの各位置を表示することで、前クリップの撮像カメラと後クリップの撮像カメラに応じた適切なカメラワークを指定し易くすることができる。 Furthermore, in the second information processing apparatus of the embodiment, the display processing unit indicates, on the camera work designation screen, the positions of the camera that is the movement start point of the viewpoint and the camera that is the movement end point of the plurality of cameras. Processing is performed to display the placement position information and the end point placement position information (see FIG. 61 and the like).
This makes it possible for the user to grasp from which camera position the movement of the viewpoint starts and ends at which camera position.
Therefore, when specifying the camera work information used for creating the free viewpoint image, it is possible to make it easier for the user to find the desired camera work information. In particular, when generating an image in which the front clip and the rear clip are connected to the free viewpoint image as described above, a camera that serves as a viewpoint movement starting point is used so that the connection between the clips becomes natural. It is desirable to match the image camera of the front clip and the camera that is the end point of movement of the viewpoint with the image camera of the rear clip, but as described above, the positions of the camera that is the start point of movement and the camera that is the end point of movement are displayed. By doing so, it is possible to easily specify appropriate camera work according to the image pickup camera of the front clip and the image pickup camera of the rear clip.
　また、実施形態の第二の情報処理装置においては、表示処理部は、始点配置位置情報及び終点配置位置情報と、複数のカメラのうち移動始点となるカメラ及び移動終点となるカメラ以外の配置位置情報とを異なる態様で表示する処理を行っている。
　これにより、視点の移動がどのカメラ位置から開始しどのカメラ位置で終了するカメラワークであるのかをユーザに直感的に把握させることが可能となる。
　従って、自由視点画像の作成に用いるカメラワーク情報を指定するにあたり、ユーザが所望するカメラワーク情報をより見つけ易くすることができる。 Further, in the second information processing apparatus of the embodiment, the display processing unit has the start point arrangement position information and the end point arrangement position information, and the arrangement positions other than the camera that is the movement start point and the camera that is the movement end point among the plurality of cameras. Processing is performed to display information in a different manner.
This makes it possible for the user to intuitively grasp from which camera position the movement of the viewpoint starts and ends at which camera position.
Therefore, when specifying the camera work information used for creating the free viewpoint image, it is possible to make it easier for the user to find the desired camera work information.
　また、実施形態の第二の情報処理装置においては、表示処理部は、カメラワーク指定画面において、視点の移動速度を視覚化した情報を表示する処理を行っている（図６８、図６９を参照）。
　視点を移動させる期間において、どの期間で視点の移動速度を変化させるかは、自由視点画像の絵作りにおいて重要な要素とされる。
　従って、上記のように視点の移動速度の視覚化情報を表示することで、ユーザが所望するカメラワーク情報をより見つけ易くすることができ、カメラワーク情報の指定に要する時間の短縮化を図ることができる。 Further, in the second information processing apparatus of the embodiment, the display processing unit performs a process of displaying information that visualizes the moving speed of the viewpoint on the camera work designation screen (see FIGS. 68 and 69). ).
In the period in which the viewpoint is moved, the period in which the movement speed of the viewpoint is changed is an important factor in creating a free viewpoint image.
Therefore, by displaying the visualization information of the moving speed of the viewpoint as described above, it is possible to make it easier for the user to find the desired camera work information, and to shorten the time required to specify the camera work information. Can be done.
　さらに、実施形態の第二の情報処理装置においては、表示処理部は、視点の移動速度を視覚化した情報として、移動速度が低下する期間を示す情報を表示する処理を行っている。
　視点を移動させる期間において、どの期間で視点の移動速度を低下させるかは、自由視点画像の絵作りにおいて重要な要素とされる。
　従って、上記のように視点の移動速度が低下する期間を示す情報を表示することで、ユーザが所望するカメラワーク情報をより見つけ易くすることができ、カメラワーク情報の指定に要する時間の短縮化を図ることができる。 Further, in the second information processing apparatus of the embodiment, the display processing unit performs a process of displaying information indicating a period during which the moving speed decreases as information that visualizes the moving speed of the viewpoint.
In the period in which the viewpoint is moved, the period in which the movement speed of the viewpoint is reduced is an important factor in creating a free viewpoint image.
Therefore, by displaying the information indicating the period during which the moving speed of the viewpoint decreases as described above, it is possible to make it easier for the user to find the desired camera work information, and the time required for specifying the camera work information can be shortened. Can be planned.
　さらにまた、実施形態の第二の情報処理装置においては、表示処理部は、カメラワーク指定画面において、視点からの視野範囲を視覚化した情報を表示する処理を行っている（図６１等を参照）。
　視野範囲が視覚的に示されることで、ユーザによるカメラワークの把握の容易化が図られる。
　従って、ユーザが所望するカメラワーク情報を見つけ易くすることが可能となり、カメラワーク情報の指定に要する時間の短縮化を図ることができる。 Furthermore, in the second information processing apparatus of the embodiment, the display processing unit performs a process of displaying information that visualizes the visual field range from the viewpoint on the camera work designation screen (see FIG. 61 and the like). ).
By visually showing the field of view, it is easy for the user to grasp the camera work.
Therefore, it is possible to make it easier for the user to find the desired camera work information, and it is possible to shorten the time required to specify the camera work information.
　また、実施形態の第二の情報処理装置においては、表示処理部は、カメラワーク指定画面において、視点からの視線方向を定めるターゲットを表示する処理を行っている（図６１等を参照）。
　これにより、カメラワークが三次元空間上のどの位置の被写体を対象としたものであるかをユーザに容易に把握させることが可能となる。
　従って、ユーザが所望するカメラワーク情報を見つけ易くすることが可能となり、カメラワーク情報の指定に要する時間の短縮化を図ることができる。 Further, in the second information processing apparatus of the embodiment, the display processing unit performs a process of displaying a target that determines the line-of-sight direction from the viewpoint on the camera work designation screen (see FIG. 61 and the like).
This makes it possible for the user to easily grasp which position of the subject in the three-dimensional space the camera work is intended for.
Therefore, it is possible to make it easier for the user to find the desired camera work information, and it is possible to shorten the time required to specify the camera work information.
　さらに、実施形態の第二の情報処理装置においては、カメラワーク指定画面上におけるターゲットの位置の変更に応じてカメラワーク情報におけるターゲットの位置の情報を更新するカメラワーク編集処理部（同３２ｂ）を備えている（図７０、図７１を参照）。
　これにより、自由視点画像の生成に用いるカメラワーク情報を指定する段階でカメラワーク情報の編集を行いたい場合に、カメラワーク情報を生成するためのソフトウエアを立ち上げる必要がなくなる。
　従って、カメラワーク情報の編集を要する場合であっても自由視点画像の作成作業を迅速に実行することができる。 Further, in the second information processing apparatus of the embodiment, a camera work editing processing unit (32b) that updates the target position information in the camera work information in response to a change in the target position on the camera work designation screen is provided. (See FIGS. 70 and 71).
As a result, when it is desired to edit the camera work information at the stage of specifying the camera work information used for generating the free viewpoint image, it is not necessary to start the software for generating the camera work information.
Therefore, even when it is necessary to edit the camera work information, the work of creating the free viewpoint image can be executed quickly.
　さらにまた、実施形態の第二の情報処理装置においては、表示処理部は、視点から三次元空間を観察した画像をカメラワーク指定画面において表示する処理を行っている（図６２を参照）。
　これにより、カメラワーク情報に基づき生成される自由視点画像と同様の画像をユーザにプレビュー表示することが可能となり、カメラワークの把握の容易化を図ることが可能となる。
　従って、ユーザが所望するカメラワーク情報を見つけ易くすることが可能となり、カメラワーク情報の指定に要する時間の短縮化を図ることができる。 Furthermore, in the second information processing apparatus of the embodiment, the display processing unit performs a process of displaying an image of observing the three-dimensional space from the viewpoint on the camera work designation screen (see FIG. 62).
As a result, it is possible to preview and display an image similar to the free viewpoint image generated based on the camera work information to the user, and it is possible to facilitate the grasp of the camera work.
Therefore, it is possible to make it easier for the user to find the desired camera work information, and it is possible to shorten the time required to specify the camera work information.
　また、実施形態の第二の情報処理装置においては、表示処理部は、視点から三次元空間を観察した画像として、実空間の仮想三次元モデルをレンダリングした画像を表示する処理を行っている（図６２を参照）。
　これにより、視点からの観察画像のプレビュー表示を実現するにあたり、対象とする実空間の撮像画像から生成した三次元モデルを用いたレンダリング処理を行う必要がなくなる。
　従って、視点からの観察画像のプレビュー表示に要する処理時間の短縮化を図ることができ、自由視点画像の作成作業を迅速に実行することができる。 Further, in the second information processing apparatus of the embodiment, the display processing unit performs a process of displaying an image obtained by rendering a virtual three-dimensional model of the real space as an image of observing the three-dimensional space from the viewpoint (). See FIG. 62).
As a result, in order to realize the preview display of the observed image from the viewpoint, it is not necessary to perform the rendering process using the three-dimensional model generated from the captured image in the target real space.
Therefore, the processing time required for the preview display of the observation image from the viewpoint can be shortened, and the work of creating the free viewpoint image can be executed quickly.
　さらに、実施形態の第二の情報処理装置においては、表示処理部は、複数のカメラのうち、視野範囲の変化が検知されたカメラを通知する情報の表示処理を行っている（図７２を参照）。
　自由視点画像の生成にあたり、複数のカメラによる撮像画像から三次元情報を正確に生成するためには、各カメラが予め想定した位置や向きを維持していることを要し、何れかのカメラに位置や向きの変化が生じた場合には、三次元情報の生成に用いるパラメータについてのキャリブレーションが必要となる。上記のように視野範囲の変化が検知されたカメラを通知することで、キャリブレーションが必要であるカメラをユーザに通知することが可能となる。
　従って、正確な三次元情報に基づく自由視点画像の生成が行われるように図ることができ、自由視点画像の画質向上を図ることができる。 Further, in the second information processing apparatus of the embodiment, the display processing unit performs display processing of information for notifying the camera in which the change in the visual field range is detected among the plurality of cameras (see FIG. 72). ).
In generating free-viewpoint images, in order to accurately generate 3D information from images captured by multiple cameras, it is necessary for each camera to maintain the position and orientation assumed in advance, and one of the cameras must be used. When the position or orientation changes, it is necessary to calibrate the parameters used to generate the three-dimensional information. By notifying the camera in which the change in the field of view is detected as described above, it is possible to notify the user of the camera that needs to be calibrated.
Therefore, it is possible to generate a free-viewpoint image based on accurate three-dimensional information, and it is possible to improve the image quality of the free-viewpoint image.
　また、実施形態の第二の情報処理方法は、情報処理装置が、自由視点画像における少なくとも視点の移動軌跡を示す情報であるカメラワーク情報の指定操作を受け付けるカメラワーク指定画面として、複数のカメラワーク情報のうちユーザの入力情報に応じたカメラワーク情報をフィルタリングして示す画面の表示処理を行う情報処理方法である。
　このような第二の情報処理方法によれば、上記した第二の情報処理装置と同様の作用及び効果を得ることができる。 Further, in the second information processing method of the embodiment, a plurality of camera works are used as a camera work designation screen in which the information processing apparatus accepts a camera work information designation operation which is information indicating at least the movement locus of the viewpoint in the free viewpoint image. This is an information processing method that performs screen display processing that filters and shows camera work information according to user input information.
According to such a second information processing method, the same operation and effect as the above-mentioned second information processing apparatus can be obtained.
　ここで、実施形態としては、図５２や図５３等で説明した表示処理部３４ａによる処理を、例えばＣＰＵ、ＤＳＰ（Digital Signal Processor）等、或いはこれらを含むデバイスに実行させるプログラムを考えることができる。
　即ち、実施形態の第一のプログラムは、コンピュータ装置が読み取り可能なプログラムであって、自由視点画像における少なくとも視点の移動軌跡を示す情報であるカメラワーク情報の作成操作画面として、カメラワーク情報の少なくとも一部情報を指定する操作入力を受け付ける指定操作受付領域と、操作入力による指定内容を反映させたカメラワーク情報に基づいた視点の移動軌跡を視覚化して表示するカメラワーク表示領域とを含む画面の表示処理を行う機能をコンピュータ装置に実現させるプログラムである。
　このようなプログラムにより、上述した表示処理部３４ａを情報処理装置７０としての機器において実現できる。 Here, as an embodiment, a program can be considered in which the processing by the display processing unit 34a described with reference to FIGS. 52 and 53 is executed by, for example, a CPU, a DSP (Digital Signal Processor), or a device including these. ..
That is, the first program of the embodiment is a program that can be read by a computer device, and at least as a camera work information creation operation screen which is information indicating at least the movement locus of the viewpoint in the free viewpoint image. A screen that includes a designated operation reception area that accepts operation input that specifies some information, and a camerawork display area that visualizes and displays the movement trajectory of the viewpoint based on the camerawork information that reflects the specified content of the operation input. It is a program that realizes a function to perform display processing on a computer device.
With such a program, the above-mentioned display processing unit 34a can be realized in the device as the information processing device 70.
　また、実施形態としては、図７３や図７４等で説明した表示処理部３２ａによる処理を、例えばＣＰＵ、ＤＳＰ等、或いはこれらを含むデバイスに実行させるプログラムを考えることができる。
　即ち、実施形態の第二のプログラムは、コンピュータ装置が読み取り可能なプログラムであって、自由視点画像における少なくとも視点の移動軌跡を示す情報であるカメラワーク情報の指定操作を受け付けるカメラワーク指定画面として、複数のカメラワーク情報のうちユーザの入力情報に応じたカメラワーク情報をフィルタリングして示す画面の表示処理を行う機能をコンピュータ装置に実行させるプログラムである。
　このようなプログラムにより、上述した表示処理部３２ａを情報処理装置７０としての機器において実現できる。 Further, as an embodiment, a program can be considered in which the processing by the display processing unit 32a described with reference to FIGS. 73 and 74 is executed by, for example, a CPU, a DSP, or a device including these.
That is, the second program of the embodiment is a program that can be read by a computer device, and serves as a camera work designation screen that accepts a camera work information designation operation that is information indicating at least the movement trajectory of the viewpoint in the free viewpoint image. This is a program that causes a computer device to execute a function of filtering and displaying camera work information according to user input information among a plurality of camera work information.
By such a program, the above-mentioned display processing unit 32a can be realized in the device as the information processing device 70.
　これらのプログラムはコンピュータ装置等の機器に内蔵されている記録媒体としてのＨＤＤや、ＣＰＵを有するマイクロコンピュータ内のＲＯＭ等に予め記録しておくことができる。
　あるいはまた、フレキシブルディスク、ＣＤ－ＲＯＭ(Compact Disc Read Only Memory)、ＭＯ(Magneto Optical)ディスク、ＤＶＤ(Digital Versatile Disc)、ブルーレイディスク（Blu-ray Disc（登録商標））、磁気ディスク、半導体メモリ、メモリカードなどのリムーバブル記録媒体に、一時的あるいは永続的に格納（記録）しておくことができる。このようなリムーバブル記録媒体は、いわゆるパッケージソフトウエアとして提供することができる。
　また、このようなプログラムは、リムーバブル記録媒体からパーソナルコンピュータ等にインストールする他、ダウンロードサイトから、ＬＡＮ(Local Area Network)、インターネットなどのネットワークを介してダウンロードすることもできる。 These programs can be recorded in advance in an HDD as a recording medium built in a device such as a computer device, a ROM in a microcomputer having a CPU, or the like.
Alternatively, flexible discs, CD-ROMs (Compact Disc Read Only Memory), MO (Magneto Optical) discs, DVDs (Digital Versatile Discs), Blu-ray discs (Blu-ray Discs (registered trademarks)), magnetic discs, semiconductor memories, It can be temporarily or permanently stored (recorded) on a removable recording medium such as a memory card. Such a removable recording medium can be provided as so-called package software.
In addition to installing such a program from a removable recording medium on a personal computer or the like, it can also be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.
　またこのようなプログラムによれば、実施形態の表示処理部３４ａや表示処理部３２ａの広範な提供に適している。例えばパーソナルコンピュータ、携帯型情報処理装置、携帯電話機、ゲーム機器、ビデオ機器、ＰＤＡ（Personal Digital Assistant）等にプログラムをダウンロードすることで、当該パーソナルコンピュータ等を、本開示の表示処理部３４ａや表示処理部３２ａとしての処理を実現する装置として機能させることができる。 Further, according to such a program, it is suitable for a wide range of provision of the display processing unit 34a and the display processing unit 32a of the embodiment. For example, by downloading a program to a personal computer, a portable information processing device, a mobile phone, a game device, a video device, a PDA (Personal Digital Assistant), or the like, the personal computer or the like can be displayed by the display processing unit 34a or display processing of the present disclosure. It can function as a device that realizes the processing as the unit 32a.
　なお、本明細書に記載された効果はあくまでも例示であって限定されるものではなく、また他の効果があってもよい。
It should be noted that the effects described in the present specification are merely examples and are not limited, and other effects may be obtained.
<１１．本技術>

　なお本技術は以下のような構成も採ることができる。
（１）
　自由視点画像における少なくとも視点の移動軌跡を示す情報であるカメラワーク情報の作成操作画面として、カメラワーク情報の少なくとも一部情報を指定する操作入力を受け付ける指定操作受付領域と、前記操作入力による指定内容を反映させたカメラワーク情報に基づいた前記視点の移動軌跡を視覚化して表示するカメラワーク表示領域とを含む画面の表示処理を行う表示処理部を備える
　情報処理装置。
（２）
　前記指定操作受付領域は、前記移動軌跡の始点と終点の指定操作を受け付け可能とされた
　前記（１）に記載の情報処理装置。
（３）
　前記指定操作受付領域は、前記視点の経由地点の指定操作を受け付け可能とされた
　前記（１）又は（２）に記載の情報処理装置。
（４）
　前記指定操作受付領域は、前記経由地点を前記視点が経由するタイミングの指定操作を受け付け可能とされた
　前記（３）に記載の情報処理装置。
（５）
　前記指定操作受付領域は、前記移動軌跡の形状の種類の指定操作を受け付け可能とされた
　前記（１）から（４）の何れかに記載の情報処理装置。
（６）
　前記指定操作受付領域は、前記視点の移動速度の指定操作を受け付け可能とされた
　前記（１）から（５）の何れかに記載の情報処理装置。
（７）
　前記指定操作受付領域は、前記移動軌跡中において前記移動速度を変化させる区間の指定操作を受け付け可能とされた
　前記（６）に記載の情報処理装置。
（８）
　前記指定操作受付領域は、前記視点の移動開始時点から移動終了時点までの期間を示すタイムラインに対する前記操作入力を受け付け可能とされた
　前記（１）から（７）の何れかに記載の情報処理装置。
（９）
　前記表示処理部は、
　前記カメラワーク表示領域において、前記視点からの視野範囲を視覚化した情報を表示する処理を行う
　前記（１）から（８）の何れかに記載の情報処理装置。
（１０）
　前記表示処理部は、
　前記カメラワーク表示領域において、前記視点からの視野範囲を図形により表した情報を表示する処理を行う
　前記（９）に記載の情報処理装置。
（１１）
　前記表示処理部は、
　前記視点から三次元空間を観察した画像を前記作成操作画面において表示する処理を行う
　前記（１）から（１０）の何れかに記載の情報処理装置。
（１２）
　前記指定操作受付領域は、前記視点からの視線方向を定めるターゲットの位置の指定操作を受け付け可能とされた
　前記（１）から（１１）の何れかに記載の情報処理装置。
（１３）
　前記指定操作受付領域は、前記ターゲットを向く期間の指定操作を受け付け可能とされた
　前記（１２）に記載の情報処理装置。
（１４）
　前記指定操作受付領域は、前記ターゲットの位置の指定操作として、複数の前記ターゲットの位置の指定操作を受け付け可能とされた
　前記（１２）又は（１３）に記載の情報処理装置。
（１５）
　情報処理装置が、
　自由視点画像における少なくとも視点の移動軌跡を示す情報であるカメラワーク情報の作成操作画面として、カメラワーク情報の少なくとも一部情報を指定する操作入力を受け付ける指定操作受付領域と、前記操作入力による指定内容を反映させたカメラワーク情報に基づいた前記視点の移動軌跡を視覚化して表示するカメラワーク表示領域とを含む画面の表示処理を行う
　情報処理方法。
（１６）
　コンピュータ装置が読み取り可能なプログラムであって、
　自由視点画像における少なくとも視点の移動軌跡を示す情報であるカメラワーク情報の作成操作画面として、カメラワーク情報の少なくとも一部情報を指定する操作入力を受け付ける指定操作受付領域と、前記操作入力による指定内容を反映させたカメラワーク情報に基づいた前記視点の移動軌跡を視覚化して表示するカメラワーク表示領域とを含む画面の表示処理を行う機能を前記コンピュータ装置に実現させる
　プログラム。 <11. This technology>

The present technology can also adopt the following configurations.
(1)
As an operation screen for creating camera work information, which is information indicating at least the movement trajectory of the viewpoint in the free viewpoint image, a designated operation reception area for accepting an operation input for specifying at least a part of the camera work information, and a designated content specified by the operation input. An information processing device including a display processing unit that performs screen display processing including a camera work display area that visualizes and displays the movement locus of the viewpoint based on camera work information that reflects the above.
(2)
The information processing device according to (1) above, wherein the designated operation receiving area is capable of receiving designated operations of the start point and the end point of the movement locus.
(3)
The information processing device according to (1) or (2) above, wherein the designated operation receiving area is capable of receiving a designated operation of a waypoint of the viewpoint.
(4)
The information processing device according to (3), wherein the designated operation receiving area can receive a designated operation at a timing when the viewpoint passes through the waypoint.
(5)
The information processing device according to any one of (1) to (4) above, wherein the designated operation receiving area can receive a designated operation of the shape type of the movement locus.
(6)
The information processing device according to any one of (1) to (5) above, wherein the designated operation receiving area is capable of receiving a designated operation of a moving speed of the viewpoint.
(7)
The information processing device according to (6) above, wherein the designated operation receiving area can receive a designated operation of a section for changing the moving speed in the moving locus.
(8)
The information processing according to any one of (1) to (7) above, wherein the designated operation reception area can accept the operation input to the timeline indicating the period from the movement start time to the movement end time of the viewpoint. Device.
(9)
The display processing unit
The information processing device according to any one of (1) to (8) above, which performs a process of displaying information that visualizes a visual field range from the viewpoint in the camera work display area.
(10)
The display processing unit
The information processing apparatus according to (9) above, wherein in the camera work display area, a process of displaying information representing a visual field range from the viewpoint in a graphic shape is performed.
(11)
The display processing unit
The information processing apparatus according to any one of (1) to (10) above, which performs a process of displaying an image of a three-dimensional space observed from the viewpoint on the creation operation screen.
(12)
The information processing device according to any one of (1) to (11) above, wherein the designated operation receiving area can receive a designated operation of a target position that determines a line-of-sight direction from the viewpoint.
(13)
The information processing device according to (12) above, wherein the designated operation receiving area is capable of receiving designated operations during a period facing the target.
(14)
The information processing apparatus according to (12) or (13), wherein the designated operation receiving area can accept a plurality of target position designation operations as target position designation operations.
(15)
Information processing device
As an operation screen for creating camera work information, which is information indicating at least the movement trajectory of the viewpoint in the free viewpoint image, a designated operation reception area for accepting an operation input for specifying at least a part of the camera work information, and a designated content specified by the operation input. An information processing method that performs screen display processing including a camera work display area that visualizes and displays the movement locus of the viewpoint based on camera work information that reflects the above.
(16)
A program that can be read by a computer device
As an operation screen for creating camera work information, which is information indicating at least the movement trajectory of the viewpoint in the free viewpoint image, a designated operation reception area for accepting an operation input for specifying at least a part of the camera work information, and a designated content specified by the operation input. A program that enables the computer device to perform a screen display process including a camera work display area that visualizes and displays the movement locus of the viewpoint based on the camera work information that reflects the above.
２　自由視点画像サーバ
８　ユーティリティサーバ
１０　撮像装置
３４　カメラワーク生成処理部
３４ａ　表示処理部
Ｇｓ　カメラワーク指定画面
Ｇｇ　作成操作画面
５１　プリセットリスト表示部
５２　カメラワークリスト表示部
５３　カメラワークウインドウ
５４　操作パネル部
５５　プレビューウインドウ
７０　情報処理装置
７１　ＣＰＵ
７２　ＲＯＭ
７３　ＲＡＭ
７４　バス
７５　入出力インタフェース
７６　入力部
７７　表示部
７８　音声出力部
７９　記憶部
８０　通信部
８１　リムーバブル記録媒体
８２　ドライブ
５６　速度調整操作部
５７　軌跡形状調整操作部
５４ａ　タイムライン操作部
Ｂ１　カメラボタン
Ｂ２　ターゲットボタン
Ｂ３　３Ｄモデルボタン
Ｂ４　フォルダ参照ボタン
Ｂ５　新規作成ボタン
Ｂ１１　再生ボタン
Ｂ１２　ポーズボタン
Ｂ１３　停止ボタン
Ｂ１７　シークバー
Ｂ１８　曲線補間ボタン
Ｂ１９　速度調整ボタン
Ｂ２０　追加ボタン
Ｂ２２　期間指定バー
Ｔｇ　ターゲット
Ｍｃ　カメラ位置マーク
Ｆｖ　視野範囲情報
Ｍｔ　ターゲットマーク
Ｍｍ　移動軌跡情報
Ｍｖ　経由地点マーク
Ｍｔｎ　ターゲット位置指定マーク
Ｍｔｔ　追加ターゲットマーク
Ｍｅｍ　到達目標タイミングマーク
Ｍｓｔ　ターゲット初期位置マーク
Ｒｆ　視野範囲
Ｄｇ　視線方向
Ｐｔｎ　新規地点 2 Free viewpoint image server 8 Utility server 10 Imaging device 34 Camera work generation processing unit 34a Display processing unit Gs Camera work designation screen Gg Creation operation screen 51 Preset list display unit 52 Camera work list display unit 53 Camera work window 54 Operation panel unit 55 Preview window 70 Information processing device 71 CPU
72 ROM
73 RAM
74 Bus 75 Input / output interface 76 Input unit 77 Display unit 78 Audio output unit 79 Storage unit 80 Communication unit 81 Removable recording medium 82 Drive 56 Speed adjustment operation unit 57 Trajectory shape adjustment operation unit 54a Timeline operation unit B1 Camera button B2 Target button B3 3D model button B4 Folder reference button B5 New button B11 Play button B12 Pause button B13 Stop button B17 Seek bar B18 Curve interpolation button B19 Speed adjustment button B20 Add button B22 Period specification bar Tg Target Mc Camera position mark Fv Field of view range information Mt Target Mark Mm Movement locus information Mv Waypoint mark Mtn Target position designation mark Mtt Additional target mark Mem Achievement target Timing mark Mst Target initial position mark Rf Field of view Dg Line-of-sight direction Ptn New point

Claims (16)

  1. 　自由視点画像における少なくとも視点の移動軌跡を示す情報であるカメラワーク情報の作成操作画面として、カメラワーク情報の少なくとも一部情報を指定する操作入力を受け付ける指定操作受付領域と、前記操作入力による指定内容を反映させたカメラワーク情報に基づいた前記視点の移動軌跡を視覚化して表示するカメラワーク表示領域とを含む画面の表示処理を行う表示処理部を備える
    　情報処理装置。     As an operation screen for creating camera work information, which is information indicating at least the movement trajectory of the viewpoint in the free viewpoint image, a designated operation reception area for accepting an operation input for specifying at least a part of the camera work information, and a designated content specified by the operation input. An information processing device including a display processing unit that performs screen display processing including a camera work display area that visualizes and displays the movement locus of the viewpoint based on camera work information that reflects the above.
  2. 　前記指定操作受付領域は、前記移動軌跡の始点と終点の指定操作を受け付け可能とされた
    　請求項１に記載の情報処理装置。     The information processing device according to claim 1, wherein the designated operation receiving area is capable of receiving designated operations of a start point and an end point of the movement locus.
  3. 　前記指定操作受付領域は、前記視点の経由地点の指定操作を受け付け可能とされた
    　請求項１に記載の情報処理装置。     The information processing device according to claim 1, wherein the designated operation receiving area is capable of receiving a designated operation of a waypoint of the viewpoint.
  4. 　前記指定操作受付領域は、前記経由地点を前記視点が経由するタイミングの指定操作を受け付け可能とされた
    　請求項３に記載の情報処理装置。     The information processing device according to claim 3, wherein the designated operation receiving area is capable of receiving a designated operation at a timing when the viewpoint passes through the waypoint.
  5. 　前記指定操作受付領域は、前記移動軌跡の形状の種類の指定操作を受け付け可能とされた
    　請求項１に記載の情報処理装置。     The information processing device according to claim 1, wherein the designated operation receiving area can receive a designated operation of the shape type of the movement locus.
  6. 　前記指定操作受付領域は、前記視点の移動速度の指定操作を受け付け可能とされた
    　請求項１に記載の情報処理装置。     The information processing device according to claim 1, wherein the designated operation receiving area is capable of receiving a designated operation of the moving speed of the viewpoint.
  7. 　前記指定操作受付領域は、前記移動軌跡中において前記移動速度を変化させる区間の指定操作を受け付け可能とされた
    　請求項６に記載の情報処理装置。     The information processing device according to claim 6, wherein the designated operation receiving area can receive a designated operation of a section for changing the moving speed in the moving locus.
  8. 　前記指定操作受付領域は、前記視点の移動開始時点から移動終了時点までの期間を示すタイムラインに対する前記操作入力を受け付け可能とされた
    　請求項１に記載の情報処理装置。     The information processing device according to claim 1, wherein the designated operation reception area is capable of receiving the operation input for a timeline indicating a period from the movement start time to the movement end time of the viewpoint.
  9. 　前記表示処理部は、
    　前記カメラワーク表示領域において、前記視点からの視野範囲を視覚化した情報を表示する処理を行う
    　請求項１に記載の情報処理装置。     The display processing unit
    The information processing device according to claim 1, wherein in the camera work display area, a process of displaying information that visualizes a visual field range from the viewpoint is performed.
  10. 　前記表示処理部は、
    　前記カメラワーク表示領域において、前記視点からの視野範囲を図形により表した情報を表示する処理を行う
    　請求項９に記載の情報処理装置。     The display processing unit
    The information processing device according to claim 9, wherein in the camera work display area, a process of displaying information representing a visual field range from the viewpoint in a graphic shape is performed.
  11. 　前記表示処理部は、
    　前記視点から三次元空間を観察した画像を前記作成操作画面において表示する処理を行う
    　請求項１に記載の情報処理装置。     The display processing unit
    The information processing apparatus according to claim 1, wherein a process of displaying an image obtained by observing a three-dimensional space from the viewpoint on the creation operation screen is performed.
  12. 　前記指定操作受付領域は、前記視点からの視線方向を定めるターゲットの位置の指定操作を受け付け可能とされた
    　請求項１に記載の情報処理装置。     The information processing device according to claim 1, wherein the designated operation receiving area can receive a designated operation of a target position that determines a line-of-sight direction from the viewpoint.
  13. 　前記指定操作受付領域は、前記ターゲットを向く期間の指定操作を受け付け可能とされた
    　請求項１２に記載の情報処理装置。     The information processing device according to claim 12, wherein the designated operation receiving area is capable of receiving designated operations during a period facing the target.
  14. 　前記指定操作受付領域は、前記ターゲットの位置の指定操作として、複数の前記ターゲットの位置の指定操作を受け付け可能とされた
    　請求項１２に記載の情報処理装置。     The information processing device according to claim 12, wherein the designated operation receiving area can accept a plurality of target position designation operations as target position designation operations.
  15. 　情報処理装置が、
    　自由視点画像における少なくとも視点の移動軌跡を示す情報であるカメラワーク情報の作成操作画面として、カメラワーク情報の少なくとも一部情報を指定する操作入力を受け付ける指定操作受付領域と、前記操作入力による指定内容を反映させたカメラワーク情報に基づいた前記視点の移動軌跡を視覚化して表示するカメラワーク表示領域とを含む画面の表示処理を行う
    　情報処理方法。     Information processing device
    As an operation screen for creating camera work information, which is information indicating at least the movement trajectory of the viewpoint in the free viewpoint image, a designated operation reception area for accepting an operation input for specifying at least a part of the camera work information, and a designated content specified by the operation input. An information processing method that performs screen display processing including a camera work display area that visualizes and displays the movement locus of the viewpoint based on camera work information that reflects the above.
  16. 　コンピュータ装置が読み取り可能なプログラムであって、
    　自由視点画像における少なくとも視点の移動軌跡を示す情報であるカメラワーク情報の作成操作画面として、カメラワーク情報の少なくとも一部情報を指定する操作入力を受け付ける指定操作受付領域と、前記操作入力による指定内容を反映させたカメラワーク情報に基づいた前記視点の移動軌跡を視覚化して表示するカメラワーク表示領域とを含む画面の表示処理を行う機能を前記コンピュータ装置に実現させる
    　プログラム。     A program that can be read by a computer device
    As an operation screen for creating camera work information, which is information indicating at least the movement trajectory of the viewpoint in the free viewpoint image, a designated operation reception area for accepting an operation input for specifying at least a part of the camera work information, and a designated content specified by the operation input. A program that enables the computer device to perform a screen display process including a camera work display area that visualizes and displays the movement locus of the viewpoint based on the camera work information that reflects the above.
PCT/JP2021/005289 2020-03-30 2021-02-12 Information processing device, information processing method, and program WO2021199715A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2020061250 2020-03-30
JP2020-061250 2020-03-30

Publications (1)

Publication Number Publication Date
WO2021199715A1 true WO2021199715A1 (en) 2021-10-07

Family

ID=77929069

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/005289 WO2021199715A1 (en) 2020-03-30 2021-02-12 Information processing device, information processing method, and program

Country Status (1)

Country Link
WO (1) WO2021199715A1 (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011089961A1 (en) * 2010-01-19 2011-07-28 富士通テン株式会社 Image processing apparatus, image processing system, and image processing method
JP2014069629A (en) * 2012-09-28 2014-04-21 Fujitsu Ten Ltd Image processor, and image processing system
WO2018030206A1 (en) * 2016-08-10 2018-02-15 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカ Camerawork generating method and video processing device
JP2018207336A (en) * 2017-06-06 2018-12-27 キヤノン株式会社 Information processing apparatus, information processing system, information processing method, and program
JP2019160318A (en) * 2018-03-13 2019-09-19 キヤノン株式会社 Information processing device, information processing method, and program

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011089961A1 (en) * 2010-01-19 2011-07-28 富士通テン株式会社 Image processing apparatus, image processing system, and image processing method
JP2014069629A (en) * 2012-09-28 2014-04-21 Fujitsu Ten Ltd Image processor, and image processing system
WO2018030206A1 (en) * 2016-08-10 2018-02-15 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカ Camerawork generating method and video processing device
JP2018207336A (en) * 2017-06-06 2018-12-27 キヤノン株式会社 Information processing apparatus, information processing system, information processing method, and program
JP2019160318A (en) * 2018-03-13 2019-09-19 キヤノン株式会社 Information processing device, information processing method, and program

Similar Documents

Publication Publication Date Title
WO2017036329A1 (en) Method and device for playing video content at any position and time
US20140368621A1 (en) Image processing apparatus, image processing method, and computer program product
JP7017175B2 (en) Information processing equipment, information processing method, program
US10205969B2 (en) 360 degree space image reproduction method and system therefor
WO2017119034A1 (en) Image capture system, image capture method, and program
US20170318274A9 (en) Surround video playback
US11368666B2 (en) Information processing apparatus, information processing method, and storage medium
TWI677860B (en) Video playback device and method
GB2539897A (en) Apparatus, method and computer program
JP2013210989A (en) Image processing device, image processing method, and image processing program
JP2019220783A (en) Information processing apparatus, system, information processing method, and program
JP2013214944A (en) Image processing apparatus, image processing method, and image processing program
JP7020024B2 (en) Information processing equipment and programs
WO2013129188A1 (en) Image processing device, image processing method, and image processing program
US20230353717A1 (en) Image processing system, image processing method, and storage medium
WO2021199715A1 (en) Information processing device, information processing method, and program
KR101880803B1 (en) Video playback program, device, and method
WO2021199714A1 (en) Information processing device, information processing method, and program
WO2022107669A1 (en) Information processing device, information processing method, and information processing system
JP2011071813A (en) Three-dimensional animation-content editing program, device, and method
WO2021199735A1 (en) Information processing device, image processing system, and information processing method
WO2022181175A1 (en) Information processing device, information processing method, program, and display system
WO2023281863A1 (en) Information processing device, information processing method, and program
JP2013211821A (en) Image processing device, image processing method, and image processing program
KR101906947B1 (en) Multi-channel play system and method

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21780432

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21780432

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: JP