WO2012035783A1 - 立体映像作成装置および立体映像作成方法 - Google Patents
立体映像作成装置および立体映像作成方法 Download PDFInfo
- Publication number
- WO2012035783A1 WO2012035783A1 PCT/JP2011/005250 JP2011005250W WO2012035783A1 WO 2012035783 A1 WO2012035783 A1 WO 2012035783A1 JP 2011005250 W JP2011005250 W JP 2011005250W WO 2012035783 A1 WO2012035783 A1 WO 2012035783A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- frame
- stereoscopic
- video
- partner
- unit
- Prior art date
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/261—Image signal generators with monoscopic-to-stereoscopic image conversion
- H04N13/264—Image signal generators with monoscopic-to-stereoscopic image conversion using the relative movement of objects in two video frames or fields
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
- G06T7/579—Depth or shape recovery from multiple images from motion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
Definitions
- the present invention relates to a stereoscopic video creation device and a stereoscopic video creation method, and more particularly to a stereoscopic video creation device and a stereoscopic video creation method for creating a 3D video from a 2D video.
- 3D-related technologies have a wide range of applications, ranging from military navigation to industrial inspection and consumer electronics.
- applications and products to which 3D technology is applied are on the market, such as 3D TVs sold by many TV manufacturers and 3D movies being screened in various 3D movie theaters.
- television broadcasting companies that are experimentally broadcasting 3D channels.
- the application of 3D-related technologies increases the opportunities for people to experience 3D in contact with stereoscopic images (3D images) and the like.
- One is a method of promoting the development of new 3D cameras and providing many 3D cameras to the market. However, it takes time to realize this. Furthermore, a burden is imposed on the user such as purchasing a new 3D camera.
- the other is a method of converting 2D video content into 3D video.
- a method of converting existing 2D video content into 3D video content a method of converting 2D video content into a 3D video content at the same time as shooting a 2D video with a normal camera or camcorder.
- This method is preferable in that a suitable 3D experience can be provided to people at low cost compared to the development of a new 3D camera.
- Patent Document 1 discloses a technique that adapts to complexity (low calculation cost) and automatically converts a 2D image (2D video) to a 3D image (3D video).
- the technique disclosed in Patent Document 1 first, frames are classified into flat images and non-flat images.
- the flat image is directly converted into a 3D stereoscopic display format, and the non-flat image is converted based on a depth map estimated in advance. Note that the three-dimensional transformation based on the depth map can deal with more types.
- Patent Document 2 discloses a technique for converting a 2D image signal into a 3D image signal and outputting the converted 3D image signal.
- the technique disclosed in Patent Document 2 first, the motion of each frame is analyzed, and these frames are classified into three types. Specifically, (1) a frame that includes horizontal motion and does not include a scene change, (2) a frame that does not include horizontal motion and a scene change, and (3) a frame that does not include horizontal motion. Classify into types. Next, when a horizontal motion is included and a scene change is not included, a three-dimensional pair is constructed by directly using the target frame and the next frame.
- Non-Patent Document 1 discloses a three-dimensional transformation method based on SFM (structure from motion).
- SFM structure from motion
- camera parameters such as position, rotation, and focal length are estimated by an SFM algorithm.
- a candidate for the left eye image and a corresponding right eye image are selected from the original video sequence based on the estimated camera parameters.
- Patent Document 1 cannot estimate a high-quality depth map in real time because the calculation is too complicated.
- the requirement for conversion to a 3D image that is comfortable to humans is not satisfied.
- the technique disclosed in Patent Document 2 has a problem that it is insufficient for comfortable 3D conditions. Specifically, frames that include only horizontal movement do not occur much in actual video. Therefore, in the technique disclosed in Patent Document 2, in many cases, 2D video is converted to 3D video based on the estimated depth map. The estimation of the depth map is likely to be influenced by noise because it is estimated based on the horizontal boundary. Moreover, it takes time to estimate a high-quality depth map.
- Non-Patent Document 1 has a problem that the performance of the stereo transformation depends on the accuracy of the SFM.
- High-precision SFM is a time-consuming process and is difficult to apply in real time.
- offline SFM is used for online conversion, its feasibility is low.
- the present invention has been made in view of the above-described circumstances, and an object thereof is to provide a stereoscopic video creation apparatus and a stereoscopic video creation method for creating an appropriate and comfortable 3D video from 2D video.
- a stereoscopic video creation device is a stereoscopic video creation device that creates a 3D video from a 2D video, a receiving unit that receives the 2D video, and the 2D video.
- Candidate frames that constitute a stereoscopic image together with the target frame from among the plurality of frames that constitute a plurality of frames in which the size of the area occupied by the common area that is the area of the image common to the target frame is greater than or equal to a predetermined value
- a selection unit that selects as a stereoscopic partner frame candidate, a determination unit that determines a stereoscopic partner frame that is a frame that forms a stereoscopic image together with the target frame, based on a predetermined criterion, from the stereoscopic partner frame candidate,
- a stereoscopic image corresponding to the target frame is formed using the target frame and the stereoscopic partner frame.
- the stereoscopic video creation apparatus of the present invention can generate a stereoscopic video without estimating a time-consuming depth map.
- the present invention is not only realized as an apparatus, but also realized as an integrated circuit including processing means included in such an apparatus, or realized as a method using the processing means constituting the apparatus as a step. It can also be realized as a program for causing a computer to execute.
- the program may be distributed via a recording medium such as a CD-ROM or a communication medium such as the Internet.
- FIG. 1 is a block diagram showing a configuration of a stereoscopic video creation device according to Embodiment 1 of the present invention.
- FIG. 2 is a block diagram showing a detailed configuration of the stabilization unit in the first embodiment of the present invention.
- FIG. 3 is a block diagram showing a detailed configuration of the stereoscopic video creation unit according to Embodiment 1 of the present invention.
- FIG. 4 is a flowchart showing the processing of the stereoscopic video conversion unit in Embodiment 1 of the present invention.
- FIG. 5 is a flowchart showing processing of the stabilization unit in Embodiment 1 of the present invention.
- FIG. 6 is a diagram illustrating an example of a scene to be captured by the handheld video camera according to Embodiment 1 of the present invention.
- FIG. 1 is a block diagram showing a configuration of a stereoscopic video creation device according to Embodiment 1 of the present invention.
- FIG. 2 is a block diagram showing a detailed configuration of the stabilization
- FIG. 7 is a diagram illustrating a 2D image before stabilization that is captured by the hand-held video camera according to Embodiment 1 of the present invention.
- FIG. 8 is a diagram illustrating a stabilized 2D image captured by the handheld video camera according to Embodiment 1 of the present invention.
- FIG. 9 is a diagram showing a common area of a target frame and a corresponding three-dimensional partner frame candidate in Embodiment 1 of the present invention.
- FIG. 10 is a flowchart showing processing of the determination unit according to Embodiment 1 of the present invention.
- FIG. 11 is a flowchart for explaining processing for generating a transformation matrix in the first embodiment of the present invention.
- FIG. 12A is a diagram for explaining the concept of terms in Embodiment 1 of the present invention.
- FIG. 12A is a diagram for explaining the concept of terms in Embodiment 1 of the present invention.
- FIG. 12B is a diagram for explaining the concept of terms in Embodiment 1 of the present invention.
- FIG. 13 is a diagram illustrating a 3D image obtained by stereoscopically converting a stabilized 2D image captured by the video camera according to Embodiment 1 of the present invention.
- FIG. 14 is a block diagram showing the minimum configuration of the stereoscopic video creating apparatus according to the present invention.
- FIG. 15 is a flowchart showing the operation of the stereoscopic image creating apparatus shown in FIG.
- FIG. 16 is a block diagram showing a configuration of the stereoscopic video creation device according to Embodiment 2 of the present invention.
- FIG. 17 is a block diagram showing a configuration of an image apparatus according to Embodiment 3 of the present invention.
- FIG. 1 is a block diagram showing a configuration of a stereoscopic video creation device according to Embodiment 1 of the present invention.
- FIG. 2 is a block diagram showing a detailed configuration of the stabilization unit in the first embodiment of the present invention.
- FIG. 3 is a block diagram showing a detailed configuration of the stereoscopic video creation unit according to Embodiment 1 of the present invention.
- the stereoscopic video creation apparatus 100 shown in FIG. 1 creates a 3D video from a 2D video and outputs the created 3D video.
- the output 3D video is displayed on the external display unit 112 or stored in the external storage / transmission device 114.
- the stereoscopic video creation device 100 includes a receiving unit 101, a stereoscopic video conversion unit 106, a video output unit 108, and an internal buffer 110.
- the receiving unit 101 receives 2D video.
- the receiving unit 101 includes, for example, a storage medium reader 102 and a video decoder 104 as shown in FIG.
- the storage medium reader 102 stores a plurality of frames (image data) constituting a 2D video.
- the video decoder 104 acquires the image data S11 from the storage medium reader 102 and outputs it to the stereoscopic video conversion unit 106.
- the video decoder 104 outputs 2D video data S13 obtained by decoding the image data S11 as necessary to the stereoscopic video conversion unit 106.
- the stereoscopic video conversion unit 106 includes a stabilization unit 116 and a stereoscopic video creation unit 118.
- the stereoscopic video conversion unit 106 stabilizes the 2D video data S13 using the stabilization unit 116 and the stereoscopic video creation unit 118, and then converts the data into 3D video (stereoscopic video).
- the stabilization unit 116 stabilizes the plurality of frames by correcting fluctuations between the plurality of frames from the plurality of frames constituting the 2D video (2D video data S13).
- the shake (shake) between a plurality of frames is, for example, a shake of a video image due to a camera shake when capturing a 2D video image.
- stabilization means, for example, correcting to an image without shaking.
- the stabilization unit 116 includes a detection unit 121 and a calculation unit 122 as shown in FIG.
- the stabilizing unit 116 calculates a projective transformation matrix calculated based on a feature point that is a characteristic point on a corresponding frame between a predetermined frame and a plurality of neighboring frames that are temporally close to the predetermined frame. By using this, the fluctuation between the plurality of frames is corrected.
- the detection unit 121 detects a plurality of feature points that are characteristic points corresponding to a predetermined frame and a neighboring frame that is a frame close to the predetermined frame.
- the calculation unit 122 stabilizes a projection transformation matrix that warps (deforms) a predetermined frame so that a plurality of feature points of the predetermined frame and a plurality of weighted feature points of the corresponding neighboring frames have the same coordinate value. Calculated as a quantization matrix. For example, the calculation unit 122 calculates the weights of a plurality of neighboring frames using a weight function. When the corresponding neighboring frame is the closest frame in time to the predetermined frame, the calculation unit 122 calculates a weight having a value closer to 1 using a weight function. On the other hand, when the corresponding neighboring frame is a frame far in time from the predetermined frame, a weight having a value smaller than 1 is calculated using a weight function.
- the stabilization unit 116 applies the calculated projective transformation matrix to a predetermined frame, and similarly processes all the frames, thereby stabilizing a plurality of frames constituting the 2D video data.
- the stereoscopic video creation unit 118 includes a selection unit 123, a determination unit 124, a stereoscopic pair generation unit 125, and a conversion unit 126. From the stabilized 2D video data S13, 3D Create a video.
- the selection unit 123 selects a plurality of frames, together with the target frame, from the plurality of frames constituting the 2D video, together with the target frame. It selects as a solid partner frame candidate which is a frame candidate constituting the image.
- the selection unit 123 selects a plurality of frames shot in the same scene as the target frame from a plurality of frames constituting the 2D video as a stereoscopic partner frame candidate. Specifically, the selection unit 123 determines that the ratio of the size of the common area in the target frame and the ratio of the size of the common area in the stereoscopic partner frame candidate are each equal to or greater than a predetermined value (for example, 0.9). In this case, it is determined that the stereoscopic partner frame candidate is a frame shot in the same scene as the target frame.
- the common area is calculated based on a feature point that is a characteristic point on the corresponding frame between the target frame and the stereoscopic partner frame candidate.
- the selection unit 123 may select, as a stereoscopic partner frame candidate, a neighboring frame that is temporally close to the target frame in a plurality of frames constituting the 2D video.
- the determining unit 124 determines a stereoscopic partner frame that is a frame that forms a stereoscopic image together with the target frame, based on the first criterion, from the stereoscopic partner frame candidates. In other words, the determination unit 124 determines whether or not the three-dimensional partner frame is satisfied by determining whether or not the first criterion is satisfied using a feature point that is a characteristic point on the frame that can be detected by performing image processing. decide.
- the stereoscopic pair generation unit 125 generates a stereoscopic pair that forms a stereoscopic image corresponding to the target frame using the target frame and the stereoscopic partner frame.
- the first standard is a standard composed of a vertical difference standard, a horizontal parallax standard, a smoothing standard, and a distortion reduction standard.
- the vertical direction difference standard stipulates that a frame whose difference between the common area of the target frame and the vertical position is substantially equal to zero is a stereoscopic partner frame.
- the horizontal parallax standard stipulates that a frame in which a difference (parallax) between a common area of a target frame and a horizontal position is included in a predetermined range is a stereoscopic partner frame.
- the smoothing standard stipulates that a frame in which the displacement of the common area is smooth between three-dimensional pairs that are temporally continuous is a three-dimensional partner frame.
- the distortion reduction standard stipulates that a frame whose area occupied by the common area is not less than a predetermined value together with the target frame is a three-dimensional partner frame.
- the conversion unit 126 generates a conversion matrix for warping the stereoscopic partner frame candidate from the target frame and the stereoscopic partner frame candidate selected by the selection unit 123, and applies the generated conversion matrix to the stereoscopic pair.
- the generated stereo pair is converted.
- the conversion unit 126 generates a conversion matrix based on feature points that are characteristic points on the frame and correspond between the target frame and the three-dimensional partner frame. Specifically, the conversion unit 126 generates the conversion matrix by combining the calculated projective conversion matrix, the oblique conversion matrix, and the translation conversion matrix. More specifically, first, the conversion unit 126 calculates a basic matrix between the target frame and the stereoscopic partner frame candidate. Next, the conversion unit 126 calculates a projective conversion matrix based on the basic matrix so as to minimize the vertical difference between the target frame and the stereoscopic partner frame candidate in the stereoscopic pair.
- the conversion unit 126 calculates the oblique transformation matrix so as to maintain the same orthogonality and aspect ratio as before the transformation of the common region in the stereoscopic partner frame transformed by the projective transformation matrix.
- the conversion unit 126 converts the translation transformation matrix into a target frame and a stereoscopic partner frame whose parallax between the target frame and the stereoscopic partner frame to which the projective transformation matrix and the oblique transformation matrix are applied is earlier than the target frame. To be the same as the parallax between.
- the internal buffer 110 corresponds to the storage unit of the present invention, and stores the solid pair converted by the conversion unit 126. Specifically, the internal buffer stores the stereoscopic pair generated by the stereoscopic video creation unit 118 as intermediate data that is data before being output to the video output unit 108. In other words, the stereoscopic video conversion unit 106 transmits the generated stereoscopic pair data S15 to the video output unit 108 via the internal buffer 110.
- the video output unit 108 corresponds to the output unit of the present invention, and outputs the converted stereoscopic pair stored in the internal buffer 110. In addition, the video output unit 108 outputs a stereo pair corresponding to each of a plurality of frames constituting the 2D video and converted by the conversion unit 126.
- the video output unit 108 adjusts the input stereoscopic pair data S15 to an output video format (stereoscopic pair data S17) according to a preferred output format, and outputs it to the display unit 112. Specifically, the video output unit 108 adjusts the output video format to match the display format of the display unit 112 for display on the display unit 112.
- the output video format includes, but is not limited to, a format for an autostereoscopic device for viewing without using glasses. For example, a gray / color anaglyph viewed using glasses may be used, or an interlaced format may be used. Also, a checkerboard format may be used, or another format for a frame sequential type stereoscopic display device that is viewed using active shutter glasses may be used.
- the video output unit 108 may store / transmit the stereoscopic pair data S17 using the storage / transmission device 114.
- the storage / transmission device 114 is, for example, a flash-based memory card, a hard drive, or an optical drive, but is not limited thereto.
- the storage / transmission device 114 may be, but is not limited to, an HDMI interface, a USB interface, a wireless interface, or a direct-to-printer interface.
- the storage / transmission device 114 may arbitrarily perform lossless or irreversible compression to store / transmit.
- the stereoscopic image creation apparatus 100 is configured.
- the stereoscopic video creation device 100 (or the stereoscopic video conversion unit 106 or the like constituting the stereoscopic video creation device) is usually an IC (integrated circuit), an ASIC (specific application integrated circuit), or an LSI (large scale integrated circuit). Although realized, it may be composed of a plurality of chips or a single chip. Note that each of the stereoscopic video creation device 100 or the stereoscopic video conversion unit 106 and the like is not limited to being realized in the form of an LSI. Depending on the degree of integration, it may be realized by what is also called an IC, a system LSI, a super LSI, or an ultra LSI.
- the stereoscopic video creation device 100 or the like may be realized by being integrated using a dedicated circuit or a general-purpose processor.
- a dedicated circuit there is a specialized microprocessor such as a DSP (digital signal processor) that can be controlled by a program instruction.
- DSP digital signal processor
- the stereoscopic image creating apparatus 100 or the like may use an FPGA (Field Programmable Gate Array) that can be programmed after manufacturing the LSI, or a processor that can reconfigure the connection or configuration of the LSI.
- FPGA Field Programmable Gate Array
- the manufacturing and processing technology will improve and a completely new technology will replace LSI, it may be realized by integration using that technology.
- the stereoscopic video creating apparatus 100 generates a stereoscopic image of a liquid crystal display device that displays an image (video) in time series, a plasma display device, a display device to which a lenticular lens layer is added, or other types of display devices. It may be incorporated in a display device capable of display.
- the stereoscopic video creation device 100 may be mounted on a digital media player device such as a digital video disc player, a Blu-ray disc player, and other types of digital media players. It may be implemented in other types of devices. In either case, the scope of the present application is not limited.
- FIG. 4 is a flowchart showing the operation of the stereoscopic video conversion unit 106 according to Embodiment 1 of the present invention.
- 2D video is input to the stereoscopic video conversion unit 106.
- the stereoscopic video conversion unit 106 receives 2D video data S13 from the reception unit 101.
- the stereoscopic video conversion unit 106 detects corresponding feature points between adjacent frames.
- the method disclosed in the prior art is used for detection (registration / tracking) of the corresponding feature point.
- a known method for tracking features is disclosed in Non-Patent Document 2, for example.
- a known method for registering features is disclosed in Non-Patent Document 3, for example.
- the feature point detection (registration / tracking) method is not limited to the methods disclosed in Non-Patent Document 2 and Non-Patent Document 3.
- the stereoscopic video conversion unit 106 stabilizes the 2D video. Specifically, the stereoscopic video conversion unit 106 performs stabilization conversion for a predetermined frame using a matrix calculated based on feature points corresponding to the target (reference) frame and a plurality of neighboring frames. More specifically, a projective transformation matrix that is calculated based on a feature point that is a characteristic point on a corresponding frame between a predetermined frame and a plurality of neighboring frames that are temporally close to the predetermined frame is used. Thus, the fluctuation between the plurality of frames is corrected. Thereby, the stereoscopic video conversion unit 106 can stabilize the 2D video.
- the stereoscopic video conversion unit 106 selects a stereoscopic partner frame candidate corresponding to the target frame.
- the stereoscopic video conversion unit 106 has a common area that is an area common to the target frame from a plurality of frames constituting the 2D video, and the size of the area occupied by the common area is determined in advance.
- a plurality of frames that are equal to or greater than the value (for example, 0.9) are selected as stereoscopic partner frame candidates that are candidates for frames that form a stereoscopic image together with the target frame.
- the stereoscopic video conversion unit 106 selects a frame including the same scene as the target frame from a plurality of frames constituting the 2D video.
- the stereoscopic video conversion unit 106 determines a stereoscopic partner frame corresponding to the target frame. Specifically, the stereoscopic video conversion unit 106 determines a stereoscopic partner frame, which is a frame that forms a stereoscopic image together with the target frame, from the stereoscopic partner frame candidates based on the first reference.
- the first standard is as described above, and includes the following standard a) to standard d). That is, for the three-dimensional partner frame and the target frame forming the three-dimensional pair, the reference a) the vertical difference is almost zero (vertical difference reference), and the reference b) has sufficient and appropriate horizontal parallax (horizontal parallax reference ), Criterion c) the parallax of the corresponding points with adjacent solid pairs is similar (smoothing criterion), and criterion d) distortion is minimal (distortion reduction criterion).
- the stereoscopic video conversion unit 106 generates a stereoscopic pair. Specifically, the stereoscopic video conversion unit 106 uses the target frame and the stereoscopic partner frame to generate a stereoscopic pair that constitutes a stereoscopic image corresponding to the target frame. That is, the stereoscopic video conversion unit 106 generates a stereoscopic pair using the stabilized target frame and the stereoscopic partner frame determined based on the first reference.
- the stereoscopic video conversion unit 106 saves or displays the generated stereoscopic pair (target stereoscopic pair). For example, the stereoscopic video conversion unit 106 stores the generated target stereoscopic pair in the internal buffer 110.
- the stereoscopic video conversion unit 106 confirms whether or not the generated target frame of the target stereoscopic pair is the last frame among a plurality of frames constituting the 2D video.
- a stereoscopic video (3D video) composed of the stereoscopic pair generated by the stereoscopic video conversion unit 106 is output.
- the stereoscopic video conversion unit 106 converts 2D video to 3D video.
- one of the problems to be solved is camera movement caused by camera shake or other reasons. That is, when a 2D image is captured by a camera that has moved due to camera shake or the like, the 2D image becomes an unstable image including shaking. Therefore, even if a 3D video is created from an unstable 2D video, the 3D video obtained by the creation becomes an unstable 3D video. Unstable 3D video will feel unhealthy and uncomfortable when viewed by a person.
- the unstable 2D image is stabilized by correcting the shake between frames due to camera shake or other reasons.
- the details of the stabilization process (S106) performed by the stabilization unit 116 will be described.
- FIG. 5 is a flowchart showing processing of the stabilization unit 116 in Embodiment 1 of the present invention.
- the stabilization unit 116 receives a 2D video image. Specifically, the stabilization unit 116 receives the 2D video data S13 from the reception unit 101.
- the stabilization unit 116 detects (tracks) corresponding feature points between adjacent frames in a plurality of frames constituting the 2D video (2D video data S13). Specifically, the detection unit 121 detects a plurality of feature points that are characteristic points corresponding to a predetermined frame and a neighboring frame that is a frame close to the predetermined frame in a plurality of frames constituting the 2D video. To detect.
- the stabilization unit 116 calculates a stabilization matrix that warps the predetermined frame based on feature points corresponding to the predetermined frame and a plurality of neighboring frames.
- the calculation unit 122 stabilizes the predetermined frame so that the plurality of feature points of the predetermined frame and the plurality of weighted feature points of the corresponding neighboring frames have the same coordinate value.
- a projective transformation matrix is calculated.
- the stabilization matrix S m is calculated as a matrix that minimizes the objective function of Equation 1.
- S is a 3 ⁇ 3 projective transformation matrix
- S m is an estimated 3 ⁇ 3 stabilization matrix
- P j is a 3 ⁇ K matrix including all feature points (K) of the j-th frame. Each column of these matrices is a three-dimensional homogeneous coordinate of a feature point.
- w j is the weight of the feature point in the j-th frame. When the jth frame is far from the mth target frame, the weight is small. When the jth frame is close to the mth frame (predetermined frame), the weight is increased. The weight is calculated by a Gaussian function shown in Equation 2.
- ⁇ is a Gaussian function of variance
- the stabilization unit 116 applies the calculated stabilization matrix to a predetermined frame. Specifically, the stabilization unit 116 corrects the shake between the predetermined frames by applying the calculated projective transformation matrix to the predetermined frames. In this way, the plurality of frames can be stabilized.
- the stabilization unit 116 confirms whether or not the predetermined frame to which the stabilization matrix is applied is the last frame among a plurality of frames constituting the 2D video.
- the stabilization unit 116 outputs a stabilized 2D video.
- FIG. 6 is a diagram showing an example of a scene (scenery) to be photographed by the handheld video camera according to Embodiment 1 of the present invention.
- FIG. 7 is a diagram illustrating a 2D image before stabilization, which is captured by a handheld video camera.
- FIG. 8 is a diagram illustrating a stabilized 2D image captured by a handheld video camera.
- FIG. 7 shows a scene 502 photographed using a hand-held monocular video camera 542.
- the user shoots the scene 502 as a 2D image while shifting the video camera 542 at a constant speed in the horizontal direction at the same height.
- the trajectory 562 fluctuates in the vertical direction due to hand movement.
- the user has photographed the scene 502 as a 2D image along the trajectory 562 of the video camera 542 that fluctuates up and down due to camera shake. That is, the video camera 542 shoots, for example, a video frame 582 (2D video) corresponding to the video sample point 522 shown in FIG. 7, that is, an unstable 2D video with vertical fluctuation. Therefore, as described above, when a 3D video is created from the 2D video, this fluctuation affects the stability of the 3D video.
- the video sample point 522 is one of the video sample points of the trajectory 562 of the video camera 542, and the video frame 582 is a video frame corresponding to the video sample point 522.
- FIG. 8 shows a state in which a scene 502 is photographed using a hand-held monocular video camera 542 and the photographed 2D image is stabilized.
- a state is shown in which a virtual video camera 544 captures a scene 502 along a virtual trajectory 566 in which the vertical fluctuation of the trajectory 562 is virtually stabilized, thereby generating a stabilized 2D video.
- a virtual video frame 584 corresponding to the virtual video sample point 524 is shown.
- fluctuations due to camera shake are greatly reduced after stabilization.
- the trajectory 562 is more smoothly stabilized and the video sample points are more uniform.
- the stabilized 2D video is greatly reduced in fluctuation.
- the 3D video obtained from the stabilized 2D video is preferable because it feels healthy and comfortable when viewed by a person.
- the stereoscopic video conversion unit 106 has a common area that is an area common to the target frame from a plurality of frames constituting the 2D video, and the size of the area occupied by the common area is a predetermined value.
- the plurality of frames as described above are selected as stereoscopic partner frame candidates.
- the stereoscopic video conversion unit 106 selects a stereoscopic partner frame candidate based on the ratio between the size of the common area and the size of the target frame. More specifically, when the ratio r mn between the m-th frame (target frame) and the n-th frame is larger than a predetermined value called ratio_inumum, the n-th frame is one of the three-dimensional partner frame candidates. Selected.
- C (m) includes all indicators of candidate solid partner frames.
- FIG. 9 is a diagram showing a common area of the target frame and the corresponding three-dimensional partner frame candidate according to Embodiment 1 of the present invention.
- FIG. 9A shows the target frame 602
- FIG. 9B shows the three-dimensional partner frame candidate 604.
- a rectangle 606 and a rectangle 608 indicate the outer shape of a common area which is an area common to both frames.
- the common area is a stereoscopic pair viewing area including the target frame 602 and the corresponding stereoscopic partner frame candidate 604.
- the centroids of a plurality of feature points in the mth frame are calculated.
- centroid of a plurality of feature points (common areas) corresponding to the target frame image is calculated in the nth frame (frame image).
- the centroids of the target frame image and the nth frame image may be calculated simultaneously.
- a vertical difference 636 shown in FIG. 9A is obtained as the difference in the vertical position of the common area between the target frame image and the nth frame image.
- a horizontal difference 638 shown in FIG. 9A is obtained as the horizontal difference between the target frame image and the nth frame image.
- the size of the common area can be derived from the original frame size (the size of the target frame) and the difference between the centroids of corresponding feature points in the target frame image and the nth frame image.
- this method may not be able to derive an accurate common area. That is, a difference may occur between the calculated common area and the actual common area. However, this method is sufficient for use in the present invention.
- FIG. 10 is a flowchart showing processing of the determination unit 124 according to Embodiment 1 of the present invention.
- the stereoscopic video converting unit 106 selects a stereoscopic partner frame for the target frame based on the first criterion including the following criteria a) to d) in S110. To do.
- the following criteria a) to d) correspond to the above-mentioned criteria a) to d).
- the y-direction difference in the common area between the target frame and the three-dimensional partner frame is almost zero (vertical direction difference reference).
- the y direction corresponds to the orthogonal direction of the binocular parallax direction.
- the x-direction difference (parallax) of the common area between the target frame and the stereoscopic partner frame is within a predetermined range (horizontal parallax reference) that is sufficient to obtain a 3D effect and provides a comfortable 3D video.
- the x direction corresponds to the binocular parallax direction.
- the determination unit 124 excludes a stereoscopic partner frame candidate that does not satisfy the vertical difference in the above a) based on a corresponding feature point between the target frame and the stereoscopic partner frame candidate. To do.
- the determination unit 124 uses the variance for evaluating the vertical direction difference. That is, the determination unit 124 calculates the variance of the vertical difference between the target frame and the stereoscopic partner frame candidate. If the calculated vertical difference variance is too large, the stereoscopic partner frame candidate is excluded from the candidate set.
- the determination unit 124 excludes a stereoscopic partner frame candidate that does not satisfy the horizontal parallax of the above b) based on the corresponding feature point between the target frame and the stereoscopic partner frame candidate that is not excluded in S302. To do.
- the determination unit 124 uses variance for evaluation of horizontal parallax. That is, the determination unit 124 calculates the variance of the horizontal parallax between the target frame and the stereoscopic frame candidate, and excludes the stereoscopic partner frame candidate that does not satisfy the horizontal parallax of the above b) based on the calculated variance. This is because if the horizontal parallax is too large, the 3D image created thereafter becomes unpleasant, and conversely if the horizontal parallax is too small, the 3D effect of the 3D video created thereafter becomes low.
- the determination unit 124 determines the filtered three-dimensional partner frame candidate (included in the predetermined range) using Equation 4.
- the determination unit 124 calculates the variance of the horizontal parallax and the variance of the vertical difference using Formula 5 and Formula 6, respectively.
- p i m is the i-th point in the m-th frame.
- Equation 7 [d] x represents the first component of the vector d, and [d] y represents the second component of the vector d.
- the determination unit 124 determines a three-dimensional partner frame from the remaining three-dimensional partner frame candidates based on the smoothing criteria shown in c) above.
- the determination unit 124 uses variance for evaluation of the smoothing criterion. That is, the determination unit 124 calculates the variance of horizontal parallax at corresponding points between two adjacent solid pairs. Then, the determination unit 124 determines a stereoscopic partner frame based on the smoothing criterion, that is, using Equation 8.
- idx (m) indicates a three-dimensional partner frame index.
- the stereo partner frame may be determined from the selected stereo partner frame candidates based on the first criteria of a) to c) without using the distortion reduction criteria of d).
- the selected stereoscopic partner frame candidate cannot be used as a suitable stereoscopic partner frame corresponding to the target frame as it is. This is because the selected steric partner frame candidate usually has distortion. That is, when the selected stereoscopic partner frame candidate is used as a stereoscopic partner frame, the final 3D video is affected and a comfortable 3D effect cannot be achieved.
- a matrix H n m is a projective transformation matrix calculated based on corresponding points between the m-th frame and the n-th frame.
- the matrix W m is an oblique transformation matrix calculated based on the common area.
- the determination is made with the distortion reduced using the oblique transformation matrix.
- the determination is not limited thereto.
- the determination may be made by reducing distortion using a transformation matrix generated by combining a projective transformation matrix, a slope transformation matrix, and a translation transformation matrix.
- the process of generating the transformation matrix will be described with reference to FIG.
- FIG. 11 is a flowchart for explaining the process of generating the transformation matrix in the first embodiment of the present invention.
- the transformation matrix is generated by the transformation unit 126 based on the target frame and the corresponding stereoscopic frame candidate.
- the determination unit 124 determines a stereoscopic partner frame from the stereoscopic partner frame candidates using the generated transformation matrix.
- the conversion unit 126 calculates a basic matrix based on the feature points of the target frame and the corresponding feature points (feature points of the common region) of the stereoscopic partner frame candidate.
- the conversion unit 126 calculates a projective transformation matrix based on the basic matrix so as to minimize the vertical difference between the common region of the target frame and the common region of the stereoscopic partner frame candidate.
- the projection transformation matrix shown in Equation 9 is constrained using epipolar geometry as shown in Equation 10.
- the projection transformation matrix shown in Expression 9 that matches or is compatible with the basic matrix is restricted as shown in Expression 10 (see, for example, Non-Patent Document 4).
- the degree of freedom of normal projective transformation is reduced to 3 instead of 8.
- Equation 10 H represents a 3 ⁇ 3 projective transformation matrix, and F represents the 3 ⁇ 3 basic matrix calculated in S402.
- E ′ is the 3 ⁇ 1 homogeneous coordinate of the epipole candidate for the stereo partner frame. This can be derived from the base matrix F.
- [A] x b can also be expressed as a ⁇ b as shown by the outer product.
- v T denotes a 1 ⁇ 3 vector including a parameter with 3 degrees of freedom.
- the method of calculating the projective transformation matrix based on the basic matrix so as to minimize the vertical difference is not limited to the above method. Other methods may be used as long as H can be obtained under similar restrictions, and are included in the scope of the present invention.
- the conversion unit 126 calculates an oblique transformation matrix in which the common area in the stereoscopic partner frame transformed by the projective transformation matrix maintains the same orthogonality and aspect ratio as before the transformation.
- a slope transformation matrix for reducing distortion is calculated for the selected stereoscopic partner frame candidate. More specifically, a slope transformation matrix having a slope amount that restores the orthogonality between the aspect ratio and the common area is calculated.
- 12A and 12B are diagrams for explaining the concept of terms in the first embodiment of the present invention. First, the concept of the common area will be described with reference to FIG. 8, and the concept of aspect ratio and orthogonality will be described with reference to FIGS. 12A and 12B.
- a rectangle 606 and a rectangle 608 indicate the outer shape of a common region that is a region common to the target frame 602 and the three-dimensional partner frame candidate 604. The regions outside the rectangle 506 and the rectangle 508 appear only in one of the target frame 602 and the solid partner frame candidate 604.
- a stereoscopic pair (stereoscopic image) is formed as it is using the target frame 602 and the stereoscopic partner frame candidate 604, these outer regions feel uncomfortable when a human sees the stereoscopic image. This is because a person views a stereoscopic image as one image, and the human brain cannot extract disparity information for perceiving depth.
- the rectangle 606 and the rectangle 608 are referred to as a common region, but are also referred to as a visual field region.
- FIG. 12A shows a three-dimensional partner frame candidate 604 before warping
- FIG. 12B shows a three-dimensional partner frame candidate 712 after warping.
- warp is synonymous with the application of a transformation matrix
- a transformation matrix generated by combining a projective transformation matrix, a slope transformation matrix, and a translation transformation matrix is applied. Means.
- the rectangle 608 (common area) shown in FIG. 12A has a predetermined aspect ratio.
- This aspect ratio is represented by the ratio between the distance between the points 718 and 722 and the distance between the points 720 and 724.
- corner 726 is a right angle.
- the warped common area 716 shown in FIG. 12B is represented by the ratio of the distance between the warped points 728 and 730 and the distance between the points 732 and 734.
- the warped common region 716 shown in FIG. 12B may have an aspect ratio different from that of the rectangle 618 (common region) shown in FIG. 12A, and the corner 736 may not be a right angle.
- the orthogonality and the aspect ratio may be restored by using a general shear transformation matrix as shown in Expression 12.
- the shear transformation matrix is a matrix calculated for restoring the orthogonality and the aspect ratio.
- the center of gravity of a plurality of feature points of the target frame is calculated.
- the center of gravity of the corresponding point (common area) corresponding to the feature point of the target frame among the feature points of the stereoscopic partner frame candidate is calculated.
- the difference of the gravity center of the corresponding point (common area) between the target frame and the solid partner frame candidate is calculated.
- the vertical difference 636 is as shown in FIG. 8
- the horizontal difference 638 is as shown in FIG. In this way, the dimension of the common region can be derived from the frame size of the target frame and the difference between the centroids of corresponding points in the target frame and the stereoscopic partner frame candidate.
- this calculation method may not be able to calculate a perfect common area. That is, there may be a difference between the calculated common area and the actual common area. However, there is no problem because sufficiently satisfactory results can be obtained by this method.
- the conversion unit 126 calculates a slope conversion matrix. At that time, a shear transformation matrix may be further calculated.
- the conversion unit 126 determines that the parallax between the target frame and the stereoscopic partner frame candidate to which the projective transformation matrix and the oblique transformation matrix are applied is the target frame and the stereoscopic partner frame candidate before the target frame.
- the translation transformation matrix is calculated so as to be the same as the parallax between.
- the conversion unit 126 calculates the translation conversion matrix shown in Expression 16 using Expression 13 to Expression 15 below in order to realize a smooth depth.
- this translation transformation matrix is applied, the difference between the horizontal parallax of the target stereo pair and the horizontal parallax of the previous stereo pair is minimized, and further, the vertical parallax of the target stereo pair is minimized.
- v represents a conversion vector and can be expressed as Equation 15.
- a transformation matrix for each stereoscopic partner frame candidate is calculated.
- the stereo partner frame candidate includes ⁇ I n , Warp (T n , I n )
- the stereo partner frame is determined based on the procedure shown in FIG. Also good.
- the stereo partner frame may be determined based on the procedure shown in FIG. 10 from the stereo partner frame candidate selected in S108 (a stereo partner frame candidate that has not been warped at this time) using the above transformation matrix.
- a stereoscopic partner frame from the stereoscopic partner frame candidate selected in S108 using the calculated transformation matrix.
- warping is performed based on the transformation matrix generated by the transformation unit 126 in S112 shown in FIG. This is preferable because it is more efficient than performing conversion (warping) in each of the procedures (S302 to S304) shown in FIG.
- the stereoscopic partner frame determined based on the smoothing criterion may be an original stereoscopic partner frame to which the above transformation matrix is not applied, or a stereoscopic partner frame to which this transformation matrix is applied. Also good.
- the effects of the stereoscopic video creation apparatus and the stereoscopic video creation method according to the present embodiment will be described with reference to FIGS. 6 to 8 and FIG.
- the video camera 542 will be described assuming that a stereoscopic video creation device is configured.
- FIG. 13 is a diagram illustrating a 3D image obtained by stereoscopically converting a stabilized 2D image captured by the video camera according to Embodiment 1 of the present invention.
- a photographer (user) of a hand-held video camera 542 uses the video camera 542 to photograph a scene 502 shown in FIG.
- the scene 502 is assumed as a typical example of a stationary object such as a landscape.
- the user shoots the scene 502 as a 2D image while shifting the video camera 542 at a constant speed in the horizontal direction at the same height. That is, the user shoots the scene 502 as a 2D image while shifting the video camera 542 along the trajectory 562.
- the trajectory 562 is fluctuated in the vertical direction due to camera shake.
- the video camera 542 shoots, for example, a video frame 582 (2D video) corresponding to the video sample point 522 shown in FIG. 7, that is, an unstable 2D video with vertical fluctuation. Only unstable and inappropriate 3D images can be created from unstable 2D images. Therefore, in this embodiment, the 2D video is stabilized before the 2D video is converted into a 3D video (stereoscopic video).
- the video camera 542 stabilizes the captured 2D video.
- the 2D video is corrected (stabilized) as if the scene 502 was shot by the virtual video camera 544 along the virtual trajectory 566 indicating the trajectory 562 that has been corrected so as to significantly reduce fluctuation due to camera shake.
- the virtual video camera 544 corresponds to, for example, taking a virtual video frame 584 corresponding to the virtual video sample point 524 shown in FIG. 8, that is, shooting a stable 2D video in which fluctuations in the vertical direction are greatly reduced.
- the trajectory is smoother and the virtual video sample points 524 are more uniform, as shown in the virtual trajectory 566 of the virtual video camera 544.
- a sound and appropriate 3D image can be created from the stabilized 2D image.
- the video camera 542 performs a stereoscopic video conversion on the stabilized 2D video to create a 3D video.
- the virtual video camera 746 creates a stabilized 3D video along the virtual trajectory 766.
- the virtual video camera 746 corresponds to creating a stereoscopic video frame pair 786 and 788 corresponding to the virtual stereoscopic video sample points 767 and 768 shown in FIG.
- the video camera 542 can create an appropriate and comfortable 3D video from the 2D video.
- the stereoscopic video creation device 100 includes the reception unit 101, the stereoscopic video conversion unit 106, the video output unit 108, and the internal buffer 110. I can't.
- the minimum configuration of the stereoscopic video creation device 100 includes a reception unit 101, and a stereoscopic video creation unit 118 having a selection unit 123, a determination unit 124, a stereoscopic pair generation unit 125, and a conversion unit 126. It is sufficient that at least the stereoscopic image creation device 150 is provided.
- FIG. 14 is a block diagram showing the minimum configuration of the stereoscopic image creating apparatus according to the present invention. Elements similar to those in FIG. 1 are denoted by the same reference numerals, and detailed description thereof is omitted.
- FIG. 15 is a flowchart showing the operation of the stereoscopic image creating apparatus shown in FIG.
- the receiving unit 101 receives 2D video.
- the selection unit 123 has a common area which is an area common to the target frame from a plurality of frames constituting the 2D video, and the size of the area occupied by the common area is determined in advance.
- a plurality of frames that are greater than or equal to the value are selected as a stereoscopic partner frame candidate that is a frame candidate that constitutes a stereoscopic image together with the target frame.
- the determination unit 124 determines a stereoscopic partner frame, which is a frame that forms a stereoscopic image together with the target frame, from the stereoscopic partner frame candidates based on the first reference.
- the stereo pair generation unit 125 generates a stereo pair constituting the stereo image using the target frame and the stereo partner frame.
- a transformation matrix is generated based on the target frame and the determined stereoscopic partner frame, and the stereoscopic pair is converted by applying the generated transformation matrix to the stereoscopic pair based on the second criterion.
- an appropriate and comfortable 3D image is created from the 2D image without performing the estimation by the SFM having a high calculation cost and the time-consuming depth map. That is, a method of selecting a stereoscopic partner frame candidate corresponding to the target frame from a plurality of frames constituting the 2D video, and determining a stereoscopic partner frame from the selected stereoscopic partner frame candidate based on the first reference. By using this, an appropriate and comfortable 3D video is created from the 2D video.
- 3D video can be created from 2D video.
- FIG. 16 is a block diagram showing a configuration of the stereoscopic video creation device according to Embodiment 2 of the present invention. Elements similar to those in FIG. 1 are denoted by the same reference numerals, and detailed description thereof is omitted.
- the stereoscopic video creation device 200 shown in FIG. 16 differs from the stereoscopic video creation device 100 according to Embodiment 1 in the configuration of the reception unit 201 and a user feedback unit 204 is added.
- the stereoscopic video creation apparatus 200 is further connected to the display unit 223.
- the receiving unit 201 receives a 2D video imaged by a user with a photographing device such as a camera. Specifically, the receiving unit 201 includes, for example, a raw medium 202, and acquires image data from a photographing device such as a camera. The receiving unit 201 outputs the 2D video data S13 to the stereoscopic video converting unit 106, and transmits the original image data S21 to the user feedback unit 204. Note that the receiving unit 201 may transmit the original image data S21 to the user feedback unit 204 only when the preview pass / preview mode is set in the stereoscopic video creation device 200, for example.
- the user feedback unit 204 includes a motion analysis unit 220 and a feedback unit 222, and transmits feedback information that feeds back 2D video captured by the user with the imaging device in real time to the display unit 223.
- the feedback information is information for guiding the user to shoot a stabilized 2D video such as an arrow indicating the shooting direction or a signal indicating the movement of the shooting device.
- the motion analysis unit 220 corresponds to the analysis unit of the present invention, and generates information for guiding the user to shoot a stabilized 2D video by analyzing the motion of the imaging device.
- the feedback unit 222 generates feedback information for feeding back to the user based on the information generated by the motion analysis unit 220.
- the display unit 223 is a camera monitor or the like, and displays generated feedback information and 2D video.
- the display unit 223 may guide the user based on the transmitted feedback information so that the stereoscopic video creation apparatus 200 captures a video in the best way for creating a suitable 3D effect.
- the stereoscopic image creation apparatus 200 is configured.
- the stereoscopic video creation apparatus 200 (or the user feedback unit 204 or the stereoscopic video conversion unit 106 or the like) is usually realized by an IC (integrated circuit), an ASIC (specific application integrated circuit), or an LSI (large scale integrated circuit). However, it may be composed of a plurality of chips or a single chip. Note that each of the stereoscopic video creation device 200, the user feedback unit 204, the stereoscopic video conversion unit 106, and the like is not limited to being realized in the form of an LSI. Depending on the degree of integration, it may be realized by what is also called an IC, a system LSI, a super LSI, or an ultra LSI.
- the stereoscopic video creation device 200 and the like may be realized by being integrated using a dedicated circuit or a general-purpose processor.
- a dedicated circuit there is a specialized microprocessor such as a DSP (digital signal processor) that can be controlled by a program instruction.
- DSP digital signal processor
- each of the stereoscopic image creation apparatus 200 and the like may use an FPGA (field programmable gate array) that can be programmed after manufacturing the LSI, or a processor that can reconfigure the connection or configuration of the LSI.
- FPGA field programmable gate array
- the stereoscopic video creating apparatus 200 generates a stereoscopic image of a liquid crystal display device that displays images (videos) in time series, a plasma display device, a display device to which a lenticular lens layer is added, or other types of display devices. It may be incorporated in a display device capable of display.
- the stereoscopic video creation device 200 may be mounted on a digital media player device such as a digital video disc player, a Blu-ray disc player, and other types of digital media players. It may be implemented in other types of devices. In either case, the scope of the present application is not limited.
- FIG. 17 is a block diagram showing a configuration of an image apparatus according to Embodiment 3 of the present invention. Elements similar to those in FIG. 1 are denoted by the same reference numerals, and detailed description thereof is omitted.
- An image apparatus 1200 shown in FIG. 17 includes an optical system 1202, a video sensor 1204, an ADC (Analog-Digital Converter) 1206, a video processor 1208, a microcomputer 1210, an external memory 1212, a driver controller 1220, An operation unit 1222, a storage / transmission device 1216, and a display device 1214 are provided.
- ADC Analog-Digital Converter
- the video processor 1208 includes an internal memory 1240, a stereoscopic video conversion unit 106, an original image processor 1242, and a color video processor 1244. Although other components such as a microphone and a speaker are not illustrated, this does not limit the scope of the present invention.
- the optical system 1202 controls an optical signal that reaches the video sensor 1204.
- the optical system 1202 includes components such as a plurality of lenses or a lens set, a zoom / focus mechanism, an actuator, a shutter, and an opening.
- the optical system 1202 is controlled by the driver controller 1220, for example.
- the driver controller 1220 is controlled by the microcomputer 1210 to control the actuator and the like in the optical system 1202. Since the driver controller 1220 can move the lens of the optical system 1202 so as to compensate for the shaking, it is possible to reduce blur caused by camera shake or camera movement.
- the operation unit 1222 receives a user operation input and transmits the electrical signal to the microcomputer 1210.
- the operation unit 1222 can control the modules such as the driver controller 1220, the video sensor 1204, and the video processor 1208 that are associated with the user input by transmitting the electrical signal.
- the microcomputer 1210 controls the driver controller 1220 and the video sensor 1204.
- the video sensor 1204 accumulates incident optical signals and converts the optical signals into electric signals.
- the video sensor 1204 is controlled by the microcomputer 1210.
- the ADC 1206 converts the electrical signal converted by the video sensor 1204 into digital data (original image data), and stores it in the internal memory 1240 or the external memory 1212.
- the original image processor 1242 acquires the original image data from the internal memory 1240 (or the external memory 1212), and performs many preprocessing such as noise reduction, linearity correction, white balance, and gamma correction.
- the original image processor 1242 outputs the preprocessed original image to the storage / transmission device 1216 or the color video processor 1244.
- the color video processor 1244 generates a color image such as RGB or YCbCr by processing the original image preprocessed by the original image processor 1242.
- the processing of the color video processor 1244 includes processing such as color interpolation, color correction, tone range adjustment, and color noise reduction, and generates a suitable color image.
- the stereoscopic video conversion unit 106 includes two submodules, a stabilization unit 116 and a stereoscopic video creation unit 118.
- the stereoscopic video conversion unit 106 takes in a video frame from the internal memory 1240, stabilizes it, and converts it into 3D video.
- the stereoscopic video conversion unit 106 outputs the converted 3D video to the display device 1214 or the external memory 1212. Note that the details of the stereoscopic video conversion unit 106 have been described above, and a description thereof will be omitted here.
- the display device 1214 is, for example, a liquid crystal monitor to which a lenticular lens layer capable of displaying a stereoscopic image viewed in 3D is attached.
- the display device 1214 can display the 3D video output from the stereoscopic video conversion unit 106.
- the display device 1214 may display the 3D video output from the stereoscopic video conversion unit 106 as a 2D video or may store the 3D video in the storage / transmission device 1216.
- the storage / transmission device 1216 stores or transmits the original video preprocessed by the original image processor 1242 and the 3D video output from the stereoscopic video conversion unit 106. Note that the storage / transmission device 1216 compresses the original video pre-processed by the original image processor 1242 and the 3D video output from the stereoscopic video conversion unit 106 by a compression unit (not shown) before storing / transmitting. You may do that.
- the storage / transmission device 1216 may be configured with, for example, a flash-based memory card, a hard drive, an optical drive, or the like, but is not limited thereto.
- the storage / transmission device 1216 includes, for example, an HDMI interface, a USB interface, a wireless interface, and a direct-to-printer interface, but is not limited thereto.
- the storage / transmission device 1216 may arbitrarily perform reversible or irreversible compression on data to be processed (stored or transmitted).
- the video processor 1208 and the internal modules are usually realized by an IC (Integrated Circuit), an ASIC (Application Specific Integrated Circuit), or an LSI (Large Scale Integrated Circuit), but may be configured by a plurality of chips. It may be composed of a chip.
- each of the stereoscopic video creation device 100 or the stereoscopic video conversion unit 106 and the like is not limited to being realized in the form of an LSI. Depending on the degree of integration, it may be realized by what is also called an IC, a system LSI, a super LSI, or an ultra LSI.
- the video processor 1208 and the internal modules may be realized by being integrated using a dedicated circuit or a general-purpose processor, respectively.
- a dedicated circuit there is a specialized microprocessor such as a DSP (digital signal processor) that can be controlled by a program instruction.
- DSP digital signal processor
- each of the video processor 1208 and the internal module may use an FPGA (Field Programmable Gate Array) that can be programmed after manufacturing the LSI, or a processor that can reconfigure the connection or configuration of the LSI. .
- FPGA Field Programmable Gate Array
- the present invention it is possible not only to generate a stereoscopic image without performing estimation by SFM having a high calculation cost, but also to generate a stereoscopic image without estimating a time-consuming depth map. Thereby, it is possible to realize a stereoscopic video creation apparatus and a stereoscopic video creation method for creating an appropriate and comfortable 3D video from a 2D video.
- the stereoscopic image creating apparatus of the present invention receives 2D images and detects (tracks) corresponding points between consecutive video frames.
- the stereoscopic video creation device of the present invention performs stabilization processing on a target frame based on corresponding points between the target frame and a plurality of neighboring frames corresponding thereto. Note that the stabilization process may not be performed on the 2D video image that has been stabilized in advance.
- the stereoscopic video creation device of the present invention selects a stereoscopic partner frame candidate for the target frame according to the size of the common area between the target frame and the stereoscopic partner frame candidate.
- the stereoscopic video creation device of the present invention calculates a projective transformation matrix that satisfies the condition that the vertical direction difference is zero and the epipolar constraint between the target frame and each stereoscopic partner frame candidate.
- the stereoscopic image creation device of the present invention calculates a slope transformation matrix to reduce distortion that occurs when applying the projective transformation matrix, and the difference in parallax between corresponding points between the target frame and neighboring frames A translation transformation matrix for minimizing is calculated.
- the stereoscopic video creation device of the present invention determines a stereoscopic partner frame for the target frame based on the first criterion using the calculated projective transformation matrix, the oblique transformation matrix, and the translational transformation matrix from the stereoscopic partner frame candidates. .
- the first standard includes a vertical direction difference standard, a horizontal parallax standard (baseline), a smoothing standard, and a distortion reduction standard.
- the vertical direction difference standard stipulates that the difference in the y direction (vertical direction) between the target frame and the three-dimensional partner frame is zero.
- the horizontal parallax reference (baseline) satisfies an appropriate range so that the x-direction (horizontal direction) parallax between the target frame and the three-dimensional partner frame is sufficient to obtain a 3D effect and provides a comfortable 3D view.
- the smoothing standard is defined in the depth direction, and is defined to satisfy a range in which the parallax of corresponding points between adjacent solid pairs (stereoscopic partner frames corresponding to the target frame) is smooth.
- the distortion reduction standard is defined to be created based on these common regions without being affected by the conventional conversion in which the stereoscopic pair causes distortion.
- the stereoscopic video creation device of the present invention applies a transformation matrix generated by combining a projection transformation matrix, a slope transformation matrix, and a translation transformation matrix to the decided stereoscopic partner frame (stereopair). Warp (deform) the solid partner frame.
- the stereoscopic video creation device of the present invention outputs the target frame and the warped stereoscopic partner frame as a stereoscopic pair constituting a 3D image.
- the stereoscopic video creation device of the present invention thus repeats until the target frame becomes the final frame constituting the 2D video, and outputs the 3D video generated from the stereoscopic pair.
- the stereoscopic video creation device and the stereoscopic video creation method of the present invention have been described based on the embodiment, but the present invention is not limited to this embodiment. Unless it deviates from the meaning of this invention, the form which carried out the various deformation
- the present invention can be used for a stereoscopic video creation device and a stereoscopic video creation method, and is particularly mounted on a liquid crystal display device, a plasma display device, a display device to which a lenticular lens layer is added, a digital video disc player, a Blu-ray disc player or the like.
- the present invention can be used for a stereoscopic video creation device and a stereoscopic video creation method.
- Stereoscopic image creation device 101, 201 Reception unit 102 Storage medium reader 104 Video decoder 106 Stereoscopic video conversion unit 108 Video output unit 110 Internal buffer 112, 223 Display unit 114 Transmitter 116 Stabilization unit 118 Stereoscopic image creation unit 121 Detection Unit 122 Calculation Unit 123 Selection Unit 124 Determination Unit 125 Solid Pair Generation Unit 126 Conversion Unit 202 Raw Media 204 User Feedback Unit 220 Motion Analysis Unit 222 Feedback Unit 502 Scene 506, 508, 606, 608, 618 Rectangle 522 Video Sample Point 524 Virtual video sample points 542 Video camera 544, 746 Virtual video camera 562 Trajectory 566, 766 Virtual trajectory 582 Video frame 584 Virtual video frame 60 Target frame 604, 712 Stereo partner frame candidate 636 Vertical difference 638 Horizontal difference 716 Common area 718, 720, 722, 724, 728, 730, 732, 734 Point 726, 736 Angle 767 Virtual stereoscopic video sample point 786 Stereo video frame pair 1200 Image device
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
- Processing Or Creating Images (AREA)
- Image Processing (AREA)
Abstract
Description
図1は、本発明の実施の形態1における立体映像作成装置の構成を示すブロック図である。図2は、本発明の実施の形態1における安定化部の詳細構成を示すブロック図である。図3は、本発明の実施の形態1における立体映像作成部の詳細構成を示すブロック図である。
図16は、本発明の実施の形態2における立体映像作成装置の構成を示すブロック図である。図1と同様の要素には同一の符号を付しており、詳細な説明は省略する。
実施の形態3では、実施の形態1および実施の形態2と異なる装置に、上述の立体映像変換部が構成される場合の例について説明する。
101、201 受信部
102 記憶媒体リーダ
104 ビデオデコーダ
106 立体映像変換部
108 映像出力部
110 内部バッファ
112、223 表示部
114 送信装置
116 安定化部
118 立体映像作成部
121 検出部
122 算出部
123 選択部
124 決定部
125 立体対生成部
126 変換部
202 ロウメディア
204 ユーザフィードバック部
220 動き分析部
222 フィードバック部
502 シーン
506、508、606、608、618 矩形
522 ビデオサンプルポイント
524 仮想ビデオサンプルポイント
542 ビデオカメラ
544、746 仮想ビデオカメラ
562 軌道
566、766 仮想軌道
582 ビデオフレーム
584 仮想ビデオフレーム
602 対象フレーム
604、712 立体パートナーフレーム候補
636 垂直差
638 水平差
716 共通領域
718、720、722、724、728、730、732、734 点
726、736 角
767 仮想立体映像サンプルポイント
786 立体映像フレーム対
1200 画像装置
1202 光学系
1204 ビデオセンサ
1208 ビデオプロセッサ
1210 マイクロコンピュータ
1212 外部メモリ
1214 表示装置
1216 送信装置
1220 ドライバコントローラ
1222 操作部
1240 内部メモリ
1242 原画像プロセッサ
1244 カラー映像プロセッサ
Claims (16)
- 2D映像から3D映像を作成する立体映像作成装置であって、
2D映像を受信する受信部と、
前記2D映像を構成する複数のフレームから、対象フレームと共通する画像の領域である共通領域が占める領域の大きさが予め定めた値以上である複数のフレームを、当該対象フレームとともに立体画像を構成するフレームの候補である立体パートナーフレーム候補として選択する選択部と、
前記立体パートナーフレーム候補から、所定の基準に基づいて、当該対象フレームとともに立体画像を構成するフレームである立体パートナーフレームを決定する決定部と、
前記対象フレームと、前記立体パートナーフレームとを用いて、前記対象フレームに対応する立体画像を構成する立体対を生成する立体対生成部と、
前記対象フレームと前記選択部により選択された前記立体パートナーフレーム候補とから、前記立体パートナーフレーム候補をワープさせるための変換行列を生成し、前記立体対に、生成した前記変換行列を適用することにより、前記立体対を変換する変換部とを備える、
立体映像作成装置。 - さらに、
前記2D映像を構成する複数のフレームから、当該複数のフレーム間のゆれを補正することで、当該複数のフレームを安定化させる安定化部を備え、
前記選択部は、前記安定化部により安定化された前記2D映像を構成する複数のフレームから、立体パートナーフレーム候補を選択する、
請求項1に記載の立体映像作成装置。 - 前記立体映像作成装置は、
前記変換部により変換された立体対を記憶するための記憶部と、
前記記憶部に記憶された当該変換された立体対を出力する出力部とを備え、
前記出力部は、前記2D映像を構成する複数のフレームそれぞれに対応する立体対であって、前記変換部により変換された立体対を出力することで、前記2D映像から3D映像を生成する、
請求項1または2に記載の立体映像作成装置。 - 前記選択部は、前記立体パートナーフレーム候補として、前記2D映像を構成する複数のフレームから、前記対象フレームと同一シーンに撮影された複数のフレームを選択する、
請求項1~3のいずれか1項に記載の立体映像作成装置。 - 前記選択部は、前記対象フレームに占める前記共通領域の大きさの割合と前記立体パートナーフレーム候補に占める前記共通領域の大きさの割合とがそれぞれ前記予め定めた値以上の場合に、前記立体パートナーフレーム候補が前記対象フレームと同一シーンに撮影されたフレームであると判断し、
前記共通領域は、前記対象フレームと前記立体パートナーフレーム候補との間で対応するフレーム上の特徴的な点である特徴点に基づき算出される、
請求項4に記載の立体映像作成装置。 - 前記選択部は、前記2D映像を構成する複数のフレームにおいて、前記対象フレームと時間的に近いフレームである近隣フレームを、前記立体パートナーフレーム候補として、選択する、
請求項4または5に記載の立体映像作成装置。 - 前記所定の基準は、垂直方向差基準と、水平視差基準と、円滑化基準と、歪低減基準とで構成され、
前記垂直方向差基準では、前記対象フレームの前記共通領域と垂直方向の位置の差がゼロに略等しいフレームを前記立体パートナーフレームとする旨が規定され、
前記水平視差基準では、前記対象フレームの前記共通領域と水平方向の位置の差が所定の範囲に含まれるフレームを前記立体パートナーフレームとする旨が規定され、
前記円滑化基準では、時間的に連続する立体対間において、前記共通領域の変位が滑らかとなるフレームを前記立体パートナーフレームとする旨が規定され、
前記歪低減基準では、前記共通領域が占める領域の大きさが前記対象フレームとともに前記予め定めた値以上であるフレームを前記立体パートナーフレームとする旨が規定されている、
請求項1~6のいずれか1項に記載の立体映像作成装置。 - 前記決定部は、画像処理を施すことによって検出可能なフレーム上の特徴的な点である特徴点を用いて、前記所定の基準を満たすか否かを判定することにより、前記立体パートナーフレームを決定する、
請求項7に記載の立体映像作成装置。 - 前記変換部は、前記変換行列を、フレーム上の特徴的な点である特徴点であって、前記対象フレームと前記立体パートナーフレームとの間で対応する特徴点に基づいて生成する、
請求項1または2に記載の立体映像作成装置。 - 前記変換部は、前記変換行列を、算出した射影変換行列と斜傾変換行列と並進変換行列とを組み合わせて生成して、前記立体対に適用し、
前記変換部は、前記対象フレームと前記立体パートナーフレーム候補との間で基本行列を算出し、
前記変換部は、前記立体対における対象フレームと前記立体パートナーフレーム候補との間の垂直方向差を最小化するように、前記基本行列に基づいて前記斜傾変換行列を算出し、
前記変換部は、前記射影変換行列を、前記射影変換行列により変換される前記立体パートナーフレームにおける前記共通領域を、変換前と同じ直交性とアスペクト比とを維持させるように、前記射影変換行列を算出し、
前記変換部は、前記対象フレームと、前記射影変換行列および前記斜傾変換行列が適用された立体パートナーフレームとの間の視差が、前記対象フレームより以前の対象フレームと立体パートナーフレームとの間の視差と同じになるように、前記並進変換行列を算出する、
請求項1または2に記載の立体映像作成装置。 - 前記安定化部は、フレーム上の特徴的な点である特徴点であって、所定フレームと、前記所定フレームと時間的に近いフレームである複数の近隣フレームとの間で対応する特徴点に基づいて算出する射影変換行列を用いることにより、当該複数のフレーム間のゆれを補正する、
請求項2に記載の立体映像作成装置。 - 前記安定化部は、
所定フレームと、前記所定フレームに隣接するフレームである近隣フレームとの間で対応する特徴的な点である複数の特徴点を検出する検出部と、
前記所定フレームの複数の特徴点と、対応する前記近隣フレームの重みづけされた複数の特徴点とが同じ座標値を有するように、前記所定フレームをワープさせる射影変換行列を算出する算出部とを備え、
前記射影変換行列を前記所定フレームに適用することで、当該複数のフレーム安定化させる、
請求項2に記載の立体映像作成装置。 - 前記複数の近隣フレームは、前記所定フレームと時間的に近いフレームである、
請求項12に記載の立体映像作成装置。 - 前記算出部は、前記複数の近隣フレームの重みを、重み関数を用いて算出し、
前記算出部は、
前記対応する近隣フレームが前記所定フレームに時間的に最も近いフレームである場合、前記重み関数を用いて1により近い値の重みを算出し、
前記対応する近隣フレームが前記所定フレームから時間的に遠いフレームである場合、前記重み関数を用いて1より小さい値の重みを算出する、
請求項12に記載の立体映像作成装置。 - 前記受信部は、ユーザが撮影装置で撮影した2D映像を受信し、
前記立体映像作成装置は、さらに、
前記撮影装置の動きを分析して、安定化した2D映像を撮影するように前記ユーザを導くための情報を生成する分析部と、
前記分析部により生成された情報に基づいて、前記ユーザにフィードバックするためのフィードバック情報を生成するフィードバック部と、
生成された前記フィードバック情報と、前記2D映像とを表示する表示部とを備える、
請求項2に記載の立体映像作成装置。 - 2D映像から3D映像を作成する立体映像作成方法であって、
2D映像を受信する受信ステップと、
前記2D映像を構成する複数のフレームから、対象フレームと共通する画像の領域である共通領域が占める領域の大きさが予め定めた値以上である複数のフレームを、当該対象フレームとともに立体画像を構成するフレームの候補である立体パートナーフレーム候補として選択する選択ステップと、
前記立体パートナーフレーム候補から、前記所定の基準に基づいて、当該対象フレームとともに立体画像を構成するフレームである立体パートナーフレームを決定する決定ステップと、
前記対象フレームと、前記立体パートナーフレームとを用いて、立体画像を構成する立体対を生成する立体対生成ステップと、
前記対象フレームと前記選択部により選択された前記立体パートナーフレーム候補とから、前記立体パートナーフレーム候補をワープさせるための変換行列を生成し、前記立体対に、生成した前記変換行列を適用することにより、前記立体対を変換する変換ステップとを含む、
立体映像作成方法。
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/509,809 US8908011B2 (en) | 2010-09-17 | 2011-09-16 | Three-dimensional video creating device and three-dimensional video creating method |
CN201180004478.6A CN102598683B (zh) | 2010-09-17 | 2011-09-16 | 立体影像制作装置以及立体影像制作方法 |
EP11824796.4A EP2618584B1 (en) | 2010-09-17 | 2011-09-16 | Stereoscopic video creation device and stereoscopic video creation method |
JP2012502347A JP5891424B2 (ja) | 2010-09-17 | 2011-09-16 | 立体映像作成装置および立体映像作成方法 |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2010-210138 | 2010-09-17 | ||
JP2010210138 | 2010-09-17 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2012035783A1 true WO2012035783A1 (ja) | 2012-03-22 |
Family
ID=45831268
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2011/005250 WO2012035783A1 (ja) | 2010-09-17 | 2011-09-16 | 立体映像作成装置および立体映像作成方法 |
Country Status (5)
Country | Link |
---|---|
US (1) | US8908011B2 (ja) |
EP (1) | EP2618584B1 (ja) |
JP (1) | JP5891424B2 (ja) |
CN (1) | CN102598683B (ja) |
WO (1) | WO2012035783A1 (ja) |
Families Citing this family (23)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP5571257B2 (ja) * | 2011-12-19 | 2014-08-13 | 富士フイルム株式会社 | 画像処理装置、方法およびプログラム |
JP5977591B2 (ja) * | 2012-06-20 | 2016-08-24 | オリンパス株式会社 | 画像処理装置及びそれを備えた撮像装置、画像処理方法、並びに画像処理プログラムを記録したコンピュータ読み取り可能な記録媒体 |
KR20140104121A (ko) * | 2013-02-20 | 2014-08-28 | 주식회사 카몬 | 복합 차원 콘텐츠 제공 시스템, 그 제공 방법 및 복합 차원 콘텐츠 파일 |
US9967549B2 (en) * | 2013-03-20 | 2018-05-08 | Mediatek Inc. | 3D image capture method with 3D preview of preview images generated by monocular camera and related electronic device thereof |
US9842423B2 (en) | 2013-07-08 | 2017-12-12 | Qualcomm Incorporated | Systems and methods for producing a three-dimensional face model |
LU92357B1 (en) * | 2014-01-21 | 2015-07-22 | Evolution & Dev Technology S A | 2D/3D Video conversion method |
US9948920B2 (en) | 2015-02-27 | 2018-04-17 | Qualcomm Incorporated | Systems and methods for error correction in structured light |
US10068338B2 (en) | 2015-03-12 | 2018-09-04 | Qualcomm Incorporated | Active sensing spatial resolution improvement through multiple receivers and code reuse |
US9530215B2 (en) | 2015-03-20 | 2016-12-27 | Qualcomm Incorporated | Systems and methods for enhanced depth map retrieval for moving objects using active sensing technology |
US10275863B2 (en) * | 2015-04-03 | 2019-04-30 | Cognex Corporation | Homography rectification |
US9635339B2 (en) | 2015-08-14 | 2017-04-25 | Qualcomm Incorporated | Memory-efficient coded light error correction |
US9846943B2 (en) | 2015-08-31 | 2017-12-19 | Qualcomm Incorporated | Code domain power control for structured light |
SG10201510773PA (en) * | 2015-12-30 | 2017-07-28 | Creative Tech Ltd | A method for creating a stereoscopic image sequence |
CN109070803B (zh) * | 2016-04-14 | 2021-10-08 | 金泰克斯公司 | 提供深度信息的车辆显示*** |
US9887497B1 (en) * | 2016-06-10 | 2018-02-06 | Amazon Technologies, Inc. | Device connector with reduced electromagnetic noise |
US9961261B2 (en) * | 2016-06-20 | 2018-05-01 | Gopro, Inc. | Image alignment using a virtual gyroscope model |
US10002284B2 (en) * | 2016-08-11 | 2018-06-19 | Ncku Research And Development Foundation | Iterative matching method and system for partial fingerprint verification |
US10967266B2 (en) | 2018-02-21 | 2021-04-06 | Roblox Corporation | Cinematic game camera in a gaming platform |
CN108712643B (zh) * | 2018-04-27 | 2020-02-21 | 苏州新光维医疗科技有限公司 | 二维图像转换成三维图像的方法、装置及三维成像*** |
GB2575843A (en) * | 2018-07-25 | 2020-01-29 | Sony Interactive Entertainment Inc | Method and system for generating an image |
JP7186298B2 (ja) * | 2018-12-14 | 2022-12-08 | スペクトラル エムディー,インコーポレイテッド | 高精度マルチアパーチャスペクトルイメージングのためのシステムおよび方法 |
CN111212278B (zh) * | 2020-01-07 | 2021-08-03 | 电子科技大学 | 一种预测位移帧的方法与*** |
CN113905147B (zh) * | 2021-09-30 | 2023-10-03 | 桂林长海发展有限责任公司 | 一种船用监控视频画面去抖方法、装置以及存储介质 |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2002324234A (ja) * | 2001-02-23 | 2002-11-08 | Sharp Corp | 立体画像を偏歪修正する方法および装置 |
JP2007316982A (ja) * | 2006-05-26 | 2007-12-06 | Olympus Imaging Corp | 画像表示制御装置、画像表示装置、カメラ、及び画像表示制御プログラム |
WO2010024479A1 (en) | 2008-08-26 | 2010-03-04 | Enhanced Chip Technology Inc. | Apparatus and method for converting 2d image signals into 3d image signals |
JP2010515303A (ja) * | 2006-12-22 | 2010-05-06 | クゥアルコム・インコーポレイテッド | 複雑適応2D−to−3Dビデオシーケンス変換 |
Family Cites Families (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR100374463B1 (ko) | 1994-09-22 | 2003-05-09 | 산요 덴키 가부시키가이샤 | 2차원영상을3차원영상으로변환하는방법 |
IL150131A (en) * | 2002-06-10 | 2007-03-08 | Rafael Advanced Defense Sys | A method for turning a series of monoscopic images into a series of stereoscopic images |
IL166305A0 (en) * | 2005-01-14 | 2006-01-15 | Rafael Armament Dev Authority | Automatic conversion from monoscopic video to stereoscopic video |
JP5284048B2 (ja) * | 2008-11-12 | 2013-09-11 | キヤノン株式会社 | 画像処理装置、撮像装置及び画像処理方法 |
US9124874B2 (en) * | 2009-06-05 | 2015-09-01 | Qualcomm Incorporated | Encoding of three-dimensional conversion information with two-dimensional video sequence |
US8421847B2 (en) * | 2010-05-21 | 2013-04-16 | Mediatek Inc. | Apparatus and method for converting two-dimensional video frames to stereoscopic video frames |
-
2011
- 2011-09-16 EP EP11824796.4A patent/EP2618584B1/en not_active Not-in-force
- 2011-09-16 JP JP2012502347A patent/JP5891424B2/ja active Active
- 2011-09-16 US US13/509,809 patent/US8908011B2/en active Active - Reinstated
- 2011-09-16 WO PCT/JP2011/005250 patent/WO2012035783A1/ja active Application Filing
- 2011-09-16 CN CN201180004478.6A patent/CN102598683B/zh active Active
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2002324234A (ja) * | 2001-02-23 | 2002-11-08 | Sharp Corp | 立体画像を偏歪修正する方法および装置 |
JP2007316982A (ja) * | 2006-05-26 | 2007-12-06 | Olympus Imaging Corp | 画像表示制御装置、画像表示装置、カメラ、及び画像表示制御プログラム |
JP2010515303A (ja) * | 2006-12-22 | 2010-05-06 | クゥアルコム・インコーポレイテッド | 複雑適応2D−to−3Dビデオシーケンス変換 |
WO2010024479A1 (en) | 2008-08-26 | 2010-03-04 | Enhanced Chip Technology Inc. | Apparatus and method for converting 2d image signals into 3d image signals |
Non-Patent Citations (4)
Title |
---|
HARRIS, C.; M. STEPHENS: "A combined corner and edge detector", ALVEY VISION CONFERENCE, 1988 |
HARTLEY, R.; A. ZISSERMAN: "Multiple View Geometry in Computer Vision", 2000, CAMBRIDGE UNIVERSITY PRESS |
LUCAS; BRUCE D.; TAKEO KANADE: "An Iterative Image Registration Technique with an Application to Stereo Vision", INTERNATIONAL JOINT CONFERENCE ON ARTIFICIAL INTELLIGENCE, 1981 |
ZHANG; GUOFENG; WEI HUA; XUEYING QIN; TIEN-TSIN WONG; HUJUN BAO: "Stereoscopic Video Synthesis from a Monocular Video", IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, vol. 13, no. 4, 2007, pages 686 - 696, XP011190841, DOI: doi:10.1109/TVCG.2007.1032 |
Also Published As
Publication number | Publication date |
---|---|
CN102598683A (zh) | 2012-07-18 |
EP2618584B1 (en) | 2016-08-10 |
US20120229609A1 (en) | 2012-09-13 |
JPWO2012035783A1 (ja) | 2014-01-20 |
EP2618584A4 (en) | 2014-05-14 |
US8908011B2 (en) | 2014-12-09 |
EP2618584A1 (en) | 2013-07-24 |
JP5891424B2 (ja) | 2016-03-23 |
CN102598683B (zh) | 2015-06-17 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP5891424B2 (ja) | 立体映像作成装置および立体映像作成方法 | |
JP5679978B2 (ja) | 立体視用画像位置合わせ装置、立体視用画像位置合わせ方法、及びそのプログラム | |
US8274552B2 (en) | Primary and auxiliary image capture devices for image processing and related methods | |
JP5414947B2 (ja) | ステレオ撮影装置 | |
JP5204350B2 (ja) | 撮影装置、再生装置、および画像処理方法 | |
JP5320524B1 (ja) | ステレオ撮影装置 | |
KR101824439B1 (ko) | 모바일 스테레오 카메라 장치 및 그 촬영방법 | |
EP2532166B1 (en) | Method, apparatus and computer program for selecting a stereoscopic imaging viewpoint pair | |
WO2012029298A1 (ja) | 撮影装置および画像処理方法 | |
JP5814692B2 (ja) | 撮像装置及びその制御方法、プログラム | |
JP2011188004A (ja) | 立体映像撮像装置、立体映像処理装置および立体映像撮像方法 | |
WO2012029299A1 (ja) | 撮影装置、再生装置、および画像処理方法 | |
JP5420075B2 (ja) | 立体画像再生装置、その視差調整方法、視差調整プログラム、及び撮影装置 | |
WO2012056685A1 (ja) | 3次元画像処理装置、3次元撮像装置および3次元画像処理方法 | |
US20110242273A1 (en) | Image processing apparatus, multi-eye digital camera, and program | |
US11415935B2 (en) | System and method for holographic communication | |
JP5889022B2 (ja) | 撮像装置、画像処理装置、画像処理方法及びプログラム | |
CN102478756B (zh) | 用来拍摄立体影片的方法与装置 | |
JP2012220603A (ja) | 3d映像信号撮影装置 | |
JP2005072674A (ja) | 三次元画像生成装置および三次元画像生成システム | |
JP2012191608A (ja) | 立体撮像装置および立体撮像方法 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 201180004478.6 Country of ref document: CN |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2012502347 Country of ref document: JP |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 11824796 Country of ref document: EP Kind code of ref document: A1 |
|
REEP | Request for entry into the european phase |
Ref document number: 2011824796 Country of ref document: EP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 13509809 Country of ref document: US Ref document number: 2011824796 Country of ref document: EP |
|
NENP | Non-entry into the national phase |
Ref country code: DE |