EP3080986A1 - Systems and methods for producing panoramic and stereoscopic videos - Google Patents
Systems and methods for producing panoramic and stereoscopic videosInfo
- Publication number
- EP3080986A1 EP3080986A1 EP14869343.5A EP14869343A EP3080986A1 EP 3080986 A1 EP3080986 A1 EP 3080986A1 EP 14869343 A EP14869343 A EP 14869343A EP 3080986 A1 EP3080986 A1 EP 3080986A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- camera
- eye
- pairs
- arrangement
- panoramic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 28
- 230000000694 effects Effects 0.000 claims description 26
- 238000004519 manufacturing process Methods 0.000 claims description 13
- 238000000926 separation method Methods 0.000 claims description 4
- 230000002708 enhancing effect Effects 0.000 abstract description 3
- 210000003128 head Anatomy 0.000 description 15
- 238000004891 communication Methods 0.000 description 7
- 238000012545 processing Methods 0.000 description 7
- 230000003068 static effect Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 230000035807 sensation Effects 0.000 description 4
- 239000007787 solid Substances 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000007547 defect Effects 0.000 description 3
- 210000000624 ear auricle Anatomy 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000012937 correction Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 239000011521 glass Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000007704 transition Effects 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 230000003796 beauty Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000001627 detrimental effect Effects 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 230000009189 diving Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000012467 final product Substances 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000010287 polarization Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
- G03B35/00—Stereoscopic photography
- G03B35/08—Stereoscopic photography by simultaneous recording
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/243—Image signal generators using stereoscopic image cameras using three or more 2D image sensors
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
- G03B37/00—Panoramic or wide-screen photography; Photographing extended surfaces, e.g. for surveying; Photographing internal surfaces, e.g. of pipe
- G03B37/04—Panoramic or wide-screen photography; Photographing extended surfaces, e.g. for surveying; Photographing internal surfaces, e.g. of pipe with cameras or projectors providing touching or overlapping fields of view
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/239—Image signal generators using stereoscopic image cameras using two 2D image sensors having a relative position equal to or related to the interocular distance
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/698—Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
- H04N13/344—Displays for viewing with the aid of special glasses or head-mounted displays [HMD] with head-mounted left-right displays
Definitions
- the present invention relates generally to panoramic and stereoscopic videos and more particularly to systems and methods for producing panoramic and stereoscopic videos.
- Stereoscopic videographic methods can also be employed to produce panoramas.
- a stereoscopic image is produced using two images of a scene recorded from two positions that are slightly horizontally displaced.
- the separation is closed to the inter-ocular distance (on average, 65 to 70 mm).
- One of these images is presented to the left eye of the viewer, and the other to the right eye, providing a perception of depth to the viewer.
- non-filmed content can be made stereoscopic by computer means, filmed videos need two cameras to be used while filming.
- film refers to any medium containing motion picture data, such as videotape, digital video files, conventional movie film, and the like.
- each eye of the viewer must not see the same image. Accordingly, each frame of the displayed video, or a given part of the frame, must be aimed at a specific eye. To do so, filters can be placed between each eye and the screen. When the image displayed has to be seen by a specific eye, it is displayed either with a given polarization or a given color set.
- the filter is either a polarizing filter or a wavelength-specific filter and the eye at which the frame or part of frame is not aimed does not see it.
- Other types of filters may be used, such as active shutters which are externally activated to block images for one eye at a time, in which case the activation is synchronized with the video.
- a head-mounted display VR device of the prior art variety generally takes the form of a screen or a double screen that is placed in front of the eyes of a viewer and held in place by some attachment means such as a headband.
- Some display devices include side elements that enclose the space between the screen and the viewer's face, so that the head-mounted display device may look like a visor or a mask for scuba diving, except that the glass part is replaced by a screen.
- Some display devices use two screens, each one of them being positioned in front of an eye. Other display devices divide the screen in two sections so that each eye sees only the section that is aimed at it. Thus, each eye sees its own video (and there is thus no need to use filters). Lenses, mirrors or other optical elements can be added to the design to ease the eye focus on the screen that is close to the eyes, or to make sure that each eye only sees the appropriate screen or screen portion.
- a system and associated method for capturing panoramic and stereoscopic image data can be utilized to produce panoramic and stereoscopic videos for use in conjunction with suitable display devices, such as head-mounted virtual reality displays.
- the system utilizes a camera arrangement consisting of a plurality of camera pairs positioned to capture a 360° environment. Each camera pair comprises a left-eye camera and a right-eye camera and the camera pair planes of each camera pair substantially parallel to each other.
- the camera pairs can be arranged in levels of varying heights in order to accommodate a greater number of camera pairs, thereby enhancing the resulting image data.
- a camera arrangement for filming videos enabled for stereoscopic and panoramic effects includes a plurality of camera pairs, each one of the plurality of camera pairs comprising a left-eye camera and a right-eye camera, and each one of the left-eye cameras and each one of the right-eye cameras having a longitudinal axis, whereby the longitudinal axis of the left-eye camera and the right-eye camera of a given camera pair define a camera pair plane for each one of the camera pairs, wherein each one of the camera pair planes is substantially parallel to each other.
- a method for producing a video enabled for stereoscopic and panoramic effects comprises the steps of filming using a plurality of camera pairs, each one of the camera pairs comprising a left- eye camera and a right-eye camera, for each one of a plurality of time steps, generating a left-eye panorama by merging the images from each one of the left-eye cameras at the each one of the plurality of time steps and a right-eye panorama by merging the images from each one of the right-eye cameras at the each one of the plurality of time steps, and outputting a left-eye video stream comprising the left-eye panoramas at the plurality of time steps and a right-eye video stream comprising the right-eye panoramas at the plurality of time steps, for display on a device enabled for panoramic and stereoscopic viewing.
- FIG. 1 illustrates a top plan view of a pair of cameras for stereoscopic video filming, in accordance with one embodiment of the present invention
- FIG. 2 shows a perspective view of an exemplary camera embodiment suitable for use in a camera arrangement, in accordance with one embodiment of the present invention
- FIG. 3 depicts an exemplary sound recording system with a plurality of microphone pairs
- FIG. 4 illustrates a top plan view of an arrangement of camera pairs for stereoscopic and panoramic video filming, in accordance with one embodiment of the present invention
- FIGS. 5A, 5B, 5C and 5D are respectively front, rear and side elevation views and a top plan view illustrating an arrangement of camera pairs for stereoscopic and panoramic video filming, in accordance with one embodiment of the present invention
- FIG. 6 is a flow diagram of an exemplary method for producing a stereoscopic and panoramic video, in accordance with one embodiment of the present invention.
- FIG. 7 is a flow diagram of an exemplary method for producing a stereoscopic and panoramic video, in accordance with one embodiment of the present invention.
- FIG. 8 is a block diagram of an exemplary computing environment in which the embodiments of the present invention may be implemented.
- FIG. 1 a pair of cameras (or camera pair), indicated generally by reference numeral 100, for use in stereoscopic video filming, in accordance with an embodiment of the present invention.
- the camera pair 100 of Figure 1 comprises two cameras 110, 120, each camera for producing images that are associated with each eye of the viewer. It is therefore convenient to identify these cameras as a left-eye camera 1 10 and a right-eye camera 120, wherein each camera 1 10, 120 of the camera pair 100 is capable of filming content that will be presented to the corresponding human eye of the viewer (being the left eye and the right eye, respectively).
- the left-eye camera 1 10 is longitudinally oriented along a left-eye camera axis 1 12, whereas the right-eye camera 120 is longitudinally oriented along a right-eye camera axis 122.
- each one of the left-eye cameras and the right-eye cameras have a longitudinal axis, whereby the longitudinal axis of the left-eye camera 1 10 and the longitudinal axis of the right- eye camera 120 of a given camera pair 100 define a camera pair plane wherein each one of the camera pair planes is substantially parallel to each other. Note that when the images captured by the camera pair 100 are displayed to the viewer, they will convey to the viewer a sense of depth due to the difference in the pictures provided to both eyes and produced by the two cameras 110, 120 of the camera pair 100.
- the left-eye camera 1 10 and the right-eye camera 120 are positioned parallel to one another.
- the left-eye camera axis 1 12 and the right-eye camera axis 122 need to be substantially parallel.
- the expression “substantially parallel” describes the configuration in which the axes are not strictly parallel (in a geometric sense) but more or less appear to be parallel since they differ by a few degrees or less. It should also be understood that both cameras 110, 120 of the camera pair 100 need to point in the same direction, or substantially the same direction.
- a central axis 132 of the camera pair 100 is defined as the spatial mean between the left-eye camera axis 1 12 and the right-eye camera axis 122.
- the central axis 132 should therefore be parallel, or otherwise substantially parallel, to both the left-eye camera axis 1 12 and the right-eye camera axis 122.
- the left-eye camera axis 1 12 and the right- eye camera axis 122 define a camera pair plane 1 15 in space. This plane 1 15 further contains the central axis 132.
- the left-eye camera 1 10 and the right-eye camera 120 are positioned side-by-side, with a spatial separation between them (defined as a camera lateral offset 130) should therefore be substantially constant along the central axis 132.
- the value of the camera lateral offset 130 i.e. the distance between the cameras 1 10 and 120
- the average distance between the centers of the eyes of a human is about 6.5cm (If the camera lateral offset 130 is around this value, the final video will display more realistic depth effects).
- Other values of the camera lateral offset 130 can be used if distortions or special effects are desired, for example, for artistic purposes.
- Each one of the left-eye camera 1 10 and the right-eye camera 120 films its own field of view, respectively a left-eye field of view 1 1 1 and a right-eye field of view 121 (each field of view is generally indicated in Figure 1). Normally, both fields of view partially overlap (as further indicated in Figure 1). Objects that are positioned at the extreme left and at the extreme right are normally viewed only in the left-eye field of view 111 and the right-eye field of view 121, respectively. Objects that are positioned in and around the center (i.e.
- Each camera 110, 120 is further characterized by a rear nodal point (1 14, 124) positioned substantially along a longitudinal axis of each camera 1 10, 120.
- a left nodal point 1 14 on the left-eye camera 1 10 and a right nodal point 124 on the right-eye camera 120 A spatial mean of these two nodal points 1 14, 124 defines an average nodal point, indicated by reference numeral 119.
- the average nodal point 119 is positioned substantially along the central axis 132.
- the nodal point of a lens system is usually in the objective or close to the objective.
- FIG. 2 there is shown an example embodiment of a camera 200 suitable for use in the camera arrangements of the present invention.
- a camera is an object that is well known in the art, and the camera parts detailed herein are presented for information purposes only.
- the cameras used in the camera arrangements of the present invention including the cameras used in each camera pair 100 can take many forms, and need not adhere to the form depicted in Figure 2.
- the camera 200 depicted in Figure 2 comprises an objective 220 (also known as a photographic objective), which is used for gathering light coming from the objects to the film 250.
- the objective 220 comprises a lens system 225 for image formation, thereby forming a filmed image 255 from the object to film 250.
- the image 255 forms on a light detector 230 (for example a charge-coupled device (CCD) or a film, although other technologies may exist).
- the light receiver is usually in a closed chamber 210 that prevents the surrounding light to cause noisy conditions for the image detector 230.
- the CCD (or other image receiving device) has to refresh at a given rate and send the accumulated information for one image elsewhere. If a film is used, it has to roll in order to present a different part of the film each time a new image 255 is taken.
- a memory 280 has to be used for storing the image information at each time step.
- the film itself or a magnetic cassette can both be a memory, but in the more common cases wherein images 255 are taken by a CCD or other electronic devices, an electronic memory is used.
- the memory 280 can be in the camera 200, but data may also be sent to an external memory.
- a sound recorder can be included either within the camera or outside of the camera, such that the recorded sound can be played along with the video during viewing and wherein the sound is in sync with the images.
- Figure 3 depicts an example sound recording system with a plurality of ear-shaped microphones in pairs, which could be used to record sound. Still referring to Figure 3, there is shown a sound recording system 290 adapted for binaural recording, meaning that microphones are placed in pairs. In order to fit with the plurality of camera pairs that is described below in reference with Figure 4, a plurality of microphone pairs can be provided.
- the microphones assembled in a pair or in a plurality or pairs can have a shape sculpted as an ear pinna.
- a sound recording system 290 optionally comprises a plurality of microphone pairs positioned in a horizontal plane and having an ear pinna shape, as depicted in Figure 3.
- the sound recording system 290 can be placed under the camera arrangement for recording the sound simultaneously while the cameras are filming.
- FIG. 4 there is shown a camera arrangement 300 for filming images that can be treated to provide both stereoscopic and panoramic features, in accordance with one embodiment of the present invention.
- a plurality of camera pairs 100 is employed. As previously indicated, the nodal point 350 of the cameras 1 10, 120 of each camera pair 100 should substantially coincide in order to minimize artifacts in the panorama.
- each camera pair 100 is characterized by a camera pair plane 1 15, which is defined as the plane containing the longitudinal axis of each camera 110, 120 in the camera pair 100.
- each camera pair 100 has a camera pair plane 115 which is substantially parallel to the camera pair plane 115 of the other camera pairs 100. For example, if a camera pair 100 is horizontal (or substantially horizontal), all camera pairs 100 would have to be horizontal (or substantially horizontal). If a camera pair 100 is inclined, then all camera pairs 100 would have to be inclined in a similar fashion so that the camera pair plane 115 is parallel (or substantially parallel) to the camera pair plane 1 15 of each of the other camera pairs 100.
- camera pairs 100 can be placed on levels of different heights.
- the use of more than one level allows a greater number of camera pairs 100 to be used.
- Figure 4 shows two levels 371, 372, persons of skill in the art will recognize that more than two levels can be used.
- each level doubles the number of camera pairs (provided that each level comprises the same number of camera pairs). Note that it is preferred that levels not be multiplied or separated by a large distance, because any significant vertical offset between camera pairs 100 can introduce artifacts or other defaults into the resulting image data. Further, it is preferred that the levels coincide with the camera pair plane 1 15 of their constituting camera pairs - the first level 371 being parallel to the second level 372 (Deviations can exist, but they introduce artifacts or defects into the image data which can be difficult, or even impossible, to correct in post-production).
- each one of the two levels 371, 372 comprises four camera pairs 100, wherein each camera pair is oriented approximately at a right angle with respect to its neighboring camera pairs 100 on the other level. Because there are only eight camera pairs 100 in the Figure 4 embodiment, each camera pair 100 has to cover at least 45° (thereby covering a whole circle (360°)) In this regard, note that in the Figure 4 embodiment, while each of the four camera pairs 100 on the first level 371 is positioned approximately 90° apart, and each of the four camera pairs 100 on the second level 372 is positioned approximately 90° apart, each camera pair 100 on the second level 372 is displaced approximately 45° from the next closest camera pair 100 on the first level 371. If the cameras (and resulting camera pairs) are smaller than the scale shown in Figure 4, then they can all be placed on the same level (e.g. eight camera pairs (or more) could be placed on the same level).
- the spatial mean of the nodal point of each camera 1 10, 120 (of each camera pair 100) has to be close from the spatial mean of the nodal point of each camera in the neighboring camera pair.
- the arrangement 300 needs to be compact so that when stereoscopic camera pairs 100 are used, the average nodal point of each camera pair 100 is not too far from the average nodal point 1 19 of each adjacent camera pair 100.
- a vertical camera 360 (or pair of cameras) can be positioned above (or below) the center 350 of the arrangement 300.
- the vertical camera 360 provides images that complete the top of the images in the resulting video (this is important in the event the uppermost camera pairs 100 cannot film completely 180° vertically.
- the vertical camera 360 does not have to be a part of a camera pair, since in many cases, it is not necessary to introduce stereoscopic effects in this direction (and accordingly, in these cases, a single camera pointing toward upward will suffice).
- FIG. 5A to 5D An alternate embodiment of the camera arrangement 300 of the present invention is illustrated in Figures 5A to 5D.
- the camera pairs 100 as described above, are arranged in a compact arrangement of five levels.
- the front portion (illustrated in Figure 5A) can point toward that object, with only one camera pair 100 being on the upper level and filming in the front direction.
- the viewer watches the resulting film (using an appropriate head-mounted display device) in the front facing direction and then turns his or her head right or left away from the principal object, the new view that is displayed has been filmed by camera pairs which were on lower level (since the front direction is filmed by one camera pair positioned at the highest level). Therefore, when the viewer turns his or her head away from the front facing direction, the viewer's viewpoint will orient slightly downward.
- the single vertical camera 360 is shown, and the central axis 350 (not shown) of the camera arrangement 300 is substantially centered upon the vertical camera 360, as in the Figure 4 embodiment.
- the system of the present invention can accommodate a plurality of camera
- the plurality of camera pairs define a horizontal plane that is substantially parallel with the horizontal plane of other camera pairs (assuming that vertical consistency is important to the filmmaker), and further provided that the average nodal point of a camera pair is not too far from the average nodal point of the neighboring camera pairs (which also means that the difference in the number of levels between neighboring camera pairs is low).
- the precise camera arrangement will therefore depend on the requirements such as available space (e.g. the need for a compact arrangement) and the need for vertical consistency between camera pairs (i.e. not too many levels, for example if verticality matters, as for environments that contain subjects close to the camera).
- An example of a suitable camera for use in the camera arrangement of the present invention is a camera with a small size (such as a 3 cm camera).
- these small cameras have a resolution of 1920x1080 and a refreshment rate of 60 images/s. Smaller cameras can also be used, but their resolution is generally weaker. Additional considerations to be taken into account when selecting a suitable camera include desired video quality, camera cost, the need for external camera components, such as external lenses and external memory, and the maximum size of the video file, among other factors.
- Camera lenses can be adapted to provide a very large field of view (up to 180°).
- each camera can be turned from a right angle (approximately). For example, if the camera has a field of view with a horizontal/vertical ratio of 16:9, the vertical and horizontal axes of the view can be interchanged so that the vertical portion of the environment is better captured. In the horizontal plane, the loss of field of view is compensated by the higher number of cameras.
- FIG. 6 there is shown a flow diagram of a method for producing a stereoscopic and panoramic video using the camera arrangement 300 described above, in accordance with one embodiment of the present invention.
- the method depicted in Figure 6 includes a first step 410 of filming using a plurality of camera pairs arranged in accordance with a camera arrangement defined by the present invention, to produce a plurality of video streams, wherein each video stream corresponds to a particular camera. While filming, the camera arrangement may remain static, or may be moved, depending on the filmmaker's preferences.
- the second step 420 involves the production of a video having panoramic and stereoscopic effects by generating a left-eye panorama and a right- eye panorama for each time point (or time step).
- the images taken by the left-eye camera 1 10 of each camera pair 100 are assembled and merged together using a software tool specialized in panorama generation, in order to produce a left-eye panorama.
- a software tool specialized in panorama generation in order to produce a left-eye panorama.
- the constanting camera arrangement 300 is similar to the embodiment shown in Figure 4, then a full 360° left- eye panorama will result (Conversely, if the camera pairs 100 are arranged such that the combined field of view of the camera pairs 100 is less than 360°, then the resulting panorama will be less than 360°).
- the resulting left-eye and right-eye panoramic videos can be outputted to the head-mounted display device by conventional means for presentation to the viewer, wherein each of the left-eye panoramic video and right-eye panoramic video is presented to the corresponding eye of the viewer.
- the camera arrangements and method for filming and producing panoramic and stereoscopic videos described herein are compatible with a variety of display devices, provided that the resulting image data is properly converted (as required) to a format readable by the device.
- FIG. 7 there is illustrated a multi- step method for generating panoramic and stereoscopic videos based on the base method depicted in Figure 6.
- the method depicted in Figure 7 includes examples of additional processing steps that may be implemented, depending on the computer programs or video post- production techniques employed by the filmmaker.
- the additional steps described below usually follow the first steps 410 and 420 in which filming and panorama generation are performed.
- step 430 consists of adding effects to the video (here, colors, contrast, light and other features of a photographic nature can be changed during this step for improving the video. Special effects can also be produced).
- the generation of panoramas at step 420 may result in the generation of artifacts (principally owing to the stereoscopic arrangement of the cameras). Correcting these defects can be performed at step 430.
- step 440 is performed, which comprises treating the videos for generating stereoscopic effects.
- the treatment step 440 can comprise time-based multiplexing, also known as frame sequential multiplexing, in which the left-eye and right-eye images are combined, wherein the eye side of the image alternates for each successive frame of the video (For example, a left-eye panorama is first presented, then right, left, right, and so on).
- the display device should include means (such as polarizing filters, wavelength-selective filters, IR/radio-activated filters, etc.) to block right-eye panorama of a given frame of the video to be seen by the left eye, and vice versa.
- frames may be cut in two, thereby presenting half of the screen to the left eye with a part of the left-eye panorama, and presenting the other half of the screen to the right eye with a part of the right-eye panorama (proportions may change).
- the screen may be divided in two along a horizontal axis (top/bottom multiplexing) or a vertical axis (side-by-side multiplexing). Again, integrated optical filters are used so that each eye of the viewer does not see the panoramas that need to be seen by the other eye, in order to provide stereoscopic effects.
- step 440 Other techniques exist and should be considered as possible variants of step 440, which basically consist of treating the video to produce 3D effects therein.
- the software tool UnityTM which usually allows creating video games for 3D applications used by head-mounted display devices, can be used to perform step 440.
- the head-mounted display device may include an associated software development tool (SDK) that provides specifications of the display device and functions to be used therewith.
- SDK software development tool
- the SDK can determine the tool to be used to perform step 440 respecting the specifications of the display device in question.
- Step 460 comprises converting the video file to a format having characteristics
- the software tool UnityTM can be used to transform the videos outputted in step 420 and modified in steps 430 and 440 into a file that is fully compatible with a given head-mounted display devices (and thereby enabling the functions that such a device could provide).
- the scale of the video can be modified to match the display device and the distance between the user's eyes and the screen. Distortions can appear and are preferably corrected before display. This can be performed at step 460. Final touch ups can be performed to improve stereoscopic effects (for example, by dealing with discontinuities between frames).
- This patching which can be performed at step 460, provides an alternative view of the floor under the cameras (e.g. based on the neighboring sections of the images) for replacing the blackout section of the view resulting from the fact that cameras cannot film themselves.
- Other features can be added to the video, including but not limited to user menus, subtitles, and film transitions.
- Patching may be performed by using a static picture of what is found under the camera arrangement 300. Accordingly, after filming, a static picture of what is under the arrangement 300 is taken and merged with the bottom of the images of the video. In combination with the vertical camera, not only is a hemispheric video produced, but a complete spherical video too.
- Step 460 can be performed in real time, in which case the input from the head-mounted display device needs to be received to know if the head of the viewer changed its orientation. It also means that the software tool developed for step 460 has to be fast enough so that, at each time point (or time step), the head orientation is determined in order to display images accordingly. Step 460 is thus the step which determines which portion of the
- step 460 panoramic/hemispheric/spherical images have to be displayed to the viewer.
- Sound treatment can be performed at this step too, so that the sound heard by the viewer corresponds to what should be heard if the head was oriented in the same way in the environment that was filmed, based on the sound that was recorded. This is why a sound recording system comprising a plurality of microphone pairs is useful. If it is possible, some tasks described as part of step 460 (like correction of images) can be performed earlier (steps 430 and 440) instead to ensure the best speed of treatment at step 460 which is performed repeatedly.
- step 450 which consists of detecting the head orientation, needs to happen before step 460.
- Step 450 is thus performed repeatedly in order to select, treat and display the right portion of the panoramic images each time the user moves the head. This repetition is illustrated as a loop in Figure 7.
- Detecting the head orientation can also be used when a menu is displayed to select an option of the menu using the head orientation, or to trigger a transition to a particular scene, for example, when the viewer has a head orientation pointing toward a particular object in the video.
- steps 430, 440 and 460, or parts of these steps can be intertwined.
- some software products have built-in functionalities that can achieve more than one of the actions that are related to steps 430, 440 and 460, and the order can be modified accordingly.
- the final video with panoramic and stereoscopic effects can be displayed to the user with the appropriate device at step 490.
- the video can comprise a spherical representation of the reality, enabling a view of 360° horizontally and 180° vertically
- the solid angle which can be viewed is less than 4 pi steradians. If the view is purely hemispheric, the solid angle is only 2 pi steradians. The maximum solid angle for viewing corresponds to what has been filmed.
- panoramic and stereoscopic videos may be filmed advantageously during sports events, music shows and other events in which a viewer would appreciate the sensation of being in the action without being physically present.
- the panoramic and stereoscopic videos may also be used for a realistic contemplation of natural or architectural environments, thereby bringing beauty to the eyes of a viewer along with realistic depth sensations, with no displacement.
- a suitable head-mounted display device which displays a video produced in accordance with the systems and methods described herein, it is possible to experience the action that is filmed. For example, if a show is filmed by a camera arrangement 300 placed on a stage, users viewing the video using a suitable head-mounted display device would be able to view the show in any direction they wish.
- the camera arrangement 300 is placed in an interesting building, such as a museum or other tourist attraction, viewers can look in every direction as they turn their head in a given direction. They can see what they want, and they see the movement of visitors around them because they were filmed and not only photographed. [0071] Similarly, if the camera arrangement 300 is placed in the street, viewers wearing the head-mounted display device and watching to the resulting video could view the street as if they were there.
- the video produced by the camera arrangement 300 would allow a user to view a panoramic and hemispheric video (not only a static picture) of the street, in which pedestrians and cars would be moving, additionally providing stereoscopic effects.
- FIG. 8 illustrates a generalized example of a suitable computing environment 600 in which the described embodiments of the present invention may be implemented.
- the computing environment 600 is not intended to suggest any limitation as to scope of use or functionality, as the techniques and tools may be implemented in diverse general-purpose or special-purpose computing environments.
- the computing environment 600 includes at least one CPU 610 and associated memory 620 as well as at least one GPU or other co-processing unit 615 and associated memory 625 (used for example for video acceleration).
- this most basic configuration 630 is included within the dashed line.
- the processing unit 610 executes computer- executable instructions and may be a real or a virtual processor. In a multi-processing system, multiple processing units execute computer-executable instructions to increase processing power.
- a host encoder or decoder process offloads certain computationally intensive operations to the GPU 615.
- the memory 620,125 may be volatile memory (e.g., registers, cache, RAM), nonvolatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of the two.
- the memory (620, 625) stores software 680 for a decoder implementing one or more of the decoder innovations described herein.
- a computing environment may have additional features.
- the computing environment 600 includes storage 640, one or more input devices 650, one or more output devices 660, and one or more communication connections 670.
- An interconnection mechanism such as a bus, controller, or network interconnects the components of the computing environment 600.
- operating system software provides an operating environment for other software executing in the computing environment 600, and coordinates activities of the components of the computing environment 600.
- the storage 640 may be removable or non-removable, and includes magnetic disks, magnetic tapes or cassettes, CD-ROMs, DVDs, or any other medium which can be used to store information and which can be accessed within the computing environment 600.
- the storage 640 stores instructions for the software 680.
- the input device(s) 650 may be a touch input device such as a keyboard, mouse, pen, or trackball, a voice input device, a scanning device, or another device that provides input to the computing environment 600.
- the input device(s) 650 may be a sound card, video card, TV tuner card, or similar device that accepts audio or video input in analog or digital form, or a CD-ROM or CD-RW that reads audio or video samples into the computing environment 600.
- the output device(s) 660 may be a display, printer, speaker, CD- writer, or another device that provides output from the computing environment 600.
- the communication connection(s) 670 enable communication over a communication medium to another computing entity.
- the communication medium conveys information such as computer-executable instructions, audio or video input or output, or other data in a modulated data signal.
- a modulated data signal is a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.
- communication media include wired or wireless techniques implemented with an electrical, optical, RF, infrared, acoustic, or other carrier.
- Computer-readable media are any available media that can be accessed within a computing environment.
- Computer-readable media are any available media that can be accessed within a computing environment.
- computer readable media include memory 620, storage 640, communication media, and combinations of any of the above.
- program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform particular tasks or implement particular abstract data types.
- the functionality of the program modules may be combined or split between program modules as desired in various embodiments.
- Computer- executable instructions for program modules may be executed within a local or distributed computing environment.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
Abstract
Description
Claims
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201361915873P | 2013-12-13 | 2013-12-13 | |
PCT/CA2014/000895 WO2015085406A1 (en) | 2013-12-13 | 2014-12-15 | Systems and methods for producing panoramic and stereoscopic videos |
Publications (2)
Publication Number | Publication Date |
---|---|
EP3080986A1 true EP3080986A1 (en) | 2016-10-19 |
EP3080986A4 EP3080986A4 (en) | 2017-11-22 |
Family
ID=53370402
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP14869343.5A Withdrawn EP3080986A4 (en) | 2013-12-13 | 2014-12-15 | Systems and methods for producing panoramic and stereoscopic videos |
Country Status (3)
Country | Link |
---|---|
EP (1) | EP3080986A4 (en) |
CA (1) | CA2933704A1 (en) |
WO (1) | WO2015085406A1 (en) |
Families Citing this family (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
DE102015118997A1 (en) * | 2015-11-05 | 2017-05-11 | Berliner Kta Shareholder Gmbh | Camera mount for stereoscopic panorama shots |
WO2017115349A1 (en) * | 2016-01-03 | 2017-07-06 | Humaneyes Technologies Ltd. | Stitching frames into a panoramic frame |
CN109076200B (en) | 2016-01-12 | 2021-04-23 | 上海科技大学 | Method and device for calibrating panoramic stereo video system |
EP3196838A1 (en) * | 2016-01-25 | 2017-07-26 | Nokia Technologies Oy | An apparatus and associated methods |
EP3206082A1 (en) | 2016-02-12 | 2017-08-16 | Vroxtechnology B.V. | System, method and computer program for recording a non-virtual environment for obtaining a virtual representation |
CN107316273B (en) * | 2016-04-27 | 2021-05-18 | 深圳看到科技有限公司 | Panoramic image acquisition device and acquisition method |
US10447993B2 (en) | 2016-09-27 | 2019-10-15 | Laduma, Inc. | Stereoscopic 360 degree digital camera systems |
WO2018067680A1 (en) * | 2016-10-05 | 2018-04-12 | Hidden Path Entertainment, Inc. | System and method of capturing and rendering a stereoscopic panorama using a depth buffer |
US10237537B2 (en) | 2017-01-17 | 2019-03-19 | Alexander Sextus Limited | System and method for creating an interactive virtual reality (VR) movie having live action elements |
CN115222793A (en) * | 2017-12-22 | 2022-10-21 | 展讯通信(上海)有限公司 | Method, device and system for generating and displaying depth image and readable medium |
US10955657B2 (en) * | 2018-12-20 | 2021-03-23 | Acclarent, Inc. | Endoscope with dual image sensors |
DE102019101192A1 (en) * | 2019-01-17 | 2020-07-23 | Innogy Se | Camera system and home automation system with one camera system |
CN110446021A (en) * | 2019-03-23 | 2019-11-12 | 杨秀海 | A kind of implementation method of mobile phone and its dual camera |
CN116386520B (en) * | 2023-05-29 | 2023-09-19 | 长春希达电子技术有限公司 | Spherical inner display screen correction method and device, electronic equipment and storage medium |
Family Cites Families (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
IL136128A0 (en) * | 1998-09-17 | 2001-05-20 | Yissum Res Dev Co | System and method for generating and displaying panoramic images and movies |
US20040001138A1 (en) * | 2002-06-27 | 2004-01-01 | Weerashinghe W.A. Chaminda P. | Stereoscopic panoramic video generation system |
GB2444533B (en) * | 2006-12-06 | 2011-05-04 | Sony Uk Ltd | A method and an apparatus for generating image content |
US20080298674A1 (en) | 2007-05-29 | 2008-12-04 | Image Masters Inc. | Stereoscopic Panoramic imaging system |
WO2011037964A1 (en) * | 2009-09-22 | 2011-03-31 | Tenebraex Corporation | Systems and methods for correcting images in a multi-sensor system |
US20130016181A1 (en) * | 2010-03-30 | 2013-01-17 | Social Animal Inc. | System and method for capturing and displaying cinema quality panoramic images |
US8896671B2 (en) * | 2010-04-09 | 2014-11-25 | 3D-4U, Inc. | Apparatus and method for capturing images |
US8581961B2 (en) | 2011-03-31 | 2013-11-12 | Vangogh Imaging, Inc. | Stereoscopic panoramic video capture system using surface identification and distance registration technique |
EP2695385B8 (en) | 2011-04-08 | 2016-03-16 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Capturing panoramic or semi-panoramic 3d scenes |
-
2014
- 2014-12-15 WO PCT/CA2014/000895 patent/WO2015085406A1/en active Application Filing
- 2014-12-15 EP EP14869343.5A patent/EP3080986A4/en not_active Withdrawn
- 2014-12-15 CA CA2933704A patent/CA2933704A1/en active Pending
Also Published As
Publication number | Publication date |
---|---|
WO2015085406A1 (en) | 2015-06-18 |
CA2933704A1 (en) | 2015-06-18 |
EP3080986A4 (en) | 2017-11-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20160344999A1 (en) | SYSTEMS AND METHODs FOR PRODUCING PANORAMIC AND STEREOSCOPIC VIDEOS | |
WO2015085406A1 (en) | Systems and methods for producing panoramic and stereoscopic videos | |
US11575876B2 (en) | Stereo viewing | |
CN108337497B (en) | Virtual reality video/image format and shooting, processing and playing methods and devices | |
US20150358539A1 (en) | Mobile Virtual Reality Camera, Method, And System | |
US7660472B2 (en) | System and method for managing stereoscopic viewing | |
CN106796390A (en) | For the camera apparatus with big visual field of three-dimensional imaging | |
WO2003081921A1 (en) | 3-dimensional image processing method and device | |
US10631008B2 (en) | Multi-camera image coding | |
CN108141578A (en) | Camera is presented | |
JP2013514004A (en) | 3D recording and display system using proximal and distal focused images | |
US9161018B2 (en) | Methods and systems for synthesizing stereoscopic images | |
Naimark | Elements of real-space imaging: a proposed taxonomy | |
CA2112470A1 (en) | Improved integral virtual reality and/or image recording, projection-visualization-listening system | |
KR101686634B1 (en) | layered Hologram System using Auto-stereoscopic 3D image | |
KR101433082B1 (en) | Video conversing and reproducing method to provide medium feeling of two-dimensional video and three-dimensional video | |
Mayhew et al. | Parallax scanning methods for stereoscopic three-dimensional imaging | |
Perisic | A New Single Camera System for Stereoscopic Image Acquisition | |
CN109729335A (en) | A kind of method, apparatus and system of parallax adjustment | |
US20140267635A1 (en) | Enhanced 3D Display Method and System | |
KR20160095500A (en) | 3D Hologram System using Auto-stereoscopic 3D image |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
17P | Request for examination filed |
Effective date: 20160711 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
DAX | Request for extension of the european patent (deleted) | ||
RIC1 | Information provided on ipc code assigned before grant |
Ipc: H04N 13/02 20060101AFI20170607BHEP Ipc: G03B 37/00 20060101ALI20170607BHEP Ipc: H04N 5/247 20060101ALI20170607BHEP Ipc: G03B 35/08 20060101ALI20170607BHEP |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G03B 37/00 20060101ALI20170911BHEP Ipc: H04N 13/02 20060101AFI20170911BHEP Ipc: G03B 35/08 20060101ALI20170911BHEP Ipc: H04N 5/247 20060101ALI20170911BHEP |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: H04N 5/247 20060101ALI20171009BHEP Ipc: G03B 37/00 20060101ALI20171009BHEP Ipc: G03B 35/08 20060101ALI20171009BHEP Ipc: H04N 13/02 20060101AFI20171009BHEP |
|
A4 | Supplementary search report drawn up and despatched |
Effective date: 20171019 |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: H04N 13/02 20060101AFI20171013BHEP Ipc: G03B 35/08 20060101ALI20171013BHEP Ipc: H04N 5/247 20060101ALI20171013BHEP Ipc: G03B 37/00 20060101ALI20171013BHEP |
|
17Q | First examination report despatched |
Effective date: 20190111 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN |
|
18D | Application deemed to be withdrawn |
Effective date: 20190522 |