US9799097B1 - Method and system for defining a virtual reality resolution distribution - Google Patents
Method and system for defining a virtual reality resolution distribution Download PDFInfo
- Publication number
- US9799097B1 US9799097B1 US15/156,069 US201615156069A US9799097B1 US 9799097 B1 US9799097 B1 US 9799097B1 US 201615156069 A US201615156069 A US 201615156069A US 9799097 B1 US9799097 B1 US 9799097B1
- Authority
- US
- United States
- Prior art keywords
- virtual reality
- resolution
- reality content
- content
- user
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
- 238000000034 method Methods 0.000 title claims abstract description 105
- 238000009826 distribution Methods 0.000 title claims abstract description 66
- 238000012545 processing Methods 0.000 claims abstract description 54
- 230000033001 locomotion Effects 0.000 claims description 16
- 238000013507 mapping Methods 0.000 claims description 15
- 230000000007 visual effect Effects 0.000 claims description 15
- 238000003860 storage Methods 0.000 claims description 8
- 208000029257 vision disease Diseases 0.000 claims description 8
- 230000004393 visual impairment Effects 0.000 claims description 8
- 230000001225 therapeutic effect Effects 0.000 claims description 4
- 210000005252 bulbus oculi Anatomy 0.000 claims description 3
- 239000003086 colorant Substances 0.000 claims description 3
- 230000005540 biological transmission Effects 0.000 abstract description 14
- 230000003044 adaptive effect Effects 0.000 abstract description 7
- 230000004438 eyesight Effects 0.000 abstract description 3
- 238000005457 optimization Methods 0.000 abstract description 3
- 230000006870 function Effects 0.000 description 43
- 230000008569 process Effects 0.000 description 25
- 238000007906 compression Methods 0.000 description 22
- 230000006835 compression Effects 0.000 description 22
- NCGICGYLBXGBGN-UHFFFAOYSA-N 3-morpholin-4-yl-1-oxa-3-azonia-2-azanidacyclopent-3-en-5-imine;hydrochloride Chemical compound Cl.[N-]1OC(=N)C=[N+]1N1CCOCC1 NCGICGYLBXGBGN-UHFFFAOYSA-N 0.000 description 12
- 238000006243 chemical reaction Methods 0.000 description 8
- 238000004422 calculation algorithm Methods 0.000 description 7
- 210000003128 head Anatomy 0.000 description 7
- 238000002716 delivery method Methods 0.000 description 5
- 238000007654 immersion Methods 0.000 description 5
- 238000003672 processing method Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 238000013021 overheating Methods 0.000 description 4
- 238000013459 approach Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 210000001508 eye Anatomy 0.000 description 3
- 230000004424 eye movement Effects 0.000 description 3
- 230000010354 integration Effects 0.000 description 3
- 230000035945 sensitivity Effects 0.000 description 3
- 238000002560 therapeutic procedure Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000013144 data compression Methods 0.000 description 2
- 230000001419 dependent effect Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000004886 head movement Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 208000006992 Color Vision Defects Diseases 0.000 description 1
- 208000027534 Emotional disease Diseases 0.000 description 1
- 241000282412 Homo Species 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000004931 aggregating effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 201000007254 color blindness Diseases 0.000 description 1
- 239000012141 concentrate Substances 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005315 distribution function Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000005304 joining Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 210000001747 pupil Anatomy 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 210000001525 retina Anatomy 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 208000012672 seasonal affective disease Diseases 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 238000011282 treatment Methods 0.000 description 1
- 230000004304 visual acuity Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
- G09G5/003—Details of a display terminal, the details relating to the control arrangement of the display terminal and to the interfaces thereto
- G09G5/005—Adapting incoming signals to the display format of the display terminal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/08—Projecting images onto non-planar surfaces, e.g. geodetic screens
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4092—Image resolution transcoding, e.g. by using client-server architectures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/012—Head tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/10—Geometric effects
- G06T15/20—Perspective computation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/10—Geometric effects
- G06T15/20—Perspective computation
- G06T15/205—Image-based rendering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
-
- G06T3/005—
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
- G09G5/36—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
- G09G5/39—Control of the bit-mapped memory
- G09G5/395—Arrangements specially adapted for transferring the contents of the bit-mapped memory to the screen
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/131—Protocols for games, networked simulations or virtual reality
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/52—Network services specially adapted for the location of the user terminal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/04—Indexing scheme for image data processing or generation, in general involving 3D image data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/08—Indexing scheme for image data processing or generation, in general involving all processing steps from image acquisition to 3D model generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
- G06T2219/20—Indexing scheme for editing of 3D models
- G06T2219/2016—Rotation, translation, scaling
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2340/00—Aspects of display data processing
- G09G2340/02—Handling of images in compressed format, e.g. JPEG, MPEG
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2340/00—Aspects of display data processing
- G09G2340/04—Changes in size, position or resolution of an image
- G09G2340/0407—Resolution change, inclusive of the use of different resolutions for different screen areas
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2350/00—Solving problems of bandwidth in display systems
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2370/00—Aspects of data communication
- G09G2370/16—Use of wireless transmission of display information
Definitions
- the present invention relates generally to the field of image processing in the virtual reality space and more particularly to an improved method for processing and delivering virtual reality content using spherical projections.
- virtual reality technology has rapidly increased over the last few years.
- new virtual reality applications in an array of fields including, for example, education and training, art, architectural design, movie production, advertisement, entertainment, video games, therapy, and medical fields, have been adopted or shown potential interest.
- the implementation of virtual reality in each field however often presents new challenges and requirements for the virtual reality application to be useful and functional. In resolving those challenges to meet each application's requirements, data content exponentially increases, compromising delivery reliability, feasibility, speeds and resolution.
- Virtual reality experiences require high resolution content that is reliably delivered to the user for the experience to by fully immersive. Without a fully immersive experience, users of virtual reality are unlikely to adopt and implement the virtual reality technology for their particular application. In nearly all current virtual reality applications, the constraint on resolution generally results from the bandwidth available, as opposed to any limit on content generation. Depending on the playback device, whether it is streaming content or accessing locally stored content, and the application, additional constraints also result from the image processing and resolution due to limitations of the central processing unit (CPU), memory, screen maximum resolution, graphics processing unit (GPU) and storage capacity, power consumption, and overheating of the electronic device. These constraints in most cases decrease the level of immersion for the user and greatly compromise the reliability needed for some applications.
- CPU central processing unit
- GPU graphics processing unit
- This method by its own solution does not encode the full 360° of content, but rather only transmits and dewarps a rectangular subset of the data, irrecoverably limiting the delivered content. Because the content is irrevocably limited to only the subset transmitted, the viewer will experience dislocated views outside the transmitted region, disrupting immersion.
- the method changes the number of macroblocks spatially, it must be fully integrated with the compression algorithm, rather than being an independent preprocessing step.
- This method also introduces compression artifacts which appear blocky and greatly hinder immersion, especially when present in stereoscopic content, as well as restricting the final resolution to be constant within all individual macroblocks.
- a common fundamental problem of the aforementioned compression methods is that traditional compression algorithms have hardware support for encoding and decoding on appropriate devices, so trying to force adoption of new compression algorithms is challenging.
- this can be accomplished by enabling one or more of: using desired resolution distribution of projections in conjunction with adaptive focus, selectively transmitting content according to the view direction, converting spherical virtual reality content to rectangular content for encoding and/or compression, dynamically sending resolution that does not exceed the max DPI for the screen, using crowd sourced data to control the projections and/or transmission of data and lowering bandwidth requirements.
- an exemplary system for processing and delivering virtual reality content including: one or more computer servers comprising one or more processors and one or more storage devices and executable software stored on at least one of the one or more storage devices and executable on demand.
- the software operative with at least one of the one or more processors to cause the one or more computer servers to: select one or more sets of parameters affecting the virtual reality content's resolution when inversely projected to a three-dimensional shape; define a desired resolution distribution using at least some of the one or more sets of parameters selected, wherein the desired resolution distribution is able to counteract all or some constraints corresponding to the selected one or more sets of parameters; create a two-dimensional resolution-defined projection according to the desired resolution distribution; map the virtual reality content to the two-dimensional resolution-defined projection spaces; derive one or more two-dimensional renditions from the mapped virtual reality content to the two-dimensional resolution-defined projection spaces; and access the one or more two-dimensional renditions for display by a device configured to inversely project the virtual reality image data to a sphere or other three-dimensional shape.
- At least one of the sets of parameters can correspond to a virtual reality content capturing set up and the software can be additionally operative with at least one of the one or more processors to cause the one or more computer servers to: match the virtual reality content's pixel resolution resulting from the parameters corresponding to the virtual content capturing set up to a spatial resolution that can be used to define the desired, resolution distribution.
- the parameters corresponding to the virtual content capturing set up can include, for example, one or more of a physical arrangement of image capturing devices, lens properties, and stitching, methods, in additional embodiments, at least one of the sets of parameters may, additionally or alternatively, correspond to a user's virtual reality device and the software is additionally operative with at least one of the one or more processors to cause the one or more computer servers to: match the virtual reality content's pixel resolution resulting from the parameters corresponding to the user's virtual reality device to a spatial resolution that can be used to define the desired resolution distribution.
- the parameters corresponding to user's virtual reality device can include one or more of a display's specifications, power consumption, connection properties, and processing speeds.
- At least one of the sets of parameters may, additionally or alternatively, correspond to a user's visual capacity and the software can be additionally operative with at least one of the one or more processors to cause the one or more computer servers to: match the virtual reality content's pixel resolution resulting from the parameters corresponding to the user's visual capacity to a spatial resolution that can be used to define the desired resolution distribution.
- the parameters corresponding to the user's visual capacity can include, for example, one or more of a viewing direction, a fovea field of view, eyeball movement, and visual impairments. Where the visual impairments are accounted for, the desired resolution distribution implemented can correct or provide a therapeutic visual effect to the user, for example.
- At least one of the sets of parameters may, additionally or alternatively, correspond to crowdsourced data and the software can be additionally operative with at least one of the one or more processors to cause the one or more computer servers to: match the virtual reality content's pixel resolution resulting from the parameters corresponding to the crowdsourced data to a spatial resolution that can be used to define the desired resolution distribution.
- the sets of parameters corresponding to feedback loop data defining one or more user's preferences and the software can be additionally operative with at least one of the one or more processors to cause the one or more computer servers to: match the virtual reality content's pixel resolution resulting from the parameters corresponding to the feedback loop data defining one or more user's preferences to a spatial resolution that can be used to define the desired resolution distribution.
- At least one of the sets of parameters correspond to content metrics and the software may be additionally operative with at least one of the one or more processors to cause the one or more computer servers to: match the virtual reality content's pixel resolution resulting from the parameters corresponding to the content metrics to a spatial resolution that can be used to define the desired resolution distribution.
- the parameters corresponding to the content metrics can include one or more of virtual reality content contrast, virtual reality content detected movement, and colors contained in the virtual reality content.
- the one or more image capturing devices may be any image capturing device that is enabled to capture all or a portion of the immersive image content.
- the image capturing devices can include 360° image capturing devices.
- one or more of the capturing devices can include computers enabled to capture immersive content from a computer generated environment (i.e., in virtual space).
- the generating of the one or more renditions can include creating a two-dimensional resolution-defined projection according to a desired resolution distribution that is modified during playback of the virtual reality content according to the retrieved parameters.
- one or both of lowering the resolution and changing parameters of at least one of the one or more renditions is a function of the modified desired resolution distribution.
- a method corresponding to the exemplary system for processing and delivering virtual reality content including: selecting one or more sets of parameters affecting the virtual reality content's resolution when inversely projected to a three-dimensional shape; defining a desired resolution distribution using at least some of the one or more sets of parameters selected, wherein the desired resolution distribution is able to counteract all or some constraints corresponding to the selected one or more sets of parameters; creating a two-dimensional resolution-defined projection according to the desired resolution distribution; mapping the virtual reality content to the two-dimensional resolution-defined projection spaces; deriving one or more two-dimensional renditions from the mapped virtual reality content to the two-dimensional resolution-defined projection spaces; and accessing the one or more two-dimensional renditions for display by a device configured to inversely project the virtual reality image data to a sphere or other three-dimensional shape.
- Additional general optional features can additionally include, for example, predicting and/or identifying resolution distributions to control processing according to a change to take place, or be encouraged, in the viewing direction and/or orientation of the user.
- Other aspects of the invention will be understood by those of skill in the art upon review of the teachings herein.
- Other aspects of the invention may involve combinations of the above noted aspects of the invention.
- These other aspects of the invention may provide various combinations of the aspects presented above as well as provide other configurations, systems, functional relationships, and processes that have not been specifically set forth above.
- FIG. 1 is a schematic of an exemplary system that can be used according to aspects of the present disclosure
- FIG. 2 is a schematic of an exemplary arrangement of functional components that can be used according to aspects of the present disclosure
- FIG. 3 is a flowchart illustrating method steps that may be implemented for the processing and delivery of virtual reality content according to some aspects of the disclosure
- FIG. 4 is a flowchart illustrating method steps that may be implemented to generate a resolution-defined projection according to some aspects of the disclosure
- FIG. 5 is a schematic representation illustrating the method steps of FIG. 4 ;
- FIG. 6 is a flowchart illustrating method steps that may be implemented to transmit virtual reality content according to additional aspects of the disclosure
- FIG. 7 is a schematic representation illustrating the method steps of FIG. 6 ;
- FIG. 8 is a flowchart illustrating method steps that may be implemented to process and deliver virtual reality content according to yet additional aspects of the disclosure
- FIG. 9 is a schematic representation illustrating the method steps of FIG. 8 .
- Adaptive focus can refer to the optimized processing and/or transmission of a selected different virtual reality image(s) that may at least partially dependent on a desired resolution distribution.
- the desired resolution distribution may depend on a fovea field of view for the corresponding viewer's direction so that the resolution can be best for whichever direction the viewer is looking, contrast of the virtual reality image data, movement of the virtual reality image data content, capturing camera arrangement and geometry, capturing camera lenses, display parameters, and crowd sourced data relating to the virtual reality image content.
- Crowd sourced data can refer to a collection of data collected from at least one other virtual reality user or person associated with the virtual reality content
- the data, collected from at least one other virtual reality user may be in relation to him/her viewing the virtual reality content and include, for example, data descriptive of head movement patterns (specific or non-specific to displayed content), connection, geographic location, time duration, frequency between action patterns, and the such.
- the data collected from at least one other person associated with the virtual reality content may include, for example, data collected from the at least one other person that is part of or has directly viewed the virtual reality content (either live or in the past).
- Desired resolution distribution can refer to a spatial resolution function that can be defined by one or more of the fovea field of view for a particular viewing direction, contrast of the virtual reality image data, movement of the virtual reality image data content, camera arrangement, camera lenses, and crowd sourced data.
- “Fovea field of view”, as used herein can refer to a correlation of spherical virtual reality image content and the human's field of view that is of higher visual acuity/sensitivity near the center of the retina. More specifically, about 200 degrees horizontally, of the 360 degrees of content, and 135 degrees vertically, of the 180 degrees of content, with higher resolutions, for example, at about 10 degrees from direction of focus to gradually degraded towards the periphery.
- Image content” or “image data” as used herein can include 360 content in either one or both of three-dimensional and two-dimensional formats for processing and display by virtual reality devices and, in some embodiments, immersive displays of wireless devices, smartphones, tablets, PCs, desktops, smart TVs and the such.
- the virtual reality devices or immersive displays are capable of processing selected portions of the 360 content in still images and/or video clips. Accordingly, image content or image data can include both captured still images or portions thereof and/or video.
- projection can refer to the mapping of virtual reality spherical image data to flat rectangular data.
- the projection can be, a resolution-defined projection as to enable compression and higher resolution virtual reality experiences with lower virtual reality content transmission bandwidths.
- Resolution-defined projection can refer to a projection that is defined by a resolution distribution.
- the projection may be able to adapt to a separable resolution distribution.
- the separable resolution distribution function may be, for example, a product of two different resolution functions, one for each of azimuth and elevation. Accordingly, the basis for a resolution-defined projection may be equirectangular or any bijective projection that has a separable Jacobian.
- “User specific learned behavior”, as used herein may refer to the customized delivery of virtual reality content to a user according at least in part to feedback data from one or more previous virtual reality content experiences (i.e. feedback loop data) of the user.
- “Virtual reality content” or “virtual reality image data”, as used herein refer to any one or more of “image content”/“image data” recorded, received, processed (including during encoding, storing, and compression), and transmitted, in one of a 3 dimensional spherical format or any other 3 dimensional shape, 2 dimensional rectangular format or projected (e.g., as video onto a spherical display) according to the different aspects of the process disclosed.
- a projection into a sphere or any other three dimensional shape can refer to the way the image is processed to generate an immersive image/video that can be displayed and not to the shape of the screen of the device.
- the spherical or 3D shape projecting aspect is a function of the software in generating the rendering and not of the display shape.
- Virtual reality content can be immersive because a user can be entirely surrounded by a virtual environment. As opposed to traditional image or video displayed on a screen, virtual reality immersive experiences fill every possible viewing direction. A virtual reality experience thus requires processing of a full sphere of image content. This sphere of content often must be delivered both in high resolution and over a bandwidth-constrained network(s). Compression of the content can be essential, according to aspects of the disclosure to optimally utilize existing algorithms for image and video data compression, the virtual reality input can be mapped to be flat and rectangular. Accordingly, as further described in the exemplary embodiments herein, a projection can be used along with desired resolution distributions to convert the three-dimensional (3D) spherical content in virtual reality space to the two-dimensional (2D) rectangular representation for input to encoding methods. In this description, various exemplary projections and resolution functions are further disclosed to demonstrate the inventive aspects of the resolution-defined projections useful for virtual reality content processing and delivery.
- this disclosure enables at least: (a) encoding and compression agnostic image content processing solutions; (b) process flexibility as to the projection functions that can be used to process and deliver image content; (c) solutions that can work using arbitrary resolution functions; (d) a virtual reality content processing and delivery method that can provide compression and content flexibility and lower bandwidth requirements without compromising the virtual resolution experience, for example, due to lower resolution and/or playback problems; (e) a solution to can lower the amount of image data processed to reduce power consumption and overheating of the device; (f) method that enables useful distributions of resolution, such that data is uniformly and maximally useful; (g) a method that can enable the changing of the resolution sent based on display and/or processing parameters; and (h) a virtual reality content processing and delivery method that lowers the hardware and software requirements of display devices.
- image content 1005 can be transmitted to a computer server 1010 . Transmission may take place via-wireless or wired connections automatically or upon request. For example, image content 1005 may be downloaded or streamed from one or an array of 360 image content capturing device(s).
- the system may be a distributed system with a main computer server 1010 handling the loading of the image content inputted and scheduling and/or transferring content to various processors 1010 a that can work in parallel, for example, to independently re-sample the image content and map it to new projection spaces according to the methods described herein.
- processors 1010 a can work in parallel, for example, to independently re-sample the image content and map it to new projection spaces according to the methods described herein.
- Processed image content 1010 b can then include variously projected versions in the variety of new projection spaces.
- Processors 1010 a and/or additional distributed processors 1010 c which may also work in parallel can downsample and/or encode the image content to a variety of video sizes, bitrates, and settings.
- the variously projected and variously encoded versions 1010 d may be further processed and/or arranged to be stored in database 1015 as renditions aggregated and made ready for playback on demand.
- FIG. 2 a schematic diagram 2000 including an exemplary arrangement of functional components that can be used according to aspects of the present disclosure is depicted.
- the schematic diagram 2000 showing interaction between the system components for carrying out the improved methods of processing and transmission of image content.
- variously projected and variously encoded versions of the image content 2005 can, be saved in a database managed by the provider.
- the image content 2005 can be transmitted via a network, for example, a wireless network 2010 , to an antenna 2015 capable of transmitting requests for content, along with related information for the request of specific renditions, and receiving the image content.
- the antenna 2015 can be in communication with a central processing unit 2025 that can receive the image content 2005 and include or be in connection with a video decoder 2020 enabled to process the image content and get pixel values.
- the content may be downloaded, for example, via a direct connection, or the system may be fully integrated.
- the central processing unit can receive and process one or more input(s) 2030 including, for example, from an accelerometer, gyroscope, crowdsource data, viewing parameters, or other sensor.
- Input(s) 2030 may come from sensors associated with the display device 2040 for the user 2045 or from stored parameters and preferences or data relating to prior experiences from other users (i.e. crowdsourced data).
- the display device 2040 can receive the data from a graphics processing unit 2035 in, direct or indirect, communication with the video encoder 2020 .
- Adaptive focus can refer to the optimized processing and/or transmission of a selected different virtual reality image(s) that may at least partially dependent on a viewer's direction.
- the selection of may additionally depend on a resolution distribution based on the fovea field of view for the corresponding viewer's direction, so that the resolution can be best for whichever direction the viewer is looking.
- virtual reality content can be, transmitted in a manner that is compressed to give the most immersive experience for a set of view directions.
- This approach is contrary to current virtual reality content processes that require virtual reality encoding be performed for all of the same data, regardless of view direction, thus increasing processing and transmission bandwidth requirements.
- the method steps disclosed can greatly reduce the required bandwidth for virtual reality content, even while potentially boosting the observed resolution.
- the amount of image content that needs to be processed for display is significantly reduced. By reducing this processing power consumption can also be reduced, improving reliability of the delivery and preventing overheating of the device.
- the virtual reality content delivery process disclosed in flowchart 100 can generally consist of four main parts—projection generation 110 , projection conversion 120 , encoding 130 , processing, including accessing, or transmission 135 and display 140 .
- projection generation a mapping from a sphere to a flat rectangle can be generated.
- This projection can be used to map the spherical virtual reality content to a rectangular format, in part to enable encoding with traditional algorithms such as JPEG or H.264.
- at step 112 at least one existing projection may be specified/selected to convert the 360° three-dimensional spherical content to two dimensional rectangular content.
- the specified/selected projection must have a separable Jacobian.
- the two-dimensional projection(s) specified/selected can be used to create a projection that is defined by a desired resolution distribution.
- the desired resolution distribution should describe a desired resolution function than is separable when expressed in the projection space specified/selected.
- the desired resolution distribution can refer to, a smooth two dimensional rectangular resolution distribution that generally adjusts according one or more of the fovea field of view for a particular viewing direction, contrast of the virtual reality image data, movement of the virtual reality image data content, and crowd sourced data.
- data from the captured 360° spherical content 101 , the viewing direction of the user 146 , crowdsourcing 150 , user's feedback 144 may be transmitted (shown as (A), (C) and/or (D)) to the system for use during the projection generation 110 .
- the resolution adjustments that define the projection can result at least in part from the mathematical derivation process in the generation of the new projection (the resolution-defined projection).
- a few examples, including generation of the novel Gaussian and Cauchy projections is disclosed with reference to FIGS. 2 and 3 .
- this resolution-defined projection can match the desired resolution function used to the create the projection (i.e., by first selecting the total number of pixels desired in the output, it is possible to then calculate the pixel dimensions for any projection.)
- this method could be used to generate a resolution that matches the user's FOV, encoding settings, and/or to highlight any region of the virtual reality content, for example, to guide the attention of the user, or apply warping effects to spherical content.
- Projection conversion 120 can take place once a resolution-defined projection has been created.
- the 360° spherical virtual reality content may be received at step 122 .
- the 360° spherical virtual reality content can be applied to the resolution-defined projection.
- the virtual reality content may then be stored in equirectangular format, although the disclosed process would support any known format regardless of the original content's projection. Taking the equirectangular format, for example, a grid of pixel points matching the dimensions of the output video may be used to map each pixel to its coordinates in the equirectangular format. Lanczos interpolation may be used to minimize projection conversion artifacts.
- the coordinate mapping can be defined by the projection equations and interpolation may be defined by the Lanczos interpolation method.
- any other image interpolation method could be used, including nearest neighbor, bilinear, or bicubic.
- This coordinate transformation and image interpolation can be processor intensive.
- the process may be parallelized, for example, by each color dimension (RGB, YCbCr, or any other color space representation) so that each color may be interpolated independently.
- each frame of a video may also be independently interpolated. In doing so, it can be important to ensure the processing time to calculate coordinate mappings and interpolation filter values runs only once for each video with specific settings. This may be done by performing the calculation and programming the specific settings before the first frame is processed, such that all frames simply lookup the coordinate mapping and filter values.
- GPU acceleration of this process may also be implemented.
- the interpolation and method steps can enable virtual reality content generation taking form as a plurality of renditions that can have much higher resolutions on a subset of the original sphere, while still serving data to cover the entire or a greater portion of the sphere.
- This is different than the less sophisticated method currently known which simply crops the spherical content to only display a subset of the sphere, leaving the majority of the sphere content-less.
- it can be advantageous to serve content to cover the entire or a greater portion of the sphere because the user could turn her/his head quicker than the latency of displaying a new video.
- the resolution distribution to enable larger content delivery without increasing the bandwidth required, if this happens, because the disclosed method still serves content everywhere almost everywhere on the sphere (although low resolution), the user can maintain some level of immersion.
- Encoding 130 of the plurality of renditions can then take place.
- Encoding 130 can include encoding two or more renditions at least one resolution and bitrate 132 .
- the renditions of virtual reality content may be encoded, for example, using a H.264 at a bitrate that matches the pixel dimensions. This too may be performed in a distributed computing setup to process a single virtual reality content video input. For example, for each rendition in terms of bitrate, dimensions, and view direction, a different processor may speed up the task of encoding many video renditions. Since the costliest operation can often be the interpolation needed for projection conversion, in parallel a downscaled version of the original input virtual reality content may be generated and made available so a version is quickly available for the uploader.
- the rest of the virtual reality content may go through conversion to a variety of projections, projection parameters, dimensions, bitrates, and view directions.
- the larger of the versions by the standard process may be generated. It is the possible to downsize and reduce the bitrate to generate any smaller versions of the same video. This can greatly reduce computation time for these smaller version as compared with generating them directly from the original equirectangular content.
- notification flags can be sent to a database when their individual task is complete so that a centralized record of the status of the conversions is available.
- the virtual reality content renditions can be grouped in sets according to one or more predetermined common properties between them.
- the renditions can be grouped in sets of versions in which only view direction changes so that a set can include all of the versions of the same virtual reality content that can be served to a user depending on their view direction. Compression of the renditions 134 may take place as part of the encoding in 132 or separately as it may be desired.
- Each of the virtual reality content sets may be split using a video display frequency (e.g. virtual reality image content included in a short duration virtual reality video).
- the sets of virtual reality content may be organized in a standard structure folder structure database(s) that can allow the displaying device to easily request a different set depending on the view direction of the user detected at step 146 .
- Head motion recorded by the virtual reality content display device and/or determined using data from the viewing direction detected, head motion can be used to dynamically select at least one of the projection versions, individually or as a data set, according to the head motion at step 136 . This dynamic selection may be done, for example, similarly to the adaptive streaming methods in which the bitrate of the video displayed adaptively matches available bandwidth. However, instead of only including video quality, viewing direction can additionally be included for the dynamic selection.
- the dynamically selected virtual reality content is delivered 142 to a virtual reality device of the user.
- Virtual reality devices can include all head-mounted devices (HMD) for virtual reality that include head tracking, and even mobile devices which can use internal accelerometers to achieve head tracking or immersive displays of PCs, tablets, smart TVs and the such.
- Virtual reality devices can include but are not limited to the Samsung GearVRTM system or any Google CardboardTM setup.
- the delivery can take into account specifications of the viewing device, the data connection, and the view direction, to provide content that achieves the best visible resolution for the user. In some embodiments with integrated components, the delivery can simply refer to the processing and selection of the specific image content for display within the device.
- resolution optimization can also be a result of the dynamic selection of pre-encoded array of videos at various sizes, at various bitrates, at various rotations, and using different resolution-defined projections that take into account the small field of view of humans (i.e., fovea field of view).
- the fovea field of view being relatively minor in comparison to the full sphere of virtual reality content to allow higher resolution where according to the eye's resolution and is also in line by the constraints of virtual reality display devices.
- the Oculus RiftTM has a 110° field of view while the Samsung GearVRTM has a 96° field of view.
- Google CardboardTM setups have a range of field of views, but all being less than 96′.
- the field of view constraints mean that a user will, at any single instant, only experience resolution in a small subset of the full sphere of content.
- Their eye resolution means any user's limited ability to only appreciate extremely high resolution in his/her direction of focus.
- an infinite number of video versions could be generated to match every possible viewing direction of the user where limited virtual reality content is at issue. Having the different versions, at almost any instant, the video version could allow for it to be updated to exactly match the user's field of view. I.e., taking the user's fovea field of view resolution sensitivity and the device-limited field of view to generate a desired resolution profile for the virtual reality content transmitted to the virtual reality device to achieve minimum bandwidth requirements while still giving the viewer a perfect viewing experience.
- feedback data during the display of content including, for example, resulting data from eye movement tracking, may be recorded at 144 for future processing and/or set up of user preferences.
- Eye movement may be tracked to signal attention diversion, discomfort, moving patterns and the such which additionally may be used for vision condition diagnostics or therapy treatments.
- the feedback may be used for the early detection of color blindness or abnormal sensitivity to flashing light patterns, seasonal affective disorder therapy and similar related diagnostic therapeutic applications.
- pre-recorded and/or transmitted crowdsourced data 150 may be used to define projections, conditions, and/or parameters relating to the delivery of the specific virtual reality content for the implementation.
- the resolution function can be a smoothed version of the user's exact viewing resolution distribution in order to account for deviations, including, for example, the fact that a user could move his eyes before the new rendition may be delivered. It can also allow for a narrower number of view-direction renditions as every additional view-direction rendition requires processing time to encode and storage space, so may be incentives to reduce the total number depending on the amount of content and application.
- the aforementioned aspects may function via data input/output by the server and virtual reality device of the user.
- processing various combinations of video parameters, and joining videos that may only differ in view direction in to an adaptive streaming playlist, split into small-duration chunks, can be achieved for display 140 .
- the viewing direction can be determined for a specific rendition request.
- an estimate the bandwidth of the connection between the two can be determined using the limits imposed by processing power on the size of the video that the device can decode and a request for content that may be mapped to a specific rendition of the video. Accordingly, all or at least some of these different possible videos can be requested adaptively, with the new video arriving within one to two chunks from request.
- a new video chunk may be indicated by the inclusion of a keyframe at the beginning of every chunk.
- the virtual reality device can be programmed, or run an application that is capable of determining which frame starts a chunk from a new video parameter set because of a flag included with this keyframe. Upon seeing a keyframe flag, the video can be reverse-projected using the latest parameters used to request this newest chunk.
- the requested video chunk arrives and is decoded, it can be ready to be displayed as a reverse-projection.
- the same method as projection can be used by creating a distribution of points in the final mapping space, usual a three-dimensional sphere, mapping each pixel to its corresponding coordinates in projection space, and interpolating the pixels values near the target coordinates.
- the video may then be in a standard format as a texture to be applied to a three-dimensional shape in virtual reality space with higher resolutions in the areas we would like.
- At least some aspects could be used on 350° panoramas or any 360° virtual reality content device that would benefit from having a projection generation process for customizable resolution profiles that allows us to create virtual reality content renditions that are optimized for a range of bandwidths, devices, and/or view directions. More specifically, a projection generation process that can work both on the fly, for example, to adjust to bandwidth and device parameters, independent of viewing direction and/or according to the desired resolution based on the viewing direction.
- FIGS. 4 and 5 a flowchart 210 and corresponding schematic representation 310 illustrating the method steps that may be implemented to generate a resolution-defined projection according to some aspects of the disclosure are depicted. Both FIGS. 4 and 5 will reference common numeral identifiers as appropriate to help illustrate the relationship between method steps in the generating a resolution-defined projection process. Whilst specific mathematical equations and additional steps are, disclosed to better enable and explain the different novel aspects of the disclosure, the scope of those aspects is to be defined by the claim limitations as interpreted in view of the more simplified steps in flowchart 210 or their corresponding equivalents.
- one or more of the fovea field of view for a particular viewing direction, contrast of the virtual reality image data, movement of the virtual reality image data content, camera arrangement, camera lenses, and crowd sourced data will inform the specification of a desired resolution distribution.
- an existing bijective projection from a sphere to a rectangle having a separable Jacobian, such as the equirectangular projection may be selected to function as an intermediary projection.
- a desired resolution distribution informed by one or more items in 111 that is separable in the existing projection space of 115 is specified.
- a resolution-defined projection may be generated for the content to be mapped.
- the resolution-defined projection and any associated parameters generated can also be used for encoding and/or compression.
- the exemplary projections considered map 3D Cartesian coordinates, (x, y, z), to a new coordinate space, (u, v, w).
- P ⁇ a ⁇ 3 is set to be any such projection and p (•) ⁇ 3 ⁇ 1
- P ( x,y,z )
- ( u,v,w ) ( p u ( x,y,z ), p v ( x,y,z ), p w ( x,y,z )) (1)
- the projection can be invertible, with a defined P ⁇ 1 ⁇ 3 ⁇ 3 and associated p (•) ⁇ 1 ⁇ 3 ⁇ 1 .
- u ⁇ [u a ,u b ), v ⁇ [v a ,v b ) and w+w 0 can define the unit-sphere rectangle in projection space.
- may be the area of the unit-sphere rectangle in projection space and ⁇ P can be a surface area element in projection space.
- the subscript C can refers to a value in Cartesian coordinates while the subscript P refers to a value in projection coordinates.
- the area of the unit sphere in Cartesian space can be known.
- a C can be the area of the unit sphere in Cartesian space and let ⁇ C be a surface area element in Cartesian space.
- the Jacobian of a projection is accounted for.
- the Jacobian can be a function of the inverse individual coordinate projection equations (i.e. p (•) ⁇ 1 ).
- J ⁇ ( u , v ) [ ⁇ p x - 1 ⁇ u ⁇ p x - 1 ⁇ v ⁇ p x - 1 ⁇ w ⁇ p y - 1 ⁇ u ⁇ p y - 1 ⁇ v ⁇ p y - 1 ⁇ w ⁇ p z - 1 ⁇ u ⁇ p z - 1 ⁇ v ⁇ p z - 1 ⁇ w ] ⁇ ⁇ ( u , v , w 0 ) ( 7 )
- Constraints can be defined on the projections P that may be considered valid, and the associated functions P ⁇ 1 , p(•), p ⁇ 1 , and J, all of which may be derived from any valid P.
- the surface integral of a function in Cartesian space can be evaluated in projection space by treating f C , (x, y, z) as any function in Cartesian space.
- ⁇ f C ( x,y,z ) d ⁇ C ⁇ f C ( P ⁇ 1 ( u,v ) ⁇ J ( u,v )
- J ⁇ ( u , v ) [ ⁇ p x - 1 ⁇ u ⁇ p x - 1 ⁇ v ⁇ p x - 1 ⁇ w ⁇ p y - 1 ⁇ u ⁇ p y - 1 ⁇ v ⁇ p y - 1 ⁇ w ⁇ p z - 1 ⁇ u ⁇ p z - 1 ⁇ v ⁇ p z - 1 ⁇ w ] ⁇
- the Jacobian of a projection can define the warping of space due to a projection. This in essence may be the same or significantly similar to the warping of space which can affect the resolution of the spherical output content.
- Spatial resolution r can be defined as the local density of independent pixels, or the local ratio of independent pixels ⁇ N to the area they occupy ⁇ . This can be the information used as the density of an image. Spatial resolution is not necessarily constant over an image.
- Pixel density ⁇ can be defined as the ratio of discrete pixels M to the area they occupy
- Pixel density ⁇ and spatial resolution r may not necessary be equivalent as they may only be the same when each pixel is independent from all other pixels. Spatial resolution cannot exceed pixel density, but pixel density could be higher than spatial resolution, as it may be the case with a blurred or upsampled image. r ⁇ (12)
- i A C can be the spatial resolution, pixel density, independent pixel count, and total area of the spherical input content, respectively. It may be assumed that the content could have been generated (via cameras or computers) in a way such that the spatial resolution can be constant and equal to the pixel density.
- the spherical input data may cover a unit sphere, so the total area of the content i A C can be taken as 4 ⁇ .
- the resolution of the mapped data r P can be the same as the pixel density ⁇ P if we limit the pixel count N P such that no oversampling of the spherical input content occurs. This may be a reasonable assumption since the pixel count of the mapped content can typically be much lower than the largest available version of the spherical input content.
- the spatial resolution of the spherical output content r C may be constrained by the mapped content resolution.
- the pixel density of the spherical output data ⁇ C can be determined by the field of view and screen size of the display device. Considering current virtual reality headset resolutions and mobile network bandwidths, the pixel density can typically be much higher than the spatial resolution, which is often bandwidth constrained. This can be the first case where the pixel density can be higher than the spatial resolution, due to oversampling when inverse projecting the mapped content to spherical output content.
- the spatial resolution may not be uniform due to the warping of the uniform pixels of the mapped content during reverse projection. This warping can be exactly captured by the magnitude of the determinant of the Jacobian.
- the Equirectangular projection can be a projection from the unit sphere to a rectangle in latitutde-longititude space. This may be the spherical coordinate representation of a sphere, with longitude u ⁇ [ ⁇ , ⁇ ), latitude
- Equation 18 may be used to solve for the resolution function resulting from the Equirectangular projection.
- This thus can demonstrate how the projection chosen can be used to determine the shape of the spatial resolution of the final content—through the projection's Jacobian.
- a specified appropriate resolution function can be used to derive an application-specific projection according to the various aspects of the disclosure.
- Np can be the number of pixels in the unit-sphere rectangle in the resolution-defined projection space and
- J is the Jacobian for the resolution-defined projection P.
- J includes nine degrees of freedom, so solving Equation 31 for all the elements may be open ended.
- the constraint can be an existing projection from the unit sphere to a rectangle, and to which the projections can be applied to warp the rectangle. For purposes of this disclosure this may be referred to as an intermediate projection ⁇ tilde over (P) ⁇ .
- This projection can map from the 3D unit sphere to a rectangle spanning (a, b) in (a, b, c) coordinate space.
- the next projection can be the first coordinate remapping ⁇ hacek over (P) ⁇ , which may warp the rectangle along the a axis, preserving the b and c coordinates.
- This first coordinate remapping can map from (a, b, c) coordinates to (d, e, f) coordinates.
- the final projection can be the second coordinate remapping ⁇ circumflex over (P) ⁇ , which then may warp the rectangle in (d, e, f) space along the e-axis.
- This second coordinate remapping can map from (d, e, f) space to (u, v, w) space.
- the combination of these three projections may be referred to as a hybrid projection P .
- the Jacobian of the hybrid projection J can simply be a product of its component projections.
- the distributive property of the determinant and then the constraints can also be applied to ⁇ hacek over (P) ⁇ and ⁇ circumflex over (P) ⁇ .
- Solving for ⁇ tilde over (J) ⁇ (a, b, c) may be possible using the intermediate projection.
- the remapping Jacobians may also have only one variable element in each.
- a multivariable first-order differential equation can be generated to solve in order to derive the resolution-defined hybrid projection P .
- Equation 37 Two additional constraints may be applied.
- a ⁇ tilde over (J) ⁇ ( a ) b ⁇ tilde over (J) ⁇ ( b ) c ⁇ tilde over (J) ⁇ ( c ) (38)
- r C ( x,y,z ) a r C ( ⁇ tilde over (p) ⁇ a ( x,y,z )) b r C ( ⁇ tilde over (p) ⁇ b ( x,y,z )) c r C (( x,y,z )) (39)
- r C ( ⁇ tilde over (P) ⁇ ⁇ 1 ( a,b,c )) a r C (
- the absolute value can be dropped since and ⁇ J ⁇ are positive values. Dropping the absolute values can create an even stricter equation, but the differential equations can result in a solvable form with the following solution.
- Equation 27 shows that the Equirectangular Jacobian can be separable.
- the target uniform resolution could be any positive value k.
- the Lambert Cylindrical Equal-Area projection derived may be used to solve for the number of pixels in projection space N P in order to exactly match the target resolution function. a ⁇ [ ⁇ , ⁇ ) (63)
- the Lambert projection with a total of 2 ⁇ 2 k pixels.
- a bivariate gaussian shape for the resolution function may be desired.
- the resolution can then fall off smoothly in azimuth and elevation with standard deviations ⁇ a and ⁇ b, respectively and the equirectangular projection may again be used as an intermediate projection.
- N P may match any value of k in the original resolution function.
- a p _ ⁇ 2 ⁇ ( ⁇ ⁇ ⁇ a 2 ⁇ erf ⁇ ( ⁇ 2 ⁇ ⁇ a
- ) ) ⁇ 2 ⁇ ( sin ( ⁇ ⁇ ⁇ b 2 ⁇ erf ( ⁇ 2 2 ⁇ ⁇ b ) ) ) ⁇ 2 ⁇ 2 ⁇ ⁇ ⁇ ⁇ a ⁇ erf ⁇ ( ⁇ 2 ⁇ ⁇ a ) ⁇ sin ( ⁇ ⁇ ⁇ b 2 ⁇ erf ( ⁇ 2 2 ⁇ ⁇ b ) ) ( 86 )
- the final projection equations for the gaussian resolution function can be fairly computationally expensive. In addition, it may also be useful to use a function with fatter tails, so the resolution decrease isn't quite so dramatic. Following is an example on how the Cauchy function can be used to solve these issues.
- any virtual reality content may be received, uploaded and/or recorded, (i.e., transmitted) to the system so long as the projection is known or can be determined from the content.
- the content may be uploaded in two dimensional equirectangular format and come from one or multiple virtual reality recording devices.
- the content may be in 360° spherical three dimensional format.
- custom projections can be implemented to magnify the range of distortions across a projection (as explained above). This can exaggerate the differences in resolution depending on viewing angle, thus enabling the delivery of higher resolution projections for the most relevant view direction (as compared to those portions of the virtual reality content that are not immediately visible). All of this can be accomplished while also reducing the bitrate of the transmitted video, as the content in its new projection format can be encoded in such a way that its bitrate is lower than the original content, but the resolution in the view direction is the same as the original content.
- the content can be mapped using a resolution-defined projection.
- the resolution-defined projection may be generated, for example, according to one or more of the developed projections with Gaussian, Cosine, Uniform, and Cauchy resolution distributions or combinations thereof.
- the mapping of the content may be repeated as to generate at least one rendition using variously-rotated and/or variously-shaped projections that can be encoded at 415 for transmission at 420 and decoding at 425 at the VR device. Storing (not shown) may optionally occur at one or more stages of the process.
- the content is encoded at 416 , stored at 421 and accessed accordingly at 426 , without a need of either or both transmission or decoding.
- the rendition(s) can use the same or different projections rotation in azimuth and elevation as the base projections, or could even implement combinations thereof.
- the renditions generated herein are fundamentally different because of the different projections used to provide optimized resolutions and encoding.
- the different projections can generally be pre-determined defined, for example, by a desired resolution distribution according to the view direction and/or, on the fly, for example, by a desired resolution distribution that adapts to parameters of the device, bandwidth limitations, and the such.
- data definitive of the mapping and/or the desired resolution may be included along the virtual reality content for organized transmission at 420 of dynamically selected projection versions according to the viewing direction and decoding 425 at the virtual reality projection device.
- the dynamically selected projection versions may include only a subset of a generated rendition.
- the virtual reality projection device may include an app installed that is configured to transmit virtual reality content requests (including, for example, viewing direction defined requests) and receive the corresponding data for decoding accordingly.
- virtual reality content requests including, for example, viewing direction defined requests
- one or more renditions may be transmitted to the user's virtual reality device.
- a reverse projection is applied to the content to map it back to the full sphere of content it was derived from.
- the virtual reality device already has knowledge of which rendition it requested, so using the correct parameters for projection may be trivial.
- any additional renditions transmitted may be used as a small buffer to the user's viewing direction, for example.
- inverse-projection content may be projected to a sphere at 430 .
- the virtual reality content can be projected in accordance with the desired resolution distribution at 435 , so that it is of higher resolution in the areas of interest.
- the virtual reality user views virtual reality content with higher resolution in their view direction without data unnecessarily transmitted to perfectly recreate content in their periphery or out of their field of view.
- This method can be used to reduce virtual reality content bitrate without sacrificing visible resolution, or to improve visible resolution without increasing bitrate, or a combination of improved visible resolution and reduced bitrate.
- FIGS. 8 and 9 a flowchart 600 and corresponding schematic representation 700 illustrating method steps that may be implemented to process and deliver virtual reality content according to some aspects of the disclosure are shown. Both FIGS. 6 and 7 will reference common numeral identifiers as appropriate to help illustrate the relationship between method steps for processing and transmitting virtual reality content.
- 360° virtual reality content is transmitted to a processing device.
- the content in order to use traditional media encoding and compression techniques, the content can be re-mapped to a two-dimensional rectangular projection to avoid extraneous data.
- a set of projections and/or rotations can then be mapped.
- each of those renditions can then be encoded at one or more bitrates.
- head and/or eye movement tracking can be used to determine a viewing direction of the user. This data can be obtained from the head mounted virtual reality devices' gyroscopes and/or a sensor tracking the user's pupils during viewing of content or other methods of view direction tracking preexisting in the virtual reality device.
- the viewing direction data may be used to select a subset of renditions to be delivered to the virtual reality device at 625 .
- detection of available bandwidth and/or viewing device parameters may be used to determine the selection of the projections.
- viewing direction is used for the selection, for example, by detecting the user's viewing direction and, taking into account the fact that human vision degrades away from the focus point
- the subset of virtual reality content that can be selected is optimized for that viewing direction.
- the viewing device parameters are used for the selection, for example, the maximum DPI of the screen may be used for the selection, to ensure the subset closely matches the maximum DPI of the screen and avoid unnecessary processing.
- the subset of renditions selected are then transmitted to the virtual reality device of the user for display of the virtual reality content at 630 .
- the process steps can further be used to automatically generate projections based on the content or recorded moving patterns.
- a resolution distribution that includes greater resolution in those corresponding regions could be determined.
- the higher definition in those regions then provide a more immersive experience to the user.
- this method could also be used in conjunction with the implementation of crowd-sourced data to define, at least in part, which areas of the virtual reality content should include the higher distributions and/or establish moving patterns that can be used for predictive generation of the renditions. More specifically, by aggregating data from many viewers of the same or other virtual reality content (viewing direction, moving patterns, focus, emotional reactions and/or physiological changes), the data may be used to improve the virtual reality experience, including, for example, to create desired resolution distribution profiles specific to the virtual reality content or viewer or to select one or more existing renditions.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computer Graphics (AREA)
- Human Computer Interaction (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Architecture (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
- Controls And Circuits For Display Device (AREA)
Abstract
Description
P(x,y,z)=|(u,v,w)=(p u(x,y,z),p v(x,y,z),p w(x,y,z)) (1)
(x,y,z)=P −1(u,v,w)=(p z −1(u,v,w),p y −1(u,v,w),p z −1(u,v,w)) (2)
P(x,y,z)|(x,y,z)εu.s.=(u,v,w 0)=(u,v)|w
P −1(u,v)≡P −1(u,v)|w
∫∫f C(x,y,z)dΣ C =∫∫f C(P −1(u,v)∥J(u,v)|dΣ P (8)
r≦ρ (12)
and radius w(w0=1).
p z −1(u,v,w)=w cos(v)cos(u) (19)
p y −1(u,w)=w cos(v)sin(u) (20)
p z −1(u,v,w)=w sin(v) (21)
c∥J(u,v,w)|=w 2 cos(v)| (26)
and w0=1.
∥J(u,v)∥=cos(v) (27)
|A P|=|(v b −v a)(u b −u a)=2π2 (28)
P(x,y,Z)={circumflex over (P)}({hacek over (P)}({tilde over (P)}(x,y,z))))=(u,v,w) (32)
∥
{tilde over (J)}(a,b,c)|=a {tilde over (J)}(a)b {tilde over (J)}(b)c {tilde over (J)}(c) (38)
r C(x,y,z)=a r C({tilde over (p)} a(x,y,z))b r C({tilde over (p)} b(x,y,z))c r C((x,y,z)) (39)
r C({tilde over (P)} −1(a,b,c))=a r C(a)b r C(b)c r C(c) (40)
f(x)=y s.t.x=∫g(γ)h(γ)dγ| γ=y (45)
{tilde over (J)}(a,b,c)=cos(b) (46)
a {tilde over (J)}(a)=1 (47)
c {tilde over (J)}(b)=cos(b) (48)
c {tilde over (J)}(c)=1 (49)
r C(x,y,z)=k=r C({tilde over (P)} −1(a,b,c)) (50)
{dot over (p)} a −1(u)=u (54)
{hacek over (p)} e −1(v)=sin−1(v) (55)
{hacek over (p)} u(e)=sin(e) (57)
and a simple mapping from 3D space to 2D space can be solved for.
aε[−π,π) (63)
uε{circumflex over (p)} u([−π,π))=[−π,π) (65)
A
which may result in the final mapping from (x,y,z) to our unit-sphere rectangle in (u,v) space.
aε[−π,π) (82)
and a final mapping from (x, y, z) to our unit-sphere rectangle in (u, v) space can result.
aε[−π,π) (101)
Claims (24)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/156,069 US9799097B1 (en) | 2016-04-29 | 2016-05-16 | Method and system for defining a virtual reality resolution distribution |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/143,424 US9721393B1 (en) | 2016-04-29 | 2016-04-29 | Method for processing and delivering virtual reality content to a user |
US15/156,069 US9799097B1 (en) | 2016-04-29 | 2016-05-16 | Method and system for defining a virtual reality resolution distribution |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/143,424 Continuation US9721393B1 (en) | 2016-04-29 | 2016-04-29 | Method for processing and delivering virtual reality content to a user |
Publications (2)
Publication Number | Publication Date |
---|---|
US9799097B1 true US9799097B1 (en) | 2017-10-24 |
US20170316549A1 US20170316549A1 (en) | 2017-11-02 |
Family
ID=59382646
Family Applications (4)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/143,424 Active US9721393B1 (en) | 2016-04-29 | 2016-04-29 | Method for processing and delivering virtual reality content to a user |
US15/151,041 Active 2036-07-23 US9959834B2 (en) | 2016-04-29 | 2016-05-10 | Method and system for adaptively changing display parameters of virtual reality content |
US15/151,322 Active 2036-06-16 US9865092B2 (en) | 2016-04-29 | 2016-05-10 | Method and system for predictive processing of virtual reality content |
US15/156,069 Active 2036-06-12 US9799097B1 (en) | 2016-04-29 | 2016-05-16 | Method and system for defining a virtual reality resolution distribution |
Family Applications Before (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/143,424 Active US9721393B1 (en) | 2016-04-29 | 2016-04-29 | Method for processing and delivering virtual reality content to a user |
US15/151,041 Active 2036-07-23 US9959834B2 (en) | 2016-04-29 | 2016-05-10 | Method and system for adaptively changing display parameters of virtual reality content |
US15/151,322 Active 2036-06-16 US9865092B2 (en) | 2016-04-29 | 2016-05-10 | Method and system for predictive processing of virtual reality content |
Country Status (1)
Country | Link |
---|---|
US (4) | US9721393B1 (en) |
Families Citing this family (48)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10868848B2 (en) * | 2016-07-25 | 2020-12-15 | Peraso Technologies Inc. | Wireless multimedia communications system and method |
US10412412B1 (en) | 2016-09-30 | 2019-09-10 | Amazon Technologies, Inc. | Using reference-only decoding of non-viewed sections of a projected video |
US10553029B1 (en) * | 2016-09-30 | 2020-02-04 | Amazon Technologies, Inc. | Using reference-only decoding of non-viewed sections of a projected video |
KR20230070062A (en) | 2016-10-04 | 2023-05-19 | 주식회사 비원영상기술연구소 | Image data encoding/decoding method and apparatus |
US20190253624A1 (en) * | 2017-07-17 | 2019-08-15 | Ki Baek Kim | Image data encoding/decoding method and apparatus |
KR20180071012A (en) * | 2016-12-19 | 2018-06-27 | 삼성전자주식회사 | Electronic apparatus and controlling method thereof |
US10334238B2 (en) * | 2017-01-03 | 2019-06-25 | Black Sails Technology Inc. | Method and system for real-time rendering displaying high resolution virtual reality (VR) video |
JPWO2018128097A1 (en) * | 2017-01-06 | 2019-11-07 | ソニー株式会社 | Information processing apparatus and information processing method |
US10609356B1 (en) | 2017-01-23 | 2020-03-31 | Amazon Technologies, Inc. | Using a temporal enhancement layer to encode and decode stereoscopic video content |
US11057643B2 (en) | 2017-03-13 | 2021-07-06 | Mediatek Inc. | Method and apparatus for generating and encoding projection-based frame that includes at least one padding region and at least one projection face packed in 360-degree virtual reality projection layout |
US11004173B2 (en) * | 2017-03-13 | 2021-05-11 | Mediatek Inc. | Method for processing projection-based frame that includes at least one projection face packed in 360-degree virtual reality projection layout |
US10904508B2 (en) * | 2017-03-15 | 2021-01-26 | Avago Technologies International Sales Pte. Limited | 360 degree video with combined projection format |
WO2018175625A1 (en) | 2017-03-22 | 2018-09-27 | Magic Leap, Inc. | Depth based foveated rendering for display systems |
US11070724B2 (en) * | 2017-03-22 | 2021-07-20 | Sony Corporation | Image processing apparatus and method |
US10467775B1 (en) * | 2017-05-03 | 2019-11-05 | Amazon Technologies, Inc. | Identifying pixel locations using a transformation function |
US11049219B2 (en) | 2017-06-06 | 2021-06-29 | Gopro, Inc. | Methods and apparatus for multi-encoder processing of high resolution content |
US9998664B1 (en) * | 2017-06-20 | 2018-06-12 | Sliver VR Technologies, Inc. | Methods and systems for non-concentric spherical projection for multi-resolution view |
JP6938237B2 (en) * | 2017-06-21 | 2021-09-22 | キヤノン株式会社 | Information processing equipment, information processing methods and programs |
US10776992B2 (en) * | 2017-07-05 | 2020-09-15 | Qualcomm Incorporated | Asynchronous time warp with depth data |
CN107317987B (en) * | 2017-08-14 | 2020-07-03 | 歌尔股份有限公司 | Virtual reality display data compression method, device and system |
TWI681662B (en) | 2017-08-18 | 2020-01-01 | 聯發科技股份有限公司 | Method and apparatus for reducing artifacts in projection-based frame |
US10818087B2 (en) | 2017-10-02 | 2020-10-27 | At&T Intellectual Property I, L.P. | Selective streaming of immersive video based on field-of-view prediction |
GB201717011D0 (en) | 2017-10-17 | 2017-11-29 | Nokia Technologies Oy | An apparatus a method and a computer program for volumetric video |
EP3724858A4 (en) * | 2017-12-14 | 2021-01-13 | Samsung Electronics Co., Ltd. | Method and apparatus for managing immersive data |
US10554953B2 (en) * | 2017-12-17 | 2020-02-04 | Google Llc | Distortion of video for seek in 360 degree video |
US10659815B2 (en) | 2018-03-08 | 2020-05-19 | At&T Intellectual Property I, L.P. | Method of dynamic adaptive streaming for 360-degree videos |
EP3765943A4 (en) | 2018-03-16 | 2021-12-22 | Magic Leap, Inc. | Depth based foveated rendering for display systems |
CN110278459A (en) * | 2018-03-16 | 2019-09-24 | 上海小麦互动企业发展有限公司 | Content control system and its control method |
US10977871B2 (en) * | 2018-04-25 | 2021-04-13 | International Business Machines Corporation | Delivery of a time-dependent virtual reality environment in a computing system |
US11153482B2 (en) * | 2018-04-27 | 2021-10-19 | Cubic Corporation | Optimizing the content of a digital omnidirectional image |
US10735765B2 (en) | 2018-06-07 | 2020-08-04 | Hong Kong Applied Science and Technology Research Institute Company, Limited | Modified pseudo-cylindrical mapping of spherical video using linear interpolation of empty areas for compression of streamed images |
US10573060B1 (en) * | 2018-06-14 | 2020-02-25 | Kilburn Live, Llc | Controller binding in virtual domes |
US10740957B1 (en) * | 2018-06-14 | 2020-08-11 | Kilburn Live, Llc | Dynamic split screen |
US10826964B2 (en) | 2018-09-05 | 2020-11-03 | At&T Intellectual Property I, L.P. | Priority-based tile transmission system and method for panoramic video streaming |
US11461942B2 (en) * | 2018-12-21 | 2022-10-04 | Koninklijke Kpn N.V. | Generating and signaling transition between panoramic images |
US11470140B2 (en) * | 2019-02-20 | 2022-10-11 | Dazn Media Israel Ltd. | Method and system for multi-channel viewing |
US11457053B2 (en) * | 2019-02-20 | 2022-09-27 | Dazn Media Israel Ltd. | Method and system for transmitting video |
US10460516B1 (en) * | 2019-04-26 | 2019-10-29 | Vertebrae Inc. | Three-dimensional model optimization |
WO2020232281A1 (en) * | 2019-05-14 | 2020-11-19 | Intel Corporation | IMMERSIVE VIDEO CODING TECHNIQUES FOR THREE DEGREE OF FREEDOM PLUS/METADATA FOR IMMERSIVE VIDEO (3DoF+/MIV) AND VIDEO-POINT CLOUD CODING (V-PCC) |
EP3739880A1 (en) * | 2019-05-14 | 2020-11-18 | Axis AB | Method, device and computer program product for encoding a distorted image frame |
US11228781B2 (en) | 2019-06-26 | 2022-01-18 | Gopro, Inc. | Methods and apparatus for maximizing codec bandwidth in video applications |
US11109067B2 (en) * | 2019-06-26 | 2021-08-31 | Gopro, Inc. | Methods and apparatus for maximizing codec bandwidth in video applications |
US11432009B2 (en) * | 2019-07-02 | 2022-08-30 | Intel Corporation | Techniques for encoding and decoding immersive video |
KR102582407B1 (en) * | 2019-07-28 | 2023-09-26 | 구글 엘엘씨 | Methods, systems, and media for rendering immersive video content with foveated meshes |
US11481863B2 (en) | 2019-10-23 | 2022-10-25 | Gopro, Inc. | Methods and apparatus for hardware accelerated image processing for spherical projections |
US11792420B2 (en) * | 2019-11-04 | 2023-10-17 | Qualcomm Incorporated | Methods and apparatus for foveated compression |
US20220058999A1 (en) * | 2020-08-18 | 2022-02-24 | Roka Labs, Inc. | Automated vision care diagnostics and digitally compensated smart eyewear |
US20220103655A1 (en) | 2020-09-29 | 2022-03-31 | International Business Machines Corporation | Proactively selecting virtual reality content contexts |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6005611A (en) | 1994-05-27 | 1999-12-21 | Be Here Corporation | Wide-angle image dewarping method and apparatus |
US20160021373A1 (en) | 2014-07-17 | 2016-01-21 | Yahoo! Inc. | System and method of spherical image compression |
US20160353146A1 (en) | 2015-05-27 | 2016-12-01 | Google Inc. | Method and apparatus to reduce spherical video bandwidth to user headset |
-
2016
- 2016-04-29 US US15/143,424 patent/US9721393B1/en active Active
- 2016-05-10 US US15/151,041 patent/US9959834B2/en active Active
- 2016-05-10 US US15/151,322 patent/US9865092B2/en active Active
- 2016-05-16 US US15/156,069 patent/US9799097B1/en active Active
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6005611A (en) | 1994-05-27 | 1999-12-21 | Be Here Corporation | Wide-angle image dewarping method and apparatus |
US20160021373A1 (en) | 2014-07-17 | 2016-01-21 | Yahoo! Inc. | System and method of spherical image compression |
US20160353146A1 (en) | 2015-05-27 | 2016-12-01 | Google Inc. | Method and apparatus to reduce spherical video bandwidth to user headset |
Non-Patent Citations (13)
Title |
---|
Aakash Patel, Gregory Rose, Conduit, "Efficient Video Compression for Live VR Streaming", https://avp.github.io/conduit/ , Apr. 25, 2016. |
Ben Lang, Road to VR, "Hands-on: Nokia's OZO VR Cam Impresses with Capture Quality, Unique Playback Solution Has Pros and Cons", Jul. 29, 2015, http://www.roadtovr.com/hands-on-nokias-ozo-vr-cam-impresses-with-capture-quality-unique-playback-solution-has-pros-and-cons/. |
Bitmovin, "VR and 360o Video from Encoding to Playout," Apr. 23, 2016, http://web.archive.org/web/20160423045729/https://bitmovin.com/tutorials/vr-360-video-encoding-playout/#mobile-menu. |
David Pio, Facebook/Oculus Connect 2: New Techniques for Streaming VR Video, YouTube video, https://www.youtube.com/watch?v=zpuUUxMfaJs, Sep. 23-25, 2015. |
Evgeny Kuzyakov, David Pio, Facebook, Next-generation video encoding techniques for 260 video and VR, https://code.facebook.com/posts/1126354007399553/next-generation-video-encoding-techniques-for-360-video-and-vr/ , Jan. 21, 2016. |
Frank Nielsen, "High Resolution Full Spherical Videos", Sony Computer Science Laboratories, Inc., Proceedings of the International Conference on Information Technology: Coding and Computing (ITCC'02), dated 2002, in 8 pages. |
Frank Nielsen, "Surround Video: A Multihead Camera Approach", Sony Computer Science Laboratories, The Visual omputer, published Feb. 3, 2005. |
Henry Segerman, EleVR, "Spherical video editing effects with Mobius transformations", http://elevr.com/spherical-video-editing-effects-with-mobius-transformations/ , Jan. 22, 2016. |
Ivana Tosic, Pascal Rossard, Signal Processing Laboratory (LTS4), Low Bit-Rate Compression of Omnidirectional Images, http://infoscience.epfl.ch/record/130365/files/omni-cr.pdf , 2009. |
Ivana Tosic, Pascal Rossard, Signal Processing Laboratory (LTS4), Low Bit-Rate Compression of Omnidirectional Images, http://infoscience.epfl.ch/record/130365/files/omni—cr.pdf , 2009. |
Joseph Bertolami, "Cannes Video Codec", http://bertolami.com/index.php?engine=portfolio&content=compression&detail=cannes-video-codec , Mar. 8, 2016. |
Joseph Bertolami, "Virtual Reality Video Compression" blog, http://bertolami.com/index.php?engine=blog&content=posts&detail=virtual-reality-video-codec, Oct. 11, 2015. |
Sebastian Perez-Duarte, David Swart, Proeedings of Bridges 2013: Mathematics, Music, Art, Architexture, Culture, "The Mercator Redemption", http://archive.bridgesmathart.org/2013/bridges2013-217.pdf. |
Also Published As
Publication number | Publication date |
---|---|
US20170316753A1 (en) | 2017-11-02 |
US9721393B1 (en) | 2017-08-01 |
US9865092B2 (en) | 2018-01-09 |
US9959834B2 (en) | 2018-05-01 |
US20170316609A1 (en) | 2017-11-02 |
US20170316549A1 (en) | 2017-11-02 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9799097B1 (en) | Method and system for defining a virtual reality resolution distribution | |
US20220174252A1 (en) | Selective culling of multi-dimensional data sets | |
Fan et al. | A survey on 360 video streaming: Acquisition, transmission, and display | |
US10602126B2 (en) | Digital camera device for 3D imaging | |
US10536693B2 (en) | Analytic reprocessing for data stream system and method | |
Chiariotti | A survey on 360-degree video: Coding, quality of experience and streaming | |
US10958834B2 (en) | Method to capture, store, distribute, share, stream and display panoramic image or video | |
US11295425B2 (en) | Gaze direction-based adaptive pre-filtering of video data | |
US10977764B2 (en) | Viewport independent image coding and rendering | |
CN107439010B (en) | Streaming spherical video | |
CN112204993B (en) | Adaptive panoramic video streaming using overlapping partitioned segments | |
JP2021103327A (en) | Apparatus and method for providing and displaying content | |
US20180084283A1 (en) | Behavioral Directional Encoding of Three-Dimensional Video | |
JP2019526178A (en) | Stream spatially tiled omnidirectional video | |
US10748250B2 (en) | Method and apparatus for managing immersive data | |
US20190335166A1 (en) | Deriving 3d volumetric level of interest data for 3d scenes from viewer consumption data | |
US11461871B2 (en) | Virtual reality cinema-immersive movie watching for headmounted displays | |
US20200334788A1 (en) | Playback apparatus and method, and generation apparatus and method | |
WO2018221211A1 (en) | Image processing device and method, file generation device and method, and program | |
US20200404241A1 (en) | Processing system for streaming volumetric video to a client device | |
WO2018069570A1 (en) | Display of visual data with a virtual reality headset | |
WO2018004936A1 (en) | Apparatus and method for providing and displaying content | |
TWI846808B (en) | Image signal representing a scene |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: YOUVISIT LLC, NEW YORK Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:DUNN, CHARLES;BADERKHAN, TAHER;MACHACEK, VINCENT;AND OTHERS;REEL/FRAME:041422/0701 Effective date: 20160429 |
|
AS | Assignment |
Owner name: IMMERSIVE ENTERPRISES, LLC, NEW YORK Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:YOUVISIT LLC;REEL/FRAME:041868/0081 Effective date: 20170113 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
CC | Certificate of correction | ||
AS | Assignment |
Owner name: YOUVISIT LLC, NEW YORK Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:IMMERSIVE ENTERPRISES, LLC;REEL/FRAME:048710/0568 Effective date: 20190325 |
|
AS | Assignment |
Owner name: MACQUARIE CAPITAL FUNDING LLC, AS COLLATERAL AGENT, ILLINOIS Free format text: SECURITY INTEREST;ASSIGNOR:YOUVISIT LLC;REEL/FRAME:051741/0022 Effective date: 20200205 Owner name: CORTLAND CAPITAL MARKET SERVICES LLC, AS COLLATERAL AGENT, ILLINOIS Free format text: SECURITY INTEREST;ASSIGNOR:YOUVISIT LLC;REEL/FRAME:051741/0062 Effective date: 20200205 |
|
AS | Assignment |
Owner name: EAB GLOBAL, INC., DISTRICT OF COLUMBIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:YOUVISIT LLC;REEL/FRAME:052098/0413 Effective date: 20191217 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YR, SMALL ENTITY (ORIGINAL EVENT CODE: M2551); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY Year of fee payment: 4 |
|
AS | Assignment |
Owner name: MACQUARIE CAPITAL FUNDING LLC, AS COLLATERAL AGENT, ILLINOIS Free format text: FIRST LIEN PATENT SECURITY AGREEMENT;ASSIGNORS:EAB GLOBAL, INC.;ROYALL & COMPANY, LLC (F/K/A ROYALL & COMPANY);REEL/FRAME:057350/0322 Effective date: 20210816 Owner name: UBS AG, STAMFORD BRANCH, AS COLLATERAL AGENT, CONNECTICUT Free format text: SECOND LIEN PATENT SECURITY AGREEMENT;ASSIGNORS:EAB GLOBAL, INC.;ROYALL & COMPANY, LLC (F/K/A ROYALL & COMPANY);REEL/FRAME:057335/0908 Effective date: 20210816 |
|
AS | Assignment |
Owner name: YOUVISIT LLC, DISTRICT OF COLUMBIA Free format text: RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:CORTLAND CAPITAL MARKET SERVICES LLC, AS COLLATERAL AGENT;REEL/FRAME:057336/0079 Effective date: 20210816 Owner name: YOUVISIT LLC, DISTRICT OF COLUMBIA Free format text: RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:MACQUARIE CAPITAL FUNDING LLC, AS COLLATERAL AGENT;REEL/FRAME:057336/0072 Effective date: 20210816 |
|
AS | Assignment |
Owner name: ROYALL & COMPANY, LLC, DISTRICT OF COLUMBIA Free format text: RELEASE OF SECOND LIEN SECURITY INTEREST IN PATENTS (REEL/FRAME 057335/0908);ASSIGNOR:UBS AG, STAMFORD BRANCH;REEL/FRAME:067799/0153 Effective date: 20240618 Owner name: EAB GLOBAL, INC., DISTRICT OF COLUMBIA Free format text: RELEASE OF SECOND LIEN SECURITY INTEREST IN PATENTS (REEL/FRAME 057335/0908);ASSIGNOR:UBS AG, STAMFORD BRANCH;REEL/FRAME:067799/0153 Effective date: 20240618 |