NZ789927A - System and methods of holographic sensory data generation, manipulation and transport - Google Patents
System and methods of holographic sensory data generation, manipulation and transportInfo
- Publication number
- NZ789927A NZ789927A NZ789927A NZ78992717A NZ789927A NZ 789927 A NZ789927 A NZ 789927A NZ 789927 A NZ789927 A NZ 789927A NZ 78992717 A NZ78992717 A NZ 78992717A NZ 789927 A NZ789927 A NZ 789927A
- Authority
- NZ
- New Zealand
- Prior art keywords
- energy
- data
- ofthe
- data points
- locations
- Prior art date
Links
- 230000001953 sensory Effects 0.000 title description 65
- 239000000463 material Substances 0.000 claims description 84
- 238000000034 method Methods 0.000 claims description 64
- 230000000007 visual effect Effects 0.000 claims description 30
- 150000002500 ions Chemical class 0.000 claims description 13
- 230000035943 smell Effects 0.000 claims description 8
- 229940035295 Ting Drugs 0.000 claims description 7
- 230000011218 segmentation Effects 0.000 claims description 6
- 210000001217 Buttocks Anatomy 0.000 claims description 3
- 230000001809 detectable Effects 0.000 claims description 3
- 230000000875 corresponding Effects 0.000 claims description 2
- 230000002401 inhibitory effect Effects 0.000 description 50
- 239000000203 mixture Substances 0.000 description 22
- 238000005070 sampling Methods 0.000 description 22
- 230000004807 localization Effects 0.000 description 18
- 230000003287 optical Effects 0.000 description 18
- 230000005540 biological transmission Effects 0.000 description 16
- 239000000306 component Substances 0.000 description 14
- 238000010586 diagram Methods 0.000 description 13
- 230000001902 propagating Effects 0.000 description 12
- 238000009877 rendering Methods 0.000 description 12
- 238000004458 analytical method Methods 0.000 description 11
- 230000003238 somatosensory Effects 0.000 description 11
- 239000000835 fiber Substances 0.000 description 10
- 230000000644 propagated Effects 0.000 description 10
- 230000004044 response Effects 0.000 description 10
- 230000004438 eyesight Effects 0.000 description 9
- 238000005286 illumination Methods 0.000 description 9
- 238000003384 imaging method Methods 0.000 description 9
- 238000004519 manufacturing process Methods 0.000 description 9
- 230000014860 sensory perception of taste Effects 0.000 description 9
- 230000002457 bidirectional Effects 0.000 description 8
- 238000005516 engineering process Methods 0.000 description 8
- 235000019640 taste Nutrition 0.000 description 8
- 230000035917 taste Effects 0.000 description 8
- 230000000694 effects Effects 0.000 description 7
- 230000001339 gustatory Effects 0.000 description 7
- 239000010410 layer Substances 0.000 description 7
- 230000004304 visual acuity Effects 0.000 description 7
- 210000002265 Sensory Receptor Cells Anatomy 0.000 description 6
- 238000009826 distribution Methods 0.000 description 6
- 239000003365 glass fiber Substances 0.000 description 6
- 230000000051 modifying Effects 0.000 description 6
- 108091008013 sensory receptors Proteins 0.000 description 6
- 102000027722 sensory receptors Human genes 0.000 description 6
- 230000001419 dependent Effects 0.000 description 5
- 239000011521 glass Substances 0.000 description 5
- 230000035945 sensitivity Effects 0.000 description 5
- 239000000126 substance Substances 0.000 description 5
- 238000006243 chemical reaction Methods 0.000 description 4
- 230000002452 interceptive Effects 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000006011 modification reaction Methods 0.000 description 4
- 230000035807 sensation Effects 0.000 description 4
- 235000019615 sensations Nutrition 0.000 description 4
- 238000000926 separation method Methods 0.000 description 4
- 241000229754 Iva xanthiifolia Species 0.000 description 3
- HWHLPVGTWGOCJO-UHFFFAOYSA-N Trihexyphenidyl Chemical compound C1CCCCC1C(C=1C=CC=CC=1)(O)CCN1CCCCC1 HWHLPVGTWGOCJO-UHFFFAOYSA-N 0.000 description 3
- 238000002835 absorbance Methods 0.000 description 3
- 238000007792 addition Methods 0.000 description 3
- 230000001427 coherent Effects 0.000 description 3
- 150000001875 compounds Chemical class 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 230000000873 masking Effects 0.000 description 3
- 238000005259 measurement Methods 0.000 description 3
- 108020003175 receptors Proteins 0.000 description 3
- 102000005962 receptors Human genes 0.000 description 3
- 230000008929 regeneration Effects 0.000 description 3
- 238000011069 regeneration method Methods 0.000 description 3
- XUIMIQQOPSSXEZ-UHFFFAOYSA-N silicon Chemical compound [Si] XUIMIQQOPSSXEZ-UHFFFAOYSA-N 0.000 description 3
- 239000010703 silicon Substances 0.000 description 3
- 229910052710 silicon Inorganic materials 0.000 description 3
- 230000000638 stimulation Effects 0.000 description 3
- 230000002123 temporal effect Effects 0.000 description 3
- 244000153665 Ficus glomerata Species 0.000 description 2
- 235000012571 Ficus glomerata Nutrition 0.000 description 2
- 101710028361 MARVELD2 Proteins 0.000 description 2
- 210000000412 Mechanoreceptors Anatomy 0.000 description 2
- 235000016247 Mentha requienii Nutrition 0.000 description 2
- 235000002899 Mentha suaveolens Nutrition 0.000 description 2
- 101700028182 OGT Proteins 0.000 description 2
- 206010063834 Oversensing Diseases 0.000 description 2
- 210000003491 Skin Anatomy 0.000 description 2
- 235000015125 Sterculia urens Nutrition 0.000 description 2
- IVBPIRYPDAOPPJ-UHFFFAOYSA-N [4-amino-2-(4-methoxyanilino)-1,3-thiazol-5-yl]-phenylmethanone Chemical compound C1=CC(OC)=CC=C1NC1=NC(N)=C(C(=O)C=2C=CC=CC=2)S1 IVBPIRYPDAOPPJ-UHFFFAOYSA-N 0.000 description 2
- 230000004308 accommodation Effects 0.000 description 2
- 230000001058 adult Effects 0.000 description 2
- 230000006399 behavior Effects 0.000 description 2
- 235000006682 bigleaf mint Nutrition 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 2
- 238000005253 cladding Methods 0.000 description 2
- 235000009508 confectionery Nutrition 0.000 description 2
- 210000002858 crystal cell Anatomy 0.000 description 2
- 230000003247 decreasing Effects 0.000 description 2
- 230000001747 exhibiting Effects 0.000 description 2
- 230000001788 irregular Effects 0.000 description 2
- 230000000670 limiting Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 238000004020 luminiscence type Methods 0.000 description 2
- 108091008026 mechanoreceptors Proteins 0.000 description 2
- 235000006679 mint Nutrition 0.000 description 2
- 238000004806 packaging method and process Methods 0.000 description 2
- 230000036961 partial Effects 0.000 description 2
- 229920000642 polymer Polymers 0.000 description 2
- 235000019633 pungent taste Nutrition 0.000 description 2
- 235000019600 saltiness Nutrition 0.000 description 2
- 230000003068 static Effects 0.000 description 2
- 230000002194 synthesizing Effects 0.000 description 2
- 244000020998 Acacia farnesiana Species 0.000 description 1
- 235000002566 Capsicum Nutrition 0.000 description 1
- 230000036536 Cave Effects 0.000 description 1
- 241000083551 Ena Species 0.000 description 1
- 101700015817 LAT2 Proteins 0.000 description 1
- 101710032250 MICAL1 Proteins 0.000 description 1
- 240000006217 Mentha pulegium Species 0.000 description 1
- 235000016257 Mentha pulegium Nutrition 0.000 description 1
- 206010027646 Miosis Diseases 0.000 description 1
- 210000001706 Olfactory Mucosa Anatomy 0.000 description 1
- 210000003254 Palate Anatomy 0.000 description 1
- 239000006002 Pepper Substances 0.000 description 1
- 235000016761 Piper aduncum Nutrition 0.000 description 1
- 235000017804 Piper guineense Nutrition 0.000 description 1
- 240000000129 Piper nigrum Species 0.000 description 1
- 235000008184 Piper nigrum Nutrition 0.000 description 1
- 210000001779 Taste Buds Anatomy 0.000 description 1
- 210000003051 Thermoreceptors Anatomy 0.000 description 1
- 241001080526 Vertica Species 0.000 description 1
- 230000002730 additional Effects 0.000 description 1
- 230000000996 additive Effects 0.000 description 1
- 239000000654 additive Substances 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 235000019568 aromas Nutrition 0.000 description 1
- BSYNRYMUTXBXSQ-UHFFFAOYSA-N aspirin Chemical compound CC(=O)OC1=CC=CC=C1C(O)=O BSYNRYMUTXBXSQ-UHFFFAOYSA-N 0.000 description 1
- 230000003416 augmentation Effects 0.000 description 1
- 230000003190 augmentative Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 229910052799 carbon Inorganic materials 0.000 description 1
- OKTJSMMVPCPJKN-UHFFFAOYSA-N carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 239000004568 cement Substances 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 239000008358 core component Substances 0.000 description 1
- 230000002596 correlated Effects 0.000 description 1
- 238000010192 crystallographic characterization Methods 0.000 description 1
- 238000009792 diffusion process Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 238000005315 distribution function Methods 0.000 description 1
- 239000000839 emulsion Substances 0.000 description 1
- 230000002255 enzymatic Effects 0.000 description 1
- 230000001815 facial Effects 0.000 description 1
- 238000011049 filling Methods 0.000 description 1
- 239000000796 flavoring agent Substances 0.000 description 1
- 235000019634 flavors Nutrition 0.000 description 1
- 238000005755 formation reaction Methods 0.000 description 1
- 239000010438 granite Substances 0.000 description 1
- 108091005688 gustatory receptors Proteins 0.000 description 1
- 102000035397 gustatory receptors Human genes 0.000 description 1
- 235000001050 hortel pimenta Nutrition 0.000 description 1
- 125000001867 hydroperoxy group Chemical group [*]OO[H] 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 238000007654 immersion Methods 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 238000010297 mechanical methods and process Methods 0.000 description 1
- 230000003547 miosis Effects 0.000 description 1
- 238000002156 mixing Methods 0.000 description 1
- 239000002365 multiple layer Substances 0.000 description 1
- 101700042652 old-1 Proteins 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 239000003973 paint Substances 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 238000000059 patterning Methods 0.000 description 1
- 235000006678 peppermint Nutrition 0.000 description 1
- 235000015132 peppermint Nutrition 0.000 description 1
- 235000007735 peppermint Nutrition 0.000 description 1
- 230000002093 peripheral Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000007639 printing Methods 0.000 description 1
- 230000000717 retained Effects 0.000 description 1
- 230000008786 sensory perception of smell Effects 0.000 description 1
- 230000013707 sensory perception of sound Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000004936 stimulating Effects 0.000 description 1
- 238000003860 storage Methods 0.000 description 1
- 230000001360 synchronised Effects 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 108091008011 thermoreceptors Proteins 0.000 description 1
- 230000036410 touch Effects 0.000 description 1
- 238000002834 transmittance Methods 0.000 description 1
- 230000001720 vestibular Effects 0.000 description 1
- 239000011800 void material Substances 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Abstract
method determines four dimensional (4D) plenoptic coordinates for content data by receiving content data; determining locations of data points with respect to a first surface to creating a digital volumetric representation of the content data, the first surface being a reference surface; determining 4D plenoptic coordinates of the data points at a second surface by tracing the locations the data points in the volumetric representation to the second surface where a 4D function is applied; and determining energy source location values for 4D plenoptic coordinates that have a first point of convergence. ng 4D plenoptic coordinates of the data points at a second surface by tracing the locations the data points in the volumetric representation to the second surface where a 4D function is applied; and determining energy source location values for 4D plenoptic coordinates that have a first point of convergence.
Description
A method determines four dimensional (4D) plenoptic coordinates for t data by receiving
content data; determining locations of data points with respect to a first surface to creating a
digital volumetric representation of the content data, the first surface being a reference surface;
ining 4D tic coordinates of the data points at a second surface by tracing the
locations the data points in the volumetric representation to the second surface where a 4D
function is applied; and determining energy source location values for 4D plenoptic coordinates
that have a first point of convergence.
NZ 789927
SYSTEM AND S OF HOLOGRAPHIC SENSORY DATA GENERATION,
LAHON AND TRANSPORT
TECHNICAL -FIELD
This disclosure is generally related to generation of holographic content compnsmg
sensory ation, and more specifically to generation of holographic content from nonholographic
information.
llACKGROUND
The dream ofan interactive virtual vvorld within a "holodeck" chamber as popularized by
Gene Roddenberry's Star Trek and originally envisioned by author Alexander I\foszkowski in the
early 1900s has been the inspiration for science fiction and technological im1ovation for nearly a
century. However, no compelling implementation of this experience exists outside of literature,
media, and the collective imagination of children and adults alike.
SUMMARY
[To be drafted after claims are finalized]]
These and other advantages ofthe t sure will become apparent to those skilled
in the art from the following ed description and the appended claims.
BRIEF PTION OF THE DRAWINGS
FIG. l is a schematic diagram illustrating design parameters for an energy directing system:
is a schematic diagram illustrating an energy system having an active device area
with a mechanical envelope;
is a schematic diagram illustrating an energy relay system:
is a schematic diagram illustrating an embodiment ofenergy relay elements adhered
together and fastened to a base stmcture;
is a schematic diagram rating an example ofa relayed image through multi
core optical :
FIG. SB is a tic diagram illustrating an example of a relayed image h an
energy relay that exhibits the properties ofthe Transverse Anderson Localization principle;
[OOH] is a schematic diag ram showing rays propagated from an energy surface to a view-er;
[0(H2] illustrates a perspective view of an energy waveguide system having a base
stmcture, four energy devices, and four energy relay elements forming a seamless energy
surface, in accordance with one embodiment of the present disclosure;
illustrates an energy relay system according to one embodiment of the
present disclosure;
illustrates a top-down perspective vie\v of an embodiment of an energy
waveguide system according to one embodiment of the present disclosure;
illustrates a front perspective view of the embodiment shown in ;
FIGS. 7E-7L illustrate various embodiments of an energy inhibiting element;
is a flow chart illustrating an embodiment of an embodiment of a process for
processing holographic sensory data;
is a schematic diagram of a virtual nment constmcted from sensory data;
is exemplii\1ing schematic m illustrating an embodiment ofenergy g;
is exemplifying schematic diagram illustrating anembodiment of ,
DETAILED DESCRIPTION
An ment ofa Holodeck (collectively calIed "Holodeck Design Parameters")
provide sufficient energy stimulus to fool the human sensory ors into believing that
received energy impulses within a virtual, social and interactive environment are real,
providing: l) binocular ity without external ories, head-mounted r, or
other peripherals; 2) accurate motion parallax, occlusion and y throughout a viewing
volume simultaneously for any number of viewers; 3) visual focus through synchronous
convergence, accommodation and miosis of the eye for all ved rays of light; and 4)
converging energy wave propagation of sufficient density and tion to exceed the
human sensory "resolution" for vision, hearing, touch, taste, smell, and/or balance.
Based upon conventional technology to date, we are decades, if not centuries away
from a technology capable of providing for ail receptive fields in a compelling way as
suggested by the Holodeck Design Parameters including the visual, auditory,
somatosensory, f,lUStatory, ory, and ular systems.
In this disclosure, the terms light field and holographic may be used
interchangeably to define the energy propagation for ation of any sensory receptor
response. While initial disclosures may refer to examples ofenergy and mechanical energy
propagation through energy surfaces for holographic imagery and volumetric haptics, all
fom1s of sensory receptors are envisioned in this disclosure. Furthermore, the principles
disclosed herein for energy propagation along propagation paths may be applicable to both
energy emission and energy capture.
Many technologies exist today that are often unfortunately confused with
holograms ing lenticular printing, Pepper's Ghost, glasses-free stereoscopic displays,
horizontal ax displays, head-mounted VR and AR displays (HJ\.1D), and other such
illusions generalized as "fauxlography." These technologies may exhibit some of the
desired properties of a tme holographic display, however, lack the ability to stimulate the
human visual sensory response in any way sufficient to address at least two of the four
identified Holodeck Design Parameters.
These challenges have not been successfully implemented by conventional
technology to produce a seamless energy surface sufficient for holographic energy
propagation. There are various approaches to implementing volumetric and ion
multiplexed light field displays ing parallax barriers, hogels, voxels, diffractive
optics, multi-view projection, holographic ers, rotational mirrors, multilayered
displays, time sequential displays, head mounted display, etc., however, conventional
approaches may involve a compromise on image quality, resolution, angular sampling
density, size, cost, safety, frame rate, etc., ultimately resulting in an unviable logy.
To achieve the ck Design Parameters for the visual, ry, somatosensory
systems, the human acuity of each of the tive systems is studied and understood to
propagate energy waves to sufficiently fool the human sensory receptors. The visual system
is capable of resolving to approximately l arc rnin, the auditory system may distin,guish
the difference in ent as little as three degrees, and the somatosensory system at the
hands are capable of discerning points ted by 2 - 12mm. \Vhile there are various and
cting ways to measure these es, these values are sufficient to understand the
systems and s to stimulate perception of energy propagation.
Of the noted sensory receptors, the human visual system is by far the most sensitive
given that even a single photon can induce sensation. For this reason, much of this
introduction \Vill focus on visual energy wave propagation, and vastly lower resolution
energy systems coupled within a disclosed energy guide surface may converge
appropriate signals to induce holographic sensory tion. Unless otherwise noted, ail
disclosures apply to all energy and sensory domains.
When calculating for effective design parameters ofthe energy propagation for the
visual system given a viewing volume and viewing distance, a d energy surface may
be designed to include many gigapixels of ive energy location density. For wide
viewing volumes, or near field viewing, the design parameters of a desired energy e
rnay include ds of gigapixels or more of effective energy location density By
comparison, a desired energy source may be designed to have 1 to 250 effective megapixels
of energy location density for ultrasonic propagation of volumetric haptics or an array of
36 to 3,600 effective energy locations for acoustic propagation of aphic sound
depending on input environmental variables. \Vhat is important to note is that with a
disclosed bi-directional energy surface architecture, all components may be configured to
form the appropriate structures forany energy domain to enable holographic propagation.
1-ImNever, the main nge to enable the Holodeck today involves available
visual technologies and energy device limitations. ic and ultrasonic devices are less
challenging given the orders of magnitude difference in desired density based upon sensory
acuity in the respective receptive field, although the complexity should not be
underestimated. While holographic emulsion exists with resolutions exceeding the desired
density to encode interference ns in static imagery, state-of-the-art y devices
are limited by resolution, data throughput and manufacturing feasibility. To date, no
singular y device has been able to gfully produce a light field having near
holographic resolution for visual acuity.
Production of a single silicon-based device capable of meeting the desired
resolution for a compelling light field display may not cal and may involve extremely
complex fabrication processes beyond the current manufacturing capabilities. The
limitation to tiling le existing display devices together involves the seams and gap
formed by the physical size of packaging, electronics, enclosure, optics and a number of
other challenges that inevitably result in an unviable technology from an imaging, cost
and/or a size standpoint.
The embodiments disclosed herein may provide a real-world path to building the
Holodeck.
e embodiments will now be described hereinafter with reference to the
accompanying drawings, which form a part : and which illustrate example
ments which may be practiced. As used in the disclosures and the appended claims,
the terms "embodiment", "example embodiment", and "exemplary embodiment" do not
necessarily refer to a single embodiment, although they may, and various example
embodiments may be readily combined and interchanged, out departing from the scope
or spirit ofexample embodiments. Furthermore, the terminology as used herein is for the
purpose of describing example embodiments only and is not intended to be tions. In
this respect, as used herein, the term "in" may include "in" and "on", and the tem1s "a,"
"an" and "the" may include singular and plural references. Furthermore, as used herein,
the term "by" may also mean "from", depending on the context. Furthermore, as used
herein, the term "if'' may also mean "when" or "upon," depending on the context.
Furthermore, as used , the words "and/or" may refer to and encompass any and all
possible combinations of one or more ofthe associated listed items.
Holographic System Considerations:
Overview of Light Field Energv ation tion
Light field and holographic display is the result of a plurality of projections where
energy surface locations provide angular, color and intensity information propagated
within a viewing volume. The disclosed energy surface es opportunities for
additional information to coexist and propagate through the same surface to induce other
sensory system ses. Unlike a stereoscopic display, the viewed position of the
converged energy propagation paths in space do not vary as the viewer moves around the
viewing volume and any number ofvievvers may simultaneously see propagated objects in
real-world space as if it was truly there. In some embodiments, the ation of energy
may be located in the same energy propagation path but in opposite directions. For
example, energy emission and energy capture along an energy propagation path are both
possible in some embodiments of the present disclosed.
is a schematic diagram illustrating variables relevant for stimulation of
sensory receptor response. These variables may include surface al l O 1, surface
h i 02, surface height 103, a determined target seating distance 118, the target seating
field of view field of view from the center of the display 104, the number of intermediate
samples demonstrated here as samples between the eyes 105, the average adult inter-ocular
separation 106, the average resolution of the human eye in arcmin 107, the horizontal field
of view formed between the target viewer location and the surface width 108, the vertical
field of view formed between the target viewer location and the surface height 109, the
resultant horizontal waveguide element resolution, or total number of elements, across the
surface 110, the resultant vertical waveguide element resolution, or total number of
ts, across the surface 111, the sample distance based upon the inter-ocular g
between the eyes and the number of intermediate samples for angular projection between
the eyes 112, the r sampling may be based upon the sample distance and the target
seating distance 113, the total resolution Horizontal per waveguide element derived from
the angular sampling desired 114, the total resolution Vertical per waveguide t
derived from the angular sampling desired 115, device ntal is the count of the
detennined number ofdiscreet energy s d 116, and device Vertical is the count
of the determined number of discreet energy sources desired 117.
A method to understand the desired minimum resolution may be based upon the
following criteria to ensure sufficient stimulation of visual (or other) sensory or
response: surface size (e.g., 84" diagonal), surface aspect ratio (e.g., 16:9), seating distance
(e.g., 128" from the display), seating field of view (e.g., 120 degrees or+/- 60 degrees
about the center of the display), desired intermediate samples at a distance (e.g., one
additional propagation path between the eyes), the average inter-ocular separation of an
adult (approximately 65mm), and the average resolution of the human eye (approximately
1 arcmin). These example values should be considered placeholders depending on the
specific application design parameters.
r, each of the values attributed to the visual sensory receptors may be
ed with other systems to determine d propagation path ters. For other
energy propagation embodiments, one may consider the auditory system's angular
sensitivity as low as three s, and the somatosensory system's spatial resolution of
the hands as small as 2 - 12mm.
While there are various and conflicting ways to e these sensory acuities,
these values are sufficient to understand the systems and methods to stimulate perception
of virtual energy propagation. There are many ways to consider the design resolution, and
the below proposed methodology combines pragmatic product considerations with the
biological resolving limits of the y systems. As will be appreciated by one of
ordinary skill in the art, the following overview is a fication of any such system
design, and should be considered for exemplary purvoses only.
With the tion limit of the sensory system understood, the total energy
waveguide elernent density rnay be calculated such that the receiving y systern
cannot discern a single energy waveguide element from an adjacent element, given:
i d_t
, ur ace S , f Aspec. t R t .a w =
_w_Height (H)
_h_(;...¼ ....;..
Surface Horizontal Size = Surface al* ( , 1
\I ' ·1rv/11+ (!2..)2
• Surface Vertical Size= Surface Diagonal* ( 1 w)
✓ p+ (H)
Surface Horizontal Size
II- onzonta.. l F. ldie of rr·v 1ew = 2 * atan ( )
2 * Seating Distance
Surface Verticle Size
Vertica. l 1 ... , ie . old 1. V.iew = 2* atan ( )
2 * Seating Distance
Horizontal Element Resolution= Horizontal FoV * 0
Eye Resolution
al Element Resolution= Vertical FoV* ----60
Eye Resolution
The above calculations result in approximately a 32xl8° field of view resulting in
imately l 920xl080 ed to nearest format) energy 'vvavef,:ruide elements being
desired. One may also constrain the variables such that the field of view is consistent for
both (u, v) to provide a more regular spatial sampling ofenergy locations (e.g. pixel aspect
ratio). The angular ng of the system assumes a defined target viewing volume
location and additional propagated energy paths between two points at the optimized
distance, given:
• inter--·Ocular Distance
S .amp el D • tis ·ance =.
r of Desired Intermediate Samples+l)
• Sample Distance
An, l S, z · atan ------
m.q =
gu.ar, amp " (Seating Distance)
In this case, the inter-ocular distance is leveraged to calculate the sample distance
although any metric may be ged to account for appropriate number of samples as a
given distance. With the above variables considered, approximately one ray per 0.57° may
be desired and the total system resolution per independent sensory system may be
detennined, given:
Locatwns Per nt(N)_ • _ � , = Seating FoV. . .
Anguiar Sampling
• Total Resolution H = N * Horizontal Element Resolution
• Total Resolution V= N * al Element Resolu.tion
With the above io given the size of energy surface and the r resolution
addressed for the visual acuity systern, the resultant energy surface rnay desirably include
approximately 400k x 225k pixels of energy resolution locations, or 90 gigapixels
holographic propagation density. These variables provided are for ary purposes
only and many other sensory and energy metrology considerations should be considered
for the optimization of holographic ation of energy. In an additional embodiment, l
gigapixel of energy resolution locations may be desired based upon the input variables. In
an additional embodiment, 1,000 gigapixels of energy resolution locations may be desired
based upon the input variables.
CmTent Technologv tions:
Active Area, Device Electronics, Packaging, and the nical Envelope
illustrates a device 200 having an active area 220 with a certain mechanical
form factor The device 200 may include s 230 and electronics 240 for powering and
interface to the active area 220, the active area having a dimension as shown by the x and
y arrows. This device 200 does not take into account the cabling and mechanical structures
to drive, power and cool components, and the mechanical footprint may be further
rninimized by introducing a flex cable into the device 200. The minimum footprint for such
a device 200 may also be referred to as a mechanical envelope 210 having a dimension as
shown by the M:x and M:y arrows. This device 200 is for illustration purposes only and
custom electronics designs may further decrease the mechanical envelope overhead, but in
almost all cases may not be the exact size of the active area of the device. In an embodiment,
this device 200 illustrates the dependency of electronics as it relates to active image area
220 for a micro OLED, DLP chip or LCD panel, or any other technology with the purpose
ofimage illumination.
In some ments, it may also be le to consider other prqjection
technologies to ate multiple images onto a larger overall display. However, this may
come at the cost ofgreater complexity for throw distance, minimum focus, optical y,
unifom1 field resolution, chromatic aberrntion, thermal properties, ation, alignment,
additional size or form factor. For most practical applications, g tens or hundreds of
these projection sources 200 may result in a design that is much larger with less reliability.
For exemplary purposes only, assuming energy devices with an energy location
density of 3840 x 2160 sites, one may determine the number of individual energy s
(e.g., device 100) desired for an energy surface, given:
• 1)euces
_ o_t a_l_R_e_so_l_ut_ i,_ -ri _ t ,_'-!
1• I- { _
, - _ T Device Resolution H
uevices n . V = ------ Tota.! Resolution V
Device Resolution V
Given the above resolution considerations, approximately 105 x 105 devices
r to those shown in may be desired. It should be noted that many devices
consist of various pixel strnctures that may or may not map to a regular grid. In the event
that there are additional sub-pixels or locations within each full pixel, these may be
exploited to generate additional tion or angular density Additional signal processing
may be used to determine how to convert the light field into the correct (u,v) coordinates
depending on the specified location of the pixel structure(s) and can be an explicit
characteristic of each device that is known and calibrated. Further, other energy domains
may involve a different handling of these ratios and device strnctures, and those d in
the art will understand the direct intrinsic relationship between each of the desired
frequency domains. This will be shown and discussed in more detail in subsequent
disclosure.
The resulting calculation may be used to understand how many of these dual
s may be desired to produce a full resolution energy surface. In this case,
approximately 105 x 105 or approximately 11,080 devices may be desired to achieve the
visual acuity threshold. The challenge and novelty exists within the fabrication of a
seamless energy surface from these available energy locations for sufficient sensory
holographic propagation.
Summary of Seamless Energy Surfaces:
(;g_nfigurations and Designs for Arrays of Energy Relay�
In some embodiments, approaches are disclosed to address the nge of
ting high energy location density from an array of dual devices without seams
due to the limitation of mechanical structure for the devices. In an embodiment, an energy
propagating relay system may allow for an increase the effective size of the active device
area to meet or exceed the mechanical dimensions to configure an array of relays and form
a singular seamless energy e.
illustrates an embodiment of such an energy relay system 300. As shown,
the relay system 300 may include a device 310 mounted to a mechanical envelope 320,
with an energy relay element 330 propagating energy from the device 310. The relay
element 330 may be configured to e the y to mitigate any gaps 340 that may be
produced when multiple mechanical envelopes 320 of the device are placed into an array
of multiple devices 310.
For example, if a device's active area 310 is 20mm x l0mm and the mechanical
envelope 320 is 40mm x 20mm, an energy relay element 330 may be designed with a
magnification of 2:1 to e a tapered form that is approximately 20mm x 10mm on a
rninified end (arrow A) and 40rnm x 20mm on a rnagnified end (arrnw B), providing the
ability to align an array of these elements 330 together seamlessly without altering or
colliding with the mechanical envelope 320 of each device 310. Mechanically, the relay
elements 330 may be bonded or fosed together to align and polish ensuring minimal seam
gap 340 between devices 310. In one such embodiment, it is possible to achieve a seam
gap 340 smaller than the visual acuity limit of the eye.
rates an example of a base structure 400 having energy relay ts
410 formedtogether and securely fastened to an additional mechanical structure 430. The
mechanical structure of the seamless energy surface 420 provides the ability to couple
nmltiple energy relay ts 410, 450 in series to the same base structure through
bonding or other mechanical processes to mount relay elements 410, 450. In some
embodiments, each relay element 410 may be fused, bonded, adhered, pressure fit, aligned
or ise attached together to form the resultant seamless energy surface 420. In some
embodiments, a device 480 may be mounted to the rear of the relay t 410 and
aligned passively or actively to ensure appropriate energy location alignment within the
detennined tolerance is maintained.
In an embodiment, the seamless energy surface comprises one or more energy
locations and one or more energy relay elernent stacks comprise a first and second side and
each energy relay t stack is ed to form a sin!:,:rular seamless energy smface
directing energy along propagation paths extending between one or more energy locations
and the seamless energy surface, and where the separation between the edges of any two
adjacent second sides of the terminal energy relay ts is less than the minimum
perceptible contour as defined by the visual acuity of a human eye having better than 20i40
vision at a distance greater than the width of the singular seamless energy surface.
In an embodiment, each of the seamless energy surfaces comprise one or more
energy relay elements each with one or more strnctures forming a first and second surface
with a erse and longitudinal orientation. The first relay surface has an area different
than the second resulting in positive or negative magnification and configured with explicit
surface contours for both the first and second s passing energy through the second
relay surface to substantially fill a +i- 10 degree angle with respect to the normal of the
surface contour across the entire second relay surface.
In an ment, multiple energy domains may be configured within a single, or
between multiple energy relays to direct one or more y holographic energy
propagation paths including visual, acoustic, tactile or other energy domains.
In an embodiment, the seamless energy surface is configured with energy relays
that comprise two or more first sides for each second side to both receive and emit one or
more energy domains simultaneously to e bi-directional energy propagation
throughout the system.
In an embodiment, the energy relays are provided as loose coherent elements.
uction to Component Engineered Structures:
Disclosed Advances in Transverse Anderson Localization Energv Relavs
The properties of energy relays may be significantly optimized according to the
ples disclosed herein for energy relay elements that induce Transverse Anderson
Localization. Transverse Anderson Localization is the propagation of a ray transported
through a transversely disordered but longitudinally consistent al.
This implies that the effect ofthe materials that produce the Anderson Localization
ena may be less impacted by total internal reflection than by the randomization
between multiple-scattering paths where wave interference can cornpletely limit the
propagation in the transverse orientation while continuing in the longitudinal orientation.
Of significant additional benefit is the ation of the cladding of traditional
core optical fiber als. The cladding is to functionally eliminate the scatter of
energy between fibers, but simultaneously act as a barrier to rays of energy thereby
reducing transmission by at least the core to clad ratio (e.g., a core to clad ratio of 70:30
,vill transmit at best 70% ofreceived energy transmission) and additionally forms a strong
ted patterning in the propagated energy.
illustrates an end view of an example of one such non-Anderson
zation energy relay 500, wherein an irnage is relayed through multi-core optical
fibers where pixilation and fiber noise may be exhibited due to the intrinsic properties of
the optical . \Vith traditional mode and multi-core optical fibers, relayed images
may be intrinsically pixelated due to the properties oftotal internal reflection ofthe discrete
array of cores where any cross-talk between cores will reduce the modulation transfer
function and increase blurring. The resulting imagery produced with traditional nmlti-core
optical fiber tends to have a residual fixed noise fiber n similar to those shown in FIG.
, illustrates an e of the same relayed image 550 h an energy
relay comprising materials that exhibit the properties ofTransverse Anderson Localization,
where the relayed pattern has a greater density grain stmctures as compared to the fixed
fiber pattern from . In an embodiment, relays comprising randomized microscopic
component ered strnctures induce Transverse Anderson Localization and transport
light more efficiently with higher propagation of resolvable resolution than commercially
available multi-mode glass optical fibers.
There is significant age to the Transverse Anderson Localization material
ties in terms of both cost and weight, where a similar optical grade glass material,
may cost and weigh upwards of 10 to 100-fold more than the cost for the same material
generated within an embodiment, wherein disclosed systems and methods comprise
ized microscopic component engineered ures demonstrating significant
opportunities to improve both cost and quality over other technologies known in the art.
In an embodiment, a relay element exhibiting Transverse Anderson Localization
may comprise a plurality ofat least t\vo different component engineered structures in each
of three orthogonal planes ed in a dimensional lattice and the plurality of structures
form randomized distributions of material wave propagation properties in a transverse
plane within the dimensional lattice and channels of similar values of material wave
propagation properties in a longitudinal plane within the dimensional lattice, wherein
localized energy waves propagating h the energy relay have higher transport
ency in the longitudinal orientation versus the transverse orientation.
In an embodiment, le energy domains may be configured within a single, or
between multiple Transverse Anderson Localization energy relays to direct one or rnore
y holographic energy propagation paths including visual, acoustic, tactile or other
energy domains.
In an embodiment, the seamless energy surface is configured with Transverse
Anderson Localization energy relays that comprise two or more first sides for each second
side to both receive and emit one or more energy domains simultaneously to provide bidirectional
energy propagation throughout the system.
In an embodiment, the Transverse Anderson zation energy relays are
confi!:, :rured as loose coherent or flexible energy relay elements.
Considerations for 4D Plenoptic Functions:
Selective Propagation of Energy through Holographic \Vaveguide Arrays
As sed above and herein throughout, a light field display system generally
includes an energy source (e.g., illumination source) and a ss energy surface
confi,gured with sufficient energy location density as articulated in the above discussion. A
plurality of relay elements may be used to relay energy from the energy devices to the
seamless energy surface. Once energy has been delivered to the seamless energy surface
with the requisite energy location density, the energy can be propagated in accordance with
a 4D plenoptic on through a disclosed energy waveguide system. As will be
iated by one of ordinary skill in the art, a 4D plenoptic function is well known in the
art and will not be elaborated further herein.
The energy 'vvaveguide system selectively propagates energy h a plurality of
energy locations along the seamless energy surface representing the l coordinate of
the 4D plenoptic function with a structure configured to alter an angular direction of the
energy waves passing h representing the angular component of the 4D tic
function, wherein the energy waves propagated may converge in space in accordance with
a plurality of propagation paths directed by the 4D plenoptic function.
Reference is now made to illustrating an e of light field energy
surface in 4D image space in accordance 'vvith a 4D plenoptic function. The fi!:, :rure shows
ray traces of an energy e 600 to a viewer 620 in describing how the rays of energy
converge in space 630 from various positions within the vievving volume. As shown, each
waveguide element 610 defines four dimensions of infonnation describing energy
propagation 640 through the energy surface 600. Two l dimensions (herein referred
to as x and y) are the physical plurality of energy locations that can be viewed in image
space, and the r components theta and phi (herein referred to as u and v), which is
viewed in virtual space when projected through the energy waveguide array. In general,
and in accordance with a 4D plenoptic function, the plurality ofwaveguides (e.g., lenslets)
are able to direct an energy location from the x, y dimension to a unique location in virtual
space, along a direction defined by the u, v an,gular component, in g the holographic
or light field system described herein.
However, one skilled in the art will understand that a significant challenge to light
field and aphic display technologies arises from uncontrolled propagation ofenergy
due designs that have not accurately accounted for any of diffraction, scatter, diffusion,
angular direction, calibration, focus, collimation, ure, uniformity, element cross-talk,
as well as a multitude of other parameters that contribute to decreased ive resolution
as well as an inability to accurately converge energy with sufficient fidelity
In an embodiment, an approach to selective energy propagation for addressing
challenges associated with aphic display may include energy inhibiting elements and
substantially filling waveguide apertures with near-collimated energy into an environment
d by a 4D plenoptic function.
In an embodiment, an array of energy waveguides may define a plurality of energy
ation paths for each waveguide element configured to extend through and
substantially fill the waveguide element's effective aperture in unique directions defined
by a ibed 4D function to a plurality of energy locations along a seamless energy
surface inhibited by one or more elements positioned to limit propagation of each energy
location to only pass h a single waveguide element.
In an embodiment, multiple energy s may be configured within a single, or
between multiple energy waveguides to direct one or more sensory holographic energy
propagations including visual, acoustic, tactile or other energy domains.
In an embodiment, the energy \vaveguides and ss energy e are
confi!:, :rured to both receive and emit one or more energy domains to provide bi-directional
energy propagation throughout the system.
In an embodiment, the energy waveguides are configured to propagate non-linear
or non-regular distributions ofenergy, including non-transmitting void regions, leveraging
digitally encoded, diffractive, refractive, reflective, grin, holographic, Fresnel, or the like
waveguide configurations for any seamless energy surface orientation including wall, table,
floor, ceiling, room, or other geornetry based environments. In an additional embodiment,
an energy waveguide element may be configured to produce various geometries that
provide any surface e and/or tabletop viewing allowing users to vie\v holographic
imagery from al! around the energy surface in a 360-degree configuration.
In an embodiment, the energy waveguide array elements may be reflective surfaces
and the arrangement of the elements may be hexagonal, square, irregular, semi-regular,
curved, anar, spherical, cylindrical, tilted regular, tilted irregular, spatially varying
and/or multi-layered.
For any component within the seamless energy surface, waveguide, or relay
ents rnay include, but not limited to, l fiber, silicon, glass, r, l
relays, diffractive, holographic, refractive, or reflective elements, optical face plates,
energy combiners, beam splitters, prisms, polarization elements, spatial light modulators,
active , liquid crystal cells, transparent displays, or any similar materials exhibiting
on zation or total internal reflection.
Realizing the Holodeck:
Aggregation of Bi-directional Seamless Energy Surface Svstems To Stimulate Human
Sensorv Receptors \.Vithin Holographic Enviromnents
It is possible to construct large-scale environments of seamless energy surface
systems by , fusing, bonding, attaching, and/or stitching multiple seamless energy
surfaces together forming arbitrary sizes, shapes, contours or form-factors including entire
rooms. Each energy surface system may comprise an assembly having a base structure,
energy surface, relays, ide, devices, and electronics, collectively configured for bidirectional
aphic energy propagation, emission, reflection, or sensing.
In an embodiment, an environment of tiled seamless energy systems is aggregated
to form large seamless planar or curved walls including installations comprising up to all
surfaces in a given nment, and configured as any combination of searnless,
discontinuous planar, faceted, curved, cylindrical, spherical, geometric, or gular
geometries.
In an embodiment, aggregated tiles of planar surfaces form wall-sized systems for
theatrical or venue-based holographic entertainment. In an ment, aggregated tiles
of planar surfaces cover a room with four to six walls ing both ceiling and floor for
cave-based holographic lations. In an embodiment, aggregated tiles of curved
surfaces produce a cylindrical seamless nment for imrnersive holographic
installations. In an embodiment, aggregated tiles of seamless spherical es form a
holographic dome for immersive Holodeck-based experiences.
In an embodiment, aggregates tiles ofseamless curved energy waveguides provide
mechanical edges following the precise n along the boundary of energy inhibiting
elements within the energy waveguide structure to bond, align, or fuse the adjacent tiled
mechanical edges of the adjacent waveguide surfaces, resulting in a modular and seamless
energy waveguide system.
In a further embodiment of an aggregated tiled environment, energy is propagated
bi-directionally for multiple simultaneous energy domains. In an additional embodiment,
the energy surface provides the ability to both display and capture simultaneously from the
same energy surface with waveguides designed such that light field data may be projected
by an illmnination source through the wave.guide and simultaneously received through the
same energy surface. In an additional embodiment, onal depth sensing and active
scanning technologies may be leveraged to allow for the interaction between the energy
propagation and the viewer in correct world coordinates. In an additional embodiment, the
energy e and waveguide are operable to emit, reflect or converge frequencies to
induce tactile sensation or volumetric haptic feedback. In some embodiments, any
combination of ectional energy propagation and ated surfaces are possible.
In an embodiment, the system comprises an energy waveguide capable of bidirectional
emission and sensing of energy through the energy smface with one or more
energy devices independently paired with two-or-more-path energy ers to pair at
least t\vo energy devices to the same portion of the seamless energy surface, or one or more
energy devices are secured behind the energy surface, proxirnate to an onal
component secured to the base stmcture, or to a location in front and outside of the FOY
of the waveguide for off-axis direct or reflective tion or sensing, and the resulting
energy surface provides for bi-directional transmission of energy allowing the waveguide
to converge , a first device to emit energy and a second device to sense energy, and
where the information is processed to perform computer vision related tasks including, but
not limited to, 4D plenoptic eye and l tracking or sensing of interference 'vvithin
propagated energy patterns, depth estimation, proximity, motion tracking, image, color, or
sound formation, or other energy frequency is. In an additional ment, the
tracked positions actively calculate and modify positions of energy based upon the
interference between the bi-directional captured data and projection information.
In some embodiments, a plurality of combinations of three energy s
comprising an ultrasonic sensor, a e energy display, and an onic emitting device
are configured together for each of three first relay surfaces propagating energy combined
into a single second energy relay surface with each of the three first surfaces comprising
engineered ties specific to each device's energy domain, and two engineered
waveguide elements configured for ultrasonic and energy respectively to provide the ability
to direct and converge each device's energy independently and substantially unaffected by
the other waveguide elements that are configured for a separate energy domain.
In some embodiments, disclosed is a calibration procedure to enable efficient
cturing to rernove system cts and produce a geometric mapping ofthe resultant
energy surface for use with ng/decoding technologies as well as dedicated ated
systems for the conversion of data into calibrated information appropriate for energy
propagation based upon the calibrated configuration files.
In some embodiments, additional energy waveguides in series and one or more
energy s may be integrated into a system to produce opaque holographic pixels.
In some embodiments, additional waveguide elements may be integrated
sing energy inhibiting elements, beam-splitters, prisms, active parallax barriers or
polarization technologies in order to provide spatial and/or r resolutions greater than
the diameter of the waveguide or for other super-resolution purposes.
In some embodiments, the disclosed energy system may also be configured as a
wearable bi-directional , such as virtual reality (VR) or augmented reality (AR). In
other embodiments, the energy system may e adjustment optical element(s) that
cause the displayed or received energy to be focused proximate to a ined plane in
space for a viewer. In some embodiments, the waveguide array may be incorporated to
holographic head-mounted-display. In other embodiments, the system may include
nmltiple optical paths to allow for the viewer to see both the energy system and a real
'vvorld environment (e.g., transparent holographic display). In these instances, the system
may be ted as near field in on to other methods.
In some embodiments, the transmission of data comprises encoding processes with
selectable or variable compression ratios that receive an arbitrary dataset of information
and metadata; analyze said dataset and receive or assign material ties, vectors,
surface IDs, new pixel data forming a more sparse dataset, and n the received data
may comprise: 2D, stereoscopic, multi-view, metadata, light field, holographic, geometry,
vectors or vectorized metadata, and an encoder/decoder may provide the ability to convert
the data in real-time or off-line comprising image sing for: 2D; 2D plus depth,
metadata or other vectorized infom1ation; stereoscopic, stereoscopic plus depth, metadata
or other ized information; view; multi-view plus depth, metadata or other
vectorized information; aphic; or light field content; through depth estimation
algorithms, with or without depth metadata; and an e ray tracing methodology
approp1iately maps the resulting converted data produced by inverse ray tracing from the
various 2D, scopic, nmlti-view, volmnetric, light field or holographic data into real
world nates through a characterized 4D plenoptic function. In these embodiments,
the total data transmission desired may be nmltiple orders of magnitudes less transmitted
information than the raw light field dataset.
Energy Directing Devices Suitable for Presentim1, Holographic Senso:rv Data
In an embodiment, the optomechanical y device may be capable of emitting
and guiding light to fonn 2D, stereoscopic, multiview, plenoptic, 4D, volumetric, light
field, holographic, or any other visual representation of light.
is an example of a light field optomechanical system if configured with
emissive display s, optical relays, and a waveguide that is realized as an array of
refractive elements such as a micro lens array, where a visible image from one or more
displays may be optically relayed before being transmitted to the energy surface, where the
array of refractive elements provides a g between each location on the energy
surface and the direction of projection of the light from that location, such that a 4D
tric light field image may be projected.
In an embodiment, the waveguide may be operable to converge rays of light to
induce both vergence and accommodation from an observer point of view.
In an embodiment, the waveguides and energy relays may be formed or ed
with various surface geometries. In an embodiment, the energy relays include elements that
induce transverse Anderson localization. In an embodiment, the energy relays are
bidirectional and may both emit and/or project energy.
In one embodiment, an energy system configured to direct energy according to a
four-dimensional ( 4D) plenoptic function includes a plurality of energy s. In some
embodiments, the plurality of energy devices include illumination s emitting image
infonnation, where the image infonnation includes emissive, projection, or reflective
display technologies, leveraging e, IR, UV, coherent, laser, infrared, polarized or any
other omagnetic illumination source. In other ments, the plurality of energy
devices include mechanical energ)1 en1itting devices configured to e irr11nersive
audio or volumetric tactile sensation from an acoustic field.
In some embodiments, the energy system as confi,gured above may further include
a base structure (e.g., 72) such that the plurality ofenergy devices, the energy relay system,
and the energy waveguide system rnay all be coupled to the base structure. In other
embodiments, the plurality of energy devices, the energy relay system and the energy
waveb:ruide system may be coupled to the base structure with one or more mounting
brackets.
In some embodiments the pluralitv of ener0v s include ener0v devices for, .; b.1 b.1
ing or sensing energy, including ical, chemical, transfer, thermal, electric,
potential, kinetic, ic, gravitational, radiant, energy, structured, unstructured, or
other forrns ofenergy. In other embodiments, the plurality gy devices include energy
devices for propagating or emitting energy, including mechanical, chemical, transfer,
thermal, electric, potential, kinetic, magnetic, gravitational, radiant, energy, structured,
unstrnctured, or other forms of energy. In yet other embodiments, the plurality of energy
devices include acoustic receiving s confib:rured to provide sensory feedback or
audible controls
In one embodiment, the energy system further includes an energy relay system
(e.g., 6110 as best shown in ) having one or more energy relay elements, where
each ofthe one or more energy relay ts includes a first surface and a second surface,
the second surface of the one or more energy relay elements being arranged to form a
singular seamless energy surface of the energy relay system, and where a first plurality of
energy propagation paths extend from the energy locations in the plurality of energy
s through the ar seamless energy surface of the energy relay system. This will
be discussed in more detail below.
Reference is now made to illustrating an energy relay system 6110, in an
orthogonal view in accordance with one embodiment of the present disclosure. In one
embodiment, the energy relay system 6110 may include t\vo or more relay elements 6112,
each relay element 6112 formed ofone or more structures, each relay element 6112 having
a first surface 6114, a second surface 6116, a transverse orientation (generally parallel to
the es 6114, 6116) and a longitudinal orientation (generally dicular to the
es 6114, 6116). In one ment, the surface area ofthe first surface 6114 may be
different than the surface area ofthe second surface 6116. For example, the surface area of
the first surface 6114 may be r or lesser than the surface area of the second surface
6116. In another embodiment, the e area ofthe first surface 114 may be the same as
the surface area of the second surface 6116. Energy waves can pass from the first surface
6114 to the second surface 6116, or vice versa.
In one embodiment, the relay element 6112 of the energy relay system 6110
includes a sloped profile portion 6118 between the first surface 6114 and the second surface
6116. In operation, energy waves propagating between the first surface 6114 and the
second surface 6116 may have a higher transport efficiency in the longitudinal orientation
than in the transverse orientation, and energy waves passing through the relay element 6112
may result in spatial rnagnification or spatial de-magnification. In other words, energy
waves passing through the relay element 6112 of the relay element device 6110 may
experience sed magnification or decreased rnagnification. In some embodiments, the
one or more ures for forming the energy relay element 61 l O may include glass,
carbon, optical fiber, optical film, c, polymer, or mixtures thereof.
In one embodiment, the energy waves passing through the first surface 6114 has a
first resolution, while the energy waves passing through the second surface 6116 has a
second tion, and the second resolution is no less than about 50 %) of the first
resolution. In another embodiment, the energy waves, while having a m profile when
presented to the first surface, may pass through the second surface radiating in every
direction with an energy density in the forward direction that substantially fills a cone with
an opening angle of+/- l O degrees relative to the normal to the second surface, irrespective
ofiocation on the second relay surface.
In some embodiments, the first surface 6114 may be configured to receive energy
from an energy wave source, the energy wave source including a mechanical envelope
having a width different than the width of at least one of the first e 6114 and the
second surface 6116.
In each relay 6112, energy is transported between first and second surfaces which
defines the longitudinal orientation, the first and second surfaces of each of the relays
extends generally along a transverse orientation defined by the firstand second directions,
where the longitudinal orientation is substantially normal to the transverse orientation. In
one embodiment, energy waves propagating through the ity of relays have higher
transport efficiency in the longitudinal orientation than in the transverse orientation due to
randomized refractive index variability in the erse orientation coupled with l
refractive index variation in the longitudinal orientation. In some embodiments where each
relay is constructed of multicore fiber, the energy waves propagating within each relay
element may travel in the udinal orientation determined by the alignment of fibers in
this orientation.
In an embodiment, a separation between the edges of any two nt second sides
of the terminal energy relay elements may be less than a minimum perceptible contour as
defined by the visual acuity of a human eye having better than 20/40 vision at a distance
from the seamless energy surface that is greater than the lesser of a height of the singular
seamless energy smface or a width of the singular seamless energy surface.
In one embodiment, the plurality of energy relay ts in the stacked
configuration may e a plurality of faceplates. In some embodiments, the plurality of
faceplates may have different lengths or are loose nt optical relays. In other
embodiments, the ity of elements may have sloped profile ns similar to that of
, where the sloped profile portions may be angled, linear, curved, d, faceted
or aligned at a non-perpendicular angle relative to a nmmal axis of the relay element. In
yet another embodiment, energy waves propagating through the plurality of relay elements
have higher transport efficiency in the longitudinal orientation than in the transverse
orientation due to ized tive index variability in the transverse orientation
coupled with minimal refractive index variation in the longitudinal orientation. In
embodiments where each energy relay is constructed of multicore fiber, the energy waves
propagating within each relay element may travel in the longitudinal orientation
ined by the alignment of fibers in this orientation.
In some embodiments, the one or more relay elements (e.g., 6112) includes fused
or tiled rnosaics, where any seams between adjacent fused or tiled mosaics are separated
by or are less than the minimum perceptible contour as defined by the visual acuity of a
human eye having better than 20/40 vision at a distance at or greater than the width or
height of the singular seamless energy surface.
In other embodiments, the one or more relay elements (e.g., 6112) includes: l
fiber, silicon, glass, polymer, optical relays, diffractive elements, holographic optical
elements, refractive ts, reflective elements, optical face plates, l combiners,
beam splitters, prisms, polarization components, spatial light modulators, active pixels,
liquid crystal cells, transparent displays, or any similar als having on
localization or total internal reflection properties for forming the singular seamless energy
surface.
In yet other embodiments, the one or more relay elements (e.g., 6112) are
confi gu red to accommodate a shape of the singular seamless energy e including
planar, spherical, cylindrical, conical, faceted, tiled, regular, non-regular, or any other
geometric shape for a specified application.
In another embodiment, the system further es an energy waveguide system
(e.g., 7100 as best shown in FIGS. 7C-7L) having an array of energy waveguides, where a
second plurality of energy propagation paths extend from the singular seamless energy
surface through the array of energy waveguides in directions determined by a 4D plenoptic
function.
illustrates a top-down perspective view of an embodiment of an energy
ide system 7100 operable to define a plurality of energy propagation paths 7108.
Energy waveguide system 7100 comprises an army of energy waveguides 7112 confi gu red
to direct energy therethrough along the plurality of energy propagation paths 7108. In an
embodiment, the plurality of energy propagation paths 7108 extend through a plurality of
energy locations 7118 on a first side of the array 7116 to a second side of the array 7114.
ing to FIGS. 7C and 7L, in an embodiment, a first subset 7290 of the ity
of energy ation paths 7108 extend through a first energy location 7122. The first
energy waveguide 7104 is ured to direct energy along a first energy propagation path
7120 of the first subset 7290 of the plurality of energy propagation paths 7108. The first
energy propagation path 7120 may be defined by a first chief ray 7138 formed between the
first energy location 7122 and the first energy waveguide 7104. The first energy
propagation path 7120 may comprise rays 7138A and 71388, formed between the first
energy location 7122 and the first energy waveguide 7104, which are directed by first
energy ide 7104 along energy propagation paths 7120A and 7120B, respectively.
The first energy propagation path 7120 may extend from the first energy waveguide 7104
towards the second side of the array 7114. In an embodiment, energy directed along the
first energy propagation path 7120 comprises one or more energy ation paths
between or including energy propagation paths 7120A and 7120B, which are directed
h the first energy waveguide 7104 in a direction that is substantially el to the
angle propagated through the second side 7114 by the first chiefray 7138.
Embodiments may be configured such that energy directed along the first energy
propagation path 7120 may exit the first energy waveguide 7104 in a direction that is
ntially parallel to energy propagation paths 7120A and 7120B and to the first chief
ray 7138. It may be assumed that an energy propagation path extending through an energy
f,:ruide element 7112 on the second side 7114 ses a plurality of energy
propagation paths of a substantially sirnilar propagation direction.
is a front view illustration of an embodiment of energy waveguide system
7100. The first energy propagation path 7120 rnay extend tmNards the second side of the
array 7114 in a unique direction 7208 extending from the first energy waveguide 7104,
which is determined at least by the first energy location 7122. The first energy waveguide
7104 may be defined by a spatial coordinate 7204, and the unique direction 7208 which is
determined at least by first energy location 7122 may be d by an angular coordinate
7206 defining the directions of the first energy propagation path 7120. The spatial
coordinate 7204 and the angular nate 7206 may form a four-dimensional plenoptic
coordinate set 7210 which defines the unique direction 7208 ofthe first energy propagation
path 7120.
In an embodiment, energy directed along the first energy propagation path 7120
h the first energy waveguide 7104 substantially fiils a first aperture 7134 of the first
energy waveguide 7104, and propagates along one or more energy propagation paths which
lie between energy propagation paths 7120A and 7120B and are parallel to the direction of
the first energy propagation path 7120. In an embodiment, the one or more energy
propagation paths that substantially fill the first aperture 7134 may comprise greater than
50�'"o of the first aperture 7134 diameter.
[OJ 14] In a preferred embodiment, energy directed along the first energy propagation path
7120 through the first energy ide 7104 which substantially fills the first aperture
7134 may comprise between 50% to 80°/o of the first aperture 7134 diameter
Turning back to FIGS. 7C and 7E-7L, in an ment, the energy waveguide
systern 7100 rnay further comprise an energy inhibiting element 7124 positioned to limit
propagation of energy between the first side Ti 16 and the second side Ti 14 and to t
energy propagation between adjacent ides 7112. In an embodiment, the energy
inhibiting element is configured to inhibit energy propagation along a portion of the first
subset 7290 of the plurality of energy propagation paths 7108 that do not extend through
the first aperture 7134. In an embodiment, the energy ting element 7124 may be
located on the first side 7116 between the array of energy waveguides 7112 and the
plurality of energy locations 7118. In an embodiment, the energy inhibiting element 7124
may be located on the second side 7114 between the plurality ofenergy locations 7118 and
the energy propagation paths 7108. In an embodiment, the energy inhibiting element 7124
may be located on the first side 7116 or the second side 7114 orthogonal to the array of
energy waveguides 7112 or the plurality ofenergy locations 7118.
In an embodiment, energy directed along the first energy ation path 7120
may converge with energy directed along a second energy propagation path 7126 through
a second energy waveguide 7128. The first and second energy propagation paths may
converge at a location 7130 on the second side 7114 of the array 7112. In an embodiment,
a third and fourth energy propagation paths 7140, 7141 may also converge at a location
7132 on the first side 7116 of the array 7112. In an ment, a fifth and sixth energy
propagation paths 7142, 7143 rnay also converge at a location 7136 between the first and
second sides 7116, 7114 of the array 7112.
FIGS. 7E-7L are rations of various embodiments ofenergy inhibiting element
7124. For the avoidance t, these embodiments are provided for exemplary purposes
and in no way limiting to the scope of the combinations or implementations provided
within the scope of this disclosure.
illustrates an embodiment ofthe plurality ofenergy locations 7118 wherein
an energy ting elernent 7251 is placed adjacent to the e ofthe energy locations
7118 and comprises a specified refractive, diffractive, reflective, or other energy altering
property. The energy inhibiting t 725 l may be configured to limit the first subset of
energy ation paths 7290 to a smaller range of propagation paths 7253 by inhibiting
propagation ofenergy along energy propagation paths 7252. In an embodiment, the energy
inhibiting element is an energy relay with a numerical aperture less than 1.
illustrates an embodiment ofthe plurality gylocations 7118 wherein
an energy inhibiting structure 7254 is placed orthogonal between regions of energy
locations 7118, and wherein the energy inhibiting structure 7254 exhibits an absorptive
ty, and wherein the inhibiting energy stmcture 7254 has a defined height along an
energy propagation path 7256 such that certain energy propagation paths 7255 are
inhibited. In an embodiment, the energy inhibiting structure 7254 is hexagonal in shape. In
an embodiment, the energy inhibiting structure 7254 is round in shape. In an embodiment,
the energy inhibiting structure 7254 is iform in shape or size along any orientation
of the propagation path. In an embodiment, the energy inhibiting structure 7254 is
embedded within another structure with additional properties.
illustrates the plurality of energy locations 7118, wherein a first energy
ting structure 7257 is configured to substantially orient energy 7259 propagating
therethrough into a first state. A second energy inhibiting structure 7258 is configured to
allow energy 7259, which is substantially oriented in the first state, to propagate
therethrough, and to limit propagation of energy 7260 oriented substantially dissimilarly
to the first state. In an embodiment, the energy inhibiting t 7257, 7258 is an energy
polarizing element pair. In an embodiment, the energy inhibiting element 7257,7 258 is an
energy wave band pass t pair. In an embodiment, the energy inhibiting element
7257, 7258 is a diffractive ide pair
illustrates an embodiment of the plurality of energy locations 7118,
wherein an energy inhibiting element 7261 is stmctured to alter energy propagation paths
7263 to a certain extent depending upon which of the plurality of energy ons 7118
the energy ation paths 7263 extends through. Energy inhibiting element 7261 may
alter energy propagation paths 7263 in a uniform or iform way along energy
ation paths 7263 such that certain energy propagation paths 7262 are inhibited. An
energy inhibiting stmcture 7254 is placed orthogonal between regions ofenergy locations
7118, and wherein the energy inhibiting structure 7254 exhibits an absorptive property,
and wherein the inhibiting energy structure 7254 has a defined height along an energy
ation path 7263 such that certain energy ation paths 7262 are inhibited. In an
embodiment, an inhibiting element 7261 is a field lens. In an embodiment, an inhibiting
element 7261 is a diffractive waveguide. In an embodiment, an inhibiting element 7261 is
a curved waveguide surface.
illustrates an embodiment ofthe plurality of energy locations 7118, wherein
an energy inhibiting element 7264 provides an absorptive property to limit the propagation
of energy 7266 while allowing other ation paths 7267 to pass.
illustrates an embodiment of the plurality of energy locations 7118, and the
plurality of energy waveguides 7112, wherein a first energy inhibiting structure 7268 is
configu red to substantially orient energy 7270 propagating therethrough into a first state.
A second energy inhibiting re 7271 is configured to allow energy 7270, which is
substantially oriented in the first state, to propagate therethrough, and to limit propagation
of energy 7269 oriented ntially dissimilarly to the first state. In order to further
control energy propagation h a system, exemplified by the stray energy propagation
7272, energy inhibiting structures 7268, 7271 rnay require a compound energy inhibiting
element to ensure energy propagation maintains accurate propagation paths.
illustrates an embodiment of the plurality of energy locations 7118, and
wherein an energy inhibiting element 7276 provides an absorptive property to limit the
propagation of energy along energy propagation path 7278 while allowing other energy
along energy propagation path 7277 to pass through a pair of energy ides 7112 for
an effective re 7284 within the array ofwaveguides 7112. In an embodiment, energy
inhibiting t 7276 ses black chrorne. In an embodiment, energy ting
element 7276 comprises an absorptive material. In an embodiment, energy inhibiting
element 7276 comprises a transparent pixel array. In an embodiment, energy inhibiting
element 7276 comprises an anodized material
illustrates an embodiment comprising a ity ofenergy locations 7118,
and a ity of energy waveguides 7112, wherein a first energy inhibiting structure 7251
is placed adjacent to the surface of the energy locations 7118 and comprises a specified
refractive, diffractive, tive, or other energy altering property. The energy inhibiting
structure 7251 may be confi!:, :rured to limit the first subset of energy propagation paths 7290
to a r range of propagation paths 7275 by inhibiting propagation of energy along
energy propagation paths 7274. A second energy inhibiting structure 7261 is structured to
alter energy propagation paths 7275 to a certain extent depending upon which of the
plurality of energy locations 7118 the energy propagation paths 7275 extends through.
Energy inhibiting structure 7261 may alter energy propagation paths 7275 in a uniform or
non-uniform way such that certain energy propagation paths 7274 are inhibited. A third
energy inhibiting structure 7254 is placed orthogonal between regions of energy locations
7118. The energy inhibiting structure 7254 exhibits an absorptive property, and has a
defined height along an energy propagation path 7275 such that certain energy propagation
paths 7274 are inhibited. An energy inhibiting t 7276 provides an absorptive
property to limit the propagation of energy 280 while ng energy 7281 to pass
through. A compound system of similar or ilar waveguide elements 7112 are
positioned to substantially fill an effective waveguide element aperture 7285 'vvith energy
from the plurality of energy locations 7118 and to alter the propagation path 7273 of energy
as defined by a particular system.
In an irnent, the energy inhibiting structure 7124 may be located proximate
the first energy location 7122 and generally extend towards the first energy waveguide
7104. In an embodiment, the energy inhibiting structure 7124 may be located proximate
the first energy waveguide 7104 and lly extend towards the first energy location
7122.
In one embodiment, the energy system is configured to direct energy along the
second plurality of energy propagation paths h the energy 'vvavef,:ruide system to the
singular seamless energy surface, and to direct energy along the first plurality of energy
propagation paths from the singular ss energy surface through the energy relay
system to the plurality of energy devices.
In another embodiment, the energy system is configured to direct energy along the
first plurality of energy propagation paths from the plurality of energy devices through the
energy reiay system to the singular seamless energy surface, and to direct energy along the
second plurality of energy propagation paths from the ar seamless energy surface
through the energy ide system.
In yet r embodiment, the singular seamless energy surface is operable to
guide localized light transmission to within three or less wavelengths of visible light.
Sensory Data Suitable For Holographic Displax�
The plenopic 4D function through the e from an energy ing surface provides
for two spatial coordinates Xi, Yi from a first plane comprising energy locations and directed
through a second coordinate along a second plane comprising waveguiding parameters u1 • v1
ng a vector of an energy propagation path ft(x1 • y1 • ui, vi). In consideration of a plurality of
energy directing es, the plenoptic SD function provides for three spatial coordinates x1, y1, z1
from a first coordinate comprising one or more energy locations and directed through a second
coordinate along a plane comprising waveguiding parameters u1, v1 defining a vector of an energy
propagation path fi (x1, y1, z1, u1, v1). For each of 4D or 5D, additional variables for time and color
f1 (J1, t1) may be considered and assumed to be ive of any of the tic functions as
necessary for an application even vvhen not explicitly noted for simplicity of the function and
discussion. For the avoidance of doubt, the referenceto an energy directing surface is for exemplary
purposes only and may comprise any additional point location, ion, or plane in space for the
localization of a SD coordinate, and collectively referred to as an energy "directing surface".
Figure 8 is a flow chart diagram illustrating an embodiment of a process 800 for
determining four dimensional (4D)plenoptic coordinates for content data. The process 800 may
include a step 802 in vvhich content data is received, which may include any signal perceptible by
a visuaL audio, textural, sensational, or ory sensor. Figure 8 is a tic diagram illustrating
an embodiment of the content data, which may e at least one of the follovving: an object
location, a al property (such as material properties 906, 907, and 908), a virtual light source
904, geometry 902 at non-object location, content out of the reference surface, a virtual camera
position 914, a segmentation 9 l 0 of objects, background texture 912, and layered ts.
Referring to Figures 8 and 9, the process 800 may further include a step 804 in which
locations of data points are detem1ined with respect to a first e 920 to creating a digital
volumetric representation 922 of the content data. The first surface 920 may be used as a reference
surface for ng the ons of data points in space. In an embodiment, the process 800 may
further include a step 806 in vvfoch 4D tic coordinates of the data points are determined at a
second surface by tracing the locations of the data points in the volumetric representation to the
second surface where a 4D function is applied. In an embodiment, the process 800 may further
include a step 808 in which energy source location values are detennined for 4D plenoptic
coordinates that have a first point of convergence.
The content data received in step 802 may include N views, where N is one or more. A
single view may be presented with or without a depth channel. Stereoscopic vievvs may be
presented with or without a depth cham1e:l. Multi-view imagery may be presented with or without
a depth channel. Fmther, a 4D light field may be presented vvith or without a depth channel.
The tracing of step 806 may use prior knowledge of a calibrated geometry of an energy
, which may be stored in memory as a global model or an individually characterized system
or some combination of the two ologies.
In an embodiment, the mapping between the input data and the output energy source
provides a methodology to accurately map between various bitrate sources. The tracing ofstep 806
provides the ability to infer the foll volumetric 4D data set from the above listed partial s.
Depth infommtion either needs to be provided or calculated from the available data. With the depth
infom1ation known or calculated, the N view(s) may be inverse traced by triangulation of the
samples from the known volumetric presentation based upon depth coordinate into the 4D space.
The triangulation may assume that each available energy source location in the N source
content are representative ofa energy source location for each energy ide in the event that
a mapping between energy waveguide: and energy source location fom1at resolution are provided.
ln the event that the N source content tion are lower, super-resolution or scaling algorithms
may be implemented. In the event that the resolution of the N source image(s) are higher than the
number gy waveguides in the energy directing , interpolation between super-sampled
energy source ons may be perfom1e:d to produce higher amount of energy source locations
per energy waveguide in the resultant 4D inverse ray trace.
The above s ce information may be determined from the depth maps which
may or may not be accurate depending on the fom1 of depth infimnation provided or calculated,
and with the distance information known or assumed, the distance: information in combination with
the x-y energy source location coordinate and the (u,v) angular information as dete:rrnined by the
energy directing device properties may then be considered a 4D or 5D light field \vith d
imaging data samples. The imaging s, based upon the distance: information, are triangulated
back to the appropriate: energy source locations that may exist behind each energy waveguide
respectively, and missing data may be generated in step 808 through the disclosures contained
herein.
Referring to Figs. 7C, 8, 9, 10 in an embodiment, the energy locations may be located in
the first surface 920, and the second surface where a 4D function is applied may correspond to a
waveguide system 7100 ofan energy ing device, and energy is le to be directed through
the waveguide system according to the 4D plenoptic coordinates of the data points to fom1 a
detectable volumetric representation ofthe content data.
In an embodiment, the process 800 may farther comprise a step 810, in which energy
source location values are determined for 4 D coordinates that have a first point ofconvergence. To
provide an example implementation of the present disclosure, illustrates an ment
ofan energy directing device l 000 going through a tracing process where content data in the fom1
ofan image 1002 is provided with a distance position 1004, which may be provided or calculated,
within a determined minimum position l006 and maximum position 1008 in reference to the energy
locations 1010. In an embodiment, the energy locations 1010 may comprise an energy directing
device surface. l11e known geometry from the energy locations ]010 defined by the 4D plenoptic
function allows for the triangulation ofa point 10l 4 on the virtual surface of the image 1002 to be
traced back along rays 1016 ific energy locations 1018, each having a unique x-y nate.
Missing samples may be computationally calculated based upon the available inforrnation
contained within the dataset
When additional N san1ples are provided, the same methodology is applied with the
additional multi-perspective g data producing a richer set ofinverse ray traced s and
provide superior holographic results. The depth infom1ation from a multiple N samples may be
provided through a single depth map, or up to N, or greater than N depth maps with a kmnvn
mapping n the source location (the N+X perspective) and the source depth map (the N+X
depth map) to ensure appropriate inverse ray tracing is performed.
Inthe event that a singular depth map forthe, for example, Nperspective is provided,
the onal depth maps may be interpolated by ating for the dispaiity betvveen each ofthe
adjacent views to accurately map the source and target location between the N and the N+X
ints. With this method, it is possible to inverse ray trace the appropriate view dependent
mapping to the 4D light field such that the correct perspective(s) are projected to the appropriate
waveguide coordinates and results in the viewer's ability to in the correct view dependencies
in the associated viewpoints.
The encoder and decoders are robust and may interpret le data types to e, but
not limited to, 2D/flat files, 2D with depth, stereoscopic, stereoscopic vvith single depth channel,
stereoscopic with dual depth channel, N+X multi-view with no depth, N+X multi-view with N+Y
depth, ric or vector based scene files that may include textures, geometry, lighting, material
properties and the like to reconstruct an environment, deep imaging files wherein multiple RGBAZ
values may be provided for each x-y coordinate, 4D or 5D (4D plus depth) light fields, or ed
as a N+X vievv plus N+Y delta l dataset wherein the depth channel provides a lovver
bandvvidth methodology for only rendering a ce1tain amount of energy source location data as
required for a ined energy directing device field ofview. TI1e processors are able to inverse
ray trace at up to, or exceeding, real-time , in order to provision the appropriate 4 D light field
to present to the view·e:r with and without vvorld coordinate: locations, vvith and without compensated
minimum and maximum projected world locations and in consideration of the energy directing
device sic as characterized and/or designed.
In an embodiment, the process 800 may further comprise a step 812, in which a mapping
between energy locations 7122 on a first side of the waveguide system 7100 and the angular
directions ofthe energy propagation paths 7120 from the waveguide element 7100 on a second side
ofthe ide system 7100 is applied. Doing so may allow a plurality of energy locations on
the first side of the ide system 7100 corresponding to the 4 D tic coordinates of the
data points to be be determined.
[OJ 44] Figure 12 is a schematic diagram of a processmg system l 200 compnsmg a data
input/output ace l 20 l in communication with a processing subsystem having a sensory data
processor l 202, a vecotrization engine 1204, and a tracing engine 1206. It is to be appreciated tht
the sensory data processor 1202, the vecotrization engine 1204, and the tracing engine 1206 may
be implement on one or more processors, whether individually or any combination f. Step
802 ofthe process 800 may input content data through the data input/output interface l20 l to the
processing subsystem 1220. Step 804 may be performed by the sensory data processor 1202 to
create a volumetric representation ofthe content data. Step 806
In an ment, applying the mapping may comprise: calibrating for a distortion in the
\vaveguide system 7 l 00, which may further comprise calibrating for at least one tion selected
from a group consisting of: a spatial distortion, angular distortion, intensity distortion, and color
dist01tion.
In an embodiment, the energy directing device may further comprise a relay system 61lOon
the first side of the ide system 7100, the relay system having a first surface 61l 6 adjacent
to the waveguide system 7100, and the energy locations 7112 on the first side of the waveguide
system may be positioned adjacent to a second surface 6114 ofthe relay system 6110.
In an embodiment ng the g may include calibrating for a distrntion in the
\vaveguide system 7100. In an embodiment, ng the mapping may include calibrating both
for a distortion in the relay system 6110 anda distortion in the waveguide system 7100. In an
embodiment. the distortion to be calibrated may include at least one distortion selected from a
group consisting of: a spatial distortion, angular disto1tion, intensity tion, and color distortion.
[Discuss process 800 with respect to system in Fig. 12]
In an embodiment, a p01tion of the method may be carried out in real time, or the method
may be ly carried out in real time, or at least two portions of the method may be carried out
in different time periods.
2D to Light Field Conversion
In an ment, content data may comprise data points in a two dimensional (2D) space,
and detem1ining ons of step 704 may se applying a depth map to the data points in a
two dimensional space.
There are several methods to convert two-dimensional or t1at imagery into light field data.
These include the estimation ofdepth infimnation through depth from motion analysis, a provided
depth l through manual or rendered means, or the manual creation of disparity, depth,
occlusion, geometry and/or any other methodology known as standard for visual effects content
creation to reproduce the foll light field through regeneration of the entire environment through
manual and automated processes.
In a first embodiment, a system that includes a real-time or offline sor to perform
tion of depth from available energy source location information is possible. This may be
perforrned at the energy directing device, as a set top box or as an offline process. Additional
computation for missing volumetric data may be perfom1ed leveraging temporal infom1ation and/or
state of the art texture synthesis or other logies known in the art.
In a second embodiment, depth infomrntion is provided as an image stream and may be
embedded into the image format. Similarly, additional computation may be performed for missing
volumetric data.
In a third embodiment, an artist or a process 1s leveraged to generate the m1ssmg
environmental infommtion which may include a process to isolate or segment each object in a
scene, track said objects over time manually, semi- automatically or tically, place objects
into space leveraging disparity space, energy directing device space, optical space or world
coordinates, synthesizing background and foreground missing infimnation h visual effects
ses known in the art for tmction of backgrounds, transparencies, edge details, etc. to
regenerate the environment. For the avoidance of doubt, the implemented ses may be any,
none or all of the listed embodiments for the reconstruction of these environments. The generated
environmental information should include as much of the missing ation as possible as
determined by the energy directing device angles of view, and these angles ofview may be known
by the artist to ensure that appropriate occlusion and view dependent infom1ation is generated
appropriately.
[OJ 55] Additionally, the surface model for each object in the scene may be generated, either as a
partial model or as a completely built model and textures from the image data are projected onto
the surfaces ofthe geometry to provide riate shape for the following inverse ray tracing.
Additionally, material ties may be calculated or manually introduced to ensure that
view dependent lighting may be introduced with virtual illumination sources to further increase the
accuracy ofthe regeneration of the 4 D light field.
Further, the addition of CG or synthetic content may be introduced to augment the ng
ted als. The addition of tric data may also be incorporated. The inter-mixing
of N+X content may be introduced as vvell to provide a seamless blend n CG, 2D,
stereoscopic, multiview and/or 4D media into a single composite.
The resultant 2D to light field converted content may be retained as a geometric scene file
including geometry, textures, lighting, materials, etc. as indicated in the CG scene itself, rendered
as N+X viev,/S \vith N+D depth channels, rendered as a 4D or 5D (4D + depth) light field, a deep
image which is a fonnat that allmvs for multiple RGBAZ samples per x-y energy source location
coordinate with or without a limitation of ng ofZ samples per x-y nate, or provided as
a N+X view plus N+Y delta l dataset wherein the depth channelprovides a lower bandwidth
methodology for only rendering a ceitain an10unt of energy source location data as ed for a
detem1ined energy directing device field ofview. Tools may be provided to allow for the generation
ofall, some or one ofthese respective output fi.mnats.
Stereoscopic and Multi-view to Light .Field Conversion
The process from above ging single view content may be applied to stereoscopic and
multi-view materials. The estimation of depth information is obtained through depth from motion
analysis, as well as from stereoscopic, multi-view and/or disparity analysis, a ed depth
channel or provided depth channels through manual or rendered means, or the manual creation of
disparity, depth, occlusion, geometry and/or any other ology known as rd for visual
effects content creation to reproduce the full light field through regeneration of the entire
environment through manual and automated processes and leveraging the appropriate data to
fmther retain the view dependent content as available in the provided imaging materials.
[OJ 60] In an embodiment, the content data received in step 102 may comprise data points in a
three dimensional (3D) space, and determining locations may comprise adjusting the data points in
the 3 D space.
In an embodiment, adjusting the data points in the 3D space may include applying a depth
map to the data points in the 3D space, adding new data points, reconstructing occluded data points,
or any combination thereof.
The significant advantage to this approach exists in that the accuracy of stereoscopic
disparity tion is far greater than from motion parallax or other similar 2D estimation
processes alone. Further the image quality ofthe resultant converted 4D light field is more accurate
due to the bility of some of the view dependent conditions, ing but not limited to
illumination, transparencies, materials, occlusion, etc.
The ability to retain the explicit angular encies of the multi-view image data relies
on the ability to calculate the surface normals in relation to the center viewpoint camera, or some
other defined center point. With these normals and disparity or depth information known, it is
possible to interpolate between viewpoints based upon energy directing device angle of view,
which is then either directly applied to the inverse ray tracing, or sized as a n of the
texture sis during the inverse ray tracing.
For brevity, all of the previously disclosed methodologies for the truct ion of 2D to
light field imagery may be applied to the reconstrnction of stereoscopic or multi-view ts.
Generation of NxN RGB Images from 4D or 5D Light Fields
By ging 4D or 5D light fields, it is possible to generate NxN or any value of up to
NxN number of RGB multi-view images. This process is accommodated by considering ea.ch
bottom left coordinate under each ide, assuming a square grid, the 0,0 position, and the top
right position as the N,N position. The grid is only exemplary and any other mapping methodology
may be leveraged. For each 0,0 to N,N position, it is possible to fimn full resolution images from
the light field with the widest possible depth of field based upon the capture system leveraged
wherein each waveguide in the an-ay is considered a single energy source location and each
coordinate under each vvaveguide is a single energy source location of the larger energy source
location array for each complete image from 0,0 to N,N tively. This may be repeated for a
5D light field for the depth infom1ation as well. ln this fashion, it is possible to easily ate
n the 4D or 5D light field to any subset of the dataset that is desired for various distribution
reasons to include 2D, stereoscopic, multi-view, point cloud, CG scene file, or any other desired
combination of data that may be derived from the 4D or 5D light field. For non-regular or square
packed 4 D or 5 D structures, further interpolation is required to align energy source locations to a
regular grid, or a linear mapping between energy source locations and non-square packed structures
may be implemented wherein the resultant images may not appear rectilinear and may also contain
energy source location artifacts.
exemplifies the methodology to convert from a 4D or 5D light field into multiple
viewpoints by arranging the energy ons 1102 from underneath of each energy waveguide
element 1104 according to energy waveguide element position and energy location coordinate
respectively. This provides the ability toseamlessly transfer between light field and smaller datasets
seainlessly.
N+X RGB and N+Y Depth Datasets
The ideal t format that provides the highest quality with the balance of data
transmission size includes the use ofN+X RGB and N+Y Depth+ vectorized channels n
N+X RGB information contains N RGB images that may represent a certain resolution ai1d ,
and X that may represent a different tion and format for RGB data to include lower
resolutions, delta ation and the like, and N+Y Depth+ vect01ized channels that contains N
depth + vectorized channels that may represent a n resolution and format and Y that may
represent a different resolution and fomiat for depth+ vector data to include lower resolutions,
delta information and the like.
The number of N+X views may be generated on a regular grid, from a radius around a
center point with or without a center view, from multiple radii around a center point with or without
a center view, or any methodology to ine the mapping of the number of views and the
associated packing or perspective locations. The configuration for the perspectives may be
contained in the metadata of the file, or the depth+ vectorized channels ed may include a
direct mapping to world nates such that the imaging data aligns to the same coordinate in
XYZ space without other necessary metadata.
40 Disk inversion and Energy directing device Compatibility Processing
For any data captured with a plenoptic or light field 4D or SD , including potentially
those captured with l rigs with optical simulation ofa 4D or 5D light field , the resultant
fly's eye perspectives contain discs that represent the uv vectors for the light field. However, these
coordinates assume energy focusing elements that may not exist in an energy directing device. In
the proposed energy directing device solution, the focusing elements may be the viewer's eye, and
the mapping between the capture system and the mapping between the original capture
methodology and the viewed energy directing device are no longer correct.
To invert this and correct for the additionally g energy directing element in the
system when compared to the capture system, it is possible to individually flip each disc
independently, wherein the x-y location ofeach (u,v) coordinate is retargeted based upon the center
point of each waveguide respectively. In this fashion, the ion of the image that fom1s as a
result of the main waveguide is inverted and allows for the light field energy directing device to
project the rays in the correct x-y-u-v orientation.
A further embodiment ofthis may implement a hardware modification wherein leveraging
an energy -vvaveguide array provides a direct ion ofevery presented energy waveguide energy
source location. For light field energy directing devices, this is advantageous to have a direct
mapping betvveen a potential capture system and energy directing device. This may further be
ageous an embodiment comprising HMD systems or volumetric opacity energy directing
devices such that a group ofenergy waveguides in the overall array may be ated by removing
the necessity to relay additional times for accurate x-y-u-v coordinates.
r, not all light fields are identical. They may be captured with differing NAs, FOVs,
N values, optical prescriptions, etc. The intrinsics and extrinsics ofthe input light field data may
be tood and convert to the energy directing device characteristics. This may be performed
by embodiments contained within this disclosure for universal pararneti zation graphic and
light field data.
Universal Parameterization of Holographic Sensory Data Transport h
Inverse EnergyTracing and ization of Sensory Properties for an Energy
Directing System
The plenopic 4D function through the surface from an energy directing surface provides
for two spatial coordinates x1, y1 from a first plane comprising energy locations and directed
through a second coordinate along a second plane comprising waveguiding parameters u1, v1
defining a vector of an energy propagation path ft(x1, y1, u1, vi). In consideration of a plurality of
energy directing surfaces, the tic SD on provides for three spatial coordinates x1, y1, z1
from a first coordinate sing one or more energy locations and directed through a second
coordinate along a plane comprising vvaveguiding parameters u1, v1 defining a vector of an energy
propagation path [1 (x1 • y1, z1, u1, v1) For each of4D or 5D, additional variables for time and color
[1 (Jl • ti) may be considered and assumed to be inclusive of any of the plenoptic functions as
necessary for an ation even when not explicitly noted t\'.)r simplicity of the function and
discussion. For the avoidance ofdoubt, the reference to an energy ing surface is for exemplary
,,..,
purposes only and may comprise any onal point location, ion, or plane in space for the
localization of a 5D coordinate, and collectively referred to as an energy "directing surface".
Along a first vector of an energy propagation path, a plurality of intersection points
comprising convergence of energies may occur together with additional energy propagation paths.
At this intersection point, a 3D point or depth paran1eter forms at location X1 , Y1,Z1 among the
plurality of energy propagation paths \vith the 4D or 5D functions, \vherein the 3D point of
convergence X1 , Y1 ,Z1 among the plurality of energy propagation paths, where for each x1 , y1 or
x1 , y1 , z, nate contained within the energy directing surface or surfaces, there is only a single
u1, v1 propagation path that fonns between a first coordinate and the converging 3D point. The 4D
function fz(x 1 • Yt , u 1 , vi ) or 5D function fz(x1 , y1 , z1 , u.1 , v1 ) collectively define all 4D Xt , Yt , or 5D
Xt , Yt · z1 coordinates and commensurate u1 , v1 propagation paths that exist for each converging
point at X1 , Y1 , Z1 .
At a first 5D coordinate resulting from the convergence of es along a plurality of
energy propagation paths through the energy ing surface )( , Y1 , Z1 , the coordinate may
represent a point vvithin a larger object volume, particle or localized energy parameter, wherein
ging energies at additional coordinates proximate to the first 5D coordinate may exhibit
additional ized properties for sensory energies within an environment or holographic dataset.
These vectorized properties may comprise information for each SD coordinate, for each energy
location coordinate vvithin the 4D dataset, for regions within either of the 4D or 5D datasets, or
other ts of coordinates comprising the energy surface.
In an embodiment, the universal parameterization of 4D and 5D holographic sensory
energy properties for propagation of , auditory, somatosensory, gustatory, olfactory,
ular or other desired energies for sensory system response for raster and vector 2D, 3D, 4D
and 5D datasets are disclosed, wherein the 2D data may comprise a single angular , 3D data
may comprise two or more angular samples in a single dimension, 4 D data may comprise a plurality
of angular san1ples in two dimensions, or 5D data may comprise a plurality of angular s in
three or more dimensions, in reference to the second coordinate of the second plane of the 4D
energy directing surface.
Embodiments of received sample data may comprise any of:
l ). 2D or monoscopic, flat, point cloud, uv-mapped geometry, intrinsic geometry, deep
images, layered images, CAD files (intrinsic), single pointsampling, single camera capture,
single projector projection, volumetric (monoscopic single sample points with vectors in a
volume), s of 3 Degrees of Freedom (DoF: raster with monoscopic x, y, z rotation
about a single point), sources ofnon-light field 6 DoF (raster+ vectors from monoscopic
samples), volumetric energy directing device (monoscopic samples in a volume), sources
of 's Ghost (single point projection), sources of2D AR HMD (monoscopic single
or multiple focus planes; layered monoscopic), sources of 2D VR HMD (monoscopic
single or multiple focus planes; layered monoscopic), or any other represent,,tion oftwodimensional
raster or vector infom1ation:
2). 3D or stereoscopic, triscopic (single baseline), multiview (]D), JD multi-sample, lD
perspective, horizontal or al only parallax, lD projection array, two point
sampling, lD point sampling, horizontal or vertical array, bullet time, sources of 3 DoF
(raster; stereoscopic x, y, z rotation about a singlepoint), sources of3 DoF (3D raster within
stereoscopic x, y, z rotation about a single point), sources of non-light field 6 DoF (3D
raster + vectors from stereoscopic samples), sources of ID volumetric energy directing
device (JD ax contained samples), sources of autostereoscopic data, sources of
horizontal multiview energy directing device, sources of3D AR Ffl\iiD (stereoscopic single
or multiple focus plane; d stereoscopic), sources of3D VRHMD (stereoscopic single
or multiple focus planes: layered scopic), or any other representation of threedimensional
raster or vector inforn1ation;
3) 4D or plenoptic (SD), multiscopic, integral image, light field (4D), holographic (4D),
2D multiview, 2D multi-sample, 2D multi-perspective, 2D parallax, horizontal and vertical
parallax, 2D pr(�ection array, 2D point sampling, motion capture stage (along a surface),
planar array, witness camera array, rendered or raytraced geometric representations (4D
representations), extrinsic ry (4D representation), s oflight field 6 DoF (4D
raster within planar light field samples), sources of free-viewpoint 6 DoF (4D +
vectors from 4D light field samples), sources of4D volumetric energy directing device (2D
parallax ned samples), sources oflight field energy directing device (4D sampling),
s of light field HMD (near field 4D sampling), sources of holographic energy
directing device (4D san1pling), or any other representation offour-dimensional raster or
vector information;
4). 5D or plenoptic + depth, light field+ depth, aphic (5D ng, 4D + depth),
ary multivievv (along all x, y and z axis), multi-sample (along all xyz), multi-
perspective (along all xyz), volumetric ax (along all xyz), tion array (along all
xyz), point sampling (along all xyz), motion capture stage (along all xyz), witness camera
array (arbitrary xyz configurations), rendered or raytraced geometric representations (5D
representations), cubic or volumetric rendering (along all xyz), extrinsic geometry (5D
representation), sources of light field 6 Dof (SD raster \vithin volumetric light field
samples), s of ieyvpoint 6 Dof (SD raster + vectors from SD light field
samples), sources of SD volumetric energy directing device (multiplanar 4D sampling),
sources of 5D light field energy directing device (5D sainpling, 4D + multiple planes),
sources of 5D light field HJVrD (near field SD sampling, 4D + multiple planes), sources of
aphic energy directing device (SD sampling, 4D + multiple planes), or any other
representation of five-dimensional raster or vector information.
At each ofthe second coordinates, the provided data may comprise a sub-set or a super-set
of either raster or vector samples and in samples may represent and include additional
vectorized information to enable transfom1ation into increased sampling density through
interpretation or processing of the sub-set or super-set ofraster or vector samples.
For each of 2D, 3D, 4D or SD provided datasets, the infom1ation is ted through
vectorized infom1ation, manual identification, computer vision analysis, automated processing, or
other means to transform the ed samples from the original dataset into a 5D nate
system. For each of 2D, 3D, 4D or 5D provided ts, the information may comprise multiple
samples or layers of samples as well as additional vectorized properties in respect to the originating
angular sampling component for each provided dataset in reference to the second coordinate of the
second plane of the 4 D energy directing surface, or may comprise a combination of contributing
samples for any of 2D, 3D, 4D or 5D additional provided datasets.
Each of the provided samples compnse sic energy for each desired coordinate,
wherein the intrinsic energy may include additional extrinsic energy utes, where the intrinsic
energy represents value at a given SD coordinate in the absence of other external samples,
properties or environmental ions. In the electromagnetic spectrnm, this may be referred to as
the albedo as the ionless measurement for reflectance c01Tesponding to a white body that
ts all incident radiation, but explicitly extended to each desired sensory energy wherein the
range of dimensionless values is commensurate to the specified sensory energy. \Vithin the visual
y systems, this range is approximately 400nm to 700um, and in the auditory sensory s,
this range is approximately 20Hz to 20kHz.
Over the past l decades, vast technological improvements enabling the reproduction
of human senses artificially leveraging sophisticated pattern recognition of detected sensation,
aromas and flavors through electronic means. For other systems that may exist outside of the
electromagnetic spectmm, these dimensionless values may be characterized in the same way based
upon sensed acuity se. While holographic sensory energy technologies are newly emerging,
sed within this embodiment comprises a system, method and fom1at for the stimulation of all
human senses in a vi1tual environment to articulate the universal construct for various sensory
parameters whereby provisioning for the appropriate data handling, transmission, storage,
vectorization, translation to, from and between any sensory energy parameter or device desired for
complete immersion of the constrncted virtual environment and embodiments of energy
propagation for holographic sensory technologies will be disclosed in future applications. It is
onally the intent of this disclosure to enable other analogue devices, including novelties like
the classic "smell-o-vision," or contemporary versions like FeelReal's smelling VR headset to
leverage the parameterized values provided for within the vectorization of the dataset herein.
In an ment, the somatosensory system may be defined based upon the components
that define sensitivity including mechanoreceptors for textures with a pressure sensitivity range in
the skin that may be nonnalized between 50Hz to 300Hz, them1oreceptors with a temperature
sensitivity range in the skin that may be normalized n 0°c to 50°c (although this range may
be much vvider range with upper and lower bounds defined by the extremes of temperature) or
surface defonnability defining the range of lastic behaviors of a material measure both
viscous and elastic characteristics when undergoing defomiations between stress and strain over
time which provides for a multiplicity of physics including les for time, strain, modulus,
among other dynamics, and for the purposes of this disclosure is simplified to a dimensionless
normalized scale vvith a value of O for unmovable solids such as granite, and 1 for low viscosity
s such as water. Those skilled in the art will understand that the actual vectors provided will
comprise the ary physics to appropriately define the viscoelasticity of the material, and
ized for exemplary purposes only.
Finally, state of the art advances in artificial electronic g including gustatory and
olfactory devices demonstrate a viable path to fu1ther vectorizing the sensory parameters disclosed
for the Holodeck design ters, as well as enable the electronic reproduction of artificial taste
and smell through a aphic waveguiding means as described herein. Artificial electronic taste
and smell receptors have made considerable progress through emerging nanodevices, wherein
frequency-based artificial taste receptors using an enzymatic biosensor to sample the intensity of
al stimulus h the encoding and conversion to frequency based pulses to both
repeatedly and accurately detect taste as frequencies of the sampled chemical compositions through
a pattern ition system resulting in the detection of the tastes that compose the human palate.
lt is believed that the technology may be extended to all types of able tastes and similar
advances in artificial olfactory system have demonstrated l interfaces for stimulating ones
smell receptors using weak electrical pulses targeting the nasal conchae with ongoing studies to
further parameterize the ns contained within frequencies of particular olfactory responses
through variation in electrical signals.
With the path established for the arbitrary generation of frequencies and complex electronic
patterns to represent olfactory, gustatory and other sensory system, in one embodiment, the acuity
se for taste may be vectorized to comprise a normalized scale for each of electronically
lled parameters along a scale from Oto J to represent the minimum and maximum gustatory
response to saturate the average human's 2,000 - 8,000 taste buds, potentially comprising but not
limited to vectors for sourness, saltiness, bitter (spiciness), sweetness, and savoriness (unmami)
wherein the vector and the spatial coordinate of the vectorized signals may infom1 the production
for the complex olfactory implementations.
In r embodiment, the acuity response for smell may be r vectorized to
comprise a normalized scale for each of electronically controlled parameters along a scale from 0
to 1 to represent the minimum and m olfactory response to saturate the average human's
l O cm2 of olfactory epithelium, for each of the highly complex olfactory spaces potentially
comprising but not limited to vectors for fragrant, fmity, citms, woody (resinous), chemical, sweet,
mint (peppermint), toasted (nutty), t and decayed n the vector and the spatial
nate of the ized signals may infom1 the production for the complex olfactory
implementations.
Each of these vectors may provide the normalized values representing these patterns for
taste, smell or other sensory s, converted to a wave, amplitude, magnitude or other attribute
as required for the appropriate application of the provided vectorized values. \Vhile the sense of
smell and taste are two of the most highly debased senses within the sensory system, with
parameterized values to vectorize complex amalgamations, it is additionally possible in an
embodiment to provide for user based interactive control over the sensitivity of any such sensory
energy to provide for ization ofindividualization ofeach ofvisual, auditory, sensory,
gustatory, olfactory, vestibular or other desired sensory system responses.
In an embodiment, each ofthe represented sensory albedo energy values ofthe sample may
additionally comprise extrinsic energy attributes baked into the single sample value representing
the additive result of each ed sample respective of other external samples, properties or
environmental conditions. In this uration, the compound sample value may or may not
exhibit latent attributes of other energies from other s in a physically based or simulated
environment. The most efficient and pure methodology to transmit the paraineterized and
reconstrncted holographic t is based upon the singular intrinsic sample information providing
for simplified and lovver bandwidth frequency infom1ation, although this is not always possible to
receive outside of entirely synthetic environments, particularly for physically based imaging or
acoustic systems. In any real-world environment, there is always some amount of extrinsic
contribution to the resultant sample information. n systems like the Light Stage, or other
s known in the art to facilitate the estimation of reflectance, shape, texture, and motion
capture leverage some fom1 of stmctured illumination and one or more imaging devices vvhich
provide for the direct or indirect analysis of the albedo, depth infom1ation, surface nomial and
bidirectional ring distribution surface properties.
The bidirectional scattering distribution on (BSDF) is a generalized superset of the
bidirectional transmittance distribution function , the ctional texture function (BTF),
and the bidirectional reflectance distribution on (BRDF), which are often ented by the
generalized function fr(w 1 . Wr), collectively act as a model to parameterize and identify surface
properties in computer graphics and vision algorithms known in the art. The fonction describes how
visible light is reflected, transmitted or otherwise interacts with a surface given an incoming
incident direction w, and outgoing reflected or transmitted direction Wrfor an energy propagation
path, where the surface nom1al is perpendicular to the tangent ofthe object surface and the function
describes the ratio ofreflected radiance exiting along the ng path Wr to the irradiance incident
on the surface along the incoming path w1, wherein each ofw1, Wr may comprise a 4D function to
define a parameterized azimuth and zenith angle for each ofthe incoming light path and the exiting
light path.
The functions may further be articulated for a first location x1 of energy At striking a
surface, and exit afl:er al properties ally scatter the energy to a second location Xr of
energy }.r to account for visible \vavelength effects like iridescence, luminescence, subsurface
scattering, non-local scattering effects, specularity, shadowing, masking, inten-et1ections, or the
like, resultant output energy based upon material properties of a surface, the input energies and
locations, the output energies and locations across the e of an object, , or point.
Therefore, the generalized properties to describe how energy is transported betvveen any
two energy rays that strike a surface, to include wavelength or frequency dependency and spatially
varying material properties or surfaces, may be ented as a 1OD fonction, and specified
as f� (At, xi, w1, ,in Xr, w r) for each or any of the ble or provided samples within a dataset to
account for input energy, the impact of a vectorized surface e, and the output reflected,
refracted, specular, transmitted, scattered, diffused, or other material property result from any
energy domain given the generalization of the function j�.
In eration now of the energy directing surface, the plenopic 4D function es for
two spatial coordinates x1, y1 from a first plane comprising energy locations and directed h a
second nate along a second plane comprising waveguiding parameters ui, Vi defining a
vector of an energy propagation path [1 (xi, Yi, ul, vl). In consideration of a plurality of energy
directing surfaces, the plenoptic 5D function provides for three spatial coordinates x1, y1, z1 from a
first coordinate comprising one or more energy locations and directed h a second coordinate
along a plane comprising waveguiding parameters ul, vl defining a vector of an energy propagation
path f1 (x1,y1, z1, u1, vi). For each of 4D or 5D, additional variables for time and color f1 (Ji. 1, ti)
may be considered and assumed to be inclusive of any of the plenoptic functions as necessary for
an application even when not explicitly noted for simplicity of the function and discussion.
Along a first vector of an energy propagation path, a plurality of intersection points
comprising convergence of energies may occur together with additional energy propagation paths.
At this ection point, a 3D point or depth ter forms at location X1, Y1 .Z1 an1ong the
plurality of energy propagation paths with the 4D or 5D plenoptic functions, wherein the 3D point
of convergence X,, Y1 • Z1 among the plurality of energy propagation paths, where for each Xi, Yi or
x1, y1, z1 coordinate contained within the energy directing 4D surface or 5D surfaces, there is only
a single Ut, Vt propagation path angle that fom1s between a first nate and the converging 3D
point. The 4D fonction fz(x i, Yi, u1, v1 ) or 5D fonction fz(x,, Yi, z1 • ui, vi) collectively define all
4D Xi, Yi, or 5D x1, y1, Zt coordinates and commensurate u1, v1 propagation paths that exist for each
converging point at Xi, Y1, Z1.
At a ging coordinate X1, Y1, Z1, a surface is fom1ed and the smface may comprise a
point, volume, object or other embodiment comprising a 3D position of converging energy
propagation paths. The provided samples for each surface location may comprise one or more
surface properties, vectors, materials, terizations, or other identif'.ving property Vt to
characterize or otherwise process the resulting energy, as well as one or more input energy sources
striking a given point proximate to the e location vvherein the reflectance function now
comprises a generalized vector for the various ties of the surface and represented as an l lD
universal object parameterization function frCAt , x1, w1, A,-, x,., Wr, VJ.
[OJ 93] The llD universal holographic parameterization on J;.(A 1, Xt , w1, Ar, Xr, Wr, V,)
s the resultant values for a given environment and vectorized object ties and the 4D
function ft(x1, y1, u1, vt) defines the energy propagation paths from an energy directing device
surface, may therefore be further generalized as an 15D universal holographic paramete1ization
function J;.(..1 1, Xt, w1, Ar, Xr, w'r(x1, y1, u1, v1), VJ where the transmitted direction Wr defines and
equals the propagation path of Ut, Vt , whereby defining the spatial coordinate x1, y1 and for each
transmitted direction w,. there may be only one f1 (x1, Yi, u1, vi) set ofvalues to satisfy w,. = u1 • v1 .
Those skilled in the art will appreciate the various transforms and mathematical constrncts in
addition to the rendering requirements associated with the disclosed universal paran1eterization of
4D and 5D holographic sensory energy properties.
[OJ 94] With the complete l5D function describing the vectorization of all sensory energy
properties to coincide with surfaces fom1ed from converging points in space, multiple orders of
magnitude of required data have been fundamentally eliminated provisioning for a viable path to
enabling the transmission of truly holographic datasets.
[OJ 95] The vectorized properties strive to provide accurate s for each of the y
domains for properties that may be synthetically programmed, captured, or computationally
ed, n Vt may prescribe attributes for each surface, volume or 3D coordinate Xi, Yt, Z1
vectorized ties about an o�ject for a given sample n a provided dataset for general
system rnetadata or for each or any sensory energy domain, comprising:
l .) system metadata may provide for any ofthe sensory energy specific utes or system
wide references for surface properties for each sample including normals, depth
tion, nmental properties, multiple angular samples for a given 3D coordinate,
procedural textures, geometry, point clouds, deep image data, static frames, temporal
frames, video data, e IDs, surface passes, coordinate maps, virtual camera
coordinates, virtual illumination and visible energy infonnation, environment maps, scene
information outside of the field ofthe visual sensory san1ple infom1ation, curves, vertices,
temporal ation, networked data, databases, object recognition, energy devices,
external data foeds, sensors for system modifications and ctivity, system status, voice
recognition, olfactory detection, auditory detection, facial recognition, somatosensory
recognition, gustatory recognition, UI, UX, user profiles, flmv and motion vectors, layers,
s, arency, segments, animation, sequence infom1ation, procedural infom1ation,
cement maps, or any other scene data that is necessary to provide sufficient data for
the appropriate processing of each sample;
2.) visual sensory energy may provide surface properties to define the appropriate
rendering of visible or non-visible electromagnetic energy, iridescence, luminescence,
subsurface scattering, non-local scattering effects, specularity, shadowing, absorbance,
transmission, masking, interreflections, , transparency, physics, dynamics,
reflection, tion, diffraction, optical effects, atmospheric effects, frequency,
tion, surface profiles, textures, displacement maps, physics and dynamics to
specifically interrelate to other sensory energies and respond based upon provisioned
energies (e.g. vibrations of sound altering tance properties or tactile material
deformation causing surface defom1ations), layers, regions, transparency, ts,
, ion, sequence infi.mnation, procedural information, size of material,
environmental conditions, room dynamics, or other related material properties for a
surface, environment, room, , point, volume or the like:
3.) auditory y : vectors related to the placement of localized sound fields,
magnitude, amplitude, mass, material propagation parameters, absorbance, transmission,
material properties infom1ing acoustic reflectance, diffosion, transmission, augmentation,
masking, scattering, localization, frequency dependence or modulation, pitch, tone,
viscosity, smoothness, texture, modulus, any other paran1eters that determine the
propagation of acoustic waves within the object, surface, medium or othenvise, s
and dynamics to specifically interrelated to other sensory energies and respond based upon
provisioned energies (e.g. temperature changing the sound of a material), layers, regions,
transparency, segments, , ion, ce infonnation, procedural infom1ation,
size of material, nmental conditions, room dynamics, or other related material
ties for a e, environment, room, object, point, volume or the like;
4) somatosensory energy vectors related to the mechanoreceptors for textures, pressure,
thermoreceptors, temperature, surface defonnability parameters and vectors defining the
range of viscoelastic behaviors of a material measure both viscous and elastic
characteristics when undergoing deformations between stress and strain over time vvhich
provides for a licity of physics including variables for time, strain, modulus, ainong
other dy11amics, layers, regions, transparency, segments, , animation, sequence
infonnation, procedural infonnation, size of material, environmental conditions, room
dynamics, or other related material properties for a e, environment, room, object,
point, volume or other somatosensory ters;
.) ory sensory energy s for fragrant, frnity, citrns, woody (resinous), chemical,
sweet, mint (peppem1int), toasted (nutty), pungent and decayed \vherein the vector and the
l coordinate of the vectorized signals may inform the production for the x
ory implementations and further provide duration, ude, frequency, length,
time, radius, modulation, layers, regions, transparency, segments, curves, animation,
sequence infomrntion, procedural information, size of material, environmental conditions,
room dynamics, or other related material properties for a surface, environment, room,
object, point, volume or other gustatory sensory parameters;
6 ) olfactory sensory energy vectors for sourness, saltiness, bitter (spiciness), sweetness,
and sav01iness (unmami) wherein the vector and the spatial coordinate of the vectorized
signals may inform the production for the complex olfactory implementations and further
provide duration, magnitude, frequency, length, time, radius, modulation, layers, regions,
trai1sparency, segments, curves, animation, sequence information, procedural information,
size of material, environmental conditions, room dynamics, or other related material
prope1ties for a surface, environment, room, object, point, volume or other olfactory
parameters;
7.) or other interrelated sensory dynamics based upon physical, synthetic, mitted, or
computational interdependencies from any other sensory sample dataset, sensory system
s as needed, designed, or required and any additional sensory properties where
parameterization of a particular characteristic is beneficial for the reconstruction, e,
processing or transmission of generalized holographic constmcted data.
With the received dataset sing 2D data having a single angular sample, 3D data
having t\vo or more angular samples in a single dimension, 4D data having a plurality of r
samples in two dimensions, or SD data having a plurality of angular samples in three or more
dimensions.
For all ed source materials, each source material may undergo additional processes
to appropriately prepare for efficient vectorization of the aphic dataset. For any provided
source materials that exhibit lower spatial or r resolution that the energy directing surface, a
transfonnation process may be required in order to accurately convert the originating source to a
4D or SD dataset.
For appropriate preparation, m an embodiment, provided 2D or 3D source materials
comprise raphic e from a standard imaging system. Within this sequence of images
are rastered reflections, refractions, transparent elements and other similar examples of al
property interaction with physically based illumination.
In the event that the content is prepared by simply identifying surface IDs for the surfaces
\vith the already rastered material ties, the effective data may be sufficient for converging
into a 4D coordinate system, however, any additional rendering applied to these surfaces will
exhibit a double image for the s of both the photographic, as well as the parameterized
synthetic rendered reflectance properties. The ideal source dataset for efficient holographic
transmission comprises an albedo representation ofthe sample source infonnation, plus vectorized
material properties for each of the specified energy s with metadata fi.mning an -based
volumetric sarnpling of the albedo multi-view samples, and wherein all material properties provide
for accurate surtace identification and rendering as well as the localization or projection of other
sensory energies accurately based upon the specified vectorized surface properties.
In an embodiment, manuaL utomated, computer vision, or automated processes are
provisioned to algorithmically or manually assess the content -vvithin the source san1ple dataset, and
in a manual or algorithmic analysis is performed whereby segmentation and other object
isolation methodologies known in the art are performed to identify the regions that include
undesired physically rasterized effects. ln an ment, a person is photographed in front of a
background wherein the material properties ofthe person include reflections from the environment,
and the ound objects are occluded by the photographed person. After these regions have
been identified as undesirable, a process may be leveraged to l) isolate the objects in question; 2)
separate all object elements into the core components to account for occlusion, transparency, edges,
or other element; 3) through image analysis, al analysis, energy analysis, with the facilitation
ofmachine learning, computer vision, extra hardware and energy devices that additionally captured
1ation about the scene, s and/or environment, or h completely manual means, the
object elements are provisioned such that any e that should exhibit a material property has
any such baked-in al properties removed through computer , algorithms, processors,
or manual visual effects wherein the manual processes are generally known in the art for methods
to perform ,vire removals, paint fix, clean plates, image restoration, alpha matte creation, occlusion
g, object recreation, image projection, motion tracking, camera tracking, rotoscope, optical
flow, and the like for the e of rating the intrinsic material property in the absence of
the extrinsic material properties thereby preparing the content for the most efficient transmission
and propagation for said dataset; 4) An additional process of the above involves the manual or
computer assisted identification of depth or 3D coordinate values for each ofthe desired samples;
and 5) Further within this embodiment is the identification of the associated material properties,
each h represent a point region ofdata, surface, object or other representation ofa material
such that the dat1, may easily be further rendered ,vithin the energy directing device's y
drivers or within any additional system capable ofeither encoding and decoding the parameterized
dataset.
In an embodiment, the dataset from the above comprises 3D multiview samples that are
prepared \vith albedo visual energy samples, each of which having multiple layers of rgba
information, a collection of vectorized material properties to associate each segmented al
with a surface 1D and series ofsurface parameters to closely reconstmct the original source dataset
prior to the removal ofthe extrinsic image data, and wherein an acoustic dat1,set is provisioned with
vectorized al properties associated with the material properties of the visual energy system
as well as multiple sound channels each having identified frequency, tion, spatial placement
and other sound localization properties, and wherein a somatosensory sensory energy dataset is
provided for a subset of the surfaces contained within the visual energy dataset, to onally
comprise viscoelastic and temperature vectorized material properties, both ofwhich are correlated
to the other vectorized datasets.
From any provided dataset, each provided sample from the visual energy dataset is assessed
for a relative depth position in relation to the energy ing device surface, and wherein each of
the samples for any of the visual energy samples are placed into a 3D coordinate system, and
\vherein the energy propagation path length for each ofthe ed samples is assessed in relation
to the on that correlates each 3D coordinate in relation to the plurality of coexisting
converging energy ation paths that intersection a first 3D point at location Xt, Vi, Z1 among
the plurality of energy propagation paths within the 4D or SD plenoptic ons, where for each
Xi, Yi or Xt, Yt, z1 coordinate contained within the energy directing 4D surface or SD surfaces, there
is only a single u1, v1 propagation path angle that forms between a first coordinate and the
converging 3D point. The 4D function fz(x 1, y1, u1, vt) or SD function .fz(x1, Yt, Zi, Ut, vi)
collectively define all 4D Xi, Yi, or SD Xi, Yl• z1 coordinates ned within the energy directing
device and commensurate ui, Vi propagation paths that exist for each converging point at Xi, Yi, Z1
and wherein the total number ofsamples per presented or available 4D x1, y1, or 5D x1, y1, z1 spatial
coordinates is known after performing this analysis process, and wherein the total energy
propagation path length n each 3D point at location Xi, Yt, Z1 to the 4D or 5D coordinate
location is known, and wherein a weighted distribution based upon total available samples per 4D
or SD coordinate and m path length to the sampled 3D coordinate values from the available
plurality of 3D coordinate data provides for a complete sampling of the 4D or 5D light field from
an arbitrary dataset.
As a further embodiment of the above, after each of the samples for any of the l) visuaL
acoustic, somatosensory, and any other provided energy samples are 2) placed into a 3D coordinate
system based upon the provided dataset, additional processing, or additional vectorized properties,
and before perfom1ing a coordinate analysis; 3) the 15D universal holographic parameterization
function fr CAt, X1, Wt, A,., x,., w,.(x[,Yi, U1, vt), va is assessed wherein 4) additional knmvn
environmental scene, geometry, metadata or the like is provided, each with independent vectorized
material properties; 5) virtual illumination infonnation is provided and the additional sensory
energy metadata properties are ed for any potential interference between the properties that
may altering the rendering functions and; 6) the 15D parameterization on assesses for each
provided 3D coordinate and commensurate vectorized material ty to: 7) perform a rendering
process through on-line, ne, ime, processor, ASIC, FPGA, cloud, or other forrn of
ing process to result in a new plurality of angularly g material properties given the
arbitrary provided t, and wherein 8) the rendering process is specific to each of the
transmitted direction Wr defining and equal to each of the propagation paths u1, v1 , whereby
defining the spatial coordinate Xi,Yi , and for each transmitted direction Wr there may be only
one fi(x1, y1, u1, v1) set of values to satisfy Wr '" u1, V1, and wherein 9) based upon the rendered
results and resultant available new angularly varying material properties, for each of the 4D or SD
coordinates comprising the energy propagation path length for each of the provided samples are
assessed in relation to the function that coITelates each 3D coordinate in relation to the plurality of
ting converging energy propagation paths that intersection a first 3D point at location
Xt , }'1, Z1 among the plurality of energy propagation paths within the 4D or 5D plenoptic ons,
where for each x1, y1 or x1, y1, z1 coordinate contained within the energy directing 4D surface or 5D
es, there is only a single u/, v! propagation path angle that forms between a first coordinate
and the converging 3D point. The 4D function .fz(x/, Y!, u1, v1) or 5D function , Y!, Zi, u1, vi)
collectively define all 4D Xi, Yi, or 5D Xi, Yl• z1 coordinates contained within the energy directing
device and commensurate ui, Vi propagation paths that exist for each ging point at Xi, Yi, Z1
and wherein the total number ofsamples per presented or available 4D x1, y1, or 5D x1, y1, z1 spatial
coordinates is known after performing this analysis process, and wherein the total energy
propagation path length between each 3D point at location X1, Yt, Z1 to the 4D or 5D coordinate
location is known, and wherein a weighted distribution based upon total available samples per 4D
or 5D coordinate and minimum path length to the san1pled 3D coordinate values from the ble
plurality of 3D coordinate data provides for a complete sampling of the 4D or SD light field for all
provided sensory energies from an arbitrary dataset.
An additional embodiment of the above system wherein the rendering onally
accounts for a ctional energy directing surface such that sensed omagnetic energy
representing the illumination of the real-world environment, or the absorbance of n acoustic
ncies within the environment may result in the dynamic or off-line update to the rendering
process or other sensed interactive real-world element is assessed, and wherein the illumination
and acoustic or other sources are adjusted to accommodate for the modification in environmental
conditions.
Turning back to Fig. 8, in view of the principles disclosed above, in an embodiment of
process 800, the received content data may r comprise vectorized material property dat'l, and
wherein the s 800 further comprises a step 830, in which digital volumetric representation of
the content data is associated with the vectorized al property data; and wherein, in step 804,
determining energy source location values is based on at least the vectorized material ty data
associated with the volumetric representation of the content data.
Referring to Figs. 9 and ] 3, in an embodiment, a vectorization process ] 300 may include
a step 1302 in which first content data is ed and a step l304 in which identifying a surface
915 in the content data. In an embodiment, identifying the surface 915 may comprise using
segmentation data in the content data The vectorization process l300 may further include a step
1306 in which a surface identification of the surface 9] 5 is detem1ined and a step 1308 in \"-foch
material property data ofthe surface 915 is determined. In an embodiment, determining the material
property data may comprise manual detem1ination, or using a predetermined process. After steps
1306 and 1308, the vectorization process 1300 may further include a step 13] 0 in which the surface
identification is associated with the material property data of the surface 915. The vectorization
process ] 300 may further include a step of1312 in which the vectors ofthe material property data
is created. The vectorization process 1300 may farther include a step 13l 4 in which vectorized
material property data is generated based on the created vectors.
In an embodiment, the process 1300 may ally include a step 1316 in which al
property data is d from the first content data and ed by the ized material
property data is generated in step 1314. In an embodiment the vectorized material property data is
generated in step 1314 may used in process 800 as discussed above to detem1ine 4D plenoptic
coordinates for the energy directing s ofthe present disclosure as discussed above.
The process 1300 may be carried out using any processing system ofthe present sure,
including processin system 1200. In an embodidment, content data may be received in step l 302
h the data output interface 1201, and steps 1304 thru 1314 of the vectorization process
] 300 may be carried out using the vectorization engine ] 204. Additionally, the vectorized material
property data generated in step 1314 may be used by the sensory data processor ] 202 and tracing
engine 1206 for processsing according to the steps of process 800 as discussed above. Steps 808
and 812 may be performed by the tracing engine to determine 4D coordinates for holographic
presentation. Step 810 may be perfom1ed by the sensory data processor 1202. The output of the
processing subsystem may be provided to a comprsession engine 1210, from which compressed
data may be stored in a memory or d to the data input out interface 1201 for transmission to
anenergy directing system either conneted locally or remotely to the system 1210. Datamay also
be stored in the memory 1208 until a later time to be retrieved.
While various embodiments in accordance with the principles sed herein have been
described above, it should be understood that they have been presented by way ofexample only,
and are not limiting. Thus, the breadth and scope ofthe invention(s) should not be limited by any
of the above-described exemplary embodiments, but should be defined only in ance with
the claims and their equivalents issuing from this disclosure. Furthermore, the above advantages
and features are provided in described embodiments, but shall not limit the application of such
issued claims to processes and structures accomplishing any or all of the above advantages.
It will be understood that the pal es of this sure can be employed in
s embodiments without depaiting from the scope of the disclosure. Those skilled in the art
\Vill recognize, or be able to ascertain using no more than routine experimentation, numerous
equivalents to the specific procedures described herein. Such equivalents are considered to be
within the scope ofthis disclosure and are covered by the claims.
Additionally, the section headings herein are provided for consistency with the suggestions
under 37 CFR] .77 or ise to provide organizational cues. These headings shall not limit or
characterize the invention(s) set out in any claims that may issue from this sure. Specifically,
and by way of e, although the headings refer to a "Field of Invention," such claims should
not be limited by the ge under this g to describe the so-called technical field. r,
a description of technology in the "Background of the ion" section is not to be constrned as
an admission that technology is prior art to any invention(s) in this disclosure. Neither is the
"Summary" to be considered a characterization of the invention(s) set forth in issued claims.
Furthermore, any reference in this disclosure to "invention" in the ar should not be used to
argue that there is only a single point of novelty in this disclosure. Multiple inventions may be set
fo1th according to the limitations of the multiple claims issuing from this disclosure, and such
claims ingly define the invention(s), and their equivalents, that are protected thereby. In all
instances, the scope ofsuch claims shall be considered on their own merits in light of this disclosure,
but should not be constrained by the headings set forth herein.
The use of the word "a" or "an" when used in conjunction with the tenn "comprising" in
the claims and/or the specification may mean '"one," but it is also consistent with the meaning of
''one or more," "at least one," and "one or more than one:' The use of the term ·'or" in the claims
is used to mean "and/or" unless explicitly indicated to refer to alternatives only or the alternatives
are mutually exclusive, although the disclosure supports a definition that refers to only alternatives
and "and/or." Throughout this application, the term t" is used to indicate that a value includes
the inherent variation of error for the device, the method being employed to detern1ine the value,
or the variation that exists among the study subjects. In general, but subject to the preceding
discussion, a numerical value herein that is modified by a word of approximation such as "about"
may vary from the stated value by at least :H, 2, 3, 4, 5, 6, 7, 10, 12 or 15%,.
As used in this specification and claim(s), the words ·'comprising"' (and any form of
comprising, such as "comprise"' and "comprises''), g" (and any fom1 of having, such as
"have" and "has"), "including" (and any fom1 of including, such as "includes" and de") or
"containing" (and any fom1 of containing, such as ins" and "contain") are inclusive or openended
and do not exclude additional, unrecited elements or method steps.
Words of comparison, measurement and timing such as "at the time," "equivalent,"
"during," "complete," and the like should be understood to mean "substantially at the time,"
"substantially equivalent" "substantially during," "substantially complete," etc , \vhere
"substantially" means that such comparisons, measurements, and timings are practicable to
accomplish the implicitly or expressly stated desired result. Words relating to relative: position of
ts such as "near," "proximate to," and "adjacent to" shall mean sufficiently close to have a
al effect upon the respective system t interactions. Other vvords of approximation
similarly refer to a ion that when so modified is understood to not necessarily be absolute or
perfect but would be considered close enough to those of ordinary skill in the art to warrant
designating the ion as being present. The extent to which the description may vary will
depend on how great a change can be instituted and still have one of ry skilled in the art
ize the modified feature as still having the required characteristics and capabilities of the
unmodified feature.
The tem1 "or combinations thereof' as used herein refers to all pennutations and
combinations ofthe listed items preceding the tenn. For example, "A, B, C, or combinations thereof
is ed to include at least one of: A, B, C, AB, AC, BC, or ABC, and iforder is important in a
particular context, also BA, CA, CB, CBA, BCA, ACB, BAC, or CAB. Continuing with this
example, expressly included are combinations that contain repeats of one or more item or term,
such as BB, AAA, AR BBC, AAABCCCC, CBBAAA, CABA.BR and so foith. The skilled
artisan \Vill understand that typically there is no limit on the number of items or tem1s in any
combination, unless ise apparent from the context.
All of the compositions and/or methods disclosed and claimed herein can be made and
executed without undue experimentation in lightofthepresent disclosure. While the compositions
and methods of this disclosure have been described in tem1s ofprefe1Ted embodiments, it will be
apparent to those of skill in the art that variations may be applied to the compositions and/or
methods and in the steps or in the sequence of steps of the method described herein without
ing from the concept, spirit and scope of the disclosure:. All such similar substitutes and
modifications nt to those d in the art are deemed to be within the spiiit, scope and
concept of the disclosure as defined by the appended claims.
Claims (48)
1. A method of detem1ining four dimensional ( 4D) tic coordinates for content data, method comprising: receiving content data; determining ons of data points with respect to a first e to creating a digital volumetric representation of the content data, the first surface being a reference surface; determining 4D plenoptic coordinates ofthe data points at a second surface by tracing the locations the data points in the volumetric representation to the second surface where a 4D function is applied; and determining energy source location values for 4D plenoptic coordinates that have a first point ofconvergence.
2. The method ofclaim 1, -vvherein the content data comprises a signal perceptible by a , audio, textural, sensational, or smell senor.
3. The method ofclaim 1, wherein the content data comprises at least one of the ing: an object location, a material property, a virtual light source, content for geometry at non-object location, content out of the reference surface, a virtual camera position, a segmentation ofobjects, and layered contents.
4. The method ofclaim 1, wherein the content data comprises data points in a two dimensional (2D) space, and wherein determining locations comprises applying a depth map to the data points in a two dimensional space.
5. The method ofclaim 1, wherein the content data comprises data points in a three dimensional (3D) space, and wherein detem1ine locations comprises adjusting the data points in the 3 D space.
6. The method ofclaim 5, wherein adjusting comprises ng a depth map to the data points in the 3 D space.
7. The method ofclaim 5, in ing comprises adding new data points.
8. The method ofclaim 5, wherein adjusting comp1ises reconstructing occluded data points.
9. The method ofclaim 1, wherein the second surface corresponds to a waveguide system of an energy directing device, and energy is operable to be directed through the guide system according to the 4D tic coordinates ofthe data points to fom1 a detectable volumetric representation of the content data.
10. The method ofclaim 9, wherein the method further comprises ng a mapping between energy locations on a first side ofthe ide system and the angular directions ofthe energy propagation paths from the waveguide element on a second side of the \vaveguide system, wherein a plurality ofenergy locations on the first side ofthe waveguide system corresponding to the 4D plenoptic coordinates ofthe data points are determined by applying the mapping.
11. The method ofclaim l 0, applying the mapping comprises ating for a distortion in the waveguide system.
12. The method ofclaim 11, calibrating for the distortion in the waveguide system comprises calibrating for at least one distortion selected from a group consisting of: a spatial distortion, angular distortion, intensity distortion, and color distortion.
13. The method ofclaim 9, ein the energy directing device further comprises a relay system on the first side ofthe waveguide system, the relay system having a first surface nt to the waveguide system, and further wherein the energy locations on the first side ofthe waveguide system are positioned adjacent to a second surface ofthe relay system.
14. The method m 13, wherein applying the mapping comprises ating for a distortion in the waveguide system.
15. The method ofclaim 13, n applying the g comprises calibrating for a ion in the relay system.
16. The method ofclaim 15, wherein applying the mapping comprises calibrating for a distortion in the waveguide system.
17. The method ofclaim ] 5, wherein calibrating for the tion in the relay system comprises calibrating for at least one tion selected from a group consisting of: a spatial dist01tion, angular distortion, intensity distortion, and color dist01tion.
18. The method ofclaim 9, wherein the energy locations are located in the first surface.
19. The method ofclaim Lwherein the received content data farther comprises vectorized material property data, and n the method farther comprises ating the digital volumetric representation of the content data with the vectorized material property data; and \vherein detem1ining energy source location values is based on at least the vectorized material ty data associated with the volumetric representation ofthe content data.
20. The method ofclaim 1, wherein at least a portion ofthe method is carried out in real time.
21. The method ofclaim L vvherein method is entirely carried out in real time.
22. The method ofclaim 1, wherein at least two portions of the method is carried out in different time periods.
23. A method ofdetem1ining four dimensional ( 4D) plenoptic coordinates for content data, method comprising: receiving content data; determining locations of data points with respect to a reference point location; izing the data point by creating vectors of the data points based on the reference point location; detem1ining, based on the vectorized data points, locations of data points vvith respect to a first surface to creating a digital volumetric entation of the content data, the first surface being a reference surface: and determining 4D tic coordinates of the data points at a second surface by tracing locations the data points in the volumetric representation to the second smface where a 4D function is applied.
24. The method ofclaim 23, wherein the content data ses a signal perceptible by a visual, audio, al, sensational, or smell senor.
25. The method ofclaim 23, wherein the content data comprises at least one of the following: an object location, a material property, a virtual light source, t for geometry at non-object location, content out ofthe reference surtace, a virtual camera position, a segmentation ofobjects, and layered contents.
26. The method ofclaim 23, wherein the content data comprises data points in a t\vo dimensional (2D) space, and wherein determining locations ses applying a depth map to the data points in a tvvo dimensional space.
27. The method ofclaim 23, wherein the content data comprises data points in a three dimensional (3D) space, and wherein determine locations comprises adjusting the data points in the 3D space.
28. The method ofclaim 27, wherein adjusting comprises ng a depth map to the data points in the 3D space.
29. L .....-,Q The method ofclaim 27, wherein adjusting ses adding new data points.
30. The method m 27, wherein adjusting comprises reconstructing occluded data points.
31. The method ofclaim 23, wherein the second surface corresponds to a waveguide system ofan energy directing device, and energy is operable to be directed through the waveguide system according to the 4D plenoptic coordinates ofthe data points to form a detectable volumetric entation ofthe content data.
32. The method ofclaim 3l,wherein the method further comprises applying a mapping between energy locations on a first side ofthe waveguide system and the angular directions ofthe energy propagation paths from the waveguide t on a second side ofthe waveguide system, n a plurality ofenergy ons on the first side ofthe uide system con-esponding to the 4D plenoptic coordinates ofthe data points are determined by applying the mapping.
33. The method ofclaim 32, applying the mapping comprises calibrating for a distortion in the ide system.
34. The method of claim 33, calibrating fort he distoition in the vvaveguide system ses calibrating t\'.)r at least one dist01tion selected from a group ting of: a spatial distortion, angular distortion, intensity distortion, and color distortion.
35. The method ofclaim 31, wherein the energy directing device further comprises a relay system on the first side of the waveguide , the relay system having a first surface nt to the waveguide system, and further wherein theenergy locations on the first side of the waveguide system are positioned adjacent to a second surface of the relay .
36. The method of claim 35, wherein applying the mapping comprises calibrating for a tion in the vvaveguide system.
37. The method ofclaim 35, wherein applying the mapping comprises calibrating for a tion in the relay .
38. The method of claim 37, wherein applying the mapping comprises calibrating for a dist01tion in the waveguide system.
39. The method of claim 37, \vherein calibrating forthe distortion in the relay system comprises calibrating for at least one distortion selected from a group consisting of: a spatial distortion, angular distortion, intensity distortion, and color distortion.
40. The method ofclaim 3l, wherein the energy locations are located in the first surface.
41. 4l. The method ofclaim 23,wherein the received content data further comprises vectorized material prope1ty data, and wherein the method further comprises associating the digital volumetric entation ofthe content data with the vectorized material ty data: and wherein detennining energy source location values is based on at least the vectorized material property data associated vvith the volumetric representation ofthe content data.
42. A method ofvectorization, comprising: receiving first content data; identifying a surface in the content data; ining a surface identification of the surface; determining material property data of the surface; associating the surface identification with the material prope1ty data ofthe e; creating the vectors ofthe material property data ; and generating vectorized material property data based on the created vectors.
43. The method ofclaim 42, wherein identifying the e comprises using segmentation data in the content data .
44. The method of claim 42, wherein determining the material property data ses manual detem1inations.
45. The method of claim 42, wherein detem1ining the al property data comprises using a ermined process.
46. The method of claim 42, further comprising: ining locations of data points of the surface with respect to a reference surface to creating a digital volumetric representation of the first content data; associating the digital volumetric representation ofthe first content data vvith the ized material property data; determining 4D plenoptic coordinates of the data points of the surface at a 4D application surface by tracing the locations of the data points in the volumetric representation to the second surface where a 4D function is applied; and detem1ining energy source location values for 4D nates that have a first point of convergence, wherein detennining energy source location values is based on at least the vectorized al property data associated vvith the volumetric representation of the first content data.
47. The method of claim 46, further comprising removing material property data.
48. A system compring an input-output interface a processing subsystem in cornmunicatino with the interface a ssion. . engme; an optional memery C""- .,0............ r� ...... "'°) .. c::::i,-
Applications Claiming Priority (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US62/362,602 | 2016-07-15 | ||
US62/366,076 | 2016-07-24 | ||
US62/507,500 | 2017-05-17 | ||
USPCT/US2017/042275 | 2017-07-14 | ||
USPCT/US2017/042276 | 2017-07-14 |
Publications (1)
Publication Number | Publication Date |
---|---|
NZ789927A true NZ789927A (en) | 2022-07-01 |
Family
ID=
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11874493B2 (en) | System and methods of universal parameterization of holographic sensory data generation, manipulation and transport | |
US20230341592A1 (en) | Systems and methods for rendering data from a 3d environment | |
WO2018014047A1 (en) | System and methods of holographic sensory data generation, manipulation and transport | |
WO2018014040A1 (en) | Holographic superimposition of real world plenoptic opacity modulation through transparent waveguide arrays | |
US20240151896A1 (en) | System and methods of universal parameterization of holographic sensory data generation, manipulation and transport | |
NZ789927A (en) | System and methods of holographic sensory data generation, manipulation and transport | |
박길배 | STUDY ON THREE-DIMENSIONAL ANAMORPHIC VOLUMETRIC DISPLAY SYSTEM |