US8878900B2 - Non photorealistic rendering of augmented reality - Google Patents
Non photorealistic rendering of augmented reality Download PDFInfo
- Publication number
- US8878900B2 US8878900B2 US12/666,957 US66695708A US8878900B2 US 8878900 B2 US8878900 B2 US 8878900B2 US 66695708 A US66695708 A US 66695708A US 8878900 B2 US8878900 B2 US 8878900B2
- Authority
- US
- United States
- Prior art keywords
- image
- npr
- transparency
- captured
- saliency map
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/50—Lighting effects
- G06T15/503—Blending, e.g. for anti-aliasing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/02—Non-photorealistic rendering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2210/00—Indexing scheme for image generation or computer graphics
- G06T2210/62—Semi-transparency
Definitions
- the present invention relates to a method of rendering images, in particular to provide occlusion cues, for example in medical augmented reality displays.
- Augmented reality is becoming a valuable tool in surgical procedures.
- Providing real-time registered preoperative data during a surgical task removes the need to refer to off-line images and aids the registration of these to the real tissue.
- the visualization of the objects of interest becomes accessible through the “see-through” vision that AR provides.
- MIS Minimally Invasive Surgery
- the unique operational setting of the surgical robot provides an ideal platform for enhancing the visual field with pre-operative/intra-operative images or computer generated graphics.
- the effectiveness and clinical benefit of AR has been well recognized in neuro and orthopedic surgery. Its application to cardiothoracic or gastrointestinal surgery, however, remains limited as the complexity of tissue deformation imposes significant challenges to the AR display.
- NPR non-photorealistically rendered
- the NPR image may be rendered as a 2 dimensional view or a second image may be captured to define a stereoscopic view.
- a virtual (for example AR) object may be rendered behind the captured image.
- the assignment of the transparency values may be done inside a window such that the NPR captured image remains opaque outside the window, occluding the object when it is not seen through the window.
- the transparency values may be gradually blended from within the window to outside of it.
- the window position may be defined in dependence upon the viewer's gaze, continuously tracking the viewer's gaze or updating the position only when an update request is received from the viewer.
- the windows in the two (left and right) NPR images may be offset by an amount determined by the camera positions and parameters in accordance with the stereoscopic view.
- the transparency values may be determined as a function of a normalised image intensity gradient at corresponding locations in the captured image. Calculating the image intensity gradient may include determining a partial derivative with respect to an image coordinate divided by image intensity at the corresponding locations.
- Determining NPR image transparency as set out above can be seen as an example of a method of setting the transparency values by defining a saliency map for an area of the NPR image and assigning transparency values as a function of values of respective corresponding locations in the same saliency map.
- the saliency map may be arranged to capture salient features of the image, for example features which are salient because they protrude from the background of the underlying scene or because of colour and/or intensity contrast.
- the saliency map may be defined as a function of local slopes in a scene underlying the image, for example as estimated based on shading in the image. The local slopes may be estimated as a function of respective normalised intensity gradients in the image.
- the saliency map may also be used to assign a colour value to a pixel of the NPR image, for example using a colour scale.
- the transparency and colour values may be assigned such that an object rendered behind the NPR image is being perceived as being viewed through the transparent area (window) while being occluded by pixels within the area which have high values in the saliency map.
- the virtual object may be derived from medical imaging data, for example CT or MRI images of a tumour.
- the images may have been captured using a stereoscopic endoscope, for example during thoracic keyhole surgery.
- the rendering method described above is not limited to medical AR applications but is more generally applicable to AR applications were a virtual object is rendered out of a viewer's normal view behind a captured scene.
- NPR image short for Non-Photorealistically Rendered image
- NPR image is used here to designate the captured and processed digital image, applied for example as a texture to a plane corresponding to an image plane in a 3D computer graphics model which may also contain the AR object.
- this model may be rendered as a 2D or stereoscopic 3D image.
- FIG. 1 is a flow diagram of a method for non-photorealistic rendering of at least part of an image, for example to reveal an object rendered behind the scene captured in the image;
- FIG. 2 is a flow diagram of an algorithm for rendering of a corresponding NPR image and AR object in 2D or 3D;
- FIG. 3 depicts a corresponding system
- FIG. 4 depicts a mask function used in the processing
- FIG. 5 depicts an example of an AR view rendered using the described method.
- the underlying idea for the present method of rendering an image as applied to medical AR displays is to render an exposed anatomical surface as a translucent layer while keeping sufficient details to aid navigation and depth cueing.
- One embodiment is based on pq-space based Non-Photorealistic Rendering (NPR) for providing a see through vision of the embedded virtual object whilst maintaining salient anatomical details of the exposed anatomical surface.
- NPR Non-Photorealistic Rendering
- surface geometry based on a pq-space representation is first derived, where p and q represent the slope of the imaged surface along the x, y axes, respectively. For example, this can be achieved with photometric stereo by introducing multiple lighting conditions.
- E ⁇ ( x , y ) s 0 ⁇ ⁇ ⁇ ( x , y ) ⁇ cos ⁇ ⁇ ⁇ r 2 ( 1 )
- s o is the light source intensity constant
- ⁇ (x,y) is the albedo or reflection coefficient
- r is the distance between the light source and the surface point (x,y,z)
- ⁇ is the angle between the incident light ray and the normal to the surface ⁇ circumflex over (n) ⁇ .
- the normal vector to the surface is equal to
- n ⁇ ( p , q , - 1 ) 1 + p 2 + q 2 ( 2 )
- p and q represent surface slopes in directions x and y respectively.
- E ⁇ ( x , y ) s 0 ⁇ ⁇ average ⁇ ⁇ ( 1 - p 0 ⁇ x - q 0 ⁇ y ) 3 Z 0 2 ⁇ ( 1 - p 0 ⁇ x 0 - q 0 ⁇ y 0 ) 2 ⁇ ( 1 + p 0 2 + q 0 2 ) 1 / 2 ⁇ ( 1 + x 2 + y 2 ) 3 / 2 ( 3 ) which defines the relationship between the image intensity E(x,y) at the point (x,y) and scene radiance at the corresponding surface point (x o Z o ,y o Z o ,Z o ) with surface normal (p o ,q o , ⁇ 1), where ⁇ average denotes the average albedo in a small neighborhood of the surface and s o is the light source intensity constant.
- Lambertian surface under point source illumination is an idealised surface material that satisfie
- This can be rewritten as two linear equations in p 0 and q 0 (the normalised partial derivatives or normalised gradient R x/y at image location (x,y) being determinable from the image intensity at (x,y) and its neighbourhood):
- a 1 p 0 +B 1 q 0 +C 1 0
- p 0 B 2 ⁇ C 1 - B 1 ⁇ C 2 B 1 ⁇ A 2 - A 1 ⁇ B 2 ( 7 )
- q 0 - A 2 ⁇ C 1 + A 1 ⁇ C 2 B 1 ⁇ A 2 - A 1 ⁇ B 2 ( 8 )
- the p and q values of the imaged surface capture 3D details of the exposed anatomical structure and are used to accentuate salient features (that is features protruding from the surface and hence having a high gradient) while making the smoothly varying background surface semi-transparent. It will be understood that the p and q values may be calculated using any suitable technique.
- surfaces of the scene that are parallel to the viewing plane are rendered as more or less transparent whilst sloped structures (high p, q) are accentuated and rendered more or less opaque.
- the logarithm squashes high values of p and q to limit the dynamic range for display purposes.
- a smooth background map B is created by applying a wide Gaussian filter to the saliency map S, thereby smoothing out high frequency variations in the image likely to represent noise or minor surface variations rather than “true” salient features.
- the salient and background images are combined using a mask such that low pixel values of S are replaced with the value of B at the corresponding (x,y) pixel, as described in detail below.
- a region of interest is defined for NPR processing. This may include the entire image or a sub area thereof.
- the region of interest is pre-processed including converting the pixel colour values (if in colour) to grey-scale and applying a mild smoothing function such as a 3 ⁇ 3 pixels Gaussian.
- the saliency map is calculated from the captured image in the ROI as described above using any known method for calculating the partial derivatives, for example simply differencing the pixel values of the pixel in question with a neighbouring pixel in the relevant (e.g. x) direction.
- the map is calculated from the partial derivative and image intensity (for example grey-scale value) at each pixel or location in the ROI.
- the saliency map is denoised by combining it with the smooth background map B (for example B can be derived from S by applying a broad Gaussian filter of 6 ⁇ 6 pixels with a spread of 7 pixels).
- the mask function can be defined using splines with a few control points (for example a Catmull Rom spline).
- a suitable mask function is depicted in FIG. 5 as a function of S(x,y).
- Other mask functions can also be employed, for example a step function with a suitable threshold value for S(x,y).
- Another example could be a suitable polynomic function.
- the pixels within the denoised region of interest are assigned colour values in accordance with the saliency map S(x,y) using a colour scale.
- a suitable colour scale can range from black for the minimum value of S through blue to white for the maximum value.
- These artificial colours are applied to the NPR image in accordance with a window function, for example a radial window function f(r) such that the artificial colour is applied within the window and the original colour of the image remains outside the window.
- f(r) is a step function, in which case the window would have a sharp edge discretely switching from artificial to image colour.
- f(r) defines a transition region in another example
- f ⁇ ( r ) ⁇ r 2 r 0 2 for ⁇ ⁇ r ⁇ r 0 1 for ⁇ ⁇ r ⁇ r 0 ( 11 )
- r 2 (x ⁇ x center ) 2 +(y ⁇ y center ) 2 and r 0 determines the window size.
- the NPR image pixel transparency within the same window is set in accordance with S(x,y) and the window function f(r) such that the lowest value of S corresponds to fully (or nearly fully, say 95%) transparent and the maximum value of S corresponds to fully (or nearly fully, say 95%) opaque and the transparency values are blended with the remainder of the NPR image using f(r), for example.
- NPR image f ( r ) ⁇ CI+(1 ⁇ f ( r )) ⁇ NPI. (12)
- the window position is predefined based on the position of an AR object to be rendered behind the image plane, in particular if there is only a single object to be rendered within the field of view.
- the window position may be determined based on a viewer's gaze (for example detected using an eye tracker device), either updated continuously or in accordance with a viewer's fixations when the viewer issues a request for the window position to be updated (or in some embodiments, if no window is presently displayed, the window is displayed in accordance with the viewer's fixation when the window is switched on). This dynamic window display may be particularly useful if a full AR scene is being rendered rather than a single object.
- an image of the scene is captured by an imaging device 22 and transmitted to a processing unit 24 .
- the captured image is treated as a texture projected onto a scene plane object to be displayed in a computer graphics model in some embodiments.
- the computer graphics model may be implemented in a number of ways, for example using the OpenGL graphics library in a C++ program in some embodiments.
- the correct perspective (and disparity in the case of 3D rendering) is already contained in the captured image (left and right captured image in the case of a stereo camera imaging device) but the depth information about the augmented reality object in the computer graphics model is important to consistently handle occlusion.
- the position of the augmented reality object in the left and right view has to match disparity in the captured images.
- the captured images are displayed in the same position for both left and right views as they already contain the disparity, whereas the augmented reality object is rendered at different positions for left and right views so as to match the disparity of the captured images.
- the transformations obtained as part of a stereo camera calibration are utilized using OpenGL transformations in some embodiments.
- an OpenGL window is opened for each (left and right) display and the NPR image is displayed (for example, applied to a scene plane object acting like a projection screen) in each window at an appropriate depth (see below).
- the AR object is then rendered for each window using the camera calibration for the respective (left and right) camera.
- the relative position of the ROI and/or window in each of the OpenGL windows is set using the camera calibration data to be consistent with the stereoscopic view in some implementations.
- smoothly blended windows will assist in fusing the two windows in a stereoscopic view even if the window displacement is not exactly in accordance with the stereoscopic view.
- one approximation suitable for a stereoscopic endoscope camera is to use the camera displacement as the window displacement in some implementations.
- the z coordinate of the NPR image (the scene object plane) must be closer to the camera position in the computer graphics model than the position of the AR object in the camera frame such that pixels take the colour values of the scene object where the scene object occludes the AR object behind it (or where low transparency parts of the scene object dominate the colouring of the corresponding pixels).
- the occlusion cues provided by the non-photorealistic processing discussed above are only necessary when the AR object lies behind the surface captured by the imaging device 22 (otherwise the real object would be visible in the captured image), it is assumed for rendering in some embodiments that the scene object is always in front of the AR object and, for example, the frontal plane of the viewing frustrum of the computer graphics model is used as the scene object on which the captured image(s) are projected in some implementations. This ensures correct handling of occlusion without influencing the perspective or disparity of the captured image(s) which are defined by the real camera positioning.
- the NPR image is positioned at or close to the camera focal plane or the depth of the scene plane could be recovered using range finding to set the NPR image depth (see for example Stoyanov D., et al, Computer Aided Surgery July 2005, 10(4): 199-208, herewith incorporated herein by reference).
- Techniques which are used for depth recovery include the use of the observer's vergence from eye tracking (Mylonas GP et al, Proceedings of the second International Workshop on Medical Imaging and Augmented Reality, MIAR (2004), Beijing, 311-319), depth recovery from stereo such as sparse techniques, shape from shading or a combination of the two (Stoyanov D et al, MICCAI (2) 2004: 41-48) or the use of structured light (Koninckx T P and Van Gool L, IEEE PAMI, vol 28, no. 3, 2006), fiducial markers or laser range finding (Mitsuhiro H et al, Medical Image Analysis 10 (2006) 509-519), all herewith incorporated herein by reference.
- An AR object input 26 provides coordinate data of the AR object obtained from medical imaging, for example from MRI or CT data. This data must be expressed in coordinates in a frame of reference fixed on a patient's body, for example using fiducial markers fixed on the patient's body before the images are obtained to convert the image data from the medical imagining device frame to a body frame of reference.
- the AR objects coordinates from the body frame of reference defined by the fiducial markers need to be transformed into the imagining device camera frame of reference from reference frame input 28 which tracks both camera position and orientation as well as the fiducial markers, for example by tracking the fiducial markers using a 3 dimensional tracking device to determine the position and orientation of the body frame of reference relative to the camera frame of reference, as is well known in the art.
- the camera position is known (for example in the case of a robotic surgery console in which the location of the camera can be derived from the position of the robotic arm supporting the camera) this known position in a coordinate frame fixed to the operating room is used together with the tracker data to perform the necessary coordinate transformation in some implementations particularly relevant to applications in a robotic surgery console.
- this known position in a coordinate frame fixed to the operating room is used together with the tracker data to perform the necessary coordinate transformation in some implementations particularly relevant to applications in a robotic surgery console.
- camera position also needs to be tracked to be able to perform the required coordinate transform. Details of the required measurements and transformation are described in Vogt S. et al, Reality Augmentation for Medical Procedures: System Architecture, Single Camera Marker Tracking and System Evaluation, International Journal of Computer Vision 2006, 70(2): 179-190, herewith incorporated herein by reference.
- NPR processing of the captured image is carried out as described above with reference to FIG. 1 , possibly taking an input about the window position from window position input 30 which, in some embodiments, includes an eye tracker and user interface for switching the window on or off and selecting a window position, for the window described above.
- the scene object and AR object are rendered for display on the display device 32 at step 20 .
- This may be a 2D view of the corresponding computer graphics model where the display device is a simple monitor or a 3D view consisting of a left and right image where the display device is stereo capable.
- the system and method described above are incorporated in a minimally invasive robotic surgery console, for example the da Vinci robotic surgical console by Intuitive Surgical, Inc, Mountain View, USA.
- the console provides robotic manipulation for remotely controlling minimally invasive surgical tools and stereo visual feedback via a fixed position stereo display device providing respective left and right images from a stereoscopic endoscope to each eye of the operator.
- the captured images are processed as described above and can then be rendered together with an AR object (e.g. representing a tumour).
- FIGS. 5 a and b show respective left and right eye views of lung tissue from a robotic assisted lung lobectomy.
- FIGS. 5 c and d show the same view with a transparent AR overlay of an AR object in which both the scene and the object are rendered transparently as in the see-through video approach of Vogt et al, referenced above.
- FIGS. 5 e and f depict respective left and right views of the scene of FIGS. 5 a and b processed using the method described above together with the AR object and figures g and h show the views of a and b and e and f, respectively, combined using a smooth radial window.
- the window provides for a more natural interpretation of the 3 dimensional scene as the AR object is seen through the window while the preserved features in the window maintain reference information for the surgeon and also provide additional depth cues by occlusion.
- one embodiment may be in hardware, such as implemented to operate on a device or combination of devices, for example, whereas another embodiment may be in software.
- an embodiment may be implemented in firmware, or as any combination of hardware, software, and/or firmware, for example.
- one embodiment may comprise one or more articles, such as a carrier or storage medium or storage media.
- the storage media such as, one or more CD-ROMs solid state memory, mageneto-optical disk and/or magnetic disks or tapes, for example, may have stored thereon instructions, that when executed by a system, such as a computer system, computing platform, or other system, for example, may result in an embodiment of a method in accordance with claimed subject matter being executed, such as one of the embodiments previously described, for example.
- a system such as a computer system, computing platform, or other system, for example
- One embodiment may comprise a carrier signal on a telecommunications medium, for example a telecommunications network.
- suitable carrier signals include a radio frequency signal, an optical signal, and/or an electronic signal.
- a computing platform or computer may include one or more processing units or processors, one or more input/output devices, such as a display, a keyboard and/or a mouse, and/or one or more memories, such as static random access memory, dynamic random access memory, flash memory, to and/or a hard drive.
Landscapes
- Engineering & Computer Science (AREA)
- Computer Graphics (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Processing Or Creating Images (AREA)
- Image Generation (AREA)
- Image Processing (AREA)
Abstract
Description
where so is the light source intensity constant, ρ(x,y) is the albedo or reflection coefficient, r is the distance between the light source and the surface point (x,y,z), and θ is the angle between the incident light ray and the normal to the surface {circumflex over (n)}. In gradient space, the normal vector to the surface is equal to
where p and q represent surface slopes in directions x and y respectively. For a smooth Lambertian surface in the scene, image intensity given by equation Eq. 1 can be reduced to
which defines the relationship between the image intensity E(x,y) at the point (x,y) and scene radiance at the corresponding surface point (xoZo,yoZo,Zo) with surface normal (po,qo,−1), where ρaverage denotes the average albedo in a small neighborhood of the surface and so is the light source intensity constant. Lambertian surface under point source illumination is an idealised surface material that satisfies two conditions: (1) it appears equally bright from all viewing directions, and (2) it reflects all incident light.
This can be rewritten as two linear equations in p0 and q0 (the normalised partial derivatives or normalised gradient Rx/y at image location (x,y) being determinable from the image intensity at (x,y) and its neighbourhood):
A 1 p 0 +B 1 q 0 +C 1=0
A 2 p 0 +B 2 q 0 C 2=0 (5)
with
A 1=(−x 0 ·R x+3)·(1+x 0 2 +y 0 2)−3·x 0 2
B 1 =−R x·(1+x 0 2 +y 0 2)·y 0−3·x 0 ·y 0
C 1 =R x·(1+x 0 2 +y 0 2)+3·x 0
A 2 =−R y·(1+x 0 2 +y 0 2)·x 0−3·x 0 ·y 0
B 2=(−y 0 ·R y+3)·(1+x 0 2 +y 0 2)−3·y 0 2
C 2 =R y·(1+x 0 2 +y 0 2)+3·y 0 (6)
which gives the following expressions for p0 and q0 at each point (x,y) of the image:
The p and q values of the imaged
S(x,y)=log(abs(p 0)+abs(q 0)+1) (9)
where high values of S(x,y) correspond to large gradients. In effect, this provides a saliency map or salient image. The logarithm squashes high values of p and q to limit the dynamic range for display purposes. A smooth background map B is created by applying a wide Gaussian filter to the saliency map S, thereby smoothing out high frequency variations in the image likely to represent noise or minor surface variations rather than “true” salient features. The salient and background images are combined using a mask such that low pixel values of S are replaced with the value of B at the corresponding (x,y) pixel, as described in detail below.
S(x,y)=mask(x,y)·S(x,y)+(1−mask(x,y))·B(x,y) (10)
such that the saliency map is blended with the background map whereby S dominates where S is high and B dominates where S is low. The mask function can be defined using splines with a few control points (for example a Catmull Rom spline). A suitable mask function is depicted in
where r2=(x−xcenter)2+(y−ycenter)2 and r0 determines the window size. By defining a smooth transition, fusing the two (left and right) windows in a stereo image (see below) may be helped if applicable.
NPR image=f(r)·CI+(1−f(r))·NPI. (12)
Claims (19)
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
GB0712690.7 | 2007-06-29 | ||
GBGB0712690.7A GB0712690D0 (en) | 2007-06-29 | 2007-06-29 | Imagee processing |
PCT/GB2008/002139 WO2009004296A1 (en) | 2007-06-29 | 2008-06-23 | Non-photorealistic rendering of augmented reality |
Publications (2)
Publication Number | Publication Date |
---|---|
US20100177163A1 US20100177163A1 (en) | 2010-07-15 |
US8878900B2 true US8878900B2 (en) | 2014-11-04 |
Family
ID=38420978
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US12/666,957 Active 2030-12-07 US8878900B2 (en) | 2007-06-29 | 2008-06-23 | Non photorealistic rendering of augmented reality |
Country Status (10)
Country | Link |
---|---|
US (1) | US8878900B2 (en) |
EP (1) | EP2174297B1 (en) |
JP (1) | JP5186561B2 (en) |
KR (1) | KR20100051798A (en) |
CN (1) | CN101802873B (en) |
AT (1) | ATE500578T1 (en) |
DE (1) | DE602008005312D1 (en) |
ES (1) | ES2361228T3 (en) |
GB (1) | GB0712690D0 (en) |
WO (1) | WO2009004296A1 (en) |
Cited By (24)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20190088019A1 (en) * | 2016-03-16 | 2019-03-21 | Koninklijke Philips N.V. | Calculation device for superimposing a laparoscopic image and an ultrasound image |
US10628995B2 (en) | 2017-04-17 | 2020-04-21 | Microsoft Technology Licensing, Llc | Anti-aliasing of graphical elements defined based on functions |
US10646289B2 (en) * | 2015-12-29 | 2020-05-12 | Koninklijke Philips N.V. | System, controller and method using virtual reality device for robotic surgery |
US10646283B2 (en) | 2018-02-19 | 2020-05-12 | Globus Medical Inc. | Augmented reality navigation systems for use with robotic surgical systems and methods of their use |
US10650594B2 (en) | 2015-02-03 | 2020-05-12 | Globus Medical Inc. | Surgeon head-mounted display apparatuses |
US10685498B2 (en) | 2014-05-13 | 2020-06-16 | Nant Holdings Ip, Llc | Augmented reality content rendering via albedo models, systems and methods |
US11153555B1 (en) | 2020-05-08 | 2021-10-19 | Globus Medical Inc. | Extended reality headset camera system for computer assisted navigation in surgery |
US11207150B2 (en) | 2020-02-19 | 2021-12-28 | Globus Medical, Inc. | Displaying a virtual model of a planned instrument attachment to ensure correct selection of physical instrument attachment |
US11382700B2 (en) | 2020-05-08 | 2022-07-12 | Globus Medical Inc. | Extended reality headset tool tracking and control |
US11382699B2 (en) | 2020-02-10 | 2022-07-12 | Globus Medical Inc. | Extended reality visualization of optical tool tracking volume for computer assisted navigation in surgery |
US11410398B2 (en) | 2018-11-21 | 2022-08-09 | Hewlett-Packard Development Company, L.P. | Augmenting live images of a scene for occlusion |
US11464581B2 (en) | 2020-01-28 | 2022-10-11 | Globus Medical, Inc. | Pose measurement chaining for extended reality surgical navigation in visible and near infrared spectrums |
US11510750B2 (en) | 2020-05-08 | 2022-11-29 | Globus Medical, Inc. | Leveraging two-dimensional digital imaging and communication in medicine imagery in three-dimensional extended reality applications |
US11607277B2 (en) | 2020-04-29 | 2023-03-21 | Globus Medical, Inc. | Registration of surgical tool with reference array tracked by cameras of an extended reality headset for assisted navigation during surgery |
US11737831B2 (en) | 2020-09-02 | 2023-08-29 | Globus Medical Inc. | Surgical object tracking template generation for computer assisted navigation during surgical procedure |
US11750794B2 (en) | 2015-03-24 | 2023-09-05 | Augmedics Ltd. | Combining video-based and optic-based augmented reality in a near eye display |
US11766296B2 (en) | 2018-11-26 | 2023-09-26 | Augmedics Ltd. | Tracking system for image-guided surgery |
US11801115B2 (en) | 2019-12-22 | 2023-10-31 | Augmedics Ltd. | Mirroring in image guided surgery |
US11896445B2 (en) | 2021-07-07 | 2024-02-13 | Augmedics Ltd. | Iliac pin and adapter |
US11925423B2 (en) | 2018-01-10 | 2024-03-12 | Covidien Lp | Guidance for positioning a patient and surgical robot |
US11974887B2 (en) | 2018-05-02 | 2024-05-07 | Augmedics Ltd. | Registration marker for an augmented reality system |
US11980506B2 (en) | 2019-07-29 | 2024-05-14 | Augmedics Ltd. | Fiducial marker |
US11992373B2 (en) | 2019-12-10 | 2024-05-28 | Globus Medical, Inc | Augmented reality headset with varied opacity for navigated robotic surgery |
US12044856B2 (en) | 2022-09-13 | 2024-07-23 | Augmedics Ltd. | Configurable augmented reality eyewear for image-guided medical intervention |
Families Citing this family (43)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP5859441B2 (en) * | 2009-10-02 | 2016-02-10 | コーニンクレッカ フィリップス エヌ ヴェKoninklijke Philips N.V. | Viewpoint selection to generate additional views in 3D video |
SG183408A1 (en) * | 2010-04-08 | 2012-09-27 | Univ Nanyang Tech | Image generating devices, graphics boards, and image generating methods |
US20110287811A1 (en) * | 2010-05-21 | 2011-11-24 | Nokia Corporation | Method and apparatus for an augmented reality x-ray |
CN102271262B (en) * | 2010-06-04 | 2015-05-13 | 三星电子株式会社 | Multithread-based video processing method for 3D (Three-Dimensional) display |
KR20110137634A (en) * | 2010-06-17 | 2011-12-23 | 삼성전자주식회사 | Method for generating sketch image and display apparaus applying the same |
KR101188715B1 (en) * | 2010-10-04 | 2012-10-09 | 한국과학기술연구원 | 3 dimension tracking system for surgery simulation and localization sensing method using the same |
JP5738569B2 (en) * | 2010-10-15 | 2015-06-24 | 任天堂株式会社 | Image processing program, apparatus, system and method |
US8514295B2 (en) * | 2010-12-17 | 2013-08-20 | Qualcomm Incorporated | Augmented reality processing based on eye capture in handheld device |
EP3654146A1 (en) | 2011-03-29 | 2020-05-20 | QUALCOMM Incorporated | Anchoring virtual images to real world surfaces in augmented reality systems |
KR20120117165A (en) * | 2011-04-14 | 2012-10-24 | 삼성전자주식회사 | Method of generating 3-dimensional image and endoscope apparatus using the same |
EP2521097B1 (en) * | 2011-04-15 | 2020-01-22 | Sony Interactive Entertainment Europe Limited | System and Method of Input Processing for Augmented Reality |
US8970693B1 (en) * | 2011-12-15 | 2015-03-03 | Rawles Llc | Surface modeling with structured light |
DE102011089233A1 (en) * | 2011-12-20 | 2013-06-20 | Siemens Aktiengesellschaft | Method for texture adaptation in medical image for repairing abdominal aorta aneurysm on angiography system for patient, involves adjusting portion of image texture that is designed transparent such that visibility of object is maintained |
US9734633B2 (en) | 2012-01-27 | 2017-08-15 | Microsoft Technology Licensing, Llc | Virtual environment generating system |
WO2013118065A1 (en) * | 2012-02-06 | 2013-08-15 | Biooptico Ab | Camera arrangement and image processing method for quantifying tissue structure and degeneration |
CN102663788B (en) * | 2012-04-12 | 2014-09-10 | 云南大学 | Pen light-colored artistic effect drawing method based on unreality feeling |
US9576397B2 (en) | 2012-09-10 | 2017-02-21 | Blackberry Limited | Reducing latency in an augmented-reality display |
EP2706508B1 (en) * | 2012-09-10 | 2019-08-28 | BlackBerry Limited | Reducing latency in an augmented-reality display |
US9265458B2 (en) | 2012-12-04 | 2016-02-23 | Sync-Think, Inc. | Application of smooth pursuit cognitive testing paradigms to clinical drug development |
US9380976B2 (en) | 2013-03-11 | 2016-07-05 | Sync-Think, Inc. | Optical neuroinformatics |
CN104346777B (en) * | 2013-08-09 | 2017-08-29 | 联想(北京)有限公司 | A kind of method and device for adding real enhancement information |
CN103489214A (en) * | 2013-09-10 | 2014-01-01 | 北京邮电大学 | Virtual reality occlusion handling method, based on virtual model pretreatment, in augmented reality system |
KR101652888B1 (en) * | 2014-08-20 | 2016-09-01 | 재단법인 아산사회복지재단 | Method for displaying a surgery instrument by surgery navigation |
KR20170040342A (en) * | 2014-09-09 | 2017-04-12 | 노키아 테크놀로지스 오와이 | Stereo image recording and playback |
US10241569B2 (en) | 2015-12-08 | 2019-03-26 | Facebook Technologies, Llc | Focus adjustment method for a virtual reality headset |
US10445860B2 (en) | 2015-12-08 | 2019-10-15 | Facebook Technologies, Llc | Autofocus virtual reality headset |
EP3211599A1 (en) * | 2016-02-29 | 2017-08-30 | Thomson Licensing | Adaptive depth-guided non-photorealistic rendering method, corresponding computer program product, computer-readable carrier medium and device |
US11106276B2 (en) | 2016-03-11 | 2021-08-31 | Facebook Technologies, Llc | Focus adjusting headset |
JP6493885B2 (en) | 2016-03-15 | 2019-04-03 | 富士フイルム株式会社 | Image alignment apparatus, method of operating image alignment apparatus, and image alignment program |
WO2017165566A1 (en) * | 2016-03-25 | 2017-09-28 | The Regents Of The University Of California | High definition, color images, animations, and videos for diagnostic and personal imaging applications |
US10379356B2 (en) | 2016-04-07 | 2019-08-13 | Facebook Technologies, Llc | Accommodation based optical correction |
JP6698824B2 (en) | 2016-04-11 | 2020-05-27 | 富士フイルム株式会社 | Image display control device, method and program |
US10429647B2 (en) | 2016-06-10 | 2019-10-01 | Facebook Technologies, Llc | Focus adjusting virtual reality headset |
CN106780760A (en) * | 2016-12-12 | 2017-05-31 | 大连文森特软件科技有限公司 | System is viewed and emulated in a kind of plastic operation based on AR virtual reality technologies |
US10310598B2 (en) | 2017-01-17 | 2019-06-04 | Facebook Technologies, Llc | Varifocal head-mounted display including modular air spaced optical assembly |
US10818028B2 (en) * | 2018-12-17 | 2020-10-27 | Microsoft Technology Licensing, Llc | Detecting objects in crowds using geometric context |
CN110009720B (en) * | 2019-04-02 | 2023-04-07 | 阿波罗智联(北京)科技有限公司 | Image processing method and device in AR scene, electronic equipment and storage medium |
US10881353B2 (en) * | 2019-06-03 | 2021-01-05 | General Electric Company | Machine-guided imaging techniques |
KR102216587B1 (en) * | 2019-10-24 | 2021-02-17 | 주식회사 레티널 | Optical device for augmented reality which can prevent ghost images |
EP4093311A4 (en) * | 2020-01-22 | 2023-06-14 | Beyeonics Surgical Ltd. | System and method for improved electronic assisted medical procedures |
CA3231531A1 (en) * | 2021-09-10 | 2023-03-16 | Boston Scientific Scimed, Inc. | Method of robust surface and depth estimation |
CN116540872B (en) * | 2023-04-28 | 2024-06-04 | 中广电广播电影电视设计研究院有限公司 | VR data processing method, device, equipment, medium and product |
CN117041511B (en) * | 2023-09-28 | 2024-01-02 | 青岛欧亚丰科技发展有限公司 | Video image processing method for visual interaction enhancement of exhibition hall |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6151009A (en) * | 1996-08-21 | 2000-11-21 | Carnegie Mellon University | Method and apparatus for merging real and synthetic images |
WO2006099490A1 (en) * | 2005-03-15 | 2006-09-21 | The University Of North Carolina At Chapel Hill | Methods, systems, and computer program products for processing three-dimensional image data to render an image from a viewpoint within or beyond an occluding region of the image data |
US20070070063A1 (en) * | 2005-09-29 | 2007-03-29 | Siemens Medical Solutions Usa, Inc. | Non-photorealistic volume rendering of ultrasonic data |
US7295720B2 (en) * | 2003-03-19 | 2007-11-13 | Mitsubishi Electric Research Laboratories | Non-photorealistic camera |
US7355597B2 (en) * | 2002-05-06 | 2008-04-08 | Brown University Research Foundation | Method, apparatus and computer program product for the interactive rendering of multivalued volume data with layered complementary values |
EP2034442A1 (en) * | 2007-09-06 | 2009-03-11 | Thomson Licensing | Method for non-photorealistic rendering of an image frame sequence |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2005108108A (en) * | 2003-10-01 | 2005-04-21 | Canon Inc | Operating device and method for three-dimensional cg and calibration device for position/attitude sensor |
-
2007
- 2007-06-29 GB GBGB0712690.7A patent/GB0712690D0/en not_active Ceased
-
2008
- 2008-06-23 KR KR1020107002014A patent/KR20100051798A/en not_active Application Discontinuation
- 2008-06-23 WO PCT/GB2008/002139 patent/WO2009004296A1/en active Application Filing
- 2008-06-23 AT AT08762451T patent/ATE500578T1/en not_active IP Right Cessation
- 2008-06-23 JP JP2010514098A patent/JP5186561B2/en active Active
- 2008-06-23 EP EP08762451A patent/EP2174297B1/en active Active
- 2008-06-23 ES ES08762451T patent/ES2361228T3/en active Active
- 2008-06-23 DE DE602008005312T patent/DE602008005312D1/en active Active
- 2008-06-23 US US12/666,957 patent/US8878900B2/en active Active
- 2008-06-23 CN CN200880022657.0A patent/CN101802873B/en active Active
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6151009A (en) * | 1996-08-21 | 2000-11-21 | Carnegie Mellon University | Method and apparatus for merging real and synthetic images |
US7355597B2 (en) * | 2002-05-06 | 2008-04-08 | Brown University Research Foundation | Method, apparatus and computer program product for the interactive rendering of multivalued volume data with layered complementary values |
US7295720B2 (en) * | 2003-03-19 | 2007-11-13 | Mitsubishi Electric Research Laboratories | Non-photorealistic camera |
WO2006099490A1 (en) * | 2005-03-15 | 2006-09-21 | The University Of North Carolina At Chapel Hill | Methods, systems, and computer program products for processing three-dimensional image data to render an image from a viewpoint within or beyond an occluding region of the image data |
US20070070063A1 (en) * | 2005-09-29 | 2007-03-29 | Siemens Medical Solutions Usa, Inc. | Non-photorealistic volume rendering of ultrasonic data |
EP2034442A1 (en) * | 2007-09-06 | 2009-03-11 | Thomson Licensing | Method for non-photorealistic rendering of an image frame sequence |
Non-Patent Citations (30)
Title |
---|
Bichlmeier et al. "Improving Depth Perception in Medical AR A Virtual Vision Panel to the Inside of the Patient", Proceedings of BVM 2007, p. 217-221. |
Bruckner et al. "Illustrative Context-Preserving Exploration of Volume Data", IEEE Transactions on Visualization and Computer Graphics Nov./ Dec. 2006, vol. 12, No. 6, p. 1559-1569. |
Chris Furmanski et al., Augmented-Reality Visualizations Guided by Cognition; Perceptual Heuristics for Combining Visible and Obscured Information, HRL Laboratories, LLC, Malibu, CA, USA. |
Christoph Bichlmeier et al., Virtual Window for Improved Depth Perception in Medical AR, Chair for Computer Aided Medical Procedures (CAMP), TU Munich, Germany. |
Csebfalvi. "Fast Visualization of Object Contours by Non-Photorealistic Volume Rendering", Eurographics 2001, vol. 20, No. 3, 10 Pages. |
Danail Stoyanov et al., A Practical Approach Towards Accurate Dense 3D Depth Recovery for Robotic Laparoscopic Surgery, Computer Aided Surgery, Jul. 2005; 10(4): 199-208, Imperial College London, London, UK. |
Edwards et al. "Clinical Experience and Perception in Stereo Augmented Reality Surgical Navigation", MIAR 2004, LNCS 3150, p. 369-376. |
Ève Coste-Manìere, Louaï Adhami, Fabien Mourgues, Alain Carpentier, Planning, simulation, and augmented reality for robotic cardiac procedures: The STARS system of the ChIR team, Seminars in Thoracic and Cardiovascular Surgery, vol. 15, Issue 2, Apr. 2003, pp. 141-156, ISSN 1043-0679, 10.1016/S1043-0679(03)70022-7. * |
Ève Coste-Manìere, Louaï{umlaut over ( )} Adhami, Fabien Mourgues, Alain Carpentier, Planning, simulation, and augmented reality for robotic cardiac procedures: The STARS system of the ChIR team, Seminars in Thoracic and Cardiovascular Surgery, vol. 15, Issue 2, Apr. 2003, pp. 141-156, ISSN 1043-0679, 10.1016/S1043-0679(03)70022-7. * |
F. Sauer et al., An Augmented Reality Navigation System with a Single-Camera Tracker: System Design and Needle Biopsy Phantom Trial, Imaging & Visualization Dept., Siemens Corporate Research, Princeton, NJ, USA. |
George P. Mylonas et al., Gaze Contingent Depth Recovery and Motion Stabilisation for Minimally Invasive Robotic Surgery, Royal Society/Wolfson Medical Image Computing Library, Imperial College London, London, UK. |
H.U. Rashid et al., Differential Algorithm for the Determination of Shape From Shading Using a Point Light Source, vol. 10, No. 2, Mar. 1992, Imperial College of Science, Technology and Medicine, London, UK. |
Hui Xu et al., Stylized Rendering of 3D Scanned Real World Environments, University of Minnesota at Twin Cities. |
J. Edward Swann II et al., Egocentric Depth Judgments in Optical, See-Through Augmented Reality, IEEE Transactions on Visualization and Computer Graphics, vol. 13, No. 3, May/Jun. 2007. |
Jan Fischer et al., Stylized Augmented Reality for Improved Immersion, IEEE Virtual Reality, Mar. 12-16, 2005, Bonn, Germany. |
Jurgen Dollner et al., Real-Time Expressive Rendering of City Models, Hasso-Plattner-Institute at the University of Potsdam. |
King et al. "Stereo Augmented Reality in the Surgical Microscope", Presence Aug. 2000, vol. 9, No. 4, p. 360-368. |
LG Johnson, et al., Surface Transparency Makes Stereo Overlays Unpredictable: the Implications for Augmented Reality, Stud Health Technol Inform, 2003; 94; 131-6. |
Michael Bajura et al., Merging Virtual Objects With the Real World: Seeing Ultrasound Imagery Within the Patient, Computer Graphics, 26, 2, Jul. 1992, Department of Computer Science, University of North Carolina, Chap Hill, NC. |
Mirna Lerotic et al., PQ-Space Based Non-Photorealistic Rendering for Augmented Reality, Lecture Notes in Computer Science, vol. 4792, 2007, Springer Berlin, Heidelberg. |
Mitsuhiro Hayashibe et al., Laser-Scan Endoscope System for Intraoperative Geometry Acquisition and Surgical Robot Safety Management, Medical Image Analysis 10 (2006) 509-519. |
Norihisa Shigemura, MD et al., Long-Term Outcomes After a Variety of Video-Assisted Thoracoscopic Lobectomy Approaches for Clinical Stage IA Lung Cancer: a Multi-Institutional Study, the Journal of Thoracic and Cardiovascular Surgery, vol. 132, No. 3. |
Philip J. Edwards et al., Design and Evaluation of a System for Microscope-Assisted Guided Interventions (MAGI), IEEE Transactions on Medical Imaging, vol. 19, No. 11, Nov. 2000, London, UK. |
R.J. Lapeer et al., PC-based Volume Rendering for Medical Visualisation and Augmented Reality based Surgical Navigation, School of Computing Sciences, University of East Anglia, Norwich, UK. |
Ronald Azuma et al., Performance Analysis of an Outdoor Augmented Realty Tracking System that Relies Upon a Few Mobile Beacons, Proceedings of the 5th Int'l Symp. On Mixed and Augmented Reality, pp. 101-104, Oct. 2006, Santa Barbara, CA. |
Sebastian Vogt et al., Reality Augmentation for Medical Procedures: System Architecture, Single Camera Marker Tracking, and System Evaluation, International Journal of Computer Vision 70(2), 179-190, 2006. |
Stoyanov et al. "Dense 3D Depth Recovery for Soft Tissue Deformation During Robotically Assisted Laparoscopic Surgery", MICCAI 2004, LNCS3217, p. 41-48. |
Takayuki Okatani et al., Shape Reconstruction from an Endoscope Image by Shape from Shading Technique for a Point Light Source at the Projection Center, Computer Vision and Image Understanding, vol. 66, No. 2, May, pp. 119-131, 1997, Article No. IV970613, University of Tokyo, Bunkyo-ku, Tokyo 113, Japan. |
Thomas P. Koninckx et al., Real-Time Range Acquisition by Adaptive Structured Light, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, No. 3, Mar. 2006. |
Tobias Sielhorst et al., Depth Perception-A Major Issue in Medical AR: Evaluation Study by Twenty Surgeons, MICCAI 2006, LNCS 4190, pp. 364-372, 2006. |
Cited By (42)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10685498B2 (en) | 2014-05-13 | 2020-06-16 | Nant Holdings Ip, Llc | Augmented reality content rendering via albedo models, systems and methods |
US11710282B2 (en) | 2014-05-13 | 2023-07-25 | Nant Holdings Ip, Llc | Augmented reality content rendering via Albedo models, systems and methods |
US11176754B2 (en) | 2014-05-13 | 2021-11-16 | Nant Holdings Ip, Llc | Augmented reality content rendering via albedo models, systems and methods |
US12002171B2 (en) | 2015-02-03 | 2024-06-04 | Globus Medical, Inc | Surgeon head-mounted display apparatuses |
US11734901B2 (en) | 2015-02-03 | 2023-08-22 | Globus Medical, Inc. | Surgeon head-mounted display apparatuses |
US11763531B2 (en) | 2015-02-03 | 2023-09-19 | Globus Medical, Inc. | Surgeon head-mounted display apparatuses |
US11062522B2 (en) | 2015-02-03 | 2021-07-13 | Global Medical Inc | Surgeon head-mounted display apparatuses |
US10650594B2 (en) | 2015-02-03 | 2020-05-12 | Globus Medical Inc. | Surgeon head-mounted display apparatuses |
US11176750B2 (en) | 2015-02-03 | 2021-11-16 | Globus Medical, Inc. | Surgeon head-mounted display apparatuses |
US11461983B2 (en) | 2015-02-03 | 2022-10-04 | Globus Medical, Inc. | Surgeon head-mounted display apparatuses |
US11217028B2 (en) | 2015-02-03 | 2022-01-04 | Globus Medical, Inc. | Surgeon head-mounted display apparatuses |
US11750794B2 (en) | 2015-03-24 | 2023-09-05 | Augmedics Ltd. | Combining video-based and optic-based augmented reality in a near eye display |
US11413099B2 (en) | 2015-12-29 | 2022-08-16 | Koninklijke Philips N.V. | System, controller and method using virtual reality device for robotic surgery |
US10646289B2 (en) * | 2015-12-29 | 2020-05-12 | Koninklijke Philips N.V. | System, controller and method using virtual reality device for robotic surgery |
US20190088019A1 (en) * | 2016-03-16 | 2019-03-21 | Koninklijke Philips N.V. | Calculation device for superimposing a laparoscopic image and an ultrasound image |
US10628995B2 (en) | 2017-04-17 | 2020-04-21 | Microsoft Technology Licensing, Llc | Anti-aliasing of graphical elements defined based on functions |
US11925423B2 (en) | 2018-01-10 | 2024-03-12 | Covidien Lp | Guidance for positioning a patient and surgical robot |
US10646283B2 (en) | 2018-02-19 | 2020-05-12 | Globus Medical Inc. | Augmented reality navigation systems for use with robotic surgical systems and methods of their use |
US11980507B2 (en) | 2018-05-02 | 2024-05-14 | Augmedics Ltd. | Registration of a fiducial marker for an augmented reality system |
US11980508B2 (en) | 2018-05-02 | 2024-05-14 | Augmedics Ltd. | Registration of a fiducial marker for an augmented reality system |
US11974887B2 (en) | 2018-05-02 | 2024-05-07 | Augmedics Ltd. | Registration marker for an augmented reality system |
US11410398B2 (en) | 2018-11-21 | 2022-08-09 | Hewlett-Packard Development Company, L.P. | Augmenting live images of a scene for occlusion |
US11980429B2 (en) | 2018-11-26 | 2024-05-14 | Augmedics Ltd. | Tracking methods for image-guided surgery |
US11766296B2 (en) | 2018-11-26 | 2023-09-26 | Augmedics Ltd. | Tracking system for image-guided surgery |
US11980506B2 (en) | 2019-07-29 | 2024-05-14 | Augmedics Ltd. | Fiducial marker |
US11992373B2 (en) | 2019-12-10 | 2024-05-28 | Globus Medical, Inc | Augmented reality headset with varied opacity for navigated robotic surgery |
US11801115B2 (en) | 2019-12-22 | 2023-10-31 | Augmedics Ltd. | Mirroring in image guided surgery |
US11464581B2 (en) | 2020-01-28 | 2022-10-11 | Globus Medical, Inc. | Pose measurement chaining for extended reality surgical navigation in visible and near infrared spectrums |
US11883117B2 (en) | 2020-01-28 | 2024-01-30 | Globus Medical, Inc. | Pose measurement chaining for extended reality surgical navigation in visible and near infrared spectrums |
US11382699B2 (en) | 2020-02-10 | 2022-07-12 | Globus Medical Inc. | Extended reality visualization of optical tool tracking volume for computer assisted navigation in surgery |
US11207150B2 (en) | 2020-02-19 | 2021-12-28 | Globus Medical, Inc. | Displaying a virtual model of a planned instrument attachment to ensure correct selection of physical instrument attachment |
US11690697B2 (en) | 2020-02-19 | 2023-07-04 | Globus Medical, Inc. | Displaying a virtual model of a planned instrument attachment to ensure correct selection of physical instrument attachment |
US11607277B2 (en) | 2020-04-29 | 2023-03-21 | Globus Medical, Inc. | Registration of surgical tool with reference array tracked by cameras of an extended reality headset for assisted navigation during surgery |
US11839435B2 (en) | 2020-05-08 | 2023-12-12 | Globus Medical, Inc. | Extended reality headset tool tracking and control |
US11838493B2 (en) | 2020-05-08 | 2023-12-05 | Globus Medical Inc. | Extended reality headset camera system for computer assisted navigation in surgery |
US11510750B2 (en) | 2020-05-08 | 2022-11-29 | Globus Medical, Inc. | Leveraging two-dimensional digital imaging and communication in medicine imagery in three-dimensional extended reality applications |
US11382700B2 (en) | 2020-05-08 | 2022-07-12 | Globus Medical Inc. | Extended reality headset tool tracking and control |
US11153555B1 (en) | 2020-05-08 | 2021-10-19 | Globus Medical Inc. | Extended reality headset camera system for computer assisted navigation in surgery |
US11737831B2 (en) | 2020-09-02 | 2023-08-29 | Globus Medical Inc. | Surgical object tracking template generation for computer assisted navigation during surgical procedure |
US11896445B2 (en) | 2021-07-07 | 2024-02-13 | Augmedics Ltd. | Iliac pin and adapter |
US12044856B2 (en) | 2022-09-13 | 2024-07-23 | Augmedics Ltd. | Configurable augmented reality eyewear for image-guided medical intervention |
US12044858B2 (en) | 2022-09-13 | 2024-07-23 | Augmedics Ltd. | Adjustable augmented reality eyewear for image-guided medical intervention |
Also Published As
Publication number | Publication date |
---|---|
WO2009004296A1 (en) | 2009-01-08 |
EP2174297A1 (en) | 2010-04-14 |
JP5186561B2 (en) | 2013-04-17 |
ES2361228T3 (en) | 2011-06-15 |
DE602008005312D1 (en) | 2011-04-14 |
JP2010532035A (en) | 2010-09-30 |
CN101802873B (en) | 2012-11-28 |
US20100177163A1 (en) | 2010-07-15 |
EP2174297B1 (en) | 2011-03-02 |
CN101802873A (en) | 2010-08-11 |
ATE500578T1 (en) | 2011-03-15 |
GB0712690D0 (en) | 2007-08-08 |
KR20100051798A (en) | 2010-05-18 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8878900B2 (en) | Non photorealistic rendering of augmented reality | |
Mori et al. | A survey of diminished reality: Techniques for visually concealing, eliminating, and seeing through real objects | |
US8090174B2 (en) | Virtual penetrating mirror device for visualizing virtual objects in angiographic applications | |
US9681925B2 (en) | Method for augmented reality instrument placement using an image based navigation system | |
Bichlmeier et al. | Contextual anatomic mimesis hybrid in-situ visualization method for improving multi-sensory depth perception in medical augmented reality | |
Sielhorst et al. | Advanced medical displays: A literature review of augmented reality | |
US7232409B2 (en) | Method and apparatus for displaying endoscopic images | |
Lerotic et al. | Pq-space based non-photorealistic rendering for augmented reality | |
US11961193B2 (en) | Method for controlling a display, computer program and mixed reality display device | |
Vogt et al. | Reality augmentation for medical procedures: System architecture, single camera marker tracking, and system evaluation | |
US20230050857A1 (en) | Systems and methods for masking a recognized object during an application of a synthetic element to an original image | |
Wang et al. | Autostereoscopic augmented reality visualization for depth perception in endoscopic surgery | |
Kutter et al. | Real-time volume rendering for high quality visualization in augmented reality | |
CN113906479A (en) | Generating synthetic three-dimensional imagery from local depth maps | |
WO2010081094A2 (en) | A system for registration and information overlay on deformable surfaces from video data | |
US20220215539A1 (en) | Composite medical imaging systems and methods | |
Wieczorek et al. | GPU-accelerated rendering for medical augmented reality in minimally-invasive procedures. | |
Macedo et al. | A semi-automatic markerless augmented reality approach for on-patient volumetric medical data visualization | |
Fischer et al. | A hybrid tracking method for surgical augmented reality | |
Martin-Gomez et al. | Augmented mirrors | |
Reichard et al. | Intraoperative on-the-fly organ-mosaicking for laparoscopic surgery | |
Özgür et al. | [POSTER] visualizing in-organ tumors in augmented monocular laparoscopy | |
Dey et al. | Mixed reality merging of endoscopic images and 3-D surfaces | |
Shakya et al. | Remote surgeon hand motion and occlusion removal in mixed reality in breast surgical telepresence: rural and remote care | |
EP4322114A1 (en) | Projective bisector mirror |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: IMPERIAL INNOVATIONS LIMITED, UNITED KINGDOM Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:YANG, GUANG-ZHONG;LEROTIC, MIRNA;SIGNING DATES FROM 20100208 TO 20100215;REEL/FRAME:024216/0472 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
FEPP | Fee payment procedure |
Free format text: SURCHARGE FOR LATE PAYMENT, LARGE ENTITY (ORIGINAL EVENT CODE: M1554) |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551) Year of fee payment: 4 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 8 |
|
AS | Assignment |
Owner name: IP2IPO INNOVATIONS LIMITED, UNITED KINGDOM Free format text: CHANGE OF NAME;ASSIGNOR:IMPERIAL INNOVATIONS LIMITED;REEL/FRAME:063116/0819 Effective date: 20190301 |