Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T19/00—Manipulating 3D models or images for computer graphics
  • G06T19/006—Mixed reality
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/80—Camera processing pipelines; Components thereof
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2200/00—Indexing scheme for image data processing or generation, in general
  • G06T2200/21—Indexing scheme for image data processing or generation, in general involving computational photography

Definitions

• the present invention relates to augmented reality methods and apparatus, in particular to a method and to various apparatus for adding annotations to data corresponding to a scene.
• augmented-reality applications are known where a user points a portable device towards a scene, e.g. a landscape, a building, a poster, or a painting in a museum, and the display shows the image together with superimposed information concerning the scene.
• a scene e.g. a landscape, a building, a poster, or a painting in a museum
• information can include names, e.g. for mountains and habitations, people names, historical information for buildings, and commercial information such as advertising, e.g. a restaurant menu.
• An example of such a system is described in EP1246080 and in EP22071 13.
• Many annotation methods include a step of comparing an image, such as a 2D image produced by a standard pin-hole camera with a standard CCD or CMOS sensor, or a computer generated image, with a set of reference images stored in a database.
• an aim of the comparison algorithm is to remove the influence of these parameters.
• WO2008134901 describes a method where a first image is taken using a digital camera associated with a communication terminal. Query data related to the first image is transmitted via a communication network to a remote recognition server, where a matching reference image is identified. By replacing a part of the first image with at least a part of the annotated image, an augmented image is generated and displayed at the communication terminal. The augmentation of the first image taken with the camera occurs in the planar space and deals with two-dimensional images and objects only.
• the invention is also achieved by way of an apparatus for capturing and annotating data corresponding to a scene, comprising:
• a plenoptic capturing device for capturing data representing a light field
• the invention also provides an apparatus for determining annotations, comprising:
• program code for causing said processor to receive data representing a light field, to match said data with one reference data, to determine an annotation from said store in plenoptic format associated with said reference data, and to send either said annotation in plenoptic format or data corresponding to an annotated image in plenoptic format to a remote device when said program code is executed.
• annotations in a plenoptic format permits a more realistic integration of the annotation in the image in plenoptic format; the annotation seems to be an element of the captured scene, instead of just a text superimposed over an image.
• An annotation in a plenoptic format (also called "plenoptic annotation" in the present application) contains a more complete description of the light field than a conventional annotation, including information of how light rays are modified.
• the provision of annotations in a plenoptic format also permits a selection of the annotation that should be displayed, depending on a focus distance and/or on a viewpoint selected by the user during the rendering of the image, or automatically selected, for example based on his interests.
• the computational expense for rendering the plenoptic data in a human understandable format is reduced.
• the rendering process is identical for both.
• a single rendering process can be used for rendering the images and the associated annotations.
• the projection parameters selected for the plenoptic rendering process (such as selection of focus, depth, change of view point, ...) also apply on plenoptic annotations. For example, when changing the focus or viewpoint of a plenoptic image, the same
• the transformation can be used for displaying the plenoptic annotations at various distances.
• the effect of the annotation is applied to the captured plenoptic image, and a rendering of the modified plenoptic image is performed.
• a plenoptic annotation i.e., an annotation in plenoptic format
• a plenoptic annotation may contain as much information about light rays as images captured by plenoptic capturing device.
• the annotation can retain the characteristics of light reflection on the surface of an annotated object which is not possible with a conventional annotation system. In this sense, annotated views seem more realistic.
• the present invention also relates to a method for attaching annotations to a reference image in plenoptic format, comprising:
• This method may be carried out with a suitable authoring system, such as a suitable software application or web site.
• Fig. 1 schematically illustrates a plenoptic capturing device for capturing data representing a light field of scene with an object at a first distance.
• Fig. 2 schematically illustrates a plenoptic capturing device for capturing data representing a light field of scene with an object at a second distance.
• Fig. 3 schematically illustrates a plenoptic capturing device for capturing data representing a light field of scene with an object at a third distance.
• Fig. 4 schematically illustrates a system comprising various apparatus elements that together embody the invention.
• Figs. 5A to 5B show annotated views rendered from the same plenoptic data, wherein the viewpoint selected by the user during the rendering has changed between the two views, resulting in a same annotation rendered in a different way.
• Figs. 6A to 6B show annotated views rendered from the same plenoptic data, wherein the viewpoint selected by the user during the rendering has changed between the two views, resulting in a first annotation made visible on the first view and a second annotation made visible on the second view.
• Figs. 7A to 7B show annotated views rendered from the same plenoptic data, wherein the focusing distance selected by the user during the rendering has changed between the two views, resulting in a first annotation made visible on the first view and a second annotation made visible on the second view.
• Fig. 8 is a block diagram of a method for generating and rendering a view with annotations in plenoptic format.
• Fig. 9 is a block diagram of a method for modifying the rendering of annotations when the viewer selects a different viewing direction and/or a different focusing distance on a view.
• Fig. 10 is a block diagram of a method for associating an annotation in a plenoptic format with a reference data.
• Fig. 1 1 is a block diagram of a method of continuous annotation of a series of plenoptic images, such as video plenoptic images or plenoptic images captured by a user in movement.
• a complete light field may comprise up to 7 parameters for describing each light ray (or for describing the light rays at a given position): 3 for the position, 2 for the direction, 1 for the wavelength and (in the case of video) 1 for the time.
• Some current plenoptic cameras deliver plenoptic data comprising 2 parameters for the position, 2 for the
• plenoptic data representing a so-called plenoptic light field, i.e., a matrix indicating at least the position and the direction of the light rays. It means that plenoptic data generated by a plenoptic capturing device contains more information about the light field than a conventional 2D image data generated by a conventional 2D camera.
• plenoptic sensors that can record such a plenoptic light field: Lytro and Raytrix. Their two cameras are slightly different in terms of design, but the main idea is to decompose the different directions of the light that is supposed to fall on a single photosite (or pixel) in a standard camera sensor. To that aim, as illustrated on Fig. 1 , an array of micro-lenses 20 is placed behind the main lens 1, in place of the sensor of conventional cameras.
• the micro-lenses 21 redirect the light rays according to their incident angle and the redirected light rays reach different pixels 210 of the sensor 21 .
• the amount of light measured by each of the N X M pixels 210 making a sub image depends on the direction of the light beams that hit the micro-lens 20 in front of that sub image.
• each sub image thus includes a patch of brightness values indicating the amount of light coming from various directions onto that sub-image.
• the array of micro-lenses 20 is located on the image plane formed by the main lens 1 of the plenoptic capturing device, and the sensor 21 is located at a distance f from the micro-lenses, where f is the focal length of the micro-lenses.
• This design allows a high angular resolution but suffers from relatively poor spatial resolution (the effective number of pixels per rendered image is equal to the number of micro- lenses).
• This problem is addressed by other plenoptic capturing devices where the micro-lenses focus on the image plane of the main lens, thus creating a gap between the micro-lenses and the image plane. The price to pay in such a design is poorer angular resolution.
• the plenoptic light field corresponding to a scene with a single point 3 in this example depends on the distance from the point 3 to the main lens 1 .
• all the light beams from this object reach the same micro-lens 20, thus resulting in a plenoptic light field where all the pixels in the sub-image corresponding to this micro-lens record a first positive light intensity while all other pixels corresponding to other lenses record a different, null light intensity.
• Fig. 2 where the object 3 is closer to the lens 1 , some light beams originating from the point 3 reach pixels of other sub images, i.e., sub images
• the digital data 22 delivered by the sensor 21 depends on the distance to the object 3.
• the plenoptic sensor 21 thus delivers plenoptic data 22
• each pixel of a sub image corresponds to the intensity measure of a light ray hitting the sensor with a certain incidence angle phi (in the plane of the page) and theta (perpendicular to the plane of the page).
• Fig. 4 schematically illustrates a block diagram of an annotation system embodying the invention.
• the system comprises a user device 4, such as a hand-held device, a smartphone, a tablet, a camera, glasses, goggles, contact lenses, etc.
• the device 4 includes a plenoptic capturing device 41 such as the camera illustrated in Figs. 1 to 3, for capturing data representing a light field on a scene 3, a processor such as a microprocessor 400 with a suitable program code, and a communication module 401 such as a WIFI and/or cellular interface for connecting the device 4 to a remote server 5, for example a cloud server, over a network such as the Internet 6.
• a remote server 5 for example a cloud server
• the server 5 includes a storage 50 with a database such as a SQL database, a set of XML documents, a set of images in plenoptic format, etc , for storing a collection of reference plenoptic data representing images and/or one or a plurality of global models, and a processor 51 , including a microprocessor with computer code for causing the microprocessor to perform the operations needed in the annotation method.
• the annotations and corresponding positions can also be stored in storage 50 along with the reference plenoptic data.
• the program code executed by the user device 4 could include for example an application software, or app, that can be downloaded and installed by the user in the user device 4.
• the program code could also include part of the operating code of the user device 4.
• the program code could also include code embedded in web page or executed in a browser, including for example Java, Javascript, HTML5 code, etc.
• the program code may be stored as a computer program product in a tangible apparatus readable medium, such as a Flash memory, a hard disk, or any type of permanent or semi-permanent memory.
• the program code is executed by the microprocessor 400 in the user device 4 for causing this microprocessor to send at least some of the captured data sets corresponding to light fields, or features of those data sets, to the remote server 5.
• the program code is arranged for sending the data in a "plenoptic format", i.e., without losing the information about the direction of the light rays.
• the program code can also cause the
• microprocessor 400 to receive from the server 5 annotated data in a plenoptic format, or annotated images, or annotations related to the previously sent plenoptic data, and for rendering a view corresponding to the captured data with annotations.
• the plenoptic annotation method may comprise two parts: an offline process and an online process.
• the main purpose of the offline process is to associate annotations with reference images in a plenoptic format, or with other 2D, stereoscopic, or 3D reference images.
• the offline process may comprise for example the following steps:
• This offline process can be performed either on the server 5, in the user device 4, or in yet another equipment such as a personal computer, a tablet, etc. Typically, this offline process is executed only once for each annotation associated with the reference image. If the selected annotation is initially not available in a plenoptic format, it may be converted into a plenoptic format.
• the main purpose of the online process is to add plenoptic annotations to plenoptic images.
• the online process may comprise two phases. The first one may be carried out by a program code executed by a microprocessor in the server 5, which may include executable programs or other codes for causing the server 5 to carry out at least some of the following tasks:
• this matching could be done locally in the user's device with a set of locally stored reference images or with a model locally stored in the device.
• the server 5 is embarked on the user device 4.
• the online process can be executed several times in accordance with user's request.
• the second phase of the online process may be carried out by a program code executed by a microprocessor in the device 4, which may include executable programs or other codes for causing the device 4 to carry out at least some of the following tasks:
• this step could be done on the server's 5 side. In this case, either the final rendered view is transmitted back to the device 4, or the entire annotated light field.
• a user can associate annotations with a particular position and orientation with respect to a rendered view of a plenoptic reference image, and indicate one or plurality of light field parameters that the annotation should use in this specific view.
• a same annotation may be rendered differently depending on the viewpoint selected by the viewer during the rendering of the view.
• a first annotation may be replaced by a second annotation at the same location if the viewer selects a different viewpoint as the light field parameters of the annotation may change.
• FIG. 10 An example of flowchart for the offline process is illustrated on Fig. 10.
• This flowchart illustrates a method which allows a user to choose the annotation that has to be associated with a reference image, and the position, orientation and light field parameters with this annotation, so that this annotation will be applied to the captured plenoptic images matching this plenoptic reference image.
• This method may use an annotation authoring system, which may be run locally in the user's device 4.
• the annotation authoring system may also be hosted on the server 5 where a web platform presents some tools to manage annotations and relate them to plenoptic reference images.
• Services such as augmented reality usage statistics, may also be available from the web platform.
• the annotation authoring system may also be run in a different server or equipment, including a user's personal computer, tablet, etc.
• step 1 50 a user selects a reference image, such as an image in a plenoptic format.
• the image is uploaded on the plenoptic authoring system and serves as a support image for the annotations.
• a viewer renders the uploaded data to the user in a way such that the user can view it. If the data is in a plenoptic format, which cannot be understood easily as such by a human, this might include using a plenoptic rendering module for rendering the plenoptic model in a space understandable by the user.
• the viewer constitutes a tool to manipulate the plenoptic data and place annotations at the desired position and orientation with respect to a given view, but all processing and combination with the plenoptic annotation are done directly in the plenoptic space.
• the plenoptic model can be rendered as a 2D view so that the user can visualize it from one viewpoint at a time, and with one focus distance at a time, allowing him to understand and edit the plenoptic model.
• controls are available such that upon request, another 2D view can be displayed.
• the plenoptic model might be rendered as a partial 3D scene, where different directions of the rays can be visualized.
• a major difference with standard complete 3D scene is that the 3D scene exploration is limited when rendered from a plenoptic model. For instance, the view directions as well as the view position are limited to what has been captured by the plenoptic capturing device.
• the user selects a plenoptic annotation he wants to associate with a particular element or location of the plenoptic model.
• the plenoptic annotation is defined in the plenoptic space and thus described with light rays.
• Those light rays can describe for instance a text, an image, a video, or other elements directly acting on plenoptic image light rays.
• the plenoptic annotation may be retrieved from a library of plenoptic annotations in a database or in a file explorer for example.
• the plenoptic annotation can also be created on the fly, for example by capturing it with a plenoptic capturing device, by entering a text with a text editor, by drawing an image and/or by recording a sound or a video.
• the plenoptic annotation can be presented in a library or a list on the authoring system as previews.
• Plenoptic annotation previews correspond to the rendering of the annotation for a default view. This default view can be taken randomly or in a preferred embodiment, as corresponding to the middle view with respect to the plenoptic annotation range of positions and directions.
• the previews allow the user to get a quick and clear idea about what does the plenoptic annotation corresponds to.
• the preview illustrates the annotation applied to the center of the current model view rendered by the authoring system. Therefore, if this type of annotation has only the effect of rotating all model rays by 10°, the preview will be composed of the center part of the current model rendered view, where each ray has been rotated by 10°.
• step 1 52 the user selects with the plenoptic annotation authoring system a position in the coordinate system of the rendered view of the selected reference model at which he wants to add the plenoptic annotation. This can be done for example by dragging the annotation from the annotation preview list on top of the displayed view at the desired location, and possibly by translating, rotating, resizing, cropping and/or otherwise editing the annotation. Alternatively, the user may also enter the coordinates as values in a control panel. [0051] In step 152', the user can adjust the parameters of the annotation light rays to generate another view of the annotation.
• the light rays of the annotation are combined with light rays of the plenoptic model and a new 2D view is generated in the viewer for each new position or new orientation. This is made possible as the user mouse pointer and its movements are projected to the plenoptic space. The movement of the pointer is then applied to the annotation in the plane parallel to the virtual one corresponding to the 2D rendered view.
• a captured plenoptic data can contain information on a direction of light rays, a wave length (i.e. color) for each light ray, thus an annotation can be considered as a modification of those parameters. For instance, attaching a text on the surface of an object can be seen as a modification of the wave length of the light rays at a specific area on the surface.
• the type of effect produced by an annotation is determined by the annotation itself.
• the plenoptic annotation is for example only composed of opaque text. In this case, the model
• the rays of the model may have their direction changed by the annotation in order to reflect the new texture.
• the model ray positions may be changed by the annotation.
• the plenoptic annotation can be seen as a filter modifying light rays. This offers more possibilities of displaying annotated scenes.
• One further example of this processing is to alter the directions of light rays.
• a glow effect can be applied to the light rays incoming from a specific object in the captured plenoptic image by adding
• An annotated object can be made reflective.
• Another example is modification of the property of surface such as modification of texture information. Since a plenoptic annotation allows modifying the variables of light ray such as the direction and wave length, it is possible to modify the surface of an object as if a texture is added on it by combining the modifications in the variables. For instance, the plenoptic annotation enables to change a flat surface with red color to a lumpy surface with yellow color by modifying the direction and the wave length.
• the information describing the effect of the annotation on the model rays may be stored in the plenoptic annotation array as will be described in step 154.
• step 153 the user selects one or a plurality of annotation light field parameters. This could be for example the wavelength of the annotation in order to change its color.
• the user may also define different appearances for the same annotation viewed from different directions, or even a different annotation associated to a same element viewed from different directions.
• the user can choose to navigate to another view of the plenoptic viewer.
• the plenoptic annotations are automatically reported on the new view of the plenoptic model.
• the user can then decide to edit the annotation, change its light field parameters or appearance for this particular view. He can proceed the same way for all available views of the plenoptic model.
• An interpolation process may take place between a first and a second view of the plenoptic annotation to prevent the user from having to navigate through all views of the plenoptic model. These two views of the plenoptic annotation do not have to be consecutive. The user has to specify the appearance of the annotation in the two views and the plenoptic authoring system will automatically generate the in-between views of the plenoptic annotations. Other views of the plenoptic model that have not been associated with the annotation will not display it, resulting in the possibility to not render an annotation for particular view points or focal planes of the scene.
• the plenoptic annotation may comprise data corresponding to light rays and described with a set of parameters.
• the viewer sets some parameters and allows the user to modify the others. Navigating from this view to a second one, the user changes the parameters that have to be fixed by the viewer, while being able to modify the others.
• interpolation process automatically computes the ray parameters of the plenoptic annotation between these two views.
• annotation may be as follows: 3 (or possibly 2) parameters for the ray position in space, 2 parameters for their direction, 1 parameter for their wavelength and possibly 1 for the time.
• the parameters of position, direction and time may for instance be set by the viewer.
• the user could then change the parameters not fixed by the viewer, in this example corresponding to the wavelength of the rays. Let us assume that the user sets it to a first value v1. Now, for another view of the annotation, i.e. for different values of the position, direction and time parameters, let us assume that the user changes the wavelength value for the second view, and set it for instance to v2.
• the interpolation process aims at computing the annotation values between v1 and v2 for views in between the position, direction and time parameters associated with the first and second views.
• the interpolation may also consider computing values for other parameters of the plenoptic data as well, including position, direction, wavelength and/or time.
• interpolation examples include for instance a change in the color of plenoptic annotation passing for example from an orange color to a more reddish one, a change in the visibility of the annotation where for a specific view, the annotation is visible while hidden for another view.
• an action can also be associated to all or some of the annotations when the annotation is displayed on a captured image.
• These actions can be triggered by the user or executed automatically using for instance timers.
• Actions include launching a web browser with a specific URL, animating the annotations such as making one annotation move, appear or disappear, playing a video, launching a menu presenting further possible actions, launching a slide show or playing an audio file.
• Actions that allow the view of the plenoptic data presented to the user to be modified are also possible, for instance actions that allow to focus the view of the plenoptic data at a given focal length.
• the plenoptic annotation is stored and associated in a memory, for example in database 51 or in the user's device, with the corresponding position, orientation and with the selected reference plenoptic model. Knowing the annotations which are needed, it is possible to store in a plenoptic format the annotations attached to each reference plenoptic model. Each annotation is stored as a separate plenoptic file.
• the plenoptic annotated reference data is generated from the plenoptic reference data and the corresponding one or plurality of plenoptic annotations.
• This augmented reality model takes the form of a file containing all the information required to render back the plenoptic model with its associated annotations. It therefore describes the relations between the plenoptic reference data and its annotations.
• the plenoptic annotated reference data can be rendered directly on the plenoptic annotation authoring system to pre-visualize the results, but also on the client side to render some plenoptic augmented reality.
• the information describing the effect of the annotation on the model rays is stored in the plenoptic annotation data.
• the modification defined by the annotation acts on the model ray parameters.
• an annotation can describe for example a modification of the model light ray directions, positions, time or wavelength. In other words, this information describes a function of the model rays.
• each rays of the annotation are assigned a unique identifier.
• the annotation rays unique identifiers are matched to their corresponding rays of the model.
• each ray of the model is assigned a annotation ray identifier which is then used by the system when it has to apply ray by ray the annotation on the model, as it is for instance mainly the case in the online phase.
• the annotation information can be stored in a 2-dimensional array, where each ray contains the information about its effect on the model for each parameter.
• the unique identifier of the annotation rays is then used to define the corresponding ray effect in the array for each parameter.
• the first dimension of the array corresponds to the rays, referred by their identifier, and the second dimension to their parameters, i.e. light field parameters.
• Any annotation can be fully represented using this format as any modification of the model ray for any parameter can be represented in the array.
• an annotation can for instance modify all model rays direction by 10° for one angle.
• the 2-dimensional array then contains 10° in the column of the parameter corresponding to the direction angle.
• the column reads 10° for all rays as it is assumed they are all acting the same way.
• the system will first identify the annotation and model ray pairs, extract the unique identifier corresponding to the annotation ray, look into the annotation table to see what effect this annotation ray has in order to finally apply this change to the model ray.
• the angle of all model rays affected by the annotation will be rotated by 10°.
• the user may want to add a text annotation to a scene containing a building. Moreover, the text annotation color will need to vary from a viewpoint to another. The following step will then be done by the user:
• a plenoptic capture of the building is uploaded to the plenoptic
• a 2D view is rendered from the captured plenoptic image
• the user select the text annotation type from an annotation type list, enter his text and drag the text annotation onto the rendered 2D view 4.
• the user can move the viewpoint of the rendered 2D view or the annotation position and orientation so that the annotation appears exactly as the user wants.
• the plenoptic annotation authoring system performs the following tasks to generate the proper annotation model based on the previously described user action steps for the text annotation:
• a 2D view is rendered to the user based on the viewpoint setting initially set to a default value
• a plenoptic version of the text annotation is generated by tracing rays from the text object to the virtual viewpoint. This creates a set of light rays, each one described by a unique identifier. This set of rays describes the text. These light rays are represented in memory by an array corresponding to the modifications which have to be applied to the reference plenoptic image. In this case, the array will contain the value of the wavelength that the light rays which are matched with the annotation light rays have to take.
• the annotation is initially lying at a default position pre-defined in the authoring tool.
• the annotation light rays are combined with the reference plenoptic image rays. These relations between the rays of the reference image and the annotation are stored for future use by using the rays unique identifiers. 4.
• the different light rays of the annotation are combined with other light rays of the captured plenoptic image and a new 2D view is generated for each position or orientation modification. This is made possible as the user mouse pointer is project into the plenoptic space.
• the translation of the pointer is then applied to the annotation in the plane parallel to the virtual one corresponding to the 2D rendered view.
• annotation is moved, the relations of light rays between the reference image and the annotation are changed and updated according to the annotation position or orientation change.
• the wavelength value of the annotation array is changed to match the chosen color.
• the wavelength value of the annotation array corresponding to the light rays used to generate this new rendered view is changed.
• the wavelength value in-between the first viewpoint and the second viewpoint are interpolated using standard or ad-hoc interpolation methods.
• the plenoptic annotation array is saved with the uploaded plenoptic reference model so that it can be used in the online phase.
• annotation process happens when a user capturing a plenoptic image wants that image to be annotated.
• the online phase of the annotation process is applied to the input plenoptic image to get a final plenoptic annotated image.
• This consists of matching the input image with some reference models, retrieving the annotations of the matched reference model, combining the annotations with the input plenoptic image, rendering the annotated view to the user in an understandable form, and possibly treating user interactions in order to generate the different actions defined on the annotations.
• the annotation content composed of light rays is in a plenoptic format and the captured image is also in plenoptic format, those two data sets lie in the same space.
• the annotation can thus be applied directly to the plenoptic image without further projections needed.
• the modified plenoptic space where the annotations have been applied to can then be projected, for example, into a 2D view.
• projection parameters selected for the plenoptic rendering process such as selection of focus, depth, change of view point, ...) also implicitly apply on plenoptic annotations. For example, when changing the focus or viewpoint of the rendering process, the annotations will have the effects applied to them.
• the online plenoptic annotation process comprises a first step 100 during which data representing a light field in plenoptic format (plenoptic data) is retrieved.
• the plenoptic data might be retrieved by the device 4 capturing the data with a plenoptic capturing device, or retrieved by the apparatus, such as a server 5, that receives the plenoptic data from the device 4 over a communication link.
• step 101 the retrieved data is matched with reference data.
• This step might be performed in the device 4 and/or in the server 5. This step might involve determining a set of features in the captured data, finding a matching reference data representing a reference image with matching features, and registering the captured data with the reference data as described for example in US13645762.
• the reference data may represent images in plenoptic format, or other images, and might be stored in a memory 51 , such as a database, accessible from a plurality of devices. Identification of a matching reference data might be based on user's location, time, hour, signal received from elements of the scene, indication given by the user's and/or image similarities.
• the registration process aims at finding a geometrical relation between the user position and the reference data so that a transformation between the light rays of the captured plenoptic image and the ones from the matched plenoptic reference image can be deduced.
• a plenoptic annotation associated with the matching reference data is retrieved, for example from the memory 51.
• This annotation is in a plenoptic format, i.e. described with light rays.
• Those annotation light rays may represent for example a text, a still image, a video image, a logo, and/or other elements directly acting on plenoptic image light rays.
• the annotations might include sounds in the plenoptic space, e.g., sounds attached to a specific group of rays of the plenoptic reference image, so that the sound will be played only for some directions where the selected rays are also visible and/or in focus in the plenoptic image.
• step 103 the retrieved annotation in plenoptic format is combined with the captured plenoptic data to generate annotated data representing an annotated image in plenoptic format.
• This combination might be made in the server 5, or in the device 4. In the latter case, the server 5 might send the annotated data to the device 4, which then makes the combination.
• This annotation combination is made possible as the transformation projecting the light rays of the reference image to the captured plenoptic image is known from the matching step (Step 101). The annotation can therefore be also applied to the captured plenoptic image.
• the plenoptic annotations can be applied to the captured plenoptic image using the following method:
• the array is used as a lookup-table where the light rays, which can be identified thanks to the selection process of step (1 ) and (2), and the parameters of the transformation (such as wavelength, directions, ...) are used as lookup-keys.
• the annotation array will contain a single non-null light field parameter which is the wavelength corresponding to the text color.
• the captured plenoptic image light rays will thus be modified by increasing/decreasing the wavelength of the rays by a factor stored in the annotation array. This factor is looked-up in the array by using the transformation between light rays computed in the registration process.
• a view is rendered from the annotated data, for example a 2D or stereoscopic view, and presented, for example displayed on display 40 or with another apparatus, to the user/viewer.
• This view rendering process is described in more details below in conjunction with Fig. 9.
• the interaction with the annotation is made possible.
• the system is capable of reacting to different events in order to execute specific actions previously defined in the offline part of the annotation process.
• Such an event can be a user interaction with an annotation.
• the user is able to point and interact with a given annotation. This interaction would generate an interaction event which can trigger specific actions defined in the offline phase of the annotation process.
• Another possible type of event is the events triggered when a specific change in the scene is detected. As explained later in this section, an occlusion by an object of the reference model in the captured plenoptic image can be detected. This occlusion event can trigger an action
• a sound recognition module can be used in order to trigger certain actions based on certain types of detected sounds.
• Fig. 9 illustrates the rendering of a view, and various possibilities for the viewer to modify subsequently the rendering.
• an augmented reality view is rendered in step 104 from
• the rendered view may be a standard 2D view as produced by a pin-hole camera, a
• a dynamic image module could be an HTML5/Javascript web page able to render a plenoptic image as a function of the commands values or as a Flash object or any other technologies allowing a dynamic presentation of several images.
• FIGs. 5A, 6A, and 7A Examples of views that may be rendered during step 104 are shown on Figs. 5A, 6A, and 7A.
• the view on Figs. 5A and 6A includes an object 60 with annotation 61 .
• Refocusing or change of viewpoint can be triggered manually by the user (for example by selecting an object or position on or around the image), or automatically (for example when the user moves).
• step 105 the user enters a command for modifying the viewpoint, in order to produce during step 107 a novel view from the same plenoptic data, corresponding to the same scene observed from a different viewpoint.
• Algorithms for generating from plenoptic data various 2D images of a scene as seen from different viewpoints or viewing directions are known as such, and described for example in US6222937.
• An example of modified 2D image produced by this command and executed by a viewpoint selection module 403 is illustrated on Fig. 5B. As can be seen, not only the perspective of the object 60 has been modified by this command, but also the perspective of the annotation 61.
• Some annotations may be visible only from a first set of viewing directions, but not from other directions. Therefore, as illustrated with Figure 6B, a change of viewpoint during step 105 may result in a new view where one annotation 61 is made invisible but a new annotation 64 associated with the same object is revealed.
• a plurality of annotations may be associated with a single location of the reference image, but with different viewing directions.
• the annotation itself might also look differently when rendered from a first viewing direction compared to second different view direction due to the different annotation light field parameter set in the offline phase of the annotation process.
• the change of appearance can be defined by the annotation itself but it could also be a function of the input plenoptic image.
• step 106 of Fig. 9 the user enters a command for refocusing the image and for generating from the data in plenoptic format a new image focused at a different distance.
• This command might be executed by a refocusing module 402.
• a refocusing module 402. As can be seen on Figures 7A and 7B, this might result in a first annotation 61 visible at a first focus distance to disappear, or become less sharp, at the second focus distance shown on Fig. 7B, whereas a second annotation 63 only appears at this second focus distance.
• the different commands used in step 105 and 106 to change the rendered views can also be issued automatically with respect to user movements.
• the user movements are tracked with a Inertial Measurement Unit (IMU) embedded in the plenoptic capturing device.
• IMU Inertial Measurement Unit
• the rendered view is automatically updated as user moves. For example, when the user moves on the left, the viewing direction is slightly translated to the left.
• the same principle is applied when a user moves forward, where the focusing range is moved also forward, yielding to sharper objects in the background planes, and softer objects at the foreground planes, compared to the previously rendered view.
• the present invention is not restricted to the use of an IMU to track user movements. Other means such as using directly plenoptic image content to track user movements can also be used.
• the online plenoptic annotation process is continuously applied to a stream of plenoptic images produced by a plenoptic capturing device of a user in movement.
• This continuous processing allows a user to continuously move, or to move his plenoptic capturing device, and have the plenoptic annotations updated in real time.
• the stream of plenoptic images has to be processed in real-time as well as the rendering of the views (step 104 of Fig. 8) so that the users perceive the annotations as if they were part of the scene.
• the fact that the viewing direction can be modified afterwards without the need to have another plenoptic capture allows to accomplish the same effect with a much lower number of plenoptic images that needs to be processed from the stream.
• step 200, 201 , 202, 203 of Fig. 20 are similar or equivalent to steps 100, 101 , 102, 103 in Fig. 8.
• step 204 viewing directions parameters are computed as a result of the registration process of step 201 .
• step 205 a view is rendered based on the viewing direction computed in the previous step.
• step 206 the Inertial Measurement Unit (IMU) is used to determine the user movement with respect to the time the step 200 has been computed. A decision is then taken to either go back to step 200 for processing a new plenoptic image or going directly to step 204 to update the viewing direction parameters based on the IMU movement estimation. The amount of movement is used to determine whether or not the previously captured plenoptic data can be used to generate a novel view. This typically depends on the field of view of the plenoptic capturing device.
• IMU Inertial Measurement Unit
• the rendering of plenoptic annotations may consider possible occlusions.
• a plenoptic annotation may be occluded if the target element to annotate is hidden from the capturing device eyesight by another object lying in the input plenoptic image.
• the rendering module takes advantage of the plenoptic format of the captured data to visually hide the annotation behind the irrelevant object.
• the rendering module knows from the plenoptic reference data the properties of the captured rays that should come from each element of the captured plenoptic image. If the captured rays have different properties than the expected rays of the element, it could mean that an occluding object is in front of the element, and thus, that the annotation does not have to be displayed for this element.
• the rendering module could use this information to detect occlusions.
• color information of rays can also be used to determine whether a captured element is occluded or not. However, the color information is not sufficient as an occluding object might have the same color as the target element.
• a first example of application is the use of a plenoptic
• plenoptic images of objects/scenes could be captured by users with their plenoptic capturing device.
• the captured plenoptic image can then be annotated by the user using all sort of annotations, including plenoptic image previously captured and used as annotation.
• Their annotated scene can then be shared to users' friends using social networks so that those friends can experience the annotated scene when they are capturing it with their own plenoptic capturing device.
• the advantage of using the plenoptic annotation process in this case is leveraged in the fact that annotations are already lying in the plenoptic space as they are plenoptic images. Therefore doing the annotation process in the same plenoptic space is more computationally efficient and yields to a more realistic annotated scene.
• a second example of application which exploits the different information of the plenoptic space, is the use of specially designed plenoptic annotations in the field of architectural design.
• a plenoptic annotation is composed of light rays which are combined with the plenoptic image light rays in the online phase. The way this light rays are combined is defined in the offline part of the annotation process. This combination can be such that rays from the plenoptic image are not replaced by other light rays from the annotation, but, for example, only their direction is changed.
• plenoptic annotations can be
• simulation of weather conditions can be applied to the captured plenoptic image.
• An annotation simulating rain can be applied to the scene. This will yields to an annotated image with a rain effect applied to it so that user can visually see how the scene would like in case of rain or other different weather conditions, where the different light reflexions and refractions are properly handled and computed in a realistic way thanks to the plenoptic information.
• treasure hunt is a popular application in conventional two-dimensional augmented reality solutions. It consists in attaching annotations to physical objects and by giving hints to friends or other people, let them search for these annotations (called treasures).
• treasures when someone comes close to the hidden object, he can scan the surrounding objects with his plenoptic capturing device to determine whether they are associated with an annotation.
• plenoptic annotations the treasure hunt becomes more exciting since we can limit the annotation visibility to some viewing directions or focus distances. For instance, a user can attach an annotation to a statue, and decide to make this annotation only visible when a future hunter is placed in front of the statue and therefore that he sees it from that angle.
• Another application concerns a city guide in an urban
• the plenoptic annotations could be made dependent on the user point of view and focus. For instance, elements of an image captured by the user with a particular view angle (or in a particular range of view angle) could be displayed with a lower importance than elements which are faced by the user. In one
• low importance annotations can only be displayed as titles or points on the screen (which can be extended when the user clicks on them), while more important interest points present more details or have a larger size or emphasis on the image.
• any suitable means capable of performing the operations such as various hardware and/or software component(s), circuits, and/or module(s).
• any operations described in the application may be performed by corresponding functional means capable of performing the operations.
• the various means, logical blocks, and modules may include various hardware and/or software component(s) and/or module(s), including, but not limited to a circuit, an application specific integrated circuit (ASIC), or a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field
• FPGA programmable gate array signal
• PLD programmable logic device
• a general purpose processor may be a microprocessor, but in the alternative, the processor may be any commercially available processor, controller, microcontroller or state machine.
• a processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
• a server may be implemented as a single machine, as a set of machine, as a virtual server, or as a cloud server.
• the expression "plenoptic data” designates any data generated with a plenoptic capturing device, or computed from other types of data, and describing a light field image of a scene, i.e., an image where not only the brightness and colour of the light is stored, but also the direction of this light.
• a 2D or stereographic projection rendered from such a plenoptic data is not considered to be a plenoptic image, since this direction of light is lost.
• the expression "plenoptic space” may designate a multi-dimensional space with which a light field, i.e., a function that describes the amount of light in every direction in space, can be described.
• a plenoptic space may be described by at least two parameters for the position of the ray two for its orientation and one for its wavelength and possibly one parameter for the time (in case of video).
• the term “annotation” encompasses a wide variety of possible elements, including for example text, still images, video images, logos, sounds and/or other elements that could be superimposed or otherwise merged into the plenoptic space represented by plenoptic data.
• annotation encompasses the different way to alter the different parameters of the plenoptic space light rays represented by the plenoptic data.
• Annotations may be dynamic and change their position and/or appearance over time.
• annotations may be user interactive and react to a user's operations (e.g. move or transform upon user interaction).
• pixel may designate one single monochrome photosite, or a plurality of adjacent photosites for detecting light in different colours. For example, three adjacent photosites for detecting red, green and blue light could form a single pixel.
• determining encompasses a wide variety of actions. For example, “determining” may include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, a database or another data structure), ascertaining, estimating and the like. Also, “determining” may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory) and the like. Also, “determining” may include resolving, selecting, choosing, establishing and the like.
• Capturing an image of a scene involves using a digital pin-hole camera for measuring the brightness of light that reaches the image sensor of the camera.
• Capturing plenoptic data may involve using a plenoptic capturing device, or may involve generating the light field data from a virtual 3D model or other description of the scene and light sources.
• Retrieving an image may involve capturing the image, or retrieving the image over a communication link from a different device.
• rendering a view for example “rendering a 2D view from plenoptic data” encompasses the action of computing or generating an image, for example computing a 2D image or an
• a software module may reside in any form of storage medium that is known in the art. Some examples of storage media that may be used include random access memory (RAM), read only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM and so forth.
• RAM random access memory
• ROM read only memory
• flash memory EPROM memory
• EEPROM memory EEPROM memory
• registers a hard disk, a removable disk, a CD-ROM and so forth.
• a software module may comprise a single instruction, or many instructions, and may be distributed over several different code segments, among different programs, and across multiple storage media.
• a software module may consist of an executable program, a portion or routine or library used in a complete program, a plurality of interconnected programs, an "apps" executed by many smartphones, tablets or computers, a widget, a Flash application, a portion of HTML code, etc.
• a storage medium may be coupled to a processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor.
• a database may be implemented as any structured collection of data, including a SQL database, a set of XML documents, a semantical database, or set of information available over an IP network, or any other suitable structure.
• certain aspects may comprise a computer program product for performing the operations presented herein.
• a computer program product may comprise a computer readable medium having instructions stored (and/or encoded) thereon, the instructions being executable by one or more processors to perform the operations described herein.
• the computer program product may include packaging material.

Landscapes

• Engineering & Computer Science (AREA)
• Computer Graphics (AREA)
• Computer Hardware Design (AREA)
• General Engineering & Computer Science (AREA)
• Software Systems (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Processing Or Creating Images (AREA)
• User Interface Of Digital Computer (AREA)

Applications Claiming Priority (1)

Publications (1)

Family

ID=47553021

Family Applications (1)

Country Status (5)

Families Citing this family (9)

Family Cites Families (6)

• 2012
  • 2012-12-21 WO PCT/EP2012/076643 patent/WO2014094874A1/en active Application Filing
  • 2012-12-21 KR KR1020157016284A patent/KR20150106879A/ko not_active Application Discontinuation
  • 2012-12-21 CN CN201280077894.3A patent/CN104969264A/zh active Pending
  • 2012-12-21 JP JP2015548228A patent/JP2016511850A/ja active Pending
  • 2012-12-21 EP EP12813361.8A patent/EP2936442A1/en not_active Withdrawn

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events