WO2011105048A1 - コンピュータ・グラフィックス映像合成装置と方法、及び表示装置 - Google Patents
コンピュータ・グラフィックス映像合成装置と方法、及び表示装置 Download PDFInfo
- Publication number
- WO2011105048A1 WO2011105048A1 PCT/JP2011/000978 JP2011000978W WO2011105048A1 WO 2011105048 A1 WO2011105048 A1 WO 2011105048A1 JP 2011000978 W JP2011000978 W JP 2011000978W WO 2011105048 A1 WO2011105048 A1 WO 2011105048A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- video
- data
- eye
- mapping
- stereoscopic
- Prior art date
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/275—Image signal generators from 3D object models, e.g. computer-generated stereoscopic image signals
Definitions
- the present invention relates to a technique for synthesizing and displaying a stereoscopic video image on a computer graphics (CG) video image.
- CG computer graphics
- Recent video game machines and digital televisions have a viewer function and a slide show function. By using these functions, it is possible to display a photograph / moving picture taken by a user with a digital camera on a display screen.
- various visual effects are added to photos / moving pictures by combining with CG images.
- thumbnail images of a plurality of photos and a plurality of moving images are displayed as a list on the screen, and the user is allowed to select any one of the photos / moving images using a graphics user interface (GUI).
- GUI graphics user interface
- the switching from the list display screen to the specific photo / moving image display screen accompanying the selection operation is expressed by animation using CG video.
- by synthesizing the photograph / moving image with the CG image it is possible to give entertainment to the user's operation and improve the operability of the GUI.
- a CG image called “object” is expressed by a collection of figures such as points, lines, and polygons (polygons). Those figures may be expressed as being drawn on a two-dimensional (2D) screen or may be expressed as being arranged in a three-dimensional (3D) virtual space.
- a two-dimensional image (texture) representing one frame of a photograph / moving image is projected onto the surface of the object. This operation is called “mapping”.
- the object is placed in the 3D virtual space, it is further projected on a “planar view image” when the object is viewed from a specific viewpoint, that is, a single plane placed between the object and the viewpoint. The image of the object is calculated.
- the planar view image obtained by combining one frame of the photograph / moving image with the object is displayed on the screen of the display.
- the photograph / moving image reflected on the surface of the object is similarly deformed or displaced accordingly.
- the “stereoscopic video” refers to a combination of a left-eye video and a right-eye video representing one scene, which is viewed by a viewer as a stereoscopic video of the scene.
- the “left-eye video (left view)” refers to a planar video image that is viewed by the viewer's left eye
- the “right-eye video (right view)” refers to a planar video image that is displayed to the viewer's right eye.
- There is a difference between the left-eye video and the right-eye video due to the binocular parallax of the viewer.
- a “frame sequential method” is generally used.
- a left-eye image and a right-eye image are alternately displayed on the display screen.
- the viewer views the screen through shutter glasses.
- the shutter glasses make only the left lens transparent during the period when the image for the left eye is displayed on the screen, and make only the right lens transparent during the period when the image for the right eye is displayed.
- only the video for the left eye appears in the left eye of the viewer, and only the video for the right eye appears in the right eye.
- the difference between the left-eye video and the right-eye video is perceived by the viewer as a binocular parallax of one scene, so that the two planar video images appear as one stereoscopic video image. .
- two digital cameras may be placed side by side as if they are viewed by the viewer's eyes, and they can be simultaneously shot.
- a technique for converting a CG image into a stereoscopic image is also known.
- the left eye video and the right eye video of the object are created by shifting the display position of the object on the 2D screen to the left and right.
- Such a technique for expressing a photograph / moving image and a CG image as a stereoscopic image is desired to be applied to the viewer function and the slide show function.
- the use of stereoscopic video images is expected to provide a sense of reality that cannot be expressed with planar video images, and open up new possibilities for visual effects.
- a planar video when viewing the object from the viewpoint corresponding to the left eye of the viewer that is, a CG video for the left eye
- a planar video when viewed from another viewpoint corresponding to the right eye that is, the right eye.
- Create a CG video for use Next, the left-eye CG video is mapped to the left-eye CG video, and the right-eye photo / video is mapped to the right-eye CG video.
- the left eye and right eye photos / videos are shot so that the subject looks the most three-dimensional when viewed from the direction of the camera. Therefore, if the surface of the object is flat like a plate and tilted nearly perpendicular to the viewing direction, the stereoscopic image mapped on the surface will rise or sink from the surface. It looks in. However, if the surface of the object is undulating or curved with a relatively high curvature, or if the surface is greatly inclined with respect to the viewing direction, the stereoscopic image mapped on the surface is distorted, so It is difficult to see.
- the object is represented by a planar shape projected on the 2D screen instead of a three-dimensional shape in the 3D virtual space, if the object moves at a relatively high speed, it is mapped to the surface. Stereoscopic images are difficult to see stereoscopically.
- the amount of calculation required for the mapping process is generally large. Therefore, in order to further improve the image quality of the CG image synthesized with the photo / video, the load necessary for the mapping process is reduced as much as possible under the condition that the photo / video is difficult to see stereoscopically, It is desirable to further smooth the process.
- An object of the present invention is to solve the above-described problems, and in particular, to provide a CG video composition device that can further improve the quality of a composite video by adjusting the mapping process according to the shape and movement of an object. There is to do.
- the CG video composition device is a device that maps a video to an object and composes it into a single CG video, and includes a memory unit, a determination unit, and a mapping unit.
- the memory unit stores a pair of shape data and video data.
- the shape data represents the shape of the object.
- the video data pair includes left-eye and right-eye video data representing one stereoscopic video.
- the determination unit determines whether the object is suitable for mapping of the stereoscopic video from the shape data.
- the mapping unit generates left-eye CG data by combining the left-eye video data with the shape data, and the right-eye video data. Is combined with shape data to create CG data for the right eye.
- the mapping unit further synthesizes only the left-eye video data or the right-eye video data into the shape data when the determination unit determines that the object is not suitable for the mapping of the stereoscopic video.
- CG data and CG data for the right eye are created.
- the determination unit may determine whether the object is suitable for stereoscopic video mapping as follows. The determination unit obtains the representative point of the object from the 2D data for each frame of the CG video, and when the displacement of the representative point of the object is less than a predetermined threshold between two consecutive frames, the object is stereoscopically viewed. If it is determined that the object is suitable for video mapping and the magnitude of the displacement exceeds the threshold, it is determined that the object is not suitable for mapping of the stereoscopic video.
- the determination unit determines whether the object is suitable for stereoscopic video mapping by any of the following three methods. May be.
- the determination unit first calculates the normal vector of each polygon constituting the object from the 3D data, and evaluates the curvature of the surface of the object from the set of calculated normal vectors. Next, when the curvature is less than a predetermined threshold, the determination unit determines that the object is suitable for stereoscopic video mapping, and when the curvature exceeds the threshold, the object is suitable for stereoscopic video mapping. Judge that there is no.
- the determination unit first calculates the illuminance at the apex of each polygon constituting the object from the 3D data. The determination unit then determines that the object is suitable for stereoscopic video mapping when the difference between the maximum value and the minimum value of the calculated overall illuminance is less than a predetermined threshold value, and the difference is a threshold value. If it exceeds, it is determined that the object is not suitable for stereoscopic video mapping.
- the determination unit first obtains a representative point of the object from the vertices of each polygon constituting the object for each frame of the CG video. The determination unit then determines that the object is suitable for stereoscopic video mapping if the displacement of the representative point of the object between two consecutive frames is less than a predetermined threshold, and the displacement magnitude Is greater than the threshold value, it is determined that the object is not suitable for stereoscopic video mapping.
- the CG video composition apparatus determines from the shape data of an object whether or not the object is suitable for mapping of a stereoscopic video. Thereby, the CG video composition device can adjust the mapping process in accordance with the shape and movement of the object, so that the quality of the composite video can be further improved.
- FIG. 2 is a block diagram showing a hardware configuration of a CG video composition unit 160 shown in FIG. 1.
- 6 is a flowchart of processing for creating a frame for a right eye and a left eye by mapping a stereoscopic video / moving image to a CG video of an object by the display device according to the first embodiment of the present invention.
- (A) to (c) are schematic diagrams showing different representation formats of a collection of polygons constituting the shape of an object. It is a flowchart of the process which step S11 shown by FIG. 1, ie, the display apparatus 10 by Embodiment 1 of this invention produces the flame
- FIG. (A) is a schematic diagram which shows a normal vector in case the aggregate
- B) is a schematic diagram showing a normal vector in a case where an aggregate of polygons is expressed by the triangular fan shown in (b) of FIG.
- C) is a schematic diagram showing a normal vector in a case where an aggregate of polygons is expressed by the triangular mesh shown in (c) of FIG.
- surface which shows an example of a polygon buffer. 4 is a flowchart of step S12 shown in FIG.
- step S27 shown in FIG. 5 is a flowchart of step S27 shown in FIG. 5 and step S45 shown in FIG.
- (A) is a schematic diagram showing a CG image of an object POB having a flat surface.
- (B) represents the left-eye CG image POL of the object POB, and (c) represents the right-eye CG image POR.
- (D) represents a left-eye planar image constituting one frame of a stereoscopic video image / moving image to be mapped on the surface of the object POB, and (e) represents a right-eye planar view of the same frame. Represents a video.
- (F) shows a composite video of the left-eye CG video POL of the object and the “tree” left-eye video LV
- (g) shows the right-eye CG video POR of the object and the “tree” right-eye video.
- (H) is a schematic diagram showing a stereoscopic video image that can be seen by a viewer when each synthesized video image shown in (f) and (g) is viewed by each viewer's eye.
- (A) is a schematic diagram showing a CG image of an object COB having a curved surface.
- (B) represents the left-eye CG image COL for the object COB
- (c) represents the right-eye CG image COR.
- (D) represents a planar image for the left eye that constitutes one frame of a photograph / moving picture of a stereoscopic image to be mapped on the surface of the object COB, and (e) is a planar view for the right eye of the same frame.
- (F) represents a composite image of the left-eye CG image COL of the object and the “tree” right-eye image RV, and (g) represents the right-eye CG image COR of the object and the “tree” right-eye image.
- FIG. (H) is a schematic diagram showing a stereoscopic video image that can be seen by a viewer when each synthesized video image shown in (f) and (g) is viewed by each viewer's eye. It is a flowchart at the time of the display apparatus by Embodiment 2 of this invention performing step S25 shown by FIG.
- (A) is a schematic diagram which shows the normal vector of the polygon which belongs to the triangular mesh which comprises a part of surface of an object, and the normal vector of a vertex.
- (B) is a schematic diagram showing a CG image of a flat object.
- FIG. (C) shows stereoscopic CG images BO1, BO2, and BO3 of three flat objects, and stereoscopic images VST, VTR, and VCR of “star”, “tree”, and “circle” mapped to each of them. It is a schematic diagram which shows. It is a flowchart at the time of the display apparatus by Embodiment 3 of this invention performing step S25 shown by FIG. (A) is a schematic diagram showing a stereoscopic CG image PL1 of a flat object and a “tree” stereoscopic image TR1 mapped thereon.
- FIG. (B) is a schematic diagram showing a stereoscopic CG video PL2 of an object and a stereoscopic video TR2 of “tree” in a frame subsequent to the frame shown in (a).
- (C) is a schematic diagram showing an object stereoscopic CG video PL3 and a “tree” stereoscopic video TR3 several frames after the frame shown in (b). It is a flowchart at the time of the display apparatus by Embodiment 4 of this invention performing step S11 shown by FIG. It is a flowchart of step S84 shown by FIG.
- (A) is a schematic diagram showing a planar view CG image MV1 of a rectangular object and a “tree” stereoscopic image ST1 mapped thereon.
- FIG. 10 is a flowchart of processing for creating a right-eye frame and a left-eye frame by mapping a stereoscopic video image / moving image to a CG video image of an object when the display device displays a plurality of objects in one frame.
- shape data of an object is 2D data, it is a flowchart of a process in which the display device maps a photograph / moving image as a planar video to the planar CG video of the object.
- FIG. 1 is a functional block diagram of a display device according to Embodiment 1 of the present invention.
- This display device is a digital television receiver.
- the display device 10 may be incorporated in various electronic devices such as a mobile phone, a portable information terminal, a personal computer, a digital still camera, a digital video camera, and a car navigation system.
- the display device 10 includes a video data input unit, an operation unit 140, a system bus 150, a CG video synthesis unit 160, a display unit 170, and a left / right signal transmission unit 180.
- the video data input unit includes a receiving unit 100, an optical disk drive 110, a card reader 120, and a hard disk drive (HDD) 130.
- HDD hard disk drive
- the receiving unit 100 receives a digital terrestrial television broadcast wave through an antenna 190 or an external network 191 such as a cable TV.
- the receiving unit 100 further extracts video data of a desired channel from the broadcast wave and sends it to the CG video synthesizing unit 160.
- the optical disc drive 110 is built in the display device 10, reads video data from an optical disc 192 such as a DVD or a BD (Blu-ray Disc, Blu-ray Disc (registered trademark)), and sends it to the CG video composition unit 160.
- the optical disk drive 110 may be a single device externally attached to the display device 10 or a device built in the optical disk playback device.
- the card reader 120 reads video data from a semiconductor memory card 193 such as an SD card and sends it to the CG video composition unit 160.
- the HDD 130 is built in the display device 10 and stores video data transmitted from the receiving unit 100, the optical disc drive 110, or the card reader 120. Also, the HDD 130 sends the stored video data to the CG video composition unit 160.
- the operation unit 140 detects a user operation through the remote controller 194 or the front panel of the display device 10, and requests the CG video composition unit 160 to perform processing corresponding to the type of the operation.
- the remote controller 105 includes a plurality of buttons. Each button is associated with each function of the display device 10 such as turning on / off the power of the display device 10, tuning, or adjusting the volume.
- the remote controller 194 detects the pressing of each button by the user, and transmits the identification information of the button to the display device 10 by infrared or wireless signal IR.
- the operation unit 140 identifies the function associated with the button indicated by the identification information from the signal IR and notifies the CG video composition unit 160 of the function.
- the system bus 150 connects the video data input unit 100-130 and the operation unit 140 to the CG video composition unit 160 so that data can be exchanged.
- the system bus 150 conforms to the PCI-Express standard.
- the system bus 150 may conform to the PCI standard or the AGP standard.
- the system bus 150 includes a chipset.
- the chip set converts video data transmitted from the input units 100-130 into a format that can be processed by the CG video compositing unit 160.
- the CG video composition unit 160 includes a control unit 161 and a rendering unit 162.
- the control unit 161 and the rendering unit 162 are each mounted on one integrated circuit.
- Each integrated circuit may be configured as a dedicated circuit or may be configured using a general-purpose processor.
- the control unit 161 and the rendering unit 162 may be integrated into one LSI.
- the control unit 161 controls each functional unit of the display device 10 according to a program such as firmware.
- the control unit 161 controls the rendering unit 162 to map the video represented by the video data received from the input units 100-130 to the CG video of the object.
- the control unit 161 includes a determination unit 161A and a memory unit 161B.
- the determination unit 161A determines whether or not the object is suitable for mapping of the stereoscopic video from the shape data of the object.
- “Shape data” refers to data representing the shape of an object.
- the memory unit 161B is a memory device built in the control unit 161, stores a program such as firmware, and is used as a working memory area.
- the memory unit 161B stores object shape data, acquires video data to be mapped from the input units 100-130, and temporarily holds the video data.
- a pair of video data for left eye and right eye representing the stereoscopic video is held by the memory unit 161B.
- the rendering unit 162 performs processing (bitBLT) to transfer video data from the memory unit 162A to the frame buffer 162B according to the control unit 161, and performs various processing such as scaling and visual effects by CG on the video data.
- the rendering unit 162 maps the video represented by the video data to the CG video of the object.
- the rendering unit 162 includes a mapping unit 162A, a frame buffer 162B, and an output unit 162C.
- the mapping unit 162A combines the video data for the left eye or the right eye held in the memory unit 161B with the shape data of the object according to the determination result by the determination unit 161A.
- the mapping unit 162A when the determination unit 161A determines that the object is suitable for stereoscopic video mapping, the mapping unit 162A generates left-eye CG data by combining the left-eye video data with the shape data. At the same time, the right-eye CG data is created by combining the right-eye video data with the shape data. On the other hand, when the determination unit 161A determines that the object is not suitable for stereoscopic video mapping, the left eye CG data and the right eye CG data are generated by combining only the right eye video data with the shape data. . Details thereof will be described later.
- the frame buffer 162B is an area of a memory element built in the rendering unit 162, and holds CG data synthesized by the mapping unit 162A frame by frame.
- the output unit 162C reads the frame from the frame buffer 162B, converts it into an NTSC, PAL, or SECAM output format and sends it to the display unit 170.
- the output unit 162C also performs various processes on the video data.
- the types of processing include scaling, IP conversion, noise reduction, and frame rate conversion.
- Scaling is a process for enlarging / reducing the size of an image.
- IP conversion is a process of converting a scanning method between a progressive method and an interlace method.
- Noise reduction is a process for removing noise from an image.
- Frame rate conversion is a process of converting the frame rate.
- the display unit 170 causes the display panel 172 to display the frame created by the CG video composition unit 160.
- the display unit 170 alternately displays a left-eye CG image represented by the left-eye CG data and a right-eye CG image represented by the right-eye CG data when the image data represents a stereoscopic image.
- the display unit 170 includes a display driving unit 171 and a display panel 172.
- the display driving unit 171 controls the display panel 172 according to a signal from the output unit 162C.
- the display panel 172 is a liquid crystal display panel (LCD).
- the display panel 172 may be of other types such as a plasma display panel or an organic EL display panel.
- the left / right signal transmission unit 180 transmits the left / right signal LR to the shutter glasses 195 by infrared rays or wirelessly.
- the left / right signal LR indicates whether the video currently displayed on the screen of the display panel 172 is a frame for the left eye or for the right eye.
- the control unit 161 detects frame switching by identifying a left-eye frame and a right-eye frame from a control signal such as a synchronization signal accompanying the video data and auxiliary data.
- the control unit 161 further causes the left / right signal transmission unit 180 to change the left / right signal LR in synchronization with the detected frame switching.
- the shutter glasses 195 include two liquid crystal display panels 195L and 195R and a left / right signal receiving unit.
- the liquid crystal display panels 195L and 195R constitute left and right lens portions.
- the left / right signal receiving unit receives the left / right signal LR, and causes the left and right liquid crystal display panels 195L and 195R to uniformly transmit or block light as a whole according to the change.
- the left / right signal LR indicates the display of the left-eye frame
- the left-eye liquid crystal display panel 195L transmits light
- the right-eye liquid crystal display panel 195R blocks light.
- the left / right signal LR indicates the display of the right eye frame.
- the two liquid crystal display panels 195L and 195R alternately transmit light in synchronization with frame switching.
- the left-eye frame appears only in the viewer's left eye
- the right-eye frame appears only in the right eye.
- the viewer perceives the difference between the images shown in each eye as binocular parallax with respect to the same stereoscopic object, so that the video looks stereoscopic.
- FIG. 2 is a block diagram illustrating a hardware configuration of the CG video composition unit 160.
- the control unit 161 includes a first internal bus 210, a first input / output interface (I / O) 211, a CPU 212, and a main memory 213.
- the rendering unit 162 includes a second internal bus 220, a second I / O 221, a graphics processing unit (GPU) 222, and a video memory (VRAM) 223.
- the first internal bus 210 connects the functional units 211, 212, and 213 of the control unit 161 so that data can be exchanged.
- the first I / O 211 is a bus bridge that connects the first internal bus 210 to the system bus 150.
- I / O 211 is compliant with the PCI-Express standard.
- the first I / O 211 may conform to the PCI standard or the AGP standard.
- the CPU 212 executes a program stored in the main memory 213, gives video data to be processed to the rendering unit 162 according to the program, and controls the operation of each functional unit of the rendering unit 162.
- the CPU 212 causes the rendering unit 162 to transfer video data from the main memory 213 to the VRAM 223.
- the CPU 212 further functions as the determination unit 161A shown in FIG.
- the main memory 213 is a synchronous DRAM (SDRAM), preferably a DDR (Double-Data-Rate) -SDRAM.
- SDRAM synchronous DRAM
- DDR Double-Data-Rate
- the main memory 213 is used as the memory unit 161B shown in FIG.
- the second internal bus 220 connects the functional units 221, 222, 162 C, and 223 of the rendering unit 162 so that data can be exchanged.
- the second I / O 221 is a bus bridge that connects the second internal bus 220 to the system bus 150.
- the second I / O 221 complies with the PCI-Express standard.
- the second I / O 221 may be compliant with the PCI standard or the AGP standard.
- the GPU 222 is a logic circuit such as a chip specialized for arithmetic processing necessary for graphics display.
- CG processing performed by the GPU 222 includes geometry processing and rendering processing.
- the geometry processing the arrangement when each object arranged in the 3D virtual space is projected onto the 2D screen is determined by geometric calculation, particularly coordinate transformation.
- the rendering process video data representing a video to be actually displayed on the screen of the display panel 172 is created based on the arrangement of each object on the 2D screen determined by the geometry process.
- the rendering process includes hidden surface removal, shading, shadowing, texture mapping, and the like.
- the GPU 222 functions as the mapping unit 162A shown in FIG.
- the GPU 222 includes a vertex shader 222A and a pixel shader 222B.
- the vertex shader 222A is an arithmetic unit dedicated to geometry processing, and is used for geometric calculations necessary for geometry processing, particularly calculations related to coordinate transformation.
- the vertex shader 222A may be a calculator provided for each type of geometric calculation, or may be a calculator capable of processing various geometric calculations by a program.
- the pixel shader 222B is a computing unit dedicated to rendering processing, and is used for computation related to color information of each pixel necessary for rendering processing, that is, pixel data processing.
- the pixel shader 222B reads video data from the VRAM 223 in units of pixels, and calculates the sum and product between the components in units of pixels.
- the pixel shader 222B may be an arithmetic unit prepared for each type of calculation related to pixel data processing, or an arithmetic unit capable of processing various calculations in units of pixels by a program. Furthermore, one programmable calculator may be used as both the vertex shader 222A and the pixel shader 222B depending on the program.
- the VRAM 223 is an SDRAM built in the rendering unit 162, and is preferably a DDR-SDRAM or a GDDR (Graphic-DDR) -SDRAM.
- VRAM 223 specifically includes a frame buffer 162B shown in FIG. Each memory cell of the VRAM 223 included in the frame buffer 162B is associated with one pixel of the display panel 172, and pixel data for the pixel is stored.
- the display device 10 provides a stereoscopic effect to the viewer function or the slide show function using the elements shown in FIGS. Specifically, the display device 10 acquires a stereoscopic video image / video through the input units 100-130, maps the photo / video to the CG video of the object using the CG video synthesis unit 160, and displays the display panel. Display on 172 screen. At that time, as described below, the display device 10 according to the first embodiment switches the video represented by the mapping target photograph / moving image between the stereoscopic video and the planar video according to the shape of the mapping destination object. .
- FIG. 3 is a flowchart of processing in which the display device 10 creates a frame for the right eye and for the left eye by mapping a stereoscopic video image / moving image to the CG image of the object.
- the display device 10 starts this process when the user is instructed to execute the viewer function or the slide show function through the operation unit 140.
- step S11 the display apparatus 10 creates a frame for the right eye from the stereoscopic video image / moving image acquired through the input unit 100-130.
- the display device 10 creates a frame for the left eye from those photos / moving pictures.
- the frame data created in steps S11 and S12 is sent to the display unit 170.
- the display unit 170 displays these frames alternately on the screen of the display panel 172.
- the control unit 161 causes the left / right signal transmission unit 180 to switch the state of the left / right signal LR.
- the display device 10 checks whether or not the next frame exists in those photos / moving pictures. If there is a next frame, the process is repeated from step S11. If there is no next frame, the process ends.
- the types of shape data include 2D data and 3D data.
- the 2D data represents the shape of the object on the 2D screen.
- the 3D data represents the shape of the object in the 3D virtual space.
- the shape data represents the shape of the object as a collection of triangles called “polygons”.
- each vertex Ve [m] are represented by 2D vectors (x, y) in 2D data and 3D vectors (x, y, z) in 3D data.
- the coordinates of each vertex Ve [m] may be represented by four-dimensional homogeneous coordinates (x, y, z, 1).
- the seven polygons P0, P1, P2,..., P6 shown in (a) of FIG. 4 are represented in a first expression called “triangle strip”.
- the triangular strips P0 to P7 are connected so that two adjacent polygons share one side.
- the first polygon P0 and the second polygon P1 share the side L0 connecting the second vertex Ve [1] and the third vertex Ve [2]
- the second polygon P1 and the third polygon P2 Share the side L1 connecting the third vertex Ve [2] and the fourth vertex Ve [3].
- the four polygons P0, P1, P2, and P3 shown in FIG. 4B are expressed in a second expression format called “triangular fan”.
- the triangular fans P0 to P3 are connected so that a plurality of polygons share one vertex Ve [0].
- the shape data defines the triangle strip and the triangle fan as follows. First, the first polygon P0 is defined by three vertices Ve [0], Ve [1], and Ve [2]. Next, the other polygons P1, P2,... Are defined by one vertex different from the already-existing vertices. For example, the second polygon P1 is defined by the fourth vertex Ve [3], and the third polygon P2 is defined by the fifth vertex Ve [4].
- the seven polygons P0, P1, P2,..., P6 shown in (c) of FIG. 4 are represented in a third expression format called “triangular mesh”.
- a triangular mesh is defined by a set of indices (p, q, r) of three vertices Ve [p], Ve [q], Ve [r] for each polygon.
- the first polygon P0 is defined by the index set (0, 1, 2) of the first vertex Ve [0], the second vertex Ve [1], and the third vertex Ve [2].
- the 2-polygon P1 is defined by an index set (0, 2, 3) of the first vertex Ve [0], the third vertex Ve [2], and the fourth vertex Ve [3].
- Shape data further includes attribute values given to each vertex.
- the attribute value includes a normal vector and texture coordinates.
- the normal vector is an average of the normal vectors of polygons sharing the vertex. This normal vector is used to calculate the illuminance of the polygon.
- the texture coordinates represent 2D coordinates of a portion of the planar view image mapped to the polygon, that is, the portion mapped to the vertex in the texture.
- the shape data may include a vertex color value instead of the normal vector.
- the vertex color value represents the color coordinate value of each vertex.
- the shape data may further include an arbitrary attribute processed by the vertex shader 222A.
- FIG. 5 is a flowchart of step S11 shown in FIG. 1, that is, a process in which the display device 10 creates a frame for the right eye.
- step S21 the control unit 161 prepares the shape data of each object in the memory unit 161B. Thereafter, the process proceeds to step S22.
- the control unit 161 obtains video data for the right eye and camera parameters for the right eye from the video data received through the input units 100-130.
- the camera parameter is a parameter included in the auxiliary data added to the video data, and represents the position, orientation, and angle of view of the camera when shooting the video data.
- the camera parameters specifically include a coordinate transformation matrix.
- the coordinate transformation matrix defines the viewer's viewpoint and viewing direction, and is used for visual field coordinate transformation.
- View coordinate conversion refers to conversion from the global coordinate system to the view coordinate system.
- Global coordinate system refers to an orthogonal coordinate system defined in the 3D virtual space.
- the “field-of-view coordinate system” refers to an orthogonal coordinate system including two axes orthogonal to each other on a 2D screen that projects an object in the 3D virtual space and an axis perpendicular to the screen.
- the coordinate transformation matrix represented by the camera parameters is represented by a 4 ⁇ 4 matrix.
- the stereoscopic video data is provided with a left-eye camera parameter that defines the viewer's left-eye viewpoint and a right-eye camera parameter that defines the right-eye viewpoint.
- the control unit 161 extracts right-eye camera parameters from the video data and stores them in the memory unit 161B. On the other hand, the control unit 161 extracts video data for the right eye from the video data and sends it to the mapping unit 162A. Thereafter, the process proceeds to step S23.
- FIG. 6 is a schematic diagram showing visual field coordinate conversion.
- a global coordinate system (x, y, z) is defined in the 3D virtual space in which the object OBJ is arranged, and a visual field coordinate system (2D screen SCL for the left eye and 2D screen SCR for the right eye is respectively displayed.
- u, v, n are defined.
- Each screen SCL, SCR represents a screen that can be seen from the viewpoints VPL, VPR of each eye of the viewer.
- the n-axis perpendicular to each screen SCL, SCR is directed from each screen SCL, SCR to each viewpoint VPL, VPR.
- the global coordinates (x, y, z) of each vertex of the object OBJ are the visual field coordinates (u, v, n) of the vertex of the object OBL projected on the left-eye 2D screen SCL using the left-eye camera parameters. ) And converted to the visual field coordinates (u, v, n) of the vertex of the object OBR projected onto the right-eye 2D screen SCL using the right-eye camera parameters.
- the determination unit 161A refers to the shape data of the object prepared in the memory unit 161B, and determines whether or not the shape data is 2D data. If the shape data is 2D data, the process proceeds to step S24. If the shape data is not 2D data, the process proceeds to step S25.
- step S24 the determination unit 161A sets the texture mode to “left eye”.
- the texture mode is an environment variable that is referred to when creating the frame for the left eye in step S12.
- the image to be mapped to the CG image for the left eye of the object is a photo / image for either the left eye or the right eye. Specify whether it is a video. If the object has a 2D shape, the stereoscopic video mapped on the object looks right stereoscopic.
- step S24 since the shape data of the object is 2D data, the texture mode is set so that the left eye photograph / moving image is mapped to the CG image for the left eye of the object.
- the texture mode is stored in the memory unit 161B. Thereafter, the process proceeds to step S26.
- step S25 since the shape data is 3D data, the determination unit 161A further determines from the object shape data whether or not the shape of the object is suitable for stereoscopic video mapping. Determination unit 161A sets the texture mode according to the determination result. Details of the determination will be described later. After step S25, the process proceeds to step S26.
- step S ⁇ b> 26 the control unit 161 causes the rendering unit 162 to use the right-eye camera parameter to perform visual field coordinate conversion on the shape data of the object. Specifically, the control unit 161 causes the rendering unit 162 to first convert the vertex coordinate value represented by the shape data into a four-dimensional homogeneous coordinate value.
- the shape data is 2D data
- step S26 2D data is also converted into four-dimensional homogeneous coordinates, but this is not a limitation. If it is clear in advance that only two-dimensional coordinate transformation is performed on 2D data, the processing may be performed while the 2D data is expressed in two-dimensional coordinates.
- step S27 the control unit 161 causes the mapping unit 162A to use the shape data represented by the visual field coordinate values to map the right eye photograph / moving image to the CG image of the object. Details of the mapping process will be described later. In this way, the right eye photograph / moving image is combined with the CG image of the object to create a frame for the right eye.
- FIG. 7 is a flowchart of step S25 shown in FIG.
- the determination unit 161A determines whether or not the shape of the object is suitable for stereoscopic video mapping.
- the determination unit 161A uses the curvature of the surface of the object as follows to determine the suitability.
- step S32 the determination unit 161A causes the rendering unit 162 to evaluate the curvature of the surface of the object from the normal vector N [k] of each polygon Pk. The specific evaluation method will be described later. Thereafter, the process proceeds to step S33.
- step S33 the determination unit 161A compares the curvature value evaluated in step S32 with a predetermined threshold value, and determines whether or not the value is equal to or greater than the threshold value.
- the threshold value represents a curvature value when the stereoscopic image mapped on the surface becomes invisible in three dimensions when the curvature of the surface of the object is gradually increased from zero.
- the threshold value is stored in advance in the memory unit 161B by the control unit 161. If the curvature is equal to or greater than the threshold value, the process proceeds to step S34. If the curvature is less than the threshold value, the process proceeds to step S35.
- step S34 the determination unit 161A sets the texture mode to “for right eye”. Since the curvature of the surface of the object is equal to or greater than the threshold value, the stereoscopic video mapped on the surface cannot be viewed stereoscopically. Therefore, the texture mode is set so that the same right eye photograph / moving image as the right eye CG image is mapped to the left eye CG image of the object. The texture mode is stored in the memory unit 161B. Thereafter, the process proceeds to step S26 shown in FIG.
- step S35 the determination unit 161A sets the texture mode to “left eye”. Since the curvature of the surface of the object is less than the threshold value, the stereoscopic image mapped on the surface looks right stereoscopic. Accordingly, the texture mode is set so that the left eye photograph / moving image is mapped to the left eye CG image of the object. The texture mode is stored in the memory unit 161B. Thereafter, the process proceeds to step S26 shown in FIG.
- step S31 shown in FIG. 7 the normal vector of each polygon is calculated by one of the following two methods.
- the normal vector of each polygon is calculated from the coordinate value of each vertex.
- N [0] Normalize (v0 ⁇ v1).
- the operator “ ⁇ ” represents a vector product.
- Nh [i] Mv- T * Ne [i] .
- Nh [i] Mv- T * Ne [i] .
- N [0] Normalize ⁇ (Nh [0] + Nh [1] + Nh [2]) / 3 ⁇ .
- the operator “+” represents a vector sum
- the operator “/” represents a quotient for each vector component.
- the length of the normal vector N [0] is equal to 1.
- any one of the normal vectors Nh [i] obtained in the same manner as in the second method may be selected as the normal vector N [0] of the polygon P0.
- step S32 shown in FIG. 7 the curvature of the surface of the object is (1) the angle formed by the normal vectors of two adjacent polygons, or (2) the maximum value of each component of the normal vector of the polygon. It is evaluated using either.
- FIG. 8 is a schematic diagram showing a case where a collection of polygons is represented by the triangular strip shown in (a) of FIG.
- the operator “ ⁇ ” represents an inner product of vectors.
- the letter P represents the total number of polygons constituting the triangular strip.
- FIG. 8 is a schematic diagram showing a case where a polygonal aggregate is represented by the triangular fan shown in (b) of FIG.
- the letter P represents the total number of polygons constituting the triangular fan.
- the angle ⁇ [i] is of (P-1) types, and when the triangular fan is closed, the angle ⁇ [i] is of P types.
- FIG. 8 is a schematic diagram showing a case where a collection of polygons is expressed by the triangular mesh shown in (c) of FIG. In that case, the determination unit 161A first configures the polygon buffer in the memory unit 161B with reference to the 3D data of the object.
- Polygon buffer is a correspondence table between a set of indexes defining each polygon and a normal vector of the polygon.
- FIG. 9 is a table showing an example of the polygon buffer. Referring to FIG. 9, serial numbers 0, 1, 2,..., P-1 are assigned to each polygon as unique IDs. Here, the letter P represents the total number of polygons constituting one surface of the object.
- Each polygon ID is associated with one set of vertex indices constituting the corresponding polygon and one normal vector of the polygon.
- the determination unit 161A searches the polygon buffer for normal vectors associated with each set of indexes, and calculates the angle formed by these normal vectors.
- the determination unit 161A When determining the curvature of the surface of the object using the maximum value of each component of the polygon normal vector, the determination unit 161A first calculates one of the normal vectors calculated in step S31. Select and assume that the x-axis component is the maximum value maxNx. Next, the determination unit 161A selects another normal vector and compares the x-axis component Nx with the maximum value maxNx. When the x-axis component Nx is larger than the maximum value maxNx, the determination unit 161A replaces the maximum value maxNx with the x-axis component Nx. This process is repeated for the normal vectors of all polygons constituting one surface of the object, whereby the true maximum value of the x-axis component is obtained.
- the determination unit 161A calculates the length
- FIG. 10 is a flowchart of step S12 shown in FIG.
- the display device 10 creates a frame for the left eye as follows.
- step S41 the control unit 161 reads the texture mode from the memory unit 161B, and determines whether or not it is set to “for left eye”. If the texture mode is set to “for left eye”, the process proceeds to step S42, and if not set, the process proceeds to step S43.
- step S42 since the texture mode is set to “left eye”, the left eye photograph / moving image should be mapped to the CG image for the left eye of the object. Therefore, the control unit 161 extracts the video data for the left eye from the video data received through the input units 100-130 and sends it to the mapping unit 162A. Thereafter, the process proceeds to step S43.
- step S43 the control unit 161 extracts the camera parameter for the left eye from the video data and stores it in the memory unit 161B. Thereafter, the process proceeds to step S44.
- step S44 the control unit 161 causes the rendering unit 162 to use the left-eye camera parameter to perform visual field coordinate conversion on the object shape data. Thereafter, the process proceeds to step S45.
- step S45 the control unit 161 causes the mapping unit 162A to use the shape data represented by the visual field coordinate values to map the photograph / moving image to the CG image of the object.
- the texture mode is set to “for left eye”
- the video data for the left eye is used for the mapping process in step S42.
- the texture mode is set to “for the right eye”
- the video data for the left eye is not sent to the mapping unit 162A, and therefore the existing video data for the right eye is used for the mapping process. Details of the mapping process will be described later.
- the texture mode when the texture mode is set to “left eye”, the left eye photo / movie is combined with the CG image of the object, and when the texture mode is set to “right eye”, the right eye Photos / moving pictures are combined with the CG video of the object.
- the synthesized video is sent out as a frame for the left eye.
- FIG. 11 is a flowchart of step S27 shown in FIG. 5 and step S45 shown in FIG. Data to be processed is different between step S27 and step S45. However, the specific processing is common to all steps.
- step S51 the rendering unit 162 uses the vertex shader 222A to perform processing on the vertices of each polygon constituting the object, and calculates the attribute value of each vertex.
- the attribute values include, for example, illuminance and color coordinate values.
- the illuminance calculation uses a Lambert model, a Phong model, or a unique shading model described by the vertex shader 222A. Thereafter, the process proceeds to step S52.
- step S52 the rendering unit 162 performs viewport conversion on the shape data of the object using the vertex shader 222A.
- Viewport refers to an area where video is actually displayed in a window displayed on the screen of the display panel 172.
- Viewport conversion refers to a process of obtaining a CG video of an object projected on a 2D screen, and further converting the CG video into a video for display in a viewport.
- any method known in the technical field of 3DCG may be used.
- a method defined in OpenGL is used.
- the rendering unit 162 further calculates an inclination parameter for the shape data that has undergone viewport conversion.
- “Inclination parameter” is a parameter that represents the inclination of a straight line corresponding to each side of a polygon in a CG image of an object to be displayed in the viewport. Thereafter, the process proceeds to step S53.
- step S53 the rendering unit 162 causes the pixel shader 222B to use the attribute value of each vertex calculated in step S51 and the shape data and the inclination parameter calculated in step S52, and perform rasterization processing on the shape data.
- the rendering unit 162 calculates color information (pixel data) of each pixel in the viewport, and expresses a CG image of an object to be displayed in the viewport as raster data.
- any method known in the technical field of 3DCG may be used. For example, DDA (Digital Differential Analyzer) is used.
- the obtained raster data is written into the frame buffer 162B. Thereafter, the process proceeds to step S54.
- step S54 the rendering unit 162 causes the pixel shader 222B to map the video represented by the video data transferred from the memory unit 161B as a texture to the CG video of the object.
- Any method known in the 3DCG technical field may be used for the texture mapping.
- the rendering unit 162 first obtains texture coordinates for each pixel in which pixel data is written in step S53.
- the rendering unit 162 combines the pixel data with the portion of the video data indicated by the texture coordinates of each pixel data.
- any method known in the technical field of 3DCG may be used.
- a method defined in OpenGL is used.
- the synthesized pixel data is written into the frame buffer 162B.
- raster data representing a composite video in which the video represented by the video data is mapped to the CG video of the object is configured in the frame buffer 162B as a frame for the right eye or the left eye.
- FIG. 12A is a schematic diagram showing a CG image of an object having a flat surface.
- the object POB is a triangular pyramid placed in the 3D virtual space, and the surfaces are all flat triangles.
- the planar CG image POR obtained by projecting the triangular pyramid POB onto the right-eye 2D screen SCR is calculated from the right-eye camera parameters corresponding to the viewer's right-eye viewpoint VPR.
- the planar CG image POL obtained by projecting the triangular pyramid POB onto the left-eye 2D screen SCL is calculated from the left-eye camera parameters corresponding to the viewer's left-eye viewpoint VPL.
- (B) in FIG. 12 represents a planar CG image POL for the left eye of the triangular pyramid POB
- (c) represents a planar CG image POR for the right eye.
- the depth of the triangular pyramid POB is deeper than the screen.
- the distance DL1 between the CG image POL for the left eye and the left end of the screen SCR is equal to the CG image POR for the right eye and the screen SCR. It is smaller than the distance DR1 between the left end.
- FIG. 12 represents a left-eye planar image constituting one frame of a stereoscopic video image / moving image to be mapped on the surface of the triangular pyramid POB, and (e) represents the same frame.
- a right-eye planar image is shown.
- the depth of the “tree” stereoscopic image captured in one frame is in front of the screen.
- the distance DL2 between the “tree” left-eye video LV and the left end of the screen SCR is the right-eye video of the “tree”. It is greater than the distance DR2 between RV and the left edge of the screen SCR.
- the texture mode is set to “left eye” in step S25 shown in FIG.
- the left-eye video LV of “tree” shown in (d) is mapped to the left-eye CG video POL of the triangular pyramid shown in (b) of FIG. 12, and shown in (c).
- the “tree” right-eye image RV shown in (e) is mapped.
- FIG. 12 represents a composite image of a triangular pyramid left-eye CG image POL and a “tree” left-eye image LV
- (g) shows a triangular pyramid right-eye CG image POR and “tree”.
- the distance DL3 between the left end of the left-eye CG image POR and the left-eye image LV is the left end of the right-eye CG image POR and the right-eye image RV. The distance between is greater than DR3.
- H of FIG.
- FIG. 12 is a schematic diagram showing a stereoscopic video that can be seen by a viewer when each synthesized video shown in (f) and (g) is viewed by each viewer. As shown in (h) of FIG. 12, the “tree” stereoscopic video SV1 appears to protrude forward from the triangular pyramid stereoscopic video SV2.
- FIG. 13 is a schematic diagram showing a CG image of an object having a curved surface.
- the object COB is an elliptical cylinder placed in the 3D virtual space, and the surface is greatly curved.
- the planar CG image COR obtained by projecting the elliptic cylinder COB onto the right-eye 2D screen SCR is calculated from the right-eye camera parameters corresponding to the viewer's right-eye viewpoint VPR.
- the planar CG image POL obtained by projecting the elliptic cylinder COB on the left-eye 2D screen SCL is calculated from the left-eye camera parameters corresponding to the viewer's left-eye viewpoint VPL.
- FIG. 13 represents a planar CG image COL for the left eye of the elliptical column COB, and (c) represents a planar CG image COR for the right eye.
- the depth of the elliptic cylinder COB is deeper than the screen.
- the distance DL4 between the CG image COL for the left eye and the left end of the screen SCR is equal to the CG image POR for the right eye and the screen SCR. It is smaller than the distance DR4 between the left end.
- FIG. 13 represents a left-eye planar image constituting one frame of a stereoscopic video image / moving image to be mapped on the surface of the elliptic cylinder COB, and (e) represents the same frame.
- a right-eye planar image is shown.
- the depth of the “tree” stereoscopic image captured in one frame is in front of the screen.
- the distance DL2 between the “tree” left-eye video LV and the left edge of the screen SCR is the right-eye video of the “tree”. It is greater than the distance DR2 between RV and the left edge of the screen SCR.
- FIG. 13 represents a composite image of the left-eye CG image COL of the elliptic cylinder and the “tree” right-eye image RV, and (g) of the right-eye CG image COR of the ellipse and the “tree”.
- the distance DL5 between the left end of the left-eye CG image COR and the right-eye image RV is equal to the left end of the right-eye CG image COR and the right-eye image RV. Is approximately equal to the distance DR5.
- H of FIG.
- FIG. 13 is a schematic diagram showing a stereoscopic video that can be seen by a viewer when each synthesized video shown in (f) and (g) is viewed by each viewer.
- the “tree” stereoscopic video SV3 is located at the same depth as the elliptical stereoscopic video SV4, and particularly appears to be attached to the surface thereof. .
- the display device 10 As shown in each of FIGS. 12 and 13 (h), the display device 10 according to the first embodiment of the present invention has a relatively large curvature of the surface when mapping the image of the photograph / moving image to the surface of the object. When it is low, a stereoscopic image of a photograph / moving image is mapped on the surface, and when it is relatively high, a plane image (for the right eye) is mapped. Thereby, if the surface of the object is flat, the display device 10 can make the photograph / moving image appear to float or sink from the surface.
- the display device 10 can display the right-eye photo / video. Only the rendering unit 162 needs to process. As a result, the load necessary for the mapping process, particularly the load necessary for bitBLT from the memory unit 161B to the rendering unit 162 is reduced, so that the process of synthesizing the photograph / moving image with the CG video of the object is further facilitated. As a result, the display device 10 can further improve the image quality of these synthesized videos.
- the display device according to the second embodiment of the present invention is not the curvature of the surface of the object but the surface of the object in the determination of the appropriateness of the mapping of the stereoscopic video in step S25 shown in FIG.
- the illuminance at the apex of each polygon that constitutes is evaluated.
- the display device according to the second embodiment has the same configuration and function as those according to the first embodiment. Therefore, in the following description, of the display device according to the second embodiment, a changed part and an extended part from those according to the first embodiment will be described. For the same parts as those of the display device according to the first embodiment, the description of the first embodiment is cited.
- FIG. 14 is a flowchart of step S25 shown in FIG.
- the determination unit 161A determines whether or not the mapping of the photo / moving image to the surface of the object according to the illuminance of the vertex of the polygon constituting the surface of the object is as follows. to decide. In the following description, it is assumed that the normal vector of each vertex is calculated by the first method. If the 3D data of the object defines a normal vector for each vertex, steps S61 and S62 are skipped.
- step S61 the determination unit 161A causes the rendering unit 162 to refer to the 3D data of the object to calculate the normal vector of each polygon. Thereafter, the process proceeds to step S62.
- step S62 the determination unit 161A causes the rendering unit 162 to calculate the normal vector of each vertex by averaging the normal vectors of each polygon. The specific evaluation method will be described later. Thereafter, the process proceeds to step S63.
- step S63 the determination unit 161A causes the rendering unit 162 to use a predetermined reflected light model and calculate the illuminance of the vertex from the normal vector of each vertex.
- a Lambert model or a Phong model is used as the reflected light model. Thereafter, the process proceeds to step S64.
- step S64 the determination unit 161A determines the maximum value and the minimum value from the illuminance at each vertex, and calculates the difference between them. Thereafter, the process proceeds to step S65.
- step S65 the determination unit 161A compares the difference calculated in step S64 with a predetermined threshold value and determines whether or not the difference is equal to or greater than the threshold value.
- the threshold value is stored in the memory unit 161B in advance by the control unit 161, and the maximum value and the minimum value of the illuminance at each vertex when the surface of the object is in any of the following states: Represents the difference between values.
- the first state represents a state where the stereoscopic video mapped on the surface becomes invisible stereoscopically when the curvature of the surface of the object is gradually increased from zero. When the curvature of the surface is sufficiently large, any vertex of the polygon that constitutes the surface is hidden by the shadow of the surface itself.
- the difference between the maximum and minimum values of the illuminance at the apex can be regarded as evaluating the curvature of the surface by its shadow density.
- the second state represents a state in which when the surface of the object is gradually tilted from a state perpendicular to the viewing direction, the stereoscopic video mapped on the surface is tilted so as not to be viewed stereoscopically.
- the illuminance is greatly different between a vertex located in a portion near the viewpoint and a vertex located in a portion far from the viewpoint.
- the difference between the maximum value and the minimum value of the illuminance at the vertices can be regarded as evaluating the inclination of the surface by the difference in distance between each vertice and the viewpoint. If the difference between the maximum value and the minimum value of the illuminance at the apex is equal to or greater than the threshold value, the process proceeds to step S66. If the difference is less than the threshold value, the process proceeds to step S67.
- step S66 the determination unit 161A sets the texture mode to “for right eye”. Since the difference between the maximum value and the minimum value of the illuminance at the vertex is equal to or greater than the threshold value, the surface of the object is greatly curved or greatly inclined with respect to the viewing direction. Therefore, the stereoscopic image mapped on the surface cannot be seen stereoscopically. Therefore, the texture mode is set so that the same right eye photograph / moving image as the right eye CG image is mapped to the left eye CG image of the object. The texture mode is stored in the memory unit 161B. Thereafter, the process proceeds to step S26 shown in FIG.
- step S67 the determination unit 161A sets the texture mode to “left eye”. Since the difference between the maximum value and the minimum value of the illuminance at the vertex is less than the threshold value, the stereoscopic image mapped on the surface looks right stereoscopic. Accordingly, the texture mode is set so that the left eye photograph / moving image is mapped to the left eye CG image of the object. The texture mode is stored in the memory unit 161B. Thereafter, the process proceeds to step S26 shown in FIG.
- FIG. 15A is a schematic diagram showing a normal vector of a polygon and a normal vector of a vertex belonging to a triangular mesh constituting a part of the surface of the object.
- the rendering unit 162 averages the normal vectors among the polygons sharing one vertex, and determines the average vector as the vertex normal vector.
- FIG. 15A is a schematic diagram showing a normal vector of a polygon and a normal vector of a vertex belonging to a triangular mesh constituting a part of the surface of the object.
- FIG. 15B is a schematic diagram showing a CG image of a flat object.
- the object BOB is greatly inclined with respect to the viewing direction from the viewpoint VP in the 3D virtual space. That is, the surface of the object BOB facing the viewpoint VP is greatly inclined with respect to the projection plane SCR for the viewpoint VP.
- the illuminance at the corner CR1 closest to the viewpoint VP is maximum, and the illuminance at the angle CR2 farthest from the viewpoint VP is minimum.
- the inclination of the surface of the object can be evaluated from the difference between the maximum value and the minimum value of illuminance.
- FIG. 15 (c) shows stereoscopic CG images BO1, BO2, and BO3 of three flat objects, and stereoscopic images VST and VTR of “star”, “tree”, and “circle” mapped respectively. It is a schematic diagram which shows VCR. As shown in FIG. 15C, the stereoscopic CG image B01 of the first object and the stereoscopic CG image B03 of the third object are greatly inclined with respect to the screen SCR.
- the stereoscopic images VST and VCR of “star” and “circle” are It is located at the same depth as the stereoscopic images BO1 and BO3 of each object, and in particular, it appears to be attached to each surface.
- the stereoscopic CG image B02 of the second object is substantially parallel to the screen SCR.
- a “tree” right-eye video is mapped to the right-eye CG video of the second object
- a “tree” left-eye video is mapped to the left-eye CG video of the second object.
- the “tree” stereoscopic video VTR appears to protrude forward from the stereoscopic video BO2 of the second object.
- the display device maps a picture / moving image to the surface of an object between the vertices of polygons constituting the surface.
- a stereoscopic image of a photograph / moving image is mapped on the surface, and when it is relatively large, a planar image (for the right eye) is mapped.
- the display device can make the photo / moving image appear to float or sink from the surface if the surface of the object is relatively parallel to the screen.
- the display device can Only the moving image may be processed by the rendering unit 162.
- the load necessary for the mapping process particularly the load necessary for bitBLT from the memory unit 161B to the rendering unit 162 is reduced, so that the process of synthesizing the photograph / moving image with the CG video of the object is further facilitated.
- the display device can further improve the quality of the synthesized video.
- the display device according to the third embodiment of the present invention is not the curvature of the surface of the object but the center of gravity in the determination of the appropriateness of the mapping of the stereoscopic video in step S25 shown in FIG. Evaluate the movement. Except for this point, the display device according to the third embodiment has the same configuration and function as those according to the first embodiment. Accordingly, in the following description, of the display device according to the third embodiment, a changed part and an extended part from those according to the first embodiment will be described. For the same parts as those of the display device according to the first embodiment, the description of the first embodiment is cited.
- FIG. 16 is a flowchart of step S25 shown in FIG.
- the determination unit 161A performs a photograph / image on the surface of the object according to the amount of displacement of the center of gravity of the object between two consecutive frames as follows. Determine whether video mapping is appropriate.
- the character NV is the vertex.
- step S72 the determination unit 161A first reads the coordinate value VG0 of the center of gravity of the object in the frame immediately before the current frame from the memory unit 161B. Next, the determination unit 161A obtains the magnitude
- step S73 the determination unit 161A compares the displacement amount
- the threshold value is stored in the memory unit 161B in advance by the control unit 161, and the object in one frame period when the stereoscopic video mapped on the surface of the object becomes invisible stereoscopically is displayed. Represents the displacement amount of the center of gravity, that is, the center of gravity speed of the object. If the displacement amount
- step S74 the determination unit 161A sets the texture mode to “for right eye”. Since the center-of-gravity speed of the object is equal to or higher than the threshold value, the movement of the surface of the object is too fast for stereoscopic viewing of the stereoscopic video mapped on the surface. Therefore, the texture mode is set so that the same right eye photograph / moving image as the right eye CG image is mapped to the left eye CG image of the object. The texture mode is stored in the memory unit 161B. Thereafter, the process proceeds to step S26 shown in FIG.
- step S75 the determination unit 161A sets the texture mode to “left eye”. Since the center-of-gravity speed of the object is less than the threshold value, the stereoscopic video mapped on the surface looks right stereoscopic. Accordingly, the texture mode is set so that the left eye photograph / moving image is mapped to the left eye CG image of the object. The texture mode is stored in the memory unit 161B. Thereafter, the process proceeds to step S26 shown in FIG.
- FIG. 17A is a schematic diagram showing a stereoscopic CG image PL1 of a flat object and a “tree” stereoscopic image TR1 mapped thereon.
- the stereoscopic CG image PL1 of the object is stationary with respect to the screen SCR. In that case, the center of gravity of the stereoscopic CG video PL1 of the object is not displaced between consecutive frames. Accordingly, the “tree” right-eye video is mapped to the right-eye CG video of the object, and the “tree” left-eye video is mapped to the left-eye CG video of the object.
- the “tree” stereoscopic video TR1 appears to protrude forward from the stereoscopic video PL1 of the object.
- FIG. 17 is a schematic diagram showing the stereoscopic CG video PL2 of the object and the stereoscopic video TR2 of “tree” in the frame next to the frame shown in (a).
- the center of gravity of the stereoscopic CG image PL2 of the object is greatly displaced from the center of gravity of the stereoscopic CG image PL1 in the previous frame.
- only the “tree” right-eye video is mapped to both the left-eye CG video and the right-eye CG video of the object.
- the “tree” stereoscopic image TR2 is located at the same depth as the stereoscopic CG image PL2 of the object, and particularly appears to be attached to the surface thereof.
- the center of gravity of the stereoscopic CG image of the object is greatly displaced every time the frame is switched, only the right-eye image of “tree” is mapped to the stereoscopic CG image. Therefore, during that time, the stereoscopic image of “tree” appears to move together with the stereoscopic CG image while being attached to the stereoscopic CG image of the object.
- FIG. 17 is a schematic diagram showing the stereoscopic CG video PL3 of the object and the stereoscopic video TR3 of “tree” after several frames from the frame shown in (b).
- the stereoscopic CG video PL3 of the object is still with respect to the screen SCR.
- the center of gravity of the stereoscopic CG video PL3 of the object is not displaced between successive frames.
- the “tree” right-eye video is mapped to the right-eye CG video of the object
- the “tree” left-eye video is mapped to the left-eye CG video of the object.
- the “tree” stereoscopic video TR3 appears to protrude forward from the stereoscopic video PL3 of the object.
- the display device when mapping a photograph / moving image on the surface of an object, between two consecutive frames.
- the stereoscopic image of the photograph / moving image is mapped on the surface of the object, and when the displacement is relatively large, the planar image (for the right eye) is mapped.
- the display device can make the photo / video appear to float or sink from its surface if the movement of the object is relatively slow.
- the display unit can cause the rendering unit 162 to process only the right eye photo / video. That's fine.
- the load necessary for the mapping process particularly the load necessary for bitBLT from the memory unit 161B to the rendering unit 162 is reduced, so that the process of synthesizing the photograph / moving image with the CG video of the object is further facilitated.
- the display device can further improve the quality of the synthesized video.
- Embodiment 4 Unlike the display device according to the first embodiment, the display device according to the fourth embodiment of the present invention replaces step S24 shown in FIG. 5 in step S11 for creating the right-eye frame shown in FIG. A step of determining the movement of the planar image of the object is performed. Except for this point, the display device according to the fourth embodiment has the same configuration and function as those according to the first embodiment. Accordingly, in the following description, of the display device according to the fourth embodiment, a changed part and an extended part from those according to the first embodiment will be described. For the same parts as those of the display device according to the first embodiment, the description of the first embodiment is cited.
- FIG. 18 is a flowchart when the display device according to the fourth embodiment of the present invention performs step S11 shown in FIG.
- the determination unit 161A refers to the shape data of the object and determines whether or not the shape data is 2D data. If the shape data is 2D data, the process proceeds to step S84. If the shape data is not 2D data, the process proceeds to step S25.
- step S84 the object is a planar view CG image, and the whole is a planar image that can be seen at the same depth as the screen.
- the determination unit 161A determines the texture mode from the amount of displacement of the representative point of the object between two consecutive frames as shown below.
- the texture mode is stored in the memory unit 161B. Thereafter, the process proceeds to step S26.
- FIG. 19 is a flowchart of step S84 shown in FIG.
- the determination unit 161A represents a representative point of an object between two consecutive frames, for example, the center of gravity. In accordance with the amount of displacement, it is determined whether or not the mapping of the photo / moving image to the surface of the object is appropriate.
- the representative point may be a point fixed to any part such as the boundary of the object in addition to the center of gravity.
- the obtained coordinate value of the center of gravity is stored in the memory unit 161B. Thereafter, the process proceeds to step S92.
- step S92 the determination unit 161A first reads the coordinate value VG0 of the center of gravity of the object in the frame immediately before the current frame from the memory unit 161B. Next, the determination unit 161A obtains the magnitude
- step S93 the determination unit 161A compares the displacement amount
- the threshold value is stored in the memory unit 161B in advance by the control unit 161, and represents the center-of-gravity speed of the object when the stereoscopic video mapped on the surface of the object becomes invisible three-dimensionally. If the displacement amount
- step S94 the determination unit 161A sets the texture mode to “for right eye”. Since the center-of-gravity speed of the object is equal to or higher than the threshold value, the movement of the surface of the object is too fast for stereoscopic viewing of the stereoscopic video mapped on the surface. Therefore, the texture mode is set so that the same right eye photograph / moving image as the right eye CG image is mapped to the left eye CG image of the object. The texture mode is stored in the memory unit 161B. Thereafter, the processing proceeds to step S26 shown in FIG.
- step S95 the determination unit 161A sets the texture mode to “left eye”. Since the center-of-gravity speed of the object is less than the threshold value, the stereoscopic video mapped on the surface looks right stereoscopic. Accordingly, the texture mode is set so that the left eye photograph / moving image is mapped to the left eye CG image of the object. The texture mode is stored in the memory unit 161B. Thereafter, the processing proceeds to step S26 shown in FIG.
- FIG. 20A is a schematic diagram showing a planar CG video MV1 of a rectangular object and a “tree” stereoscopic video ST1 mapped thereon.
- the planar view CG image MV1 of the object is stationary on the screen SCR. In that case, the center of gravity of the planar view CG image MV1 of the object is not displaced between consecutive frames. Accordingly, the right-eye video of “tree” is mapped to the CG video of the object, and the right-eye planar video is synthesized. On the other hand, the left-eye video of “tree” is mapped to the CG video of the same object, and the left-eye planar video is synthesized. As a result, as shown in FIG. 20A, the “tree” stereoscopic image ST1 appears to protrude forward from the planar CG image MV1 of the object.
- FIG. 20 is a schematic diagram showing a planar view CG image MV2 of an object and a “tree” stereoscopic image ST2 in a frame subsequent to the frame shown in (a).
- the center of gravity of the planar view CG image MV2 of the object is greatly displaced from the center of gravity of the planar view CG image MV1 in the previous frame.
- only the “tree” right-eye video is mapped to the CG video of the object, and the left-eye and right-eye planar images are combined.
- the stereoscopic image ST2 of “tree” appears to be pasted on the screen in the same manner as the planar view CG image MV2 of the object.
- the center of gravity of the planar CG image of the object is greatly displaced each time the frame is switched, only the “tree” right-eye video is mapped to the planar CG image. Therefore, in the meantime, the stereoscopic image of “tree” appears to move on the screen together with the planar CG image of the object.
- FIG. 20 is a schematic diagram showing a planar view CG image MV3 of an object and a “tree” stereoscopic image ST3 after several frames from the frame shown in (b).
- the stereoscopic CG image MV3 of the object is still on the screen SCR. In that case, the center of gravity of the planar view CG image MV3 of the object is not displaced between consecutive frames. Accordingly, the right-eye video of “tree” is mapped to the CG video of the object, and the planar video for the right eye is synthesized, and the left-eye video of “tree” is mapped to the CG video of the object. A planar video is synthesized. As a result, as shown in FIG. 20C, the “tree” stereoscopic video ST3 appears to protrude forward from the planar CG video MV3 of the object.
- the display device when mapping a photograph / moving image on the surface of an object, displays the object between consecutive frames.
- the stereoscopic image of the photograph / moving image is mapped on the surface of the object, and when the displacement is relatively large, the planar image (for the right eye) is mapped.
- the display device can make the photo / video appear to float or sink from its surface if the movement of the object is relatively slow.
- the display device is for the right eye. Only the photo / video may be processed by the rendering unit 162. As a result, the load necessary for the mapping process, particularly the load necessary for bitBLT from the memory unit 161B to the rendering unit 162 is reduced, so that the process of synthesizing the photograph / moving image with the CG video of the object is further facilitated. As a result, the display device can further improve the quality of the synthesized video.
- the right-eye frame is generated before the left-eye frame when one frame of stereoscopic video is displayed.
- the left-eye frame may be generated before the right-eye frame.
- the setting of the texture mode is the reverse of the above example.
- the texture mode is set to “for right eye”.
- the process proceeds to step S35, and when it is less than the threshold, the process proceeds to step S34.
- step S35 since the curvature of the surface of the object is equal to or greater than the threshold value, the determination unit 161A sets the texture mode to “for left eye” and the same left eye CG image as the left eye CG image is displayed on the object. Make sure that your photos / videos are mapped.
- step S34 since the curvature of the surface of the object is less than the threshold, the determination unit 161A sets the texture mode to “for right eye”, and the right eye photograph / video is included in the right eye CG image of the object. To be mapped.
- a polarization display method or a lenticular method may be used.
- filters having different polarization directions are installed in the display area of the left-eye frame (for example, odd-numbered lines) and the display area of the right-eye frame (for example, even-numbered lines).
- the viewer is made to see the screen through polarized glasses.
- a filter having a different polarization direction is provided for each lens. Accordingly, since the left-eye and right-eye frames are visible only to the viewer's eyes, the viewer can be shown a 3D image.
- the “lenticular method” divides the left-eye and right-eye frames into small strip-shaped areas that are elongated in the vertical direction, and alternately arranges the small areas of those frames horizontally in one screen. Display at the same time.
- the surface of the screen is covered with a lenticular lens.
- the lenticular lens is a single sheet formed by arranging a plurality of elongate eyelid lenses in parallel. Each heel lens extends vertically on the surface of the screen.
- the output unit 162C shown in FIG. 1 converts the frame output format in accordance with the 3D video display method by the display unit 170.
- the output unit 162C uses a built-in buffer memory to combine a pair of a left-eye frame and a right-eye frame into one frame.
- the output unit 162C temporarily stores the previously synthesized right-eye frame in its buffer memory and holds it.
- the output unit 162C then combines the left-eye frame and further combines with the right-eye frame held in the buffer memory.
- the left-eye frame and the right-eye frame are alternately arranged in units of lines and reconfigured into one frame.
- the output unit 162C uses the built-in buffer memory to synthesize a pair of the left-eye frame and the right-eye frame into one frame. Specifically, the output unit 162C temporarily stores the previously synthesized right-eye frame in its buffer memory and holds it. The output unit 162C then combines the left-eye frame and further combines with the right-eye frame held in the buffer memory. In the synthesis, the left-eye frame and the right-eye frame are each divided into strip-shaped small areas that are elongated in the vertical direction, and each small area is alternately arranged in the horizontal direction in one frame. Reconstructed into frames. Thus, one synthesized frame is sent to the display unit 170.
- FIG. 21 is a flowchart of processing in which the display device creates a frame for the right eye and for the left eye by mapping a stereoscopic video image / moving image to the CG image of the object. The display device starts this processing when the user is instructed to execute the viewer function or the slide show function through the operation unit 140 shown in FIG.
- step S101 the display device initializes the integer value variable N to “1”.
- step S11 the display device creates a video in which a right eye photo / moving image is mapped to the CG video for the right eye of the Nth object, and synthesizes it with the frame for the right eye.
- step S102 the display device determines whether or not the variable N has reached the total number of objects. If the variable N is smaller than the total number of objects, the process proceeds to step S103. If the variable N is equal to the total number of objects, the process proceeds to step S104. In step S103, a value obtained by adding 1 to the variable N is set as a new variable N. Thereafter, the process is repeated from step S11.
- the right eye photograph / moving image is mapped to the right eye CG image of all the objects, and these are combined into one right eye frame.
- the data of the frame is sent to the display unit 170.
- a list of texture modes for all objects is stored in the memory unit 161B.
- step S104 the display device initializes the integer value variable N to “1” again.
- step S12 the display device first refers to the list of texture modes and determines whether the left-eye or right-eye photo / moving image should be mapped to the left-eye CG image of the Nth object. .
- the display device maps the photograph / moving image indicated by the determination result to the left-eye CG image of the Nth object, and synthesizes the obtained image into the left-eye frame.
- step S105 the display device determines whether or not the variable N has reached the total number of objects. If the variable N is smaller than the total number of objects, the process proceeds to step S106. If the variable N is equal to the total number of objects, the process proceeds to step S13.
- step S106 a value obtained by adding 1 to the variable N is set as a new variable N. Thereafter, the process is repeated from step S12. In this way, by repeating steps S12 to S106, photographs / moving images are mapped to the CG images for the left eye of all the objects, and these are combined into one left eye frame. The data of the frame is sent to the display unit 170.
- step S13 the display device checks whether the next frame exists in the photograph / moving picture. If there is a next frame, the process is repeated from step S101. If there is no next frame, the process ends.
- the photograph / moving image is mapped as a stereoscopic video to the planar CG video of the object. Is done.
- the photograph / moving image may be mapped to the planar view CG image of the object as the planar view image. In that case, the CG video of the object and the photo / moving image are displayed at the same depth.
- FIG. 22 is a flowchart of a process in which the display device maps a photograph / moving image to a CG image of an object in the above case.
- the display device starts this processing when the user is instructed to execute the viewer function or the slide show function through the operation unit 140 shown in FIG.
- the flowchart shown in FIG. 22 is different from those shown in FIGS. 3 and 5 and includes steps S111 and S112 for creating a frame of a planar view video, that is, a 2D frame, instead of step S24. Except for this point, the flowchart shown in FIG. 22 is the same as that shown in FIGS. Therefore, in the following, of the steps shown in FIG. 22, the changed part and the extended part from those shown in the actual drawings 3 and 5 will be described. For the steps similar to those shown in FIGS. 3 and 5, the description of FIGS.
- step S23 the determination unit 161A refers to the shape data of the object and determines whether or not the shape data is 2D data. If the shape data is 2D data, the process proceeds to step S111. If the shape data is not 2D data, the process proceeds to step S25.
- the vertex shader 222A is used for these operations. Thereafter, the process proceeds to step S112.
- step S112 the control unit 161 causes the mapping unit 162A to use the shape data represented by the visual field coordinate values to map the right eye photograph / moving image to the CG image of the object.
- the details of the mapping process are the same as those described above. In this way, the right eye photograph / moving image is combined with the CG image of the object to create a 2D frame.
- the process skips step S12 for creating a frame for the left eye and proceeds to step S13 shown in FIG.
- the display device may use the operation unit 140 to allow the user to select whether or not to display a composite video of a photo / video and an object as a stereoscopic video.
- the present invention relates to a stereoscopic video display technology, and, as described above, maps and synthesizes another video to the CG video of the object.
- the present invention is clearly industrially applicable.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
- Processing Or Creating Images (AREA)
- Controls And Circuits For Display Device (AREA)
Abstract
Description
[表示装置の構成]
図1は、本発明の実施形態1による表示装置の機能ブロック図である。この表示装置はデジタルテレビ受像器である。表示装置10はその他に、携帯電話機、携帯情報端末、パーソナルコンピュータ、デジタルスチルカメラ、デジタルビデオカメラ、カーナビゲーション・システム等、多様な電子機器に組み込まれたものであってもよい。図1を参照するに、表示装置10は、映像データの入力部、操作部140、システムバス150、CG映像合成部160、表示部170,及び左右信号送信部180を備えている。映像データの入力部は、受信部100、光ディスクドライブ110、カードリーダ120、及びハードディスクドライブ(HDD)130を含む。
図2は、CG映像合成部160のハードウェア構成を示すブロック図である。図2を参照するに、制御部161は、第1内部バス210、第1入出力インタフェース(I/O)211、CPU212、及びメインメモリ213を含む。レンダリング部162は、出力部162Cの他に、第2内部バス220、第2I/O221、グラフィックス処理専用プロセッサ(GPU:Graphic Processor Unit)222、及びビデオメモリ(VRAM)223を含む。
表示装置10は、図1、2に示されている要素を利用して、ビューワー機能又はスライドショー機能に立体視効果を与える。具体的には、表示装置10は入力部100-130を通して立体視映像の写真/動画を取得し、CG映像合成部160を利用してその写真/動画をオブジェクトのCG映像にマッピングして表示パネル172の画面に表示する。そのとき、実施形態1による表示装置10は、以下に説明するように、マッピング先のオブジェクトの形状に応じてマッピング対象の写真/動画の表す映像を立体視映像と平面視映像との間で切り換える。
形状データの種類には2Dデータと3Dデータとがある。2Dデータはオブジェクトの2D画面上での形状を表す。3Dデータはオブジェクトの3D仮想空間内での形状を表す。
図5は、図1に示されているステップS11、すなわち、表示装置10が右目用のフレームを作成する処理のフローチャートである。
図7は、図5に示されているステップS25のフローチャートである。ステップS25では、オブジェクトの形状が立体視映像のマッピングに適しているか否かを判断部161Aが判断する。本発明の実施形態1では、判断部161Aはその適否判断に、以下のように、オブジェクトの表面の曲率を利用する。
図7に示されているステップS31では、各ポリゴンの法線ベクトルが次の二つの方法のいずれかで算定される。
図7に示されているステップS32では、オブジェクトの表面の曲率が、(1)隣接する二つのポリゴンの法線ベクトルの成す角度、又は、(2)ポリゴンの法線ベクトルの各成分の最大値のいずれかを利用して評価される。
図10は、図3に示されているステップS12のフローチャートである。ステップS12では、表示装置10が、以下のように、左目用のフレームを作成する。
図11は、図5に示されているステップS27、及び図10に示されているステップS45のフローチャートである。ステップS27とステップS45とでは処理対象のデータは異なる。しかし、いずれのステップも具体的な処理は共通である。
図12の(a)は、表面が平面的なオブジェクトのCG映像を示す模式図である。図12の(a)を参照するに、そのオブジェクトPOBは、3D仮想空間内に置かれた三角錐であって、表面はいずれも平たい三角形である。上記のとおり、その三角錐POBを右目用2D画面SCRに投影した平面視CG映像PORは、視聴者の右目の視点VPRに対応する右目用カメラ・パラメータから計算される。一方、その三角錐POBを左目用2D画面SCLに投影した平面視CG映像POLは、視聴者の左目の視点VPLに対応する左目用カメラ・パラメータから計算される。
本発明の実施形態2による表示装置は、実施形態1によるものとは異なり、図5に示されているステップS25による立体視映像のマッピングの適否判断において、オブジェクトの表面の曲率ではなく、その表面を構成する各ポリゴンの頂点の照度を評価する。その点を除き、実施形態2による表示装置は実施形態1によるものと構成及び機能が等しい。従って、以下では、実施形態2による表示装置のうち、実施形態1によるものからの変更部分及び拡張部分について説明する。実施形態1による表示装置と同様な部分については、上記の実施形態1についての説明を援用する。
図14に示されているステップS61では、頂点の法線ベクトルが次のように算定される。図15の(a)は、オブジェクトの表面の一部を構成する三角形メッシュに属するポリゴンの法線ベクトルと頂点の法線ベクトルとを示す模式図である。レンダリング部162はまず、頂点シェーダ222Aを利用して、オブジェクトの3Dデータから各頂点の座標値を読み取り、更にそれらの座標値から各ポリゴンの法線ベクトルN[k]を求める(k=0、1、2、…)。レンダリング部162は次に、一つの頂点を共有するポリゴンの間で法線ベクトルを平均して、その頂点の法線ベクトルとして決定する。図15の(a)に示されている例では1番目の頂点Ve[0]が6個のポリゴンに共有されている。従って、まず、それらのポリゴンの法線ベクトルN[k](k=0、1、2、…、5)の平均値が1番目の頂点Ve[0]の法線ベクトルNv[0]として決定される:Nv[0]=Normalize{(N[0]+N[1]+…+N[5])/6}。同様に、2番目の頂点Ve[1]と3番目の頂点Ve[2]とのそれぞれを共有するポリゴンの法線ベクトルの平均値が、各頂点Ve[1]、Ve[2]の法線ベクトルNv[1]、Nv[2]として決定される。
図15の(b)は、平板状のオブジェクトのCG映像を示す模式図である。図15の(b)を参照するに、そのオブジェクトBOBは、3D仮想空間内では、視点VPからの視方向に対して大きく傾いている。すなわち、視点VPに面したオブジェクトBOBの表面は、その視点VPに対する投影面SCRに対して大きく傾いている。その結果、その表面の中では、例えば、視点VPに最も近い角CR1の照度が最大であり、視点VPから最も遠い角CR2の照度が最小である。このように、オブジェクトの表面の形状が平面的であっても、視方向からの傾きが十分に大きければ、その表面の各部の照度に差が生じる。更に、一般には、その傾きが大きいほどその差は大きい。従って、照度の最大値と最小値との間の差からオブジェクトの表面の傾きを評価することができる。
本発明の実施形態3による表示装置は、実施形態1によるものとは異なり、図5に示されているステップS25による立体視映像のマッピングの適否判断において、オブジェクトの表面の曲率ではなく、その重心の動きを評価する。その点を除き、実施形態3による表示装置は実施形態1によるものと構成及び機能が等しい。従って、以下では、実施形態3による表示装置のうち、実施形態1によるものからの変更部分及び拡張部分について説明する。実施形態1による表示装置と同様な部分については、上記の実施形態1についての説明を援用する。
図17の(a)は、平板状のオブジェクトの立体視CG映像PL1と、それにマッピングされた“木”の立体視映像TR1とを示す模式図である。オブジェクトの立体視CG映像PL1は画面SCRに対して静止している。その場合、連続するフレーム間でそのオブジェクトの立体視CG映像PL1の重心は変位していない。従って、そのオブジェクトの右目用CG映像には“木”の右目用映像がマッピングされ、そのオブジェクトの左目用CG映像には“木”の左目用映像がマッピングされる。その結果、“木”の立体視映像TR1は、図17の(a)に示されているように、そのオブジェクトの立体視映像PL1よりも手前に飛び出しているように見える。
本発明の実施形態4による表示装置は、実施形態1によるものとは異なり、図3に示されいてる右目用のフレームを作成するステップS11において、図5に示されているステップS24に代えて、オブジェクトの平面視映像の動きを判定するステップを行う。その点を除き、実施形態4による表示装置は実施形態1によるものと構成及び機能が等しい。従って、以下では、実施形態4による表示装置のうち、実施形態1によるものからの変更部分及び拡張部分について説明する。実施形態1による表示装置と同様な部分については、上記の実施形態1についての説明を援用する。
図20の(a)は、矩形状のオブジェクトの平面視CG映像MV1と、それにマッピングされた“木”の立体視映像ST1とを示す模式図である。オブジェクトの平面視CG映像MV1は画面SCR上に静止している。その場合、連続するフレーム間でそのオブジェクトの平面視CG映像MV1の重心は変位していない。従って、そのオブジェクトのCG映像に“木”の右目用映像がマッピングされて右目用の平面視映像が合成される。一方、同じオブジェクトのCG映像に“木”の左目用映像がマッピングされて左目用の平面視映像が合成される。その結果、“木”の立体視映像ST1は、図20の(a)に示されているように、そのオブジェクトの平面視CG映像MV1よりも手前に飛び出しているように見える。
(A)本発明の上記の実施形態では、図3に示されているように、1フレームの立体視映像を表示する際に右目用のフレームを左目用のフレームよりも先に生成する。その他に、左目用のフレームを右目用のフレームよりも先に生成してもよい。その場合、テクスチャ・モードの設定は上記の例とは逆である。例えば、図5に示されているステップS24では、オブジェクトの右目用のCG映像には右目用の写真/動画がマッピングされるべきであるので、テクスチャ・モードが“右目用”に設定される。また、図7では、オブジェクトの表面の曲率が閾値以上である場合、処理はステップS35へ進み、その閾値未満である場合、処理はステップS34へ進む。ステップS35では、オブジェクトの表面の曲率は閾値以上であるので、判断部161Aはテクスチャ・モードを“左目用”に設定して、そのオブジェクトの右目用CG映像に、左目用CG映像と同じ左目用の写真/動画がマッピングされるようにする。一方、ステップS34では、オブジェクトの表面の曲率は閾値未満であるので、判断部161Aはテクスチャ・モードを“右目用”に設定して、そのオブジェクトの右目用CG映像に右目用の写真/動画がマッピングされるようにする。
POL オブジェクトPOBの左目用の平面視CG映像
POR オブジェクトPOBの右目用の平面視CG映像
SCL 左目用2D画面
SCR 右目用2D画面
VPL 視聴者の左目の視点
VPR 視聴者の右目の視点
LV “木”の左目用映像
RV “木”の右目用映像
SV1 “木”の立体視映像
SV2 オブジェクトの立体視CG映像
Claims (12)
- オブジェクトに映像をマッピングして一つのコンピュータ・グラフィックス(CG)映像に合成する装置であって、
前記オブジェクトの形状を表す形状データ、及び、一つの立体視映像を表す左目用と右目用との映像データの対を記憶しているメモリ部、
前記形状データから、前記オブジェクトが前記立体視映像のマッピングに適しているか否かを判断する判断部、並びに、
前記オブジェクトが前記立体視映像のマッピングに適していると前記判断部が判断した場合、前記左目用の映像データを前記形状データに合成して左目用のCGデータを作成すると共に、前記右目用の映像データを前記形状データに合成して右目用のCGデータを作成し、前記オブジェクトが前記立体視映像のマッピングに適していないと前記判断部が判断した場合、前記左目用の映像データと前記右目用の映像データとのいずれかのみを前記形状データに合成して左目用のCGデータと右目用のCGデータとを作成するマッピング部、
を備えているCG映像合成装置。 - 前記形状データは、前記オブジェクトの3次元(3D)仮想空間内での形状を表す3Dデータであり、
前記判断部は、前記3Dデータから、前記オブジェクトを構成する各ポリゴンの法線ベクトルを計算し、計算された法線ベクトルの集合から前記オブジェクトの表面の曲率を評価し、前記曲率が所定の閾値未満である場合、前記オブジェクトが前記立体視映像のマッピングに適していると判断し、前記曲率が前記閾値を超えている場合、前記オブジェクトが前記立体視映像のマッピングに適していないと判断する、
請求項1に記載のCG映像合成装置。 - 前記形状データは、前記オブジェクトの3D仮想空間内での形状を表す3Dデータであり、
前記判断部は、前記3Dデータから、前記オブジェクトを構成する各ポリゴンの頂点における照度を計算し、計算された照度全体での最大値と最小値との間の差が所定の閾値未満である場合、前記オブジェクトが前記立体視映像のマッピングに適していると判断し、前記差が前記閾値を超えている場合、前記オブジェクトが前記立体視映像のマッピングに適していないと判断する、
請求項1に記載のCG映像合成装置。 - 前記形状データは、前記オブジェクトの3D仮想空間内での形状を表す3Dデータであり、
前記判断部は、前記CG映像のフレームごとに、前記オブジェクトを構成する各ポリゴンの頂点から前記オブジェクトの代表点を求め、連続する二つのフレームの間で前記オブジェクトの代表点の変位の大きさが所定の閾値未満である場合、前記オブジェクトが前記立体視映像のマッピングに適していると判断し、前記変位の大きさが前記閾値を超えている場合、前記オブジェクトが前記立体視映像のマッピングに適していないと判断する、
請求項1に記載のCG映像合成装置。 - 前記形状データは、前記オブジェクトの2次元(2D)画面上での形状を表す2Dデータであり、
前記判断部は、前記CG映像のフレームごとに前記2Dデータから前記オブジェクトの代表点を求め、連続する二つのフレームの間で前記オブジェクトの代表点の変位の大きさが所定の閾値未満である場合、前記オブジェクトが前記立体視映像のマッピングに適していると判断し、前記変位の大きさが前記閾値を超えている場合、前記オブジェクトが前記立体視映像のマッピングに適していないと判断する、
請求項1に記載のCG映像合成装置。 - 前記形状データは、前記オブジェクトの2D画面上での形状を表す2Dデータと、前記オブジェクトの3D仮想空間内での形状を表す3Dデータとのいずれかであり、
前記判断部は、前記形状データが前記2Dデータと前記3Dデータとのいずれであるかを識別し、前記形状データが前記2Dデータである場合、前記オブジェクトが前記立体視映像のマッピングには適していると判断し、前記形状データが前記3Dデータである場合、前記3Dデータから、前記オブジェクトが前記立体視映像のマッピングに適しているか否かを更に判断する、
請求項1に記載のCG映像合成装置。 - 平面視映像と立体視映像とのいずれが前記オブジェクトにマッピングされるべきかを示すユーザ操作を受け付ける操作部を更に備え、
前記判断部は、前記平面視映像が前記オブジェクトにマッピングされるべきことを前記ユーザ操作が示す場合、前記オブジェクトが前記立体視映像のマッピングには適していないと判断する、
請求項1に記載のCG映像合成装置。 - オブジェクトに映像をマッピングして一つのCG映像に合成する方法であって、
前記オブジェクトの形状を表す形状データから、前記オブジェクトが、左目用と右目用との映像データの対で表される立体視映像のマッピングに適しているか否かを判断するステップ、
前記オブジェクトが前記立体視映像のマッピングに適していると判断された場合、前記左目用の映像データを前記形状データに合成して左目用のCGデータを作成すると共に、前記右目用の映像データを前記形状データに合成して右目用のCGデータを作成するステップ、及び、
前記オブジェクトが前記立体視映像のマッピングに適していないと判断された場合、前記左目用の映像と前記右目用の映像データとのいずれかのみを前記形状データに合成して左目用のCGデータと右目用のCGデータとを作成するステップ、
を有するCG映像合成方法。 - オブジェクトに映像をマッピングして一つのCG映像に合成する処理をコンピュータに実行させるためのプログラムであって、前記処理が、
前記オブジェクトの形状を表す形状データから、前記オブジェクトが、左目用と右目用との映像データの対で表される立体視映像のマッピングに適しているか否かを判断するステップ、
前記オブジェクトが前記立体視映像のマッピングに適していると判断された場合、前記左目用の映像データを前記形状データに合成して左目用のCGデータを作成すると共に、前記右目用の映像データを前記形状データに合成して右目用のCGデータを作成するステップ、及び、
前記オブジェクトが前記立体視映像のマッピングに適していないと判断された場合、前記左目用の映像と前記右目用の映像データとのいずれかのみを前記形状データに合成して左目用のCGデータと右目用のCGデータとを作成するステップ、
を有するプログラム。 - オブジェクトに映像をマッピングして一つのCG映像に合成する処理をコンピュータに実行させるためのプログラムを記録した記録媒体であって、前記処理が、
前記オブジェクトの形状を表す形状データから、前記オブジェクトが、左目用と右目用との映像データの対で表される立体視映像のマッピングに適しているか否かを判断するステップ、
前記オブジェクトが前記立体視映像のマッピングに適していると判断された場合、前記左目用の映像データを前記形状データに合成して左目用のCGデータを作成すると共に、前記右目用の映像データを前記形状データに合成して右目用のCGデータを作成するステップ、及び、
前記オブジェクトが前記立体視映像のマッピングに適していないと判断された場合、前記左目用の映像と前記右目用の映像データとのいずれかのみを前記形状データに合成して左目用のCGデータと右目用のCGデータとを作成するステップ、
を有する記録媒体。 - オブジェクトに映像をマッピングして一つのCG映像に合成する装置に搭載される集積回路であって、
前記オブジェクトの形状を表す形状データ、及び、一つの立体視映像を表す左目用と右目用との映像データの対を記憶しているメモリ部、
前記形状データから、前記オブジェクトが前記立体視映像のマッピングに適しているか否かを判断する判断部、並びに、
前記オブジェクトが前記立体視映像のマッピングに適していると前記判断部が判断した場合、前記左目用の映像データを前記形状データに合成して左目用のCGデータを作成すると共に、前記右目用の映像データを前記形状データに合成して右目用のCGデータを作成し、前記オブジェクトが前記立体視映像のマッピングに適していないと前記判断部が判断した場合、前記左目用の映像と前記右目用の映像データとのいずれかのみを前記形状データに合成して左目用のCGデータと右目用のCGデータとを作成するマッピング部、
を備えている集積回路。 - オブジェクトに映像をマッピングして一つのCG映像として表示する装置であって、
前記オブジェクトの形状を表す形状データ、及び、一つの立体視映像を表す左目用と右目用との映像データの対を記憶しているメモリ部、
前記形状データから、前記オブジェクトが前記立体視映像のマッピングに適しているか否かを判断する判断部、
前記オブジェクトが前記立体視映像のマッピングに適していると前記判断部が判断した場合、前記左目用の映像データを前記形状データに合成して左目用のCGデータを作成すると共に、前記右目用の映像データを前記3Dデータに合成して右目用のCGデータを作成し、前記オブジェクトが前記立体視映像のマッピングに適していないと前記判断部が判断した場合、前記左目用の映像と前記右目用の映像データとのいずれかのみを前記形状データに合成して左目用のCGデータと右目用のCGデータとを作成するマッピング部、並びに、
表示パネルを含み、前記左目用のCGデータの表す左目用のCG映像と、前記右目用のCGデータの表す右目用のCG映像とを交互に前記表示パネルに表示させる表示部、
を備えている表示装置。
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201180001535.5A CN102362294B (zh) | 2010-02-23 | 2011-02-22 | 计算机图形影像合成装置和方法、以及显示装置 |
JP2011534967A JP5628819B2 (ja) | 2010-02-23 | 2011-02-22 | コンピュータ・グラフィックス映像合成装置と方法、及び表示装置 |
US13/258,730 US8866887B2 (en) | 2010-02-23 | 2011-02-22 | Computer graphics video synthesizing device and method, and display device |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2010-036794 | 2010-02-23 | ||
JP2010036794 | 2010-02-23 | ||
JP2010-182039 | 2010-08-17 | ||
JP2010182039 | 2010-08-17 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2011105048A1 true WO2011105048A1 (ja) | 2011-09-01 |
Family
ID=44506481
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2011/000978 WO2011105048A1 (ja) | 2010-02-23 | 2011-02-22 | コンピュータ・グラフィックス映像合成装置と方法、及び表示装置 |
Country Status (4)
Country | Link |
---|---|
US (1) | US8866887B2 (ja) |
JP (1) | JP5628819B2 (ja) |
CN (1) | CN102362294B (ja) |
WO (1) | WO2011105048A1 (ja) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2015008373A1 (ja) * | 2013-07-19 | 2015-01-22 | 富士通株式会社 | 情報処理装置、検査範囲の計算方法、及びプログラム |
Families Citing this family (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9165393B1 (en) * | 2012-07-31 | 2015-10-20 | Dreamworks Animation Llc | Measuring stereoscopic quality in a three-dimensional computer-generated scene |
JP6116164B2 (ja) * | 2012-09-11 | 2017-04-19 | 株式会社キーエンス | 形状測定装置、形状測定方法および形状測定プログラム |
US9965893B2 (en) * | 2013-06-25 | 2018-05-08 | Google Llc. | Curvature-driven normal interpolation for shading applications |
JP2015169684A (ja) * | 2014-03-05 | 2015-09-28 | カシオ計算機株式会社 | 画像表示装置、画像表示方法、およびプログラム |
CN105741344A (zh) * | 2014-12-10 | 2016-07-06 | 富泰华工业(深圳)有限公司 | 点云精简方法及*** |
US10388069B2 (en) * | 2015-09-09 | 2019-08-20 | Futurewei Technologies, Inc. | Methods and systems for light field augmented reality/virtual reality on mobile devices |
CN106971384A (zh) * | 2017-03-16 | 2017-07-21 | 福建中金在线信息科技有限公司 | 一种图像处理方法及装置 |
CN111489411B (zh) * | 2019-01-29 | 2023-06-20 | 北京百度网讯科技有限公司 | 线条绘制方法、装置、图像处理器、显卡及车辆 |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2001076180A (ja) * | 1999-09-07 | 2001-03-23 | Nec Corp | 3次元グラフィックス表示装置 |
JP2004178581A (ja) * | 2002-11-12 | 2004-06-24 | Namco Ltd | 画像生成システム、画像生成方法、プログラム及び情報記憶媒体 |
Family Cites Families (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH11234703A (ja) * | 1998-02-09 | 1999-08-27 | Toshiba Corp | 立体表示装置 |
JP2004145832A (ja) | 2002-08-29 | 2004-05-20 | Sharp Corp | コンテンツ作成装置、コンテンツ編集装置、コンテンツ再生装置、コンテンツ作成方法、コンテンツ編集方法、コンテンツ再生方法、コンテンツ作成プログラム、コンテンツ編集プログラム、および携帯通信端末 |
JP2005267655A (ja) | 2002-08-29 | 2005-09-29 | Sharp Corp | コンテンツ再生装置、コンテンツ再生方法、コンテンツ再生プログラム、コンテンツ再生プログラムを記録した記録媒体、および携帯通信端末 |
US7321682B2 (en) | 2002-11-12 | 2008-01-22 | Namco Bandai Games, Inc. | Image generation system, image generation method, program, and information storage medium |
JP4365573B2 (ja) | 2002-11-13 | 2009-11-18 | 株式会社ソフィア | 遊技機 |
EP1628490A1 (en) * | 2003-05-27 | 2006-02-22 | Sanyo Electric Co., Ltd. | Image display device and program |
AU2003252270A1 (en) | 2003-05-30 | 2005-01-21 | Lattice Technology, Inc. | 3-dimensional graphics data display device |
JP4764305B2 (ja) | 2006-10-02 | 2011-08-31 | 株式会社東芝 | 立体画像生成装置、方法およびプログラム |
JP4320030B2 (ja) * | 2006-11-02 | 2009-08-26 | 株式会社ソフィア | 遊技機 |
JP4836814B2 (ja) * | 2007-01-30 | 2011-12-14 | 株式会社東芝 | 3dディスプレイ用cg画像生成装置、3dディスプレイ用cg画像生成方法、及びプログラム |
-
2011
- 2011-02-22 US US13/258,730 patent/US8866887B2/en not_active Expired - Fee Related
- 2011-02-22 JP JP2011534967A patent/JP5628819B2/ja not_active Expired - Fee Related
- 2011-02-22 WO PCT/JP2011/000978 patent/WO2011105048A1/ja active Application Filing
- 2011-02-22 CN CN201180001535.5A patent/CN102362294B/zh not_active Expired - Fee Related
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2001076180A (ja) * | 1999-09-07 | 2001-03-23 | Nec Corp | 3次元グラフィックス表示装置 |
JP2004178581A (ja) * | 2002-11-12 | 2004-06-24 | Namco Ltd | 画像生成システム、画像生成方法、プログラム及び情報記憶媒体 |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2015008373A1 (ja) * | 2013-07-19 | 2015-01-22 | 富士通株式会社 | 情報処理装置、検査範囲の計算方法、及びプログラム |
JPWO2015008373A1 (ja) * | 2013-07-19 | 2017-03-02 | 富士通株式会社 | 情報処理装置、検査範囲の計算方法、及びプログラム |
US10133256B2 (en) | 2013-07-19 | 2018-11-20 | Fujitsu Limited | Information processing apparatus and method for calculating inspection ranges |
Also Published As
Publication number | Publication date |
---|---|
US20120013616A1 (en) | 2012-01-19 |
CN102362294A (zh) | 2012-02-22 |
JP5628819B2 (ja) | 2014-11-19 |
JPWO2011105048A1 (ja) | 2013-06-20 |
CN102362294B (zh) | 2015-07-22 |
US8866887B2 (en) | 2014-10-21 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP5628819B2 (ja) | コンピュータ・グラフィックス映像合成装置と方法、及び表示装置 | |
US8750599B2 (en) | Stereoscopic image processing method and apparatus | |
US8791989B2 (en) | Image processing apparatus, image processing method, recording method, and recording medium | |
Tsai et al. | A real-time 1080p 2D-to-3D video conversion system | |
US20120056885A1 (en) | Image generation system, image generation method, and information storage medium | |
JP2011090400A (ja) | 画像表示装置および方法、並びにプログラム | |
TW201921921A (zh) | 基於紋理圖與網格之3d影像資訊的處理 | |
WO2015196791A1 (zh) | 双目三维图形渲染方法及相关*** | |
US9495795B2 (en) | Image recording device, three-dimensional image reproducing device, image recording method, and three-dimensional image reproducing method | |
US11417060B2 (en) | Stereoscopic rendering of virtual 3D objects | |
US20190266802A1 (en) | Display of Visual Data with a Virtual Reality Headset | |
US20210382313A1 (en) | Image generation appratus, head-mounted display, content processing system, and image display method | |
JP2009163724A (ja) | グラフィックスインターフェイス、グラフィックスデータをラスタ化する方法およびコンピュータ読み取り可能な記録媒体 | |
US10771758B2 (en) | Immersive viewing using a planar array of cameras | |
CN114513646A (zh) | 一种三维虚拟场景中全景视频的生成方法及设备 | |
US8976171B2 (en) | Depth estimation data generating apparatus, depth estimation data generating method, and depth estimation data generating program, and pseudo three-dimensional image generating apparatus, pseudo three-dimensional image generating method, and pseudo three-dimensional image generating program | |
JP6166985B2 (ja) | 画像生成装置および画像生成プログラム | |
WO2012132267A1 (ja) | 全方向ステレオ画像出力装置 | |
TWM630947U (zh) | 立體影像播放裝置 | |
TW202332263A (zh) | 立體影像播放裝置及其立體影像產生方法 | |
JP7365183B2 (ja) | 画像生成装置、ヘッドマウントディスプレイ、コンテンツ処理システム、および画像表示方法 | |
Ludé | New Standards for Immersive Storytelling through Light Field Displays | |
Stone | Omnidirectional Stereoscopic Projections for VR. | |
CN116708737A (zh) | 立体影像播放装置及其立体影像产生方法 | |
Marrinan et al. | Image Synthesis from a Collection of Depth Enhanced Panoramas: Creating Interactive Extended Reality Experiences from Static Images |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 201180001535.5 Country of ref document: CN |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2011534967 Country of ref document: JP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 13258730 Country of ref document: US |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 11747020 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 11747020 Country of ref document: EP Kind code of ref document: A1 |