US20170094267A1 - Stereoscopic image reproduction device and method, stereoscopic image capturing device, and stereoscopic display device - Google Patents
Stereoscopic image reproduction device and method, stereoscopic image capturing device, and stereoscopic display device Download PDFInfo
- Publication number
- US20170094267A1 US20170094267A1 US15/377,353 US201615377353A US2017094267A1 US 20170094267 A1 US20170094267 A1 US 20170094267A1 US 201615377353 A US201615377353 A US 201615377353A US 2017094267 A1 US2017094267 A1 US 2017094267A1
- Authority
- US
- United States
- Prior art keywords
- image
- parallax
- display
- viewpoint
- stereoscopic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 238000000034 method Methods 0.000 title claims description 14
- 238000006073 displacement reaction Methods 0.000 claims abstract description 53
- 238000012545 processing Methods 0.000 claims description 79
- 230000004927 fusion Effects 0.000 claims description 42
- 230000006835 compression Effects 0.000 abstract description 25
- 238000007906 compression Methods 0.000 abstract description 25
- 238000010586 diagram Methods 0.000 description 31
- 239000003550 marker Substances 0.000 description 19
- 239000004973 liquid crystal related substance Substances 0.000 description 16
- 230000003287 optical effect Effects 0.000 description 15
- 210000003128 head Anatomy 0.000 description 11
- 238000004891 communication Methods 0.000 description 10
- 238000012937 correction Methods 0.000 description 8
- 238000001514 detection method Methods 0.000 description 8
- 230000000007 visual effect Effects 0.000 description 8
- 238000003384 imaging method Methods 0.000 description 7
- 230000033001 locomotion Effects 0.000 description 7
- 230000004888 barrier function Effects 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 239000011521 glass Substances 0.000 description 4
- 238000009825 accumulation Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000011156 evaluation Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 230000002411 adverse Effects 0.000 description 1
- 230000003321 amplification Effects 0.000 description 1
- 208000003464 asthenopia Diseases 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 230000002950 deficient Effects 0.000 description 1
- 230000007274 generation of a signal involved in cell-cell signaling Effects 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 230000010287 polarization Effects 0.000 description 1
- 238000001454 recorded image Methods 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
-
- H04N13/0497—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
-
- H04N13/0022—
-
- H04N13/0033—
-
- H04N13/0055—
-
- H04N13/0066—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/128—Adjusting depth or disparity
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/144—Processing image signals for flicker reduction
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/172—Processing image signals image signals comprising non-image signal components, e.g. headers or format information
- H04N13/178—Metadata, e.g. disparity information
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/189—Recording image signals; Reproducing recorded image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/356—Image reproducers having separate monoscopic and stereoscopic modes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/356—Image reproducers having separate monoscopic and stereoscopic modes
- H04N13/359—Switching between monoscopic and stereoscopic modes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/398—Synchronisation thereof; Control thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440218—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440263—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by altering the spatial resolution, e.g. for displaying on a connected PDA
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/76—Television signal recording
- H04N5/765—Interface circuits between an apparatus for recording and another apparatus
- H04N5/77—Interface circuits between an apparatus for recording and another apparatus between a recording apparatus and a television camera
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/76—Television signal recording
- H04N5/907—Television signal recording using static stores, e.g. storage tubes or semiconductor memories
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N9/00—Details of colour television systems
- H04N9/79—Processing of colour television signals in connection with recording
- H04N9/80—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback
- H04N9/82—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback the individual colour picture signal components being recorded simultaneously only
- H04N9/8205—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback the individual colour picture signal components being recorded simultaneously only involving the multiplexing of an additional signal and the colour video signal
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N9/00—Details of colour television systems
- H04N9/79—Processing of colour television signals in connection with recording
- H04N9/80—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback
- H04N9/82—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback the individual colour picture signal components being recorded simultaneously only
- H04N9/8205—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback the individual colour picture signal components being recorded simultaneously only involving the multiplexing of an additional signal and the colour video signal
- H04N9/8227—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback the individual colour picture signal components being recorded simultaneously only involving the multiplexing of an additional signal and the colour video signal the additional signal being at least another television signal
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2370/00—Aspects of data communication
- G09G2370/10—Use of a protocol of communication by packets in interfaces along the display data pipeline
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2370/00—Aspects of data communication
- G09G2370/12—Use of DVI or HDMI protocol in interfaces along the display data pipeline
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2380/00—Specific applications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/361—Reproducing mixed stereoscopic images; Reproducing mixed monoscopic and stereoscopic images, e.g. a stereoscopic image overlay window on a monoscopic image background
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N2213/00—Details of stereoscopic systems
- H04N2213/002—Eyestrain reduction by processing stereoscopic signals or controlling stereoscopic devices
Definitions
- the present invention relates to a stereoscopic image reproduction device and method, a stereoscopic image capturing device, and a stereoscopic display device, and in particular, to a technique for appropriately reproducing and displaying a stereoscopic image recorded in a three-dimensional image file.
- a stereoscopic image capturing device photographs the same subject from left and right viewpoints using two image capturing units (image capturing part) arranged left and right with parallax, acquires an image for a left eye and an image for a right eye, and records the images in a recording medium.
- the acquired left and right images are read from the recording medium, are input to a 3D display which can perform three-dimensional (3D) display, and are displayed such that the image for the left eye and the image for the right eye are visible separately with the left and right eyes.
- 3D display can perform three-dimensional (3D) display
- the 3D displays have various screen sizes, and the amount of parallax of the recorded left and right image may not be appropriate for the size of the 3D display on which the left and right images are reproduced and displayed.
- the degree of protrusion or recession becomes excessive, and there is a problem in that the images may not be recognized as a natural stereoscopic image.
- an intermediate viewpoint image which should be viewed from an arbitrary viewpoint is generated from an image for a left eye and an image for a right eye.
- JP2003-333621A discloses a technique in which information regarding the imaging size of a camera capturing an image and the distance between the lens center and the imaging surface is transmitted in the transmission of images of two or more viewpoints. According to this technique, information regarding the viewing angle during imaging can be obtained on the display side.
- an object of the invention is to provide a stereoscopic image reproduction device and method, a stereoscopic image capturing device, and a stereoscopic display device which display a stereoscopic image with an appropriate amount of parallax on the basis of auxiliary information recorded in a three-dimensional image file.
- a stereoscopic image reproduction device includes first acquisition part for reading a three-dimensional image file having recorded therein a plurality of viewpoint images for stereoscopic vision and auxiliary information for the plurality of viewpoint images, the auxiliary information including the maximum display size enabling binocular fusion when the plurality of viewpoint images are displayed on a stereoscopic display, and acquiring the plurality of viewpoint images and the auxiliary information from the three-dimensional image file, second acquisition part for acquiring the display size of the stereoscopic display as an output destination, determination part for comparing the acquired display size of the stereoscopic display with the maximum display size and determining whether or not the display size of the stereoscopic display is greater than the maximum display size, image processing part for generating a plurality of viewpoint images with smaller parallax on at least a distant view side between the plurality of viewpoint images on the basis of the plurality of viewpoint images acquired by the first acquisition part if it is determined that the display size of the
- distal view side refers to a distal side away from the image capturing means toward the subject
- near view side refers to a proximal side close to the image capturing part from the subject contrary to the distal side.
- the display size of the stereoscopic display is compared with the maximum display size acquired from the three-dimensional image file, and if it is determined that the display size of the stereoscopic display is greater, a plurality of viewpoint images with smaller parallax on at least the distant view side are generated and output to the stereoscopic display. Therefore, a stereoscopic image can be displayed with an appropriate amount of parallax.
- a stereoscopic image reproduction device includes first acquisition part for reading a three-dimensional image file having recorded therein a plurality of viewpoint images for stereoscopic vision and auxiliary information including the maximum amount of parallax on a distant view side in the plurality of viewpoint images, and acquiring the plurality of viewpoint images and the auxiliary information from the three-dimensional image file, second acquisition part for acquiring the display size of a stereoscopic display as an output destination, determination part for determining whether or not binocular fusion is enabled when a plurality of viewpoint images are displayed on the stereoscopic display on the basis of the acquired maximum amount of parallax on the distant view side, the display size of the stereoscopic display, and a predetermined value representing an interocular distance of a person, image processing part for generating a plurality of viewpoint images with smaller parallax on the distant view side between the a plurality of viewpoint images on the basis of the plurality of viewpoint images acquired by the
- the second aspect of the invention it is determined whether or not binocular fusion is enabled when a plurality of viewpoint images acquired from the three-dimensional image file are displayed on the stereoscopic display on the basis of the maximum amount of parallax on at least the distant view side acquired from the three-dimensional image file, the display size of the stereoscopic display, and the predetermined value representing the interocular distance of the person, and when it is determined that binocular fusion is disabled, a plurality of viewpoint images with smaller parallax on at least the distant view side are generated and output to the stereoscopic display. Therefore, a stereoscopic image can be displayed with an appropriate amount of parallax.
- the determination part has image deviation amount calculation part for calculating the amount of image deviation on the stereoscopic display corresponding to the maximum amount of parallax when the plurality of viewpoint images are displayed on the stereoscopic display on the basis of the acquired maximum amount of parallax on the distant view side and the display size of the stereoscopic display, and determines whether or not binocular fusion is enabled depending on whether or not the calculated amount of image deviation exceeds the predetermined value representing the interocular distance of the person.
- the image processing part performs parallax displacement between the plurality of viewpoint images such that the maximum amount of parallax on the distant view side of the plurality of viewpoint images on the stereoscopic display is equal to or smaller than a predetermined value representing an interocular distance of a person, and generating viewpoint shift images subjected to the parallax displacement.
- viewpoint images can be appropriately generated.
- the auxiliary information of the three-dimensional image file includes the maximum amount of parallax on a near view side in the plurality of viewpoint images
- the stereoscopic image reproduction device further includes addition part for adding the maximum amount of parallax on the near view side of the plurality of viewpoint images on the stereoscopic display and the amount of deviations of the viewpoint shift images on the stereoscopic display, and determination part for determining whether or not the amount of image deviation corresponding to the added maximum amount of parallax on the near view side exceeds a binocular fusion limit when being displayed on the stereoscopic display, and if the determination part determines that the amount of image deviation exceeds the binocular fusion limit, the output part outputs one of the plurality of viewpoint images acquired by the first acquisition part to the stereoscopic display to display a two-dimensional image.
- the image processing part has parallax amount calculation part for calculating the amount of parallax representing the amount of deviation between feature points having common features from the plurality of viewpoint images acquired by the first acquisition part, and virtual viewpoint image generation part for generating one or a plurality of virtual viewpoint images corresponding to an arbitrary virtual viewpoint on the basis of at least one viewpoint image from the plurality of viewpoint images acquired by the first acquisition part and the amount of parallax calculated by the parallax amount calculation part.
- the output part outputs one of the plurality of viewpoint images acquired by the first acquisition part to the stereoscopic display to display a two-dimensional image if the determination part determines that the display size of the stereoscopic display is greater than the maximum display size or if the calculated amount of image deviation exceeds a predetermined value representing an interocular distance of a person, and outputs the virtual viewpoint images instead of the two-dimensional image if the virtual viewpoint images are generated by the image processing part.
- the stereoscopic image reproduction device further includes display image generation part for generating a plurality of display images of size smaller than the viewpoint images on the basis of the plurality of viewpoint images, wherein the image processing part generates a plurality of display images with smaller parallax on at least the distant view side between the plurality of display images using the plurality of generated display images instead of the plurality of viewpoint images acquired by the first acquisition part, and the output part outputs the display images generated by the display image generation part or the display images generated by the image processing part to the stereoscopic display.
- images of appropriate size can be output to the stereoscopic display.
- the stereoscopic image reproduction device further includes recording part for adding or overwriting at least one of the viewpoint images generated by the image processing part and the display images generated by the display image generation part to the three-dimensional image file.
- a stereoscopic image capturing device includes image capturing part for acquiring a plurality of viewpoint images of the same subject from a plurality of viewpoints, parallax amount calculation part for calculating the amount of parallax representing the amount of displacement between feature points having common features from the plurality of acquired viewpoint images, maximum parallax amount acquisition part for acquiring the maximum amount of parallax on a distant view side from the calculated amount of parallax of each feature point, maximum display size acquisition part for acquiring the maximum display size enabling binocular fusion when a stereoscopic image based on the plurality of viewpoint images is displayed on a stereoscopic display on the basis of the acquired maximum amount of parallax on the distant view side, recording part for generating a three-dimensional image file in which the plurality of viewpoint images are recorded, and recording the three-dimensional image file in a recording medium, the recording part recording the plurality of viewpoint images in the three-dimensional image file and also
- a stereoscopic image can be displayed with an appropriate amount of parallax.
- a stereoscopic display device includes a stereoscopic display as an output destination, and the above-described stereoscopic image reproduction device.
- a stereoscopic image can be displayed with an appropriate amount of parallax.
- a stereoscopic image reproduction method includes first acquisition part for reading a three-dimensional image file having recorded therein a plurality of viewpoint images for stereoscopic vision and auxiliary information for the plurality of viewpoint images, the auxiliary information including the maximum display size enabling binocular fusion when the plurality of viewpoint images are displayed on a stereoscopic display, and acquiring the plurality of viewpoint images and the auxiliary information from the three-dimensional image file, a second acquisition step of acquiring the display size of the stereoscopic display as an output destination, a determination step of comparing the acquired display size of the stereoscopic display with the maximum display size and determining whether or not the display size of the stereoscopic display is greater than the maximum display size, an image processing step of generating a plurality of viewpoint images with smaller parallax on at least a distant view side between the plurality of viewpoint images on the basis of the plurality of viewpoint images acquired in the first acquisition step if it is determined that the
- a stereoscopic image reproduction method includes a first acquisition step of reading a three-dimensional image file having recorded therein a plurality of viewpoint images for stereoscopic vision and auxiliary information for the plurality of viewpoint images, the auxiliary information including the maximum amount of parallax on a distant view side in the plurality of viewpoint images, and acquiring the plurality of viewpoint images and the auxiliary information from the three-dimensional image file, a second acquisition step of acquiring the display size of a stereoscopic display as an output destination, a determination step of determining whether or not binocular fusion is enabled when a plurality of viewpoint images are displayed on the stereoscopic display on the basis of the acquired maximum amount of parallax on the distant view side, the display size of the stereoscopic display, and a predetermined value representing an interocular distance of a person, an image processing step of generating a plurality of viewpoint images with smaller parallax on at least the distant view side between
- a stereoscopic image with an appropriate amount of parallax can be displayed on the basis of auxiliary information recorded in a three-dimensional image file.
- FIG. 1 is a diagram showing a condition in which two viewpoint images are photographed.
- FIG. 2 is a diagram schematically showing the data structure of a 3D image file.
- FIG. 3 is a flowchart showing photographing and recording processing.
- FIGS. 4A and 4B are diagrams showing an example of two viewpoint images.
- FIG. 5 is a diagram showing a condition in which four viewpoint images are photographed.
- FIG. 6 is a diagram schematically showing the data structure of a 3D image file.
- FIGS. 7A to 7C are diagrams schematically showing another form of the data structure of a 3D image file.
- FIG. 8 is a diagram illustrating a virtual viewpoint.
- FIGS. 9A and 9B are diagrams schematically showing the data structure of a 3D image file.
- FIG. 10 is a diagram illustrating the principle of parallax displacement.
- FIGS. 11A and 11B are diagrams showing left and right viewpoint images and parallax displacement.
- FIG. 12 is a flowchart showing photographing and recording processing.
- FIG. 13 is a diagram schematically showing the data structure of a 3D image file.
- FIG. 14 is a flowchart showing reproduction and display processing.
- FIG. 15 is a flowchart showing reproduction and display processing based on the maximum amount of parallax.
- FIG. 16 is a flowchart showing processing for recording a 3D image file.
- FIG. 17A and 17B are diagrams schematically showing the data structure of a 3D image file.
- FIG. 18 is a flowchart showing processing for performing 2D display.
- FIG. 19 is a flowchart showing processing taking into consideration the maximum amount of parallax (near view).
- FIG. 20A and 20B are diagrams showing the appearance of a stereoscopic image capturing device.
- FIG. 21 is a block diagram showing the internal configuration of a stereoscopic image capturing device.
- FIG. 22 is a diagram showing the overall structure of a stereoscopic image reproduction device and a 3D display.
- FIG. 23 is a block diagram showing the internal configuration of a stereoscopic image reproduction device.
- a stereoscopic image reproduction device of the invention includes first acquisition part, second acquisition part, determination part, image processing part, and output part.
- the first acquisition part reads a three-dimensional image file having recorded therein a plurality of viewpoint images for stereoscopic vision and auxiliary information for the plurality of viewpoint images, auxiliary information including the maximum display size enabling binocular fusion when the plurality of viewpoint images are displayed on a stereoscopic display, and acquires the plurality of viewpoint images and the auxiliary information from the three-dimensional image file (first acquisition step).
- the second acquisition part acquires the display size of the stereoscopic display as an output destination (second acquisition step).
- the determination part compares the acquired display size of the stereoscopic display with the maximum display size to determine whether or not the display size of the stereoscopic display is greater than the maximum display size (determination step).
- the image processing part generates a plurality of viewpoint images with smaller parallax on at least a distant view side between the plurality of viewpoint images on the basis of the plurality of viewpoint images which are determined that the display size of the stereoscopic display is greater than the maximum display size and acquired by the first acquisition part (image processing step).
- the output part outputs the viewpoint images generated by the image processing part to the stereoscopic display, and outputs the viewpoint images acquired by the first acquisition part to the stereoscopic display if it is determined that the display size of the stereoscopic display is equal to or smaller than the maximum display size (output step).
- the display size of the stereoscopic display is compared with the maximum display size acquired from the three-dimensional image file, and when it is determined that the display size of the stereoscopic display is greater, a plurality of viewpoint images with smaller parallax on at least the distant view side can be generated and output to the stereoscopic display. As a result, it becomes possible to display a stereoscopic image with an appropriate amount of parallax.
- the second acquisition part acquires the display size of the stereoscopic display as an output destination
- the output part outputs the viewpoint images generated by the image processing part to the stereoscopic display if the determination part determines that binocular fusion is disabled and outputs the viewpoint images acquired by the first acquisition part to the stereoscopic display if it is determined that binocular fusion is enabled.
- binocular fusion it is determined whether or not binocular fusion is enabled when a plurality of viewpoint images acquired from the three-dimensional image file are displayed on the stereoscopic display on the basis of the maximum amount of parallax on at least the distant view side acquired from the three-dimensional image file, the display size of the stereoscopic display, and the predetermined value representing the interocular distance of the person, and when it is determined that binocular fusion is disabled, it becomes possible to generate a plurality of viewpoint images with smaller parallax on at least the distant view side and to output a plurality of viewpoint images to the stereoscopic display. As a result, a stereoscopic image can be displayed with an appropriate amount of parallax.
- the determination part has image deviation amount calculation part for calculating the amount of image deviation on the stereoscopic display corresponding to the maximum amount of parallax when a plurality of viewpoint images are displayed on the stereoscopic display on the basis of the acquired maximum amount of parallax on the distant view side and the display size of the stereoscopic display, and it is determined whether or not binocular fusion is enabled depending on whether or not the calculated amount of image deviation exceeds the predetermined value representing the interocular distance of the person. This is because it becomes possible to appropriately determine whether or not binocular fusion is enabled.
- the image processing part performs parallax displacement between the plurality of viewpoint images such that the maximum amount of parallax on the distant view side of the plurality of viewpoint images on the stereoscopic display is equal to or smaller than the predetermined value representing the interocular distance of the person, and generates viewpoint shift images subjected to the parallax displacement. This is because it becomes possible to appropriately generate viewpoint images.
- the auxiliary information of the three-dimensional image file includes the maximum amount of parallax on the near view side in the plurality of viewpoint images
- the stereoscopic image reproduction device further includes addition part for adding the maximum amount of parallax on the near view side of the plurality of viewpoint images on the stereoscopic display and the amount of deviations of the viewpoint shift images on the stereoscopic display, and determination part for determining whether or not the amount of image deviation corresponding to the added maximum amount of parallax on the near view side exceeds a binocular fusion limit when being displayed on the stereoscopic display, and if the determination part determines that the amount of image deviation exceeds the binocular fusion limit, the output part outputs one of the plurality of viewpoint images acquired by the first acquisition part to the stereoscopic display to display a two-dimensional image.
- the image processing part includes parallax amount calculation part for calculating the amount of parallax representing the amount of deviation between feature points having common features from the plurality of viewpoint images acquired by the first acquisition part, and virtual viewpoint image generation part for generating one or a plurality of virtual viewpoint images corresponding to an arbitrary virtual viewpoint on the basis of at least one viewpoint image from the plurality of viewpoint images acquired by the first acquisition part and the amount of parallax calculated by the parallax amount calculation part. This is because it becomes possible to generate appropriate viewpoint images.
- the output part outputs one of the plurality of viewpoint images acquired by the first acquisition part to the stereoscopic display to display a two-dimensional image if the determination part determines that the display size of the stereoscopic display is greater than the maximum display size or if the calculated amount of image deviation exceeds a predetermined value representing an interocular distance of a person, and outputs the virtual viewpoint images instead of the two-dimensional image if the virtual viewpoint images are generated by the image processing part. This is because, even when the generation of the virtual viewpoint images requires a lot of time, it becomes possible to display images on the stereoscopic display.
- the stereoscopic image reproduction device further includes display image generation part for generating a plurality of display images of size smaller than the viewpoint images on the basis of the plurality of viewpoint images, wherein the image processing part generates a plurality of display images with smaller parallax on at least the distant view side between the plurality of display images using the plurality of generated display images instead of the plurality of viewpoint images acquired by the first acquisition part, and the output part outputs the display images generated by the display image generation part or the display images generated by the image processing part to the stereoscopic display. This is because it becomes possible to output images of appropriate size to the stereoscopic display.
- the stereoscopic image reproduction device further includes recording part for adding or overwriting at least one of the viewpoint images generated by the image processing part and the display images generated by the display image generation part to the three-dimensional image file. This is because it becomes possible to display images with an appropriate amount of parallax without performing image processing during the next display.
- a 3D image file according to a first embodiment will be described.
- FIG. 1 is a diagram showing a condition in which two viewpoint images are photographed from different viewpoints with respect to a subject 100 using image capturing part of two 2 image capturing devices 101 - 2 and 101 - 3 (image capturing step).
- FIG. 2 is a diagram schematically showing the data structure of a 3D image file in which two viewpoint images 201 - 2 and 201 - 3 photographed by the image capturing part of the image capturing devices 101 - 2 and 101 - 3 shown in FIG. 1 are recorded.
- the 3D image file shown in FIG. 2 is an MP file based on an MP format, and a plurality of viewpoint images of the same subjects are connected and collected in one file.
- an SOI Start of Image
- EOI End of Image
- Each piece of viewpoint image data has an image region and an auxiliary information region.
- auxiliary information including a maximum display size (width and height, unit: mm) acquired during image recording, a visual distance (the distance between a viewer who observes a stereoscopic image and a display) (unit: mm) supposed during image recording, a maximum amount of parallax (near view) (%), a maximum amount of parallax (distant view) (%), a convergence angle of a device photographing each viewpoint image, a base line length, an image capturing unit arrangement (viewpoint number), and the number of times of photographing when each viewpoint image is acquired is recorded.
- Photographing and recording processing for recording such a 3D image file will be described with reference to the flowchart of FIG. 3 .
- a plurality of viewpoint images are acquired (Step S 11 ).
- photographing is performed using the two image capturing devices 101 - 2 and 101 - 3 one by one with respect to the subject 100 to acquire the two viewpoint images 201 - 2 and 201 - 3 .
- the image capturing device 101 - 2 has a viewpoint number 2
- the image capturing device 101 - 3 has a viewpoint number 3 .
- a plurality of feature points are extracted from the two viewpoint images 201 - 2 and 201 - 3 using parallax amount calculation part (Step S 12 ), and the amount of parallax of each feature point is calculated (Step S 13 ).
- the amount of parallax indicates the difference in the distance from the left end of each viewpoint image of the corresponding feature points between the viewpoint images, and the unit is pixel.
- the maximum amount of parallax on the distant view side and the maximum amount of parallax on the near view side are acquired from the calculated amount of parallax of each feature point using maximum parallax amount acquisition part (Step S 14 ).
- FIGS. 4A and 4B are diagrams showing an example of two viewpoint images.
- FIG. 4A shows a left viewpoint image 201 - 2
- FIG. 4B shows a right viewpoint image 201 - 3 .
- the maximum amount of parallax on the near view side is 213 N
- the feature point (the position of the maximum amount of parallax (near view)) having this amount of parallax is 211 N and 212 N
- the maximum amount of parallax on the distant view side is 213 F
- the feature point (the position of the maximum amount of parallax (distant view)) having this amount of parallax is 211 F and 212 F.
- the ratios (%) of the maximum amount of parallax 213 N on the near view side and the maximum amount of parallax 213 F on the distant view side with respect to the image width become a maximum amount of parallax (near view) (%) and a maximum amount of parallax (distant view) (%).
- the maximum display size is acquired on the basis of the calculated maximum amount of parallax using the maximum display size acquisition part (Step S 15 ). If the amount of parallax on the distant view side exceeds the interocular distance of a person, binocular fusion is disabled at the position of the image, and stereoscopic vision cannot be achieved. Accordingly, the maximum display size of which parallax does not exceed the interocular distance of the person is acquired from the ratio (%) of the maximum amount of parallax (distant view) with respect to the image width calculated in Step S 14 .
- the maximum display width which is allowed for stereoscopic vision becomes 500 mm. That is, in the case of a display having a width equal to or smaller than 500 mm, display is performed such that the amount of parallax of the maximum parallax position (distant view) does not exceed the interocular distance 50 mm, and as a result, the viewer can view stereoscopic vision.
- the maximum display height becomes 281.25 mm.
- the interocular distance of the person may be determined appropriately for the intended viewer. For example, when only an adult is intended, the value of a wide width of 65 mm or the like may be set.
- the maximum display width which is allowed for stereoscopic vision becomes about 433 mm.
- the maximum display height becomes about 244 mm.
- a table in which the maximum display size corresponding to the maximum amount of parallax is recorded may be prepared in advance, and the maximum display size may be acquired with reference to the table.
- the two viewpoint images 201 - 2 and 201 - 3 and the auxiliary information are recorded as one 3D image file using recording part (Step S 16 ).
- the maximum display size acquired in Step S 15 the supposed visual distance, the maximum amount of parallax (near view) (%) and the maximum amount of parallax (distant view) (%) acquired in Step S 14 , the convergence angle of the image capturing devices 101 - 2 and 101 - 3 in Step S 11 , the base line length, the viewpoint number, and the number of times of photographing are recorded to be included in the auxiliary information. Thereafter, image information of the viewpoint image 201 - 2 and the EOI marker are recorded.
- the SOI marker, the auxiliary information, the image information, and the EOI marker of the viewpoint image 201 - 3 are recorded.
- the auxiliary information region of the viewpoint image 201 - 3 only Exif auxiliary information may be recorded.
- the 3D image file shown in FIG. 2 can be recorded.
- the 3D image file recorded in this way is read by a stereoscopic image reproduction device when being displayed on a 3D display.
- the maximum display size recorded in the auxiliary information of the 3D image file is compared with the display size of the 3D display for display, thereby easily determining whether or not stereoscopic vision is possible. Accordingly, when it is determined that stereoscopic vision is possible, the recorded image may be displayed directly, and processing for adjusting the amount of parallax can be reduced depending on the screen size in the stereoscopic image reproduction device.
- the amount of parallax can be appropriately adjusted using information regarding the supposed visual distance or the number of times of photographing recorded in the auxiliary information.
- the maximum display size is determined on the basis of only the maximum amount of parallax on the distant view side
- the maximum display size may be determined taking into consideration the maximum amount of parallax on the near view side. Taking into consideration the maximum amount of parallax on the near view side, the maximum display size capable of appropriately achieving stereoscopic vision on the near view side as well as the distant view side can be determined.
- the display size with the amount of parallax of the near view equal to or smaller than 50 mm may be the maximum display size. This is because, if the amount of parallax on the near view side increases, the viewer feels fatigued in stereoscopic vision, and thus it is preferable that the amount of parallax is equal to or smaller than a predetermined amount.
- the number of viewpoints of a stereoscopic image according to the invention is not limited to two, and multi-viewpoint images may be used.
- FIG. 5 is a diagram showing a condition in which four viewpoint images are photographed from different viewpoints with respect to the subject 100 using four image capturing devices 101 - 1 to 101 - 4 . It is assumed that the viewpoint numbers of the image capturing devices 101 - 1 to 101 - 4 are 1 to 4 in order.
- FIG. 6 is a diagram schematically showing the data structure of a 3D image file in which four viewpoint images photographed by the four image capturing devices 101 - 1 to 101 - 4 are recorded.
- the 3D image file in addition to the four viewpoint images, an image for display obtained by reducing the size of each viewpoint image generated by display image generation part is recorded.
- an image for display a full HD image having resolution of 1920 ⁇ 1080 is used.
- these images are recorded in order of a viewpoint image (hereinafter, called a viewpoint image ( 1 )) of a viewpoint number 1 , an image (hereinafter, called a thumbnail image ( 1 )) for display of the viewpoint number 1 , a viewpoint image (hereinafter, called a viewpoint image ( 2 )) of a viewpoint number 2 , an image (hereinafter, called a thumbnail image ( 2 )) for display of the viewpoint number 2 , a viewpoint image (hereinafter, called a viewpoint image ( 3 )) of a viewpoint number 3 , an image (hereinafter, called a thumbnail image ( 3 )) for display of the viewpoint number 3 , a viewpoint image (hereinafter, called a viewpoint image ( 4 )) of a viewpoint number 4 , and an image (hereinafter, called a thumbnail image ( 4 )) for display of the viewpoint number 4 .
- a viewpoint image ( 1 ) an image for display of the viewpoint number 1
- the viewpoint image ( 1 ) photographed with the viewpoint number 1 as a reference viewpoint is recorded as a head image, and the recording region has an SOI marker, auxiliary information region, an image information region, and an EOI marker.
- auxiliary information in addition to Exif auxiliary information having photographing device information, information regarding a photographing mode, and the like, auxiliary information including a maximum display size, a supposed visual distance, a maximum amount of parallax (near view), a maximum amount of parallax (distant view), a convergence angle of a device photographing each viewpoint image, a base line length, an image capturing unit arrangement (viewpoint number), the number of times of photographing when each viewpoint image is acquired, and the like is recorded.
- auxiliary information including a maximum display size, a supposed visual distance, a maximum amount of parallax (near view), a maximum amount of parallax (distant view), a convergence angle of a device photographing each viewpoint image, a base line length, an image capturing unit arrangement (viewpoint number), the number of times of photographing when each viewpoint image is acquired, and the like is recorded.
- the maximum display size the maximum amount of parallax, and the like
- the values where the entire 3D image file is used are recorded. Specifically, the maximum amount of parallax calculated from a viewpoint image (in this case, the viewpoint image ( 4 )) with the greatest amount of parallax on the basis of the viewpoint image ( 1 ) photographed with the reference viewpoint, and the maximum display size acquired from the maximum amount of parallax are recorded.
- the convergence angle, the base line length, and the like the convergence angle and the base line length to a device photographing the same viewpoint image (in this case, the viewpoint image ( 4 )) are recorded.
- image information of the viewpoint image ( 1 ) is recorded, and the EOI marker is subsequently recorded.
- the thumbnail image ( 1 ) generated from the viewpoint image ( 1 ) is recorded, and as before, the recording region has an SOI marker, an auxiliary information region, an image information region, and an EOI marker.
- the auxiliary information region normal Exif auxiliary information is recorded.
- the recording region of the viewpoint image ( 2 ) also has an SOI marker, an auxiliary information region, an image information region, and an EOI marker.
- auxiliary information including a maximum display size calculated from the viewpoint image ( 2 ) and the viewpoint image ( 1 ) photographed with the reference viewpoint, a supposed visual distance, a maximum amount of parallax (near view), a maximum amount of parallax (distant view), a convergence angle of a device photographing two viewpoint images, a base line length, a viewpoint number, the number of times of photographing of two viewpoint images, and the like is included and recorded.
- the thumbnail image ( 2 ) generated from the viewpoint image ( 2 ) is recorded, and thereafter, the viewpoint image ( 3 ) is recorded.
- the recording region of the viewpoint image ( 3 ) also has an SOI marker, an auxiliary information region, an image information region, and an EOI marker.
- auxiliary information region in addition to normal Exif auxiliary information, the maximum display size calculated from the viewpoint image ( 3 ) and the viewpoint image ( 1 ) photographed with the reference viewpoint, and the like are recorded in the same way.
- the maximum display size as auxiliary information of each viewpoint image in the relation with the head image is recorded.
- the 3D image file recorded in this way is read by the stereoscopic image reproduction device when being displayed on the 3D display.
- the maximum display size of each viewpoint image recorded in the auxiliary information of the 3D image file is compared with the display size of the 3D display for display, thereby appropriately and easily determining whether or not stereoscopic vision is possible.
- the recording order of multi-viewpoint images is not limited to the order shown in FIG. 6 .
- the viewpoint image ( 1 ), the thumbnail image ( 1 ) to the thumbnail image ( 4 ), and the viewpoint image ( 2 ) to the viewpoint image ( 4 ) may be recorded in this order. If the thumbnail images for display are recorded earlier, image reading during file reading when display is performed can be quickened, and a required time until images are displayed on the 3D display can be reduced.
- Each viewpoint image is primarily used for printing, and since printing requires a predetermined time, even when viewpoint image is recorded in the second half portion of the file, there is little adverse effect.
- the recording order of the thumbnail images may be set such that recommended images when being displayed on the 3D display are recorded earlier. For example, if stereoscopic display using the thumbnail image ( 2 ) and the thumbnail image ( 3 ) is recommended, as shown in FIG. 7B , the thumbnail image ( 2 ) and the thumbnail image ( 3 ) may be recorded next to the viewpoint image ( 1 ) as the head image, and thereafter, the thumbnail image ( 1 ) and the thumbnail image ( 4 ) may be recorded.
- the thumbnail image ( 1 ) and the thumbnail image ( 3 ) may be recorded next to the viewpoint image ( 1 ) as the head image, and thereafter, the thumbnail image ( 2 ) and the thumbnail image ( 4 ) may be recorded.
- the maximum display size, supposed visual distance, and the maximum amount of parallax in the recommended images may be recorded in the auxiliary information of the viewpoint image ( 1 ) as the head image.
- All multi-viewpoint images in the second embodiment may not be images which are actually photographed, and may include virtual viewpoint images corresponding to virtual viewpoints.
- two viewpoint images are photographed from different viewpoints (viewpoint number 1 and viewpoint number 4 ) with respect to the subject 100 using two image capturing devices 101 - 1 and 101 - 4 .
- a viewpoint image 2 and a viewpoint image 3 of a viewpoint number 2 and a viewpoint number 3 with virtual viewpoints which are different from a viewpoint number 1 and a viewpoint number 4 and not actually present may be generated using virtual viewpoint image generation part.
- the method is not particularly limited.
- FIG. 9A is a diagram schematically showing the data structure of a 3D image file in which the viewpoint images acquired in this way are recorded.
- two viewpoint images ( 1 ) and ( 4 ) actually photographed, thumbnail images ( 1 ) and ( 4 ) as images for display of the viewpoint images, and thumbnail image ( 2 ) and ( 3 ) as images for display of the virtual viewpoint images are recorded.
- the viewpoint image ( 1 ) is recorded as a head image, then, the thumbnail image ( 2 ), the thumbnail image ( 3 ), the thumbnail image ( 1 ), and the thumbnail image ( 4 ) are recorded, and subsequently, the viewpoint image ( 4 ) is recorded.
- the thumbnail images may be recorded in a recommendation order, may be recorded in an arrangement order of viewpoints, or may be recorded in order from the centermost viewpoint.
- the thumbnail image ( 2 ) and the thumbnail image ( 3 ) may be generated from the thumbnail image ( 1 ) and the thumbnail image ( 4 ), or may be generated from the viewpoint image ( 1 ) and the viewpoint image ( 4 ).
- the recording region of each image has an SOI marker, an auxiliary information region, an image information region, and an EOI marker.
- viewpoint information representing whether each viewpoint number is a viewpoint (actual viewpoint) with which photographing is actually performed or a virtual viewpoint is recorded.
- Images for display may not be provided, and only viewpoint images for printing may be recorded.
- two viewpoint images ( 1 ) and ( 4 ) actually photographed and viewpoint images ( 2 ) and ( 3 ) which are virtual viewpoint images are recorded.
- the recording order may be a recommendation order of printing or display.
- the viewpoint image ( 2 ) is recorded as a head image, and subsequently, the viewpoint image ( 3 ), the viewpoint image ( 1 ), and the viewpoint image ( 4 ) are recorded.
- the viewpoint images may be recorded in an order of viewpoints.
- recording may be performed after the amount of parallax of images is adjusted such that stereoscopic vision is possible even when the images are displayed on the display of this size.
- parallax displacement is performed as the adjustment of the amount of parallax.
- FIG. 10 is a diagram illustrating the principle of parallax displacement.
- FIG. 11A is a diagram showing a left viewpoint image
- FIG. 11B is a diagram showing a right viewpoint image.
- the left eye of the viewer is at the coordinates (0,D)
- the right eye of the viewer is at the coordinates (X B ,D).
- parallax displacement is performed, the adjustment of the amount of parallax can be performed. Accordingly, if parallax displacement is performed when the amount of parallax on the distant view side exceeds the interocular distance of the person, the amount of parallax can fall within the interocular distance of the person, making it possible to appropriately perform stereoscopic vision.
- FIG. 12 is a flowchart showing a photographing and recording principle for recording a 3D image file according to this embodiment.
- a plurality of viewpoint images are acquired (Step S 21 ). Here, it is assumed that two viewpoint images are photographed.
- a plurality of feature points are extracted from each two viewpoint images (Step S 22 ), and the amount of parallax of each feature point is calculated (Step S 23 ).
- the maximum amount of parallax on the near view side and the maximum amount of parallax on the distant view side are acquired from the calculated amount of parallax of each feature point (Step S 24 ).
- the display width is acquired.
- the display width of the supposed display may be determined in advance and stored, or when a 3D display is connected through an external connection terminal, such as an HDMI terminal, the size of the 3D display may be read through communication based on the connection.
- Step S 25 it is determined whether or not the maximum parallax position on the distant view side can be stereoscopically viewed. Specifically, the product of the display width and the maximum amount of parallax (%) on the distant view side is calculated, and it is determined whether or not the calculated product is equal to or greater than the interocular distance of the person, 50 mm.
- the two photographed viewpoint images are recorded directly (Step S 27 ).
- the viewpoint images may be recorded as the 3D image file having the data structure shown in FIG. 2 , or the thumbnail images for display or virtual viewpoint images may be generated and recorded.
- parallax displacement is performed such that the product becomes smaller than 50 mm and the maximum parallax position on the distant view side can be stereoscopically viewed (Step S 26 ).
- parallax displacement may be performed by displacing the right viewpoint image in the left direction, or may be performed by displacing the left viewpoint image in the right direction.
- the left and right viewpoint images may be displaced in a direction in which the images are brought close to each other.
- FIG. 13 is a diagram schematically showing the data structure of a 3D image file to be recorded.
- the 3D image file in addition to two photographed original images, images for display and thumbnail images obtained by reducing the size of the original images after parallax displacement are recorded.
- the images for display are images of full HD size
- the thumbnail images are images of VGA size.
- the thumbnail images are used, for example, when being displayed for image search on a small liquid crystal display provided on the rear surface of the image capturing device.
- these images are recorded in order of the image for display of the viewpoint number 1 , the image for display of the viewpoint number 2 , the thumbnail image of the viewpoint number 1 , the thumbnail image of the viewpoint number 2 , the original image of the viewpoint number 1 , and the original image of the viewpoint number 2 , and the recording region of each image has an SOI marker (not shown in the drawing), an auxiliary information region, an image information region, and an EOI marker (not shown in the drawing).
- the image for display of the viewpoint number 1 as the reference viewpoint is recorded as a head image, and in the auxiliary information region, the maximum amount of parallax is recorded.
- the image for display of the viewpoint number 2 is next recorded, and this image is an image after parallax displacement is performed.
- this image is an image after parallax displacement is performed.
- information indicating that the parallax displacement processing is completed and the shift amount (unit: pixel) are recorded.
- the thumbnail image of the viewpoint number 1 is recorded, and in the auxiliary information region thereof, the maximum amount of parallax is recorded.
- the thumbnail image of the viewpoint number 2 is further recorded.
- the thumbnail image may be an image after parallax displacement, or may be an image generated from the original image of the viewpoint number 2 .
- the original image of the viewpoint number 1 and the original image of the viewpoint number 2 are recorded.
- the maximum amount of parallax is recorded.
- the maximum parallax position on the distant view side can be constantly stereoscopically viewed, and thus, the viewer can appropriately view stereoscopic vision.
- the supposed display size may be recorded in the auxiliary information region of the head image. If the supposed display size is recorded, in the stereoscopic image reproduction device, the read supposed display size is compared with the display size of the 3D display for display, thereby appropriately and easily determining whether or not stereoscopic vision is possible.
- Step S 31 a 3D image file is read.
- Step S 32 the size of a display on which a 3D image is now displayed is acquired.
- the maximum display size of which viewpoint images for 3D display can be appropriately 3D-displayed is acquired from the auxiliary information of the read 3D image file (Step S 33 ).
- Step S 34 the lateral width of the display on which the 3D image is now displayed is compared with the width of the maximum display size.
- the width of the actual display is greater, if the read viewpoint images are displayed directly, the amount of parallax of the maximum parallax position on the distant view side exceeds the interocular distance of the person, and this portion may not be stereoscopically viewed. Accordingly, it is necessary to adjust the amount of parallax of the left and right viewpoint images.
- parallax displacement or parallax compression adjusts the amount of parallax by displacing one or both of the left and right viewpoint images.
- parallax compression adjusts the amount of parallax by generating images of virtual viewpoints with the amount of parallax smaller than the left and right viewpoint images and displaying the generated images of the virtual viewpoints.
- parallax displacement or parallax compression may be determined in advance or may be selected by the user. In regard to the adjustment of the amount of parallax, even when either parallax displacement or parallax compression is used, the parallax on at least the distant view side can become small.
- the amount of parallax is adjusted to the amount of parallax suitable for the width of the display being used, and the adjusted images are 3D-displayed (Step S 36 ).
- the maximum display size recorded as the auxiliary information is read and compared with the display size of which display is actually performed.
- the maximum display size is greater, it is determined that there is no problem in stereoscopic vision, and the recorded 3D image is displayed directly.
- the display size of which display is actually performed is greater, it is determined that there is a region where stereoscopic vision is not possible, the recorded 3D image is displayed after the amount of parallax is adjusted, making it possible to constantly display an appropriate 3D image.
- Such processing may be performed by reading the maximum amount of parallax recorded in the auxiliary information of the 3D image file.
- FIG. 15 is a flowchart when reproduction and display processing is performed on the basis of the maximum amount of parallax.
- the portions common to the flowchart shown in FIG. 14 are represented by the same reference numerals, and detailed description thereof will not be repeated.
- the maximum amount of parallax (distant view) is acquired from the auxiliary information of the read 3D image file (Step S 41 ).
- the maximum amount of parallax (distant view) is recorded as the ratio (%) with respect to the image width.
- Step S 42 it is determined whether or not the product of the width of the display acquired in Step S 32 and the maximum amount of parallax (distant view) is equal to or greater than 50 mm.
- the product of the width of the display and the maximum amount of parallax becomes an actual amount of parallax at the maximum parallax amount position (distant view) on a displayed image.
- the amount of parallax becomes equal to or greater than the interocular distance of the person, 50 mm, this portion cannot be stereoscopically viewed. For this reason, after the adjustment of the amount by parallax displacement or parallax compression is performed for the read viewpoint images (Step S 35 ), 3D display is performed (Step S 36 ).
- reproduction and display can be controlled on the basis of the maximum amount of parallax (distant view) read from the 3D image file.
- this is effective when the maximum display size is not recorded in the auxiliary information of the 3D image file, and only the maximum amount of parallax is recorded.
- 3D display when it is determined that 3D display cannot be appropriately performed, 3D display is performed on the entire display after the amount of parallax is adjusted by parallax displacement or parallax compression, 3D display may be performed in an image size with an appropriate amount of parallax, that is, an image size corresponding to the maximum display size.
- FIG. 16 is a flowchart when processing for recording images in a 3D image file is performed.
- images generated by parallax displacement or parallax compression in Step S 35 are added or rewritten to a 3D image file as thumbnail images for display in Step S 43 .
- the addition or rewriting to a 3D image file may be performed after 3D display in Step S 36 .
- FIGS. 17A and 17B are diagrams schematically showing the data structure of a 3D image file in which four viewpoint images and four thumbnail images for display generated from the viewpoint images are recorded.
- FIG. 17A shows a 3D image file when being read in Step S 31 of FIG. 16
- FIG. 17B shows a 3D image file after rewriting in Step S 43 .
- the maximum amount of parallax (distant view) of a viewpoint image ( 1 ) and a viewpoint image( 4 ) is recorded in the auxiliary information of the viewpoint image ( 1 ) as a head image.
- Step S 35 parallax displacement or parallax compression is performed.
- the thumbnail image ( 4 ) after parallax displacement is recorded instead of the thumbnail image ( 4 ).
- Information representing that the parallax displacement processing is completed and the shift amount (unit: pixel) are recorded in the auxiliary information of the thumbnail image ( 4 ) after parallax displacement.
- Step S 35 If parallax compression is performed in Step S 35 , the thumbnail image ( 4 ) after parallax compression is recorded as the thumbnail image ( 4 ). In this case, the maximum amount of parallax (distant view) after parallax compression may be recorded as the auxiliary information.
- thumbnail image ( 4 ) after parallax amount adjustment is recorded instead of the thumbnail image ( 4 )
- the thumbnail image ( 4 ) after parallax amount adjustment may be additionally recorded.
- thumbnail images for display may be generated and recorded as a 3D image file shown in FIG. 17B .
- Parallax adjustment is not limited to the thumbnail images, and the parallax adjustment of the viewpoint images may be performed. For example, if the parallax adjustment of the viewpoint image ( 4 ) is performed, the viewpoint image ( 4 ) and the thumbnail image ( 4 ) generated on the basis of the viewpoint image ( 4 ) may be overwritten or additionally recorded.
- the product of the width of the display and the maximum amount of parallax (distant view) calculated in Step S 42 may be recorded in the auxiliary information region of the head image of the 3D image file as the maximum display size.
- parallax compression processing generation of virtual viewpoint images
- 2D display is performed on the 3D display.
- FIG. 18 is a flowchart showing processing in which 2D display is performed.
- the portions common to the flowchart shown in FIG. 16 are represented by the reference numerals, and detailed description thereof will not be repeated.
- Step S 42 If it is determined in Step S 42 that the product of the width of the display and the maximum amount of parallax (distant view) is equal to or greater than the interocular distance of the person, 50 mm, either the viewpoint images or the thumbnail images for display recorded in the 3D image file are 2D-displayed on the display (Step S 51 ). Thereafter, the parallax compression processing is performed (Step S 35 ).
- the thumbnail images for display of the 3D image file are added or rewritten (Step S 43 ), and the images subjected to parallax compression are 3D-displayed on the display (Step S 36 ). After 3D display is performed, the thumbnail images for display may be recorded in the 3D image file.
- 2D display may be performed while the parallax displacement processing is performed. Simultaneously with 2D display, the effect that the parallax amount adjustment processing is performed may be displayed. If display is performed in this way, the viewer can know the reason for performing 2D display on the 3D display.
- the adjustment of the amount of parallax is performed on the basis of the maximum amount of parallax (distant view), and 3D display is performed, in regard to the adjustment of the amount of parallax, if parallax displacement is performed, there is a drawback in that the amount of parallax on the near view side increases. Accordingly, when parallax displacement is performed, display may be performed taking into consideration the maximum amount of parallax (near view).
- FIG. 19 is a flowchart when parallax displacement processing is performed taking into consideration the maximum amount of parallax (near view).
- the portions common to the flowchart shown in FIG. 16 are represented by the same reference numerals, and detailed description thereof will not be repeated.
- Step S 42 it is determined whether or not the product of the width of the display which is used for display and the maximum amount of parallax (distant view) is equal to or greater than the interocular distance of the person, 50 mm.
- the product is smaller than 50 mm, since there is no difficulty even when the 3D image recorded in the 3D image file is displayed directly, the read 3D image is displayed directly (Step S 64 ).
- a parallax displacement amount (necessary displacement amount) in which the product becomes smaller than 50 mm is calculated (Step S 61 ).
- Step S 62 when parallax displacement is performed by the calculated necessary displacement amount, it is determined whether or not the maximum parallax amount position on the near view side can be appropriately stereoscopically viewed. Specifically, it is determined whether or not a value obtained by adding the necessary displacement amount to the product of the width of the display which is used for display and the maximum amount of parallax (near view) is equal to or smaller than 50 mm (Step S 62 ). Although it is determined that stereoscopic vision is appropriately possible when the amount of parallax of the near view is equal to or smaller than 50 mm, the value of 50 mm may be appropriately determined.
- parallax displacement is executed (Step S 63 ), and the images subjected to parallax displacement are 3D-displayed on the 3D display (Step S 64 ).
- the images subjected to parallax displacement may be added to rewritten to the 3D image file.
- the adjustment of the amount of parallax is performed taking into consideration the maximum amount of parallax (near view) as well as the maximum amount of parallax (distant view), thereby displaying an appropriate 3D image.
- FIGS. 20A and 20B are diagrams showing the appearance of a stereoscopic image capturing device according to the invention.
- FIG. 20A is a perspective view when a stereoscopic image capturing device is viewed from the front side
- FIG. 20B is a rear view.
- a stereoscopic image capturing device (fly-eye camera) 10 is a digital camera in which 2D/3D still images and 2D/3D motion images can be recorded. As shown in FIGS. 20A and 20B , a shutter button 11 and a zoom button 12 are provided on the top surface of a thin rectangular parallelepiped camera body.
- a lens barrier 13 substantially having the same width as the width of the camera body in the left-right direction is provided on the front surface of the camera body to freely move in the up-down direction.
- the lens barrier 13 is moved in the up-down direction between a position indicated by a two-dot-chain line and a position indicated by a solid line to simultaneously open/close the front surfaces of a pair of left and right photographing optical systems 14 - 1 and 14 - 2 .
- zoom lenses of bent optical systems are used as the photographing optical systems 14 - 1 and 14 - 2 .
- a camera power supply can be turned on/off in conjunction with the opening/closing operation of the lens front surface by the lens barrier 13 .
- a liquid crystal monitor 16 for 3D is provided in the center portion on the rear surface of the camera body.
- the liquid crystal monitor 16 can display a plurality of parallax images (right viewpoint image and left viewpoint image) as directional images having predetermined directivity by a parallax barrier.
- a liquid crystal monitor using a lenticular lens a liquid crystal monitor in which a right viewpoint image and a left viewpoint image can be individually viewed using dedicated glasses, such as polarization glasses or liquid crystal shutter glasses, or the like may be applied.
- An operating switch 18 A is a selector switch for switching between still image photographing and motion image photographing
- an operating switch 18 B is a parallax adjustment switch for adjusting the amount of parallax of a right viewpoint image and a left viewpoint image
- an operating switch 18 C is a selector switch for switching between 2D imaging and 3D imaging.
- An operating switch 18 D is a seesaw key serving as a MENU/OK button and a play button
- an operating switch 18 E is a multi-function cross key
- an operating key 18 F is a DISP/BACK key.
- the MENU/OK button is an operating switch which has a function as a menu button for giving an instruction to display a menu on the screen of the liquid crystal monitor 16 and a function as an OK button for instructing the confirmation and execution of a selected content, or the like.
- the play button is a button which is used for switching from a photographing mode to a reproduction mode.
- the cross key is an operating switch which inputs instructions in four directions of up, down, left, and right, and a macro button, a flash button, a self-timer button, and the like are assigned. When the menu is selected, the cross key functions as a switch (cursor movement operating part) for switching an item from the menu screen or for instructing the selection of various setting items from each menu.
- the left/right key of the cross key functions as a frame-by-frame playback (forward/reverse playback) button in the reproduction mode.
- the DISP/BACK key is used to switch the display form of the liquid crystal monitor 16 , to cancel an instruction content on the menu screen, to return an operation state to a previous operation state, or the like.
- reference numeral 15 denotes a stereo microphone.
- FIG. 21 is a block diagram showing the internal configuration of the stereoscopic image capturing device 10 .
- the stereoscopic image capturing device 10 primarily has a plurality of image capturing units 20 - 1 and 20 - 2 , a central processing unit (CPU) 32 , an operating unit 34 including the shutter button 11 and the zoom button 12 described above and various operating switches, a display control unit 36 , a liquid crystal monitor 16 , a recording control unit 38 , a compression/expansion processing unit 42 , a digital signal processing unit 44 , an AE (Automatic Exposure) detection unit 46 , an AF (Auto Focus) detection unit 48 , an AWB (Automatic White Balance) detection unit 50 , a VRAM 52 , a RAM 54 , a ROM 56 , an EEPROM 58 , and the like.
- the image capturing units 20 - 1 and 20 - 2 capture two parallax images of an image for a left eye and an image for a right eye having parallax, three or more image capturing units 20 may be provided.
- the image capturing unit 20 - 1 which captures the image for the left eye includes a prism (not shown), a photographing optical system 14 - 1 ( FIGS. 20A and 20B ) having a focus lens/zoom lens 21 , optical units having an aperture diaphragm 22 and a mechanical shutter 23 , a solid-state image capturing element (CCD) 24 , an analog signal processing unit 25 , an A/D converter 26 , an image input controller 27 , a lens driving unit 28 , an aperture diaphragm driving unit 29 , and a shutter control unit 30 for driving the optical units, and a CCD control unit 31 for controlling the CCD 24 .
- the image capturing unit 20 - 2 which captures the image for the right eye has the same configuration as the image capturing unit 20 - 1 which captures the image for the left eye, thus description of a specific configuration will be omitted.
- the CPU 32 performs overall control of the operation of the entire camera in accordance with a predetermined control program on the basis of an input from the operating unit 34 .
- the calculation of the maximum display size, parallax displacement, the generation of the virtual viewpoint image, and the like are performed by the CPU 32 .
- the ROM 56 stores a control program executed by the CPU 32 and various kinds of data necessary for control
- the EEPROM 58 stores various kinds of information representing the adjustment result of adjustment before product shipment, for example, defective pixel information of the CCD 24 , correction parameters which are used for image processing, a correspondence table of the maximum amount of parallax and the maximum display size, and the like.
- the VRAM 52 is a memory which temporarily stores image data for display on the liquid crystal monitor 16
- the RAM 54 includes a computational work area of the CPU 32 and a temporary storage area of image data.
- the focus lens/zoom lens 21 included in the photographing optical system is driven by the lens driving unit 28 and moves back and forth along the optical axis.
- the CPU 32 controls the driving of the lens driving unit 28 to control the position of the focus lens and to perform focus adjustment such that the subject comes into focus, and controls the zoom position of the zoom lens in accordance with a zoom instruction from the zoom button 12 of the operating unit 34 to change a zoom magnification.
- the aperture diaphragm 22 has, for example, an iris diaphragm, and is driven and operated by the aperture diaphragm driving unit 29 .
- the CPU 32 controls the amount (aperture value) of opening of the aperture diaphragm 22 through the aperture diaphragm driving unit 29 , and controls the amount of incident light on the CCD 24 .
- the mechanical shutter 23 opens/closes the optical path to determine the exposure time in the CCD 24 , and prevents unwanted light from being incident on the CCD 24 when reading an image signal from the CCD 24 , thereby preventing the occurrence of smear.
- the CPU 32 outputs a shutter closing signal synchronized with the exposure end time corresponding to a shutter speed to the shutter control unit 30 , and controls the mechanical shutter 23 .
- the CCD 24 has a two-dimensional color CCD solid-state image capturing element. Multiple photodiodes are two-dimensionally arranged on the light-receiving surface of the CCD 24 , and color filters are disposed in each photodiode with a predetermined arrangement.
- the optical image of the subject focused on the CCD light-receiving surface through the optical units having the above-described configuration is converted to a signal charge according to the amount of incident light by the photodiodes.
- the signal charges accumulated in each photodiode are sequentially read from the CCD 24 as a voltage signal (image signal) according to the signal charges in accordance with an instruction of the CPU 32 on the basis of a driving pulse from the CCD control unit 31 .
- the CCD 24 has an electronic shutter function, and controls a charge accumulation time into the photodiode to control the exposure time (shutter speed).
- the charge accumulation start time corresponding to the shutter speed is controlled by the electronic shutter, and the mechanical shutter 23 is closed to control the exposure end time (charge accumulation end time).
- the CCD 24 is used as an image capturing element, an image capturing element having a different configuration, such as a CMOS sensor, may be used.
- R, G, and B analog signals read from the CCD 24 are subjected to double-correlated sampling (CDS) or amplification by the analog signal processing unit 25 , and converted to R, and B digital signals by the A/D converter 26 .
- CDS double-correlated sampling
- the image input controller 27 has an internal line buffer of a predetermined capacity, temporarily accumulates R, G, and B image signals (CCD RAW data) A/D converted by the A/D converter 26 , and stores the R, G, and B image signals in the RAM 54 through a bus 60 .
- the CPU 32 controls the image capturing unit 20 - 2 which captures the right viewpoint image in the same manner as the image capturing unit 20 - 1 which captures the left viewpoint image.
- the AE detection unit 46 calculates subject luminance necessary for AE control on the basis of the image signals introduced when the shutter button 11 is half-pressed, and outputs a signal representing subject luminance (photographing EV value) to the CPU 32 .
- the CPU 32 sets the shutter speed (exposure time), the aperture value, and photographing sensitivity in a plurality of image capturing units 20 - 1 and 20 - 2 in accordance with a predetermined program diagram on the basis of the input photographing EV value.
- the AF detection unit 48 integrates the absolute value of a high-frequency component of an image signal of an AF area introduced when the shutter button 11 is half-pressed, and outputs the integrated value (AF evaluation value) to the CPU 32 .
- the CPU 32 moves the focus lens from a close range side to an infinite side, searches for a focusing position where the AF evaluation value detected by the AF detection unit 48 is maximal, and moves the focus lens to the focusing position, thereby performing focus adjustment on the subject (main subject).
- so-called mountain-climbing control is performed to move the focus lens such that the AF evaluation value constantly has the maximum value.
- the AWB detection unit 50 automatically obtains a light source type (color temperature of field) on the basis of the R, G, and B image signals acquired during this image capturing, and reads a corresponding white balance gain from a table which stores R, G, and B white balance gains (white balance correction values) according to the light source type in advance.
- a light source type color temperature of field
- the digital signal processing unit 44 includes a white balance correction circuit, a gradation conversion processing circuit (for example, a gamma correction circuit), a synchronization circuit which interpolates spatial deviations of color signals of R, G, B, and the like depending on the color filter arrangement of a single-plate CCD to position the color signals, a contour correction circuit, a luminance/color difference signal generation circuit, and the like, and performs image processing on the R, G, and B image signals (CCD RAW data) stored in the RAM 54 .
- a white balance correction circuit for example, a gamma correction circuit
- a synchronization circuit which interpolates spatial deviations of color signals of R, G, B, and the like depending on the color filter arrangement of a single-plate CCD to position the color signals
- a contour correction circuit for example, a luminance/color difference signal generation circuit, and the like
- R, G, and B CCD RAW data are subjected to white balance correction by multiplying the white balance gains detected by the AWB detection unit 50 in the digital signal processing unit 44 , then subjected to the predetermined processing, such as gradation conversion processing (for example, gamma correction), and converted to YC signals having a luminance signal (Y signal) and a color-difference signal (Cr and Cb signals).
- the YC signals processed by the digital signal processing unit 44 are stored in the RAM 54 .
- the digital signal processing unit 44 also includes a distortion correction circuit which corrects lens distortion of the photographing optical systems of a plurality of image capturing units 20 - 1 and 20 - 2 , and an image clipping circuit which clips images of predetermined clipping areas from the left and right viewpoint images to correct optical axis deviations of the photographing optical systems of a plurality of image capturing units 20 - 1 and 20 - 2 .
- the compression/expansion processing unit 42 compresses the YC signals stored in the RAM 54 in accordance with an instruction from the CPU 32 during recording in a memory card 40 , or expands compressed data recorded in the memory card 40 to obtain the YC signals.
- the recording control unit 38 converts compressed data compressed by the compression/expansion processing unit 42 to an image file of a predetermined format (for example, a 3D still image becomes an MP file, and a 3D motion image becomes a motion image file of motion PEG H.264, MPEG4, or MPEG4-MVC), records the image file in the memory card 40 , or reads an image file from the memory card 40 .
- a predetermined format for example, a 3D still image becomes an MP file, and a 3D motion image becomes a motion image file of motion PEG H.264, MPEG4, or MPEG4-MVC
- the recording control unit 38 records the maximum display size, the supposed visual distance, the maximum amount of parallax (near view) (%), the maximum amount of parallax (distant view) (%), and the like as auxiliary information in the memory card 40 along with Exif auxiliary information, when recording an MP file according to the invention.
- the liquid crystal monitor 16 is used as an image display unit which displays photographed images, and is also used as a GUI (Graphical User Interface) during various settings.
- the liquid crystal monitor 16 is also used as an electronic view-finder which displays live view images (hereinafter, referred to as “through images” for confirming an image angle in the photographing mode.
- the display control unit 36 alternately displays a left viewpoint image and a right viewpoint image held in the VRAM 52 pixel by pixel.
- left and right images alternately arranged pixel by pixel are separately viewed by the left and right eyes of the user who observes at a predetermined distance. Accordingly, stereoscopic vision is possible.
- the embodiment can be realized.
- the stereoscopic image capturing device 10 described herein includes the two image capturing units for photographing images of left and right two viewpoints, three or more image capturing units may be provided to photograph three or more viewpoint images.
- four image capturing units such as the four image capturing devices 101 - 1 to 101 - 4 shown in FIG. 5 , may be provided to photograph four viewpoint images.
- Fifth to eighth embodiments may be realized in a stereoscopic image reproduction device in which no image capturing unit is provided.
- FIG. 22 is a diagram showing the overall configuration of a stereoscopic image reproduction device 300 and a 3D display 320 according to the invention. As shown in FIG. 22 , the stereoscopic image reproduction device 300 and the 3D display 320 are separate devices and are communicably connected together by a communication cable 310 .
- the 3D display 320 is a parallax barrier-type or a lenticular-type display, and a left viewpoint image and a right viewpoint image input from the stereoscopic image reproduction device 300 to the 3D display 320 are alternately displayed line by line.
- the 3D display 320 may alternately switch and display the left viewpoint image and the right viewpoint image in a temporal manner. In this case, the viewer views the 3D display 320 using special glasses.
- FIG. 23 is a block diagram showing the internal configuration of the stereoscopic image reproduction device 300 .
- the stereoscopic image reproduction device 300 includes a CPU 301 , a recording control unit 305 , a memory card 306 , a display control unit 307 , a communication interface 308 , and the like.
- the CPU 301 performs overall control of the operation of the entire stereoscopic image reproduction device 300 on the basis of a control program recorded in a ROM 302 .
- a RAM 303 is used as a computational work area of the CPU 301 .
- the recording control unit 305 and the display control unit 307 are connected to the CPU 301 through a bus 304 .
- the recording control unit 305 controls data reading and writing of a 3D image file with respect to the memory card 306 .
- the memory card 306 is, for example, the same as the memory card 40 of the stereoscopic image capturing device 10 shown in FIG. 21 , and records a 3D image file including the viewpoint images photographed in the stereoscopic image capturing device 10 and auxiliary information.
- the communication interface 308 is a connector unit to which a communication cable 310 is connected, and the display control unit 307 displays a stereoscopic image on the 3D display 320 through the communication interface 308 and the communication cable 310 .
- the communication interface 308 and the communication cable 310 those based on the HDMI standard may be used.
- the stereoscopic image reproduction device 300 can acquire the display size of the 3D display 320 connected thereto through the communication cable 310 .
- Fly-eye image capturing part for photographing viewpoint images may be provided in the stereoscopic image reproduction device 300 , and the photographed viewpoint images may be recorded in the memory card 306 .
- the stereoscopic image reproduction device 300 and the 3D display 320 may be constituted as a single device.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Library & Information Science (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
- Processing Or Creating Images (AREA)
Abstract
A stereoscopic image is displayed with an appropriate amount of parallax on the basis of auxiliary information recorded in a three-dimensional image file. A 2D image file is read (Step S31), and the size of a display which performs 2D display is acquired (Step S32). The maximum display size capable of appropriately performing 2D display of a viewpoint image is acquired from metadata of the read 2D image file (Step S33). The size acquired in Step S32 is compared with the maximum display size acquired in Step S33 (Step S34). When the maximum display size is smaller, parallax displacement or parallax compression is performed, (Step S35), and then 2D display is performed (Step S36). Therefore, display can be appropriately performed on the basis of the maximum display size.
Description
- This application is a Divisional of copending application Ser. No. 13/659,638, filed on Oct. 24, 2012, which is a Continuation of International Application No. PCT/JP2011/060095 filed on Apr. 25, 2011, which claims the benefit of Patent Appliction No. 2010-103905 filed in Japan, on Apr. 28, 2010. The entire contents of all of the above application are hereby incorporated by reference into the present application.
- 1. Field of the Invention
- The present invention relates to a stereoscopic image reproduction device and method, a stereoscopic image capturing device, and a stereoscopic display device, and in particular, to a technique for appropriately reproducing and displaying a stereoscopic image recorded in a three-dimensional image file.
- 2. Description of the Related Art
- A stereoscopic image capturing device photographs the same subject from left and right viewpoints using two image capturing units (image capturing part) arranged left and right with parallax, acquires an image for a left eye and an image for a right eye, and records the images in a recording medium. The acquired left and right images are read from the recording medium, are input to a 3D display which can perform three-dimensional (3D) display, and are displayed such that the image for the left eye and the image for the right eye are visible separately with the left and right eyes. In this way, the images can be recognized as a stereoscopic image.
- On the other hand, the 3D displays have various screen sizes, and the amount of parallax of the recorded left and right image may not be appropriate for the size of the 3D display on which the left and right images are reproduced and displayed. In this case, the degree of protrusion or recession becomes excessive, and there is a problem in that the images may not be recognized as a natural stereoscopic image.
- In regard to this problem, in a reproduction device, an intermediate viewpoint image which should be viewed from an arbitrary viewpoint is generated from an image for a left eye and an image for a right eye. In this way, in order to generate an intermediate viewpoint image from a plurality of images of different viewpoints, it is necessary to obtain pixel correspondence between the images to estimate the depth.
- However, there is a problem in that, when display is performed without causing eye fatigue (for example, parallax control), a depth range is not known. When performing such display, it is necessary to appropriately determine an observation distance along with the conditions of the viewing angle or the size of an imaging surface during photographing, the distance between the lens center and the imaging surface, and the like.
- In order to solve this problem, JP2003-333621A discloses a technique in which information regarding the imaging size of a camera capturing an image and the distance between the lens center and the imaging surface is transmitted in the transmission of images of two or more viewpoints. According to this technique, information regarding the viewing angle during imaging can be obtained on the display side.
- However, in the technique of JP2003-333621A, there is a problem in that processing on the display device is complicated. Furthermore, since the technique relates to data transmission, when a stereoscopic image is recorded in a recording medium, how information is recorded is not disclosed.
- The invention has been accomplished in consideration of this situation, and an object of the invention is to provide a stereoscopic image reproduction device and method, a stereoscopic image capturing device, and a stereoscopic display device which display a stereoscopic image with an appropriate amount of parallax on the basis of auxiliary information recorded in a three-dimensional image file.
- In order to attain the above-described object, a stereoscopic image reproduction device according to a first aspect of the invention includes first acquisition part for reading a three-dimensional image file having recorded therein a plurality of viewpoint images for stereoscopic vision and auxiliary information for the plurality of viewpoint images, the auxiliary information including the maximum display size enabling binocular fusion when the plurality of viewpoint images are displayed on a stereoscopic display, and acquiring the plurality of viewpoint images and the auxiliary information from the three-dimensional image file, second acquisition part for acquiring the display size of the stereoscopic display as an output destination, determination part for comparing the acquired display size of the stereoscopic display with the maximum display size and determining whether or not the display size of the stereoscopic display is greater than the maximum display size, image processing part for generating a plurality of viewpoint images with smaller parallax on at least a distant view side between the plurality of viewpoint images on the basis of the plurality of viewpoint images acquired by the first acquisition part if it is determined that the display size of the stereoscopic display is greater than the maximum display size, and output part for outputting the viewpoint images generated by the image processing part to the stereoscopic display if the determination part determines that the display size of the stereoscopic display is greater than the maximum display size and outputting the viewpoint images acquired by the first acquisition part to the stereoscopic display if it is determined that the display size of the stereoscopic display is equal to or smaller than the maximum display size. The term “distant view side” used herein refers to a distal side away from the image capturing means toward the subject, and the term “near view side” refers to a proximal side close to the image capturing part from the subject contrary to the distal side.
- According to the first aspect of the invention, the display size of the stereoscopic display is compared with the maximum display size acquired from the three-dimensional image file, and if it is determined that the display size of the stereoscopic display is greater, a plurality of viewpoint images with smaller parallax on at least the distant view side are generated and output to the stereoscopic display. Therefore, a stereoscopic image can be displayed with an appropriate amount of parallax.
- In order to attain the above-described object, a stereoscopic image reproduction device according to a second aspect of the invention includes first acquisition part for reading a three-dimensional image file having recorded therein a plurality of viewpoint images for stereoscopic vision and auxiliary information including the maximum amount of parallax on a distant view side in the plurality of viewpoint images, and acquiring the plurality of viewpoint images and the auxiliary information from the three-dimensional image file, second acquisition part for acquiring the display size of a stereoscopic display as an output destination, determination part for determining whether or not binocular fusion is enabled when a plurality of viewpoint images are displayed on the stereoscopic display on the basis of the acquired maximum amount of parallax on the distant view side, the display size of the stereoscopic display, and a predetermined value representing an interocular distance of a person, image processing part for generating a plurality of viewpoint images with smaller parallax on the distant view side between the a plurality of viewpoint images on the basis of the plurality of viewpoint images acquired by the first acquisition part if the determination part determined that binocular fusion is disabled, and output part for outputting the viewpoint images generated by the image processing part to the stereoscopic display if the determination part determines that binocular fusion is disabled and outputting the viewpoint images acquired by the first acquisition part to the stereoscopic display if it is determined that binocular fusion is enabled.
- According to the second aspect of the invention, it is determined whether or not binocular fusion is enabled when a plurality of viewpoint images acquired from the three-dimensional image file are displayed on the stereoscopic display on the basis of the maximum amount of parallax on at least the distant view side acquired from the three-dimensional image file, the display size of the stereoscopic display, and the predetermined value representing the interocular distance of the person, and when it is determined that binocular fusion is disabled, a plurality of viewpoint images with smaller parallax on at least the distant view side are generated and output to the stereoscopic display. Therefore, a stereoscopic image can be displayed with an appropriate amount of parallax.
- In the stereoscopic image reproduction device according to the second aspect of the invention, it is preferable that the determination part has image deviation amount calculation part for calculating the amount of image deviation on the stereoscopic display corresponding to the maximum amount of parallax when the plurality of viewpoint images are displayed on the stereoscopic display on the basis of the acquired maximum amount of parallax on the distant view side and the display size of the stereoscopic display, and determines whether or not binocular fusion is enabled depending on whether or not the calculated amount of image deviation exceeds the predetermined value representing the interocular distance of the person.
- Accordingly, it is possible to appropriately determine whether or not binocular fusion is enabled.
- In the stereoscopic image reproduction device according to the first and second aspects of the invention, it is preferable that the image processing part performs parallax displacement between the plurality of viewpoint images such that the maximum amount of parallax on the distant view side of the plurality of viewpoint images on the stereoscopic display is equal to or smaller than a predetermined value representing an interocular distance of a person, and generating viewpoint shift images subjected to the parallax displacement.
- Accordingly, viewpoint images can be appropriately generated.
- In the stereoscopic image reproduction device according to the first and second aspects of the invention, it is preferable that the auxiliary information of the three-dimensional image file includes the maximum amount of parallax on a near view side in the plurality of viewpoint images, the stereoscopic image reproduction device further includes addition part for adding the maximum amount of parallax on the near view side of the plurality of viewpoint images on the stereoscopic display and the amount of deviations of the viewpoint shift images on the stereoscopic display, and determination part for determining whether or not the amount of image deviation corresponding to the added maximum amount of parallax on the near view side exceeds a binocular fusion limit when being displayed on the stereoscopic display, and if the determination part determines that the amount of image deviation exceeds the binocular fusion limit, the output part outputs one of the plurality of viewpoint images acquired by the first acquisition part to the stereoscopic display to display a two-dimensional image.
- Accordingly, appropriate images can be displayed.
- In the stereoscopic image reproduction device according to the first and second aspects of the invention, it is preferable that the image processing part has parallax amount calculation part for calculating the amount of parallax representing the amount of deviation between feature points having common features from the plurality of viewpoint images acquired by the first acquisition part, and virtual viewpoint image generation part for generating one or a plurality of virtual viewpoint images corresponding to an arbitrary virtual viewpoint on the basis of at least one viewpoint image from the plurality of viewpoint images acquired by the first acquisition part and the amount of parallax calculated by the parallax amount calculation part.
- Accordingly, appropriate viewpoint images can be generated.
- At this time, in the stereoscopic image reproduction device according to the first and second aspects of the invention, it is preferable that the output part outputs one of the plurality of viewpoint images acquired by the first acquisition part to the stereoscopic display to display a two-dimensional image if the determination part determines that the display size of the stereoscopic display is greater than the maximum display size or if the calculated amount of image deviation exceeds a predetermined value representing an interocular distance of a person, and outputs the virtual viewpoint images instead of the two-dimensional image if the virtual viewpoint images are generated by the image processing part.
- Accordingly, even when the generation of the virtual viewpoint images requires a lot of time, images can be displayed on the stereoscopic display.
- It is preferable that the stereoscopic image reproduction device according to the first and second aspects of the invention further includes display image generation part for generating a plurality of display images of size smaller than the viewpoint images on the basis of the plurality of viewpoint images, wherein the image processing part generates a plurality of display images with smaller parallax on at least the distant view side between the plurality of display images using the plurality of generated display images instead of the plurality of viewpoint images acquired by the first acquisition part, and the output part outputs the display images generated by the display image generation part or the display images generated by the image processing part to the stereoscopic display.
- Accordingly, images of appropriate size can be output to the stereoscopic display.
- It is preferable that the stereoscopic image reproduction device according to the first and second aspects of the invention further includes recording part for adding or overwriting at least one of the viewpoint images generated by the image processing part and the display images generated by the display image generation part to the three-dimensional image file.
- Accordingly, images with an appropriate amount of parallax can be displayed during the next display without image processing.
- In order to attain the above-described object, a stereoscopic image capturing device according to a third aspect of the invention includes image capturing part for acquiring a plurality of viewpoint images of the same subject from a plurality of viewpoints, parallax amount calculation part for calculating the amount of parallax representing the amount of displacement between feature points having common features from the plurality of acquired viewpoint images, maximum parallax amount acquisition part for acquiring the maximum amount of parallax on a distant view side from the calculated amount of parallax of each feature point, maximum display size acquisition part for acquiring the maximum display size enabling binocular fusion when a stereoscopic image based on the plurality of viewpoint images is displayed on a stereoscopic display on the basis of the acquired maximum amount of parallax on the distant view side, recording part for generating a three-dimensional image file in which the plurality of viewpoint images are recorded, and recording the three-dimensional image file in a recording medium, the recording part recording the plurality of viewpoint images in the three-dimensional image file and also recording the maximum amount of parallax on the distant view side and the maximum display size in the three-dimensional image file as auxiliary information, and the above-described stereoscopic image reproduction device.
- Accordingly, a stereoscopic image can be displayed with an appropriate amount of parallax.
- In order to attain the above-described object, a stereoscopic display device according to a fourth aspect of the invention includes a stereoscopic display as an output destination, and the above-described stereoscopic image reproduction device.
- Accordingly, a stereoscopic image can be displayed with an appropriate amount of parallax.
- In order to attain the above-described object, a stereoscopic image reproduction method according to a fifth aspect of the invention includes first acquisition part for reading a three-dimensional image file having recorded therein a plurality of viewpoint images for stereoscopic vision and auxiliary information for the plurality of viewpoint images, the auxiliary information including the maximum display size enabling binocular fusion when the plurality of viewpoint images are displayed on a stereoscopic display, and acquiring the plurality of viewpoint images and the auxiliary information from the three-dimensional image file, a second acquisition step of acquiring the display size of the stereoscopic display as an output destination, a determination step of comparing the acquired display size of the stereoscopic display with the maximum display size and determining whether or not the display size of the stereoscopic display is greater than the maximum display size, an image processing step of generating a plurality of viewpoint images with smaller parallax on at least a distant view side between the plurality of viewpoint images on the basis of the plurality of viewpoint images acquired in the first acquisition step if it is determined that the display size of the stereoscopic display is greater than the maximum display size, and an output step of outputting the viewpoint images generated in the image processing step to the stereoscopic display if it is determined in the determination step that the display size of the stereoscopic display is greater than the maximum display size and outputting the viewpoint images acquired in the first acquisition step to the stereoscopic display if it is determined that the display size of the stereoscopic display is equal to or smaller than the maximum display size.
- In order to attain the above-described object, a stereoscopic image reproduction method according to a sixth aspect of the invention includes a first acquisition step of reading a three-dimensional image file having recorded therein a plurality of viewpoint images for stereoscopic vision and auxiliary information for the plurality of viewpoint images, the auxiliary information including the maximum amount of parallax on a distant view side in the plurality of viewpoint images, and acquiring the plurality of viewpoint images and the auxiliary information from the three-dimensional image file, a second acquisition step of acquiring the display size of a stereoscopic display as an output destination, a determination step of determining whether or not binocular fusion is enabled when a plurality of viewpoint images are displayed on the stereoscopic display on the basis of the acquired maximum amount of parallax on the distant view side, the display size of the stereoscopic display, and a predetermined value representing an interocular distance of a person, an image processing step of generating a plurality of viewpoint images with smaller parallax on at least the distant view side between the plurality of viewpoint images on the basis of the plurality of viewpoint images acquired in the first acquisition step if it is determined in the determination step that binocular fusion is disabled, and an output step of outputting the viewpoint images generated in the image processing step to the stereoscopic display if it is determined in the determination step that binocular fusion is disabled and outputting the viewpoint images acquired in the first acquisition step to the stereoscopic display if it is determined that binocular fusion is enabled.
- According to the invention, a stereoscopic image with an appropriate amount of parallax can be displayed on the basis of auxiliary information recorded in a three-dimensional image file.
-
FIG. 1 is a diagram showing a condition in which two viewpoint images are photographed. -
FIG. 2 is a diagram schematically showing the data structure of a 3D image file. -
FIG. 3 is a flowchart showing photographing and recording processing. -
FIGS. 4A and 4B are diagrams showing an example of two viewpoint images. -
FIG. 5 is a diagram showing a condition in which four viewpoint images are photographed. -
FIG. 6 is a diagram schematically showing the data structure of a 3D image file. -
FIGS. 7A to 7C are diagrams schematically showing another form of the data structure of a 3D image file. -
FIG. 8 is a diagram illustrating a virtual viewpoint. -
FIGS. 9A and 9B are diagrams schematically showing the data structure of a 3D image file. -
FIG. 10 is a diagram illustrating the principle of parallax displacement. -
FIGS. 11A and 11B are diagrams showing left and right viewpoint images and parallax displacement. -
FIG. 12 is a flowchart showing photographing and recording processing. -
FIG. 13 is a diagram schematically showing the data structure of a 3D image file. -
FIG. 14 is a flowchart showing reproduction and display processing. -
FIG. 15 is a flowchart showing reproduction and display processing based on the maximum amount of parallax. -
FIG. 16 is a flowchart showing processing for recording a 3D image file. -
FIG. 17A and 17B are diagrams schematically showing the data structure of a 3D image file. -
FIG. 18 is a flowchart showing processing for performing 2D display. -
FIG. 19 is a flowchart showing processing taking into consideration the maximum amount of parallax (near view). -
FIG. 20A and 20B are diagrams showing the appearance of a stereoscopic image capturing device. -
FIG. 21 is a block diagram showing the internal configuration of a stereoscopic image capturing device. -
FIG. 22 is a diagram showing the overall structure of a stereoscopic image reproduction device and a 3D display. -
FIG. 23 is a block diagram showing the internal configuration of a stereoscopic image reproduction device. - A stereoscopic image reproduction device of the invention includes first acquisition part, second acquisition part, determination part, image processing part, and output part. First, the first acquisition part reads a three-dimensional image file having recorded therein a plurality of viewpoint images for stereoscopic vision and auxiliary information for the plurality of viewpoint images, auxiliary information including the maximum display size enabling binocular fusion when the plurality of viewpoint images are displayed on a stereoscopic display, and acquires the plurality of viewpoint images and the auxiliary information from the three-dimensional image file (first acquisition step). The second acquisition part acquires the display size of the stereoscopic display as an output destination (second acquisition step). Thereafter, the determination part compares the acquired display size of the stereoscopic display with the maximum display size to determine whether or not the display size of the stereoscopic display is greater than the maximum display size (determination step). The image processing part generates a plurality of viewpoint images with smaller parallax on at least a distant view side between the plurality of viewpoint images on the basis of the plurality of viewpoint images which are determined that the display size of the stereoscopic display is greater than the maximum display size and acquired by the first acquisition part (image processing step). Finally, the output part outputs the viewpoint images generated by the image processing part to the stereoscopic display, and outputs the viewpoint images acquired by the first acquisition part to the stereoscopic display if it is determined that the display size of the stereoscopic display is equal to or smaller than the maximum display size (output step). With the above-described configuration, the display size of the stereoscopic display is compared with the maximum display size acquired from the three-dimensional image file, and when it is determined that the display size of the stereoscopic display is greater, a plurality of viewpoint images with smaller parallax on at least the distant view side can be generated and output to the stereoscopic display. As a result, it becomes possible to display a stereoscopic image with an appropriate amount of parallax.
- Alternatively, after the first acquisition part acquires the plurality of viewpoint images and the auxiliary information, and the second acquisition part acquires the display size of the stereoscopic display as an output destination, it is determined whether or not binocular fusion is enabled when a plurality of viewpoint images are displayed on the stereoscopic display on the basis of the maximum amount of parallax on the distance view side acquired by the determination part, the display size of the stereoscopic display, and a predetermined value representing an interocular distance of a person. If the determination part determines that binocular fusion is disabled, a plurality of viewpoint images with smaller parallax on at least the distance view side between the plurality of viewpoint images are generated on the basis of the plurality of viewpoint images acquired by the first acquisition part using the image processing part. Thereafter, the output part outputs the viewpoint images generated by the image processing part to the stereoscopic display if the determination part determines that binocular fusion is disabled and outputs the viewpoint images acquired by the first acquisition part to the stereoscopic display if it is determined that binocular fusion is enabled. With this, it is determined whether or not binocular fusion is enabled when a plurality of viewpoint images acquired from the three-dimensional image file are displayed on the stereoscopic display on the basis of the maximum amount of parallax on at least the distant view side acquired from the three-dimensional image file, the display size of the stereoscopic display, and the predetermined value representing the interocular distance of the person, and when it is determined that binocular fusion is disabled, it becomes possible to generate a plurality of viewpoint images with smaller parallax on at least the distant view side and to output a plurality of viewpoint images to the stereoscopic display. As a result, a stereoscopic image can be displayed with an appropriate amount of parallax. At this time, it is preferable that the determination part has image deviation amount calculation part for calculating the amount of image deviation on the stereoscopic display corresponding to the maximum amount of parallax when a plurality of viewpoint images are displayed on the stereoscopic display on the basis of the acquired maximum amount of parallax on the distant view side and the display size of the stereoscopic display, and it is determined whether or not binocular fusion is enabled depending on whether or not the calculated amount of image deviation exceeds the predetermined value representing the interocular distance of the person. This is because it becomes possible to appropriately determine whether or not binocular fusion is enabled.
- In various stereoscopic image reproduction devices described above, it is preferable that the image processing part performs parallax displacement between the plurality of viewpoint images such that the maximum amount of parallax on the distant view side of the plurality of viewpoint images on the stereoscopic display is equal to or smaller than the predetermined value representing the interocular distance of the person, and generates viewpoint shift images subjected to the parallax displacement. This is because it becomes possible to appropriately generate viewpoint images. It is preferable that the auxiliary information of the three-dimensional image file includes the maximum amount of parallax on the near view side in the plurality of viewpoint images, the stereoscopic image reproduction device further includes addition part for adding the maximum amount of parallax on the near view side of the plurality of viewpoint images on the stereoscopic display and the amount of deviations of the viewpoint shift images on the stereoscopic display, and determination part for determining whether or not the amount of image deviation corresponding to the added maximum amount of parallax on the near view side exceeds a binocular fusion limit when being displayed on the stereoscopic display, and if the determination part determines that the amount of image deviation exceeds the binocular fusion limit, the output part outputs one of the plurality of viewpoint images acquired by the first acquisition part to the stereoscopic display to display a two-dimensional image. This is because it becomes possible to display appropriate images. It is preferable that the image processing part includes parallax amount calculation part for calculating the amount of parallax representing the amount of deviation between feature points having common features from the plurality of viewpoint images acquired by the first acquisition part, and virtual viewpoint image generation part for generating one or a plurality of virtual viewpoint images corresponding to an arbitrary virtual viewpoint on the basis of at least one viewpoint image from the plurality of viewpoint images acquired by the first acquisition part and the amount of parallax calculated by the parallax amount calculation part. This is because it becomes possible to generate appropriate viewpoint images. At this time, it is preferable that the output part outputs one of the plurality of viewpoint images acquired by the first acquisition part to the stereoscopic display to display a two-dimensional image if the determination part determines that the display size of the stereoscopic display is greater than the maximum display size or if the calculated amount of image deviation exceeds a predetermined value representing an interocular distance of a person, and outputs the virtual viewpoint images instead of the two-dimensional image if the virtual viewpoint images are generated by the image processing part. This is because, even when the generation of the virtual viewpoint images requires a lot of time, it becomes possible to display images on the stereoscopic display. It is preferable that the stereoscopic image reproduction device further includes display image generation part for generating a plurality of display images of size smaller than the viewpoint images on the basis of the plurality of viewpoint images, wherein the image processing part generates a plurality of display images with smaller parallax on at least the distant view side between the plurality of display images using the plurality of generated display images instead of the plurality of viewpoint images acquired by the first acquisition part, and the output part outputs the display images generated by the display image generation part or the display images generated by the image processing part to the stereoscopic display. This is because it becomes possible to output images of appropriate size to the stereoscopic display. It is preferable that the stereoscopic image reproduction device further includes recording part for adding or overwriting at least one of the viewpoint images generated by the image processing part and the display images generated by the display image generation part to the three-dimensional image file. This is because it becomes possible to display images with an appropriate amount of parallax without performing image processing during the next display.
- Hereinafter, preferred embodiments of the invention will be described with reference to the accompanying drawings.
- A 3D image file according to a first embodiment will be described.
-
FIG. 1 is a diagram showing a condition in which two viewpoint images are photographed from different viewpoints with respect to a subject 100 using image capturing part of two 2 image capturing devices 101-2 and 101-3 (image capturing step).FIG. 2 is a diagram schematically showing the data structure of a 3D image file in which two viewpoint images 201-2 and 201-3 photographed by the image capturing part of the image capturing devices 101-2 and 101-3 shown inFIG. 1 are recorded. - The 3D image file shown in
FIG. 2 is an MP file based on an MP format, and a plurality of viewpoint images of the same subjects are connected and collected in one file. In each of the connected viewpoint images, an SOI (Start of Image) marker is recorded at the top of data, and an EOI (End of Image) marker is recorded at the end of data. Accordingly, the read start position and end position of each image can be recognized. - Each piece of viewpoint image data has an image region and an auxiliary information region. In the auxiliary information region, in addition to Exif auxiliary information having photographing device information, information regarding a photographing mode, and the like, in this embodiment, auxiliary information including a maximum display size (width and height, unit: mm) acquired during image recording, a visual distance (the distance between a viewer who observes a stereoscopic image and a display) (unit: mm) supposed during image recording, a maximum amount of parallax (near view) (%), a maximum amount of parallax (distant view) (%), a convergence angle of a device photographing each viewpoint image, a base line length, an image capturing unit arrangement (viewpoint number), and the number of times of photographing when each viewpoint image is acquired is recorded.
- Photographing and recording processing for recording such a 3D image file will be described with reference to the flowchart of
FIG. 3 . - First, a plurality of viewpoint images are acquired (Step S11). Here, as shown in
FIG. 1 , photographing is performed using the two image capturing devices 101-2 and 101-3 one by one with respect to the subject 100 to acquire the two viewpoint images 201-2 and 201-3. Note that the image capturing device 101-2 has aviewpoint number 2, and the image capturing device 101-3 has aviewpoint number 3. - Next, a plurality of feature points are extracted from the two viewpoint images 201-2 and 201-3 using parallax amount calculation part (Step S12), and the amount of parallax of each feature point is calculated (Step S13). Here, the amount of parallax indicates the difference in the distance from the left end of each viewpoint image of the corresponding feature points between the viewpoint images, and the unit is pixel. The maximum amount of parallax on the distant view side and the maximum amount of parallax on the near view side are acquired from the calculated amount of parallax of each feature point using maximum parallax amount acquisition part (Step S14).
-
FIGS. 4A and 4B are diagrams showing an example of two viewpoint images.FIG. 4A shows a left viewpoint image 201-2, andFIG. 4B shows a right viewpoint image 201-3. - In the example of
FIGS. 4A and 4B , the maximum amount of parallax on the near view side is 213N, and the feature point (the position of the maximum amount of parallax (near view)) having this amount of parallax is 211N and 212N. The maximum amount of parallax on the distant view side is 213F, and the feature point (the position of the maximum amount of parallax (distant view)) having this amount of parallax is 211F and 212F. That is, when stereoscopic vision of a stereoscopic image based on the two viewpoint images 201-2 and 201-3 is achieved, the position of the maximum amount of parallax (near view) is viewed most closely, and the position of the maximum amount of parallax (distant view) is viewed most distantly. - The ratios (%) of the maximum amount of
parallax 213N on the near view side and the maximum amount ofparallax 213F on the distant view side with respect to the image width become a maximum amount of parallax (near view) (%) and a maximum amount of parallax (distant view) (%). - The maximum display size is acquired on the basis of the calculated maximum amount of parallax using the maximum display size acquisition part (Step S15). If the amount of parallax on the distant view side exceeds the interocular distance of a person, binocular fusion is disabled at the position of the image, and stereoscopic vision cannot be achieved. Accordingly, the maximum display size of which parallax does not exceed the interocular distance of the person is acquired from the ratio (%) of the maximum amount of parallax (distant view) with respect to the image width calculated in Step S14.
- For example, if it is assumed that the interocular distance of the person is 50 mm, and the ratio of the maximum amount of parallax (distant view) with respect to the image width is 10%, the maximum display width which is allowed for stereoscopic vision becomes 500 mm. That is, in the case of a display having a width equal to or smaller than 500 mm, display is performed such that the amount of parallax of the maximum parallax position (distant view) does not exceed the
interocular distance 50 mm, and as a result, the viewer can view stereoscopic vision. When a display having an aspect ratio of 16:9 is assumed, the maximum display height becomes 281.25 mm. - The interocular distance of the person may be determined appropriately for the intended viewer. For example, when only an adult is intended, the value of a wide width of 65 mm or the like may be set.
- For example, if it is assumed that the interocular distance of the person is 65 mm, and the ratio of the maximum amount of parallax (distant view) with respect to the image width is 15%, the maximum display width which is allowed for stereoscopic vision becomes about 433 mm. When a display having an aspect ratio of 16:9 is assumed, the maximum display height becomes about 244 mm.
- Instead of calculation using the interocular distance of the person, a table in which the maximum display size corresponding to the maximum amount of parallax is recorded may be prepared in advance, and the maximum display size may be acquired with reference to the table.
- Finally, as shown in
FIG. 2 , the two viewpoint images 201-2 and 201-3 and the auxiliary information are recorded as one 3D image file using recording part (Step S16). - That is, subsequent to the SOI marker for the viewpoint image 201-2, the maximum display size acquired in Step S15, the supposed visual distance, the maximum amount of parallax (near view) (%) and the maximum amount of parallax (distant view) (%) acquired in Step S14, the convergence angle of the image capturing devices 101-2 and 101-3 in Step S11, the base line length, the viewpoint number, and the number of times of photographing are recorded to be included in the auxiliary information. Thereafter, image information of the viewpoint image 201-2 and the EOI marker are recorded.
- The SOI marker, the auxiliary information, the image information, and the EOI marker of the viewpoint image 201-3 are recorded. In the auxiliary information region of the viewpoint image 201-3, only Exif auxiliary information may be recorded.
- As described above, the 3D image file shown in
FIG. 2 can be recorded. - The 3D image file recorded in this way is read by a stereoscopic image reproduction device when being displayed on a 3D display. At this time, in the stereoscopic image reproduction device, the maximum display size recorded in the auxiliary information of the 3D image file is compared with the display size of the 3D display for display, thereby easily determining whether or not stereoscopic vision is possible. Accordingly, when it is determined that stereoscopic vision is possible, the recorded image may be displayed directly, and processing for adjusting the amount of parallax can be reduced depending on the screen size in the stereoscopic image reproduction device.
- When it is necessary to adjust the amount of parallax in the stereoscopic image reproduction device, the amount of parallax can be appropriately adjusted using information regarding the supposed visual distance or the number of times of photographing recorded in the auxiliary information.
- Although in this embodiment, the maximum display size is determined on the basis of only the maximum amount of parallax on the distant view side, the maximum display size may be determined taking into consideration the maximum amount of parallax on the near view side. Taking into consideration the maximum amount of parallax on the near view side, the maximum display size capable of appropriately achieving stereoscopic vision on the near view side as well as the distant view side can be determined.
- For example, the display size with the amount of parallax of the near view equal to or smaller than 50 mm may be the maximum display size. This is because, if the amount of parallax on the near view side increases, the viewer feels fatigued in stereoscopic vision, and thus it is preferable that the amount of parallax is equal to or smaller than a predetermined amount.
- Although in the first embodiment, an example where two viewpoint images are photographed from two viewpoints has been described, the number of viewpoints of a stereoscopic image according to the invention is not limited to two, and multi-viewpoint images may be used.
-
FIG. 5 is a diagram showing a condition in which four viewpoint images are photographed from different viewpoints with respect to the subject 100 using four image capturing devices 101-1 to 101-4. It is assumed that the viewpoint numbers of the image capturing devices 101-1 to 101-4 are 1 to 4 in order. -
FIG. 6 is a diagram schematically showing the data structure of a 3D image file in which four viewpoint images photographed by the four image capturing devices 101-1 to 101-4 are recorded. In the 3D image file, in addition to the four viewpoint images, an image for display obtained by reducing the size of each viewpoint image generated by display image generation part is recorded. As an image for display, a full HD image having resolution of 1920×1080 is used. - As shown in
FIG. 6 , these images are recorded in order of a viewpoint image (hereinafter, called a viewpoint image (1)) of aviewpoint number 1, an image (hereinafter, called a thumbnail image (1)) for display of theviewpoint number 1, a viewpoint image (hereinafter, called a viewpoint image (2)) of aviewpoint number 2, an image (hereinafter, called a thumbnail image (2)) for display of theviewpoint number 2, a viewpoint image (hereinafter, called a viewpoint image (3)) of aviewpoint number 3, an image (hereinafter, called a thumbnail image (3)) for display of theviewpoint number 3, a viewpoint image (hereinafter, called a viewpoint image (4)) of aviewpoint number 4, and an image (hereinafter, called a thumbnail image (4)) for display of theviewpoint number 4. - First, the viewpoint image (1) photographed with the
viewpoint number 1 as a reference viewpoint is recorded as a head image, and the recording region has an SOI marker, auxiliary information region, an image information region, and an EOI marker. - In the auxiliary information region, as in the first embodiment, in addition to Exif auxiliary information having photographing device information, information regarding a photographing mode, and the like, auxiliary information including a maximum display size, a supposed visual distance, a maximum amount of parallax (near view), a maximum amount of parallax (distant view), a convergence angle of a device photographing each viewpoint image, a base line length, an image capturing unit arrangement (viewpoint number), the number of times of photographing when each viewpoint image is acquired, and the like is recorded.
- In regard to the maximum display size, the maximum amount of parallax, and the like, the values where the entire 3D image file is used are recorded. Specifically, the maximum amount of parallax calculated from a viewpoint image (in this case, the viewpoint image (4)) with the greatest amount of parallax on the basis of the viewpoint image (1) photographed with the reference viewpoint, and the maximum display size acquired from the maximum amount of parallax are recorded. In regard the convergence angle, the base line length, and the like, the convergence angle and the base line length to a device photographing the same viewpoint image (in this case, the viewpoint image (4)) are recorded.
- In the image information region, image information of the viewpoint image (1) is recorded, and the EOI marker is subsequently recorded.
- Subsequently to the viewpoint image (1), the thumbnail image (1) generated from the viewpoint image (1) is recorded, and as before, the recording region has an SOI marker, an auxiliary information region, an image information region, and an EOI marker. In the auxiliary information region, normal Exif auxiliary information is recorded.
- Subsequently to the thumbnail image (1), the viewpoint image (2) is recorded. The recording region of the viewpoint image (2) also has an SOI marker, an auxiliary information region, an image information region, and an EOI marker.
- In the auxiliary information region, in addition to normal Exif auxiliary information, auxiliary information including a maximum display size calculated from the viewpoint image (2) and the viewpoint image (1) photographed with the reference viewpoint, a supposed visual distance, a maximum amount of parallax (near view), a maximum amount of parallax (distant view), a convergence angle of a device photographing two viewpoint images, a base line length, a viewpoint number, the number of times of photographing of two viewpoint images, and the like is included and recorded.
- Subsequently to the viewpoint image (2), the thumbnail image (2) generated from the viewpoint image (2) is recorded, and thereafter, the viewpoint image (3) is recorded.
- The recording region of the viewpoint image (3) also has an SOI marker, an auxiliary information region, an image information region, and an EOI marker. In the auxiliary information region, in addition to normal Exif auxiliary information, the maximum display size calculated from the viewpoint image (3) and the viewpoint image (1) photographed with the reference viewpoint, and the like are recorded in the same way.
- Hereinafter, in regard to the viewpoint image (4), the same is recorded.
- In this way, when recording multi-viewpoint images, it is preferable that the maximum display size as auxiliary information of each viewpoint image in the relation with the head image is recorded.
- The 3D image file recorded in this way is read by the stereoscopic image reproduction device when being displayed on the 3D display. At this time, in the stereoscopic image reproduction device, the maximum display size of each viewpoint image recorded in the auxiliary information of the 3D image file is compared with the display size of the 3D display for display, thereby appropriately and easily determining whether or not stereoscopic vision is possible.
- The recording order of multi-viewpoint images is not limited to the order shown in
FIG. 6 . - For example, as shown in
FIG. 7A , the viewpoint image (1), the thumbnail image (1) to the thumbnail image (4), and the viewpoint image (2) to the viewpoint image (4) may be recorded in this order. If the thumbnail images for display are recorded earlier, image reading during file reading when display is performed can be quickened, and a required time until images are displayed on the 3D display can be reduced. Each viewpoint image is primarily used for printing, and since printing requires a predetermined time, even when viewpoint image is recorded in the second half portion of the file, there is little adverse effect. - The recording order of the thumbnail images may be set such that recommended images when being displayed on the 3D display are recorded earlier. For example, if stereoscopic display using the thumbnail image (2) and the thumbnail image (3) is recommended, as shown in
FIG. 7B , the thumbnail image (2) and the thumbnail image (3) may be recorded next to the viewpoint image (1) as the head image, and thereafter, the thumbnail image (1) and the thumbnail image (4) may be recorded. - When the amount of parallax of two images is small, even in a large display, display can be performed such that stereoscopic vision is possible. An image with a viewpoint close to the center is used appropriately for stereoscopic vision. Accordingly, in such a case, saying images in the
viewpoint number 2 and theviewpoint number 3 are recommended, it is preferable the thumbnail image (2) and the thumbnail image (3) are recorded earlier. - Similarly, if stereoscopic display using the thumbnail image (1) and the thumbnail image (3) is recommended, as shown in
FIG. 7C , the thumbnail image (1) and the thumbnail image (3) may be recorded next to the viewpoint image (1) as the head image, and thereafter, the thumbnail image (2) and the thumbnail image (4) may be recorded. - When there are recommended images during stereoscopic display, the maximum display size, supposed visual distance, and the maximum amount of parallax in the recommended images may be recorded in the auxiliary information of the viewpoint image (1) as the head image.
- All multi-viewpoint images in the second embodiment may not be images which are actually photographed, and may include virtual viewpoint images corresponding to virtual viewpoints.
- For example, as shown in
FIG. 8 , two viewpoint images are photographed from different viewpoints (viewpoint number 1 and viewpoint number 4) with respect to the subject 100 using two image capturing devices 101-1 and 101-4. Aviewpoint image 2 and aviewpoint image 3 of aviewpoint number 2 and aviewpoint number 3 with virtual viewpoints which are different from aviewpoint number 1 and aviewpoint number 4 and not actually present may be generated using virtual viewpoint image generation part. When generating virtual viewpoint images, there is a method of internally dividing each pixel of a plurality of photographed images, a method of generating virtual viewpoint images using a parallax map generated from a plurality of photographed images and one photographed image, or the like, the method is not particularly limited. -
FIG. 9A is a diagram schematically showing the data structure of a 3D image file in which the viewpoint images acquired in this way are recorded. In the example ofFIG. 9A , two viewpoint images (1) and (4) actually photographed, thumbnail images (1) and (4) as images for display of the viewpoint images, and thumbnail image (2) and (3) as images for display of the virtual viewpoint images are recorded. - First, the viewpoint image (1) is recorded as a head image, then, the thumbnail image (2), the thumbnail image (3), the thumbnail image (1), and the thumbnail image (4) are recorded, and subsequently, the viewpoint image (4) is recorded. Here, the thumbnail images may be recorded in a recommendation order, may be recorded in an arrangement order of viewpoints, or may be recorded in order from the centermost viewpoint. The thumbnail image (2) and the thumbnail image (3) may be generated from the thumbnail image (1) and the thumbnail image (4), or may be generated from the viewpoint image (1) and the viewpoint image (4).
- As before, the recording region of each image has an SOI marker, an auxiliary information region, an image information region, and an EOI marker. In the auxiliary information region of the viewpoint image (1), in addition to a maximum display size, a supposed visual distance, and a maximum amount of parallax, viewpoint information representing whether each viewpoint number is a viewpoint (actual viewpoint) with which photographing is actually performed or a virtual viewpoint is recorded.
- Images for display may not be provided, and only viewpoint images for printing may be recorded. In the example shown in
FIG. 9B , two viewpoint images (1) and (4) actually photographed and viewpoint images (2) and (3) which are virtual viewpoint images are recorded. - The recording order may be a recommendation order of printing or display. Specifically, the viewpoint image (2) is recorded as a head image, and subsequently, the viewpoint image (3), the viewpoint image (1), and the viewpoint image (4) are recorded. As before, the viewpoint images may be recorded in an order of viewpoints.
- In this way, if virtual viewpoint images are generated and recorded, it should suffice that there are two images as actual viewpoint images, making it possible to simplify and lighten an image capturing optical system of an image capturing device.
- When being displayed on a display of a supposed size, if it is determined that the amount of parallax on the distant view side exceeds the interocular distance of the person, recording may be performed after the amount of parallax of images is adjusted such that stereoscopic vision is possible even when the images are displayed on the display of this size.
- In this embodiment, parallax displacement is performed as the adjustment of the amount of parallax.
-
FIG. 10 is a diagram illustrating the principle of parallax displacement.FIG. 11A is a diagram showing a left viewpoint image, andFIG. 11B is a diagram showing a right viewpoint image. Here, it is assumed that the left eye of the viewer is at the coordinates (0,D), and the right eye of the viewer is at the coordinates (XB,D). In the left and right viewpoint images displayed on Z=0, the subject displayed at the coordinates (XL,0) of the left viewpoint image and the coordinates (XR,0) of the right viewpoint image is viewed at the coordinates (XP,YP). - In this state, as shown in
FIG. 11B , if the right viewpoint image is shifted by XR-XR in the left direction, as shown inFIG. 10 , the coordinates of the right viewpoint image of the subject become (XR′,0), and as a result, the subject is viewed at the coordinates (XP′,YP′). - In this way, if parallax displacement is performed, the adjustment of the amount of parallax can be performed. Accordingly, if parallax displacement is performed when the amount of parallax on the distant view side exceeds the interocular distance of the person, the amount of parallax can fall within the interocular distance of the person, making it possible to appropriately perform stereoscopic vision.
-
FIG. 12 is a flowchart showing a photographing and recording principle for recording a 3D image file according to this embodiment. - First, a plurality of viewpoint images are acquired (Step S21). Here, it is assumed that two viewpoint images are photographed. Next, a plurality of feature points are extracted from each two viewpoint images (Step S22), and the amount of parallax of each feature point is calculated (Step S23). The maximum amount of parallax on the near view side and the maximum amount of parallax on the distant view side are acquired from the calculated amount of parallax of each feature point (Step S24).
- Here, for a supposed display on which display is performed, the display width is acquired. The display width of the supposed display may be determined in advance and stored, or when a 3D display is connected through an external connection terminal, such as an HDMI terminal, the size of the 3D display may be read through communication based on the connection.
- When the stereoscopic image is displayed on the display having this display width, it is determined whether or not the maximum parallax position on the distant view side can be stereoscopically viewed (Step S25). Specifically, the product of the display width and the maximum amount of parallax (%) on the distant view side is calculated, and it is determined whether or not the calculated product is equal to or greater than the interocular distance of the person, 50 mm.
- When the calculated product is smaller than 50 mm, the two photographed viewpoint images are recorded directly (Step S27). For example, the viewpoint images may be recorded as the 3D image file having the data structure shown in
FIG. 2 , or the thumbnail images for display or virtual viewpoint images may be generated and recorded. - When the calculated product is equal to or greater than 50 mm, parallax displacement is performed such that the product becomes smaller than 50 mm and the maximum parallax position on the distant view side can be stereoscopically viewed (Step S26).
- As shown in
FIGS. 11A and 11B , parallax displacement may be performed by displacing the right viewpoint image in the left direction, or may be performed by displacing the left viewpoint image in the right direction. The left and right viewpoint images may be displaced in a direction in which the images are brought close to each other. - After the parallax displacement processing ends, recording is performed (Step S27).
FIG. 13 is a diagram schematically showing the data structure of a 3D image file to be recorded. In the 3D image file, in addition to two photographed original images, images for display and thumbnail images obtained by reducing the size of the original images after parallax displacement are recorded. Here, the images for display are images of full HD size, and the thumbnail images are images of VGA size. The thumbnail images are used, for example, when being displayed for image search on a small liquid crystal display provided on the rear surface of the image capturing device. - As shown in
FIG. 13 , these images are recorded in order of the image for display of theviewpoint number 1, the image for display of theviewpoint number 2, the thumbnail image of theviewpoint number 1, the thumbnail image of theviewpoint number 2, the original image of theviewpoint number 1, and the original image of theviewpoint number 2, and the recording region of each image has an SOI marker (not shown in the drawing), an auxiliary information region, an image information region, and an EOI marker (not shown in the drawing). - First, the image for display of the
viewpoint number 1 as the reference viewpoint is recorded as a head image, and in the auxiliary information region, the maximum amount of parallax is recorded. - The image for display of the
viewpoint number 2 is next recorded, and this image is an image after parallax displacement is performed. In the auxiliary information region of the image for display of theviewpoint number 2, information indicating that the parallax displacement processing is completed and the shift amount (unit: pixel) are recorded. - Subsequently, the thumbnail image of the
viewpoint number 1 is recorded, and in the auxiliary information region thereof, the maximum amount of parallax is recorded. - The thumbnail image of the
viewpoint number 2 is further recorded. The thumbnail image may be an image after parallax displacement, or may be an image generated from the original image of theviewpoint number 2. - Thereafter, the original image of the
viewpoint number 1 and the original image of theviewpoint number 2 are recorded. In the auxiliary information region of the original image of theviewpoint number 1, the maximum amount of parallax is recorded. - With this 3D image file, when being displayed on a display of supposed size, the maximum parallax position on the distant view side can be constantly stereoscopically viewed, and thus, the viewer can appropriately view stereoscopic vision.
- Since the thumbnail images of VGA size are also recorded, this is effective for display during image search.
- The supposed display size may be recorded in the auxiliary information region of the head image. If the supposed display size is recorded, in the stereoscopic image reproduction device, the read supposed display size is compared with the display size of the 3D display for display, thereby appropriately and easily determining whether or not stereoscopic vision is possible.
- Next, processing for reading, reproducing, and displaying the 3D image file recorded in this way will be described with reference to the flowchart of
FIG. 14 . - Initially, a 3D image file is read (Step S31). Simultaneously, the size of a display on which a 3D image is now displayed is acquired (Step S32). The maximum display size of which viewpoint images for 3D display can be appropriately 3D-displayed is acquired from the auxiliary information of the read 3D image file (Step S33).
- Next, the lateral width of the display on which the 3D image is now displayed is compared with the width of the maximum display size (Step S34).
- When the width of the maximum display size is greater, since there is no difficulty in stereoscopic vision even when the viewpoint images recorded in the 3D image file are displayed directly, the read viewpoint images are displayed directly (Step S36).
- However, when the width of the actual display is greater, if the read viewpoint images are displayed directly, the amount of parallax of the maximum parallax position on the distant view side exceeds the interocular distance of the person, and this portion may not be stereoscopically viewed. Accordingly, it is necessary to adjust the amount of parallax of the left and right viewpoint images.
- The adjustment of the amount of parallax is performed by parallax displacement or parallax compression (Step S35). As described with reference to
FIGS. 11A and 11B , parallax displacement adjusts the amount of parallax by displacing one or both of the left and right viewpoint images. As described with reference toFIG. 8 , parallax compression adjusts the amount of parallax by generating images of virtual viewpoints with the amount of parallax smaller than the left and right viewpoint images and displaying the generated images of the virtual viewpoints. When adjusting the amount of parallax, either parallax displacement or parallax compression may be determined in advance or may be selected by the user. In regard to the adjustment of the amount of parallax, even when either parallax displacement or parallax compression is used, the parallax on at least the distant view side can become small. - With the parallax displacement or parallax compression, the amount of parallax is adjusted to the amount of parallax suitable for the width of the display being used, and the adjusted images are 3D-displayed (Step S36).
- In this way, the maximum display size recorded as the auxiliary information is read and compared with the display size of which display is actually performed. When the maximum display size is greater, it is determined that there is no problem in stereoscopic vision, and the recorded 3D image is displayed directly. When the display size of which display is actually performed is greater, it is determined that there is a region where stereoscopic vision is not possible, the recorded 3D image is displayed after the amount of parallax is adjusted, making it possible to constantly display an appropriate 3D image.
- Such processing may be performed by reading the maximum amount of parallax recorded in the auxiliary information of the 3D image file.
-
FIG. 15 is a flowchart when reproduction and display processing is performed on the basis of the maximum amount of parallax. The portions common to the flowchart shown inFIG. 14 are represented by the same reference numerals, and detailed description thereof will not be repeated. - The maximum amount of parallax (distant view) is acquired from the auxiliary information of the read 3D image file (Step S41). The maximum amount of parallax (distant view) is recorded as the ratio (%) with respect to the image width.
- Next, it is determined whether or not the product of the width of the display acquired in Step S32 and the maximum amount of parallax (distant view) is equal to or greater than 50 mm (Step S42). The product of the width of the display and the maximum amount of parallax becomes an actual amount of parallax at the maximum parallax amount position (distant view) on a displayed image. When the amount of parallax becomes equal to or greater than the interocular distance of the person, 50 mm, this portion cannot be stereoscopically viewed. For this reason, after the adjustment of the amount by parallax displacement or parallax compression is performed for the read viewpoint images (Step S35), 3D display is performed (Step S36).
- When the amount of parallax is smaller than 50 mm, since there is no difficulty even when the recorded viewpoint images are displayed directly, the read viewpoint images are displayed directly (Step S36).
- In this way, reproduction and display can be controlled on the basis of the maximum amount of parallax (distant view) read from the 3D image file. In particular, this is effective when the maximum display size is not recorded in the auxiliary information of the 3D image file, and only the maximum amount of parallax is recorded.
- Although in this embodiment, when it is determined that 3D display cannot be appropriately performed, 3D display is performed on the entire display after the amount of parallax is adjusted by parallax displacement or parallax compression, 3D display may be performed in an image size with an appropriate amount of parallax, that is, an image size corresponding to the maximum display size.
- As described above, images generated by parallax displacement or parallax compression may be recorded in a 3D image file.
FIG. 16 is a flowchart when processing for recording images in a 3D image file is performed. - Here, images generated by parallax displacement or parallax compression in Step S35 are added or rewritten to a 3D image file as thumbnail images for display in Step S43. The addition or rewriting to a 3D image file may be performed after 3D display in Step S36.
-
FIGS. 17A and 17B are diagrams schematically showing the data structure of a 3D image file in which four viewpoint images and four thumbnail images for display generated from the viewpoint images are recorded.FIG. 17A shows a 3D image file when being read in Step S31 ofFIG. 16 , andFIG. 17B shows a 3D image file after rewriting in Step S43. The maximum amount of parallax (distant view) of a viewpoint image (1) and a viewpoint image(4) is recorded in the auxiliary information of the viewpoint image (1) as a head image. - If the maximum amount of parallax is read in Step S41, and it is determined in Step S42 that the product of the width of the display and the maximum amount of parallax (distant view) is equal to or greater than 50 mm, in Step S35, parallax displacement or parallax compression is performed. For example, when parallax displacement of a thumbnail image (4) is performed, in Step S43, the thumbnail image (4) after parallax displacement is recorded instead of the thumbnail image (4). Information representing that the parallax displacement processing is completed and the shift amount (unit: pixel) are recorded in the auxiliary information of the thumbnail image (4) after parallax displacement.
- If parallax compression is performed in Step S35, the thumbnail image (4) after parallax compression is recorded as the thumbnail image (4). In this case, the maximum amount of parallax (distant view) after parallax compression may be recorded as the auxiliary information.
- Although the thumbnail image (4) after parallax amount adjustment is recorded instead of the thumbnail image (4), the thumbnail image (4) after parallax amount adjustment may be additionally recorded.
- Even when there is no thumbnail image for display in the 3D image file when being read in Step S31 of
FIG. 16 , thumbnail images for display may be generated and recorded as a 3D image file shown inFIG. 17B . - Parallax adjustment is not limited to the thumbnail images, and the parallax adjustment of the viewpoint images may be performed. For example, if the parallax adjustment of the viewpoint image (4) is performed, the viewpoint image (4) and the thumbnail image (4) generated on the basis of the viewpoint image (4) may be overwritten or additionally recorded.
- The product of the width of the display and the maximum amount of parallax (distant view) calculated in Step S42 may be recorded in the auxiliary information region of the head image of the 3D image file as the maximum display size.
- Since the parallax compression processing (generation of virtual viewpoint images) requires a certain amount of time, images after parallax compression cannot be immediately displayed, and the viewer may feel inconvenience. In this embodiment, while the parallax compression processing is performed, 2D display is performed on the 3D display.
-
FIG. 18 is a flowchart showing processing in which 2D display is performed. The portions common to the flowchart shown inFIG. 16 are represented by the reference numerals, and detailed description thereof will not be repeated. - If it is determined in Step S42 that the product of the width of the display and the maximum amount of parallax (distant view) is equal to or greater than the interocular distance of the person, 50 mm, either the viewpoint images or the thumbnail images for display recorded in the 3D image file are 2D-displayed on the display (Step S51). Thereafter, the parallax compression processing is performed (Step S35).
- After the parallax compression processing ends, the thumbnail images for display of the 3D image file are added or rewritten (Step S43), and the images subjected to parallax compression are 3D-displayed on the display (Step S36). After 3D display is performed, the thumbnail images for display may be recorded in the 3D image file.
- If 2D display is performed in this way, a drawback in that images cannot be displayed while the parallax compression processing is performed can be covered.
- Although an example where parallax compression is performed has been described, when the parallax displacement processing requires a lot of time, 2D display may be performed while the parallax displacement processing is performed. Simultaneously with 2D display, the effect that the parallax amount adjustment processing is performed may be displayed. If display is performed in this way, the viewer can know the reason for performing 2D display on the 3D display.
- Although in the foregoing embodiments, the adjustment of the amount of parallax is performed on the basis of the maximum amount of parallax (distant view), and 3D display is performed, in regard to the adjustment of the amount of parallax, if parallax displacement is performed, there is a drawback in that the amount of parallax on the near view side increases. Accordingly, when parallax displacement is performed, display may be performed taking into consideration the maximum amount of parallax (near view).
-
FIG. 19 is a flowchart when parallax displacement processing is performed taking into consideration the maximum amount of parallax (near view). The portions common to the flowchart shown inFIG. 16 are represented by the same reference numerals, and detailed description thereof will not be repeated. - In Step S42, it is determined whether or not the product of the width of the display which is used for display and the maximum amount of parallax (distant view) is equal to or greater than the interocular distance of the person, 50 mm. When the product is smaller than 50 mm, since there is no difficulty even when the 3D image recorded in the 3D image file is displayed directly, the read 3D image is displayed directly (Step S64).
- When the product is equal to or greater than 50 mm, if display is performed directly, since stereoscopic vision of the maximum parallax position on the distant view side is not possible, it is necessary to adjust the amount of parallax. Accordingly, a parallax displacement amount (necessary displacement amount) in which the product becomes smaller than 50 mm is calculated (Step S61).
- Next, when parallax displacement is performed by the calculated necessary displacement amount, it is determined whether or not the maximum parallax amount position on the near view side can be appropriately stereoscopically viewed. Specifically, it is determined whether or not a value obtained by adding the necessary displacement amount to the product of the width of the display which is used for display and the maximum amount of parallax (near view) is equal to or smaller than 50 mm (Step S62). Although it is determined that stereoscopic vision is appropriately possible when the amount of parallax of the near view is equal to or smaller than 50 mm, the value of 50 mm may be appropriately determined.
- When the value is equal to or smaller than 50 mm, parallax displacement is executed (Step S63), and the images subjected to parallax displacement are 3D-displayed on the 3D display (Step S64). The images subjected to parallax displacement may be added to rewritten to the 3D image file.
- Meanwhile, when the value is greater than 50 mm, if parallax displacement is not performed, there is a portion which cannot be stereoscopically viewed on the distant view side. If parallax displacement is performed such that the maximum amount of parallax on the distant view side becomes appropriate, the near view side cannot be appropriately stereoscopically viewed. For this reason, 3D display is not performed, and 2D display is performed on the 3D display (Step S65). During 2D display, either the viewpoint images or the thumbnail images for display recorded in the 3D image file are displayed. Along with 2D display, the effect that the display size is too large and the images are not 3D-displayed may be warned and displayed.
- In this way, the adjustment of the amount of parallax is performed taking into consideration the maximum amount of parallax (near view) as well as the maximum amount of parallax (distant view), thereby displaying an appropriate 3D image.
- [Appearance of Stereoscopic Image Capturing Device]
- Next, a device for realizing the above-described embodiment will be described.
-
FIGS. 20A and 20B are diagrams showing the appearance of a stereoscopic image capturing device according to the invention.FIG. 20A is a perspective view when a stereoscopic image capturing device is viewed from the front side, andFIG. 20B is a rear view. - A stereoscopic image capturing device (fly-eye camera) 10 is a digital camera in which 2D/3D still images and 2D/3D motion images can be recorded. As shown in
FIGS. 20A and 20B , ashutter button 11 and azoom button 12 are provided on the top surface of a thin rectangular parallelepiped camera body. - A
lens barrier 13 substantially having the same width as the width of the camera body in the left-right direction is provided on the front surface of the camera body to freely move in the up-down direction. Thelens barrier 13 is moved in the up-down direction between a position indicated by a two-dot-chain line and a position indicated by a solid line to simultaneously open/close the front surfaces of a pair of left and right photographing optical systems 14-1 and 14-2. As the photographing optical systems 14-1 and 14-2, zoom lenses of bent optical systems are used. A camera power supply can be turned on/off in conjunction with the opening/closing operation of the lens front surface by thelens barrier 13. - As shown in
FIG. 20B , a liquid crystal monitor 16 for 3D is provided in the center portion on the rear surface of the camera body. The liquid crystal monitor 16 can display a plurality of parallax images (right viewpoint image and left viewpoint image) as directional images having predetermined directivity by a parallax barrier. As the liquid crystal monitor 16 for 3D, a liquid crystal monitor using a lenticular lens, a liquid crystal monitor in which a right viewpoint image and a left viewpoint image can be individually viewed using dedicated glasses, such as polarization glasses or liquid crystal shutter glasses, or the like may be applied. - Various operating switches are provided on the left and right sides of the
liquid crystal monitor 16. Anoperating switch 18A is a selector switch for switching between still image photographing and motion image photographing, anoperating switch 18B is a parallax adjustment switch for adjusting the amount of parallax of a right viewpoint image and a left viewpoint image, and anoperating switch 18C is a selector switch for switching between 2D imaging and 3D imaging. Anoperating switch 18D is a seesaw key serving as a MENU/OK button and a play button, anoperating switch 18E is a multi-function cross key, and an operating key 18F is a DISP/BACK key. - The MENU/OK button is an operating switch which has a function as a menu button for giving an instruction to display a menu on the screen of the liquid crystal monitor 16 and a function as an OK button for instructing the confirmation and execution of a selected content, or the like. The play button is a button which is used for switching from a photographing mode to a reproduction mode. The cross key is an operating switch which inputs instructions in four directions of up, down, left, and right, and a macro button, a flash button, a self-timer button, and the like are assigned. When the menu is selected, the cross key functions as a switch (cursor movement operating part) for switching an item from the menu screen or for instructing the selection of various setting items from each menu. The left/right key of the cross key functions as a frame-by-frame playback (forward/reverse playback) button in the reproduction mode. The DISP/BACK key is used to switch the display form of the
liquid crystal monitor 16, to cancel an instruction content on the menu screen, to return an operation state to a previous operation state, or the like. - In
FIG. 20A ,reference numeral 15 denotes a stereo microphone. - [Internal Configuration of Stereoscopic Image Capturing Device]
-
FIG. 21 is a block diagram showing the internal configuration of the stereoscopicimage capturing device 10. - As shown in
FIG. 21 , the stereoscopicimage capturing device 10 primarily has a plurality of image capturing units 20-1 and 20-2, a central processing unit (CPU) 32, an operatingunit 34 including theshutter button 11 and thezoom button 12 described above and various operating switches, adisplay control unit 36, aliquid crystal monitor 16, arecording control unit 38, a compression/expansion processing unit 42, a digitalsignal processing unit 44, an AE (Automatic Exposure)detection unit 46, an AF (Auto Focus)detection unit 48, an AWB (Automatic White Balance)detection unit 50, aVRAM 52, aRAM 54, aROM 56, anEEPROM 58, and the like. Although the image capturing units 20-1 and 20-2 capture two parallax images of an image for a left eye and an image for a right eye having parallax, three or more image capturing units 20 may be provided. - The image capturing unit 20-1 which captures the image for the left eye includes a prism (not shown), a photographing optical system 14-1 (
FIGS. 20A and 20B ) having a focus lens/zoom lens 21, optical units having anaperture diaphragm 22 and amechanical shutter 23, a solid-state image capturing element (CCD) 24, an analogsignal processing unit 25, an A/D converter 26, animage input controller 27, alens driving unit 28, an aperturediaphragm driving unit 29, and ashutter control unit 30 for driving the optical units, and aCCD control unit 31 for controlling theCCD 24. The image capturing unit 20-2 which captures the image for the right eye has the same configuration as the image capturing unit 20-1 which captures the image for the left eye, thus description of a specific configuration will be omitted. - The
CPU 32 performs overall control of the operation of the entire camera in accordance with a predetermined control program on the basis of an input from the operatingunit 34. The calculation of the maximum display size, parallax displacement, the generation of the virtual viewpoint image, and the like are performed by theCPU 32. - The
ROM 56 stores a control program executed by theCPU 32 and various kinds of data necessary for control, and theEEPROM 58 stores various kinds of information representing the adjustment result of adjustment before product shipment, for example, defective pixel information of theCCD 24, correction parameters which are used for image processing, a correspondence table of the maximum amount of parallax and the maximum display size, and the like. - The
VRAM 52 is a memory which temporarily stores image data for display on theliquid crystal monitor 16, and theRAM 54 includes a computational work area of theCPU 32 and a temporary storage area of image data. - The focus lens/
zoom lens 21 included in the photographing optical system is driven by thelens driving unit 28 and moves back and forth along the optical axis. TheCPU 32 controls the driving of thelens driving unit 28 to control the position of the focus lens and to perform focus adjustment such that the subject comes into focus, and controls the zoom position of the zoom lens in accordance with a zoom instruction from thezoom button 12 of the operatingunit 34 to change a zoom magnification. - The
aperture diaphragm 22 has, for example, an iris diaphragm, and is driven and operated by the aperturediaphragm driving unit 29. TheCPU 32 controls the amount (aperture value) of opening of theaperture diaphragm 22 through the aperturediaphragm driving unit 29, and controls the amount of incident light on theCCD 24. - The
mechanical shutter 23 opens/closes the optical path to determine the exposure time in theCCD 24, and prevents unwanted light from being incident on theCCD 24 when reading an image signal from theCCD 24, thereby preventing the occurrence of smear. TheCPU 32 outputs a shutter closing signal synchronized with the exposure end time corresponding to a shutter speed to theshutter control unit 30, and controls themechanical shutter 23. - The
CCD 24 has a two-dimensional color CCD solid-state image capturing element. Multiple photodiodes are two-dimensionally arranged on the light-receiving surface of theCCD 24, and color filters are disposed in each photodiode with a predetermined arrangement. - The optical image of the subject focused on the CCD light-receiving surface through the optical units having the above-described configuration is converted to a signal charge according to the amount of incident light by the photodiodes. The signal charges accumulated in each photodiode are sequentially read from the
CCD 24 as a voltage signal (image signal) according to the signal charges in accordance with an instruction of theCPU 32 on the basis of a driving pulse from theCCD control unit 31. TheCCD 24 has an electronic shutter function, and controls a charge accumulation time into the photodiode to control the exposure time (shutter speed). The charge accumulation start time corresponding to the shutter speed is controlled by the electronic shutter, and themechanical shutter 23 is closed to control the exposure end time (charge accumulation end time). Although in this embodiment, theCCD 24 is used as an image capturing element, an image capturing element having a different configuration, such as a CMOS sensor, may be used. - R, G, and B analog signals read from the
CCD 24 are subjected to double-correlated sampling (CDS) or amplification by the analogsignal processing unit 25, and converted to R, and B digital signals by the A/D converter 26. - The
image input controller 27 has an internal line buffer of a predetermined capacity, temporarily accumulates R, G, and B image signals (CCD RAW data) A/D converted by the A/D converter 26, and stores the R, G, and B image signals in theRAM 54 through a bus 60. - In a 3D image capturing mode, the
CPU 32 controls the image capturing unit 20-2 which captures the right viewpoint image in the same manner as the image capturing unit 20-1 which captures the left viewpoint image. - The
AE detection unit 46 calculates subject luminance necessary for AE control on the basis of the image signals introduced when theshutter button 11 is half-pressed, and outputs a signal representing subject luminance (photographing EV value) to theCPU 32. TheCPU 32 sets the shutter speed (exposure time), the aperture value, and photographing sensitivity in a plurality of image capturing units 20-1 and 20-2 in accordance with a predetermined program diagram on the basis of the input photographing EV value. - The
AF detection unit 48 integrates the absolute value of a high-frequency component of an image signal of an AF area introduced when theshutter button 11 is half-pressed, and outputs the integrated value (AF evaluation value) to theCPU 32. TheCPU 32 moves the focus lens from a close range side to an infinite side, searches for a focusing position where the AF evaluation value detected by theAF detection unit 48 is maximal, and moves the focus lens to the focusing position, thereby performing focus adjustment on the subject (main subject). During motion image photographing, so-called mountain-climbing control is performed to move the focus lens such that the AF evaluation value constantly has the maximum value. - The
AWB detection unit 50 automatically obtains a light source type (color temperature of field) on the basis of the R, G, and B image signals acquired during this image capturing, and reads a corresponding white balance gain from a table which stores R, G, and B white balance gains (white balance correction values) according to the light source type in advance. - The digital
signal processing unit 44 includes a white balance correction circuit, a gradation conversion processing circuit (for example, a gamma correction circuit), a synchronization circuit which interpolates spatial deviations of color signals of R, G, B, and the like depending on the color filter arrangement of a single-plate CCD to position the color signals, a contour correction circuit, a luminance/color difference signal generation circuit, and the like, and performs image processing on the R, G, and B image signals (CCD RAW data) stored in theRAM 54. That is, R, G, and B CCD RAW data are subjected to white balance correction by multiplying the white balance gains detected by theAWB detection unit 50 in the digitalsignal processing unit 44, then subjected to the predetermined processing, such as gradation conversion processing (for example, gamma correction), and converted to YC signals having a luminance signal (Y signal) and a color-difference signal (Cr and Cb signals). The YC signals processed by the digitalsignal processing unit 44 are stored in theRAM 54. - The digital
signal processing unit 44 also includes a distortion correction circuit which corrects lens distortion of the photographing optical systems of a plurality of image capturing units 20-1 and 20-2, and an image clipping circuit which clips images of predetermined clipping areas from the left and right viewpoint images to correct optical axis deviations of the photographing optical systems of a plurality of image capturing units 20-1 and 20-2. - The compression/
expansion processing unit 42 compresses the YC signals stored in theRAM 54 in accordance with an instruction from theCPU 32 during recording in amemory card 40, or expands compressed data recorded in thememory card 40 to obtain the YC signals. - The
recording control unit 38 converts compressed data compressed by the compression/expansion processing unit 42 to an image file of a predetermined format (for example, a 3D still image becomes an MP file, and a 3D motion image becomes a motion image file of motion PEG H.264, MPEG4, or MPEG4-MVC), records the image file in thememory card 40, or reads an image file from thememory card 40. - The
recording control unit 38 records the maximum display size, the supposed visual distance, the maximum amount of parallax (near view) (%), the maximum amount of parallax (distant view) (%), and the like as auxiliary information in thememory card 40 along with Exif auxiliary information, when recording an MP file according to the invention. - The liquid crystal monitor 16 is used as an image display unit which displays photographed images, and is also used as a GUI (Graphical User Interface) during various settings. The liquid crystal monitor 16 is also used as an electronic view-finder which displays live view images (hereinafter, referred to as “through images” for confirming an image angle in the photographing mode. When a 3D image is displayed on the
liquid crystal monitor 16, thedisplay control unit 36 alternately displays a left viewpoint image and a right viewpoint image held in theVRAM 52 pixel by pixel. With the parallax barrier provided in theliquid crystal monitor 16, left and right images alternately arranged pixel by pixel are separately viewed by the left and right eyes of the user who observes at a predetermined distance. Accordingly, stereoscopic vision is possible. - With the stereoscopic
image capturing device 10 configured as above, the embodiment can be realized. Although the stereoscopicimage capturing device 10 described herein includes the two image capturing units for photographing images of left and right two viewpoints, three or more image capturing units may be provided to photograph three or more viewpoint images. For example, four image capturing units, such as the four image capturing devices 101-1 to 101-4 shown inFIG. 5 , may be provided to photograph four viewpoint images. - [Stereoscopic Image Reproduction Device]
- Fifth to eighth embodiments may be realized in a stereoscopic image reproduction device in which no image capturing unit is provided.
-
FIG. 22 is a diagram showing the overall configuration of a stereoscopicimage reproduction device 300 and a3D display 320 according to the invention. As shown inFIG. 22 , the stereoscopicimage reproduction device 300 and the3D display 320 are separate devices and are communicably connected together by acommunication cable 310. - The
3D display 320 is a parallax barrier-type or a lenticular-type display, and a left viewpoint image and a right viewpoint image input from the stereoscopicimage reproduction device 300 to the3D display 320 are alternately displayed line by line. - The
3D display 320 may alternately switch and display the left viewpoint image and the right viewpoint image in a temporal manner. In this case, the viewer views the3D display 320 using special glasses. -
FIG. 23 is a block diagram showing the internal configuration of the stereoscopicimage reproduction device 300. As shown inFIG. 23 , the stereoscopicimage reproduction device 300 includes aCPU 301, arecording control unit 305, amemory card 306, adisplay control unit 307, acommunication interface 308, and the like. - The
CPU 301 performs overall control of the operation of the entire stereoscopicimage reproduction device 300 on the basis of a control program recorded in aROM 302. ARAM 303 is used as a computational work area of theCPU 301. - The
recording control unit 305 and thedisplay control unit 307 are connected to theCPU 301 through abus 304. Therecording control unit 305 controls data reading and writing of a 3D image file with respect to thememory card 306. Thememory card 306 is, for example, the same as thememory card 40 of the stereoscopicimage capturing device 10 shown inFIG. 21 , and records a 3D image file including the viewpoint images photographed in the stereoscopicimage capturing device 10 and auxiliary information. - The
communication interface 308 is a connector unit to which acommunication cable 310 is connected, and thedisplay control unit 307 displays a stereoscopic image on the3D display 320 through thecommunication interface 308 and thecommunication cable 310. As thecommunication interface 308 and thecommunication cable 310, those based on the HDMI standard may be used. According to the HDMI standard, the stereoscopicimage reproduction device 300 can acquire the display size of the3D display 320 connected thereto through thecommunication cable 310. - Fly-eye image capturing part for photographing viewpoint images may be provided in the stereoscopic
image reproduction device 300, and the photographed viewpoint images may be recorded in thememory card 306. The stereoscopicimage reproduction device 300 and the3D display 320 may be constituted as a single device.
Claims (9)
1. A stereoscopic image reproduction device comprising:
first acquisition part for reading a three-dimensional image file having recorded therein a plurality of viewpoint images for stereoscopic vision and auxiliary information including the maximum amount of parallax on a distant view side in the plurality of viewpoint images, and acquiring the plurality of viewpoint images and the auxiliary information from the three-dimensional image file;
second acquisition part for acquiring the display size of a stereoscopic display as an output destination;
determination part for determining whether or not binocular fusion is enabled when a plurality of viewpoint images are displayed on the stereoscopic display on the basis of the acquired maximum amount of parallax on the distant view side, the display size of the stereoscopic display, and a predetermined value representing an interocular distance of a person;
image processing part for generating a plurality of viewpoint images with smaller parallax on the distant view side between the a plurality of viewpoint images on the basis of the plurality of viewpoint images acquired by the first acquisition part if the determination part determined that binocular fusion is disabled; and
output part for outputting the viewpoint images generated by the image processing part to the stereoscopic display if the determination part determines that binocular fusion is disabled and outputting the viewpoint images acquired by the first acquisition part to the stereoscopic display if it is determined that binocular fusion is enabled.
2. The stereoscopic image reproduction device according to claim 1 ,
wherein the determination part has image deviation amount calculation part for calculating the amount of image deviation on the stereoscopic display corresponding to the maximum amount of parallax when the plurality of viewpoint images are displayed on the stereoscopic display on the basis of the acquired maximum amount of parallax on the distant view side and the display size of the stereoscopic display, and determines whether or not binocular fusion is enabled depending on whether or not the calculated amount of image deviation exceeds the predetermined value representing the interocular distance of the person.
3. The stereoscopic image reproduction device according to claim 1 ,
wherein the image processing part performs parallax displacement between the plurality of viewpoint images such that the maximum amount of parallax on the distant view side of the plurality of viewpoint images on the stereoscopic display is equal to or smaller than a predetermined value representing an interocular distance of a person, and generating viewpoint shift images subjected to the parallax displacement.
4. The stereoscopic image reproduction device according to claim 2 ,
wherein the image processing part performs parallax displacement between the plurality of viewpoint images such that the maximum amount of parallax on the distant view side of the plurality of viewpoint images on the stereoscopic display is equal to or smaller than a predetermined value representing an interocular distance of a person, and generating viewpoint shift images subjected to the parallax displacement.
5. The stereoscopic image reproduction device according to claim 3 ,
wherein the auxiliary information of the three-dimensional image file includes the maximum amount of parallax on a near view side in the plurality of viewpoint images,
the stereoscopic image reproduction device further includes addition part for adding the maximum amount of parallax on the near view side of the plurality of viewpoint images on the stereoscopic display and the amount of deviations of the viewpoint shift images on the stereoscopic display; and
determination part for determining whether or not the amount of image deviation corresponding to the added maximum amount of parallax on the near view side exceeds a binocular fusion limit when being displayed on the stereoscopic display, and if the determination part determines that the amount of image deviation exceeds the binocular fusion limit, the output part outputs one of the plurality of viewpoint images acquired by the first acquisition part to the stereoscopic display to display a two-dimensional image.
6. The stereoscopic image reproduction device according to claim 4 ,
wherein the auxiliary information of the three-dimensional image file includes the maximum amount of parallax on a near view side in the plurality of viewpoint images,
the stereoscopic image reproduction device further includes addition part for adding the maximum amount of parallax on the near view side of the plurality of viewpoint images on the stereoscopic display and the amount of deviations of the viewpoint shift images on the stereoscopic display; and
determination part for determining whether or not the amount of image deviation corresponding to the added maximum amount of parallax on the near view side exceeds a binocular fusion limit when being displayed on the stereoscopic display, and
if the determination part determines that the amount of image deviation exceeds the binocular fusion limit, the output part outputs one of the plurality of viewpoint images acquired by the first acquisition part to the stereoscopic display to display a two-dimensional image.
7. The stereoscopic image reproduction device according to claim 1 ,
wherein the image processing part has parallax amount calculation part for calculating the amount of parallax representing the amount of deviation between feature points having common features from the plurality of viewpoint images acquired by the first acquisition part, and virtual viewpoint image generation part for generating one or a plurality of virtual viewpoint images corresponding to an arbitrary virtual viewpoint on the basis of at least one viewpoint image from the plurality of viewpoint images acquired by the first acquisition part and the amount of parallax calculated by the parallax amount calculation part.
8. The stereoscopic image reproduction device according to claim 7 ,
wherein the output part outputs one of the plurality of viewpoint images acquired by the first acquisition part to the stereoscopic display to display a two-dimensional image if the determination part determines that the display size of the stereoscopic display is greater than the maximum display size or if the calculated amount of image deviation exceeds a predetermined value representing an interocular distance of a person, and outputs the virtual viewpoint images instead of the two-dimensional image if the virtual viewpoint images are generated by the image processing part.
9. A stereoscopic image reproduction method using the stereoscopic image reproduction device according to claim 1 comprising:
a first acquisition step of reading a three-dimensional image file having recorded therein a plurality of viewpoint images for stereoscopic vision and auxiliary information for the plurality of viewpoint images, the auxiliary information including the maximum amount of parallax on a distant view side in the plurality of viewpoint images, and acquiring the plurality of viewpoint images and the auxiliary information from the three-dimensional image file;
a second acquisition step of acquiring the display size of the stereoscopic display as an output destination;
a determination step of determining whether or not binocular fusion is enabled when a plurality of viewpoint images are displayed on the stereoscopic display on the basis of the acquired maximum amount of parallax on the distant view side, the display size of the stereoscopic display, and a predetermined value representing an interocular distance of a person;
an image processing step of generating a plurality of viewpoint images with smaller parallax on at least the distant view side between the plurality of viewpoint images on the basis of the plurality of viewpoint images acquired in the first acquisition step if it is determined in the determination step that binocular fusion is disabled; and
an output step of outputting the viewpoint images generated in the image processing step to the stereoscopic display if it is determined in the determination step that binocular fusion is disabled and outputting the viewpoint images acquired in the first acquisition step to the stereoscopic display if it is determined that binocular fusion is enabled.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/377,353 US20170094267A1 (en) | 2010-04-28 | 2016-12-13 | Stereoscopic image reproduction device and method, stereoscopic image capturing device, and stereoscopic display device |
Applications Claiming Priority (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2010-103905 | 2010-04-28 | ||
JP2010103905 | 2010-04-28 | ||
PCT/JP2011/060095 WO2011136190A1 (en) | 2010-04-28 | 2011-04-25 | Stereoscopic image reproduction device and method, stereoscopic image capturing device, stereoscopic display device |
US13/659,638 US9560341B2 (en) | 2010-04-28 | 2012-10-24 | Stereoscopic image reproduction device and method, stereoscopic image capturing device, and stereoscopic display device |
US15/377,353 US20170094267A1 (en) | 2010-04-28 | 2016-12-13 | Stereoscopic image reproduction device and method, stereoscopic image capturing device, and stereoscopic display device |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/659,638 Division US9560341B2 (en) | 2010-04-28 | 2012-10-24 | Stereoscopic image reproduction device and method, stereoscopic image capturing device, and stereoscopic display device |
Publications (1)
Publication Number | Publication Date |
---|---|
US20170094267A1 true US20170094267A1 (en) | 2017-03-30 |
Family
ID=44861489
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/659,638 Expired - Fee Related US9560341B2 (en) | 2010-04-28 | 2012-10-24 | Stereoscopic image reproduction device and method, stereoscopic image capturing device, and stereoscopic display device |
US15/377,353 Abandoned US20170094267A1 (en) | 2010-04-28 | 2016-12-13 | Stereoscopic image reproduction device and method, stereoscopic image capturing device, and stereoscopic display device |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/659,638 Expired - Fee Related US9560341B2 (en) | 2010-04-28 | 2012-10-24 | Stereoscopic image reproduction device and method, stereoscopic image capturing device, and stereoscopic display device |
Country Status (4)
Country | Link |
---|---|
US (2) | US9560341B2 (en) |
JP (1) | JP5449536B2 (en) |
CN (1) | CN102860019B (en) |
WO (1) | WO2011136190A1 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20170103733A1 (en) * | 2015-10-08 | 2017-04-13 | Xiaomi Inc. | Method and device for adjusting and displaying image |
US20180004684A1 (en) * | 2015-01-21 | 2018-01-04 | Sony Corporation | Information processing device, communication system, information processing method, and program |
Families Citing this family (35)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10298834B2 (en) | 2006-12-01 | 2019-05-21 | Google Llc | Video refocusing |
CN102860019B (en) * | 2010-04-28 | 2015-07-29 | 富士胶片株式会社 | Stereo-picture regenerating unit and method, stereo photographic device, stereoscopic display device |
CN102860017B (en) * | 2010-04-28 | 2015-06-10 | 富士胶片株式会社 | Stereoscopic image pickup device and control method therefor |
JP5750457B2 (en) | 2011-01-25 | 2015-07-22 | 富士フイルム株式会社 | Stereoscopic video processing device, stereoscopic video processing program and recording medium therefor, stereoscopic imaging device, and stereoscopic video processing method |
KR20130024504A (en) * | 2011-08-31 | 2013-03-08 | 삼성전기주식회사 | Stereo camera system and method for controlling convergence |
JP6053276B2 (en) * | 2011-12-02 | 2016-12-27 | キヤノン株式会社 | Imaging apparatus, control method therefor, and image processing apparatus |
US9521395B2 (en) | 2011-10-04 | 2016-12-13 | Canon Kabushiki Kaisha | Imaging apparatus and method for controlling same |
US9858649B2 (en) | 2015-09-30 | 2018-01-02 | Lytro, Inc. | Depth-based image blurring |
US8997021B2 (en) * | 2012-11-06 | 2015-03-31 | Lytro, Inc. | Parallax and/or three-dimensional effects for thumbnail image displays |
US10334151B2 (en) | 2013-04-22 | 2019-06-25 | Google Llc | Phase detection autofocus using subaperture images |
WO2016102338A1 (en) * | 2014-12-24 | 2016-06-30 | Koninklijke Philips N.V. | Autostereoscopic display device |
US10540818B2 (en) | 2015-04-15 | 2020-01-21 | Google Llc | Stereo image generation and interactive playback |
US10419737B2 (en) | 2015-04-15 | 2019-09-17 | Google Llc | Data structures and delivery methods for expediting virtual reality playback |
US10275898B1 (en) | 2015-04-15 | 2019-04-30 | Google Llc | Wedge-based light-field video capture |
US10469873B2 (en) | 2015-04-15 | 2019-11-05 | Google Llc | Encoding and decoding virtual reality video |
US10341632B2 (en) | 2015-04-15 | 2019-07-02 | Google Llc. | Spatial random access enabled video system with a three-dimensional viewing volume |
US10440407B2 (en) | 2017-05-09 | 2019-10-08 | Google Llc | Adaptive control for immersive experience delivery |
US10412373B2 (en) | 2015-04-15 | 2019-09-10 | Google Llc | Image capture for virtual reality displays |
US10565734B2 (en) | 2015-04-15 | 2020-02-18 | Google Llc | Video capture, processing, calibration, computational fiber artifact removal, and light-field pipeline |
US10546424B2 (en) | 2015-04-15 | 2020-01-28 | Google Llc | Layered content delivery for virtual and augmented reality experiences |
US10567464B2 (en) | 2015-04-15 | 2020-02-18 | Google Llc | Video compression with adaptive view-dependent lighting removal |
US11328446B2 (en) | 2015-04-15 | 2022-05-10 | Google Llc | Combining light-field data with active depth data for depth map generation |
US10444931B2 (en) | 2017-05-09 | 2019-10-15 | Google Llc | Vantage generation and interactive playback |
JP6618271B2 (en) * | 2015-05-01 | 2019-12-11 | キヤノン株式会社 | Image processing apparatus, control method therefor, and imaging apparatus |
US9979909B2 (en) | 2015-07-24 | 2018-05-22 | Lytro, Inc. | Automatic lens flare detection and correction for light-field images |
US10275892B2 (en) | 2016-06-09 | 2019-04-30 | Google Llc | Multi-view scene segmentation and propagation |
US10679361B2 (en) | 2016-12-05 | 2020-06-09 | Google Llc | Multi-view rotoscope contour propagation |
US10594945B2 (en) | 2017-04-03 | 2020-03-17 | Google Llc | Generating dolly zoom effect using light field image data |
US10474227B2 (en) | 2017-05-09 | 2019-11-12 | Google Llc | Generation of virtual reality with 6 degrees of freedom from limited viewer data |
US10354399B2 (en) | 2017-05-25 | 2019-07-16 | Google Llc | Multi-view back-projection to a light-field |
US10545215B2 (en) | 2017-09-13 | 2020-01-28 | Google Llc | 4D camera tracking and optical stabilization |
CN108171682B (en) * | 2017-12-04 | 2022-04-19 | 北京中科慧眼科技有限公司 | Binocular synchronous exposure rate detection method and system based on long shot, and storage medium |
US10965862B2 (en) | 2018-01-18 | 2021-03-30 | Google Llc | Multi-camera navigation interface |
US10567744B1 (en) | 2018-09-24 | 2020-02-18 | Cae Inc. | Camera-based display method and system for simulators |
US10567743B1 (en) | 2018-09-24 | 2020-02-18 | Cae Inc. | See-through based display method and system for simulators |
Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130093859A1 (en) * | 2010-04-28 | 2013-04-18 | Fujifilm Corporation | Stereoscopic image reproduction device and method, stereoscopic image capturing device, and stereoscopic display device |
Family Cites Families (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP3735617B2 (en) * | 1996-04-05 | 2006-01-18 | 松下電器産業株式会社 | Multi-view image transmission method, multi-view image display method, and multi-view image display device |
US6163337A (en) * | 1996-04-05 | 2000-12-19 | Matsushita Electric Industrial Co., Ltd. | Multi-view point image transmission method and multi-view point image display method |
GB2354389A (en) * | 1999-09-15 | 2001-03-21 | Sharp Kk | Stereo images with comfortable perceived depth |
WO2001097531A2 (en) * | 2000-06-12 | 2001-12-20 | Vrex, Inc. | Electronic stereoscopic media delivery system |
JP2002095018A (en) * | 2000-09-12 | 2002-03-29 | Canon Inc | Image display controller, image display system and method for displaying image data |
JP4138425B2 (en) * | 2002-09-25 | 2008-08-27 | シャープ株式会社 | Image display apparatus and method |
JP4490074B2 (en) | 2003-04-17 | 2010-06-23 | ソニー株式会社 | Stereoscopic image processing apparatus, stereoscopic image display apparatus, stereoscopic image providing method, and stereoscopic image processing system |
JP2005073049A (en) * | 2003-08-26 | 2005-03-17 | Sharp Corp | Device and method for reproducing stereoscopic image |
JP4578294B2 (en) * | 2005-03-18 | 2010-11-10 | 株式会社エヌ・ティ・ティ・データ三洋システム | Stereoscopic image display device, stereoscopic image display method, and computer program |
JP4763571B2 (en) * | 2006-10-24 | 2011-08-31 | シャープ株式会社 | Stereo image generating device and stereo image decoding device |
KR101427647B1 (en) * | 2007-04-25 | 2014-08-07 | 삼성전자주식회사 | Method and apparatus for packet creating and precessing |
US8390674B2 (en) * | 2007-10-10 | 2013-03-05 | Samsung Electronics Co., Ltd. | Method and apparatus for reducing fatigue resulting from viewing three-dimensional image display, and method and apparatus for generating data stream of low visual fatigue three-dimensional image |
CN101868970B (en) * | 2007-10-19 | 2015-01-21 | 三星电子株式会社 | Method of recording three-dimensional image data |
JP2010169777A (en) * | 2009-01-21 | 2010-08-05 | Sony Corp | Image processing device, image processing method and program |
US8878912B2 (en) * | 2009-08-06 | 2014-11-04 | Qualcomm Incorporated | Encapsulating three-dimensional video data in accordance with transport protocols |
US8629899B2 (en) * | 2009-08-06 | 2014-01-14 | Qualcomm Incorporated | Transforming video data in accordance with human visual system feedback metrics |
GB2473282B (en) * | 2009-09-08 | 2011-10-12 | Nds Ltd | Recommended depth value |
US20110116540A1 (en) * | 2009-11-18 | 2011-05-19 | General Instrument Corporation | Multimedia Content Handling in a Home-Network System |
US20130010064A1 (en) * | 2010-03-24 | 2013-01-10 | Panasonic Corporation | Video processing device |
CN102893613B (en) * | 2010-04-28 | 2016-06-22 | 富士胶片株式会社 | Stereo-picture regenerating unit and method, stereo photographic device, stereoscopic display device |
-
2011
- 2011-04-25 CN CN201180020899.8A patent/CN102860019B/en not_active Expired - Fee Related
- 2011-04-25 WO PCT/JP2011/060095 patent/WO2011136190A1/en active Application Filing
- 2011-04-25 JP JP2012512838A patent/JP5449536B2/en not_active Expired - Fee Related
-
2012
- 2012-10-24 US US13/659,638 patent/US9560341B2/en not_active Expired - Fee Related
-
2016
- 2016-12-13 US US15/377,353 patent/US20170094267A1/en not_active Abandoned
Patent Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130093859A1 (en) * | 2010-04-28 | 2013-04-18 | Fujifilm Corporation | Stereoscopic image reproduction device and method, stereoscopic image capturing device, and stereoscopic display device |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180004684A1 (en) * | 2015-01-21 | 2018-01-04 | Sony Corporation | Information processing device, communication system, information processing method, and program |
US10795831B2 (en) * | 2015-01-21 | 2020-10-06 | Sony Corporation | Information processing device, communication system, information processing method |
US20170103733A1 (en) * | 2015-10-08 | 2017-04-13 | Xiaomi Inc. | Method and device for adjusting and displaying image |
US10026381B2 (en) * | 2015-10-08 | 2018-07-17 | Xiaomi Inc. | Method and device for adjusting and displaying image |
Also Published As
Publication number | Publication date |
---|---|
US9560341B2 (en) | 2017-01-31 |
JPWO2011136190A1 (en) | 2013-07-18 |
CN102860019B (en) | 2015-07-29 |
WO2011136190A1 (en) | 2011-11-03 |
JP5449536B2 (en) | 2014-03-19 |
US20130093859A1 (en) | 2013-04-18 |
CN102860019A (en) | 2013-01-02 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9560341B2 (en) | Stereoscopic image reproduction device and method, stereoscopic image capturing device, and stereoscopic display device | |
US8736671B2 (en) | Stereoscopic image reproduction device and method, stereoscopic image capturing device, and stereoscopic display device | |
US8363091B2 (en) | Stereoscopic image pick-up apparatus | |
JP5584677B2 (en) | Stereo imaging device and image correction method | |
US9256116B2 (en) | Stereoscopic imaging device, image reproducing device, and editing software | |
US9310672B2 (en) | Stereoscopic image capturing device and method of controlling thereof | |
US8773506B2 (en) | Image output device, method and program | |
JP5486697B2 (en) | Stereoscopic video playback device, stereoscopic video playback program and recording medium thereof, stereoscopic display device, stereoscopic imaging device, and stereoscopic video playback method | |
JP5160460B2 (en) | Stereo imaging device and stereo imaging method | |
US8982189B2 (en) | 3D video reproduction device, non-transitory recording medium, 3D display device, 3D imaging device, and 3D video reproduction method | |
JP5750457B2 (en) | Stereoscopic video processing device, stereoscopic video processing program and recording medium therefor, stereoscopic imaging device, and stereoscopic video processing method | |
US9124866B2 (en) | Image output device, method, and recording medium therefor | |
US9094671B2 (en) | Image processing device, method, and recording medium therefor |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO PAY ISSUE FEE |