WO2017159014A1 - 画像処理装置、画像処理方法及び画像処理システム - Google Patents
画像処理装置、画像処理方法及び画像処理システム Download PDFInfo
- Publication number
- WO2017159014A1 WO2017159014A1 PCT/JP2017/000861 JP2017000861W WO2017159014A1 WO 2017159014 A1 WO2017159014 A1 WO 2017159014A1 JP 2017000861 W JP2017000861 W JP 2017000861W WO 2017159014 A1 WO2017159014 A1 WO 2017159014A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- output
- smartphone
- image processing
- zoom
- Prior art date
Links
- 238000012545 processing Methods 0.000 title claims abstract description 82
- 238000003672 processing method Methods 0.000 title claims description 5
- 238000000034 method Methods 0.000 claims description 56
- 230000010365 information processing Effects 0.000 claims description 14
- 238000003384 imaging method Methods 0.000 description 45
- 230000008569 process Effects 0.000 description 45
- 238000010586 diagram Methods 0.000 description 41
- 239000007787 solid Substances 0.000 description 17
- 230000003287 optical effect Effects 0.000 description 16
- 238000006243 chemical reaction Methods 0.000 description 13
- 238000003860 storage Methods 0.000 description 13
- 238000012937 correction Methods 0.000 description 11
- 230000000694 effects Effects 0.000 description 7
- 230000036961 partial effect Effects 0.000 description 7
- 238000004364 calculation method Methods 0.000 description 5
- 230000008859 change Effects 0.000 description 5
- 238000004519 manufacturing process Methods 0.000 description 4
- 241000167854 Bourreria succulenta Species 0.000 description 3
- 235000019693 cherries Nutrition 0.000 description 3
- 230000007423 decrease Effects 0.000 description 3
- 230000005484 gravity Effects 0.000 description 3
- 230000002093 peripheral effect Effects 0.000 description 3
- 230000004075 alteration Effects 0.000 description 2
- 239000013256 coordination polymer Substances 0.000 description 2
- 230000002950 deficient Effects 0.000 description 2
- 230000002829 reductive effect Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 241000791900 Selene vomer Species 0.000 description 1
- 238000003705 background correction Methods 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 229910044991 metal oxide Inorganic materials 0.000 description 1
- 150000004706 metal oxides Chemical class 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000003825 pressing Methods 0.000 description 1
- 230000011514 reflex Effects 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/275—Image signal generators from 3D object models, e.g. computer-generated stereoscopic image signals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/60—Editing figures and text; Combining figures or text
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/04—Texture mapping
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/04—Context-preserving transformations, e.g. by using an importance map
- G06T3/047—Fisheye or wide-angle transformations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
- G11B27/02—Editing, e.g. varying the order of information signals recorded on, or reproduced from, record carriers
- G11B27/031—Electronic editing of digitised analogue information signals, e.g. audio or video signals
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
- G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
- G11B27/19—Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier
- G11B27/28—Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier by using information signals recorded by the same method as the main recording
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
- G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
- G11B27/34—Indicating arrangements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/156—Mixing image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/66—Remote control of cameras or camera parts, e.g. by remote control devices
- H04N23/661—Transmitting camera control signals through networks, e.g. control via the Internet
- H04N23/662—Transmitting camera control signals through networks, e.g. control via the Internet by using master/slave camera arrangements for affecting the control of camera image capture, e.g. placing the camera in a desirable condition to capture a desired image
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/69—Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/695—Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/698—Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
Definitions
- the present invention relates to an image processing apparatus, an image processing method, and an image processing system.
- a method for displaying a panoramic image is known.
- a user interface hereinafter referred to as “UI” for receiving a user instruction for displaying a panoramic image is known.
- the communication terminal displays a thumbnail related to the predetermined area image in the panoramic image, and receives an instruction to select the predetermined thumbnail, thereby displaying the predetermined area image related to the received thumbnail.
- a method is known in which the user can easily find a desired panoramic image or a predetermined area image desired by the user in the panoramic image (for example, Patent Document 1).
- the computer first generates a partial image that is a partial area of the image and includes an arbitrary viewpoint on the image at the center, and a viewpoint list that lists the images and thumbnails of the partial images. Display images.
- the computer receives comment information associating the partial image with the user's comment on the partial image from the server device, and displays the number of comments for each thumbnail of the partial image indicated by the viewpoint list image. .
- a method is known in which a user can grasp a viewpoint in an image with a high degree of attention (for example, Patent Document 2).
- the receiving device downloads the additional information from the additional information management system based on the corresponding image identification information.
- a method is known in which metadata can be used for image processing such as correction of image data by a user (for example, Patent Document 3).
- the conventional method has a problem that it is impossible to add a new image and produce a production effect.
- An object of the present invention is to provide an image processing apparatus capable of adding a new image and producing a production effect.
- FIG. (1) explaining an example of the imaging device which concerns on one Embodiment of this invention.
- FIG. (2) explaining an example of the imaging device which concerns on one Embodiment of this invention.
- FIG. (3) explaining an example of the imaging device which concerns on one Embodiment of this invention.
- FIG. (1) explaining an example of the omnidirectional image which concerns on one Embodiment of this invention.
- FIG. (2) explaining an example of the omnidirectional image which concerns on one Embodiment of this invention.
- FIG. (3) explaining an example of the omnidirectional image which concerns on one Embodiment of this invention.
- FIG. (4) explaining an example of the omnidirectional image which concerns on one Embodiment of this invention.
- FIG. (4) explaining an example of the omnidirectional image which concerns on one Embodiment of this invention.
- FIG. (4) explaining an example of the omnidirectional panoramic image which concerns on one Embodiment of this invention.
- FIG. 6 is a diagram (part 1) for explaining an example of an initial image according to an embodiment of the present invention; It is FIG. (2) for demonstrating an example of the initial image which concerns on one Embodiment of this invention. It is FIG. (3) for demonstrating an example of the initial image which concerns on one Embodiment of this invention. It is FIG. (4) for demonstrating an example of the initial image which concerns on one Embodiment of this invention. It is a figure for demonstrating an example of another zoom process which concerns on one Embodiment of this invention. It is a table
- FIG. (2) for demonstrating an example of the "range” of another zoom process which concerns on one Embodiment of this invention.
- FIG. (3) for demonstrating an example of the "range” of another zoom process which concerns on one Embodiment of this invention.
- FIG. (4) for demonstrating an example of the "range” of another zoom process which concerns on one Embodiment of this invention.
- FIG. (5) for demonstrating an example of the "range” of another zoom process which concerns on one Embodiment of this invention.
- FIG. 1 is a diagram illustrating an example of the overall configuration of an image processing system according to an embodiment of the present invention.
- the image processing system 10 includes a photographing device 1 and a smartphone 2 that is an example of the image processing device.
- the photographing apparatus 1 is a camera having a plurality of optical systems. For example, the photographing apparatus 1 generates an image showing a wide range such as an omnidirectional direction (hereinafter referred to as “omnidirectional image”) based on images photographed using a plurality of optical systems. Next, the imaging device 1 outputs an omnidirectional image or the like to the smartphone 2. Then, the smartphone 2 performs image processing on an input image such as an omnidirectional image.
- the input image that is, the first image is an omnidirectional image will be described.
- the panoramic image is, for example, an omnidirectional image.
- the photographing device 1 and the smartphone 2 are connected by wire or wirelessly.
- the smartphone 2 downloads data indicating the omnidirectional image output from the imaging device 1 from the imaging device 1.
- the connection may be via a network or the like.
- the photographing device 1 and the smartphone 2 may be an integrated device.
- a display device such as a personal computer or a mobile phone may be used.
- the image processing system 10 may further include an information processing apparatus in addition to the photographing apparatus 1 and the smartphone 2.
- the photographing apparatus 1 (FIG. 1) is an apparatus described below, for example.
- FIG. 2 to 4 are diagrams for explaining an example of a photographing apparatus according to an embodiment of the present invention.
- 2 to 4 are diagrams showing an example of the appearance of the photographing apparatus 1.
- FIG. 2 is an example of a front view of the photographing apparatus 1.
- FIG. 3 is an example of a left side view of the photographing apparatus 1.
- FIG. 4 is an example of a plan view of the photographing apparatus 1.
- the photographing apparatus 1 includes a front photographing element 1H1, a rear photographing element 1H2, and a switch 1H3.
- optical systems such as the front photographing element 1H1 and the rear photographing element 1H2 are used for photographing.
- the imaging device 1 produces
- the switch 1H3 is a so-called shutter button, and is an example of an input device for the user to instruct the photographing device 1 to perform photographing.
- the photographing apparatus 1 is held by a user and when the switch 1H3 is pressed by the user, the photographing apparatus 1 performs photographing. Specifically, the photographing apparatus 1 is used for photographing as shown in FIG.
- FIG. 5 is a diagram for explaining an example of photographing by the photographing apparatus according to one embodiment of the present invention.
- the user takes a picture by holding the photographing apparatus 1 by hand and pressing a switch 1H3 shown in FIG.
- the photographing apparatus 1 can photograph all directions of the photographing apparatus 1 with the front photographing element 1H1 (FIG. 2) and the rear photographing element 1H2 (FIG. 2), as shown.
- the image shot in this way is, for example, an image as shown in FIG.
- FIG. 6 is a diagram for explaining an example of an image photographed by the photographing apparatus according to one embodiment of the present invention.
- FIG. 6A is an example of an image photographed by the front photographing element 1H1 (FIG. 2).
- FIG. 6B is an example of an image photographed by the rear imaging element 1H2 (FIG. 2).
- FIG. 6C shows an image photographed by the front photographing element 1H1, that is, the image shown in FIG. 6A and an image photographed by the rear photographing element 1H2, that is, FIG. 6B. It is an example of the image produced
- an image photographed by the front photographing element 1H1 is an image having a wide range on the front side of the photographing apparatus 1, for example, a range of 180 ° in view angle. .
- an image photographed by the front photographing element 1H1 has distortion aberration when the front photographing element 1H1 uses an optical system for photographing a wide range, for example, a so-called fisheye lens. That is, the image shown in FIG. 6A is a so-called hemispherical image (hereinafter referred to as “hemispherical image”) that shows one wide range of the photographing apparatus 1 and has distortion.
- each optical system is preferably in the range of 180 ° to 200 °.
- the angle of view exceeds 180 ° or more, there is an image region to be superimposed when the hemispherical image shown in FIG. 6A and the hemispherical image shown in FIG. Is easy to generate a spherical image.
- the image photographed by the rear photographing element 1H2 is an image having a wide range on the rear side of the photographing apparatus 1, for example, a range of 180 ° in view angle. is there.
- the image captured by the rear imaging element 1H2 is a hemispherical image similar to that shown in FIG.
- the imaging apparatus 1 performs processing such as distortion correction processing and composition processing, and based on the front hemispherical image shown in FIG. 6A and the rear hemispherical image shown in FIG. Then, the image shown in FIG. 6C is generated. That is, the image shown in FIG. 6C is an example of an image generated by a method such as a so-called Mercator projection or equirectangular projection, that is, an omnidirectional image.
- the first image is not limited to an image generated by the photographing apparatus 1.
- the first image may be, for example, an image captured by another camera or the like, or an image generated based on an image captured by another camera.
- the first image is desirably an image obtained by photographing a wide viewing angle range with a so-called omnidirectional camera or a so-called wide-angle lens camera as shown in FIG.
- the first image will be described by taking an omnidirectional image as an example, but the first image is not limited to an omnidirectional image.
- the first image may be an image taken with a compact camera, a single-lens reflex camera, a smartphone, or the like.
- the image may be a panoramic image that extends horizontally or vertically.
- the shutter button is controlled by the switch 1H3 being pressed by the user, the shutter button may be controlled remotely from a display device such as the smartphone 2.
- FIG. 7 is a block diagram for explaining an example of the hardware configuration of the photographing apparatus according to an embodiment of the present invention.
- the imaging apparatus 1 includes an imaging unit 1H4, an image processing unit 1H7, an imaging control unit 1H8, a CPU (Central Processing Unit) 1H9, and a ROM (Read-Only Memory) 1H10.
- the photographing apparatus 1 includes an SRAM (Static Random Access Memory) 1H11, a DRAM (Dynamic Random Access Memory) 1H12, and an operation I / F (Interface) 1H13.
- SRAM Static Random Access Memory
- DRAM Dynamic Random Access Memory
- I / F Interface
- the imaging device 1 includes a network I / F 1H14, a wireless I / F 1H15, and an antenna 1H16. Further, hardware included in the photographing apparatus 1 is connected via a bus 1H17, and inputs / outputs data or signals via the bus 1H17.
- the photographing unit 1H4 has a front photographing element 1H1 and a rear photographing element 1H2. Further, a lens 1H5 is installed corresponding to the front imaging element 1H1, and a lens 1H6 is installed corresponding to the rear imaging element 1H2.
- the front photographing element 1H1 and the rear photographing element 1H2 are so-called camera units.
- the front imaging element 1H1 and the rear imaging element 1H2 each have an optical sensor such as a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device).
- the front photographing element 1H1 converts light incident through the lens 1H5 and generates image data indicating a hemispherical image or the like.
- the rear imaging element 1H2 converts light incident through the lens 1H6 and generates image data indicating a hemispherical image or the like.
- the photographing unit 1H4 outputs the respective image data generated by the front photographing element 1H1 and the rear photographing element 1H2 to the image processing unit 1H7.
- the output image data is, for example, a front hemispherical image shown in FIG. 6A and a rear hemispherical image shown in FIG.
- the front imaging element 1H1 and the rear imaging element 1H2 may further include other optical elements such as a diaphragm or a low-pass filter in addition to the lens in order to perform high-quality imaging. Further, the front imaging element 1H1 and the rear imaging element 1H2 may perform so-called defective pixel correction or so-called camera shake correction in order to perform high-quality imaging.
- the image processing unit 1H7 generates an omnidirectional image shown in FIG. 6C based on the image data input from the photographing unit 1H4. Details of the processing for generating the omnidirectional image will be described later.
- the photographing control unit 1H8 is a control device that controls the hardware of the photographing device 1.
- the CPU 1H9 is a calculation device that performs calculation and data processing for realizing each processing, and a control device that controls hardware. For example, the CPU 1H9 executes each process based on a program installed in advance.
- ROM 1H10, SRAM 1H11, and DRAM 1H12 are examples of storage devices.
- the ROM 1H10 stores, for example, programs, data, parameters, and the like for causing the CPU 1H9 to execute processing.
- the SRAM 1H11 and the DRAM 1H12 store, for example, a program used by the CPU 1H9 to execute processing based on the program, data used by the program, data generated by the program, and the like.
- the photographing apparatus 1 may further include an auxiliary storage device such as a hard disk.
- the operation I / F 1H13 is an interface that is connected to an input device such as the switch 1H3 and performs a process of inputting a user operation on the photographing apparatus 1.
- the operation I / F 1H13 is an input device such as a switch, a connector for connecting the input device, a cable, a circuit that processes a signal input from the input device, a driver, and a control device.
- the operation I / F 1H13 may further include an output device such as a display.
- the operation I / F 1H13 may be a so-called touch panel in which an input device and an output device are integrated.
- the operation I / F 1H 13 may have an interface such as USB (Universal Serial Bus), and may connect a recording medium such as a flash memory to the photographing apparatus 1. Accordingly, the operation I / F 1H13 may input / output data from the photographing apparatus 1 to the recording medium.
- USB Universal Serial Bus
- the switch 1H3 may be a power switch, a parameter input switch, or the like for performing an operation other than the operation related to the shutter.
- the network I / F 1H14, the wireless I / F 1H15, and the antenna 1H16 connect the external apparatus and the photographing apparatus 1 by wireless or wired.
- the imaging device 1 is connected to the network via the network I / F 1H 14 and transmits data to the smartphone 2 (FIG. 1).
- the network I / F 1H14, the wireless I / F 1H15, and the antenna 1H16 may be hardware that is connected to another external device via a wired connection such as a USB. That is, the network I / F 1H14, the wireless I / F 1H15, and the antenna 1H16 may be connectors and cables.
- the bus 1H17 is used to input / output data and the like between hardware included in the photographing apparatus 1. That is, the bus 1H17 is a so-called internal bus. Specifically, the bus 1H17 is, for example, PCI Express (Peripheral Component Interconnect Bus Express) or the like.
- PCI Express Peripheral Component Interconnect Bus Express
- the imaging device 1 is not limited to the case where there are two imaging elements. For example, you may have three or more imaging elements. Furthermore, the imaging device 1 may capture a plurality of partial images by changing the imaging angle of one imaging element. Moreover, the imaging device 1 is not limited to an optical system using a fisheye lens. For example, a wide angle lens may be used.
- the processing performed by the photographing apparatus 1 may be performed by another apparatus.
- part or all of the processing may be performed by the smartphone 2 or another information processing apparatus connected via the network, with the imaging apparatus 1 transmitting data and parameters.
- the image processing system may include a plurality of information processing apparatuses and perform processing in a distributed, redundant, or parallel manner.
- FIG. 8 is a block diagram illustrating an example of a hardware configuration of a smartphone according to an embodiment of the present invention.
- the smartphone 2 as an example of the information processing apparatus includes an auxiliary storage device 2H1, a main storage device 2H2, an input / output device 2H3, a state sensor 2H4, a CPU 2H5, and a network I / F 2H6.
- the hardware included in the smartphone 2 is connected via a bus 2H7, and inputs / outputs data or signals via the bus 2H7.
- the auxiliary storage device 2H1 stores data, parameters, programs, and the like.
- the auxiliary storage device 2H1 is, for example, a hard disk, a flash SSD (Solid State Drive), or the like.
- the data stored in the auxiliary storage device 2H1 may be partially or entirely stored redundantly or alternatively by a file server or the like connected by the network I / F 2H6.
- the main storage device 2H2 is a so-called memory that becomes a storage area used by a program for executing processing. That is, the main storage device 2H2 stores data, programs, parameters, and the like. Specifically, the main storage device 2H2 is, for example, an SRAM (Static Random Access Memory), a DRAM, or the like. The main storage device 2H2 may further include a control device that performs storage and retrieval.
- SRAM Static Random Access Memory
- the input / output device 2H3 is an output device that displays an image or a processing result and an input device that inputs a user operation.
- the input / output device 2H3 is a so-called touch panel, a peripheral circuit, a driver, and the like.
- the input / output device 2H3 displays, for example, a predetermined GUI (Graphical User Interface), an image processed image, and the like to the user.
- GUI Graphic User Interface
- the input / output device 2H3 inputs an operation by the user.
- the state sensor 2H4 is a sensor that detects the state of the smartphone 2. Specifically, the state sensor 2H4 is a gyro sensor or an angle sensor. For example, the state sensor 2H4 determines whether one side of the sides of the smartphone 2 is greater than or equal to a predetermined angle with respect to the horizontal. That is, the state sensor 2H4 detects whether the smartphone 2 is in a vertical posture state or a horizontal posture state.
- the CPU 2H5 is a calculation device that performs calculation and data processing for realizing each processing, and a control device that controls hardware.
- the CPU 2H5 may include a plurality of CPUs, devices, or a plurality of cores in order to perform processing in parallel, redundancy, or distribution.
- the smartphone 2 may have a GPU (Graphics Processing Unit) or the like inside or outside for image processing.
- the network I / F 2H 6 is connected to an external device via a network by wireless or wired.
- the network I / F 2H6 includes an antenna, a peripheral circuit, a driver, and the like for inputting and outputting data and the like.
- the smartphone 2 inputs image data from the imaging device 1 (FIG. 1) or the like by the CPU 2H5 and the network I / F 2H6.
- the smartphone 2 outputs data and the like to the photographing apparatus 1 and the like by the CPU 2H5 and the network I / F 2H6.
- the information processing apparatus is not limited to a smartphone. That is, the information processing apparatus may be a computer other than a smartphone.
- the information processing apparatus may be a PC (Personal Computer), a PDA (Personal Digital Assistance), a tablet, a mobile phone, or a combination thereof.
- FIG. 9 is a sequence diagram illustrating an example of overall processing by the image processing system according to an embodiment of the present invention.
- the image processing system 10 performs the following processing to generate an output image.
- step S0701 the imaging device 1 performs processing for generating an omnidirectional image.
- the omnidirectional image is generated from, for example, the hemispherical images shown in FIGS. 6A and 6B taken in advance by the processing shown in FIGS. .
- FIG. 10 is a diagram showing the hemispherical image shown in FIG. 6A by connecting lines where the incident angles in the horizontal direction and the vertical direction are equal to the optical axis by lines.
- ⁇ the incident angle in the horizontal direction with respect to the optical axis
- ⁇ the incident angle in the direction perpendicular to the optical axis
- FIG. 11 is a diagram showing the hemispherical image shown in FIG. 6B by connecting lines where the incident angles in the horizontal direction and the vertical direction are equal to the optical axis by lines, as in FIG. .
- FIG. 12 is a diagram for explaining an example of an image processed by the Mercator projection.
- the image shown in FIG. 12 is generated when the images shown in FIGS. 10 and 11 are associated with an LUT (LookUpTable) generated in advance and processed by equirectangular projection. It is. Then, after the state shown in FIG. 12 is reached, when the images shown in FIGS. 10 and 11 are combined as shown in FIG. 13, an omnidirectional image is generated by the imaging device.
- the composition process is a process for generating an omnidirectional image using two hemispherical images in the state shown in FIG. 12, for example. Note that the composition process is not limited to a process in which hemispherical images in the state illustrated in FIG.
- the imaging device first preprocesses the hemispherical image shown in FIG. 6A and arranges it at the center of the omnidirectional image. To do. Next, the imaging device divides the image obtained by pre-processing the hemispherical image shown in FIG. 6B into sizes that can be arranged in the left and right parts, and synthesizes the hemispherical images in FIG. An omnidirectional image shown in c) may be generated.
- the process of generating the omnidirectional image is not limited to the process using equirectangular projection.
- the pixel arrangement of the hemispherical image shown in FIG. 11 and the pixel arrangement of the hemispherical image shown in FIG. 10 are upside down, and the arrangement of the respective pixels in the ⁇ direction is horizontal. There is a case that it is the reverse of the top and bottom.
- the imaging apparatus performs a 180 ° roll rotation process in order to align the hemispherical image shown in FIG. 11 with the arrangement of the pixels in the ⁇ direction and the ⁇ direction in FIG. May be performed.
- the process of generating the omnidirectional image may be a distortion correction process for correcting the respective distortion aberrations of the hemispherical images shown in FIGS.
- the processing for generating the omnidirectional image may be performed by, for example, shading correction, gamma correction, white balance, camera shake correction, optical black correction processing, defective pixel correction processing, edge enhancement processing, linear correction processing, or the like. Good. Note that, for example, when the shooting range of the hemisphere image and the shooting range of the other hemispheric image overlap, the combining process performs correction using the pixel of the subject shot in the overlapping shooting range, A hemispherical image can be synthesized with high accuracy.
- the imaging device 1 generates an omnidirectional image from a plurality of hemispheric images to be captured by such a process of generating an omnidirectional image.
- the omnidirectional image may be generated by another process.
- an example using an omnidirectional image generated by the illustrated method will be described.
- step S0702 the smartphone 2 acquires the omnidirectional image generated in step S0701 via the network or the like.
- the smartphone 2 acquires the omnidirectional image illustrated in FIG. 13 will be described as an example.
- step S0703 the smartphone 2 generates an omnidirectional panoramic image from the omnidirectional image acquired in step S0702.
- the omnidirectional panoramic image is an image as shown in FIG.
- FIG. 14 is a diagram illustrating an example of a panoramic image according to an embodiment of the present invention.
- the smartphone 2 generates the omnidirectional panoramic image shown in FIG. 14 from the acquired omnidirectional image shown in FIG.
- the omnidirectional panoramic image is an image obtained by pasting the omnidirectional image into a spherical shape (3D model).
- the process of generating the omnidirectional panoramic image is realized by, for example, an API (Application Programming Interface) such as OpenGL ES (OpenGL (registered trademark) for Embedded Systems).
- an API Application Programming Interface
- OpenGL ES OpenGL (registered trademark) for Embedded Systems
- the omnidirectional panoramic image is obtained by first dividing the pixels of the omnidirectional image into triangles, and then connecting vertices P (hereinafter referred to as “vertices P”) of the triangles to obtain polygons. It is generated by pasting as
- step S0704 the smartphone 2 inputs an operation for causing the user to select an image.
- the smartphone 2 displays an image obtained by reducing the panoramic image generated in step S0703, a so-called thumbnail image, or the like. That is, when a plurality of omnidirectional panoramic images are stored in the smartphone 2, the smartphone 2 causes the user to select an omnidirectional panoramic image to be processed from the plurality of omnidirectional panoramic images. Therefore, thumbnail images are output as a list.
- step S0704 for example, the smartphone 2 inputs an operation for selecting one thumbnail image from a list of thumbnail images by the user.
- the processing is performed on the omnidirectional panoramic image selected in step S0704 as a processing target.
- step S0705 the smartphone 2 generates an initial image based on the omnidirectional panoramic image selected in step S0704.
- the initial image is, for example, an image as shown in FIGS.
- FIG. 15 to 18 are diagrams for explaining an example of the initial image according to the embodiment of the present invention.
- FIG. 15 is a diagram illustrating a three-dimensional coordinate system for explaining an example of an initial image according to an embodiment of the present invention.
- the smartphone 2 sets up the virtual camera 3 at the position of the origin, and generates each image from the viewpoint from the virtual camera 3.
- the omnidirectional panoramic image is expressed like a solid sphere CS in the coordinate system shown in FIG.
- the virtual camera 3 corresponds to the viewpoint of the user who views the omnidirectional panoramic image with respect to the omnidirectional panoramic image that is the solid sphere CS from the installation position, that is, the origin.
- FIG. 16 is a diagram for explaining an example of the predetermined area by the virtual camera according to the embodiment of the present invention. Specifically, FIG. 16 is an example showing FIG. 15 in three views. That is, FIG.15 and FIG.16 shows the same state. Accordingly, in FIG. 16, it is assumed that the virtual camera 3 is installed at the origin as shown.
- FIG. 17 is a projection view of an example of a predetermined area by the virtual camera according to the embodiment of the present invention.
- the predetermined area T is an area obtained by projecting the viewing angle of the virtual camera 3 onto the solid sphere CS.
- the smartphone 2 generates an image for the predetermined region T. That is, the smartphone 2 generates an image based on the predetermined area T.
- the predetermined area T is determined as follows, for example.
- FIG. 18 is a diagram for explaining an example of information for determining a predetermined area by the virtual camera according to the embodiment of the present invention.
- the predetermined area T is determined by, for example, predetermined area information (x, y, ⁇ ).
- the viewing angle ⁇ is an angle indicating the angle of view of the virtual camera 3 as shown in FIG.
- the coordinates of the center point CP of the predetermined area T are indicated by (x, y) of the predetermined area information.
- the initial image is an image generated based on a predetermined area T determined in advance and determined based on a so-called initial setting.
- the zoom process is a process for generating an image in which a predetermined area is enlarged or reduced based on an input operation by the user.
- change amount dz the operation amount input by the zoom operation.
- the smartphone 2 acquires the change amount dz based on the input operation by the user.
- the smart phone 2 calculates following (2) Formula based on the variation
- ⁇ in the above equation (2) is the viewing angle ⁇ of the virtual camera 3 shown in FIG.
- m shown in the above equation (2) is a coefficient for adjusting the zoom amount, and is a value set in advance by the user.
- ⁇ 0 in the above equation (2) is the viewing angle ⁇ of the virtual camera 3 in the initial state, that is, the viewing angle ⁇ in the initial image generated in step S0705.
- the smartphone 2 determines the range of the predetermined region T shown in FIG. 18 and the like using the viewing angle ⁇ calculated based on the above equation (2) as the projection matrix.
- the smartphone 2 calculates the following equation (3).
- the viewing angle ⁇ is calculated based on the sum of the amount of change due to each zoom operation.
- the smartphone 2 can maintain a consistent operability by performing the calculation from the viewing angle ⁇ of the omnidirectional sphere.
- the zoom process is not limited to the process based on the above formula (2) or the above formula (3).
- the zoom process may be realized by combining changes in the viewing angle ⁇ and the viewpoint position of the virtual camera 3 (FIG. 16). Specifically, zoom processing as shown in FIG. 19 may be performed.
- FIG. 19 is a diagram for explaining an example of another zoom process according to an embodiment of the present invention.
- the figure is a model diagram for explaining another zoom process.
- the solid sphere CS is the same as the solid sphere CS shown in FIG. Further, in the following description, an example in which the radius of the solid sphere CS is “1” will be described.
- the origin shown is the initial position of the virtual camera 3. Then, the virtual camera 3 moves on the optical axis, that is, the Z axis shown in FIG. Next, the movement amount d of the virtual camera 3 is indicated by a distance from the origin. For example, when the virtual camera 3 is located at the origin, that is, in the initial state, the movement amount d is “0”.
- the field angle ⁇ and the viewing angle ⁇ coincide.
- the angle of view ⁇ and the viewing angle ⁇ indicate different ranges.
- Another zoom process is a process of changing the angle of view ⁇ as shown in FIG. 20, for example.
- FIG. 20 is a table for explaining an example of another zoom process according to the embodiment of the present invention.
- the explanatory table 4 shows an example when the range of the field angle ⁇ is 60 ° to 300 °.
- the smartphone 2 determines which of the viewing angle ⁇ and the movement amount d of the virtual camera 3 is preferentially changed based on the zoom designation value ZP.
- the “range” is a range determined based on the zoom designation value ZP.
- the “output magnification” is an output magnification of an image calculated based on an image parameter determined by another zoom process. Further, the “zoom designation value ZP” is a value corresponding to the angle of view to be output.
- the process for determining the movement amount d and the viewing angle ⁇ is changed based on the zoom designation value ZP.
- the range of the zoom designation value ZP is divided into four ranges “A to B”, “B to C”, “C to D”, and “D to E”.
- view angle ⁇ is the view angle ⁇ corresponding to the image parameter determined by another zoom process.
- parameter to be changed is a description for explaining parameters to be changed by four methods based on the zoom designation value ZP. “Remarks” is a note on “parameters to be changed”.
- viewWH is a value indicating the width or height of the output area. For example, if the output area is horizontally long, “viewWH” indicates a width value. On the other hand, if the output area is vertically long, “viewWH” indicates a height value. That is, “viewWH” is a value indicating the size of the output area in the longitudinal direction.
- “imgWH” is a value indicating the width or height of the output image. For example, if the output area is horizontally long, “viewWH” indicates the width value of the output image. On the other hand, if the output area is vertically long, “viewWH” indicates the value of the height of the output image. That is, “viewWH” is a value indicating the size of the output image in the longitudinal direction.
- imgDeg is a value indicating the angle of the display range of the output image. Specifically, when indicating the width of the output image, “imgDeg” is 360 °. On the other hand, when indicating the height of the output image, “imgDeg” is 180 °.
- FIGS. 21 to 25 are diagrams for explaining an example of “range” of another zoom process according to the embodiment of the present invention.
- the illustrated example shows an example of a “range” displayed on an image and an image when the above-described another zoom process is performed.
- zoom-out will be described as an example using the illustrated example.
- or FIG. 25 shows an example of the image output.
- the right diagrams in each of FIGS. 21 to 25 are diagrams showing an example of the state of the virtual camera 3 at the time of output in the same model diagram as FIG.
- FIG. 21 illustrates an example of an image and “range” that are output when a zoom designation value ZP in which “range” in the explanatory table 4 illustrated in FIG. 20 is “A to B” is input.
- the zoom designation value ZP becomes a value of “A to B”
- the zoom designation value ZP is “A to B” and the movement amount d of the virtual camera 3 is changed in a state where the viewing angle ⁇ is fixed as shown in FIG.
- the movement amount d of the virtual camera 3 is changed so as to increase in a state where the viewing angle ⁇ is fixed will be described.
- zoom-out processing can be realized by setting the zoom designation value ZP to “A to B”, fixing the viewing angle ⁇ , and increasing the movement amount d of the virtual camera 3.
- the zoom designation value ZP is “A to B”
- the movement amount d of the virtual camera 3 is from “0” to the radius of the solid sphere CS.
- the moving amount d of the virtual camera 3 is a value of “0 to 1”.
- the moving amount d of the virtual camera 3 is a value corresponding to the zoom designation value ZP.
- FIG. 22 shows an example of an image and “range” that are output when the zoom designation value ZP in which “range” in the explanatory table 4 (FIG. 20) is “B to C” is input.
- “B to C” is a value for which the zoom designation value ZP is larger than “A to B”.
- the zoom designation value ZP is set to “B to C”, and the movement amount d of the virtual camera 3 is fixed to a value at which the virtual camera 3 is positioned on the outer edge of the solid sphere CS. In this way, as shown in FIG. 22, the movement amount d of the virtual camera 3 is fixed to “1” which is the radius of the solid sphere CS.
- the viewing angle ⁇ is changed in a state where the zoom designation value ZP is “B to C” and the movement amount d of the virtual camera 3 is fixed.
- zoom-out processing can be realized by setting the zoom designation value ZP to “B to C”, fixing the movement amount d of the virtual camera 3 and increasing the viewing angle ⁇ .
- the zoom designation value ZP is “B to C”
- the viewing angle ⁇ is calculated as “ ⁇ / 2”.
- the zoom designation value ZP is “B to C”
- the range of the viewing angle ⁇ is “60 °” to “120 °” which is a fixed value when “A to B”. Up to.
- the angle of view ⁇ matches the zoom designation value ZP.
- the zoom designation value ZP is “A to B” and “B to C”
- the angle of view ⁇ increases.
- FIG. 23 shows an example of an output image and “range” when a zoom designation value ZP in which “range” in the explanatory table 4 (FIG. 20) is “C to D” is input.
- “C to D” has a larger zoom designation value ZP than “B to C”.
- the zoom designation value ZP is “C to D” and the movement amount d of the virtual camera 3 is changed in a state where the viewing angle ⁇ is fixed.
- the movement amount d of the virtual camera 3 is changed so as to increase in a state where the viewing angle ⁇ is fixed will be described.
- the movement amount d of the virtual camera 3 is calculated by an expression based on the zoom designation value ZP shown in the explanatory table 4 (FIG. 20).
- the zoom designation value ZP is “C to D”
- the movement amount d of the virtual camera 3 is changed to the maximum display distance dmax1.
- the maximum display distance dmax1 is a distance at which the solid sphere CS can be displayed at the maximum in the output area of the smartphone 2 (FIG. 1).
- the output area is, for example, the size of the screen on which the smartphone 2 outputs an image or the like.
- the maximum display distance dmax1 is in a state as shown in FIG. 24, for example.
- the maximum display distance dmax1 is calculated by the following equation (4).
- viewW is a value indicating the width of the output area in the smartphone 2.
- viewH is a value indicating the height of the output area in the smartphone 2. The same applies hereinafter.
- the maximum display distance dmax1 is calculated based on the output area of the smartphone 2, that is, the values of “viewW” and “viewH”.
- FIG. 24 shows an example of an image and “range” that are output when a zoom designation value ZP in which “range” in the explanatory table 4 (FIG. 20) is “D to E” is input.
- “D to E” is a value for which the zoom designation value ZP is larger than “C to D”.
- the zoom designation value ZP is “C to D” and the movement amount d of the virtual camera 3 is changed in a state where the viewing angle ⁇ is fixed. Further, the moving amount d of the virtual camera 3 is changed to the limit display distance dmax2.
- the limit display distance dmax2 is a distance where the solid sphere CS is inscribed in the output area of the smartphone 2. Specifically, the limit display distance dmax2 is calculated by the following equation (5). The limit display distance dmax2 is in a state as shown in FIG. 25, for example.
- the limit display distance dmax2 is calculated based on the values of “viewW” and “viewH” that are output areas in the smartphone 2.
- the limit display distance dmax2 indicates the maximum range that the smartphone 2 can output, that is, a limit value that can increase the movement amount d of the virtual camera 3.
- the smartphone 2 is input so that the zoom designation value ZP falls within the range described in the explanation table 4 (FIG. 20), that is, the value of the movement amount d of the virtual camera 3 is equal to or less than the limit display distance dmax2. May be limited.
- the smartphone 2 is in a state in which the output image is fitted to the screen that is the output area, or in a state in which the image can be output to the user at a predetermined output magnification, and zoom-out can be realized. Then, through the processes “D to E”, the smartphone 2 can recognize that the image output to the user is an omnidirectional panorama.
- the angle of view ⁇ is different from the zoom designation value ZP.
- the angle of view ⁇ is continuous between the ranges shown in the explanatory table 4 and FIGS. 21 to 25, but the angle of view ⁇ may not increase uniformly by zooming out to the wide angle side. . That is, for example, when the zoom designation value ZP is “C to D”, the angle of view ⁇ increases with the movement amount d of the virtual camera 3. On the other hand, when the zoom designation value ZP is “D to E”, the angle of view ⁇ decreases with the movement amount d of the virtual camera 3.
- the decrease in the movement amount d of the virtual camera 3 when the zoom designation value ZP is “D to E” is because the outer area of the solid sphere CS is reflected.
- the smartphone 2 when the zoom designation value ZP designates a wide field of view of 240 ° or more, the smartphone 2 outputs an image with less discomfort to the user by changing the movement amount d of the virtual camera 3, In addition, the angle of view ⁇ can be changed.
- the smartphone 2 fixes the viewing angle ⁇ of the virtual camera 3 and increases the movement amount d of the virtual camera 3. In this way, the smartphone 2 can reduce the increase in the viewing angle ⁇ of the virtual camera 3 by fixing the viewing angle ⁇ of the virtual camera 3. Furthermore, by reducing the increase in the viewing angle ⁇ of the virtual camera 3, the smartphone 2 can output an image with less distortion to the user.
- the smartphone 2 When the viewing angle ⁇ of the virtual camera 3 is fixed and the smartphone 2 increases the movement amount d of the virtual camera 3, that is, moves the virtual camera 3 away from the smartphone 2, the smartphone 2 gives the user a sense of opening wide-angle display. Can be given.
- the smartphone 2 since the movement of moving the virtual camera 3 away is similar to the movement when a human confirms a wide range, the smartphone 2 realizes zoom-out with little discomfort by moving the virtual camera 3 away. it can.
- the zoom designation value ZP When the zoom designation value ZP is “D to E”, the angle of view ⁇ decreases as the zoom designation value ZP changes in the wide-angle direction.
- the smartphone 2 can give the user a sense of moving away from the three-dimensional sphere CS by decreasing the angle of view ⁇ .
- the smart phone 2 can output an image with little discomfort to the user by giving the user a feeling of moving away from the solid sphere CS.
- the smartphone 2 can output an image with less discomfort to the user by another zoom process described in the explanatory table 4 (FIG. 20).
- the smartphone 2 is not limited to changing only the movement amount d or the viewing angle ⁇ of the virtual camera 3 described in the explanation table 4. That is, the smartphone 2 only needs to change the movement amount d or the viewing angle ⁇ of the virtual camera 3 preferentially in the explanatory table 4. For example, the fixed value is changed to a sufficiently small value for adjustment. May be.
- the smartphone 2 is not limited to zooming out. For example, the smartphone 2 may zoom in.
- the omnidirectional image as an example of the first image is an image as shown in FIG.
- FIG. 26 is a diagram illustrating an example of a first image according to an embodiment of the present invention.
- the first image IMG11 as illustrated is selected by the user's selection or the like.
- the first image is a still image.
- the first image may be a moving image.
- step S ⁇ b> 0706 the smartphone 2 inputs a user operation for selecting the second image.
- a second image an image newly added to the omnidirectional image that is an example of the first image
- the user inputs an operation for adding additional information such as a hash tag to the first image to the smartphone.
- the smart phone 2 specifies the image corresponding to the hash tag etc. which are added to a 1st image, and makes it a 2nd image. For example, assume that the following image is the second image.
- FIG. 27 is a diagram showing an example of a second image according to an embodiment of the present invention.
- the smartphone specifies the second image corresponding to the input hash tag.
- the corresponding second image IMG21 is an image showing “petals” of the same type as the flower indicated by the first image, as illustrated.
- object OBJ the object or the like indicated by the second image.
- object OBJ the image corresponding to the hash tag “sakura”, that is, the first image contains cherry blossoms as the same type of object as the hash tag.
- the second image is an image corresponding to a part of the same type of object (for example, cherry petals), or the entire same type of object (for example, cherry petals, leaves and branches). .).
- the hash tag is “sakura” (sakura) and the second image is “petal”, but the present invention is not limited to this.
- the hash tag is “ame” (may be rain or rain)
- the second image is an image of “water drop” corresponding to rain
- the hash tag is “yuki” (snow or snow, etc.).
- the second image is a “snow” image.
- the second image may be an image showing one object. A plurality of second images may be selected.
- data indicating the second image IMG21 is input to the smartphone 2 in advance.
- the data indicating the second image IMG21 may be downloaded by the smartphone 2 from a server or the like that registers the second image in advance, or may be searched and downloaded based on a hash tag by a search engine or the like.
- the smartphone 2 may recognize the image included in the first image and select the second image.
- the first image IMG11 illustrated in FIG. 26 includes an image indicating “petals”. Therefore, when image recognition processing is performed on the first image IMG11, the smartphone 2 recognizes “petals” that appear in the first image IMG11. And based on the recognition result which the recognition part recognized, the smart phone 2 is 2nd image IMG21 which shows the same content as "the petal" reflected in the 1st image shown in FIG. 26 from the 2nd image input previously. May be specified.
- the second image may not be the same as the image included in the first image, but a similar image as illustrated in FIG. 27 may be specified.
- the second image is indicated by, for example, JSON (Java Script (registered trademark) Object Notification) format 3D model data and data indicating the color and pattern of the object OBJ. Note that the second image may be shown in other formats.
- JSON Java Script (registered trademark) Object Notification) format 3D model data and data indicating the color and pattern of the object OBJ. Note that the second image may be shown in other formats.
- step S0707 the smartphone 2 places the second image on the 3D model to which the first image is pasted, that is, the solid sphere CS (FIG. 15) or the like. Accordingly, the first image is first pasted on the 3D model by the smartphone 2, and the object OBJ (FIG. 27) indicated by the second image is further added.
- step S0708 the smartphone 2 generates an output image based on the 3D model in which the second image is arranged in step S0707. Specifically, as in the case of generating the initial image illustrated in FIGS. 15 to 18, a predetermined area of the 3D model in which the second image is arranged is output as an output image.
- the output image is an image as shown in FIG.
- FIG. 28 is a diagram showing an example of an output image according to an embodiment of the present invention.
- the smartphone 2 outputs the first output image IMGOUT1 in the first frame and then the second output image IMGOUT2 in the second frame.
- the first output image IMGOUT1 and the second output image IMGOUT2 differ in that a plurality of objects OBJ indicated by the second image, that is, “petals” are added.
- the object OBJ may be copied and output more than the number of input second images.
- each object is output at a different angle and position.
- the object OBJ is, for example, in the gravitational direction (in the illustrated example, the Y-axis direction and the first direction DR1)).
- the object “OBJ” of “petal” is output so as to appear to fall in the first direction DR1 due to gravity in the image. Therefore, with respect to the first image shown in FIG. 26, the smartphone 2 can add a second image and produce an effect that “petals” dance. Note that the effect is not limited to the illustrated example.
- the second image may be a different object from the object shown in FIG.
- the object OBJ Since the output image is generated based on the 3D model, the object OBJ is output so as to move in a predetermined direction. Specifically, in the case shown in FIG. 28, for example, the second image is output so as to move as shown in FIG.
- FIG. 29 is a diagram showing an example of the movement of the second image in the output image according to the embodiment of the present invention.
- FIG. 28 is a diagram for explaining the second image in the output image shown in FIG.
- the smartphone is determined based on the parameters specified by the user, etc.
- An image showing is output. That is, an image output based on the predetermined region T is an output image.
- an output image is generated so that the object moves in the first direction DR1.
- the direction in which the object moves, the position where the object starts to be output, the speed at which the object moves, or a combination thereof may be specified by the user or the like.
- the direction in which the object moves is the first direction DR1, but is not limited to the first direction DR1.
- the moving direction of the object is the horizontal direction (X-axis direction, the second direction DR2 in the figure), the depth direction (Z-axis direction, the third direction DR3 in the figure), or a combination thereof. May be in the direction.
- the position where the object starts to be output may be input as a Y coordinate value, for example.
- a coordinate value indicating the start position SPOS may be input.
- the object is output so as to move from the start position SPOS in the first direction DR1.
- the speed at which the object moves may be input, for example, by the number of coordinates that the object moves for each frame. In this way, the user can set in detail how the object moves.
- the output image is generated based on the 3D model, the object OBJ moves in a predetermined direction even if so-called viewpoint conversion is performed in which the predetermined region T shown in FIG. 29 is changed. Is output. Specifically, for example, the output image after the viewpoint conversion is output as shown in FIG.
- FIG. 30 is a diagram illustrating an example of an output image at a looking down viewpoint according to an embodiment of the present invention.
- the output image IMGOUTDW shown in the figure is an image that is output when the predetermined area, which is the output area, is changed from the case shown in FIG. 29 and the viewpoint is converted from the top-to-bottom viewpoint in the Y-axis direction, the so-called look-down viewpoint. It is.
- the direction in which the object OBJ moves is assumed to be the first direction DR1 as in the case shown in FIG.
- the object OBJ is displayed differently from FIG. 28 on the screen.
- the object OBJ is output from the outside of the screen toward the center and gradually reduced. That is, the object OBJ is output so as to move from the near side to the far side in the depth direction of the screen. In other words, even when the viewpoint is looking down, the output of the object OBJ moves so that the object OBJ “petal” appears to fall in the first direction DR1 due to gravity.
- the smartphone can output the second image so as to move in a predetermined direction even if there is a viewpoint conversion.
- the second image is output so as to move as shown in FIG. 30
- FIG. 31 is a diagram showing an example of the movement of the second image in the output image of the looking down viewpoint according to the embodiment of the present invention.
- the figure is a figure explaining the 2nd image in the output image shown in FIG. 30, ie, the motion of an object, with a 3D model diagram.
- FIG. 31 becomes a “looking down” viewpoint by the viewpoint conversion, and the position of the predetermined region T is different.
- the smartphone generates an output image so that the object moves in the first direction DR1.
- the output image after the viewpoint conversion is output as shown in FIG.
- FIG. 32 is a diagram showing an example of an output image at a viewpoint looking up according to an embodiment of the present invention.
- the output image IMGOUTUP shown in FIG. 29 is an image that is output when the predetermined area as the output area is changed from the case shown in FIG. 29 and the viewpoint is changed from the bottom-up viewpoint in the Y-axis direction, that is, the so-called looking-up viewpoint. It is.
- the direction in which the object OBJ moves is assumed to be the first direction DR1 as in the case shown in FIG.
- the object OBJ is displayed differently from that in FIG. Specifically, in the output image IMGOUTUP, as illustrated, the object OBJ is output from the center of the screen toward the outside of the screen and gradually enlarged. That is, the object OBJ is output so as to move from the back to the front in the depth direction of the screen. That is, even when the viewpoint is looked up, the object OBJ moves so that the object OBJ “petal” appears to fall in the first direction DR1 due to gravity. As described above, when the output image is generated based on the 3D model, the smartphone can output the second image so as to move in a predetermined direction even if there is a viewpoint conversion.
- FIG. 33 is a diagram illustrating an example of the movement of the second image in the viewpoint output image according to the embodiment of the present invention.
- FIG. 29 is a diagram for explaining the second image in the output image shown in FIG.
- FIG. 33 becomes a viewpoint of “looking up” by the viewpoint conversion, and the position of the predetermined region T is different.
- the smartphone generates an output image so that the object moves in the first direction DR1.
- the output image is generated based on the 3D model, when zoom processing is performed in which the range of the predetermined region T shown in FIG. 29 is changed, the object OBJ is output to the zoom magnification or the like. Specifically, for example, an output image when zoom-out zoom processing is performed is output as shown in FIG.
- FIG. 34 is a diagram showing an example of a zoomed-out output image according to an embodiment of the present invention.
- the zoom-out output image IMGOUTZO shown in the figure is an output image that shows a wider range than the output image shown in FIG. 28 when the output image shown in FIG. 28 is zoomed out.
- Other settings are the same as those shown in FIG. Specifically, for example, it is assumed that the direction in which the object OBJ moves is the first direction DR1. Therefore, each subject is output smaller in the output image shown in FIG. 34 than the subject shown in the output image shown in FIG. Then, as shown in the figure, the object OBJ is also output by the smartphone so that it looks small according to the zoom-out magnification or the like. Further, the output image when the zoom process for zooming in is performed in the same way, and is output as shown in FIG. 35, for example.
- FIG. 35 is a view showing an example of an output image zoomed in according to an embodiment of the present invention.
- the illustrated zoom-in output image IMGOUTZI is an output image that shows a narrower range than the output image shown in FIG. 28 when the output image shown in FIG. 28 is zoomed in.
- Other settings are the same as those shown in FIG. Specifically, for example, it is assumed that the direction in which the object OBJ moves is the first direction DR1. Therefore, each subject is output larger in the output image shown in FIG. 34 than the subject shown in the output image shown in FIG.
- the zoom-in output image IMGOUTZI is an image in which the central portion of the image shown in FIG. 28 is enlarged. Then, as shown in the figure, the object OBJ is also output by the smartphone so as to appear larger in accordance with the zoom-out magnification or the like.
- the smartphone when the output image is generated based on the 3D model, the smartphone can output the second image in the interlocked direction or size even when zoom processing, viewpoint conversion, and the like are performed. Therefore, the smart phone can add a 2nd image to the 1st image and can produce a production effect.
- FIG. 36 is a functional block diagram showing an example of a functional configuration of the image processing apparatus according to an embodiment of the present invention.
- the smartphone 2 includes an input unit 2F1 and a generation unit 2F2.
- the input unit 2F1 inputs the first image and the second image to be added to the first image.
- the input unit 2F1 is realized by the input / output device 2H3 (FIG. 8) or the like.
- the generation unit 2F2 arranges the second image on the 3D model to which the first image is pasted, and generates an output image based on the 3D model.
- the generation unit 2F2 is realized by the CPU 2H5 (FIG. 8) or the like.
- the smartphone 2 may be configured to further include a recognition unit that recognizes an image included in the first image.
- the recognition unit is realized by the CPU 2H5 or the like.
- the smartphone 2 can output the second image arranged in the 3D model so as to move in a predetermined direction. That is, the smartphone 2 can output the second image so as to move in a predetermined direction even when viewpoint conversion, zoom processing, or a combination thereof is performed.
- the smartphone 2 can add a second image to the image in which the first image is captured and produce an effect that the petals dance, even if the viewpoint is converted. it can.
- the embodiment according to the present invention may be realized by a program described in a programming language or the like. That is, the embodiment according to the present invention may be realized by a program for causing a computer such as an image processing apparatus to execute an image processing method.
- the program can be stored and distributed in a recording medium such as a flash memory, an SD (registered trademark) card, or an optical disk.
- the program can be distributed through a telecommunication line such as the Internet.
- a part or all of the processing may be realized by being processed by a programmable device (PD) such as a field programmable gate array (FPGA).
- PD programmable device
- FPGA field programmable gate array
- a part or all of the processing may be realized by being processed by ASIC (Application Specific Integrated Circuit).
- the information processing apparatus is not limited to a smartphone but may be a server or the like. Further, the information processing apparatus is not limited to one information processing apparatus, and may be configured by a plurality of information processing apparatuses. That is, the embodiment according to the present invention may be realized by an image processing system having one or more information processing apparatuses.
- JP 2014-6880 A Japanese Patent Laid-Open No. 2015-18013 JP 2014-30104 A
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Studio Devices (AREA)
- Image Processing (AREA)
- Processing Or Creating Images (AREA)
Abstract
Description
第1画像及び第2画像を入力する入力部と、
前記第1画像を貼り付けた3Dモデルに、前記第2画像を配置させて出力画像を生成する生成部と
を有することを特徴とする。
図1は、本発明の一実施形態に係る画像処理システムの全体構成の一例を説明する図である。図示する例では、画像処理システム10は、撮影装置1と、画像処理装置の例であるスマートフォン2とを有する。
撮影装置1(図1)は、例えば、以下に説明するような装置である。
図7は、本発明の一実施形態に係る撮影装置のハードウェア構成の一例を説明するブロック図である。図示するように、撮影装置1は、撮影ユニット1H4と、画像処理ユニット1H7、撮影制御ユニット1H8と、CPU(Central Processing Unit)1H9と、ROM(Read-Only Memory)1H10とを有する。また、撮影装置1は、SRAM(Static Random Access Memory)1H11と、DRAM(Dynamic Random Access Memory)1H12と、操作I/F(Interface)1H13とを有する。さらに、撮影装置1は、ネットワークI/F1H14と、無線I/F1H15と、アンテナ1H16とを有する。また、撮影装置1が有するハードウェアは、バス1H17で接続され、バス1H17を介してデータ又は信号を入出力する。
図8は、本発明の一実施形態に係るスマートフォンのハードウェア構成の一例を説明するブロック図である。図示するように、情報処理装置の一例であるスマートフォン2は、補助記憶装置2H1と、主記憶装置2H2と、入出力装置2H3と、状態センサ2H4と、CPU2H5と、ネットワークI/F2H6とを有する。また、スマートフォン2が有するハードウェアは、バス2H7で接続され、バス2H7を介してデータ又は信号を入出力する。
図9は、本発明の一実施形態に係る画像処理システムによる全体処理の一例を説明するシーケンス図である。例えば、画像処理システム10は、以下のような処理を行い、出力画像を生成する。
図36は、本発明の一実施形態に係る画像処理装置の機能構成の一例を示す機能ブロック図である。図示するように、スマートフォン2は、入力部2F1と、生成部2F2とを有する。
1 撮影装置
2 スマートフォン
OBJ オブジェクト
IMG11 第1画像
IMG21 第2画像
Claims (8)
- 第1画像及び第2画像を入力する入力部と、
前記第1画像を貼り付けた3Dモデルに、前記第2画像を配置させて出力画像を生成する生成部と
を有する画像処理装置。 - 前記第1画像は、付加情報を有し、
前記第2画像は、前記付加情報に基づいて特定される請求項1に記載の画像処理装置。 - 前記第1画像に含まれる画像を認識する認識部を更に有し、
前記第2画像は、前記認識部によって認識される画像と同一又は類似の内容を示す画像である請求項1に記載の画像処理装置。 - 前記第1画像は、撮影装置で撮影された画像又は複数の撮影装置で撮影された複数の画像に基づいて生成された画像である請求項1に記載の画像処理装置。
- 前記第2画像は、前記第1画像の各フレームにおいて、所定の方向に向かって動くように出力される請求項1に記載の画像処理装置。
- 前記第2画像が動く方向、前記第2画像が出力され始める位置、前記第2画像が動く速度又はこれらの組み合わせを設定する請求項5に記載の画像処理装置。
- 1以上の情報処理装置を有する画像処理システムであって、
第1画像及び第2画像を入力する入力部と、
前記第1画像を貼り付けた3Dモデルに、前記第2画像を配置させて出力画像を生成する生成部と
を有する画像処理システム。 - 画像処理装置が行う画像処理方法であって、
前記画像処理装置が、第1画像及び第2画像を入力する入力手順と、
前記画像処理装置が、前記第1画像を貼り付けた3Dモデルに、前記第2画像を配置させて出力画像を生成する生成手順と
を有する画像処理方法。
Priority Applications (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201780016868.2A CN108781251A (zh) | 2016-03-15 | 2017-01-12 | 图像处理装置、图像处理方法和图像处理*** |
EP17766022.2A EP3432567A4 (en) | 2016-03-15 | 2017-01-12 | Image processing device, image processing method and image processing system |
JP2018505280A JPWO2017159014A1 (ja) | 2016-03-15 | 2017-01-12 | 画像処理装置、画像処理方法及び画像処理システム |
KR1020187026218A KR20180111991A (ko) | 2016-03-15 | 2017-01-12 | 화상 처리 장치, 화상 처리 방법 및 화상 처리 시스템 |
US16/118,872 US10785470B2 (en) | 2016-03-15 | 2018-08-31 | Image processing apparatus, image processing method, and image processing system |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2016-051471 | 2016-03-15 | ||
JP2016051471 | 2016-03-15 |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/118,872 Continuation US10785470B2 (en) | 2016-03-15 | 2018-08-31 | Image processing apparatus, image processing method, and image processing system |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2017159014A1 true WO2017159014A1 (ja) | 2017-09-21 |
Family
ID=59851840
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2017/000861 WO2017159014A1 (ja) | 2016-03-15 | 2017-01-12 | 画像処理装置、画像処理方法及び画像処理システム |
Country Status (6)
Country | Link |
---|---|
US (1) | US10785470B2 (ja) |
EP (1) | EP3432567A4 (ja) |
JP (1) | JPWO2017159014A1 (ja) |
KR (1) | KR20180111991A (ja) |
CN (1) | CN108781251A (ja) |
WO (1) | WO2017159014A1 (ja) |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP3461124B1 (en) * | 2017-09-25 | 2024-05-01 | Ricoh Company, Ltd. | Communication terminal, image communication system, display control method, and carrier means |
WO2020166582A1 (ja) * | 2019-02-15 | 2020-08-20 | 株式会社Jvcケンウッド | 画像調整システム、画像調整装置、及び画像調整方法 |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2009188780A (ja) * | 2008-02-07 | 2009-08-20 | Kuu-Kan Com Inc | 映像演出システム及び映像演出方法 |
JP2015001760A (ja) * | 2013-06-13 | 2015-01-05 | 任天堂株式会社 | 画像処理システム、画像処理装置、画像処理プログラム、および画像処理方法 |
Family Cites Families (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP0930585B1 (en) * | 1998-01-14 | 2004-03-31 | Canon Kabushiki Kaisha | Image processing apparatus |
CA2435935A1 (en) * | 2003-07-24 | 2005-01-24 | Guylain Lemelin | Optical 3d digitizer with enlarged non-ambiguity zone |
KR101288971B1 (ko) * | 2007-02-16 | 2013-07-24 | 삼성전자주식회사 | 모델링 방법 및 장치 |
WO2008147561A2 (en) * | 2007-05-25 | 2008-12-04 | Google Inc. | Rendering, viewing and annotating panoramic images, and applications thereof |
JP2009230635A (ja) * | 2008-03-25 | 2009-10-08 | Olympus Imaging Corp | 画像データ生成装置、画像データ生成方法、および画像データ生成用プログラム |
KR20110052124A (ko) | 2009-11-12 | 2011-05-18 | 삼성전자주식회사 | 파노라마 이미지 생성 및 조회 방법과 이를 이용한 휴대 단말기 |
JP5250598B2 (ja) * | 2010-10-05 | 2013-07-31 | 株式会社ソニー・コンピュータエンタテインメント | 画像表示装置および画像表示方法 |
US8452081B2 (en) * | 2011-01-11 | 2013-05-28 | Eastman Kodak Company | Forming 3D models using multiple images |
CN102834849B (zh) * | 2011-03-31 | 2016-08-31 | 松下知识产权经营株式会社 | 进行立体视图像的描绘的图像描绘装置、图像描绘方法、图像描绘程序 |
EP2602792A3 (en) * | 2011-11-16 | 2013-08-21 | Magix AG | System and method for generating stereoscopic 3d multimedia works from 2d input material |
US9782147B2 (en) * | 2012-03-06 | 2017-10-10 | Analogic Corporation | Apparatus and methods for localization and relative positioning of a surgical instrument |
JP6186775B2 (ja) | 2012-05-31 | 2017-08-30 | 株式会社リコー | 通信端末、表示方法、及びプログラム |
JP6011117B2 (ja) | 2012-07-31 | 2016-10-19 | 株式会社リコー | 受信装置、画像共有システム、受信方法、及びプログラム |
JP6075066B2 (ja) | 2012-12-28 | 2017-02-08 | 株式会社リコー | 画像管理システム、画像管理方法、及びプログラム |
US9618747B2 (en) * | 2013-03-13 | 2017-04-11 | Jason Villmer | Head mounted display for viewing and creating a media file including omnidirectional image data and corresponding audio data |
JP6446766B2 (ja) | 2013-07-08 | 2019-01-09 | 株式会社リコー | プログラム、表示制御装置、記録媒体及び表示制御システム |
JP5897624B2 (ja) * | 2014-03-12 | 2016-03-30 | ファナック株式会社 | ワークの取出工程をシミュレーションするロボットシミュレーション装置 |
EP2933779A1 (en) * | 2014-04-15 | 2015-10-21 | Alcatel Lucent | Capturing an environment with objects |
KR101997500B1 (ko) * | 2014-11-25 | 2019-07-08 | 삼성전자주식회사 | 개인화된 3d 얼굴 모델 생성 방법 및 장치 |
-
2017
- 2017-01-12 CN CN201780016868.2A patent/CN108781251A/zh active Pending
- 2017-01-12 WO PCT/JP2017/000861 patent/WO2017159014A1/ja active Application Filing
- 2017-01-12 JP JP2018505280A patent/JPWO2017159014A1/ja active Pending
- 2017-01-12 KR KR1020187026218A patent/KR20180111991A/ko active IP Right Grant
- 2017-01-12 EP EP17766022.2A patent/EP3432567A4/en not_active Withdrawn
-
2018
- 2018-08-31 US US16/118,872 patent/US10785470B2/en active Active
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2009188780A (ja) * | 2008-02-07 | 2009-08-20 | Kuu-Kan Com Inc | 映像演出システム及び映像演出方法 |
JP2015001760A (ja) * | 2013-06-13 | 2015-01-05 | 任天堂株式会社 | 画像処理システム、画像処理装置、画像処理プログラム、および画像処理方法 |
Non-Patent Citations (1)
Title |
---|
See also references of EP3432567A4 * |
Also Published As
Publication number | Publication date |
---|---|
KR20180111991A (ko) | 2018-10-11 |
US20180376130A1 (en) | 2018-12-27 |
US10785470B2 (en) | 2020-09-22 |
EP3432567A1 (en) | 2019-01-23 |
CN108781251A (zh) | 2018-11-09 |
JPWO2017159014A1 (ja) | 2019-01-31 |
EP3432567A4 (en) | 2019-02-27 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN111510590B (zh) | 图像处理***、图像处理方法 | |
JP6044328B2 (ja) | 画像処理システム、画像処理方法およびプログラム | |
JP5835383B2 (ja) | 情報処理方法、情報処理装置、およびプログラム | |
JP5835384B2 (ja) | 情報処理方法、情報処理装置、およびプログラム | |
CN109218606B (zh) | 摄像控制设备、其控制方法及计算机可读介质 | |
JP5743016B2 (ja) | 画像を生成する装置および方法 | |
JP6350695B2 (ja) | 装置、方法、およびプログラム | |
JP2018109946A (ja) | 表示装置、プログラム、表示方法 | |
WO2017159014A1 (ja) | 画像処理装置、画像処理方法及び画像処理システム | |
JP6673459B2 (ja) | 画像処理装置、画像処理システム及びプログラム | |
JP6583486B2 (ja) | 情報処理方法、情報処理プログラムおよび情報処理装置 | |
CN107580157B (zh) | 图像处理装置、图像处理方法 | |
JP6394682B2 (ja) | 方法および画像処理装置 | |
JP6128185B2 (ja) | 装置、方法、およびプログラム | |
JP6992829B2 (ja) | 画像処理システム、画像処理方法およびプログラム | |
JP6777208B2 (ja) | プログラム | |
JP6705477B2 (ja) | 画像処理システム、画像処理方法およびプログラム | |
JP2016021267A (ja) | 装置、方法、およびプログラム | |
JP2022043132A (ja) | 画像処理システム、画像処理方法およびプログラム | |
JP2017224330A (ja) | 装置、方法、およびプログラム |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 2018505280 Country of ref document: JP |
|
ENP | Entry into the national phase |
Ref document number: 20187026218 Country of ref document: KR Kind code of ref document: A |
|
WWE | Wipo information: entry into national phase |
Ref document number: 1020187026218 Country of ref document: KR |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2017766022 Country of ref document: EP |
|
ENP | Entry into the national phase |
Ref document number: 2017766022 Country of ref document: EP Effective date: 20181015 |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 17766022 Country of ref document: EP Kind code of ref document: A1 |