EP3739867A1 - Imaging device, control apparatus, imaging method, and storage medium - Google Patents
Imaging device, control apparatus, imaging method, and storage medium Download PDFInfo
- Publication number
- EP3739867A1 EP3739867A1 EP20173281.5A EP20173281A EP3739867A1 EP 3739867 A1 EP3739867 A1 EP 3739867A1 EP 20173281 A EP20173281 A EP 20173281A EP 3739867 A1 EP3739867 A1 EP 3739867A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- imaging
- unit
- imaging unit
- wide
- angle image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000003384 imaging method Methods 0.000 title claims abstract description 578
- 238000012545 processing Methods 0.000 claims abstract description 19
- 238000000034 method Methods 0.000 claims description 36
- 230000008859 change Effects 0.000 claims description 19
- 230000003287 optical effect Effects 0.000 claims description 19
- 238000004590 computer program Methods 0.000 claims description 4
- 230000004044 response Effects 0.000 description 47
- 230000007246 mechanism Effects 0.000 description 42
- 230000008569 process Effects 0.000 description 33
- 238000010586 diagram Methods 0.000 description 25
- 230000007704 transition Effects 0.000 description 16
- 230000006870 function Effects 0.000 description 13
- 230000007423 decrease Effects 0.000 description 8
- 230000005540 biological transmission Effects 0.000 description 6
- 238000012544 monitoring process Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000007429 general method Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
- G03B37/00—Panoramic or wide-screen photography; Photographing extended surfaces, e.g. for surveying; Photographing internal surfaces, e.g. of pipe
- G03B37/04—Panoramic or wide-screen photography; Photographing extended surfaces, e.g. for surveying; Photographing internal surfaces, e.g. of pipe with cameras or projectors providing touching or overlapping fields of view
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/95—Computational photography systems, e.g. light-field imaging systems
- H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/40—Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled
- H04N25/41—Extracting pixel data from a plurality of image sensors simultaneously picking up an image, e.g. for increasing the field of view by combining the outputs of a plurality of sensors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/45—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/62—Control of parameters via user interfaces
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/66—Remote control of cameras or camera parts, e.g. by remote control devices
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/667—Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/69—Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/695—Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/698—Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/265—Mixing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Definitions
- the present invention relates to an imaging device used for applications such as monitoring.
- multi-lens cameras images captured by a plurality of cameras
- imaging devices capable of acquiring an image in a wider imaging range (hereinafter referred to as a "wide-angle image”) than when a single camera is used have been proposed.
- Japanese Patent Laid-Open No. 2004-118786 proposes an imaging device which generates a wide-angle image by performing matching processing while shifting images captured by cameras in multi-lens cameras to acquire an amount of shifting between a plurality of images.
- imaging devices used when a user wants to change a direction desired to be monitored or the like after the imaging devices are installed imaging devices having a so-called pan tilt zoom (PTZ) function in which an imaging direction can be controlled have been proposed.
- PTZ pan tilt zoom
- an object of the present invention is to propose an imaging device including a plurality of imaging units and capable of appropriately controlling whether to generate a wide-angle image in accordance with control of the imaging ranges of the plurality of imaging units.
- an imaging device According to a first aspect, there is provided an imaging device according to claim 1. According to a second aspect, there is provided a control apparatus according to claim 11. According to a third aspect, there is provided an imaging method according to claim 14. According to a fourth aspect, there is provided a program according to claim 15. According to a fifth aspect, there is provided a computer-readable storage medium according to claim 16.
- the imaging device includes electronic devices having a plurality of imaging units such as digital still cameras, digital movie cameras, smartphones with a camera, and tablet computers with a camera.
- Fig. 1 illustrates an imaging device in this practical example and a monitoring system using the same.
- Fig. 1 is an arrangement view of an imaging device 100 when viewed from above (a +Z axis side) and Fig. 2 is an internal functional block diagram.
- the imaging device 100 includes a first imaging unit 110, a second imaging unit 120, a first driving mechanism 111, a second driving mechanism 121, a control unit 130, a combination (synthesizing) processing unit 140, and a first transmission/reception unit 150.
- Each of the first driving mechanism 111 and the second driving mechanism 121 functions as a driving unit and is configured to be able to control an imaging direction or a the imaging range of the first imaging unit 110 and the second imaging unit 120 at least in the same plane (in an XY plane in Fig. 1 ).
- the imaging device in this practical example is configured to be able to control the imaging direction in a pan direction.
- the first driving mechanism 111 and the second driving mechanism 121 illustrated in Fig. 2 include a motor and a gear and are configured to be able to rotate the first imaging unit 110 and the second imaging unit 120 using an axis 101 in Fig. 1 as a rotation axis by controlling electric power used for driving the motor.
- the electric power used for driving the motor is controlled by the control unit 130.
- the imaging device 100 is configured to be able to change an imaging direction of each of the first imaging unit 110 and the second imaging unit 120 in the XY plane.
- the imaging directions of both of the first imaging unit 110 and the second imaging unit 120 can be changed in this practical example, a driving mechanism capable of controlling at least one of the imaging directions is provided and thus the imaging directions of the first imaging unit and the second imaging unit may only be relatively changeable.
- the first imaging unit 110 includes an imaging optical system 112 and a solid-state imaging device 113
- the second imaging unit 120 includes an imaging optical system 122 and a solid-state imaging device 123
- the first imaging unit 110 and the second imaging unit 120 form subject images on the solid-state imaging devices 113 and 123 through the imaging optical systems 112 and 122 to acquire an image.
- the driving and the signal reading of each of the solid-state imaging devices 113 and 123 is controlled by the control unit 130.
- the control unit 130 has a central processing unit (CPU) as a computer built thereinto and functions as a control unit configured to execute various operations of the entire device on the basis of a computer program stored in a memory (not shown).
- CPU central processing unit
- the combination processing unit 140 functions as a combining unit and is configured to combine a first image signal 114 acquired by the first imaging unit 110 and a second image signal 124 acquired by the second imaging unit 120 to generate a wide-angle image (panoramic image) signal 134.
- a so-called pattern matching technique for obtaining a correlation coefficient while shifting an overlapping portion of images an amount of positional shifting between a plurality of images is obtained and the wide-angle image signal 134 is generated.
- an overlapping range (an amount of overlapping) between an imaging range 115 of the first imaging unit 110 and an imaging range 125 of the second imaging unit 120 is investigated. Furthermore, the wide-angle image signal 134 is not generated when the overlapping range (the amount of overlapping) is less than a first threshold value and the wide-angle image signal 134 is generated when the overlapping range (the amount of overlapping) is equal to or more than the first threshold value. Details will be described later. When the wide-angle image signal 134 is not generated in the combination processing unit 140, the first image signal 114 and the second image signal 124 are transferred to the first transmission/reception unit 150 without being combined.
- the first transmission/reception unit 150 transmits an image signal (the first image signal 114, the second image signal 124, or the wide-angle image signal 134) which has been transferred from the combination processing unit 140 to an external client device 180 (not shown) over a wired or wireless network or the like.
- the external client device 180 transmits a command for controlling the imaging device 100 to the first transmission/reception unit 150 over a second transmission/reception unit 181 and a network and the imaging device 100 receives the command and returns a response to the command to the client device 180.
- the command is, for example, for controlling the first driving mechanism 111 and the second driving mechanism 121. That is to say, a user can control directions of the first imaging unit 110 and the second imaging unit 120 from the external client device 180 (control apparatus) over a network.
- the client device is, for example, an external device such as a personal computer (PC) and the network is configured of wired local area networks (LANs), wireless LANs, or the like. Furthermore, a configuration in which electric power is supplied to the imaging device 100 over the network may be provided.
- PC personal computer
- LANs local area networks
- wireless LANs wireless LANs
- a control unit configured to control the inside of the client device 180 (control apparatus) and having a computer such as a CPU built thereinto will be denoted with reference numeral 182. Furthermore, the control unit 182 has a memory (not shown) built thereinto and a computer program configured to control an operation of the CPU in the control unit is stored in the memory.
- a display unit serving as a display unit and configured to display an image signal or the like which has been transferred from the imaging device 100 will be denoted with reference numeral 183.
- An operation unit including input units such as various switches and touch panels will be denoted with reference numeral 184. The user can give various instructions to the imaging device 100 by operating the operation unit.
- the combination processing unit 140 is provided inside the imaging device 100 in the practical example, the combination processing unit 140 may be provided in the client device 180. Furthermore, an imaging system is constituted of the imaging device 100 and the client device 180.
- a first embodiment illustrated in Fig. 2 an example in which the imaging device 100 includes the first transmission/reception unit 150, an image is transmitted to the client device 180 side, and an operation is performed through a command from the client device 180 side is illustrated.
- the imaging device 100 may integrally include a memory configured to store image data and a part of the operation unit 184 or the like such as the display unit 183 configured to display an image and a switch configured to receive the user's assistance. That is to say, the imaging device 100 itself may have a function of the client device 180 integrally built thereinto.
- the wide-angle image signal 134 is generated. Furthermore, a configuration in which, when the size of the overlapping range 135 is less than the first threshold value, the wide-angle image signal 134 is not generated is provided.
- Figs. 3 and 4 are diagram illustrating a first imaging range 115 of the first imaging unit 110, a second imaging range 125 of the second imaging unit 120, and an overlapping range 135 thereof.
- Figs. 3A and 3B illustrate a case in which the overlapping range (the amount of overlapping) 135 is equal to or more than a first threshold value 136 and Figs. 4A and 4B illustrate a case in which the overlapping range 135 is less than the first threshold value 136.
- the overlapping range 135 corresponds to a hatched region.
- an amount of positional shifting between the first image signal 114 and the second image signal 124 can be obtained.
- a general method as described in Patent Document 1 can be used. For example, a correlation coefficient such as a sum of squared difference (SSD) and a sum of absolute difference (SAD) may be calculated while shifting the first image signal 114 and the second image signal 124 so that an amount of positional shifting having the highest correlation coefficient may be obtained.
- SSD sum of squared difference
- SAD sum of absolute difference
- a correlation coefficient having a high accuracy can be obtained.
- the overlapping range 135 increases, the number of pixel signals used for calculating an amount of positional shifting increases. Therefore, when the overlapping range 135 increases, an amount of positional shifting having a high accuracy can be obtained and a wide-angle image having a high quality can be generated.
- the overlapping range 135 decreases, the detection of the amount of positional shifting decreases. Thus, it is difficult to generate a wide-angle image having a high quality.
- the wide-angle image signal 134 having a high quality When the wide-angle image signal 134 having a high quality can be generated, the visibility of the subject in the overlapping range 135 when the wide-angle image signal 134 is obtained by combining the first image signal 114 and the second image signal 124 is higher than that when the first image signal 114 and the second image signal 124 are observed independently.
- the wide-angle image signal 134 when it is difficult to generate the wide-angle image signal 134 having a high quality, even if the wide-angle image signal 134 is generated, the wide-angle image signal 134 is likely to be generated with an incorrect amount of positional shifting. That is to say, when the wide-angle image is generated, information included in the first image signal 114 and the second image signal 124 is likely to be lost.
- the visibility of the subject when the first image signal 114 and the second image signal 124 are displayed separately is superior than that when the wide-angle image signal 134 is displayed.
- the overlapping range 135 is equal to or more than the first threshold value 136.
- the wide-angle image signal 134 is generated in the combination processing unit 140.
- the wide-angle image signal is not generated in the combination processing unit 140. Furthermore, the first image signal 114 and the second image signal 124 are transmitted as they are and displayed separately. With such a configuration, it is possible to provide an image having excellent visibility of the subject regardless of the imaging directions of the first imaging unit 110 and the second imaging unit 120.
- the imaging direction of the imaging unit means a direction of an optical axis of the imaging optical system of the imaging unit.
- the first threshold value 136 can also be determined using the number of pixels in the overlapping range 135.
- the number of pixels of the first image signal 114 and the second image signal 124 included in the overlapping range 135 is preferably 100 or more and more preferably 1000 or more.
- the first threshold value 136 may be defined as a relative ratio to the first imaging range 115 or the second imaging range 125.
- the overlapping range is preferably 5% or more and more preferably 20% or more with respect to a relatively narrower imaging range of the first imaging range 115 and the second imaging range 125.
- Fig. 1 illustrates a case in which the imaging device 100 is constituted of two imaging units, i.e., the first imaging unit 110 and the second imaging unit 120
- the imaging device 100 may be constituted of three or more imaging units.
- a determination unit is configured to determine whether or not to generate the wide-angle image by the combination processing unit 140. The determination is based on a relation between an imaging range of the first imaging unit 110 and an imaging range of the second imaging unit 120, or a position relation between the first imaging unit 110 and the second imaging unit 120 in the predetermined direction.
- a wide-angle image signal is generated by combining the image signals acquired using the imaging units.
- the overlapping range between the imaging ranges of neighboring imaging units is less than the first threshold value, a configuration in which the image signals acquired using the imaging units are not combined is provided.
- Fig. 5 illustrates an example in which the imaging device 100 is constituted of four imaging units, i.e., the imaging units 110, 120, 160, and 170 arranged in this order clockwise.
- the imaging ranges of the imaging units are 115, 125, 165, and 175.
- one wide-angle image signal 134 is generated by combining all of the image signals 114, 124 and image signals 164 and 174 obtained through imaging using the imaging units.
- Fig. 6 illustrates a case in which the overlapping ranges between the imaging units 110 and 120 and between the imaging units 120 and 160 are equal to or more than the first threshold value 136 and the overlapping ranges between the imaging units 170 and 110 and between the imaging units 160 and 170 are less than the first threshold value 136.
- the wide-angle image signal 134 is generated by combining the image signals 114, 124, and 164 from the imaging units 110, 120, and 160 and the image signal 174 imaged using the imaging unit 170 is not combined with another image signal.
- Fig. 7 illustrates a case in which the overlapping ranges between the imaging units 110 and 120 and between the imaging units 160 and 170 is equal to or more than the first threshold value 136 and the overlapping ranges between the imaging units 170 and 110 and between the imaging units 120 and 160 are less than the first threshold value 136.
- two wide-angle image signals 134A and 134B are generated by combining the image signals 114 and the image signal 124 from the imaging units 110 and 120 and combining the image signal 164 and the image signal 174 from the imaging units 160 and 170.
- An imaging device 200 (not shown) in a second embodiment and the imaging device 100 illustrated in the first embodiment differ in view of only the functions of the first driving mechanism and the second driving mechanism.
- a first driving mechanism 211 (not shown) and a second driving mechanism 212 (not shown) are configured in such a manner that the imaging directions of the first imaging unit 210 (not shown) and the second imaging unit 220 (not shown) are set so that the first imaging unit 210 (not shown) and the second imaging unit 220 (not shown) can rotate in two directions orthogonal to each other.
- a rotation mechanism centering on a Z axis a so-called pan driving mechanism
- a rotation mechanism capable of controlling an angle with respect to the Z axis is provided.
- a wide-angle image signal 234 is generated.
- the wide-angle image signal 234 is not generated.
- Figs. 9A and 9B are diagrams illustrating the first imaging range 215 of the first imaging unit 210, the second imaging range 225 of the second imaging unit 220, and the overlapping range 235.
- Fig. 9A illustrates a case in which the overlapping range 235 is large and
- Fig. 9B illustrates a case in which the overlapping range 235 is small.
- the overlapping range 235 corresponds to a hatched region.
- a determination concerning whether a wide-angle image signal is generated may be performed using the product of the deviation between the imaging direction of the first imaging unit 210 and the imaging direction of the second imaging unit 220 in the pan direction thereof and the deviation thereof in the tilt direction thereof.
- a wide-angle image signal is generated and when the product of the deviations in the pan direction and the tilt direction is larger than the first threshold value, the wide-angle image signal is not generated.
- imaging signals of the first imaging unit 210 and the second imaging unit 220 are output as they are without being combined and displayed separately.
- This practical example may be applied to an imaging device including only a tilt driving mechanism instead of an imaging device including both a pan driving mechanism and a tilt driving mechanism.
- Fig. 10 is a diagram for explaining a third embodiment.
- An imaging device 300 (not shown) in the third embodiment and the imaging device 200 illustrated in the second embodiment differ in view of only the functions of a first driving mechanism and the second driving mechanism.
- a rotation mechanism capable of rotating imaging units about optical axes of the imaging units (a so-called rotation mechanism) is provided.
- a wide-angle image signal 334 is generated and when the size (the area) of the overlapping range 335 between the imaging range 315 of the first imaging unit 310 and the imaging range 325 of the second imaging unit 320 is less than the first threshold value, the wide-angle image signal 334 is not generated.
- Figs. 10A and 10B illustrate the first imaging range 315 of the first imaging unit 310, the second imaging range 325 of the second imaging unit 320, and the overlapping range 335.
- Fig. 10A illustrates a case in which the overlapping range 335 is large and
- Fig. 10B illustrates a case in which the imaging direction of each of the imaging units rotates 90 degrees to the right while the imaging direction of each of the imaging units is maintained from the state of Fig. 10A .
- the overlapping range 135 corresponds to a hatched region.
- a length thereof in a horizontal direction thereof may be longer than a length thereof in vertical direction thereof in many cases. Therefore, when the imaging unit rotates around an optical axis, an imaging range of the imaging unit changes. For this reason, as illustrated in Figs. 10A and 10B , a size of the overlapping range 335 changes in accordance with a rotation angle of the imaging unit.
- each of the first driving mechanism and the second driving mechanism may have a part of the above three functions. That is to say, for example, with regard to a rotation function, a rotation unit configured to rotate a corresponding one imaging unit of the first imaging unit and the second imaging unit about an optical axis of at least one of the first imaging unit and the second imaging unit may be provided.
- Figs. 11A and 11B and Figs. 12A and 12B illustrate an imaging device 400 in a fourth embodiment.
- the imaging device 400 in the fourth embodiment and the imaging device 100 illustrated in the first embodiment differ in that a configuration of a first driving mechanism and a second driving mechanism and a configuration of a first imaging unit and a second imaging unit in the imaging device 400 in the fourth embodiment are different from those of the imaging device 100 illustrated in the first embodiment.
- the imaging device 400 in the fourth embodiment has a configuration in which not only imaging directions of a first imaging unit 410 and a second imaging unit 420 but also an imaging range through zooming can be controlled.
- imaging optical systems in the imaging units have zoom lenses movable in an optical axis direction and imaging ranges of the imaging units are variably controlled when a first driving mechanism 411 (not shown) and a second driving mechanism 412 (not shown) drive the zoom lenses.
- a wide-angle image signal 434 is generated and when the size (the area) of the overlapping range 435 between the imaging range 415 of the first imaging unit 410 and the imaging range 425 of the second imaging unit 420 is less than the first threshold value, the wide-angle image signal 434 is not generated.
- Figs. 11A, 11B , 12A, and 12B are diagrams illustrating the first imaging range 415 of the first imaging unit 410, the second imaging range 425 of the second imaging unit 420, and the overlapping range 435.
- Figs. 11A and 11B illustrate a case in which the overlapping range 435 is equal to or more than a first threshold value 436 and Figs. 12A and 12B illustrate a case in which the overlapping range 435 is less than the first threshold value 436.
- the overlapping range 135 corresponds to a hatched region.
- the overlapping range 435 increases.
- the imaging range (an angle of view) 415 of the first imaging unit 410 and the imaging range (an angle of view) 425 of the second imaging unit 420 increase
- the overlapping range 435 increases.
- a sum of a length of the imaging range of the first imaging unit 410 and a length of the imaging range of the second imaging unit 420 in the rotation direction from the optical axis of the first imaging unit 410 toward the optical axis of the second imaging unit 420 is obtained.
- a determination concerning whether to generate a wide-angle image signal may be performed.
- a wide-angle image signal when the sum of the lengths of the imaging ranges is, for example, equal to or less than a predetermined threshold value, a wide-angle image signal may be generated and when the sum of the lengths of the imaging ranges is larger than the predetermined threshold value, the wide-angle image signal may not be generated. That is to say, assuming that directions of the first imaging unit 410 and the second imaging unit 420 in a pan direction thereof are fixed, if a sum of angles of view of the first imaging unit 410 and the second imaging unit 420 is equal to or more than a predetermined value, an overlapping range increases. Therefore, in this case, a wide-angle image signal is generated. In addition, if the sum of the angles of view of the first imaging unit 410 and the second imaging unit 420 is smaller than the predetermined value, the overlapping range decreases. Thus, the wide-angle image signal is not generated.
- the first driving mechanism and the second driving mechanism may have only a part of the pan driving mechanism, the tilt driving mechanism, and the rotation driving mechanism. That is to say, a zoom unit configured to change an imaging range of at least one of the first imaging unit and the second imaging unit may be provided.
- a user interface configure to control imaging directions or imaging ranges of a first imaging unit and a second imaging unit in an imaging device will be described.
- an imaging direction of an imaging device is controlled in a pan direction thereof will be described below as an example as in the imaging device illustrated in the first embodiment, the same user interface can also be used in the case of a direction other than the pan direction.
- Figs. 13A and 13B are diagrams for explaining a user interface 580 of an imaging device 500 illustrated in the fifth embodiment.
- the user interface 580 includes cursors 581 and 582 configured to control imaging directions of a first imaging unit 510 and a second imaging unit 520 in the imaging device 500 in Fig. 14 and a display unit 590 configured to display an image signal acquired using the imaging units.
- the user interface 580 may be provided separately on a client device 180 side or may be provided integrally in the imaging device 100.
- the imaging direction may be controlled not only using a cursor but also using dragging a range desired to image with a mouse or a touch panel or by identifying a character or a number using a character input interface such as a keyboard.
- a wide-angle image signal 534 is displayed on the display unit 590.
- the first image signal 514 and the second image signal 524 are independently displayed side by side (hereinafter referred to as a "non-wide-angle image display").
- the first image signal 514 may be displayed on the right and the second image signal 524 may be displayed on the left.
- the deviation may be arranged in an upward/downward direction instead of a leftward/rightward direction.
- the first image signal 514 and the second image signal 524 be displayed side by side in the same order in accordance with the imaging directions of the first imaging unit 510 and the second imaging unit 520. That is to say, when the first imaging unit 510 faces the left and the second imaging unit 520 faces the right, on the display unit 590, an arrangement is performed in the same order so that the first image signal 514 is arranged on the left side and the second image signal 524 is arranged on the right side.
- first imaging unit 510 and the second imaging unit 520 are not combined in the non-wide-angle image display mode, the first imaging unit 510 and the second imaging unit 520 may be arranged side by side and displayed to be adjacent to each other without a gap. Alternatively, the first imaging unit 510 and the second imaging unit 520 may be arranged with a gap.
- the user changes the imaging direction of the first imaging unit 510 or the second imaging unit 520 and a size of the overlapping range 535 is less than the first threshold value. At this time, it is desirable that the imaging device 500 return one of the following responses.
- a first response is a response for changing the direction of the imaging unit as instructed by the user. By performing this response, it is possible to change the direction of the imaging unit as intended by the user.
- Figs. 14A and 14B illustrate a user interface and Figs 14C and 14D illustrate imaging directions and imaging ranges 515 and 525 of the imaging units 510 and 520 at that time.
- Figs. 15A, 15B, 15C, and 15D before the direction of the imaging unit is changed, there is a response that asking concerning whether a transition from the wide-angle image signal 534 to the non-wide-angle image signal is to be performed by displaying a dialog box or the like.
- Figs. 15A and 15B illustrate a user interface
- Figs. 15C and 15D illustrate imaging directions and imaging ranges 515 and 525 of the imaging units 510 and 520 at that time.
- Figs. 16A, 16B, 16C, and 16D there is a response that it is possible to change the direction of the imaging unit only within a range in which the wide-angle image signal can be displayed. That is to say, as illustrated in Figs. 16A, 16B, 16C, and 16D , it is possible to change the direction of the imaging unit until the size of the overlapping range 535 is the first threshold value.
- Figs. 16A and 16B illustrate a user interface and Figs. 16C and 16D illustrate imaging directions and imaging ranges 515 and 525 of the imaging units 510 and 520 at that time.
- an image displayed on the display unit 590 is maintained as a wide-angle image.
- the use of the third response can also eliminate the discomfort of changing from a wide-angle image to a non-wide-angle image.
- this is a response that the direction of the imaging unit is changed until the size of the overlapping range 535 reaches the first threshold value and then a dialog box or the like is displayed to request the user's instruction concerning whether to further change the direction of the imaging unit.
- a dialog box or the like is displayed to request the user's instruction concerning whether to further change the direction of the imaging unit.
- a fourth response there is a response that the user selects whether to display a wide-angle image having a low accuracy or a non-wide-angle image. That is to say, when the user selects the displaying of the wide-angle image signal having a low accuracy, the wide-angle image signal is continuously displayed even after the direction of the imaging unit has been changed and when the user selects the transitioning to the non-wide-angle image signal, the direction of the imaging unit is changed and at the same time, the non-wide-angle image signal is displayed.
- Figs. 17A and 17B illustrate a user interface
- Figs. 17C and 17D illustrate imaging directions and imaging ranges 515 and 525 of the imaging units 510 and 520 at that time.
- the size of the overlapping range 535 is less than the first threshold value, it is difficult to obtain an amount of positional shifting having a high accuracy.
- the user who can tolerate an amount of positional shifting having a low accuracy may want to display the wide-angle image signal even if the accuracy is low in some cases. In such a case, it is desirable to use the fourth response.
- the user select whether to change the direction of the imaging unit until a wide-angle image signal having a high accuracy is obtained by combining the third response and the fourth response or to change the direction of the imaging unit while a wide-angle image signal having a low accuracy is also allowed using a dialog box or the like.
- an allowable size of the overlapping range changes in accordance with the user's selection.
- the imaging device 500 return one of the following responses.
- a first response is a response that the direction of the imaging unit is changed as instructed by the user, as illustrated in Figs. 18A, 18B, 18C, and 18D .
- Figs. 18A and 18B illustrate a user interface
- Figs. 18C and 18D illustrate imaging directions and imaging ranges 515 and 525 of the imaging units 510 and 520 at that time.
- the type of image displayed on the display unit changes before and after the direction of the imaging unit is changed. That is to say, before the direction of the imaging unit is changed, the non-wide-angle image signal is display. On the other hand, after the direction of the imaging unit is changed, the wide-angle image signal 534 is displayed. For this reason, the user is likely to feel uncomfortable.
- Figs. 19A, 19B, 19C, and 19D before the direction of the imaging unit is changed, there is a response that asking concerning whether to transition from a non-wide-angle image display mode to a wide-angle image display mode using a dialog box or the like.
- Figs. 19A and 19B illustrate a user interface
- Figs. 19C and 19D illustrate imaging directions and imaging ranges 515 and 525 of the imaging units 510 and 520 at that time.
- the direction of the imaging unit is changed and the image displayed on the display unit 590 is changed from the non-wide-angle image display mode to the wide-angle image display mode.
- the non-wide-angle image may be kept displayed while the direction of the imaging unit is changed.
- the user performs driving so that the size of the overlapping range 535 is smaller than the first threshold value and equal to or more than the second threshold value.
- the second threshold value is smaller than the first threshold value, but is preferably, for example, half or more of the first threshold value.
- the imaging device 500 return one of the following responses.
- a first response is a response that the direction of the imaging unit is changed as instructed by the user. By performing this response, it is possible to change the direction of the imaging unit as intended by the user. In this case, since the overlapping range is kept smaller than the first threshold value, there is no problem that the type of image displayed on the display unit changes before and after the direction of the imaging unit is changed.
- the second response is a response that displaying indicating the fact that "if the direction of the imaging unit is made a little closer, a wide-angle image can be displayed. Do you bring the imaging unit closer?" is provided for the user while the direction of the imaging unit is changed as instructed by the user.
- Figs. 20A and 20B illustrate a user interface and Figs. 20C and 20D illustrate imaging directions and imaging ranges 515 and 525 of the imaging units 510 and 520 at that time.
- the wide-angle image is kept displayed while the direction of the imaging unit is further changed until the overlapping range is equal to or more than the first threshold value.
- the non-wide-angle image is displayed while the direction of the imaging unit is kept as it is.
- the first response or the second response is changed depending on whether the user has changed the direction of the imaging unit to increase the overlapping range or has changed the direction to decrease the overlapping range.
- the user changes the direction of the imaging unit so that the overlapping range decreases, it is expected that the user wants to image different imaging ranges with the first imaging unit and the second imaging unit.
- the first response is provided.
- Fig. 21 is a flowchart for describing an operation example of the above practical example below.
- Step S1 a determination concerning whether the overlapping range between the imaging ranges of the first imaging unit and the second imaging unit is smaller than a first threshold value V1 is performed.
- Step S2 a wide-angle image is generated and displayed.
- Step S3 a determination concerning whether an operation to change a direction of at least one of the first imaging unit and the second imaging unit is performed.
- Step S3 a determination concerning whether an operation to change a direction of at least one of the first imaging unit and the second imaging unit is performed.
- Step S4 a determination concerning whether the overlapping range is smaller than the first threshold value V1 is performed.
- the process proceeds to the process of Step S5 and the imaging direction is changed without changing the wide-angle image.
- Step S4 the process proceeds to the process of Step S6 and the direction of at least one of the first imaging unit and the second imaging unit is changed so that the overlapping range is kept at the threshold value V1 as it is in the advanced wide-angle image. Furthermore, in Step S7, a dialog box "Do you release the direction even if the image quality is changed?" is displayed, and as a result, it is determined whether the release instruction has been received. Moreover, when the determination result is No in Step S7, the process returns to the process of Step S7.
- Step S8 When the determination result is Yes, the process proceeds to the process of Step S8 and it is determined that the overlapping range is smaller than a third threshold value V3.
- V3 is a threshold value that makes it difficult to generate a wide-angle image, for example, is close to zero.
- Step S9 the process proceeds to the process of Step S9 and the direction of at least one of the first imaging unit and the second imaging unit is changed while the wide-angle image is displayed.
- Step S 10 the non-wide-angle image is displayed, and the direction of at least one of the first imaging unit and the second imaging unit is changed.
- Step S1 determines whether the determination result in Step S1 is Yes.
- Step S12 it is determined whether an operation to change the direction of the at least one of the first imaging unit and the second imaging unit is performed.
- Step S13 a determination concerning whether the overlapping range is smaller than the first threshold value V1 is performed.
- the process proceeds to the process of Step S 14 and asking concerning whether to transition to the wide-angle image is performed using a dialog box.
- Step S 14 When the determination result in Step S 14 is No, the process returns to the process of Step S11.
- Step S14 When the determination result in Step S14 is Yes, the process proceeds to the process of Step S15, the direction of at least one of the first imaging unit and the second imaging unit is changed, and a transition to the wide-angle image display is performed.
- Step S13 the process proceeds to the process of Step S16 and the direction of at least one of the first imaging unit and the second imaging unit is changed without changing the non-wide-angle image.
- Step S17 a determination concerning whether the overlapping range is smaller than a second threshold value V2 is performed.
- the process proceeds to the process of Step S16.
- the determination result in Step S17 is No
- the process proceeds to the process of Step S18 and a dialog box "Would you like to make the direction of the imaging unit closer to set to a wide-angle image?" is displayed and as a result, when the instruction of No is provided, the process returns to the process of Step S16.
- Step S19 the direction of at least one of the first imaging unit and the second imaging unit is changed so that the imaging unit is made close and the display is switched to the wide-angle image display.
- the wide-angle image display mode and the non-wide-angle image display mode are appropriately switched in accordance with the overlapping range (amount) of the image obtained from the first imaging unit and the image obtained from the second imaging unit, a user-friendly display is obtained.
- this practical example can also be applied to a case in which three or more of imaging units are provided.
- the images of a plurality of imaging units are arranged side by side without being combined so that it is possible to display an image which is comfortable for the user.
- the control unit 130 detects angular positions or directions of the first imaging unit 110 and the second imaging unit 120 in a peripheral surface of the imaging device 100, so that it is determined whether the wide-angle image signal is displayed or the independent images are displayed side by side, based on a relation between the angular positions or directions.
- the first imaging unit 110 and the second imaging unit 120 have the same imaging range or angle.
- control unit 130 detects the angular positions of the first imaging unit and the second imaging unit based on the numbers of steps from a predetermined initial position of a stepping motor for changing directions of the first imaging unit and the second imaging unit.
- the angular positions can be detected, any other detecting method may be used.
- control unit 130 calculates the difference between the angular positions of the first imaging unit and the second imaging unit, and if the difference of the angular positions is less than a predetermined value, then the wide-angle image signal is displayed. If the difference of the angular positions is not less than the predetermined value, then independent images are displayed side by side.
- an encoder may be arranged to detect the angular positions or directions of respective imaging units.
- control in the present embodiments may be supplied to an imaging device over a network or various storage media using a computer program configured to realize the functions of the above-described embodiments.
- a computer or a central process unit (CPU), a microprocessor unit (MPU), or the like
- CPU central process unit
- MPU microprocessor unit
- the program and a storage medium configured to store the program constitutes the present invention.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Computing Systems (AREA)
- Studio Devices (AREA)
- Stereoscopic And Panoramic Photography (AREA)
- Cameras In General (AREA)
Abstract
Description
- The present invention relates to an imaging device used for applications such as monitoring.
- In recent years, when images captured by a plurality of cameras (hereinafter referred to as "multi-lens cameras") are combined, imaging devices capable of acquiring an image in a wider imaging range (hereinafter referred to as a "wide-angle image") than when a single camera is used have been proposed.
- Japanese Patent Laid-Open No.
2004-118786 - Also, as imaging devices used when a user wants to change a direction desired to be monitored or the like after the imaging devices are installed, imaging devices having a so-called pan tilt zoom (PTZ) function in which an imaging direction can be controlled have been proposed.
- In the imaging device described in Japanese Patent Laid-Open No.
2004-118786 - In such multi-lens cameras, images obtained by each camera can be provided to an external monitor at the same time. However, as a result of controlling the imaging direction of each camera of multi-lens cameras, overlaps of imaging ranges of neighboring cameras in the multi-lens cameras sometimes occur and sometimes not, and it is difficult for users whether overlaps are occurring or not.
- Thus, an object of the present invention is to propose an imaging device including a plurality of imaging units and capable of appropriately controlling whether to generate a wide-angle image in accordance with control of the imaging ranges of the plurality of imaging units.
- Aspects of the present invention are set out by the independent claims. According to a first aspect, there is provided an imaging device according to
claim 1. According to a second aspect, there is provided a control apparatus according toclaim 11. According to a third aspect, there is provided an imaging method according to claim 14. According to a fourth aspect, there is provided a program according toclaim 15. According to a fifth aspect, there is provided a computer-readable storage medium according to claim 16. - Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings.
-
-
Fig. 1 is an arrangement view of an imaging device in a first embodiment when viewed from above. -
Fig. 2 is a functional block diagram of the imaging device in the first embodiment. -
Fig. 3 is a diagram illustrating a case in which an overlap of the imaging ranges of imaging units in the imaging device in the first embodiment is large. -
Fig. 4 is a diagram illustrating a case in which an overlap of the imaging ranges of the imaging units in the imaging device in the first embodiment is small. -
Fig. 5 is a diagram illustrating another example of the imaging ranges and overlapping ranges of the imaging units in the imaging device in the first embodiment. -
Fig. 6 is a diagram illustrating still another example of the imaging ranges and the overlapping range of the imaging units in the imaging device in the first embodiment. -
Fig. 7 is a diagram illustrating another example of the imaging ranges and the overlapping range of the imaging units in the imaging device in the first embodiment. -
Fig. 8 is a diagram illustrating still another example of the imaging ranges and the overlapping range of the imaging units in the imaging device in the first embodiment. -
Fig. 9 is a diagram illustrating an example of the imaging ranges and overlapping ranges of imaging units in an imaging device in a second embodiment. -
Fig. 10 is a diagram illustrating an example of the imaging ranges and overlapping ranges of imaging units in an imaging device in a third embodiment. -
Fig. 11 is a diagram illustrating an example of the imaging ranges and overlapping ranges of imaging units in an imaging device in a fourth embodiment. -
Fig. 12 is a diagram illustrating another example of the imaging range and the overlapping ranges of imaging units in an imaging device in the fourth embodiment. -
Fig. 13 is a diagram illustrating a user interface of an imaging device in a fifth embodiment. -
Fig. 14 is a diagram illustrating an example of a display screen and the movement of an imaging unit when the imaging unit transitions from a wide-angle image display mode. -
Fig. 15 is a diagram illustrating another example of the display screen and the movement of the imaging unit when the imaging unit transitions from the wide-angle image display mode. -
Fig. 16 is a diagram illustrating an example of still another example of the display screen and the movement of the imaging unit when the imaging unit transitions from the wide-angle image display mode. -
Fig. 17 is a diagram illustrating an example of yet another example of the display screen and the movement of the imaging unit when the imaging unit transitions from the wide-angle image display mode. -
Fig. 18 is a diagram illustrating an example of the display screen and the movement of the imaging unit when the imaging unit transitions from a non-wide-angle image display mode. -
Fig. 19 is a diagram illustrating another example of the display screen and the movement of the imaging unit when the imaging unit transitions from the non-wide-angle image display mode. -
Fig. 20 is a diagram illustrating still another example of the display screen and the movement of the imaging unit when the imaging unit transitions from the non-wide-angle image display mode. -
Fig. 21 is a flowchart for describing an operation example of a practical example. - A practical example of an imaging device according to an embodiment of the present invention will be described below with reference to the drawings. At this time, constituent elements having the same functions in the drawings will be denoted by the same reference numerals and repeated description thereof will be omitted.
- In the practical example, an example in which the present invention is applied to a network camera as an imaging device will be described. However, the imaging device includes electronic devices having a plurality of imaging units such as digital still cameras, digital movie cameras, smartphones with a camera, and tablet computers with a camera.
-
Fig. 1 illustrates an imaging device in this practical example and a monitoring system using the same.Fig. 1 is an arrangement view of animaging device 100 when viewed from above (a +Z axis side) andFig. 2 is an internal functional block diagram. Theimaging device 100 includes afirst imaging unit 110, asecond imaging unit 120, afirst driving mechanism 111, asecond driving mechanism 121, acontrol unit 130, a combination (synthesizing)processing unit 140, and a first transmission/reception unit 150. - Each of the
first driving mechanism 111 and thesecond driving mechanism 121 functions as a driving unit and is configured to be able to control an imaging direction or a the imaging range of thefirst imaging unit 110 and thesecond imaging unit 120 at least in the same plane (in an XY plane inFig. 1 ). The imaging device in this practical example is configured to be able to control the imaging direction in a pan direction. - To be specific, the
first driving mechanism 111 and thesecond driving mechanism 121 illustrated inFig. 2 include a motor and a gear and are configured to be able to rotate thefirst imaging unit 110 and thesecond imaging unit 120 using anaxis 101 inFig. 1 as a rotation axis by controlling electric power used for driving the motor. The electric power used for driving the motor is controlled by thecontrol unit 130. - That is to say, the
imaging device 100 is configured to be able to change an imaging direction of each of thefirst imaging unit 110 and thesecond imaging unit 120 in the XY plane. Although the imaging directions of both of thefirst imaging unit 110 and thesecond imaging unit 120 can be changed in this practical example, a driving mechanism capable of controlling at least one of the imaging directions is provided and thus the imaging directions of the first imaging unit and the second imaging unit may only be relatively changeable. - The
first imaging unit 110 includes an imagingoptical system 112 and a solid-state imaging device 113, thesecond imaging unit 120 includes an imagingoptical system 122 and a solid-state imaging device 123, and thefirst imaging unit 110 and thesecond imaging unit 120 form subject images on the solid-state imaging devices optical systems state imaging devices control unit 130. Thecontrol unit 130 has a central processing unit (CPU) as a computer built thereinto and functions as a control unit configured to execute various operations of the entire device on the basis of a computer program stored in a memory (not shown). - The
combination processing unit 140 functions as a combining unit and is configured to combine afirst image signal 114 acquired by thefirst imaging unit 110 and asecond image signal 124 acquired by thesecond imaging unit 120 to generate a wide-angle image (panoramic image)signal 134. To be specific, by applying a so-called pattern matching technique for obtaining a correlation coefficient while shifting an overlapping portion of images, an amount of positional shifting between a plurality of images is obtained and the wide-angle image signal 134 is generated. - Also, in this practical example, an overlapping range (an amount of overlapping) between an
imaging range 115 of thefirst imaging unit 110 and animaging range 125 of thesecond imaging unit 120 is investigated. Furthermore, the wide-angle image signal 134 is not generated when the overlapping range (the amount of overlapping) is less than a first threshold value and the wide-angle image signal 134 is generated when the overlapping range (the amount of overlapping) is equal to or more than the first threshold value. Details will be described later. When the wide-angle image signal 134 is not generated in thecombination processing unit 140, thefirst image signal 114 and thesecond image signal 124 are transferred to the first transmission/reception unit 150 without being combined. - The first transmission/
reception unit 150 transmits an image signal (thefirst image signal 114, thesecond image signal 124, or the wide-angle image signal 134) which has been transferred from thecombination processing unit 140 to an external client device 180 (not shown) over a wired or wireless network or the like. - The
external client device 180 transmits a command for controlling theimaging device 100 to the first transmission/reception unit 150 over a second transmission/reception unit 181 and a network and theimaging device 100 receives the command and returns a response to the command to theclient device 180. - The command is, for example, for controlling the
first driving mechanism 111 and thesecond driving mechanism 121. That is to say, a user can control directions of thefirst imaging unit 110 and thesecond imaging unit 120 from the external client device 180 (control apparatus) over a network. - The client device is, for example, an external device such as a personal computer (PC) and the network is configured of wired local area networks (LANs), wireless LANs, or the like. Furthermore, a configuration in which electric power is supplied to the
imaging device 100 over the network may be provided. - A control unit configured to control the inside of the client device 180 (control apparatus) and having a computer such as a CPU built thereinto will be denoted with
reference numeral 182. Furthermore, thecontrol unit 182 has a memory (not shown) built thereinto and a computer program configured to control an operation of the CPU in the control unit is stored in the memory. A display unit serving as a display unit and configured to display an image signal or the like which has been transferred from theimaging device 100 will be denoted withreference numeral 183. An operation unit including input units such as various switches and touch panels will be denoted withreference numeral 184. The user can give various instructions to theimaging device 100 by operating the operation unit. - Although the
combination processing unit 140 is provided inside theimaging device 100 in the practical example, thecombination processing unit 140 may be provided in theclient device 180. Furthermore, an imaging system is constituted of theimaging device 100 and theclient device 180. - In a first embodiment illustrated in
Fig. 2 , an example in which theimaging device 100 includes the first transmission/reception unit 150, an image is transmitted to theclient device 180 side, and an operation is performed through a command from theclient device 180 side is illustrated. - That is to say, an example in which a
display unit 183, thecontrol unit 182, and anoperation unit 184 are provided separately from the imaging device is illustrated. However, theimaging device 100 may integrally include a memory configured to store image data and a part of theoperation unit 184 or the like such as thedisplay unit 183 configured to display an image and a switch configured to receive the user's assistance. That is to say, theimaging device 100 itself may have a function of theclient device 180 integrally built thereinto. - As described above, in the
imaging device 100 in this practical example, when a size of the overlapping range (the amount of overlapping) 135 between theimaging range 115 of thefirst imaging unit 110 and theimaging range 125 of thesecond imaging unit 120 is equal to or more than the first threshold value, the wide-angle image signal 134 is generated. Furthermore, a configuration in which, when the size of the overlappingrange 135 is less than the first threshold value, the wide-angle image signal 134 is not generated is provided. - With such a configuration, it is possible to provide an image having excellent visibility of a subject regardless of the imaging directions of the
first imaging unit 110 and thesecond imaging unit 120. Details will be provided in the following description. -
Figs. 3 and4 are diagram illustrating afirst imaging range 115 of thefirst imaging unit 110, asecond imaging range 125 of thesecond imaging unit 120, and an overlappingrange 135 thereof. -
Figs. 3A and 3B illustrate a case in which the overlapping range (the amount of overlapping) 135 is equal to or more than afirst threshold value 136 andFigs. 4A and 4B illustrate a case in which the overlappingrange 135 is less than thefirst threshold value 136. InFig. 3 , the overlappingrange 135 corresponds to a hatched region. - In order to generate the wide-
angle image signal 134, an amount of positional shifting between thefirst image signal 114 and thesecond image signal 124 can be obtained. In order to obtain the amount of positional shifting, a general method as described inPatent Document 1 can be used. For example, a correlation coefficient such as a sum of squared difference (SSD) and a sum of absolute difference (SAD) may be calculated while shifting thefirst image signal 114 and thesecond image signal 124 so that an amount of positional shifting having the highest correlation coefficient may be obtained. - Generally, if a correlation coefficient is obtained using the SSD or the SAD, when the number of pixel signals used for calculating an amount of positional shifting increases, a correlation coefficient having a high accuracy can be obtained. As can be seen from
Fig. 3 , when the overlappingrange 135 increases, the number of pixel signals used for calculating an amount of positional shifting increases. Therefore, when the overlappingrange 135 increases, an amount of positional shifting having a high accuracy can be obtained and a wide-angle image having a high quality can be generated. On the other hand, when the overlappingrange 135 decreases, the detection of the amount of positional shifting decreases. Thus, it is difficult to generate a wide-angle image having a high quality. - When the wide-
angle image signal 134 having a high quality can be generated, the visibility of the subject in the overlappingrange 135 when the wide-angle image signal 134 is obtained by combining thefirst image signal 114 and thesecond image signal 124 is higher than that when thefirst image signal 114 and thesecond image signal 124 are observed independently. On the other hand, when it is difficult to generate the wide-angle image signal 134 having a high quality, even if the wide-angle image signal 134 is generated, the wide-angle image signal 134 is likely to be generated with an incorrect amount of positional shifting. That is to say, when the wide-angle image is generated, information included in thefirst image signal 114 and thesecond image signal 124 is likely to be lost. - Therefore, when it is difficult to generate the wide-
angle image signal 134 having a high quality, the visibility of the subject when thefirst image signal 114 and thesecond image signal 124 are displayed separately is superior than that when the wide-angle image signal 134 is displayed. - Thus, in the
imaging device 100 in this practical example, the overlappingrange 135 is equal to or more than thefirst threshold value 136. When it is possible to generate a wide-angle image signal having a high quality, the wide-angle image signal 134 is generated in thecombination processing unit 140. - On the other hand, when the overlapping
range 135 is less than thefirst threshold value 136 and it is difficult to generate a wide-angle image signal having a high quality, the wide-angle image signal is not generated in thecombination processing unit 140. Furthermore, thefirst image signal 114 and thesecond image signal 124 are transmitted as they are and displayed separately. With such a configuration, it is possible to provide an image having excellent visibility of the subject regardless of the imaging directions of thefirst imaging unit 110 and thesecond imaging unit 120. - As can be seen from
Fig. 3 , when the deviation between the imaging direction of thefirst imaging unit 110 and the imaging direction of thesecond imaging unit 120 decreases, the overlappingrange 135 increases. Therefore, in theimaging device 100 in the first embodiment, a configuration in which, when the deviation between the imaging direction of thefirst imaging unit 110 and the imaging direction of thesecond imaging unit 120 is equal to or less than a predetermined threshold value, a wide-angle image signal is generated, and when the deviation between the imaging direction of thefirst imaging unit 110 and the imaging direction of thesecond imaging unit 120 is larger than the predetermined threshold value, the wide-angle image signal is not generated can also be provided. Here, the imaging direction of the imaging unit means a direction of an optical axis of the imaging optical system of the imaging unit. - Also, the
first threshold value 136 can also be determined using the number of pixels in the overlappingrange 135. To be specific, between thefirst image signal 114 and thesecond image signal 124, the number of pixels of thefirst image signal 114 and thesecond image signal 124 included in the overlappingrange 135 is preferably 100 or more and more preferably 1000 or more. - Also, the
first threshold value 136 may be defined as a relative ratio to thefirst imaging range 115 or thesecond imaging range 125. To be specific, the overlapping range is preferably 5% or more and more preferably 20% or more with respect to a relatively narrower imaging range of thefirst imaging range 115 and thesecond imaging range 125. - Although
Fig. 1 illustrates a case in which theimaging device 100 is constituted of two imaging units, i.e., thefirst imaging unit 110 and thesecond imaging unit 120, theimaging device 100 may be constituted of three or more imaging units. When theimaging device 100 is constituted of three or more imaging units, it is desirable that a configuration in which whether to generate a wide-angle image signal through combination is determined is provided in accordance with the overlapping range between the imaging ranges of the imaging units. A determination unit is configured to determine whether or not to generate the wide-angle image by thecombination processing unit 140. The determination is based on a relation between an imaging range of thefirst imaging unit 110 and an imaging range of thesecond imaging unit 120, or a position relation between thefirst imaging unit 110 and thesecond imaging unit 120 in the predetermined direction. - To be specific, when the overlapping range between the imaging ranges of neighboring imaging units is equal to or more the first threshold value, a wide-angle image signal is generated by combining the image signals acquired using the imaging units. On the other hand, when the overlapping range between the imaging ranges of neighboring imaging units is less than the first threshold value, a configuration in which the image signals acquired using the imaging units are not combined is provided.
-
Fig. 5 illustrates an example in which theimaging device 100 is constituted of four imaging units, i.e., theimaging units - As illustrated in
Fig. 5 , when the overlapping ranges between the imaging ranges of imaging units are all equal to or more than the first threshold value, one wide-angle image signal 134 is generated by combining all of the image signals 114, 124 and image signals 164 and 174 obtained through imaging using the imaging units. - On the other hand,
Fig. 6 illustrates a case in which the overlapping ranges between theimaging units imaging units first threshold value 136 and the overlapping ranges between theimaging units imaging units first threshold value 136. At this time, the wide-angle image signal 134 is generated by combining the image signals 114, 124, and 164 from theimaging units imaging unit 170 is not combined with another image signal. -
Fig. 7 illustrates a case in which the overlapping ranges between theimaging units imaging units first threshold value 136 and the overlapping ranges between theimaging units imaging units first threshold value 136. - At this time, two wide-angle image signals 134A and 134B are generated by combining the image signals 114 and the
image signal 124 from theimaging units imaging units - Also, as illustrated in
Fig. 8 , when the overlapping ranges between the imaging ranges of the imaging units are all less than thefirst threshold value 136, none of the image signals 114, 124, 164, and 174 imaged using the imaging units is combined with other image signals. - An imaging device 200 (not shown) in a second embodiment and the
imaging device 100 illustrated in the first embodiment differ in view of only the functions of the first driving mechanism and the second driving mechanism. In the imaging device 200 in the second embodiment, a first driving mechanism 211 (not shown) and a second driving mechanism 212 (not shown) are configured in such a manner that the imaging directions of the first imaging unit 210 (not shown) and the second imaging unit 220 (not shown) are set so that the first imaging unit 210 (not shown) and the second imaging unit 220 (not shown) can rotate in two directions orthogonal to each other. To be specific, in addition to a rotation mechanism centering on a Z axis (a so-called pan driving mechanism), a rotation mechanism capable of controlling an angle with respect to the Z axis (a so-called tilt driving mechanism) is provided. - Also, as illustrated in
Fig. 9A , when a size (an area) of an overlappingrange 235 between animaging range 215 of the first imaging unit 210 and animaging range 225 of the second imaging unit 220 is equal to or more than a first threshold value (an area value), a wide-angle image signal 234 is generated. Furthermore, as illustrated inFig. 9B , when the size (the area) of the overlappingrange 235 between theimaging range 215 of the first imaging unit 210 and theimaging range 225 of the second imaging unit 220 is less than the first threshold value, the wide-angle image signal 234 is not generated. - That is to say,
Figs. 9A and 9B are diagrams illustrating thefirst imaging range 215 of the first imaging unit 210, thesecond imaging range 225 of the second imaging unit 220, and the overlappingrange 235.Fig. 9A illustrates a case in which the overlappingrange 235 is large andFig. 9B illustrates a case in which the overlappingrange 235 is small. InFigs. 9A and 9B , the overlappingrange 235 corresponds to a hatched region. - As can be seen from
Figs. 9A and 9B , when the deviation between an imaging direction of thefirst imaging unit 110 and an imaging direction of thesecond imaging unit 120 in the XY plane (refer toFig. 1 ) decreases, a size of the overlappingrange 235 in a pan direction thereof (a vertical direction inFigs. 9A and 9B ) increases. Furthermore, when the deviation between the imaging direction of thefirst imaging unit 110 and the imaging direction of thesecond imaging unit 120 in a Z direction (refer toFig. 1 ) decreases, a size of the overlappingrange 235 in a tilt direction thereof (a horizontal direction inFigs. 9A and 9B ) increases. - Therefore, in the imaging device 200 in the second embodiment, a determination concerning whether a wide-angle image signal is generated may be performed using the product of the deviation between the imaging direction of the first imaging unit 210 and the imaging direction of the second imaging unit 220 in the pan direction thereof and the deviation thereof in the tilt direction thereof.
- To be specific, when the product of the deviations in the pan direction and the tilt direction is equal to or less than the first threshold value, a wide-angle image signal is generated and when the product of the deviations in the pan direction and the tilt direction is larger than the first threshold value, the wide-angle image signal is not generated.
- Also, in this case, imaging signals of the first imaging unit 210 and the second imaging unit 220 are output as they are without being combined and displayed separately.
- This practical example may be applied to an imaging device including only a tilt driving mechanism instead of an imaging device including both a pan driving mechanism and a tilt driving mechanism.
-
Fig. 10 is a diagram for explaining a third embodiment. An imaging device 300 (not shown) in the third embodiment and the imaging device 200 illustrated in the second embodiment differ in view of only the functions of a first driving mechanism and the second driving mechanism. In the imaging device 300, in addition to being able to control imaging directions of a first imaging unit 310 (not shown) and a second imaging unit 320 (not shown), a rotation mechanism capable of rotating imaging units about optical axes of the imaging units (a so-called rotation mechanism) is provided. - Also, when a size (an area) of an overlapping
range 335 between animaging range 315 of the first imaging unit 310 and animaging range 325 of the second imaging unit 320 is equal to or more than a first threshold value, a wide-angle image signal 334 is generated and when the size (the area) of the overlappingrange 335 between theimaging range 315 of the first imaging unit 310 and theimaging range 325 of the second imaging unit 320 is less than the first threshold value, the wide-angle image signal 334 is not generated. -
Figs. 10A and 10B illustrate thefirst imaging range 315 of the first imaging unit 310, thesecond imaging range 325 of the second imaging unit 320, and the overlappingrange 335.Fig. 10A illustrates a case in which the overlappingrange 335 is large andFig. 10B illustrates a case in which the imaging direction of each of the imaging units rotates 90 degrees to the right while the imaging direction of each of the imaging units is maintained from the state ofFig. 10A . InFigs. 10A and 10B , the overlappingrange 135 corresponds to a hatched region. - Generally, in a solid-state imaging device in an imaging unit used for monitoring applications and the like, a length thereof in a horizontal direction thereof may be longer than a length thereof in vertical direction thereof in many cases. Therefore, when the imaging unit rotates around an optical axis, an imaging range of the imaging unit changes. For this reason, as illustrated in
Figs. 10A and 10B , a size of the overlappingrange 335 changes in accordance with a rotation angle of the imaging unit. - Although three functions such as a pan driving mechanism, a tilt driving mechanism, and a rotation driving mechanism are all provided together with a first driving mechanism and a second driving mechanism in this practical example, each of the first driving mechanism and the second driving mechanism may have a part of the above three functions. That is to say, for example, with regard to a rotation function, a rotation unit configured to rotate a corresponding one imaging unit of the first imaging unit and the second imaging unit about an optical axis of at least one of the first imaging unit and the second imaging unit may be provided.
-
Figs. 11A and 11B andFigs. 12A and 12B illustrate animaging device 400 in a fourth embodiment. Theimaging device 400 in the fourth embodiment and theimaging device 100 illustrated in the first embodiment differ in that a configuration of a first driving mechanism and a second driving mechanism and a configuration of a first imaging unit and a second imaging unit in theimaging device 400 in the fourth embodiment are different from those of theimaging device 100 illustrated in the first embodiment. Theimaging device 400 in the fourth embodiment has a configuration in which not only imaging directions of afirst imaging unit 410 and asecond imaging unit 420 but also an imaging range through zooming can be controlled. To be specific, imaging optical systems in the imaging units have zoom lenses movable in an optical axis direction and imaging ranges of the imaging units are variably controlled when a first driving mechanism 411 (not shown) and a second driving mechanism 412 (not shown) drive the zoom lenses. - Also, when a size (an area) of an overlapping
range 435 between animaging range 415 of thefirst imaging unit 410 and animaging range 425 of thesecond imaging unit 420 is equal to or more than a first threshold value, a wide-angle image signal 434 is generated and when the size (the area) of the overlappingrange 435 between theimaging range 415 of thefirst imaging unit 410 and theimaging range 425 of thesecond imaging unit 420 is less than the first threshold value, the wide-angle image signal 434 is not generated. -
Figs. 11A, 11B ,12A, and 12B are diagrams illustrating thefirst imaging range 415 of thefirst imaging unit 410, thesecond imaging range 425 of thesecond imaging unit 420, and the overlappingrange 435. -
Figs. 11A and 11B illustrate a case in which the overlappingrange 435 is equal to or more than afirst threshold value 436 andFigs. 12A and 12B illustrate a case in which the overlappingrange 435 is less than thefirst threshold value 436. InFigs. 11A, 11B ,12A, and 12B , the overlappingrange 135 corresponds to a hatched region. - As can be seen from
Figs. 11A, 11B ,12A, and 12B , when the imaging range (an angle of view) 415 of thefirst imaging unit 410 and the imaging range (an angle of view) 425 of thesecond imaging unit 420 increase, the overlappingrange 435 increases. To be specific, when a length of the imaging range in a rotation direction from an optical axis of thefirst imaging unit 410 toward an optical axis of thesecond imaging unit 420 increases (when the angle of view increases), the overlappingrange 435 increases. - Therefore, in the
imaging device 400 in the fourth embodiment, a sum of a length of the imaging range of thefirst imaging unit 410 and a length of the imaging range of thesecond imaging unit 420 in the rotation direction from the optical axis of thefirst imaging unit 410 toward the optical axis of thesecond imaging unit 420 is obtained. Thus, a determination concerning whether to generate a wide-angle image signal may be performed. - To be specific, when the sum of the lengths of the imaging ranges is, for example, equal to or less than a predetermined threshold value, a wide-angle image signal may be generated and when the sum of the lengths of the imaging ranges is larger than the predetermined threshold value, the wide-angle image signal may not be generated. That is to say, assuming that directions of the
first imaging unit 410 and thesecond imaging unit 420 in a pan direction thereof are fixed, if a sum of angles of view of thefirst imaging unit 410 and thesecond imaging unit 420 is equal to or more than a predetermined value, an overlapping range increases. Therefore, in this case, a wide-angle image signal is generated. In addition, if the sum of the angles of view of thefirst imaging unit 410 and thesecond imaging unit 420 is smaller than the predetermined value, the overlapping range decreases. Thus, the wide-angle image signal is not generated. - In addition to a zoom driving mechanism, as in the first to third embodiments, the first driving mechanism and the second driving mechanism may have only a part of the pan driving mechanism, the tilt driving mechanism, and the rotation driving mechanism. That is to say, a zoom unit configured to change an imaging range of at least one of the first imaging unit and the second imaging unit may be provided.
- In a fifth embodiment, a user interface configure to control imaging directions or imaging ranges of a first imaging unit and a second imaging unit in an imaging device will be described. Although a case in which an imaging direction of an imaging device is controlled in a pan direction thereof will be described below as an example as in the imaging device illustrated in the first embodiment, the same user interface can also be used in the case of a direction other than the pan direction.
-
Figs. 13A and 13B are diagrams for explaining auser interface 580 of animaging device 500 illustrated in the fifth embodiment. Theuser interface 580 includescursors first imaging unit 510 and asecond imaging unit 520 in theimaging device 500 inFig. 14 and adisplay unit 590 configured to display an image signal acquired using the imaging units. Theuser interface 580 may be provided separately on aclient device 180 side or may be provided integrally in theimaging device 100. - The imaging direction may be controlled not only using a cursor but also using dragging a range desired to image with a mouse or a touch panel or by identifying a character or a number using a character input interface such as a keyboard.
- When a size of an overlapping
range 535 between animaging range 515 of thefirst imaging unit 510 and animaging range 525 of thesecond imaging unit 520 is equal to or more than a first threshold value, as illustrated inFig. 13A , a wide-angle image signal 534 is displayed on thedisplay unit 590. - On the other hand, when the size of the overlapping
range 535 is less than a first threshold value, as illustrated inFig. 13B , thefirst image signal 514 and thesecond image signal 524 are independently displayed side by side (hereinafter referred to as a "non-wide-angle image display"). In the display unit, thefirst image signal 514 may be displayed on the right and thesecond image signal 524 may be displayed on the left. Furthermore, the deviation may be arranged in an upward/downward direction instead of a leftward/rightward direction. - Here, it is desirable that the
first image signal 514 and thesecond image signal 524 be displayed side by side in the same order in accordance with the imaging directions of thefirst imaging unit 510 and thesecond imaging unit 520. That is to say, when thefirst imaging unit 510 faces the left and thesecond imaging unit 520 faces the right, on thedisplay unit 590, an arrangement is performed in the same order so that thefirst image signal 514 is arranged on the left side and thesecond image signal 524 is arranged on the right side. - Although the
first imaging unit 510 and thesecond imaging unit 520 are not combined in the non-wide-angle image display mode, thefirst imaging unit 510 and thesecond imaging unit 520 may be arranged side by side and displayed to be adjacent to each other without a gap. Alternatively, thefirst imaging unit 510 and thesecond imaging unit 520 may be arranged with a gap. - First, in a state in which the wide-
angle image signal 534 is displayed on thedisplay unit 590, the user changes the imaging direction of thefirst imaging unit 510 or thesecond imaging unit 520 and a size of the overlappingrange 535 is less than the first threshold value. At this time, it is desirable that theimaging device 500 return one of the following responses. - As illustrated in
Figs. 14A, 14B, 14C, and 14D , a first response is a response for changing the direction of the imaging unit as instructed by the user. By performing this response, it is possible to change the direction of the imaging unit as intended by the user.Figs. 14A and 14B illustrate a user interface andFigs 14C and 14D illustrate imaging directions and imaging ranges 515 and 525 of theimaging units - However, when this response is provided, a type of image displayed on the display unit changes before and after the direction of the imaging unit changes. That is to say, before the direction of the imaging unit changes, the wide-
angle image signal 534 has been displayed. On the other hand, after the direction of the imaging unit changes, a non-wide-angle image signal is displayed. For this reason, the user is likely to feel strange. - Thus, as a second response, as illustrated in
Figs. 15A, 15B, 15C, and 15D , before the direction of the imaging unit is changed, there is a response that asking concerning whether a transition from the wide-angle image signal 534 to the non-wide-angle image signal is to be performed by displaying a dialog box or the like.Figs. 15A and 15B illustrate a user interface andFigs. 15C and 15D illustrate imaging directions and imaging ranges 515 and 525 of theimaging units - As a result of asking the user, when there is a response from the user that a non-wide-angle image can be transitioned, the direction of the imaging unit is changed and an image displayed on the
display unit 590 is changed from a wide-angle image signal to a non-wide-angle image signal. On the other hand, when there is a response from the user that a transition from a wide-angle image signal to a non-wide-angle image signal cannot be performed, the wide-angle image signal is kept displayed without changing the direction of the imaging unit. When the second response as described above is used, it is possible to eliminate the discomfort of changing from the wide-angle image signal to the non-wide-angle image signal. - Also, as a third response, there is a response that it is possible to change the direction of the imaging unit only within a range in which the wide-angle image signal can be displayed. That is to say, as illustrated in
Figs. 16A, 16B, 16C, and 16D , it is possible to change the direction of the imaging unit until the size of the overlappingrange 535 is the first threshold value.Figs. 16A and 16B illustrate a user interface andFigs. 16C and 16D illustrate imaging directions and imaging ranges 515 and 525 of theimaging units - At this time, since the size of the overlapping
range 535 is equal to or more than the first threshold value, an image displayed on thedisplay unit 590 is maintained as a wide-angle image. The use of the third response can also eliminate the discomfort of changing from a wide-angle image to a non-wide-angle image. - When the third response is used, it is more desirable to use the third response together with the second response. That is to say, this is a response that the direction of the imaging unit is changed until the size of the overlapping
range 535 reaches the first threshold value and then a dialog box or the like is displayed to request the user's instruction concerning whether to further change the direction of the imaging unit. For the user who wants to change the imaging direction while keeping the wide-angle image, it is possible to change the imaging direction as much as possible and to change the imaging direction to a desired imaging direction for the user who wants to change the imaging direction even after transitioning to the non-wide-angle image. - As a fourth response, there is a response that the user selects whether to display a wide-angle image having a low accuracy or a non-wide-angle image. That is to say, when the user selects the displaying of the wide-angle image signal having a low accuracy, the wide-angle image signal is continuously displayed even after the direction of the imaging unit has been changed and when the user selects the transitioning to the non-wide-angle image signal, the direction of the imaging unit is changed and at the same time, the non-wide-angle image signal is displayed.
Figs. 17A and 17B illustrate a user interface andFigs. 17C and 17D illustrate imaging directions and imaging ranges 515 and 525 of theimaging units - As described above, when the size of the overlapping
range 535 is less than the first threshold value, it is difficult to obtain an amount of positional shifting having a high accuracy. Thus, it is difficult to obtain a wide-angle image signal having a high quality. However, the user who can tolerate an amount of positional shifting having a low accuracy may want to display the wide-angle image signal even if the accuracy is low in some cases. In such a case, it is desirable to use the fourth response. - When the size of the overlapping
range 535 is almost zero, that is, when the imaging range of the first imaging unit and the imaging range of the second imaging unit do not overlap, it is impossible to obtain an amount of positional shifting. - In this case, it is not possible to display a wide-angle image signal. Thus, it is necessary to perform the transitioning to a non-wide-angle image display even with the fourth response.
- At this time, when there is a response indicating a prohibition instruction from the user that a transition from a wide-angle image display to a non-wide-angle image display need not be performed, it is desirable that the direction of the imaging unit can be changed only until the overlapping range is zero.
- Also, it is more desirable that the user select whether to change the direction of the imaging unit until a wide-angle image signal having a high accuracy is obtained by combining the third response and the fourth response or to change the direction of the imaging unit while a wide-angle image signal having a low accuracy is also allowed using a dialog box or the like. In this case, an allowable size of the overlapping range changes in accordance with the user's selection.
- A case in which the imaging direction of the
first imaging unit 510 or thesecond imaging unit 520 is changed in a state in which the non-wide-angle image is displayed on thedisplay unit 590 will be described below. - It is assumed that the user performs driving so that the size of the overlapping
range 535 is equal to or more than the first threshold value in a state in which the non-wide-angle image is displayed on thedisplay unit 590. - At this time, it is desirable that the
imaging device 500 return one of the following responses. - A first response is a response that the direction of the imaging unit is changed as instructed by the user, as illustrated in
Figs. 18A, 18B, 18C, and 18D . By performing this response, it is possible to change the direction of the imaging unit as intended by the user.Figs. 18A and 18B illustrate a user interface andFigs. 18C and 18D illustrate imaging directions and imaging ranges 515 and 525 of theimaging units - However, when this response is provided, the type of image displayed on the display unit changes before and after the direction of the imaging unit is changed. That is to say, before the direction of the imaging unit is changed, the non-wide-angle image signal is display. On the other hand, after the direction of the imaging unit is changed, the wide-
angle image signal 534 is displayed. For this reason, the user is likely to feel uncomfortable. - Thus, as a second response, as illustrated in
Figs. 19A, 19B, 19C, and 19D , before the direction of the imaging unit is changed, there is a response that asking concerning whether to transition from a non-wide-angle image display mode to a wide-angle image display mode using a dialog box or the like.Figs. 19A and 19B illustrate a user interface andFigs. 19C and 19D illustrate imaging directions and imaging ranges 515 and 525 of theimaging units - As a result of asking the user, when there is a response from the user that the display can transition to the wide-angle image display, the direction of the imaging unit is changed and the image displayed on the
display unit 590 is changed from the non-wide-angle image display mode to the wide-angle image display mode. - On the other hand, when there is a response from the user that the mode need not to transition from the non-wide-angle image display mode to the wide-angle image display mode, the direction of the imaging unit is not changed and the non-wide-angle image is kept displayed.
- When the second response is used, it is possible to eliminate the discomfort when the mode is changed from the non-wide-angle image display mode to the wide-angle image display mode.
- When there is a response (a prohibition instruction) from the user that the mode need not to transition from the non-wide-angle image display mode to the wide-angle image display mode, the non-wide-angle image may be kept displayed while the direction of the imaging unit is changed.
- This is because when the overlapping range is equal to or more than the first threshold value, it is desirable to display a wide-angle image when importance is placed on visibility of the overlapping range, but a non-wide-angle image can also be displayed.
- Subsequently, it is assumed that, when the non-wide-angle image is displayed on the
display unit 590, the user performs driving so that the size of the overlappingrange 535 is smaller than the first threshold value and equal to or more than the second threshold value. The second threshold value is smaller than the first threshold value, but is preferably, for example, half or more of the first threshold value. - At this time, it is desirable that the
imaging device 500 return one of the following responses. - As illustrated in
Figs. 18A, 18B, 18C, and 18D , a first response is a response that the direction of the imaging unit is changed as instructed by the user. By performing this response, it is possible to change the direction of the imaging unit as intended by the user. In this case, since the overlapping range is kept smaller than the first threshold value, there is no problem that the type of image displayed on the display unit changes before and after the direction of the imaging unit is changed. - However, as described above, when importance is placed on the visibility of the overlapping range, it is desirable to display a wide-angle image. Thus, it is desirable to provide options for displaying the wide-angle image signal to the user. That is to say, as illustrated in
Figs. 20A, 20B, 20C, and 20D , the second response is a response that displaying indicating the fact that "if the direction of the imaging unit is made a little closer, a wide-angle image can be displayed. Do you bring the imaging unit closer?" is provided for the user while the direction of the imaging unit is changed as instructed by the user. -
Figs. 20A and 20B illustrate a user interface andFigs. 20C and 20D illustrate imaging directions and imaging ranges 515 and 525 of theimaging units - When the user responds that he or she wants a wide-angle image, the wide-angle image is kept displayed while the direction of the imaging unit is further changed until the overlapping range is equal to or more than the first threshold value. On the other hand, when the user returns a response indicating that the non-wide-angle image can be kept as it is, the non-wide-angle image is displayed while the direction of the imaging unit is kept as it is.
- At this time, the first response or the second response is changed depending on whether the user has changed the direction of the imaging unit to increase the overlapping range or has changed the direction to decrease the overlapping range.
- When the user changes the direction of the imaging unit so that the overlapping range increases, it is expected that the user wants to image a near imaging range with the first imaging unit and the second imaging unit. Thus, options in which a wide-angle image can be displayed by providing the second response are presented.
- On the other hand, when the user changes the direction of the imaging unit so that the overlapping range decreases, it is expected that the user wants to image different imaging ranges with the first imaging unit and the second imaging unit. Thus, the first response is provided.
-
Fig. 21 is a flowchart for describing an operation example of the above practical example below. - In
Fig. 21 , in Step S1, a determination concerning whether the overlapping range between the imaging ranges of the first imaging unit and the second imaging unit is smaller than a first threshold value V1 is performed. In addition, when the determination result is No, in Step S2, a wide-angle image is generated and displayed. - After that, in Step S3, a determination concerning whether an operation to change a direction of at least one of the first imaging unit and the second imaging unit is performed. When the determination result is No, the process returns to the process of Step S3. When the determination result is Yes, in Step S4, a determination concerning whether the overlapping range is smaller than the first threshold value V1 is performed. In addition, when the determination result is No, the process proceeds to the process of Step S5 and the imaging direction is changed without changing the wide-angle image.
- When the determination result in Step S4 is Yes, the process proceeds to the process of Step S6 and the direction of at least one of the first imaging unit and the second imaging unit is changed so that the overlapping range is kept at the threshold value V1 as it is in the advanced wide-angle image. Furthermore, in Step S7, a dialog box "Do you release the direction even if the image quality is changed?" is displayed, and as a result, it is determined whether the release instruction has been received. Moreover, when the determination result is No in Step S7, the process returns to the process of Step S7.
- When the determination result is Yes, the process proceeds to the process of Step S8 and it is determined that the overlapping range is smaller than a third threshold value V3. Here, V3 is a threshold value that makes it difficult to generate a wide-angle image, for example, is close to zero. When the determination result in Step S8 is No, the process proceeds to the process of Step S9 and the direction of at least one of the first imaging unit and the second imaging unit is changed while the wide-angle image is displayed. When the determination result in Step S8 is Yes, the process proceeds to the process of Step S 10, the non-wide-angle image is displayed, and the direction of at least one of the first imaging unit and the second imaging unit is changed.
- On the other hand, when the determination result in Step S1 is Yes, the process proceeds to the process of Step S11 and the non-wide-angle image is generated and displayed. Furthermore, in
Step S 12, it is determined whether an operation to change the direction of the at least one of the first imaging unit and the second imaging unit is performed. When the determination result is No, the process returns to the process of Step S12. When the determination result is Yes, in Step S13, a determination concerning whether the overlapping range is smaller than the first threshold value V1 is performed. In addition, if the determination result is No, the process proceeds to the process of Step S 14 and asking concerning whether to transition to the wide-angle image is performed using a dialog box. - When the determination result in Step S 14 is No, the process returns to the process of Step S11. When the determination result in Step S14 is Yes, the process proceeds to the process of Step S15, the direction of at least one of the first imaging unit and the second imaging unit is changed, and a transition to the wide-angle image display is performed.
- When the determination result in Step S13 is Yes, the process proceeds to the process of Step S16 and the direction of at least one of the first imaging unit and the second imaging unit is changed without changing the non-wide-angle image.
- Also, in Step S17, a determination concerning whether the overlapping range is smaller than a second threshold value V2 is performed. In addition, if the determination result is Yes, the process proceeds to the process of Step S16. When the determination result in Step S17 is No, the process proceeds to the process of Step S18 and a dialog box "Would you like to make the direction of the imaging unit closer to set to a wide-angle image?" is displayed and as a result, when the instruction of No is provided, the process returns to the process of Step S16. When the instruction of Yes is provided in Step S18, in Step S19, the direction of at least one of the first imaging unit and the second imaging unit is changed so that the imaging unit is made close and the display is switched to the wide-angle image display.
- As described above, according to the practical example, since the wide-angle image display mode and the non-wide-angle image display mode are appropriately switched in accordance with the overlapping range (amount) of the image obtained from the first imaging unit and the image obtained from the second imaging unit, a user-friendly display is obtained. Of course, this practical example can also be applied to a case in which three or more of imaging units are provided.
- In this case, in the non-wide-angle image display mode, the images of a plurality of imaging units are arranged side by side without being combined so that it is possible to display an image which is comfortable for the user.
- Although according to one of the previous embodiments, depending on a size of an overlapping
range 125 between animaging range 115 of thefirst imaging unit 110 and animaging range 125 of thesecond imaging unit 120, it is determined whether the wide-angle image signal is displayed or the independent images are displayed side by side. - In the sixth embodiment, the
control unit 130 detects angular positions or directions of thefirst imaging unit 110 and thesecond imaging unit 120 in a peripheral surface of theimaging device 100, so that it is determined whether the wide-angle image signal is displayed or the independent images are displayed side by side, based on a relation between the angular positions or directions. Here, for simplification, it is assumed that thefirst imaging unit 110 and thesecond imaging unit 120 have the same imaging range or angle. - Specifically, for example, the
control unit 130 detects the angular positions of the first imaging unit and the second imaging unit based on the numbers of steps from a predetermined initial position of a stepping motor for changing directions of the first imaging unit and the second imaging unit. As far as the angular positions can be detected, any other detecting method may be used. - Then the
control unit 130 calculates the difference between the angular positions of the first imaging unit and the second imaging unit, and if the difference of the angular positions is less than a predetermined value, then the wide-angle image signal is displayed. If the difference of the angular positions is not less than the predetermined value, then independent images are displayed side by side. - In this connection, angular positions or directions of respective imaging units are detected by using the number of steps of the step motor, an encoder may be arranged to detect the angular positions or directions of respective imaging units.
- As described above, the present invention has been described in detail on the basis of the preferred embodiments. However, the present invention is not limited to the above embodiments and various modifications are possible on the basis of the gist of the present invention. It is not excluded from the scope of the invention.
- Also, a part or all of the control in the present embodiments may be supplied to an imaging device over a network or various storage media using a computer program configured to realize the functions of the above-described embodiments. Furthermore, a computer (or a central process unit (CPU), a microprocessor unit (MPU), or the like) in the imaging device may read and execute a program. In this case, the program and a storage medium configured to store the program constitutes the present invention.
- While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.
- This application claims the benefit of Japanese Patent Application No.
2019-91101 filed on May 14, 2019
Claims (16)
- An imaging device (100, 200, 300, 400, 500), comprising:a first imaging unit (110) and a second imaging unit (120), each of which is movable in a predetermined direction;a combination processing unit (140) configured to combine a first image obtained by the first imaging unit (110) and a second image obtained by the second imaging unit (120) to generate a wide-angle image;a determination unit configured to determine whether or not to generate the wide-angle image by the combination processing unit (140), based on a relation between an imaging range of the first imaging unit (110) and an imaging range of the second imaging unit (120), or a position relation between the first imaging unit (110) and the second imaging unit (120) in the predetermined direction.
- The imaging device (100) according to claim 1, comprising a control unit (130) configured to perform control so that, when the determination unit determines that an overlapping range between the imaging range of the first imaging unit (110) and the imaging range of the second imaging unit (120) is equal to or more than a first threshold value, the wide-angle image is generated in the combination processing unit (140), and when the determination unit determines that the overlapping range is less than the first threshold value, the wide-angle image is not generated in the combination processing unit (140).
- The imaging device (100) according to claim 1 or claim 2, comprising a driving unit (111, 121) configured to control at least one of the first imaging unit (110) and the second imaging unit (120), wherein the driving unit (111, 121) includes a zoom unit configured to change the imaging range of at least one of the first imaging unit (110) and the second imaging unit (120).
- The imaging device (100) according to claim 1 or claim 2, comprising a driving unit (111, 121) configured to control at least one of the first imaging unit (110) and the second imaging unit (120), wherein the driving unit (111, 121) includes a rotation unit configured to rotate a corresponding one imaging unit (110, 120) of the first imaging unit (110) and the second imaging unit (120) about at least one optical axis of the first imaging unit (110) and the second imaging unit (120).
- The imaging device (100) according to any one of claims 2 to 4, wherein the first threshold value is determined using the number of pixels included in the overlapping range.
- The imaging device (100) according to any one of claims 2 to 4, wherein the first threshold value is such that an overlapping range is 20% or more with respect to a relatively narrower imaging range of a first imaging range and a second imaging range.
- The imaging device (100) according to any one of claims 2 to 6, wherein the imaging device (100) has a third imaging unit (160), and when an overlapping range between imaging ranges of two adjacent imaging units (110, 120) among the imaging units (110, 120) is equal to or more than a first threshold value, a wide-angle image is generated by combining images obtained by the imaging units (110, 120), and when the overlapping range between the imaging ranges of the two adjacent imaging units (110, 120) is less than the first threshold value, the images obtained by the imaging units (110, 120, 160) are not combined.
- The imaging device (100) according to any preceding claim, wherein the predetermined direction is a circumferential direction, wherein the determination unit detects a position of the first imaging unit (110) in the predetermined direction and a position of the second imaging unit (120) in the predetermined direction.
- The imaging device (100) according to claim 8, wherein the determination unit determines an angle between an optical axis of the first imaging unit (110) and an optical axis of the second imaging unit (120) in the predetermined direction.
- The imaging device (100) according to claim 9, wherein the control unit (130) performs control so that the wide-angle image is generated in the combination processing unit (140) when the angle between the optical axis of the first imaging unit (110) and the optical axis of the second imaging unit (120) is less than a predetermined angle.
- A control apparatus (180) configured to control an imaging device (100) according to any preceding claim, wherein the control apparatus (180) includes:a communication unit (182) configured to communicate with the imaging device (100); anda display unit (183) configured to selectively display at least one of the first image, the second image, and the wide-angle image received from the imaging device (100) via the communication unit (182).
- The control apparatus (180) according to claim 11, wherein the display unit (183) is configured to selectively display the wide-angle image or a non-wide-angle image in which the first image and the second image are displayed without combined.
- The control apparatus (180) according to claim 12, wherein the display unit (183) is configured to display a dialog box configured to request a user's instruction when switching is performed between the wide-angle image display and the non-wide-angle image display.
- An imaging method, comprising:performing imaging using a first imaging unit (110) and a second imaging unit (120), each of which is movable in a predetermined direction;generating a wide-angle image by combining a first image obtained by the first imaging unit (110) and a second image obtained by the second imaging unit (120); anddetermining whether or not to generate the wide-angle image by the combination processing unit (140), based on a relation between an imaging range of the first imaging unit (110) and the second imaging unit (120), or a position relation between the first imaging unit (110) and a second imaging unit (120) in the predetermined direction.
- A program that, when implemented by an imaging device (100), causes the imaging device (100) to perform a method according to claim 14.
- A computer-readable storage medium storing a computer program according to claim 15.
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2019091101A JP2020188349A (en) | 2019-05-14 | 2019-05-14 | Imaging device, imaging method, computer program, and storage medium |
Publications (1)
Publication Number | Publication Date |
---|---|
EP3739867A1 true EP3739867A1 (en) | 2020-11-18 |
Family
ID=70613661
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP20173281.5A Withdrawn EP3739867A1 (en) | 2019-05-14 | 2020-05-06 | Imaging device, control apparatus, imaging method, and storage medium |
Country Status (5)
Country | Link |
---|---|
US (1) | US11240446B2 (en) |
EP (1) | EP3739867A1 (en) |
JP (1) | JP2020188349A (en) |
KR (1) | KR20200131744A (en) |
CN (1) | CN111953890A (en) |
Families Citing this family (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11115604B2 (en) * | 2018-01-02 | 2021-09-07 | Insitu, Inc. | Camera apparatus for generating machine vision data and related methods |
JP7286412B2 (en) | 2019-05-22 | 2023-06-05 | キヤノン株式会社 | IMAGE PROCESSING DEVICE AND CONTROL METHOD THEREOF, IMAGING DEVICE, MONITORING SYSTEM |
JP2020202503A (en) | 2019-06-11 | 2020-12-17 | キヤノン株式会社 | Imaging device, computer program, and recording medium |
JP2021019228A (en) | 2019-07-17 | 2021-02-15 | キヤノン株式会社 | Imaging apparatus, computer program, and recording medium |
CN113784023B (en) * | 2021-08-09 | 2023-06-30 | 深圳市天视通视觉有限公司 | Position determining method, device, image pickup equipment and storage medium |
KR20230071408A (en) * | 2021-11-16 | 2023-05-23 | 삼성전자주식회사 | A method for providing preview image and an electronic device therefor |
Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2004118786A (en) | 2002-09-30 | 2004-04-15 | Sony Corp | Image processing apparatus and method for processing image, recording medium, as well as program |
US20130044240A1 (en) * | 2011-08-17 | 2013-02-21 | Nokia Corporation | Apparatus and method for generating image data |
US20140160234A1 (en) * | 2012-12-10 | 2014-06-12 | Samsung Electronics Co., Ltd. | Photographing apparatus |
US20160077422A1 (en) * | 2014-09-12 | 2016-03-17 | Adobe Systems Incorporated | Collaborative synchronized multi-device photography |
WO2018076883A1 (en) * | 2016-10-31 | 2018-05-03 | 腾讯科技(深圳)有限公司 | Video filming box and video filming method, apparatus, and storage medium |
US20180227485A1 (en) * | 2015-09-18 | 2018-08-09 | Sony Corporation | Image processing apparatus, image processing method, program, and image pickup system |
CN108900764A (en) * | 2018-06-06 | 2018-11-27 | 三星电子(中国)研发中心 | Image pickup method and electronic device and filming control method and server |
JP2019091101A (en) | 2017-11-10 | 2019-06-13 | 株式会社 ゆうちょ銀行 | Automatic transaction device, information processing method and information processing program |
Family Cites Families (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP4979525B2 (en) * | 2007-09-20 | 2012-07-18 | 株式会社日立製作所 | Multi camera system |
US10585344B1 (en) * | 2008-05-19 | 2020-03-10 | Spatial Cam Llc | Camera system with a plurality of image sensors |
US20100097444A1 (en) * | 2008-10-16 | 2010-04-22 | Peter Lablans | Camera System for Creating an Image From a Plurality of Images |
US8947502B2 (en) * | 2011-04-06 | 2015-02-03 | Qualcomm Technologies, Inc. | In camera implementation of selecting and stitching frames for panoramic imagery |
JP2012159616A (en) * | 2011-01-31 | 2012-08-23 | Sanyo Electric Co Ltd | Imaging device |
US20160295108A1 (en) * | 2015-04-01 | 2016-10-06 | Cheng Cao | System and method for panoramic imaging |
CN105635579B (en) * | 2015-12-31 | 2018-01-09 | 宇龙计算机通信科技(深圳)有限公司 | A kind of method for displaying image and device |
US10051180B1 (en) * | 2016-03-04 | 2018-08-14 | Scott Zhihao Chen | Method and system for removing an obstructing object in a panoramic image |
CN107040719A (en) * | 2017-03-21 | 2017-08-11 | 宇龙计算机通信科技(深圳)有限公司 | Filming control method and imaging control device based on double screen terminal |
TWI672670B (en) * | 2018-03-12 | 2019-09-21 | Acer Incorporated | Image stitching method and electronic device using the same |
JP7182935B2 (en) | 2018-07-30 | 2022-12-05 | キヤノン株式会社 | IMAGING DEVICE, CONTROL METHOD AND PROGRAM |
JP7210199B2 (en) | 2018-09-20 | 2023-01-23 | キヤノン株式会社 | IMAGING DEVICE, CONTROL METHOD, COMPUTER PROGRAM, AND STORAGE MEDIUM |
JP7271132B2 (en) | 2018-10-26 | 2023-05-11 | キヤノン株式会社 | Imaging device and surveillance system |
JP7250483B2 (en) | 2018-11-12 | 2023-04-03 | キヤノン株式会社 | Imaging device, computer program, and storage medium |
JP7316809B2 (en) | 2019-03-11 | 2023-07-28 | キヤノン株式会社 | Image processing device, image processing device control method, system, and program |
-
2019
- 2019-05-14 JP JP2019091101A patent/JP2020188349A/en not_active Ceased
-
2020
- 2020-04-28 US US16/860,398 patent/US11240446B2/en active Active
- 2020-05-06 KR KR1020200053682A patent/KR20200131744A/en active IP Right Grant
- 2020-05-06 EP EP20173281.5A patent/EP3739867A1/en not_active Withdrawn
- 2020-05-13 CN CN202010400199.2A patent/CN111953890A/en active Pending
Patent Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2004118786A (en) | 2002-09-30 | 2004-04-15 | Sony Corp | Image processing apparatus and method for processing image, recording medium, as well as program |
US20130044240A1 (en) * | 2011-08-17 | 2013-02-21 | Nokia Corporation | Apparatus and method for generating image data |
US20140160234A1 (en) * | 2012-12-10 | 2014-06-12 | Samsung Electronics Co., Ltd. | Photographing apparatus |
US20160077422A1 (en) * | 2014-09-12 | 2016-03-17 | Adobe Systems Incorporated | Collaborative synchronized multi-device photography |
US20180227485A1 (en) * | 2015-09-18 | 2018-08-09 | Sony Corporation | Image processing apparatus, image processing method, program, and image pickup system |
WO2018076883A1 (en) * | 2016-10-31 | 2018-05-03 | 腾讯科技(深圳)有限公司 | Video filming box and video filming method, apparatus, and storage medium |
JP2019091101A (en) | 2017-11-10 | 2019-06-13 | 株式会社 ゆうちょ銀行 | Automatic transaction device, information processing method and information processing program |
CN108900764A (en) * | 2018-06-06 | 2018-11-27 | 三星电子(中国)研发中心 | Image pickup method and electronic device and filming control method and server |
Also Published As
Publication number | Publication date |
---|---|
US20200366854A1 (en) | 2020-11-19 |
JP2020188349A (en) | 2020-11-19 |
CN111953890A (en) | 2020-11-17 |
KR20200131744A (en) | 2020-11-24 |
US11240446B2 (en) | 2022-02-01 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP3739867A1 (en) | Imaging device, control apparatus, imaging method, and storage medium | |
JP7385052B2 (en) | Photography methods, equipment, electronic equipment and storage media | |
US20080239102A1 (en) | Camera Controller and Zoom Ratio Control Method For the Camera Controller | |
KR101739318B1 (en) | Display control apparatus, imaging system, display control method, and recording medium | |
JP6312046B2 (en) | Image processing system, image processing method, and program | |
US20140218612A1 (en) | Multiple field-of-view video streaming | |
JP2019062575A (en) | Control device, control method, and program | |
JP2006191411A (en) | Camera control apparatus, image display apparatus, method for controlling camera, method for displaying image, and program | |
US9906710B2 (en) | Camera pan-tilt-zoom (PTZ) control apparatus | |
JP2019129410A (en) | Monitoring camera, control method thereof, and program | |
JP2019054369A (en) | Imaging device, control method of imaging device, and program | |
JP2009049798A (en) | Camera control method, camera control device, camera control program, and camera system | |
JP5039607B2 (en) | Imaging apparatus and control method thereof | |
EP3751838A1 (en) | Imaging device, method for controlling imaging device, and recording medium | |
JP2016096482A (en) | Image processing apparatus, image processing method, and program | |
US11516390B2 (en) | Imaging apparatus and non-transitory storage medium | |
JP7191663B2 (en) | IMAGING DEVICE, CONTROL METHOD AND PROGRAM | |
WO2023079941A1 (en) | Information processing device and information processing method | |
JP7431609B2 (en) | Control device and its control method and program | |
WO2021261099A1 (en) | Information processing device, display range determination method, and program | |
US20240022812A1 (en) | Image capturing system, control apparatus, image capturing apparatus, and display apparatus constituting the system, control method, and display method | |
EP4333448A1 (en) | Control apparatus, image pickup apparatus, control method, and program | |
CN114205532B (en) | Shooting method, device, equipment and medium of virtual camera | |
WO2023189829A1 (en) | Information processing device, information processing method, and program | |
JP2023117955A (en) | Information processing device, imaging apparatus, imaging system, method, and program |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20210518 |
|
RBV | Designated contracting states (corrected) |
Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
17Q | First examination report despatched |
Effective date: 20211208 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R079 Free format text: PREVIOUS MAIN CLASS: H04N0005232000 Ipc: H04N0023450000 |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G06T 7/55 20170101ALI20230427BHEP Ipc: G03B 37/04 20060101ALI20230427BHEP Ipc: H04N 23/90 20230101ALI20230427BHEP Ipc: H04N 23/698 20230101ALI20230427BHEP Ipc: H04N 23/695 20230101ALI20230427BHEP Ipc: H04N 23/69 20230101ALI20230427BHEP Ipc: H04N 23/667 20230101ALI20230427BHEP Ipc: H04N 23/66 20230101ALI20230427BHEP Ipc: H04N 23/62 20230101ALI20230427BHEP Ipc: H04N 23/45 20230101AFI20230427BHEP |
|
INTG | Intention to grant announced |
Effective date: 20230512 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE APPLICATION HAS BEEN WITHDRAWN |
|
18W | Application withdrawn |
Effective date: 20230907 |