WO2005015266A2

WO2005015266A2 - Electro-optical system comprising image sensors having selected spatial resolutions

Info

Publication number: WO2005015266A2
Application number: PCT/IL2004/000737
Authority: WO
Inventors: Ehud Gal; Gennadiy Liteyga
Original assignee: O.D.F. Optronics Ltd.
Priority date: 2003-08-11
Filing date: 2004-08-09
Publication date: 2005-02-17
Also published as: WO2005015266A3; IL157331A0

Abstract

The present invention provides an electro-optical system and method for presenting the observer a unified picture comprising two or more image sensors (2,3). The sensor have different apertures and obtain images having different spatial resolutions (4,5) of different regions. At least one region has a smaller aperture (4) and is at least partially located with the region having larger aperture (5).

Description

ELECTRO-OPTICAL SYSTEM COMPRISING IMAGE SENSORS HAVING SELECTED SPATIAL RESOLUTIONS

Field of the invention

The present invention relates to the field of electro-optics. More specifically

the invention describes a system and method for obtaining images from

several different electro-optical sensors and combining the separate images

to form a unified picture on a display device.

Background of the invention

The present invention provides a system and method for imaging and

transmitting imagery data from several different sensors and combining the

separate images to form a unified picture thus providing enhanced

situational awareness to the viewer.

When using an image-sensor to observe a certain area, the sensor's fixed

resolution and lens aperture dictate the ability of the viewer to observe fine

details in the image. Trying to cover wider aperture areas causes the

sensor's resolution to be divided over a larger area therefore the ability of the viewer to detect fine details in an image is reduced as the aperture is enlarged.

There are numerous applications in which a viewer is required to view as

wide an area as possible and at the same time to be able to observe details

within that area. For example, image sensors of security systems are often

required to monitor a certain area to detect intrusions or unusual events.

While it would be best to use a single image sensor covering the entire area

that needs to be monitored, this is often not practical since the ability to

observe details will then become unacceptably reduced due to the reduced spatial resolution.

Another example is remote operation of unmanned platforms (such as land,

aerial or maritime vehicles). In such applications the operator is required to view as wide area as possible in order to fully grasp the entire environment

in which the vehicle operates, and at the same time, the operator must be

able to observe a specific region of interest and observe details in that region.

Additional applications exist in the security, military, industrial, civilian and medical fields.

Existing systems which answer only partially the above-described needs include the following: Image sensors having optical zoom capabilities. While these systems provide the ability to zoom-in on an area of interest to detect details, the zoomed-in image replaces the view of the wide area. Therefore it is not possible to view both the wide area and to observe fine details in a narrow

area simultaneously. Using multiple sensors having different spatial resolution. It is

possible to employ several different sensors each having a different spatial

resolution, thus having one sensor cover a wide field of view at a certain

spatial resolution and having a second sensor cover a narrower field of view,

within the wide filed of view, at a higher spatial resolution. Existing

systems employing multiple sensors do not display the two images as a

single unified image. Furthermore, the existing systems employ dedicated

transmission channels for each of the sensors.

It is the purpose of the present invention to provide means for imaging a

wide aperture area at a certain spatial resolution and simultaneously

imaging a narrower field of view, located within the wide field of view, at a higher spatial resolution.

It is another purpose of the present invention to provide means for

transmitting the two captured images using a single communications

channel. It is yet another purpose of the present invention to provide a method for

displaying the two captured images as a single unified picture on a display

device.

It is another purpose of the present invention to provide means for imaging,

transmitting and presenting images of selected spatial resolution captured

by more than one image sensor.

Additional purposes and advantages of the present invention will become apparent as the description proceeds.

Summary of the invention

In a first aspect, the present invention provides an electro-optical system for

presenting to an observer a unified picture comprising images from two or more sensors. The system comprises: a) an image gathering unit comprising two or more sensors for gathering images and a transmission assembly comprising a transmitter for transmitting the images; and b) a control and display station comprising a receiver for receiving the images transmitted from the image gathering unit, a CPU for processing the images, and a display device for displaying the images as a unified picture. The sensors have different apertures and obtain images having different

spatial resolutions of different regions. At least one of the regions has a smaller aperture and is at least partially located within a region having larger aperture. The transmission assembly comprises a transmitter, which

transmits the images acquired by the sensors. The unified picture is

comprised of at least one image having a higher spatial resolution

superimposed upon a lower resolution image and located at the correct relative location in the lower resolution image.

In a preferred embodiment of the invention, the transmitter transmits the images captured by the image gathering unit using a single transmission

channel shared between the sensors. In this embodiment, the system can

further comprise an electronic controller capable of selecting the rate and

sequence of images transmitted from each of the sensors through the single

transmission channel.

In another embodiment of the invention, the system further comprises

motors that are capable of controlling the position of the image sensors, thus

determining which region each of the sensors will image. In this

embodiment the control and display station further comprises means for

entering control commands for activating the motors. The CPU can control

the operation of the motors of the image gathering unit according to the

analysis of the images captured by the image sensors. The control and display station can either be located in the close proximity

of the image gathering unit and connected to it by means of wires for transferring signals between the station and the unit or can be remotely located from the image gathering unit and connected to it wirelessly. The

control and display station can further comprise a transmitter for

transmitting control commands entered by the user.

The image gathering unit of the system of the invention can comprise image

sensors capable of acquiring images in the visible, Infra Red (IR), Near Infra

Red (NIR), or in different spectral regions.

In the system of the invention, the display device can be a computer screen

or a display screen, with the image from each sensor projected onto the

screen by a dedicated projector.

In another aspect, the present invention provides a method for presenting to an observer a unified picture comprising images from two or more sensors. The method of the invention comprises the following steps: a) providing images gathered from two or more sensors; b) transmitting the images; c) receiving the images at a control and display station; d) processing the images; and e) displaying the images as a unified picture.

In a preferred embodiment of the method of the invention, the sensors have

different apertures and obtain images having different spatial resolutions of

different regions. At least one of the regions has a smaller aperture and is at

least partially located within a region having larger aperture. The images

acquired by the sensors are transmitted by means of a transmitter. The unified picture is comprised of at least one image having a higher spatial

resolution superimposed upon a lower resolution image and located at the

correct relative location in the lower resolution image.

In a preferred embodiment of the method of the invention, the images acquired by the sensors are transmitted by means of a transmitter using a

single transmission channel shared between the sensors.

Brief description of the drawings

Fig. 1 schematically shows an embodiment of an image gathering

unit comprising two image sensors with different apertures and different

spatial resolutions; Fig. 2 schematically shows another embodiment of an image gathering unit comprising two image sensors with different apertures and

different spatial resolutions; Fig. 3 is a block diagram showing the main components in an

embodiment of the system of the present invention; Fig. 4 schematically shows a typical application utilizing the system of the present invention;

Fig. 5 schematically shows an embodiment of the display method of

the invention providing a unified picture comprising two images having

different spatial resolutions; Fig. 6 schematically shows another embodiment of the display

method of the invention; and

Fig. 7 schematically shows another embodiment of the electro-optical

system of the invention and the display method.

Detailed description of the drawings

For a better understanding of the invention and to show how the same may

be carried into effect, reference will now be made, purely by way of example, to the accompanying drawings. With specific reference to the drawings in detail, it is stressed that the particulars shown are by way of example and

for purposes of illustrative discussion of preferred embodiments of the

present invention only, and are presented for the purpose of providing what is believed to be the most useful and readily understood description of the

principles and conceptual aspects of the invention. In this regard, no

attempt is made to show structural details of the invention in more detail

than is necessary for a fundamental understanding of the invention. From the description taken together with the drawings it will be apparent to those

skilled in the art how the several forms of the invention may be embodied in practice.

Fig. 1 shows schematically a preferred embodiment of the image gathering unit of the electro-optical system of the present invention. In Fig. 1, the unit (1) contains a first image sensor (2) and a second image sensor (3). The first

image sensor (2) is set to capture a first region (4) having a certain aperture,

and the second image sensor (3) is set to capture a second region (5) having

an aperture narrower than the aperture of the first region (4). The second region (5) is at least partially located within the first region (4). The

resolution of the two image sensors is such that the narrower region (5) is

captured with higher spatial resolution than the wider region (4). The higher spatial resolution of the narrower region (5) enables detection and

observation of details that can not be detected, or can only be poorly

detected, with the first image sensor (2) due to its relatively low spatial

resolution. Depending on the requirements of the application, the sensors

can be chosen for gathering images in any selected wavelength region, e.g. visible, near infrared, or infrared.

The images from the two image sensors are transmitted from the unit (1)

using a transmission assembly (6). The transmission assembly (6) may be

either wired or wireless. The transmission assembly (6) may comprise two transmitters, each transmitter dedicated to transmit the image of a specific

image sensor, in such a way that both transmitters transmit simultaneously the images captured by both image sensors. Alternatively, and preferably,

the transmission assembly (6) may include only a single transmitter, having

a limited bandwidth. In the preferred case, the two sensors share the bandwidth based on a sharing rate determined by the parameters of the

system. The sharing rate can be predetermined or user selected depending

on the content of the images. The system may be programmed to send the

images from the sensors at any desired allowable rate and in any sequence;

for example, if the transmitter is capable of transmitting a bandwidth equal

to 30 image frames per second, than the system may be programmed to send

15 images from the first image sensor (2) and 15 images of the second image

sensor (3) alternately. The system (1) may optionally include a processor

(not shown) designed to analyze the images captured by the second image sensor (3), so that images captured by the second image sensor (3) will be

transmitted only when a specific event of interest occurs. In the absence of

such an event, only images from the first image sensor (1) are transmitted

and the full bandwidth is allocated to the first image sensor (1).

The embodiment described in Fig. 1 comprises two sensors. Other

embodiments may be designed to include more than two sensors and different relationships between the images in which the images may be, for

example, partly or wholly overlapping or adjacent to one another. The images transmitted from the system (1) are received and displayed at a

processing and display station, where the images are also integrated to form a unified picture on a display device, as will be described with reference to

Fig. 3 and Fig. 5.

Fig. 2 is a schematic description of another embodiment of the image gathering unit of the present invention. According to this embodiment, the

image gathering unit (1') includes a first sensor (2) and a second sensor (3).

Each of the sensors is connected to a motor, the first sensor (2) is connected

to a first motor (7) and the second sensor (3) is connected to a second motor

(8). The motors are capable of controlling the position of the sensors to aim them such that they will produce images of selected regions of interest. The

image gathering unit also contains an electronics assembly (9) that

comprises a transmitter for transmitting the images. Optionally additional

components are included such as a receiver and a processor for receiving control commands from a remote location and activating the motors

according to control commands.

Fig. 3 is a block diagram showing the main elements of a preferred

embodiment of the electro-optical system of the present invention which

includes an image gathering unit (10) as described hereinabove with

reference to Fig. 1 and Fig. 2. A control and display station (11) is designed for receiving the images that are transmitted from the image gathering unit

(10), processing the images and displaying the images. Optionally, the control and display station (11) is also capable of sending control commands to the image gathering unit (10) in order to control the aiming of the sensors

located in the unit (10), as described with reference to Fig. 2. The control

and display station includes a receiver (12) for receiving the images from the

image gathering unit (10). The receiver may be replaced by a transceiver in order to enable both the reception of images and the transmission of control

commands to the unit (10). The control and display station (11) further

comprises a control panel (13) designed to enable the operator to operate the

control and display unit (11) and to enter control commands to be

transmitted to the image gathering unit (10). The control and display

station (11) further comprises a CPU (14) designed to process control

commands entered by the user through the control panel (13) and also processes the images received from the image gathering unit (10). A display unit (15), for instance a computer screen, is provided for displaying the

images received from the sensor assembly.

Fig. 4 is a schematic description of a typical application in which the electro-

optical system of the present invention is employed. In Fig. 4, the image

gathering unit is installed on an aerial platform (16). The image gathering

unit comprises two sensors, a first sensor covering a wide aperture area (17)

and a second sensor covering a narrower aperture sector (18) located within the wide aperture area (17). The narrow sector (18) is captured with higher

spatial resolution than the wide sector (17), in such a way that it is possible to observe details of the sector that are not observable in the wide field of

view image. The images captured by the sensors can either be transmitted

to a control and display station on the aerial platform for on board viewing in the case of a manned platform or wirelessly transmitted to a remote

control and display unit (19).

Fig. 5 schematically shows an embodiment of the display method of the

invention providing a unified picture comprising two images having different spatial resolution. Fig. 5 shows how the two images captured by

the system described with reference to Fig. 4 are displayed on a computer

screen. A first image (20) corresponds to the wide aperture area (17) in Fig.

4. and a second image (21) corresponds to the narrow aperture area (18) in

Fig. 4. Since the narrow aperture area (18) is located within the wide

aperture area (19), it is desirable that in the picture presented to the viewer

on the screen of the display device, the image of the narrow sector will

occupy its actual location in the unified picture. This allows the viewer to intuitively interpret the details of the area selected to be viewed using the

narrow aperture sensor in relationship to the area surrounding it. To

accomplish this objective, the system comprises means for determining the

exact coordinates of the center of the images gathered by each of the sensors

and transmits the coordinates along with the images to the CPU. In displaying the images, the CPU generates a picture on the display device that is comprised of an image from the wide aperture sensor with a

corresponding image from the narrow aperture sensor superimposed on top of it. The CPU aligns the coordinates of each of the images to insure

placement of the image of the narrow sector at the appropriate location in

the entire image. Therefore the viewer will be able to observe a unified

picture with a selected area of the entire scene shown with higher spatial resolution, thus revealing details not observable in the wide aperture view.

It is to be noted that, for the sake of simplicity area 18 is chosen to be

exactly in the middle of area 17, such that in the unified picture shown in

Fig. 5, image 21 is superimposed over the center of image 20. Such a

simplified arrangement is of course not necessary, and it is easily within the capability of skilled persons using techniques well known in the art to provide unified pictures comprising any desired spatial relationship between

the images.

Fig. 6 schematically shows another embodiment of the method of displaying

the unified picture. According to this embodiment, two image projectors are employed. A first image projector (23) is used for projecting the wide sector

and a second projector (24) is used for projecting the narrow sector. The

projectors project the respective images onto the back side of a concave

display screen (25). The viewer (26) is positioned facing the concave side of

the screen. As in the previous embodiment, the CPU of the system uses the coordinates associated with each image to project the center of each image

to the correct relative location on the screen (25), or if the sensors have a known fixed position, the user can pre-position the projectors appropriately. It is desirable to place the projector (24) so that it can project the image of

the narrow sector to the appropriate position in the entire image without

obscuring essential parts of the image projected from the first projector (23).

Fig. 7 schematically shows another embodiment of the image gathering unit

and unified picture according to the present invention. In this embodiment,

more than two image sensors are employed to cover several different sectors.

In the example shown in Fig. 7, four separate image sensors, each with its own aperture and spatial resolution, are used to capture images of four

different regions identified by the letters A, B, C, and D respectively. All of

the image sensors are connected to a transmission assembly (27). The

transmission assembly (27) may include several separate transmitters, each transmitter dedicated to a particular image sensor. Preferably transmission assembly (27) contains only a single transmitter having a limited

communication bandwidth. In the preferred case the communication

bandwidth is shared between the image sensors based either on a predetermined rate and sequence, or by commands received from a remote

location. The images transmitted by the transmission assembly (27) are

integrated to form unified image (28) on a display device using techniques

known in the art and according to the principles described herein. Although embodiments of the invention have been described by way of

illustration, it will be understood that the invention may be carried out with many variations, modifications, and adaptations, without departing from its

spirit or exceeding the scope of the claims.

Claims

1. An electro-optical system for presenting to an observer a unified picture comprising images from two or more sensors, said system comprising: a) an image gathering unit comprising two or more sensors for gathering images and a transmission assembly comprising a transmitter for transmitting said images; and b) a control and display station comprising a receiver for receiving said images transmitted from said image gathering unit, a CPU for processing said images, and a display device for displaying said images as a unified picture;

wherein, said sensors have different apertures and obtain images having

different spatial resolutions of different regions, at least one of said regions

having a smaller aperture being at least partially located within a region

having larger aperture; said transmission assembly comprises a transmitter, which transmits said images acquired by said sensors; and said unified

picture is comprised of at least one image having a higher spatial resolution

superimposed upon a lower resolution image and located at the correct

relative location in said lower resolution image.

2. A system according to claim 1, wherein said transmitter transmits the images captured by said image gathering unit using a single transmission channel shared between said sensors

3. A system according to claim 1, further comprising motors that are capable of controlling the position of the image sensors, thus determining which region each of said sensors will image.

4. A system according to claim 3, wherein the control and display station further comprises means for entering control commands for activating the motors.

5. A system according to claim 1, wherein the control and display station is located in the close proximity of the image gathering unit and is connected to it by means of wires for transferring signals between said station and said unit.

6. A system according to claim 1, wherein the control and display station is remotely located from the image gathering unit and is connected to it wirelessly for transferring signals between said station and said unit.

7. A system according to claim 1, wherein the control and display station further comprises a transmitter for transmitting control commands entered by the user.

8. A system according to claim 3, wherein the CPU controls the operation of the motors of the image gathering unit according to the analysis of the images captured by the image sensors.

9. A system according to claim 1, wherein the image gathering unit comprises image sensors capable of acquiring images in the visible spectral region.

10. A system according to claim 1, wherein the image gathering unit comprises image sensors capable of acquiring images in the Infra Red (IR) spectral region.

11. A system according to claim 1, wherein the image gathering unit comprises image sensors capable of acquiring images in the Near Infra Red (NIR) spectral region.

12. A system according to claim 1, wherein the sensor assembly comprises image sensors capable of acquiring images in different spectral regions.

13. A system according to claim 2, further comprising an electronic controller capable of selecting the rate and sequence of images transmitted from each of the sensors through the single transmission channel.

14. A system according to claim 1, wherein the display device is a computer screen.

15. A system according to claim 1, wherein the display device is a display screen and the image from each sensor is projected onto said screen by a dedicated projector.

16. A method for presenting to an observer a unified picture comprising images from two or more sensors, said method comprising: a) providing images gathered from two or more sensors; b) transmitting said images; c) receiving said images at a control and display station; d) processing said images; and e) displaying said images as a unified picture;

wherein, said sensors have different apertures and obtain images having

having a smaller aperture being at least partially located within a region

having larger aperture; said images acquired by said sensors are transmitted by means of a transmitter; and said unified picture is comprised

of at least one image having a higher spatial resolution superimposed upon

a lower resolution image and located at the correct relative location in said lower resolution image.

17. A method according to claim 16, wherein said images acquired by said

sensors are transmitted by means of a transmitter using a single transmission channel shared between said sensors.