WO1999051023A1

WO1999051023A1 - Teleconferencing system

Info

Publication number: WO1999051023A1
Application number: PCT/IL1998/000150
Authority: WO
Inventors: Ori J. Braun
Original assignee: 3Dv Systems, Ltd.
Priority date: 1998-03-29
Filing date: 1998-03-29
Publication date: 1999-10-07
Also published as: AU6516698A

Abstract

A method of teleconferencing between a local party and at least one remote party, the method comprising: producing an image of a scene including the local party to the teleconference, in which dynamically changing background portions are effectively blanked; and transmitting the blanked image to the at least one remote party to the teleconference.

Description

TELECONFERENCING SYSTEM

FIELD OF THE INVENTION The present invention is related to the field of teleconferencing and in particular to the means and methods for providing increased privacy and more efficient transmission of images. BACKGROUND OF THE INVENTION

Teleconferencing systems are well known. Such systems operate over the telephone lines, by wireless transmission or more recently, over the Internet. In addition, local area teleconferencing systems that transfer images and sound over Ethernet or other local area networks are known. In almost all such systems, the requirement that relatively large amounts of data, mainly image data, be transmitted in real time is a serious limitation. Even after various methods of image data compression have been used, often with negative impact on image quality, the problem of bandwidth remains a serious one.

One seemingly unrelated problem, which is both less evident and relatively unnoticed, is the problem of privacy during teleconferencing. In the present context, the term privacy refers to the transmission of images which contain only the information which it is desired to transmit and not to the normal use of the term, namely, the avoidance of eavesdropping on the transmission. Herein, this later problem, which has received extensive concern, is referred to as security. During teleconferencing, images are acquired, generally by a TV camera, and are transmitted to the other party of the conference call. These images contain substantial amounts of information that the sender may not desire be seen by the other party. Such information may include the identity of others in the room, the venue of the transmission, often identified by the background, etc. In order to avoid embarrassing or damaging transmissions, teleconferencing is often limited to sterile environments and seldom to personal sites such as bedrooms, offices. banks, doctor's offices or the like, which may be more convenient.

Additionally, a major portion of the image contains such "background" information, whose transmission generally is not useful and, as indicated above, is often not desired. This transfer of unnecessary information increases the bandwidth requirements of the system. U.S. Patent 5.519,436, the disclosure of which is incorporated by reference, describes a system for reducing the amount of information to be transmitted during teleconferencing. In the described system, an acquired image is compared to a background image and only pixels that are different from the background image are compressed and transmitted to the other party. However, this patent does not provide a solution to the privacy problem, nor does it provide a

1 solution when the background is changing due to the movement of people in the background or of the camera itself.

In the past few years a number of systems have been developed for the combination of a foreground image with a background image to form a composite image. Such techniques are routinely used in news broadcasts or in other video situations where it is desirable to provide a different background for a scene than that present in reality.

One method of producing such a composite image is to image the active portion of the image against a background having a particular color, most frequently blue, and to fill the blue areas with the "new" background. Other methods segment the image using various well-known imaging techniques and place one of the segments into a different background scene. Other systems, such as described in WO 97/01111 or US 5,668,605, the disclosures of which are incorporated herein by reference, directly measure the distance to various points in a scene and place objects, or portions of the image which are within a particular distance range, into a different background. SUMMARY OF THE INVENTION

It is an object of some aspects of the present invention to provide a teleconferencing system that transmits desired video information while blocking the transmission of unnecessary or undesired information.

It is an object of some aspects of the present invention, to provide a teleconferencing system in which the user controls the quantity of background information transmitted, along with the image of the speaker. Preferably, the user (speaker) himself controls the image being transmitted without the need for technicians or other auxiliary personnel.

It is an object of some aspects of the present invention to provide a teleconferencing system having a reduced bandwidth requirement. It is an object of some aspects of the present invention to provide a system that reduces the amount of information that is transmitted and/or provides privacy, even when the background is changing.

A preferred embodiment of the invention includes a video camera, means for determining foreground and changing background portions of the image being viewed by the camera, means for blanking background poπions of the image and means for transmitting the unblanked portion of the image to the other party or parties in the teleconference.

Preferably, the foreground and the background are determined by determining the depth of portions of an image within the scene. Alternatively, the camera may be a camera that inherently produces images that are blank for distances outside a given range of distances.

? It is an object of some aspects of the invention to provide a system in which only portions of an image containing the image of the party taking part in the teleconference is tr∑tnsmitted. This allows teleconference to take place at almost any location without transmission of sensitive or disturbing background images. In a preferred embodiment of the invention, a monitor is provided for the local teleconferencing party for viewing both the images sent by the other party and the images being transmitted to the other party. A control is provided for adjusting the distance range of the images that are sent to the other party. Preferably, the control is part of or adjacent to the console so that the local party may control the depth range of the image being sent. There is thus provided, in accordance with a preferred embodiment of the invention, a method of teleconferencing between a local party and at least one remote party, the method comprising: producing an image of a scene including the local party to the teleconference, in which dynamically changing background portions are effectively blanked; and transmitting the blanked image to the at least one remote party to the teleconference. In a preferred embodiment of the invention, producing comprises: producing a first image of the scene, including the background, illuminating the local party with light of a specified wavelength; producing a second image of the portions of the scene illuminated by the specific wavelength; and blanking the first image of the scene based on the second image.

In a preferred embodiment of the invention, producing comprises: producing a first image of the scene, including the background, illuminating the background of the scene with light of a specified wavelength; producing a second image of the portions of the scene illuminated by the specific wavelength; and blanking the first image of the scene based on the second image. Preferably, illuminating comprises illuminating with infrared radiation. In a preferred embodiment of the invention, producing comprises: producing an image of the scene; determining the distance to respective portions of the image; and effectively blanking the portions of the image, which are outside a given range of distances. There is further provided, in accordance with a preferred embodiment of the invention, a method of teleconferencing between a local party and at least one remote party, the method comprising: producing an image of a scene including the local party to the teleconference, in which portions of the image which are outside a given range of distances are effectively blanked; and transmitting the blanked image to the at least one remote party to the teleconference.

Preferably, the method includes determining the distance to respective portions of the image.

Preferably, determining the distance comprises, scanning the scene with a rangefmder. In a preferred embodiment of the invention, determining the distance comprises producing an image comprises producing an image having pixel values, which comprise distance information. In a preferred embodiment of the invention, the method includes: defocusing the background of the scene; and acquiring an image with a defocused background; and filtering the acquired image to remove the background.

In a preferred embodiment of the invention producing an image comprises: acquiring -an image of a scene utilizing a camera that produces image pixel values only for pixels that represent a portion of the scene inside the range of distances. Preferably, the method comprises providing an input from the local party of the range to be transmitted.

In a preferred embodiment of the invention, the method includes compressing the effectively blanked image prior to transmission.

Preferably, the method includes displaying the image to be transmitted to the at least one remote party for viewing by the local party.

Preferably, the method includes displaying at least one image received from at least one remote party for viewing by the local party.

Preferably, effectively blanked comprises assigning a constant value to those portions of the scene outside the desired range of distances. There is further provided, in accordance with a preferred embodiment of the invention, teleconferencing apparatus comprising: at least one camera that views a scene including a local party to the teleconference and produces a first image; a light source which illuminates a portion of the scene with illumination of a given wavelength; and an imager which images regions of the scene illuminated by the illumination of the given wavelength; and blanking circuitry, which receives the second image and produces a blanked image of the scene in which background portions of the scene are effectively blanked, based on the second image, prior to transmission to a remote party to the teleconference.

In a preferred embodiment of the invention, the light source selectively illuminates the local party. Alternatively, the light source selectively illuminates portions of the image not including the local party.

There is further provided, in accordance with a preferred embodiment of the invention, teleconferencing apparatus comprising: at least one camera which views a scene including a local party to the teleconference and produces an image; and blanking circuitry which receives an indication of a desired range of distances in the scene to be transmitted to the remote party and produces an blanked image of the scene in which portions of the scene that are outside the range are effectively blanked prior to transmission to a remote party to the teleconference. Preferably, the teleconferencing apparatus includes an image rangefmder. The rangefmder produces a distance image in which the values of the image are indicative of whether the portion is within or outside the given range and the blanking circuitry is responsive to the distance image.

Preferably, the teleconferencing apparatus includes a user interface having a user input for designating said indication of distance range.

There is further provided, in accordance with a preferred embodiment of the invention, teleconferencing apparatus comprising: a camera which views a scene including a local party to the teleconference, and produces an image, the camera including optics which defocuses background portions of the image behind the local party; and blanking circuitry which effectively blanks the defocused image prior to transmission to a remote party to the teleconference.

Preferably the blanking circuitry comprises: a filter which receives the defocused image and low pass filters it to remove the defocused background portions of the image.

Preferably, the teleconferencing apparatus includes compression circuitry, which receives the effectively blanked image and compresses the effectively blanked image prior to transmission to the remote party.

Preferably, the teleconferencing apparatus includes an image controller, which receives the effectively blanked image and displays it for viewing by the local party. Preferably the image controller also receives at least one image from a remote party and displays it for viewing by the local party. In a preferred embodiment of the invention, the teleconferencing apparatus includes an image controller, which receives at least one image from a remote party and displays it for viewing by the local party.

BRIEF DESCRIPTION OF THE DRAWINGS The present invention will be more fully understood from the following description of the preferred embodiment of the invention, taken together with the following drawings in which:

Fig. 1 shows a preferred embodiment of a teleconferencing terminal, in accordance with a preferred embodiment of the invention; and

Fig. 2 shows a simplified block diagram of a portion of the terminal of Fig. 1. DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Reference is made to Fig. 1, which is a simplified illustration of a teleconferencing terminal 10, in accordance with a preferred embodiment of the invention. Terminal 10 preferably comprises an image display 12, which is generally a display of a personal computer or a dedicated video display. Terminal 10 also includes a television camera 14 that views a scene including a party to the teleconference. Terminal 10 also preferably includes a microphone 16 that picks up audio signals, such as speech of the party, for transfer, together with a video image to the other party or parties to the teleconference.

In a preferred embodiment of the invention, terminal 10 includes a keyboard 18 for dialing to the other party. Terminal 10 also includes a system 20 for determining the distance to points on the image taken by camera 14. System 20 may be any system, as known in the art for determining distances to points in an image, for example, any one of the systems described in the aforementioned WO 97/01111 or US 5,668,605. It also preferably includes one or a pair of control knobs 22 and 24 for inputting the range of distances of the image taken by television camera 14 that are to be transmitted to the other party, based on distances determined by system 20.

Reference is now made additionally to Fig. 2, which is a simplified schematic diagram of electronic circuitry 26 of terminal 10 in accordance with a preferred embodiment of the invention. As indicated above camera 14 generates receives an image of the party using terminal 10. Distance system 20 generates a second image, designated as a "distance image" on Fig. 2. Pixel values in the distance image are representative of the distance of system 20 (and hence the camera) from points in the image. A blanking image generator 28 receives the distance image and generates a binary image. The binary image represents an image of those points that are to be tr.ansmitted to the other party and those which should not. In general, the "ones" in the image can be either logical transmit or logical blank for the final image to be transmitted. Blanking image generator 28 generates the binary image by comparing the pixel values in the image with the minimum and maximum distances input to the system by knobs 22 and 24 or by any other system which allows a user to input such values. For pixels having values between the minimum and maximum distances, the pixel value is preferably replaced by a binary value which represents "tr.ansmit" for those values within the minimum/maximum range and a binary value which represents "blank" for other distance values.

A masked image is produced in an image masker 30, based on the binary image, preferably by blanking or giving some preset value to those pixels in the scene for which the binary image pixels have a logical "blank" value. Since much of the image now has a constant value, such an image is very efficiently compressed in an image compressor 32 prior to transmission to other parties to the teleconference.

In a preferred embodiment of the invention, the image being transmitted is also shown on display 12 (as indicated above with respect to Fig. 1) together with a decompressed image received from one of the other parties to the teleconference. The display is preferably controlled by a controller 34 that receives the images to be displayed and organizes the image on display 12 accordingly. If more than two parties are involved in the teleconference, controller 34 receives a plurality of images from the other parties, and arranges and displays them on display 12. In operation, in preferred embodiments of the invention, the local party adjusts the range of distances transmitted (preferably based on the image of the transmitted image which is displayed on display 12) utilizing knobs 22 and 24, to give the required privacy. The resulting masking of the image will then automatically result in a reduced bandwidth requirement for the image being transmitted.

7 The present invention has been described utilizing a particular embodiment thereof. It should be understood that the details of terminal 10 may be varied within the ambit of the invention. For example, as described above it is based on a simple camera which images a full image, which is then blanked, based on a second distance image. In another preferred embodiment of the invention, a camera such as that described in WO 97/01111 is used. In this camera, an image is generated only for objects having a given range of distances. For a system utilizing such a camera, elements 14, 20, 28 and 30 would be replaced by the camera. Alternatively, such an camera is used to generate a blanking image for a standard camera.

Furthermore, .any convenient input system may be used to input the desired range of distances; for example, knobs 22 and 24 may be replaced by toggle switches, a trackball, or keys on a computer keyboard. Furthermore, while a dedicated teleconferencing system is described in Figs. 1 and 2, all (or a portion of) the elements shown in Fig. 2 (except for cameras 14 and 20 or a single camera as described above) may be realized by software and/or hardware in a general or special purpose computer. Furthermore, while in the preferred embodiment of the invention, the criteria for separating foreground and background is a measurement of distance, in other preferred embodiments of the invention other methods of separation may be used. For example, either the background or the foreground may be illuminated with infrared radiation. An image of the portions so irradiated can then be used to separate the foreground from the background. Preferably, the infra-red image is used to generate a binary blanking image as aforesaid.

In another preferred embodiment of the invention, camera 14 has a lens with a sharply delineated field of view. In this embodiment, the local party is in focus and the background is out of focus. The camera output is filtered by a high pass filter to remove the background. In accordance with one embodiment of the invention, the smooth portions of the image are blanked and the unblanked portion is restored by a restoration filter complementary to the high pass filter. In an alternative preferred embodiment of the invention, the filtered or blanked image is used to generate a binary blanking image, which is used to blank the unfiltered image to remove the background, in the same manner as the circuitry of Fig. 2.

Claims

1. A method of teleconferencing between a local party and at least one remote party, the method comprising: producing an image of a scene including the local party to the teleconference, in which dynamically changing background portions are effectively blanked; and transmitting the blanked image to the at least one remote party to the teleconference.

2. A method of teleconferencing according to claim 1 wherein producing comprises: producing a first image of the scene, including the background, illuminating the local party with light of a specified wavelength; producing a second image of the portions of the scene illuminated by the specific wavelength; and blanking the first image of the scene based on the second image.

3. A method of teleconferencing according to claim 1 wherein producing comprises: producing a first image of the scene, including the background; illuminating the background of the scene with light of a specified wavelength; producing a second image of the portions of the scene illuminated by the specific wavelength; and blanking the first image of the scene based on the second image.

4. A method according to claim 2 or claim 3 wherein illuminating comprises illuminating with infrared radiation.

5. A teleconferencing method according to claim 1 wherein producing comprises: producing a first image of the scene, including the background; determining the distance to respective portions of the image; and effectively blanking the portions of the first image that are outside a given range of distances.

6. A method of teleconferencing between a local party and at least one remote party, the method comprising: producing an image of a scene including the local party to the teleconference, in which portions of the image which are outside a given range of distances are effectively blanked; and

9 transmitting the blanked image to the at least one remote party to the teleconference.

7. A method according to claim 6 and including determimng the distance to respective portions of the image.

8. A teleconferencing method according to claim 5 or claim 7 wherein determining the distance comprises: scanning the scene with a rangefmder.

9. A teleconferencing method according to any of claims 5, 7 or 8. wherein determining comprises: producing an image having pixel values that comprise distance information.

10. A teleconferencing method according to claim 6 and including: defocusing the background of the scene; and acquiring an image with a defocused background; and filtering the acquired image to remove the background.

11. A teleconferencing method according to claim 1 wherein producing comprises: acquiring an image of a scene utilizing a camera that produces image pixel values only for pixels, which represent a portion of the scene inside the range of distances.

12. A teleconferencing method according to any of claims 1 or 5-9, and including: providing an input from the local party of the range to be transmitted.

13. A teleconferencing method according to any of the preceding claims and including: compressing the effectively blanked image prior to transmission.

14. A teleconferencing method according to any of the preceding claims and including displaying the image to be transmitted to the at least one remote party for viewing by the local party.

10

15. A teleconferencing method according to any of the preceding claims and including displaying at least one image received from at least one remote party for viewing by the local party.

16. A teleconferencing method according to any of the preceding claims, wherein effectively blanked comprises assigning a constant value to those portions of the scene outside the desired range of distances.

17. Teleconferencing apparatus comprising: at least one camera that views a scene including a local party to the teleconference and produces a first image; a light source which illuminates a portion of the scene with illumination of a given wavelength; and an imager which images regions of the scene illuminated by the illumination of the given wavelength; and blanking circuitry, which receives the second image and produces a blanked image of the scene in which background portions of the scene are effectively blanked, based on the second image, prior to transmission to a remote party to the teleconference.

18. Teleconferencing apparatus according to claim 17 wherein the light source selectively illuminates the local party.

19. Teleconferencing apparatus according to claim 17 wherein the light source selectively illuminates portions of the image not including the local party.

20. Teleconferencing apparatus comprising: at least one camera which views a scene including a local party to the teleconference, and produces an image; and blanking circuitry which receives an indication of a desired range of distances in the scene to be transmitted to the remote party and produces an blanked image of the scene in which portions of the scene that are outside the range are effectively blanked prior to transmission to a remote party to the teleconference.

11

21. Teleconferencing apparatus according to claim 20 and including an image rangefmder which produces a distance image in which the values of the image are indicative of whether the portion is within or outside the given range, wherein the blanking circuitry is responsive to the dist.ance image.

22. Teleconferencing apparatus according to claim 20 or claim 21 and including a user interface having a user input for designating said indication of distance range.

23. Teleconferencing apparatus comprising: a camera which views a scene including a local party to the teleconference, and produces an image, the camera including optics which defocuses background portions of the image behind the local party; and blanking circuitry which effectively blanks the defocused image prior to transmission to a remote party to the teleconference.

24. Teleconferencing apparatus according to claim 23, wherein the blanking circuitry comprises: a filter which receives the defocused image and low pass filters it to remove the defocused background portions of the image.

25. Teleconferencing apparatus according to any of claims 17-24 and including compression circuitry, which receives the effectively blanked image and compresses the effectively blanked image prior to transmission to the remote party.

26. Teleconferencing apparatus according to any of claims 17-25 an including an image controller, which receives the effectively blanked image and displays it for viewing by the local p-aity.

27. Teleconferencing apparatus according to claim 26 wherein the image controller also receives at least one image from a remote party .and displays it for viewing by the local party.

28. Teleconferencing apparatus according to any of claims 17-25 and including an image controller, which receives at least one image from a remote party and displays it for viewing by the local party.

12