WO2013066290A1

WO2013066290A1 - Videoconferencing using personal devices

Info

Publication number: WO2013066290A1
Application number: PCT/US2011/058575
Authority: WO
Inventors: Mark E. Gorzynski; Michael D. Derocher; James R. Cole
Original assignee: Hewlett-Packard Development Company, L.P.
Priority date: 2011-10-31
Filing date: 2011-10-31
Publication date: 2013-05-10

Abstract

A videoconferencing system may include a conference host (130) that is associated with an environment (110) and that may detect a personal device (140) with a participant that is visiting the environment (110) during a videoconference. A video capture system (142) of the personal device (140) may then be used to provide a video stream representing the participant during the videoconference.

Description

VIDEOCONFERENCING USING PERSONAL DEVICES

BACKGROUND

A room videoconferencing system can connect a local conference site to remote conference sites allowing participants in a videoconference to see and hear each other. A room videoconferencing system typically has local audio and video capture and reproduction systems that are associated with and located in the room. The capture systems acquire images and sound from participants in the room for sharing with the remote sites.

Immersive room systems may use multiple cameras that can create panoramic images of a room and the conference participants in the room. When such images are displayed on large video systems, the natural 'trompe-l'oei effect of these systems allows remote participants to see other participants in the context of their environment and activities. Common issues for immersive videoconferencing systems may include cost and the limits on the number of participants that an immersive room system can accommodate. In particular, when a large number of people are in a room for a meeting, the individual participants generally appear smaller in the displayed view of the room. Larger displays can be used at the receiving location to provide larger images, but larger displays increase costs.

Videoconferencing systems can also use a variety of techniques for showing individual participants and activities. For example, if all participants cannot be shown at a sufficient size in an audience or panoramic view, the current speaker at a meeting can be shown in a separate image. Insuring that a video stream is focused on the correct speaker can be difficult. In particular, a room videoconferencing system can use multiple fixed cameras and voice activity detection to determine which camera and associated video stream corresponds to a speaker. However, a videoconferencing system employing a large number of fixed cameras can be costly and is still limited to the specific views of the fixed cameras. Mechanical or digital pan and zoom control can be used to capture or create a view of a speaker, but such systems may be limited by the quality of the voice activity and/or face finding algorithms that may be used when pointing a camera at the speaker. Further, mechanical camera systems have delay in camera motion and may still require multiple cameras to capture more than one speaker or participant at a time.

Web conferencing is an alternative to room conferencing. With web conferencing, each person joins a meeting individually. While web conferencing is possible when a group of people gather together in the same room, web conferencing does not coordinate audio-video capture of the participants and may not provide an optimal representation of the context and interaction of the conference participants in the room.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 is a block diagram showing an example of a room videoconferencing system that can employ and coordinate visiting devices.

Fig. 2 illustrates an example of a videoconference system connecting room conference locations and an individual endpoint.

Fig. 3 illustrates an example of how a room videoconferencing system can composite images from visiting electronic devices to generate a composite view of the audience in a room.

Fig. 4 is a flow diagram of an example of a process that a room videoconferencing system can use to coordinate permanent and visiting devices in a room.

Use of the same reference symbols in different figures may indicate similar or identical items.

DETAILED DESCRIPTION

Participants in a local conference environment for a videoconference commonly bring personal devices such as laptop computers that contain cameras with them into the local conference environment, and when the participants position those personal devices for use, the cameras may be oriented in a manner that enables the capture of close up views of the respective participants using the personal devices. A videoconferencing system associated with a local environment such as a conference room can access and use the cameras on the personal devices. In particular, video streams from visiting personal devices can provide close up views that may be used in a coordinated fashion with or without video streams from audio-video systems that are installed in the local environment.

Fig. 1 shows a videoconferencing system 100 that includes a local environment 1 10 containing a room audio-visual (AV) system 120 and a local conference host system 130. A typical example of local environment 1 10 used during a videoconference is a conference room, but more generally, local environment 1 10 could be another type of geographical location or even a mobile environment that can accommodate participants of a videoconference.

Room AV system 120 in general is set up in local environment 1 10 and is a resource that may be shared by all participants in environment. Room AV system 120 may be permanent in the sense that AV system 120 may be intended to remain in environment 1 10. AV system 120 may also in some circumstances include components that are physically attached or mounted in environment 1 10. AV system 120 in the configuration of Fig. 1 particularly includes a video capture system 122, an audio capture system 124, a video display 126, and an audio system 128. Video capture system 122 may, for example, include one or more cameras that provide one or more fixed views of environment 1 10 or may have motorized mountings and focus systems that are able to change the view or focus of one or more cameras in video capture system 122. Audio capture system 124 can similarly include one or more microphones positioned to capture sound originating at different locations in environment 1 10. Room display 126 and audio system 128 provide video and audio output under the control of AV system 120. The audio and video may particularly include audio or video created from audio or video captured by video and audio capture systems 122 and 124 and from audio and video signals that local conference host 130 may receive from a remote conference host 150. AV system 120 may additionally include other video conferencing equipment such as a computational engine for functions such as AV signal processing or network communication, a network interface able to communicate with local host 130, user interface devices which may include a keyboard, pointing device, and a video display that is separate from room display 126.

Local conference host 130 provides logical connections to AV system 120 and to visiting devices 140 in local environment 1 10. In the illustrated implementation, local conferencing host 130 includes an interface 132 for communication with room AV system 120, an interface 134 for communication with visiting devices 140, a monitoring system 136 that can monitor the available video streams or audio streams, and a source selection module 136 that selects video and audio streams for use in environment 1 10 and for sharing with remote conference hosts 150. Local conference host 130 also communicates with one or more remote conference hosts 150. In general, some of the functionality of the illustrated configuration of local conference host 130 could be implemented elsewhere in a video conferencing system, e.g., in room AV system 120 or a visiting device 140. However, having local conference host 130 select AV streams may be particularly convenient because the data coming from room AV system 120 may be digitally compressed on a network or may otherwise differ from the data streams of visiting devices 140 and local conference host 130 is well positioned to handle necessary conversions for the various devices.

Local conference host 130 generally works with one or more remote conference hosts 150 to coordinate AV streams based on a meeting state of a videoconference involving multiple geographically separated locations. Local conference host 130 can particularly select which video stream or streams from environment 1 10 are sent to remote conference host 150. For example, when a participant in environment 1 10 chooses to present, local conference host 130 can receive a video stream corresponding to the presentation, display the presentation on display 126, and send a video stream representing the presentation to remote conference host 150. Further, when a presenter provides the presentation, the presenter may also choose to offer a live local camera view from their personal device 140 to represent the presenter during the presentation. Both the presentation and live participant video streams can be displayed on display 126 and/or communicated to the remote conference hosts 150.

Conference hosts 130 and 150 may correspond to endpoints in a videoconference that includes multiple endpoints that are interconnected through one or more networks (not shown), e.g., local area networks and a wide area network such as the Internet. Such networks may further include additional videoconferencing equipment such as one or more multipoint control units (MCUs). Videoconferences generally can be implemented using many different network topologies and a variety of network protocols. For example, the H.323 protocol promulgated by the ITU Telecommunication Standardization Sector (ITU-T) for audio-video signaling over packet switched networks is currently a common protocol used for videoconferencing. However, the specifics of the connection of local host 130 to remote devices such as remote conference host 150 and the protocol employed for communications with remote devices is not critical for the current disclosure.

Local conference host 130, as mentioned above, includes a communication interface 134 capable of communicating with personal devices 140 that are visiting environment 1 10. For example, local conference host 130 may include an interface for wired communication, e.g., Ethernet or USB ports that can be connected to visiting personal devices 140 in environment 1 10 or for wireless communication, e.g., a Wi-Fi network or Bluetooth communication, with visiting personal devices 140. The nature of interface 134 may be such that local proximity of visiting devices 140 is required for communications with interface 134, but if interface 134 has an extended range, other mechanisms could optionally be used to confirm that visiting devices 140 are in local environment 1 10.

Visiting personal devices 140 may be devices such as laptop computers, pad computers, smartphones, or any other devices capable of communicating useful information to local conference host 130. Such personal devices 140 commonly have audio capture systems such as a microphone, a video capture 142 such as a built-in camera, and communication interfaces such as a Wi-Fi adapter, a Bluetooth adapter, an Ethernet port, or a USB port suitable for connection with local conference host 130. Personal devices 140 can generate many types of data that may be useful for a videoconference. For example, one personal device 140 may be able to provide camera video, microphone audio, and data (images, graphics, sounds, or text) that may be stored in a personal device 140 or generated by a personal device 140. Personal device 140 may also provide a user interface that allows a participant to interact with local conference host 130. For example, local conference participants that are in environment 1 10 may be given the option to register their personal device 140 with local conference host 130. The participant may also be able to choose whether to permit their personal device 140 to share a selected functionality or data of their personal device 140 as needed for sharing locally in environment 1 10 and/or for sharing with remote sites 150. Once a personal device 140 is registered with local conference host 130, local conference host 130 can then determine which of the permitted resources of local devices 140 are actually used, shared locally, or shared with remote participants.

Fig. 2 illustrates one example of how personal camera video streams may be made available to a local conference host and used during a videoconference. In particular, Fig. 2 illustrates a videoconference connecting three locations or environments 210, 220, and 230 that are remote from each other. Environment 210 is a conference room containing multiple participants. Environment 220 is also a conference room containing multiple participants, but environment 230 is the site of an individual participant.

Environment 210 has a local conference host 212 that connects to communal AV systems such as a room camera 213 and a room display 214 and to one or more visiting personal devices 215. In Fig. 2, conference host 212 has access to video streams from room camera 213 and at least one personal device 215. A participant at environment 210 may use their personal device 215 to provide a video stream 216 representing a presentation, e.g., a video file or a series of slides containing information that the presenter wishes to share with other participants. The presenter may thus direct local conference host 212 to display video stream 216 on the local communal video display 214 and to share video stream 216 with other environments 220 and 230. In response, conference host 212 can select video stream 216 and a live video stream 217 from the camera of visiting device 215 for sharing with remote environments 220 and 230. Local conference host 212 also shares a video stream 218 from room camera 213 to provide a room view giving a context for the presenter and other participants in environment 210.

Conference hosts 222 and 232 respectively for environments 220 and

230 can similarly provide respective video streams 226 and 236. In particular, conference host 222 for environment 220 may select a room camera 223 to provide a group shot of participants in environment 220, who are then part of the audience of the presentation. Similarly, conference host 232 for environment 230 can select a video stream 236 from a camera 234. Each environment 210, 220, and 230 may then select and display video for the participants in the respective environments. Alternatively, a multi-point control unit (not shown) could receive all of the video streams from conference hosts 212, 222, and 232 and provide video streams (e.g., composited video streams) to respective conference hosts 212, 222, and 232. In Fig. 2, room display 214 at environment 210 displays a composite video image including presentation video stream 216, presenter video stream 217, and audience video streams 226 and 236 from respective environments 220 and 230. Room display 224 at environment 220 displays a composite video including presentation video stream 216, presenter video stream 217, and audience video streams 218 and 236 from respective locations 210 and 230, and a display at environment 230 displays a composite video including presentation video stream 216, presenter video stream 217, and group audience video streams 218 and 226 from respective locations 210 and 220.

Some environments such as environment 230 may only have personal devices such as device 235. In Fig. 2, environment 230 contains only a single device 235, but more generally multiple participants could meet in the same environment whether or not the environment contains all of the communal AV resources shown in environment 1 10 of Fig. 1 . Even when no communal AV resources are available in an environment a local conference host can coordinate the data streams from visiting personal devices in the environment to improve a video conference. Fig. 3, for example, illustrates an environment containing multiple personal devices 312, 314, and 316 that provide respective video streams 332, 334, and 336 to a local conference host 320. Conference host 320 may select one or more of video streams 332, 334, and 336 for separate use or may combine one or more of video streams 332, 334, and 336 into a composited video stream. In particular, conference host 320 can combine multiple personal streams representing images 332, 334, and 336 of individual participants into a video stream representing a pseudo panorama view 330 of a local environment containing visiting devices 312, 314, and 316. Assembled view 330 of the local environment can be shared with remote environments to show interactions of the participants in the local environment.

Fig. 4 shows a flow diagram of a videoconferencing process 400 in which a local conference host may use the resources of one or more personal devices visiting a conference room or other environment. Process 400 begins in block 410 where personal devices in a room register with the local conference host. The registration of a personal device may, for example, indicate the nature of resources in the personal device that the participant controlling the personal device is willing to share with the local conference host. Registration may also indicate the identity of the participant controlling the personal device. In some situations, a visiting device could contain software that is configured according to a user preference to automatically register the personal device upon encountering a conference host that is associated with a particular environment or that satisfies other criteria. Additionally or alternatively, in some implementations, registration of the personal devices may be as simple as establishing network connections between the personal devices and the local conference host. Registration 410 may take place at any time before or during a videoconference including but not limited to when a participant enters the room associated with the conference host or when a participant wishes to take some action in a videoconference. The participant may be able to change or cancel the registration at any time.

The local conference host in block 420 monitors communications from registered visiting devices and the available video or audio streams from registered visiting devices or from the communal AV resources associated with the local conference host. In some implementations, when personal devices are brought into the conference room and registered with the local host, the local host may immediately start to capture audio or video streams from those personal devices. The video streams may not be shared as part of the videoconference at that point, but if some activity involving a visiting personal device is detected, the video stream corresponding to that personal device is immediately available can be selected for sharing locally or with a remote participant. Alternatively, in response to detecting activity at a personal device, the local host may then direct or cause that personal device to start capturing a video stream. For example, the camera in a personal device may be turned on or activated so that a video stream from the camera can be selected and shared in the videoconference after a need is detected. Monitoring 420 by the local conference host can further include negotiating with a personal device when a participant wishes to take an action such as starting a presentation or asking a question of a presenter or speaker. Monitoring 420 may also include analysis of the content of available audio or video streams. For example, a local conference host can detect sound or speaking and identify a visiting device that is associated with a speaking participant from the volumes or other characteristics of the available audio streams. The conference host may be able to associate personal video or audio streams from one or more visiting or permanent devices with specific source devices and participants, so that selection of an appropriate video capture device may be simplified when compared to systems that may attempt to point a room camera at a particular participant.

Process 400 of Fig. 4 illustrates some examples of events that may cause the local host to select a video stream from a visiting device. In block 430, the local conference host detects a change in presentation. For example, the change in a presentation may be a participant using their personal device to request that their presentation be shared during a videoconference. The determination of whether to change a presentation may require that the local conference host contact one or more remote conference hosts to avoid a conflict. If a presentation is to be provided by a visiting device, the local conference host knows the identity of the visiting device as the source of the presentation, so that a visiting device identified is known or identified in block 432. When the local conference host begins to share the presentation, the live video stream from the camera in the identified device can also be shared in block 434 to provide a view of the presenter. The system may thus be able to better display appropriate video streams to match the data currently being shared.

Another type of event that the local conference host can check may be identified from analysis of the available audio and video channels. For example, in block 440, the local conference host can detect speech in the available audio channels, and the local host can identify whether the room video capture system or the camera of a personal device would provide a better view of the speaker. A video device can thus be identified in block 442, and the video stream from the identified video device can be shared in block 444.

Some of the videoconferencing systems described herein can provide a number of advantages over current videoconferencing systems. For example, a videoconferencing system can reduce cost by making use of increasingly available personal devices to improve video capture in local environments. Further, local room conferencing systems have a choice of local room cameras or personal camera views and can select the best video stream or streams for the situation and may improve a user's videoconferencing experience. For example, a room view may provide a wide angle shot while a camera of a visiting device provides a close up view. Using a personal device to provide a close up view in a room system may also have an advantage of providing a clear view of a participant when the personal device may partially block the view that a room camera has of the participant. A system using visiting personal devices may also avoid the need for room systems to use costly complex cameras and pointing systems for capturing video images of individual presenters or speakers. Use of visiting devices providing personal views of individual participants also easily scales the number of camera views with the number of attendees without adding room cost. The systems can further coordinate local personal devices with a local conference host to insure a desirable choice of either room or personal streams and for coordinating with output to the local room displays and audio system.

Some configurations of the systems described above may be implemented using a computer-readable media, e.g., a non-transient media, such as an optical or magnetic disk, a memory card, or other solid state storage storing instructions that a computing device can execute to perform specific processes that are described herein. Such media may be or may be contained in a server or other device connected to a network such as the Internet that provides for the downloading of data and executable instructions.

Although particular implementations have been disclosed, these implementations are only examples and should not be taken as limitations.

Various adaptations and combinations of features of the implementations disclosed are within the scope of the following claims.

Claims

What is claimed is:

1 . A videoconferencing method comprising:

operating a videoconferencing system including a conference host (130) that is associated with an environment (1 10);

detecting a personal device (140) with a participant that is visiting the environment (1 10) during a videoconference;

employing a video capture system (142) of the personal device (140) to provide a video stream representing the participant during the videoconference; and

selecting by the conference host (130) of a video capture system that provides a video stream for sharing with a remote participant of the video conference, wherein the conference host (130) selects the video capture system from a set of video capture systems including the video capture system (142) of the personal device (140).

2. The method of claim 1 , wherein operating the videoconferencing system comprises employing a video capture system (122) that is installed in the environment and is in the set from which the conference hosts selects.

3. The method of claim 1 , wherein detecting the personal device (140) comprises detecting a plurality of personal devices (140) respectively with a plurality of participants visiting the environment (1 10) during the

videoconference, wherein each of the personal devices (140) has a video capture system (142) that is in the set from which the conference hosts selects.

4. The method of claim 3, selecting the video capture system comprises selecting the video capture system (142) of one of the personal devices (140).

5. The method of claim 4, further comprising sharing with the remote participant (150) a presentation provided by one of the personal devices (140), wherein in response to the personal device (140) providing the presentation, the conference host selects the video capture system (142) of the personal device (140) that provides the presentation to provide the video stream for sharing.

6. The method of claim 4, wherein selecting the video capture system comprises:

monitoring audio streams from the personal devices (140) to detect a source of a sound in the environment (1 10); and

selecting the personal device (140) that is identified as being closest to the source of the sound.

7. The method of claim 1 , wherein the personal device (140) is a portable computer.

8. A videoconferencing system that is associated with a local environment, the system comprising:

a monitoring system (136) that monitors personal devices (140) that are visiting the local environment (1 10) and in communication with the video conferencing system (130); and

a selection system (138) adapted to select one or more data streams from the local environment (1 10) for sharing with a remote participant during a video conference, wherein the selection system (138) is configured to select from among a set of data streams including one or more data streams from video capture devices (142) in the personal devices (140) visiting the local environment (1 10).

9. The system of claim 8, further comprising:

a first communication interface (134) configured to communicate with the personal devices (140); and

a second communication interface (132) configured to communicate with capture devices (120) associated with the local environment (1 10).

10. The system of claim 9, further comprising the capture devices (120) that are associated with the local environment (1 10), wherein the set of data streams from which the selection system (138) selects further includes one or more data streams from the capture devices (120) associated with the local environment.

1 1 . The system of claim 10, wherein the local environment comprises a room, and the data capture devices (120) include a camera (122) mounted in the room.

12. The system of claim 8, wherein the personal devices (140) comprise a portable computer having a built-in camera (142).

13. The system of claim 8, wherein the selection system (138) is adapted to select the data stream from one of the personal devices based on identifying audio as being detected by that personal device.

14. The system of claim 8, wherein the selection system (138) is adapted to select the data stream of one of the personal devices based on that personal device sharing content with the remote participant.

15. A non-transient computer readable media containing instructions that when executed by the processing system performs a process comprising:

selecting a video capture system that provides a video stream for sharing with a remote participant of the video conference, wherein the video capture system is from a set of video capture systems including the video capture system (142) of the personal device (140).