WO2007053286A2

WO2007053286A2 - Video conferencing systems and methods

Info

Publication number: WO2007053286A2
Application number: PCT/US2006/040253
Authority: WO
Inventors: Jacob Apelbaum
Original assignee: First Data Corporation
Priority date: 2005-10-12
Filing date: 2006-10-12
Publication date: 2007-05-10
Also published as: WO2007053286A3

Abstract

Systems and methods are provided for video conferencing geographically disperse users. Each user operates a user computer. A network connection is established among the user computers. A respective connection speed with the network connection is determined independently for each user computer. A video signal is transmitted from one of the user computers over the network connection to others of the user computers at the independently determined connection speeds.

Description

VIDEO CONFERENCING SYSTEMS AND METHODS

BACKGROUND OF THE INVENTION

[0001] This application relates to video conferencing systems and methods.

[0002] Effective collaboration in business and other environments has long been recognized as being of considerable importance. This is particularly true for the development of new ideas as interactions fostered by the collaboration may be highly productive in expanding those ideas and generating new avenues for thought. As business and other activities have become more geographically disperse, efforts to provide collaborative environments have relied on travel by individuals so that they may collaborate in person or have relied on telecommunications conferencing mechanisms.

[0003] Travel by individuals to participate in a conference may be very costly and highly inconvenient to the participants. Despite this significant drawback, it has long been, and still is, the case that in-person collaboration is viewed as much more effective than the use of telecommunications conferencing. Telephone conferences, for example, provide only a limited form of interaction among the participants, does not easily permit side conversations to take place, and is generally a poor environment for working collaboratively with documents and other visual displays. Some of these drawbacks are mitigated with video conferencing in which participants may see and hear other, but there are still weaknesses in these types of environments as they are currently implemented.

[0004] There is accordingly a general need in the art for improved conferencing capabilities that provides for high interactivity among conference participants.

BRIEF SUMMARY OF THE INVENTION

[0005] Embodiments of the invention thus provide methods of video conferencing a plurality of geographically disperse users. Each user operates a respective one of a plurality of user computers. A network connection is established among the plurality of user computers. A respective connection speed with the network connection is determined independently for each of the plurality of user computers. A video signal is transmitted from one of the user computers over the network connection to others of the user computers. The video signal is transmitted to each of the others of the user computers at the connection speed determined independently for the each of the others of the user computers.

[0006] The connection speed for at least two of the plurality of user computers may be different. In some embodiments, a bandwidth level over the network connection is monitored in real time, with the respective connection speed for at least one of the plurality of user computers being changed in accordance with the bandwidth level.

[0007] Various functionalities complementary to the transmission of the video signal may be provided in different embodiments. For instance, in one embodiment, an audio signal is also transmitted from the one of the user computers over the network connection to the others of the user computers, hi another embodiment, an instant-messaging connection is established among the plurality of user computers. This permits an instant message to be transmitted over the instant-messaging connection form one of the plurality of user computers to another of the plurality of user computers. The instant message may be transmitted over the instant-messaging connection from the one of the plurality of user computers to a plurality of others of the user computers. A record of the instant message may be saved.

[0008] In further embodiments, a directory of the plurality of user computers is provided. A data file may be transmitted from the one of the user computers over the network connection to another of the user computers. A computer program may also be shared over the network connection among the plurality of user computers. Access to a desktop of a first of the plurality of user computers may be provided over the network connection by a second of the plurality of user computers different from the first of the user computers.

[0009] Embodiments of the invention may also include a variety of techniques to improve and/or optimize transmission of the video signal. For example, in one embodiment, data that comprises a graphical object is transmitted from the one of the user computers over the network connection to the others of the user computers. The graphical object is cached, and a cache identifier identifying the graphical object is sent from the one of the user computers over the network connection to the others of the user computers. [0010] In other embodiments, a portion of the video signal that will be obscured by a graphical output is identified. The video signal is then transmitted without the portion of the video signal that will be obscured by the graphical output.

[0011] hi various embodiments, the video signal comprises a sequence of frames, hi one such embodiment, the sequence of frames is analyzed for redundant information, with the

IS redundant information being stripped from the transmitted video signal. In another such embodiment, each frame comprises a plurality of color pixels. Each frame is analyzed to identify insignificant pixels, with a color depth of the insignificant pixels being reduced.

[0012] Other embodiments of the invention provide methods and systems for managing bandwidth in transmitting multimedia over a network. An audio stream is created from transmission of audio information over the network at an audio rate. A video stream is created for transmission of video information over the network at a video rate. A data stream is created for transmission of data information over the network at a data rate. Each of the audio rate, the video rate, and the data rate is determined independently of others of the audio rate, the video rate, and the data rate. The audio stream, the video stream, and the data stream are collectively transmitted over the network and respectively at the audio rate, the video rate, and the data rate according to a priority hierarchy that gives precedence of the audio stream over the video stream and data stream, and gives precedence of the data stream over the video stream.

[0013] A bandwidth may be assigned for the video stream and data stream substantially equal to a total bandwidth for the network less a bandwidth used by the audio stream corresponding to the audio rate. A current average size of the data stream may be determined. A bandwidth for the video stream may be assigned equal to the bandwidth for the video stream and data stream less the current average size of the data stream. The video rate is then determined from the assigned bandwidth for the video stream. In some instances, the resulting assigned bandwidth for the video stream may be approximately zero, in which case the data stream and the video stream may compete for transmission over the network.

[0014] A specification may be received of a network-connection type to define the total bandwidth. Examples of network-connection types includes a 28.8 kbps modem connection, a 56.6 kbps modem connection, a cable connection, a digital subscriber line ("DSL"), an integrated services digital network ("ISDN") connection, a satellite link, and a local-area-network ("LAN") connection. The data stream may comprise whiteboard information, may compπse mstant-messaging information, or may comprise program-sharing information in certain specific embodiments.

[0015] Still other embodiments of the invention provide a method of transmitting video data from a first user device to a second user device. In some embodiments, the video data are received as a sequence of frames from the first user device at a video-transmission system. A portion of a first frame in the sequence of frames is identified as having information redundant with a portion of a second frame in the sequence of frames. The redundant information is stripped from one of the first and second frames. The stripped frame is substituted for the one of the first and second frames into a modified sequence of frames. The modified sequence of frames is transmitted with the video-transmission system to the second user device.

[0016] hi some embodiments, a portion of a third frame in the sequence of frames may also be identified as having the redundant information. The redundant information is stripped from the third frame and the stripped third frame is substituted for the third frame in the modified sequence of frames. In other embodiments, a portion of a third frame in the sequence of frames may be identified as having second information redundant with a second portion of the first frame. The second redundant information is then stripped from one of the first and third frames, and the stripped one of the first and third frames substituted for the one of the first and third frames into the modified sequence of frames.

[0017] Stripping of the redundant information from the one of the first and second frames may comprise replacing pixels of the one of the first and second frames with transparency channels. In some instances, the modified sequence of frames may be generated by removing a frame from the sequence of frames. For example, in one embodiment, an excessive-motion pattern may be identified within a plural subset of the sequence of frames. The modified sequence of frames may then be generated by removing a frame from the subset of the sequence of frames. Alternatively, a set of anchor frames may be generated from a statistical analysis of the subset of the sequence of frames. In other embodiments, pixels within a color frame may be identified as insignificant to an image represented by the color frame, with the modified sequence of frames being generated by reducing a color depth of the identified pixels.

[0018] In a particular embodiment, the redundant information comprises a graphical object. Stripping of the redundant information from the one of the first and second frames comprises storing the graphical object in a cache with a cache identifier. When the modified sequence of frames is transmitted, the one of the first and second frames is then transmitted with the cache identifier.

[0019] In other embodiments, the video data are also received as a sequence of frames from the first user device at a video-transmission system. A connection bandwidth from the video-transmission system to the second user device is factorized. A connection speed from the video-transmission system to the second user device is also factorized. A request for a change in at least one of a frame size and a frame quality for one of the frames in the sequence of frames is identified. Codecs are assigned for transmission of the sequence of frames in accordance with the factorized connection bandwidth, factorized connection speed, and identified request. The sequence of frames is transmitted in accordance with the assigned codecs with the video-transmission system to the second user device.

[0020] In some embodiments, a video hardware accelerator is also identified, with the codecs being assigned for transmission of the sequence of frames further in accordance with the identified video hardware accelerator. In other embodiments, a change in at least one of the connection bandwidth and the connection speed is identified, with the codecs being reassigned in accordance with the identified change.

[0021] In still other embodiments, the video data are also received as a sequence of frames from the first user device at a video-transmission system. A graphical object comprised by a first of the frames is identified. The identified graphical object is stored in a cache with a cache identifier. The first of the frames is transmitted with the video- transmission system to the second user device. The graphical object is identified in a second of the frames different from the first of the frames. The graphical object is stripped from the second of the frames. The stripped second of the frames and the cache identifier are transmitted with the video-transmission system to the second user device.

[0022] The various aspects of the different sets of embodiments may also be combined with each other and in different ways that set forth above in various alternative configurations.

[0023] Methods of the invention may be embodied in a computer-readable storage medium having a computer-readable program embodied therein for directing operation of a computer system to conference a plurality of geographically disperse users, each of whom operates a respective one or a plurality of user computers. The computer-readable program includes instructions to implement the methods as described above.

BRIEF DESCRIPTION OF THE DRAWINGS

[0024] A further understanding of the nature and advantages of the present invention may be realized by reference to the remaining portions of the specification and the drawings wherein like reference numerals are used throughout the several drawings to refer to similar components.

[0025] Fig. 1 is a flow diagram summarizing multiple capabilities that may be provided with a conferencing application in an embodiment of the invention;

[0026] Fig. 2 A as a flow diagram that summarizes aspects of video and audio conferencing within the conferencing application;

[0027] Fig. 2B is an exemplary screen view that illustrates aspects of Fig. 2 A;

[0028] Fig. 3 A is a flow diagram that summarizes aspects of an instant-messaging capability within the conferencing application;

[0029] Fig. 3B is an exemplary screen view that illustrates aspects of Fig. 3 A;

[0030] Fig. 4A is a flow diagram that summarizes aspects of a locator service within the conferencing application;

[0031] Fig. 4B is an exemplary screen view that illustrates aspects of Fig. 4 A;

[0032] Fig. 5 A is a flow diagram that summarizes aspects of a file-transfer capability within the conferencing application;

[0033] Fig. 5B is an exemplary screen view that illustrates aspects of Fig. 5 A;

[0034] Fig. 6A is a flow diagram that summarizes aspects of a program-sharing capability within the conferencing application;

[0035] Fig. 6B is an exemplary screen view that illustrates aspects of Fig. 6 A; [0036] Fig. 7A is a flow diagram that summarizes aspects of a desktop-sharing capability within the conferencing application;

[0037] Fig. 7B is an exemplary screen view that illustrates aspects of Fig. 7 A;

[0038] Fig. 8 A is a flow diagram that summarizes aspects of a method for sequence optimization that may be used by the conferencing application;

[0039] Fig. 8B is a set of frames that illustrates aspects of Fig. 8 A;

[0040] Fig. 9A is a flow diagram that summarizes aspects of a method for palette optimization that may be used by the conferencing application;

[0041] Fig. 9B is a set of frames that illustrates aspects of Fig. 9 A;

[0042] Fig. 1OA is a flow diagram that summarizes aspects of a method for frame- reduction optimization that may be used by the conferencing application;

[0043] Fig. 1OB is a set of frames that illustrates aspects of Fig. 1OA;

[0044] Fig. 1 IA is a flow diagram that summarizes aspects of a method for motion analysis and frame keying that may be used by the conferencing application;

[0045] Fig. 1 IB is a set of frames that illustrates aspects of Fig. 1OA;

[0046] Fig. 12A is a flow diagram that summarizes aspects of a method for video- sequence transmission that may be used by the conferencing application;

[0047] Fig. 12B is a set of frames that illustrates aspects of Fig. 12A; and

[0048] Fig. 13 is a schematic representation of a computational unit that may be used to implement the conferencing application in embodiments of the invention.

DETAILED DESCRIPTION OF THE INVENTION

1. Overview

[0049] Embodiments of the invention provide a multifunctional application that establishes a real-time communications and collaboration infrastructure. A plurality geographically distributed user computers are interfaced by the application to create a rapid work environment and establish integrated multimodal communications. In embodiments of the invention, the application may provide telephony and conferencing support to standard switched telephone lines through an analog modem; high-speed connectivity through an integrated-services digital network ("ISDN") modem and virtual private network ("VPN"), with adapter support; telephony and conferencing support through a Private Branch Exchange ("PBX"); and point-to-point or multiuser conferencing support through a data network. Using these internet-protocol ("IP") telephone features, collaborative connections may be established rapidly across private and/or public networks such as intranets and the Internet.

[0050] An overview of different types of functionality that may be provided with the application is illustrated with the flow diagram of Fig. 1. As with all flow diagrams provided herein, the identification of specific functionality within the diagram is not intended to be limiting; other functionality may be provided in addition in some embodiments or some functionality may be omitted in some embodiments. In addition, the ordering of blocks in the flow diagrams is not intended to be limiting since the corresponding functionality may be provided in a variety of different orders in different embodiments.

[0051] At block 104, audio and video conferencing capability is provided by using any of the supported environments to establish a connection among the geographically distributed user computers. For example, the connection may be established with a public switched telephone network ("PSTN"). Telephone connections made through a PSTN may have most calls transmitted digitally except while in a local loop between a particular telephone and a central switching office, where speech from a telephone is usually transmitted in analog format. Digital data from a computer is converted to analog by a modem, with data being converted back to its original form by a receiving modem. Basic telephony call support for modems is supported with the conferencing application using PSTN lines, such as dialing and call termination. In addition, computer-based support may be provided using any suitable command set known to those of skill in the art, such as the Hayes AT command set.

[0052] An ISDN may also be used in establishing the conferencing capability. An

ISDN is a digital service provided by both regional and national telecommunications companies, typically by the same company that supports the PSTN. ISDN may provide greater data-transfer rates, in one embodiment being on the order of 128 kbps, and may establish connections more quickly than PSTN connections. Because ISDN is fully digital, the lengthy process of analog modems, which may take up to about a minute to establish a connection, is not required. ISDN may also provide a plurality of channels, each of which may support voice or digital communications, as contrasted with the single channel provided by PSTN. In addition to increasing data throughput, multiple channels eliminate the need for separate voice and data lines. The digital nature of ISDN also makes it less susceptible to static and noise when compared with analog transmissions, which generally dedicate at least some bandwidth to error correction and retransmission, permitting the ISDN connections to be dedicated substantially entirely to data transmission.

[0053] A PBX is a private telephone switching system connected to a common group of PSTN lines from one or more central switching offices to provide services to a plurality of devices. Some embodiments of the invention use such PBX arrangements in establishing a connection. For example, a telephony server may be used to provide an interface between the PBX and telephony-application program-interface ("TAPI") enabled devices. A local-area- network ("LAN") based server might have multiple connections with a PBX, for instance, with TAPI operations invoked at any associated client and forwarded over the LAN to the server. The server then uses third-party call control between the server and the PBX to implement the client's call-control requests. The server may be connected to a switch using a switch-to-host link. It is also possible for a PBX to be directly connected to the LAN on which the server and associated clients reside. Within these distributed configurations, different subconfigurations may also be used in different embodiments. For instance, personal telephony may be provided to each desktop with the service provider modeling the PBX line associated with the desktop device as a single-line device with one channel; each client computer would then have one line device available. Alternatively, each third-party station may be modeled as a separate-line device to allow applications to control calls on other stations, enabling the conferencing application to control calls on other stations.

[0054] IP telephony may be used in other embodiments to provide the connections, with a device being used to capture audio and/or video signal from a user, such information being compressed and sent to intended receivers over the LAN or a public network. At the receiving end, the signals are restored to their original form and played back for the recipient. IP telephony may be supported by a number of different protocols known to those of skill in the art, including the H.323 protocols promulgated by the International Telecommunications Union ("ITU") and described in ITU Publication H.323, "Packet-based multimedia communications systems," the entire disclosure of which is incorporated herein by reference. [0055] At its most basic level, the H.323 protocol permits users to make point-to- point audio and video phone calls over the Internet. One implementation of this standard in embodiments of the invention also allows voice-only calls to be made to conventional telephones using IP-PSTN gateways, and audio-video calls to be made over the Internet. A call may be placed by the dialing user interface identifying called parties in any of multiple ways. Frequently called users may be added to speed-dial lists. After resolving a caller's identification to the IP address of the computer on which he is available, the dialer makes TAPI calls, which are routed to the H.323 telephony service provider ("TSP"). The service provider then initiates H.323 protocol exchanges to set up the call, with the media service provider associated with the H.323 TSP using audio and video resources available on the computer to connect the caller and party receiving the call in an audio and/or video conference. The conferencing application also includes a capability to listen for incoming H.323 IP telephony calls, to notify the user when such calls are detected, and to accept or reject the calls based on the user's choice.

[0056] In addition the H.323 protocol may incorporate support for placing calls from data networks to the switched circuit PSTN network and vice versa. Such a feature permits a long-distance portion of a connection to be carried on private or public data networks, with the call then being placed onto the switched voice network to bypass long-distance toll charges. For example, a user in a New York field office could call Denver, with the phone call going across a corporate network from the field office to the Denver office, where it would then be switched to a PSTN network to be completed as a local call. This technique may be used to carry audio signals in addition to data, resulting in a significant lowering of long-distance communications bills.

[0057] In some embodiments, the conferencing application may support pass-through firewalls based on simple network address translation. A simple proxy server makes and receives calls between computers separate by firewalls.

[0058] As indicated at block 108 of Fig. 1, the conferencing application may also provide instant-messaging capability. In one embodiment, a messaging engine may be provided that uses a TAPI subsystem for cross messaging, providing a common method for applications and devices to control the underlying communications network. Other functionality that may be provided by the conferencing application includes a locator service directory as indicated at block 112, a file-transfer capability as indicated at block 116, a whiteboarding capability as indicated at block 120, a program-sharing capability as indicated at block 124, and a remote-desktop-sharing capability as indicated at block 128. Each of these functionalities is described in further detail below. The whiteboarding capability may conveniently be used in embodiments of the invention to provide a shared whiteboard for all conference participants, permitting each of the participants to contribute to a collective display, importing features to the display, adding comments to the display, changing features in the display, and the like. The whiteboard is advantageously object-oriented (both vector and ASCII) in some embodiments, rather than pixel-oriented, enabling participants to manipulate the contents by clicking and dragging functions. In addition, a remote pointer or highlighting tool may be used to point out specific contents or sections of shared pages. Such a mechanism provides a productive way for the conference participants to work with documentary materials and to use graphical methods for conveying ideas as part of the conference. In addition to these functions, the conferencing application may include such convenient features as remote-control functionality, do-not-disturb features, automatic and manual silence-detection controls, dynamic network throttling, plug-and-play support and auto detection for voice and video hardware, and the like.

2. Conferencing Application

[0059] In a typical business-usage environment, the conferencing application may be used by employees to connect directly with each other via a local network to establish a whiteboard session to share drawings or other visual information in a conversation. In another application, the conferencing application may be used to place a conference voice call to several coworkers in different geographical locations to discuss the status of a project. All this may be achieved by placing calls through the computers with presence information that minimizes call cost, while application sharing and whiteboard functionality saves time and optimizing communications needs.

[0060] Gateway and gatekeeper functionality may be implemented by providing several usage fields, such as gatekeeper name, account name, and telephone number, in addition to fields for a proxy server and gateway-to-telephone/videoconferencing systems. Calls may be provided on a secure or nonsecure basis, with options for secure calls including data encryption, certificate authentication, and password protection. In some embodiments, audio and video options may be disabled in secure calls. One implementation may also provide a host for the conference with the ability to limit features that participants may enact. For example, meeting hosts may disable the right of anyone to begin any of the functionalities identified in blocks 108 - 128. Similarly, the implementation may permit hosts to make themselves the only participants who can invite or accept others into the meeting, enabling meeting names and passwords.

[0061] Further aspects of the video and audio conferencing functionalities are illustrated with the flow diagram of Fig. 2 A and the exemplary screen view of Fig. 2B. The screen view 228 shows an example of a display that may provided and includes the video stream being generated. The video and/or audio connection is established at block 204 of Fig. 2 A using one of the protocols described in detail above. With the connection established, information, ideas, applications, and the like may be shared at block 208 using the video and/or audio connections. Real-time video images may be sent over the connection as indicated at block 212; in some instances, such images may include instantly viewed items, such as hardware devices, displayed in front of a video collection lens. Options to provide playback control over video may be provided with such features as "pause," "stop," "fast forward," and "rewind." A sensitivity level of a microphone that collects audio data may advantageously be adjusted automatically at block 216 to ensure adequate audio levels for conference participants to hear each other. The conferencing application may permit video window sizes to be change during a session as indicated at block 220. The conferencing application may also include certain optimization techniques for dynamically trading off between faster video performance and better image quality as indicated generally at block 224. Further description of such techniques is provided below.

[0062] Further aspects of the instant-messaging functionalities are illustrated with the flow diagram of Fig. 3A and the exemplary screen view of Fig. 3B. The screen view 324 shows an example of a message that may be received as part of such an instant-messaging functionality and illustrates different fields for receiving and transmitting messages. This functionality is enabled by establishing an instant-messaging connection at block 304 of Fig. 3 A. Text messages typed by one user may be transmitted to one or more other users at block 308. In instances where the messages are transmitted to all conference participants, as indicated at block 312, a "chat" functionality is implemented. In instances where a private message is transmitted to a subset of the conference participants, as indicated at block 316, a "whisper" functionality is implemented. The contents of the chat session may conveniently be recorded by the conferencing application at block 320 to provide a history file for future reference.

[0063] Functions of the locator service directory are illustrated with the flow diagram of Fig. 4A and corresponding exemplary screen view 420 of Fig. 4B. The locator service directory permits users to locate individuals connected to a network and thereby initiate a conferencing session that includes them. Such functionality is centered around a directory that may be configured to identify a list of users currently running the conferencing application. The directory is provided at block 404 of Fig. 4A, enabling a user to receive a selection of another user at block 408. A connection is established between the originating user and the selected user with the conferencing application at block 412, permitting conferencing functions between the two users to be executed. As indicated at block 416, a variety of server transactions may also be performed in some embodiments, such as enabling different directories to be view, creating directory listing of available users, and the like.

[0064] The file-transfer functionality is illustrated further with the flow diagram of Fig. 5 A and corresponding exemplary screen view 520 of Fig. 5B. As indicated at block 504, this functionality permits a file to be sent in the background to conference participants. It is possible in different embodiments for the file to be sent to everyone included in a particular conference or only to selected participants, as indicated at block 508. Each participant may have the ability to accept or reject transferred files at block 512. Data-compression techniques may advantageously be used at block 516 to accelerate file transfers.

[0065] Further aspects of the file-sharing functionality are illustrated with the flow diagram of Fig. 6A and the corresponding exemplary screen view 620 of Fig. 6B. The file- sharing functionality generally enables share programs to be viewed in a frame, as indicated at block 604, a feature that makes it easy to distinguish between shared and local applications on each user's desktop. A user may thus share any program running on one computer with other participants in a conference. Participants may watch as the person sharing the program works, or the person sharing the program can allow program control to other meeting participants. Only the person sharing the program needs to have the program installed on his computer. The shared program frame may also be minimized so that the user may proceed with other functions if (s)he does not need to work in the current conference program.

Similarly, this functionality makes it easy for users to switch between shared programs using the shared-program taskbar. Limitations may be imposed at block 608 by the conference initiator to permit only a single user to work in the shared program at any particular time. Access to the shared program by additional conference participants may be permitted in accordance with an instruction by the originating user at block 612.

[0066] An illustration of the remote-desktop functionality is illustrated with the flow diagram of Fig. 7 A and corresponding exemplary screen view 712 of Fig. 7B. After the remote-desktop functionality has been enabled at block 704, users have the ability to operate a user computer from a remote location, such as by operating an office computer from home or vice versa. A secure connection with a password may be used to access the remote desktop in such configurations at block 712.

[0067] The various implementations described above may include different security features. For example, encryption protocols may be used to encode data exchanged between shared programs, transferred files, instant messages, and whiteboard content. Users may be provided with the ability to specify whether all secure calls are encrypted and secure conferences may be held in which all data are encrypted. User-authentication protocols may be implemented to verify the identity of conference participants by requiring authentication certificates. For instance, a personal certificate issued by an external certifying authority or an intranet certificate server may be required of any or all of the conference participants. Password protections may also be implemented by the originating user required specification of the password by other conference participants to join the conference.

3. Optimization

[0068] Embodiments of the invention use a number of different optimization and bandwidth-management techniques. The average bandwidth use of audio, video, and data among the computers connected for a conference may be intelligently managed on a per- client basis, hi addition, a built-in quality-of-service ("QoS") functionality is advantageously included for network that do not currently provide RSVP and QoS. Such built-in QoS delivers advanced network throttling support while ensuring that conferencing sessions do not impact live network activity. This enables a smooth operation of the separate conferencing components and limits possible consumption of bandwidth resources on the network. [0069] In one embodiment, audio, video, and data subsystems each create streams for network transmission at their own rates. The audio subsystem creates a stream at a fairly constant rate when speech is being sent. The video subsystem may produce a stream at a widely varying rate that depends on motion, quality, and size settings of the video image. The data subsystem may also produce a stream at a widely varying rate that depends on such factors as the use of file transfer, file size, the complexity of a whiteboard session, the complexity of the graphic and update information of shared programs, and the like. In a specific embodiment, the data stream traffic occurs over the secondary UDP protocol to minimize impact on main TCP arteries.

[0070] Bandwidth may be controlled by prioritizing the different streams, with one embodiment giving highest priority to the audio stream, followed by the data stream, and finally by the video stream. During a conference, the system continuously or periodically monitors bandwidth use to provide smooth operation of the applications. The bandwidth use of the audio stream is deducted from the available throughput. The data subsystem is queried for a current average size of its stream, with this value also being deducted from the available throughput. The video subsystem uses the remaining throughput to create a stream of corresponding average size. If no throughput remains, the video subsystem may operate at a minimal rate and may compete with the data subsystem to transmit over the network, hi such an instance, performance may exhibit momentary degradation as flow-control mechanisms engage to decrease the transmission rate of the data subsystem. This might be manifest with clear-sounding audio, functional data conferencing, and with visually useful video quality, even at low bit rates.

[0071] Various optimization techniques used in different embodiments are illustrated with Figs. 8A - 12B. These optimization techniques generally seek to reduce the amount of data transmitted during a conference, thereby maintaining high performance levels for the users. Figs. 8A and 8B respectively provide a flow diagram and set of frame views to illustrate a sequence optimization method. The codec assignments to the video feed are based on a number of parameters. As indicated respectively at blocks 804, 808, and 812, various parameters may be factorized, including the connection bandwidth, the RSVP and QoS provisioning, and the connection speed. Video hardware accelerators are identified at block 816 and requests for changes in frame size and quality are identified at block 820. The resulting codec assignment is implemented at block 824. [0072] Graphical information may be sent as orders in some embodiment. Instead of sending graphical updates as bitmap information exclusively, the conferencing application may instead send the information as the actual graphical commands used by a program to draw information on a user's screen, hi addition, various caching techniques may be used as part of the sequence optimization. Data that comprises a graphical object may be sent only once, with the object then stored in a cache. The next time the object is to be transmitted, a cache identifier may be transmitted instead of the actual graphical data. Maintenance of a queue of outgoing data may also minimize the impact on a local user when a program calls graphical functions faster than the conferencing application can transmit the graphics to remote conference participants. Graphical commands are queued as they are drawn to the screen, and the graphical functions are immediately returned so that the program can continue. An asynchronous process subsequently transmits the graphical command. Changes in the outgoing data queue may also be monitored. When the queue becomes too large, the conferencing application may collect information based on the area of the screen affected by the graphical orders rather than the orders themselves. Subsequently, the necessary information is transmitted collectively.

[0073] A method for color-palette optimization is illustrated with the flow diagram of

Fig. 9 A and corresponding set of frames 924 of Fig. 9B. This method reduces the color depth of insignificant pixels in order to reduce the overall size of a transmitted image by transmitting only pixels relevant to the image integrity. At block 904, global and local palettes are shrunk to reduce the color depth, and the local dependency on the client palette is removed. A global meta-palette is created at block 912, permitting the client palette to be removed at block 916 after a successful merge with a new global palette. The meta-palette is mapped to the new global palette at block 920.

[0074] A frame-reduction method may also be used, as illustrated with the flow diagram of Fig. 1OA and the corresponding set of frames 1020 of Fig. 1OB. The sequence frames are shrunk at block 1004, such as to the smallest possible rectangle. Duplicated pixels are replaced with transparency and alpha channels at block 1008, permitting creation of a complete pixel vector map for the new image at block 1012. Redundant and noncritical frames are marked and removed at block 1016. This method permits the conferencing application to check, prior to adding a new piece of graphic output to the outgoing data queue, for existing output that the new graphic output might obscure. Existing graphic output in the queue that will be obscured by the new graphic output is discarded and the obscured output never gets transmitted. This method also permits the conferencing application to analyze various image frames for redundant information, stripping that redundant information from the transmission.

[0075] A method for motion analysis and frame keying is illustrated with the flow diagram of Fig. 1 IA and the corresponding set of frames 1116 shown in Fig. 1 IB. Excessive motion patterns within a family of related frames are identified at block 1104 of Fig. 1 IA, permitting new anchor frames to be generated at block 1108, based on statistical trends and new frame variances. The intermediate frames on excessive motions may be eliminated at block 1112 so that the size of the transmission is correspondingly reduced.

[0076] A method for optimizing video-sequence transmission is illustrated with the flow diagram of Fig. 12A and the corresponding set of frames 1220 provided in Fig. 12B. This method is related to the method described in connection with Figs. 8A and 8B and results in a dynamic reassignment of codecs based on certain identified parameters. For example, at block 1204, changes in connection bandwidth, RSVP and QoS provisioning, and/or connection speed are identified. At block 1208, video hardware changes are identified. At block 1212, changes in frame size and/or in image quality are identified. Based on these identifications, the dynamic reassignment of codecs is implemented at block 1216.

[0077] The conferencing application described herein may be embodied on a computational device such as illustrated schematically in Fig. 5, which broadly illustrates how individual system elements may be implemented in a separated or more integrated manner. The computational device 500 is shown comprised of hardware elements that are electrically coupled via bus 526. The hardware elements include a processor 502, an input device 504, an output device 506, a storage device 508, a computer-readable storage media reader 510a, a communications system 514, a processing acceleration unit 516 such as a DSP or special-purpose processor, and a memory 518. The computer-readable storage media reader 510a is further connected to a computer-readable storage medium 510b, the combination comprehensively representing remote, local, fixed, and/or removable storage devices plus storage media for temporarily and/or more permanently containing computer- readable information. The communications system 514 may comprise a wired, wireless, modem, and/or other type of interfacing connection and permits data to be exchanged with external devices. [0078] The computational device 500 also comprises software elements, shown as being currently located within working memory 520, including an operating system 524 and other code 522, such as a program designed to implement methods of the invention. It will be apparent to those skilled in the art that substantial variations may be used in accordance with specific requirements. For example, customized hardware might also be used and/or particular elements might be implemented in hardware, software (including portable software, such as applets), or both. Further, connection to other computing devices such as network input/output devices may be employed.

[0079] Having described several embodiments, it will be recognized by those of skill in the art that various modifications, alternative constructions, and equivalents may be used without departing from the spirit of the invention. Accordingly, the above description should not be taken as limiting the scope of the invention, which is defined in the following claims.

Claims

WHAT IS CLAIMED IS:

1. A method ofvideo conferencing a plurality of geographically disperse users, each such user operating a respective one of a plurality of user computers, the method comprising: establishing a network connection among the plurality of user computers; determining a respective connection speed with the network connection independently for each of the plurality of user computers; and transmitting a video signal from one of the user computers over the network connection to others of the user computers, the video signal being transmitted to each of the others of the user computers at the connection speed determined independently for the each of the others of the user computers.

2. The method recited in claim 1 wherein the connection speed for at least two of the plurality of user computers is different.

3. The method recited in claim 1 further comprising: monitoring a bandwidth level over the network connection in real time; and changing the respective connection speed for at least one of the plurality of user computers in accordance with the bandwidth level.

4. The method recited in claim 1 further comprising transmitting an audio signal from the one of the user computers over the network connection to the others of the user computers.

5. The method recited in claim 1 further comprising: establishing an instant-messaging connection among the plurality of user computers; and transmitting an instant message over the instant messaging connection from one of the plurality of user computers to another of the plurality of user computers.

6. The method recited in claim 5 wherein transmitting the instant message comprises transmitting the instant message over the instant messaging connection from the one of the plurality of user computers to a plurality of others of the user computers.

7. The method recited in claim 5 further comprising saving a record of the instant message.

8. The method recited in claim 1 further comprising providing a directory of the plurality of user computers .

9. The method recited in claim 1 further comprising transmitting a data file from the one of the user computers over the network connection to another of the user computers.

10. The method recited in claim 1 further comprising sharing a computer program over the network connection among the plurality of user computers.

11. The method recited in claim 1 further comprising providing access to a desktop of a first of the plurality of user computers over the network connection by a second of the plurality of user computers different from the first of the user computers.

12. The method recited in claim 1 further comprising: transmitting data that comprising a graphical object from the one of the user computers over the network connection to the others of the user computers; caching the graphical object; and sending a cache identifier identifying the graphical object from the one of the user computers over the network connection to the others of the user computers.

13. The method recited in claim 1 further comprising identifying a portion of the video signal that will be obscured by a graphical output, wherein transmitting the video signal comprises transmitting the video signal without the portion of the video signal that will be obscured by the graphical output.

14. The method recited in claim 1 wherein the video signal comprises a sequence of frames, the method further comprising: analyzing the sequence of frames for redundant information; and stripping the redundant information from the transmitted video signal.

15. The method recited in claim 1 wherein the video signal comprises a sequence of frames, each such frame comprising a plurality of color pixels, the method further comprising: analyzing each such frame to identify insignificant pixels; and reducing a color depth of the insignificant pixels.

16. A computer-readable storage medium having a computer-readable program embodied therein for directing operation of a computer system to conference a plurality of geographically disperse users, each such user operating a respective one of a plurality of user computers, wherein the computer-readable program includes: instructions to establish a network connection among the plurality of user computers; instructions to determine a respective connection speed with the network connection independently for each of the plurality of user computers; and instructions to transmit a video signal from one of the user computers over the network connection to others of the user computers, the video signal being transmitted to each of the others of the user computers at the connection speed determined independently for the each of the others of the user computers.

17. The computer-readable storage medium recited in claim 16 wherein the connection speed for at least two of the plurality of user computers is different.

18. The computer-readable storage medium recited in claim 16 wherein the computer-readable program further includes: instructions to monitor a bandwidth level over the network connection in real time; and instructions to change the respective connection speed for at least one of the plurality of user computers in accordance with the bandwidth level.

19. The computer-readable storage medium recited in claim 16 wherein the computer-readable program further includes instructions to transmit an audio signal from the one of the user computers over the network connection to the others of the user computers.

20. The computer-readable storage medium recited in claim 16 wherein the computer-readable program further includes: instructions to establish an instant-messaging connection among the plurality of user computers; and instructions to transmit an instant message over the instant messaging connection from one of the plurality of user computers to another of the plurality of user computers.

21. The computer-readable storage medium recited in claim 20 wherein the instructions to transmit the instant message comprise instructions to transmit the instant message over the instant message connection from the one of the plurality of user computers to a plurality of others of the user computers.

22. The computer-readable storage medium recited in claim 20 wherein the computer-readable program further includes instructions to save a record of the instant message.

23. The computer-readable storage medium recited in claim 16 wherein the computer-readable program further includes instructions to transmit a data file from the one of the user computers over the network connection to another of the user computers.

24. The computer-readable storage medium recited in claim 16 wherein the computer-readable program further includes instructions to share a computer program over the network connection among the plurality of user computers.

25. The computer-readable storage medium recited in claim 16 wherein the computer-readable program further includes instructions to provide access to a desktop of a first of the plurality of user computers over the network connection by a second of the plurality of user computers different from the first of the user computers.

26. The computer-readable storage medium recited in claim 16 wherein the computer-readable program further includes; instructions to transmit data that comprises a graphical object from the one of the user computers over the network connection to the others of the user computers; instructions to cache the graphical object; and instructions to send a cache identifier identifying the graphical object from the one of the user computers over the network connection to the others of the user computers.

27. The computer-readable storage medium recited in claim 16 wherein the computer-readable program further includes instructions to identify a portion of the video signal that will be obscured by a graphical output, wherein the instructions to transmit the video signal comprise instructions to transmit the video signal without the portion of the video signal that will be obscured by the graphical output.

28. The computer-readable storage medium recited in claim 16 wherein: the video signal comprises a sequence of frames; and the computer-readable program further includes: instructions to analyze the sequence of frames for redundant information; and instructions to strip the redundant information from the transmitted video signal.

29. The computer-readable storage medium recited in claim 16 wherein: the video signal comprises a sequence of frames, each such frame comprising a plurality of color pixels; and the computer-readable program further includes: instructions to analyze each such frame to identify insignificant pixels; and instructions to reduce a color depth of the insignificant pixels.

30. A method of managing bandwidth in transmitting multimedia over a network, the method comprising: creating an audio stream for transmission of audio information over the network at an audio rate; creating a video stream for transmission of video information over the network at a video rate; creating a data stream for transmission of data information over the network at a data rate, wherein each of the audio rate, the video rate, and the data rate is determined independently of others of the audio rate, the video rate, and the data rate; and transmitting the audio stream, the video stream, and the data stream collectively over the network and respectively at the audio rate, the video rate, and the data rate according to a priority hierarchy that gives precedence of the audio stream over the video stream and data stream, and gives precedence of the data stream over the video stream.

31. The method recited in claim 30 further comprising assigning a bandwidth for the video stream and data stream substantially equal to a total bandwidth for a connection with the network less a bandwidth used by the audio stream corresponding to the audio rate.

32. The method recited in claim 31 further comprising: determining a current average size of the data stream; assigning a bandwidth for the video stream equal to the bandwidth for the video stream and data stream less the current average size of the data stream; and determining the video rate from the assigned bandwidth for the video stream.

33. The method recited in claim 32 wherein the assigned bandwidth for the video stream is approximately zero, the method further comprising having the data stream and the video stream compete for transmission over the network.

34. The method recited in claim 32 further comprising receiving a specification of a network-connection type to define the total bandwidth.

35. The method recited in claim 34 wherein the network-connection type is selected from the group consisting of a 28.8 kbps modem connection, a 56.6 kbps modem connection, a cable connection, a digital subscriber line ("DSL"), an integrated services digital network ("ISDN") connection, a satellite link, and a local-area-network ("LAN") connection.

36. The method recited in claim 30 wherein the data stream comprises whiteboard information.

37. The method recited in claim 30 wherein the data stream comprises instant-messaging information.

38. The method recited in claim 30 wherein the data stream comprises program-sharing information.

39. A system for transmitting multimedia over a network, the system comprising: an audio subsystem configured to create an audio stream for transmission of audio information over the network at an audio rate; a video subsystem configured to create a video stream for transmission of video information over the network at a video rate; a data subsystem configured to create a data stream form transmission of data information over the network at a data rate, wherein each of the audio rate, the video rate, and the data rate is determined independently of others of the audio rate, the video rate, and the data rate; and a communications system interfaced with the audio subsystem, the video subsystem, and the data subsystem and configured to transmit the audio stream, the video stream, and the data stream collectively over the network and respectively at the audio rate, the video rate, and the data rate according to a priority hierarchy that gives precedence of the audio stream over the video stream and data stream, and gives precedence of the data stream over the video stream.

40. The system recited in claim 39 further comprising a controller in communication with the audio subsystem, the video subsystem, and the data systems and configured to determine the audio rate, the video rate, and the data rate, the controller having instructions to assign a bandwidth for the video stream and data stream substantially equal to a total bandwidth for a connection with the network less a bandwidth used by the audio stream corresponding to the audio rate.

41. The system recited in claim 40 wherein the controller further has: instructions to determine a current average size of the data stream; instructions to assign a bandwidth for the video stream equal to the bandwidth for the video stream and data stream less the current average size of the data stream; and instructions to determine the video rate from the assigned bandwidth for the video stream.

42. The system recited in claim 41 wherein: the assigned bandwidth for the video stream is approximately zero; and the controller further has instructions to have the data stream and the video stream compete for transmission over the network.

43. The system recited in claim 41 wherein the controller further has instructions to receive a specification of a network-connection type to define the total bandwidth.

44. The system recited in claim 43 wherein the network-connection type is selected from the group consisting of a 28.8 kbps modem connection, a 56.6 kbps modem connection, a cable connection, a digital subscriber line ("DSL"), an integrated services digital network ("ISDN") connection, a satellite link, and a local-area-network ("LAN") connection.

45. The system recited in claim 39 wherein the data stream comprises whiteboarding information.

46. The system recited in claim 39 wherein the data stream comprises instant-messaging information.

47. The system recited in claim 39 wherein the data stream comprises program-sharing information.

48. A method of transmitting video data from a first user device to a second user device, the method comprising: receiving the video data as a sequence of frames from the first user device at a video-transmission system; identifying a portion of a first frame in the sequence of frames having information redundant with a portion of a second frame in the sequence of frames; stripping the redundant information from one of the first and second frames; substituting the stripped frame for the one of the first and second frames into a modified sequence of frames; and transmitting the modified sequence of frames with the video-transmission system to the second user device.

49. The method recited in claim 48 further comprising: identifying a portion of a third frame in the sequence of frames having the redundant information; stripping the redundant information from the third frame; and substituting the stripped third frame for the third frame into the modified sequence of frames.

50. The method recited in claim 48 further comprising: identifying a portion of a third frame in the sequence of frames having second information redundant with a second portion of the first frame; stripping the second redundant information from one of the first and third frames; and substituting the stripped one of the first and third frames for the one of the first and third frames into the modified sequence of frames.

51. The method recited in claim 48 wherein stripping the redundant information from the one of the first and second frames comprises replacing pixels of the one of the first and second frames with transparency channels.

52. The method recited in claim 48 further comprising generating the modified sequence of frames by removing a frame from the sequence of frames.

53. The method recited in claim 48 further comprising identifying an excessive-motion pattern within a plural subset of the sequence of frames.

54. The method recited in claim 53 further comprising generating the modified sequence of frames by removing a frame from the subset of the sequence of frames.

55. The method recited in claim 53 further comprising generating a set of anchor frames from a statistical analysis of the subset of the sequence of frames.

56. The method recited in claim 48 further comprising: identifying pixels within a color frame as insignificant to an image represented by the color frame; and generating the modified sequence of frames by reducing a color depth of the identified pixels.

57. The method recited in claim 48 wherein: the redundant information comprises a graphical object; stripping the redundant information from the one of the first and second frames comprises storing the graphical object in a cache with a cache identifier; and transmitting the modified sequence of frames comprises transmitting the one of the first and second frames with the cache identifier.

58. A method of transmitting video data from a first user device to a second user device, the method comprising: receiving the video data as a sequence of frames from the first user device at a video-transmission system; factorizing a connection bandwidth from the video-transmission system to the second user device; factorizing a connection speed from the video-transmission system to the second user device; identifying a request for a change in at least one of a frame size and a frame quality for one of the frames in the sequence of frames; assigning codecs for transmission of the sequence of frames in accordance with the factorized connection bandwidth, factorized connection speed, and identified request; and transmitting the sequence of frames in accordance with the assigned codecs with the video-transmission system to the second user device.

59. The method recited in claim 58 further comprising identifying a video hardware accelerator, wherein assigning the codecs for transmission of the sequence of frames is further in accordance with the identified video hardware accelerator.

60. The method recited in claim 58 further comprising: identifying a change in at least one of the connection bandwidth and the connection speed; and reassigning the codecs in accordance with the identified change.

61. The method recited in claim 58 further comprising: identifying a portion of a first frame in the sequence of frames having information redundant with a portion of a second frame in the sequence of frames; stripping the redundant information from one of the first and second frames; and substituting the stripped from for the one of the first and second frames into the sequence of frames.

62. The method recited in claim 61 wherein stripping the redundant information from the one of the first and second frames comprises replacing pixels of the one of the first and second frames with transparency channels.

63. The method recited in claim 61 further comprising: identifying an excessive-motion pattern within a plural subset of the sequence of frames; and removing a frame from the subset of the sequence of frames.

64. The method recited in claim 61 further comprising: identifying pixels within a color frame as insignificant to an image represented by the color frame; and reducing a color depth of the identified pixels.

65. A method of transmitting video data from a first user device to a second user device, the method comprising: receiving the video data as a sequence of frames from the first user device at a video-transmission system; identifying a graphical object comprised by a first of the frames; storing the identified graphical object in a cache with a cache identifier; transmitting the first of the frames with the video-transmission system to the second user device; identifying the graphical object in a second of the frames different from the first of the frames; stripping the graphical object from the second of the frames; transmitting the stripped second of the frames and the cache identifier with the video-transmission system to the second user device.

66. The method recited in claim 65 wherein stripping the second of the frames comprises replacing pixels of the second of the frames with transparency channels.

67. The method recited in claim 65 further comprising: identifying an excessive-motion pattern within a plural subset of the sequence of frames; and removing a frame from the subset of the sequence of frames.

68. The method recited in claim 65 further comprising: identifying pixels within a color frame as insignificant to an image represented by the color frame; and reducing a color depth of the identified pixels.

69. The method recited in claim 65 further comprising: factorizing a connection bandwidth from the video-transmission system to the second user device; factorizing a connection speed from the video-transmission system to the second user device; identifying a request for a change in at least one of a frame size and a frame quality for one of the frames in the sequence of frames; and assigning codecs for transmission of the sequence of frames in accordance with the factorized connection bandwidth, factorized connection speed, and identified request.

70. A method of transmitting video data from a first user device to a second user device, the method comprising: receiving the video data as a sequence of frames from the first user device at a video-transmission system; factorizing a connection bandwidth from the video-transmission system to the second user device; factorizing a connection speed from the video-transmission system to the second user device; identifying a request for a change in at least one of a frame size and a frame quality for one of the frames in the sequence of frames; identifying a video hardware accelerator; assigning codecs for transmission of the sequence of frames in accordance with the factorized connection bandwidth, the factorized connection speed, the identified request and the identified video hardware accelerator; identifying an excessive-motion pattern within a plural subset of the sequence of frames; removing a frame from the subset of the sequence of frames; identifying pixels within a color frame as insignificant to an image represented by the color frame; reducing a color depth of the identified pixels; identifying a graphical object comprised by a first of the frames; storing the identified graphical object in a cache with a cache identifier; identifying the graphical object in a second of the frames different from the first of the frames; stripping the graphical object from the second of the frames by replacing pixels of the second of the frames with transparency channels; and

transmitting the sequence of frames as modified by the foregoing steps with the cache identifier with the video-transmission system to the second user device.