CA2563866A1 - System for supporting collaborative work - Google Patents

Description

TITLE OF THE INVENTION

System for supporting collaborative work FIELD OF THE INVENTION

[0001] The present invention generally relates to a system for supporting collaborative work. More specifically, but not exclusively, the present invention is concerned with a multimodai/multi-sensorial integrated system that enables a collaborative work environment comprising common tools and applications, whereby geographically-disseminated participants thereof can interact in real-time with each other.

BACKGROUND OF THE INVENTION

[0002] The main purpose of collaborative work and collaborative research is to support cost-effective creative solutions to complex collaborative problems that reduce risk, time and space constraints by means of intensive use of information and communication technologies. Several client/server based products were developed for the collaboration work and are offered on the market. However there is a need for an improved system. For example, there is a need for robustness and reliability of the system, a need for higher performance for real-time collaborative applications and for sharing expensive or non-accessible equipments, a need for higher quality of multimedia, a need for integration of different applications together, a need for adaptability of the system to changes, a need for more visibility of all the participants along with the services and objects to be shared.

[0003] The client/server based systems for collaborative work have a central failure point in the server. If the server fails or becomes unavailable, all the participants are affected. Also, the client/server based systems do not provide sufficient flexibility since the server controls all the information, participants cannot directly share files and documents between each other.
Therefore, Peer-to-Peer (P2P) based systems for collaborative work have been developed to offer more flexibility and robustness.

[0004] Collaborative work is becoming more and more attractive and relevant, especially among the circle of reseachers where real-time applications, specialized devices and expensive equipments are usually not always affordable to everybody. Therefore, the possibility of sharing a virtual laboratory to perform (collaborative) experimentations becomes very interesting.

[0005] Furthermore, current collaborative systems are generally decoupled in their applications. For example, one has to switch from one window to the other or open another window in order to access the browser to surf the Internet or to use any other application. None is known to integrate all the collaborative applications altogether in one single window (or workspace).

[0006] Since technologies change fast, collaborative systems need to be adaptable and allow for adding or incorporating new applications and tools both easily and efficiently.

[0007] Finally, most of the collaborative systems usually indicate only the presence of its participants. No further information concerning, for example, services and objects to be shared, or activity level and history, is provided.
Therefore, there is a need for an improved group awareness functionality in such collaborative systems.

[0008] In order to overcome the above discussed drawbacks, there is a need for an improved system for supporting collaborative work.

SUMMARY OF THE INVENTION

[0009] Therefore, an object of the present invention is to provide a system for supporting collaborative work that is designed and structured and comprises features adapted in view of overcoming the above discussed drawbacks of the current systems.

[0010] The foregoing and other objects, advantages and features of the present invention will become more apparent upon reading of the following non-restrictive description of illustrative embodiments thereof, given by way of example only, with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In the appended drawings:

[0012] Figures 1 a and lb are block diagrams schematically illustrating a collaborative work environment, in accordance with an embodiment of the present invention;

[0013] Figure 2 is a schematic diagram illustrating a system architecture of the collaborative work environment of Figures 1 a and 1 b;

[0014] Figure 3 is a block diagram schematically illustrating a deployment example of the infrastructure of the collaborative work environment of Figures 1 a and 1 b;

[0015] Figure 4 is a schematic diagram illustrating a plurality of devices found in a multimedia room of the collaborative work environment of Figures 1a and 1b;

[0016] Figure 5 is a block diagram schematically illustrating a collaborative work environment comprising a laboratory room, in accordance with another embodiment of the present invention;

[0017] Figure 6 is a flow chart illustrating a method used for integrating dynamically new objects or applications in the collaborative work environment of Figure 5;

[0018] Figure 7 is a block diagram schematically illustrating an application of the collaborative work environment of Figures 1 a and 1 b for real-time telesurgery and telementoring; and

[0019] Figure 8 is an image of a graphical participant interface of the collaborative work environment of Figures 1 a and 1 b.

DETAILED DESCRIPTION

[0020] Non-restrictive illustrative embodiments of the system for supporting collaborative work in accordance with the present invention will now be described.

General description

[0021] The present invention is generally concerned with a system for supporting collaborative work. The system has an architecture that encompasses a plurality of layers, the integration of which results in a multimodal environment having the ability of gathering and integrating multi-sensorial information from a plurality of remote sites, this being described as "telepresence". The system can comprise several collaborative tools and applications, such as instant messaging, chatting, shared whiteboards, a virtual laboratory and a virtual clinic. The system for supporting collaborative work also comprises a broadband fault tolerant network that can connect together several multimedia rooms, intended for sending to other remote sites, receiving from other remote sites and processing multi-sensorial information. The system can have recourse to various video, audio and/or haptic human-machine interfaces;
it can make use of various media, such as text, voice, sound, touch feeling, graphics, gesture and motion, handwriting and shared applications.

[0022] The system for supporting collaborative work encompasses several collaborative tools and applications, all of which being integrated in the same work space. Moreover, its distributed architecture makes it possible to integrate new tools or applications thereinto.

[0023] The system is synchronous, which means that the course of the pertaining events is governed by a common phenomenon, such as a clock.
This synchronicity property allows the participants of a collaborative work session to simultaneously interact with each other; a collaborative work session is characterized by a group of participants using a set of collaborative tools and/or applications during a fixed time frame to manipulate or produce artifacts, i.e. databases, documents, files, etc.

[0024] The system for supporting collaborative work offers a multimedia support to receive or deliver information under various forms, particularly but not exclusively under visual, video, audio and haptic forms.

[0025] Participants can connect to the system either via the Internet, or by any other network, including an intranet or a local network. The system for supporting collaborative work may comprise one or a plurality of multimedia rooms, a multimedia room being (i) defined as a physical site from where one or many participants can access the system, and (ii) potentially equipped with a plurality of interfaces, i.e. video interface, audio interface, haptic interface, etc.

[0026] The system for supporting collaborative work might also comprise a remote facility, the functions of which can be remotely controlled by the participants. As special cases, this remote facility can be a laboratory, or a medical or a surgical room.

[0027] Also, the system for supporting collaborative work might include a specially dedicated multimedia room from where participants can enter the collaborative work session for learning purposes, this being described as "telementoring".

Collaborative work environment

[0028] The collaborative work environment is characterized by an intensive use of the multimodal perceptual processes, the transmission of information via broadband networks, the integration of multi-sensor interfaces to a multiple interaction network for receiving and transmitting both data and control signals, for which the applications are suitable to enhance collaborative work, for example collaborative research. Rather than using a conventional Client/Server architecture, the collaborative environment 10, as illustrated in Figures 1 a and 1 b, has an open overlay self-scaling network architecture based on the distributed Peer-to-Peer (P2P) computing paradigm, which integrates various perceptual modes of information. It enables remote access, as well as the direct sharing of resources, responsibilities and services in synchronouslasynchronous manner among geographically disseminated participants such as 22 (Figure 1 b).

[0029] The collaborative work environment 10 may be used in various contexts that need to support synchronous collaborative work, such as eLearning, eCommerce, eHealth (eDiagnostics, Teleoperation and telementoring), eDesign, eGovenement, eBusiness, eBanking, eProcurement, eNegotiation, etc.

[0030] Figure 1 a is an illustration of a collaborative work environment that comprises six geographically separate sites, numbered site 1 to site 6, linked together by a network 12. A participant 22 (as shown in Figure 1 b) can join a collaborative work session either from a multimedia room 14a-14c or from a work station 16a-16b. In the illustrative example of Figure 1 a, the collaborative work environment 10 also comprises a laboratory 18, the equipments of which can be remotely controlled by any participant 22 at sites 1-4 or 6. The network 12 is any network that links together the six sites and may comprise, for instance, Internet links, intranet links, local networks, private networks, etc.

[0031] Figure lb is a more detailed view of a collaborative work environment 10 comprising sites 1-5, wherein each site 1 to 3 comprises a system 20 for supporting collaborative work. Participants 22 interact with the collaborative work environment 10 either through a multimedia room 14, as in sites 1 and 2, or through a workstation (not shown) as in site 3. In Figure 1 b, site 3 is linked to the network 12 through the Internet. Furthermore, the collaborative work environment 10 includes a laboratory 18 at site 4, which comprises laboratory servers (not shown), and a multicast video bridge 24 for managing all the cameras (not shown) that may be disseminated at the various sites 1-4.

Multi-layer conception of the system for supporting collaborative work

[0032] As illustrated in Figure 2, the system 20 for supporting collaborative work comprises a multi-layer architecture, comprising the following layers:

(i) a participant interface layer 26;
(ii) an application layer 28;
(iii) a middieware layer 30; and (iv) a network layer 32.

[0033] The participant interface layer 26 represents various participant interfaces; it is the collections of graphical objects, i.e.
menus, commands, controls, etc., that enable participants 22 to interact with the system 20 for supporting collaborative work and to use the various, provided collaborative tools. The participant interface 102, as illustrated in Figure 8, may use a relaxed WYSIWIS (What You See Is What I See) interface metaphor and is characterized by optimal management of the displaying space, flexibility in the presentation and ease of access to collaborative services and functionalities.

[0034] The application layer 28 constitutes the bulk of the system 20 for supporting collaborative work and includes the collection of applicative logics. It uses the services of the middleware layer 30 to build collaborative tools, toolkits and applications, in order to satisfy the needs of the participants 22 in terms of communications, data sharing and collaboration. It may comprise the following key components:
= a conference metaphor for grouping functions related to work session management and for creating the illusion of a real meeting conference;
= an instant messaging tool or chat service enabling the exchange of text messages in public, in group or in private;

= an audio/video tool for synchronous multipoint videoconference (for example compliant with the Standard H.323);
= a whiteboard tool which is a collaborative drawing tool using a vector-based representation for offering a drawing space of several pages and a graphic toolbox;
= an eDrive tool or virtual drive enabling file sharing and exchanging (downloading);
= a navigator tool enabling collaborative navigation (or browsing) for example on the Web or on Intranet;
= a visioning tool enabling distance team slide show presentation; and = a distant shared on-line laboratory (virtual laboratory) versatile toolkit grouping a collection of collaborative applications; enabling to undertake experimentations (either in real mode or in simulation mode), and to control remote machines.

[0035] The middleware layer 30 provides the required implementations to develop a distributed system that ensures communication between the participating processes through a duplicated object system (DoS) architecture. The middleware layer 30 also provides a set of services such as coherence, session, collection of specialized network services, all of which allow to facilitate the development and implementation of collaborative functionalities of the application layer 28. Finally, the middieware layer 30 provides a distributed clock that ensures temporal order of information during communication processes.

[0036] The network layer 32 takes in charge all functions related to communication networks, including communication and routing protocols, bandwidth management, and quality of service. This layer 32 is characterized by the following key elements: an automatic message routing mechanism which can allow a peer to join another peer via a third peer, a configurable transport protocol, a hierarchical and cascading message passing model based on multicast groups, and message bundling techniques.

[0037] Furthermore, the application layer 28 allows control sharing (collaborative manipulation commands), result sharing (the results of experimentations), experimentation visualization (video and sound) as well as the control of the devices or cameras used for visualization. Such a tool offers various laboratories options (some of them use DataSocket Transfer Protocol of LabView) and allows the integration of new experimentations without providing an extra development effort and without affecting the other platform's components.

Duplication of objects

[0038] The system 20 for supporting collaborative work may use, for example, the Eterna middieware from Quazal inc. for duplicating objects and ensuring that all participants 22 of a collaborative work session have, on their respective work station, a copy of all objects that are in use. Every copy of a duplicated object is made from the same master copy so as to (i) ensure coherence, (ii) maintain the pertaining functionalities and (iii) control the object's instances.

Decentralized management of applications and distributed architecture

[0039] The system 20 for supporting collaborative work is decentralized and symmetric, since it is based on a P2P architecture, and comprises a broadband network 12 that can connect a plurality of laboratories 18 or multimedia rooms 14 together. The system 20 for supporting collaborative work aggregates various aspects of multimodal perceptual information in order to support collaborative work in different fields of activity.

[0040] The system 20 for supporting collaborative work has a peer-to-peer (P2P) distributed architecture, which means that it does not depend on a central server; rather, the application components and the responsibilities of maintaining network services are distributed across multiple work stations.
This renders the system 20 more robust and tolerant to failure or disconnection of any work station within the collaborative work session. In the occurrence of a failure or disconnection, the duplication master objects will migrate to another station and let the collaborative work session to seamiessly carry on.

[0041] Finally, in Figure 3, an example of deployment of the infrastructure of the system 20 for collaborative work is illustrated. Each site 1-5 is connected to the network 12 with GigaEthernet links 40 and via a switch 42.
Of course, other types of connections can be used and would be obvious to a person of ordinary skill in the art. For example, site 4 is connected to the switch

42 via an optical carrier 44. Also, a testbed network 46 can be present and connected to the network 12 for monitoring and performance analysis purposes.
Multimedia rooms [0042] In reference with Figure 4, a multimedia room 14 is illustrated.
The multimedia room 14 provides an adapted versatile environment that replicates real collaborative situations and can also provide a general-purpose virtual laboratory facility for a large experimentation of engineering problems in the context of collaborative research, for instance by using the LabVIEWTM
software. The multimedia room 14 is equipped with high-quality multi-media devices.

[0043] So, turning now to Figure 4, the multimedia room 14 will be described. The multimedia room 14 can comprise and integrate into the collaborative environment 10 a plurality of features, the information of all of which can be accessed by the participants 22. The features can comprise, amongst others, whiteboards 50, smart boards (handwritten recognition acquisition and understanding) 52, screens 54, cameras 56, haptic devices 58, databases and applications, etc. Those features are schematically illustrated in Figure 4. Multimedia rooms 14 are equipped with high-quality accessories and devices enabling intensive computing, high-definition (HD) visualization, 2D
and 3D image processing (recognition), group communication, and videoconference services for multiple participants 22. Thanks to the high degree of integration of such intelligent interactive devices, such multimedia rooms 14 are innovative in the field of collaborative work. For instance, during teleconferencing, intelligent cameras 56 are autonomously guided to track the person 60 who is currently speaking by putting him continuously on its vision field. The speaker 60 may create his drawing directly on a shared whiteboard 50 by using the smart board 52 which provides handwriting recognition acquisition and understanding. He can also use a conventional whiteboard 50 to write notes, which can be captured by the camera 56, which has generally a high resolution given by CDD
(Charged Coupled Device) matrices. Furthermore, he can also undertake a demo in which he manipulates a remote machine (not shown), such as a robot for example, by using haptic devices 58. By means of the graphical participant interface 102 (Figure 8) and a private camera 62 for each participant 22 in this multimedia room 14, distant participants 22 (from other multimedia rooms 14) to the meetings can communicate and interact with the speaker 60 as well as with each other. Also, a sound system composed of a distributed microphone network allows for accurate and smooth acquisition of voice. Therefore, there is provided a collection of speakers enabling high-quality of sound diffusion.

[0044] The graphical participant interface 102 may also be projected to produce a wide main screen 54, having a large HD format, by means of a projector assembly 64. In addition, there is a digitization center (not shown in Figure 4) for storing, distributing and sharing digital data.

[0045] The above description is a non-restrictive example of the multiple possibilities of a multimedia room 14.

Haptic devices and robots

[0046] The system may comprise one or several haptic devices 58. A
haptic device 58 is a technology which interfaces with the participant 22 through the sense of touch. By enabling a force-feedback output, a haptic device 58 makes possible to actuate teleoperators. The haptic devices 58 may be coupled to stereoscopic spectacles in order to perceive depth of field and, in this manner, adding an extra information parameter that helps the participant 22 to properly perform a tele-manipulation. Two examples of use of haptic devices 58 are (i) conducting an experiment in a remote laboratory 18 and (ii) performing a surgery on a remote patient. For instance, a haptic device 58 may comprise Phantom DesktopTM used in connection with two pantins and a universal joint wrist, e.g. a Phantom 1.5AT"" and a Freedom 6STM
. . Generally speaking, haptic devices 58 allow for controlling remote machines or robots. In the example of a remote laboratory 18, the haptic devices 58 enable to start experimentations using different robots or to perform simulations of different machines. In the case of a virtual clinic, the haptic devices 58 allow a surgeon to manipulate surgical robots and instruments.

Virtual laboratory

[0047] The system may encompass a virtual laboratory 18, which comprises a set of collaborative applications for performing experiments on remote equipments that are actuated by teleoperators. The virtual laboratory allows to remotely perform interactive experiments, possibly by using haptic devices 58. Moreover, it is also suitable for distant learning, as it enables many geographically disseminated participants 22 to simultaneously conduct an experiment, to get and share the results thereof and, possibly, to visualize in real-time a 3D video of the experiment. The virtual laboratory 18 also offers the possibility to simulate an experiment from relevant models and databases. Of course, the virtual laboratory 18 can also be equipped with HD visualization and high quality sound systems such as those found in the multimedia rooms 14.

[0048] Turning to Figure 5, a collaborative environment comprising a laboratory 18 is shown. A plurality of sites 1-6 as well as the laboratory 18 are connected to the network 12. The network 12 can be a private network, shared by a group of universities and linking them together, for example. The laboratory 18 comprises a virtual laboratory application 70 and a virtual laboratory application using LabView 72. The laboratory application 70 is completely integrated to the system 20 for supporting collaborative work and includes a machine 76 or a robot, connected to a data acquisition device 78 or card in order to monitor and save the results from experimentations. Those results are accessible at site 6 and can be shared with any participant 22 involved and connected to a collaborative work session. The laboratory 18 also comprises a virtual laboratory application using LabView 72. This laboratory application comprises a LabView server 74, linked to an acquisition device 78 or card, which is connected to a machine 76 to obtain data from that machine 76. Also, a machine 76 can be directly connected to the LabView server 74 for data processing. Finally, the LabView server 74 is linked to the network 12 so that it is accessible to all the participants 22 located at sites 1-6. Therefore, the disseminated participants 22 can perform on-line remote experimentations and share their results.

[0049] The flexibility needed for performing on-line remote experiments is assured by the ability to integrate dynamically new objects and applications. For example, the system 20 for supporting collaborative work allows for adding new virtual laboratory applications 70 or 72 during a collaborative work session. As illustrated in Figure 6, a participant 22 can create a new virtual laboratory duplicated object (step 80) which contains description information (name, type, version, etc.) and the address of the laboratory application server 74 (Labview server for example). Then, the system 20 for supporting collaborative work publishes the duplicated object over Eterna (step 82) and activates remotely the laboratory application 72 within Labview (step 84). Then the new laboratory application 72 is opened within the shared workspace dedicated to the virtual laboratory tool(s) (step 86). Thus, a new virtual laboratory application 72 is developed and deployed within the Labview development environment without affecting the system 20 for supporting collaborative work. In this way, the developers can produce new shared remote control applications, which are called virtual laboratory, in accordance to their particular needs and use in a collaborative work session.

Telemedicine

[0050] One embodiment of the present invention is its use for performing telemedicine. More specifically, the present invention can be used for performing real-time medical or surgical interventions at distance, for diagnosis or curative purposes, whether or not in a telementoring context.
Figure 7 illustrates a telesurgery environment. In this example, a surgeon 90 is located in a medically-adapted multimedia room 14 at site 2 while a distant patient 92 is located in a remote, adapted operating room 94 at site 1. One or several assistants 96 may also be present in the adapted operating room 94 to assist the surgeon 90 in his work. The medically-adapted multimedia room 14 and the adapted operating room 94 are both equipped, particularly but not exclusively, with two-way cameras 56 and haptic devices 58. The cameras 56 allow (i) the surgeon 90 and the assistants 96 to see each other and (ii) the surgeon 90 to see the distant patient 92, especially the area on which the surgery is performed. The haptic devices 58 allow the surgeon 90 to receive a touch feeling feedback from the contact of the instruments he manipulates. For example, this feedback to the surgeon 90 can be the pressure exerted by the patient's body on an instrument the surgeon 90 manipulates thereon or therein.
More specifically, the surgeon 90 performs the surgery on the distant patient via a remote robot 98 placed in the adapted operating room 94. To accomplish this task, the surgeon 90 uses a virtual laboratory 18 comprising laboratory applications 72 and the server 74 from LabView, and the surgeon 90 interacts with the robot manipulator with embedded camera 98 via the haptic device 58.
The camera embedded in the robot manipulator 98 captures a detailed video of the operation which is displayed in high resolution in a main projection screen such as the screen 54 in Figure 4. Audio/video tools facilitate the communications between the surgeon 90 and the assistant(s) 96. Furthermore, resident students 100, located at remote site 3, may participate to the operation session for the purpose of telementoring and learning. In another embodiment of the invention, several surgeons 90 may collaborate together and teleoperate on the same patient 92. The dedicated broadband network 12 guaranties secure real-time transmission and an acceptable latency for this kind of task.
High standards of communication security and fast response which are needed in a telesurgery context are provided.

Group awareness

[0051] The system 20 for supporting collaborative work features the group awareness functionality. Group awareness is defined as enabling each participant 22 to a collaborative work session to perceive the actions, activities and states of the other participants 22. Awareness is also sometimes referred to as "ubiquity". Group awareness management is made possible by functionalities through a dedicated space for awareness information comprising parameters such as presence information, identification, localisation, availability, activity level and history. As illustrated in Figure 8, which shows a graphical interface 102 to the collaborative work system 20, the space 104 displays the name of all the currently connected participants 22, along with their respective activities and tools, which are available to be shared. Furthermore, the space 106 allows each participant 22 to know who wants to share what.

Inheritance

[0052] Another interesting feature of the system 20 is the inheritance functionality. For example, inheritance allows a late-comer participant 22 to join and inherit all shared objects or collaborative objects that have been created or produced during an on-going collaborative work session. Moreover, the collaborative work session is not affected if its initiator disconnects therefrom.

[0053] In one embodiment, the inheritance property is implemented by using the Eterna middleware of Quazal inc., which was originally dedicated to the development of multi-participant games. The middleware 30 provides a set of specialized services, such as coherence synchronization, session and implementation, which facilitate the development and implementation of collaborative functionalities for the application layer 28. The middleware layer 30 makes it possible for the system 20 to inherit the performance and properties thereof, such as object duplication and migration, duplicated spaces, data extrapolation, remote procedure calls (RPC), etc. Moreover, the use of the middieware layer 30 allows the system 20 for supporting collaborative work to inherit the following characteristics: a descriptive approach with a high level of abstraction, fault tolerance, reliability, flexibility (supporting P2P, C/S, and Hybrid architectures), interoperability (it can easily interoperate with other middleware or software components or systems), data integrity (via data encryption), use of compression, ability to evolve (1024 available classes and millions instantiations) and ability to perform load balancing.

[0054] Figure 8 shows the graphical interface 102 of the system 20 for collaborative work as seen by a participant 22. A plurality of applications are integrated into one single window or workspace. Indeed, the space 104 allows for displaying all the connected and disseminated participants 22. The space 106 shows the available sharing documents from different participants 22. The space 108 allows for chatting and instant messaging to all the connected participants 22. The space 110 is a whiteboard for drawing, writing, taking notes, calculating, etc. The space 112 allows participants 22 to browse the web and do searches. The space 114 and 116 allows for displaying pictures and videos from different machines in remote laboratories 18. They also allow for displaying the results of experimentations via graphs or tables. Those results can be manipulated and processed afterwards by each participant 22. Finally a menu bar 118 shows all the different commands available to participants 22.

[0055] Although the present invention has been described hereinabove by way of non-restrictive, illustrative embodiments thereof, these embodiments can be modified without departing from the spirit and nature of the subject invention. -

Claims