CN102857531B - A kind of long-distance interactive system based on cloud computing - Google Patents
A kind of long-distance interactive system based on cloud computing Download PDFInfo
- Publication number
- CN102857531B CN102857531B CN201110183361.0A CN201110183361A CN102857531B CN 102857531 B CN102857531 B CN 102857531B CN 201110183361 A CN201110183361 A CN 201110183361A CN 102857531 B CN102857531 B CN 102857531B
- Authority
- CN
- China
- Prior art keywords
- user
- node
- application program
- microenvironment
- terminal device
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000002452 interceptive effect Effects 0.000 title claims abstract description 40
- 230000006870 function Effects 0.000 claims description 31
- 238000003491 array Methods 0.000 claims description 7
- 238000012545 processing Methods 0.000 claims description 7
- 230000006835 compression Effects 0.000 claims description 4
- 238000007906 compression Methods 0.000 claims description 4
- 238000012544 monitoring process Methods 0.000 claims description 4
- 238000001514 detection method Methods 0.000 claims description 3
- 238000000034 method Methods 0.000 description 30
- 230000008569 process Effects 0.000 description 21
- 238000003860 storage Methods 0.000 description 21
- 230000003993 interaction Effects 0.000 description 13
- 238000010586 diagram Methods 0.000 description 12
- 230000005540 biological transmission Effects 0.000 description 9
- 238000004364 calculation method Methods 0.000 description 8
- 238000005516 engineering process Methods 0.000 description 7
- 230000001360 synchronised effect Effects 0.000 description 6
- 241001362551 Samba Species 0.000 description 4
- 230000009471 action Effects 0.000 description 4
- 238000009795 derivation Methods 0.000 description 4
- 238000005096 rolling process Methods 0.000 description 4
- 230000000153 supplemental effect Effects 0.000 description 4
- 230000006399 behavior Effects 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 229910002056 binary alloy Inorganic materials 0.000 description 2
- 230000003139 buffering effect Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 230000008676 import Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000005267 amalgamation Methods 0.000 description 1
- 230000003321 amplification Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000004069 differentiation Effects 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
- 238000002407 reforming Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Landscapes
- Information Transfer Between Computers (AREA)
Abstract
The present invention relates to a kind of long-distance interactive system based on cloud computing, it comprises terminal device, web front-end, security gateway, and described security gateway comprises one or more service nodes; First terminal device accesses described front end, by obtaining the authority that connects a certain service node in security gateway after checking; Terminal device uses this authority to be connected with specific service node in cloud computing system subsequently; In described security gateway, the interface of all application programs and audio frequency output are converted into Audio and Video stream, and a relative additional information stream, and pass to described terminal device by internet. Can reach the cost that not only reduces hardware resource by this system, realize in many ways mutual object simultaneously.
Description
Technical field
The present invention relates to a kind of long-distance interactive system based on cloud computing, relate to multiple technical fields such as computer, network, Streaming Media.
Background technology
Cloud computing refers to payment and the use pattern of IT infrastructure, refers to obtain resource requirement by network in the mode of as required, easily expanding; Broad sense cloud computing refers to payment and the use pattern of service, refers to obtain required service by network in the mode of as required, easily expanding. It is relevant with software, internet that this service can be IT, also other services. Cloud computing (CloudComputing) is the product that the traditional computers such as grid computing (GridComputing), Distributed Calculation (DistributedComputing), parallel computation (ParallelComputing), effectiveness calculating (UtilityComputing), the network storage (NetworkStorageTechnologies), virtual (Virtualization), load balancing (LoadBalance) and network technical development merge.
A problem carrying out interactive program in cloud computing system is, interactive program may be designed to only be undertaken alternately by local interaction devices on the computer that it moved by user. That is user's mutual action depends on the interactive device (as keyboard, mouse, display etc.) on local computer. This means in the specific moment, in cloud system, a computing node can only provide service for unique user at most.
For large-scale cloud computing system, the cost of its total construction cost and single computing node is closely related. Normally, as shown in Figure 1, the P/C of hardware device increases along with the raising of performance than within the specific limits, and meanwhile, higher performance also contributes to computing node can bear the stricter performance requirement of program. But high-performance simultaneously also means the raising of cost, and there are two problems: 1. for the computing node of the application program of interactive behavior, the use of the computing capability of program to node is uneven in time conventionally. For example, in the application of a three-dimensional modeling, the process of user input data (low computational resource takies) may account for the major part of service time and render process (high computational resource takies) only accounts for the short time, and therefore in most time, computational resource is idle; 2. the computational resource of interactive application uses the actual upper limit that does not reach hardware performance, and this is especially obvious in interactive PC game. Because PC game is often designed to only need consume the computational resource of certain limit in the time reaching certain user's experience requirements, and often limit the consumption of resource in order to adapt to more different model hardware device.
Therefore in existing cloud computing service system, the user of interactive PC game experiences and is limited in certain level, the therefore also just consumption of the hardware resource of restriction, and final result is, purchases the hardware device cost that more computational resources pay and is wasted.
In addition, single computing node also has extra fixed cost, for example bus apparatus (mainboard), and memory device, power supply and shell, and in order to place space cost of computing node etc. Can see, in the time that the each user's of cloud computing system cost of serving is relevant with the cost of computing node, be very limited by reducing user's cost of serving that the method for hardware cost of single computing node can reduce.
Apparently, effectively the decline scheme of single user cost is that to make single computing node can be multiple users' services simultaneously. But the interactive device that as previously mentioned, interactive application need to be monopolized on computing node moves. Virtual (Virtualization) technology provides a possible implementation. But virtualized problem is to use this technology that the performance of bringing the expense of extra hardware resource, particularly graphic process unit (GPU) is tended to suffer significant loss, and have the defect in hardware supported. For to provide windows application program for main cloud computing service system, Intel Virtualization Technology will increase the windows operating system quantity of moving on computing node and then improve mandate expense; The combination of Intel Virtualization Technology itself and windows operating system is very unripe yet.
Summary of the invention
In order to solve the problems of the technologies described above, the invention provides a long-distance interactive system based on cloud computing, it comprises terminal device, web front-end, security gateway, is characterized in that: described security gateway comprises one or more service nodes; First terminal device accesses described front end, by obtaining the authority that connects a certain service node in security gateway after checking; Terminal device uses this authority to be connected with specific service node in cloud computing system subsequently; In described security gateway, the interface of all application programs and audio frequency output are converted into Audio and Video stream, and a relative additional information stream, and pass to described terminal device by internet. Further, enough competent decoding SD or H.264 video flowings of high definition of described terminal device.
Further, described service node further comprises a processing unit, between application program and system API or network, it is characterized in that: comprise a function address monitoring modular, for detection of the function address of application program in internal memory, and a function address redirection module, the function address of storing for changing the internal memory of application program, is redirected function address; And multiple interfaces, for realization and computer hardware and API between communicate by letter.
In addition, described terminal device can be realized in kinds of platform, comprises computer, TV set-top box, mobile terminal device etc.
Described service node is by the data of jointly accessing on same memory node, and the excessive data that each user is produced stores on single memory node.
Described multiple service node forms a node array, and multiple node arrays are shared the data of a memory node reading out data and recording user generation, and user is by one in multiple node arrays described in the data selection such as network delay and bandwidth.
Further, described processing unit is connected with mixed-media network modules mixed-media with the video/audio compression module on this service node, thus by the interface of application program with play up data transaction and become video/audio stream, be sent to other terminal on internet through mixed-media network modules mixed-media.
Brief description of the drawings
Fig. 1 is the P/C ratio schematic diagram of the hardware device of extensive cloud computing system;
Fig. 2 is the framework of the long-distance interactive system based on cloud computing node;
Fig. 3 is the module map of the long-distance interactive system based on cloud computing node;
Fig. 4 is the hierarchy chart of the long-distance interactive system based on cloud computing node;
Fig. 5 is the schematic diagram of a kind of embodiment of microenvironment;
Fig. 6 is redirected schematic diagram under microenvironment;
Fig. 7 is the mutual schematic diagram of mouse message in the present invention;
Fig. 8 is the schematic diagram that in the present invention, picture is carried out pipeline processes;
Fig. 9 is the schematic diagram of the long-distance interactive system access network file of cloud computing node;
Figure 10 is the schematic diagram that interactive information is distributed between different user;
Figure 11 is the schematic diagram of the present invention about coding and decoding video;
Figure 12 is the schematic diagram of the present invention's " displaying wall ";
Figure 13 is the schematic diagram of the processing mode of the present invention to the mouse action receiving in limited rectangular extent;
Figure 14 is the performed workflow diagram of user that uses cloud computing remote interaction procedure service;
Figure 15 is the deployment diagram of distributed calculation services node array.
Detailed description of the invention
Fig. 2 and Fig. 3 have listed the framework of a long-distance interactive system based on cloud computing node. It comprises terminal device, web front-end, security gateway. Described terminal device can be PC, and any computing capabilitys such as mobile terminal or TV set-top box are competent decoding SD or the H.264 equipment of video flowing of high definition enough. Described security gateway comprises one or more service nodes, and first terminal device accesses the front end of the system of cloud computing, by obtaining the authority that connects a certain service node in security gateway after checking; Equipment uses this authority to be connected with specific service node in cloud computing system subsequently.
By one specifically by hardware, software, or it realizes above-mentioned functions in conjunction with the platform of realizing or environment (being called for short " microenvironment " below), taking windows operating system as example, can be virtual its use to user interactive system of other windows application program, thereby make the computing node of single operation windows operating system can serve multiple users simultaneously. With respect to other schemes, microenvironment has following advantage:
Microenvironment makes multiple users utilize the resource (including the software resource of single operating itself) of computing node to become possibility simultaneously.
The application program being performed is with almost primary mode access hardware resource (particularly graphic process unit resource).
Microenvironment does not change the binary code of performed application program.
In the present invention, in service node, a microenvironment is the application service that user starts appointment, described service can be including application program all services, the interface of application program and audio frequency output are converted to Audio and Video stream by microenvironment, and a very little additional information stream of (optionally) data volume passes to terminal by internet. User receives after above-mentioned Audio and Video stream and (optionally) additional information stream on use terminal device, by the interface hardware (keyboard of equipment, mouse, remote controller, game console etc.) send and control the microenvironment of data to calculation services node, this is controlled the user interactive data that data receive as moved application program by microenvironment. Thereby complete a mutual process.
The realization of microenvironment is at operating system level, between application program and system API or network, those skilled in the art is to be understood that, although used " microenvironment " this term here, but should be by its extend geographic solution a functional structure body can being realized by hardware, software, an or processing unit. Consider an application program arbitrarily, for computer on hardware interactive device carry out alternately, the interface accessing hardware driving that application program must provide by operating system, and then control hardware. (as Fig. 4 left) microenvironment comprises a function address monitoring modular, for detection of the function address variable of application program in internal memory, an and function address redirection module, the function address variable of storing for changing the internal memory of application program, and an address preservation module, for preserving the function address being again directed; And comprise multiple interfaces, for realization and computer hardware and windows application programming interfaces between communicate by letter, for example program interface, picture is played up, the input of audio frequency output and control information (keyboard, the information such as mouse).
By changing the function address variable of storing in the internal memory of application program, make application program be redirected to the access of operating system interface interface that microenvironment provides and no longer contact (as right-hand in Fig. 4) with hardware layer or the hardware driving of actual PC system. For application program, microenvironment provides interface and the respondent behavior identical with operating system. Therefore in microenvironment, carry out and can't affect its normal workflow. Microenvironment has obtained application program and user's interaction data, comprises program interface, and picture is played up, the input of audio frequency output and control information (keyboard, the information such as mouse). Microenvironment can realize in cloud computing system, and be connected with mixed-media network modules mixed-media with video/audio compression module, thereby by the interface of application program with play up data transaction and become video/audio stream, be sent to other terminal (user of cloud computing service) on internet through mixed-media network modules mixed-media. Equally, the customer interaction information of terminal arrives after microenvironment by internet, the customer interaction information that has replaced the interactive device on former cause computing node to send conveys to application program, final realization by internet in the upper object of carrying out interactive application of remote computer (cloud computing service node).
Fig. 5 has shown the embodiment that the another kind of microenvironment is general. Taking windows operating system as example, the realization mechanism of microenvironment is in conjunction with the distinctive dynamic link library of windows system (DLL) mode of operation and process space mechanism independently. In operating system, microenvironment can further comprise multiple processes, and described process is connected with client, carries out specific request by client-requested, and described request can be the data of application program or transmission or reception. Microenvironment is written into internal memory by above-mentioned request subsequently, before the actual execution of above-mentioned request, microenvironment utilizes the working method of dynamic link library that redirection module (with the form of DLL) is loaded to the process space into application program, and in the memory headroom of program process, revises the loaded run time version of application program. Or in order to save the resource of PC hardware, microenvironment directly utilizes the redirected hardware module of self to revise the function address variable comprising in above-mentioned request, to change the funcall entry position of operating system application programming interfaces API of function. After application program is carried out, when routine call operating system API, the function address being modified is jumped to the function address that redirection module is stored by redirection module. Redirection module is connected by specific Communication tunnel with the microenvironment main body of moving in other process, and interactive content when program is carried out outputs in microenvironment process, finally sends to client by network after treatment.
Fig. 6 has shown typical workflow when system API is called after amendment in the memory headroom of program process: by amendment internal storage data, the machine instruction that API entrance function starts is most modified to a jump instruction, and the interface function entrance that provides of redirection module that microenvironment loads is provided. Thereby first application program is redirected in microenvironment redirection module to the use of system API. All parameters when the interface function of microenvironment has received application call system API, after this can be used one of following manner to obtain the interactive information of application program:
1) parameter of importing into when application call system API has just comprised required interactive information, for example location to mouse pointer, the path parameter of file access etc.
2) microenvironment is used the parameter obtaining, and replaces the required system API calling of its script of application call. As shown in Figure 5, in microenvironment, can first carry out the instruction that is modified the internal memory of part in former API, from former API, internal memory starts to carry out until api function finishes after being modified part again, complete the invoked procedure of an API, and in the result data returning, obtain the interactive information of application program, such as coloured program interface picture etc. from API Calls. These information microenvironment interface functions finish, and give back before application program, can first be output to the derivation module of microenvironment.
3) application program is often relevant with hardware device to the use of system API, and specific hardware (or hardware driving) can be realized and is directly connected with derivation module by microenvironment. By the parameter of amendment API Calls, microenvironment can be handed to this by application program to the access of certain hardware device and derive specific hardware of being connected of module (or driving) and carry out, thus the information that the program that is applied is exported from hardware device. The derivation of for example audio-frequency information. Microenvironment also has more effect by application program to the redirected function of hardware access, will set forth in the back.
The importing module workflow of microenvironment and derive module class seemingly: microenvironment interface function obtains from importing the information that needs module, and parameter or the return of value etc. imported into during by application call API pass to application program.
With further reference to accompanying drawing 3, in microenvironment, the mutual all information that comprise are processed separately, and finally by different information flows, be sent to client-side program, in this embodiment, microenvironment includes but not limited to lower module: sound is derived module, interface is derived module, other information derivation modules and input and imported module. Taking PC game as example, it is video flowing output that the picture of game is derived module converts by interface, and it is audio stream output that background music and effect sound sound are derived module converts. In addition in order to realize the function of user interactions, some extra information such as in application program position and the pattern etc. of cursor also by information derive module from one independently channel of information flow export. The interactive information receiving at subscription client is sent to input and imports module, and is converted to the content such as mouse and Keyboard Message of the actual acquisition of application program.
Further, the interface that microenvironment redirection module comprises sequence of operations system call, includes but not limited to Types Below:
Graphical display interface, comprises WindowsGDI and DirectX. In recent years, the PC game of the overwhelming majority utilizes DirectX interface to play up pictures whole in its implementation.
Audio frequency play interface, for example DirectSound or be windows media device interface (MultimediaDeviceAPI).
Input/output interface, comprises DirectInput and keyboard/mouse message interface.
Storage and application context interface, comprise file access API, registry access API, environmental variance and kernel objects etc. The collision problem that can avoid resource to use when these interfaces that microenvironment is redirected are carried out multiple identical application programs under same operating system. The all right record cell of microenvironment for recording and retaining these contextual informations, thereby is prepared from current context information and executing for application program next time.
Because can replacing application program, microenvironment carries out calling of system API. Therefore in the time that application program need to be used hardware resource, microenvironment does not need to simulate all Hardware Response Delay. In actual system, in microenvironment, each request of deriving module application programs is being carried out after necessary processing, conventionally all application program can be come the directed time original position of the access of hardware resource (by actual system function interface). In fact application program is compared with original working method in microenvironment, only has more extra operation required in two step jump instructions and microenvironment, and this compares the expense with much less with Intel Virtualization Technology, can think and be in close proximity to primary working method.
For example, although Windows operating system can support many group hardware resources to work, the CPU of multi-core and many GPU simultaneously. But it is upper that operating system is still confined to CPU to the Automatic dispatching of resource, and can't Automatic dispatching and the computing capability of distributing other hardware (particularly GPU). Even for example multiple GPU of computing node deploy, general application program is often only used in operating system the equipment that is designated as " main GPU " to carry out playing up of 3D picture. Even if application program can have access to the information of whole GPU equipment and can select certain GPU equipment by user's instruction, in the time moving multiple application program on computing node, for each user, it is also impossible selecting suitable GPU equipment.
Utilize microenvironment interface derive module to DirectX interface be redirected, microenvironment can be the GPU resource that certain specific application assigned is suitable at multiple GPU equipment rooms of multiple computing nodes (or service node). select the current GPU equipment with enough idle computing capabilitys, and give different resource occupation to these GPU equipment and complete a specific calculation. for this reason, in microenvironment, further comprise a logging modle, for recording the service condition of current all application programs of carrying out therein to GPU resource, further, it also comprises a supervision module, for monitoring the duty of GPU, to be the next applicable GPU equipment of application assigned of carrying out, that is to say, the GPU of each computing node (or service node) may be assigned with and participate in the calculating irrelevant with this computing node, an embodiment of described supervision module is based on reading the system interface of standard and driving the real-time parameter of interface to monitor its duty.
The redirected output that sound is derived module application programs audio frequency is implemented in hardware driving aspect. Microenvironment before intervention the form with hardware driving realize multiple (virtual) audio frequency and play up and playback equipment. After application program is realized by virtual unit calling of audio frequency play interface, boil down to audio stream outputs to long-distance user's computer by internet. Under normal circumstances, when multiple application programs are moved simultaneously, the output of the audio frequency of each program is by mixed and be output in the audio frequency apparatus of system simultaneously. On the other hand, microenvironment can be one of them of these virtual audio device of application assigned of wherein each execution, makes the audio frequency output of different application be become independently audio stream and the voice-grade channel by different microenvironments transmits.
Outside picture and audio content, application program for user correctly mutual required out of Memory also to derive module by out of Memory collected and be sent to client by accessory channel. For example, in current application program, the physical location of mouse pointer need to be sent to client, to show exactly its position and out of Memory on client screen.
Mouse pointer is the most common element in interactively windows program. As seen from Figure 7, when mutual with long-range application program, any the interactive operation of mouse pointer is all needed to obtain by Internet Transmission, after response, could show back subscription client. This means that user is in the time of client rolling mouse pointer, will observe one and the suitable delay of network delay (TTL). We observe, and user is highstrung to the delay of rolling mouse pointer; But relatively,, in the time clicking on screen element-specific, user's the tolerance to postponing is much bigger. Therefore, in microenvironment, directly the information of cursor is not merged in program interface and transmitted as video stream. But transmit the information of mouse pointer and directly draw out in client by supplemental stream. In the time of user's rolling mouse pointer, first client terminal local will respond the event that mouse moves and repaint mouse pointer, and this event is sent to the application program of long-range execution by control information simultaneously. Noticeable delay when this time strategy has been eliminated rolling mouse pointer. The movement that the delay that network delay causes occurs over just mouse pointer causes program interface to occur in the process of specific response. And as aforementioned, this delay can be tolerated by most users.
Specify the screen resolution that specific remote application may have relatively high expectations and client screen possibly cannot meet this requirement, this is particularly common on the equipment such as mobile terminal. In the case, and the interface Zoom module of deriving module coupling the output interface of application program can be narrowed down to the acceptable scope of client screen and be output as video flowing. After interface is reduced, on interface, original Partial Elements, particularly word content etc., may lose its readability owing to dwindling. Now, an extra application program will start in microenvironment, the interface of this routine access interface exporter, the picture (keeping its original size) of ad-hoc location on program interface (conventionally to be a little square region centered by mouse pointer position pointed) is marked separately, utilize video encoder to be converted into a less video flowing, be transferred to client (as Fig. 7) by supplemental stream, not at normal video flowing part coding; Normal video flowing can be for the whole image of encoding. This video flowing after client decoding, reappear module by its with remote application on identical resolution ratio show, and be superimposed on shown (reduced) the program interface picture of client screen. User can wait operation by moving of mouse pointer, from then on the amplification picture of stack, reads the picture element that dwindles rear None-identified. Client is reappeared module the part marking is separately shown by former resolution ratio, and the described picture that presents is zoomed to the resolution ratio that the display device that uses with this client adapts and shows, the described part marking separately shows as for the superiors.
The coding module that the Audio and Video of microenvironment is derived module and low delay is coupled. The required computing of coding module can utilize CPU or GPU on calculation services node to complete, or uses special coded hardware to process. The graphical information of Application Program Interface forms each frame of outputting video streams by certain time interval. At a time, Application Program Interface figure is converted to the scramble time of corresponding frame in video flowing and is not more than 10 milliseconds; On user's terminal device, this frame of video is also not more than 10 milliseconds by video decoder decodes time of being presented on terminal screen. Under normal circumstances, we expect that Internet Transmission time delay (TTL) from service system to user terminal is in 40-50 millisecond left and right. On the other hand, the frame per second per second of video flowing is limited between 25-30, therefore user with the viewed delay of interbehavior of remote application at most 100 milliseconds of left and right. For general application program, and even mutual PC game is all an acceptable scope for this.
Fig. 8 has shown video encoding module, the streamline strategy (nethermost strategy in figure) that decoder module and transport module are taked in order to reduce coding-decoding delay. The picture of program is actual is split into multiple parts, each part is coded separately into an independent video streaming to client, after being decoded by decoder module, then by reappearing module by the picture amalgamation of cutting apart and showing, the picture of wherein said independent division is with higher bandwidth for transmission. GPU carries out compressed encoding by changing program interface and is converted to video flowing, and the process of decoding at Terminal Server Client. Coding-decode procedure is quasi real time, its postpone transmit in network with data the delay causing compare be a small amount of. In order to reduce the delay of Code And Decode, Application Program Interface is actual in cataloged procedure is divided into multiple parts, and every part encodes independently-transmits-decodes, the treatment step of each several part is with the concurrent execution of pipeline system.
Be that single video flowing is compared (Fig. 8 top strategy) by whole picture coding, although lose, can effectively utilize the parallel ability of computing node in compression ratio, reduce coding delay (strategy in the middle of Fig. 8). But the decoder module of client, due to the restriction of client computing capability, does not often have parallel decoding ability, therefore can not significantly reduce decoding delay. Also can see, in this two kinds of strategies above, the computing capability of equipment is idle in transmitting procedure simultaneously. And in streamline strategy, the process of Code And Decode and the process of transmission are carried out alternately between different pictures piece, when a picture block transmission therein, be encoded module and decoder module of the computing capability of equipment is used for other picture block to process, in fact will encode-transmit-three steps of decoding partly walk abreast, thereby have reduced total mutual delay.
H.264 video decode module on the video encoding module of microenvironment and corresponding client device is used or similar standard. In Video coding, must comprise the image frame of two types. One of them (I frame) do not rely on any other image frame in the time of decoding, the information of some frame being received before the decoding of another kind of (P frame) depends on. Single I frame need to be larger than P frame byte number, the distribution of I frame in video flowing is uncertain, the time interval between the appearance of every two I frames is between 1-10 second. Video flowing between two I frames is made up of the less P frame of byte number.
Within the specific moment, may be suitable difficulty for the node of cloud computing system provides what carry out miscellaneous application program. One of distinct issues are that the binary system run time version of preserving each application program may need a large amount of stored bits units. Particularly, concerning PC game, the binary system run time version of the PC game in the present age often exceedes 4G byte (1 byte number that standard DVD can carry). If it is upper the binary code of program to be stored in respectively to the local storage (hard disk) of each computing node, need to provide for each computing node the storage capacity of several TB bytes. The cost of this storage system of giving and maintenance all bring considerable difficulty. A scheme of saving the memory space of computing node is that the binary code of program is kept on single memory device, i.e. program storage array as shown in Figure 3, and each node in the time of performing a programme by the code data on this equipment of network connected reference at a high speed.
Can this Concurrency Access can only for example, in file system Network Based (SAMBA) aspect and can not for example, realize network disk system (iSCSI) is upper. But to a lot of be only the application program that operation design on local storage at a high speed (for example nearly all PC game), program execution to user interactions before need from storage system, to read in advance a large amount of data in internal memory. In theory, in the time transmitting data with the ideal velocity of gigabit (Gigabit) network between computing node and memory node, the reading rate (~ 100MB/s) of data can match with conventional local storage (hard disk, about 70MB/s). If stand-by period that the reading of data causes and on local storage quite when (even slightly long), this wait has been familiar with by user and has been accepted, and will can not reduce user's experience. But, these data are usually deposited with the form of small documents, and the conventional NFS based on SAMBA agreement be not suitable for the concurrent task that reads of large amount of small documents under windows, result causes actual valid data reading speed to be often less than 10MB/s, before program execution, be several times and even ten several times on local storage with executory data read waiting time, seriously reduced user's experience.
By microenvironment, we have optimized NFS under the SAMBA agreement data reading performance using redundancy in the time that application program is accessed, and do not need to develop in addition other NFS that difficulty more and stability leave a question open. In the system of Fig. 3, in cloud computing system, all service nodes are connected with single storage array. As shown in Figure 9, under normal circumstances, application program is directly by the network file position (dotted arrow) on SAMBA protocol access storage array. And in microenvironment, the actual buffer memory that is arranged in local memory device that is redirected to of the access of application program. Cache contents is managed by microenvironment. In the time that computing node runs application first, by another, the agreement based on rSync scheme (we are called rSync_i) reads the network file on storage array to microenvironment. In the time reading, multiple concurrent file access requests are merged into single TCP request and are conducted interviews according to directory location, make the reading speed of file reality be approached the extreme value that network environment allows.
Multiple computing nodes are the application data of jointly accessing on same memory node, and the excessive data that each user is produced when the executive utility stores on single memory node. Importing and exporting and transmitting by microenvironment of data realizes. When microenvironment has been used specific strategy to carry out accelerating application to carry out from memory node the speed of reading out data. Multiple computing nodes are formed a node array by system, and multiple node arrays are shared a memory node (array) and read the data of performed application data and recording user generation. User can be by one in the multiple computing node arrays of the data selection such as network delay and bandwidth. Multiple memory nodes have formed distributed memory system that can phase mutually synchronization.
Subscription client is accepted discernible any user interactive and is sent in microenvironment by control information. But the input filtering module of microenvironment will be differentiated all user's operations and only allow specific operation can send in the middle of the application program being performed. This judgement is situation based on current application program interface and dynamically changing. For example, in the time that current cursor (can access and carry out the window of the file of the local store path of computing node) at the interface that is judged as " danger ", input filtering module can stop long-distance user to carry out may be to computing node and the adventurous operation of whole service system.
Derive the video/audio stream of module output and the data of supplemental stream are stored in microenvironment output buffer module as shown in Figure 4 from interface and sound. Therefore, the user's that the information such as the interface of performed application program not only can requested execution terminal is seen, also can output in one or more users' that participate in onlooker's identity terminal by output buffer module and transport module. Connect the microenvironment of same computing node as onlooker's user, flow in corresponding microenvironment but cannot send control information. Onlooker user, except watching the action with the user of program interaction, can also send specific interactive information to cloud computing system by other passage. As shown in figure 10, a specific program can start in this microenvironment, receives and gather other and watch user's exchange of information. Interface is derived module a video flowing is merged into in the interface of these exchange of information and application program, thereby these interactive information are distributed to again to user's (comprising interactive user and onlooker user) of current all these microenvironments of connection, and are may be displayed in all users' terminal. For example, in the microenvironment at a user place of carrying out PC game, onlooker can input comment in real time with written form, and these comments can send to all other onlooker and game user.
The video flowing that beholder's user and the mutual user of execution access is synchronous (except the impact causing due to different network delays between user). But the video decode module of user terminal generally can not have the frame (B frame) of reference to decode and correct display of video streams since one. In output buffer module, can preserve all videos frame data in a period of time of having exported to interactive user, wherein must comprise at least one I frame without reference. As shown in figure 11, when a beholder user is in the time connecting microenvironment at first, output buffer module by preserve from sending to beholder user when the nearest I frame of output frame of forward direction interactive user, until while having next I frame to occur, this frame will see that beholder user and interactive user export simultaneously. The time interval occurring due to two I frames may reach 10 seconds, and the terminal that this strategy has significantly reduced beholder user is the delay when there is video from successful connection.
Particular application that is called as " displaying wall " may operate in one independently in microenvironment, as shown in figure 12. The intrasystem service managerZ-HU of displaying wall access computation is obtained the application program of moving in other microenvironment, and is connected in these microenvironments as beholder user. Show the wall program all video flowings that get of decoding, and these video flowings are reorganized in a single interface with the size of preview graph. This interface is encoded as new video flowing and sends to user terminal screen. User can select to show certain preview video stream within the walls, and other users in flowing corresponding microenvironment as a beholder user with preview video exchange.
The Audio and Video stream of microenvironment is sending to before network and can both be merged into the multimedia content flows (as shown in Figure 4) of specific format by the agreement module of reforming. Such multimedia content flows can further be arrived at the client terminal program with special realization by internet. For example, the multimedia content flows generating from reformation module can be by with RTMP(RealTimeMessagingProtocol) mode of the agreement AdobeShockwaveFlash platform that carries and be widely used presents. Thereby user allows to use cloud service system on the basis of not moving special client-side program. For example, this makes any user only need to have the terminal (all PC, net book, WebOS system and most mobile terminal etc.) that can move modern Web browser and can be used as beholder user's access.
Use AdobeShockwaveFlash platform can receive the supplemental stream that microenvironment is sent, and the operation on flash platform sends to input as control information stream and imports module using user, thereby realize a complete user terminal and be not only beholder. But based on the security limitations of Flash platform, while carrying out remote application, some required interbehavior can not directly be realized. For example portion keyboard function. A typical situation is, in the PC of for example first person of some application program game, to allow mouse to do unlimited movement (corresponding to the continuous rotation at visual angle) to a specific direction. But the mouse action that the client terminal of being realized by Flash can only receive in a limited rectangular extent, when mouse moves while causing pointer movement to the edge of rectangle to a direction, follow-up movement cannot be detected again.
Figure 13 has shown this problem solution: the rectangular area that receives mouse pointer mobile message is split into two parts, and in normal region, the information that mouse moves is normally sent to long-range application program. In the time that mouse pointer moves in the buffering edge region of rectangular edges scope, on receiving once before mouse move, last mobile information can be sent to remote application at certain intervals repeatedly, continues mobile information thereby allow user to use mouse to send to specific direction. In the time of buffering edge region, specific pattern will be presented near mouse pointer focal position, to point out special mobile behavior of user to occur.
In the service system shown in Fig. 3, service managerZ-HU is responsible for controlling the situation of the application program of carrying out in whole microenvironments on all service nodes in cloud system, and itself and customer data base are associated. Use a performed typical workflow of the user of cloud computing remote interaction procedure service as shown in figure 14. User's the terminal program first front end of access system (conventionally with Web form) obtains by security gateway and the authority (1) that is connected a certain microenvironment; Front end and service managerZ-HU alternately taking on security gateway as user terminal is opened an outlet, and according to the information of customer data base, corresponding microenvironment is set. In the time that user terminal connects this microenvironment, the list of the application program that user can carry out in microenvironment is passed through database information specified (2) by service managerZ-HU; Carry out in the process of specific application program user, relevant information (contents of program of for example carrying out, time of implementation, the data that user generates etc.) is passed service managerZ-HU and sends back customer data base (3); Finally, complete when user's executive utility, user leaves microenvironment and informs the information that service managerZ-HU exits. and required information in microenvironment before collecting is synchronized to customer data base by service managerZ-HU, for user accesses prepare (4) next time.
In the application of remote interaction procedure service, network environment is experienced and has been caused the most directly restriction user. Experience in order to obtain enough users, user needs the network of a low delay and high bandwidth to the link of computing node. Conventionally, the transmission delay of network should be lower than 50 milliseconds. In addition, test shows that, under a rational level of resolution (being conventionally assumed to be 1024x768), the video flowing of transmission reaches 25-30FPS and keeps enough readability to require the bandwidth of minimum 2Mbps. This means when a cloud service system is served a large number of users at the same time, to grid, outlet will cause huge pressure to the flow of network.
Such cloud service system requires a distributed deployment scheme so that the network bandwidth that effectively utilizes differentiation is to improve efficiency of transmission and to reduce cost. As shown in figure 15, distributed calculation services node array is deployed in multiple positions in all parts of the country, the network egress of its served user terminals single line network that normally campus network is lower. The array of multiple deployment is connected with the distributed storage array on access backbone network by special line. All these distributed systems are all with high-quality but limited bandwidth is connected to central database. User's terminal is after access web front-end, and first from connecting multiple calculation services nodes and selecting connective best one, then the microenvironment that connects certain node is wherein with executive utility service. The subscriber data file producing in user and the application program interaction direct-connected storage array of service node that will write direct. After user interactions finishes, service managerZ-HU will send request, in a specific time-domain, (conventionally be less than 12 hours), and the data file that user interactions produces will be synchronized in whole storage arrays. A small amount of core data is write customer data base in real time.
This distributed schemes based on following some consider: 1. the network egress at single calculation services node array place cannot bear bandwidth load that numerous users bring and provide the JA(junction ambient) that meets the demands (bandwidth of minimum 2Mbps and postpone lower than 50ms) for all users; 2. user may produce quite a large amount of user data in the time carrying out an interactive program. For example, in the time that user carries out the game of a physical culture racing tip, can record a video and generate the data file that reaches hundreds of MB some scene in game. Single storage array cannot be in real time or the data that quasi real time complete magnanimity with whole distributed array of services synchronous, still, and limited array service node array to make real-time data be synchronously possible; 3. due to the restriction of geographical conditions, user's possible translational speed in network, by lower than completing synchronous time threshold between storage array. For example, suppose to have to be distributed in the regional storage array in Beijing and two, Guangzhou, user store into user data in the storage array of In Guangzhou Area can be in the longest 12 hours and the array of Beijing area to complete data synchronous. And in most cases, a user's (we suppose its corresponding natural person's individuality) who finishes service in In Guangzhou Area can not apply new service in existing communications and transportation level in 12 hours in Beijing area. This real time threshold makes above-mentioned distributed schemes become possibility.
Claims (1)
1. a long-distance interactive system based on cloud computing, it comprises terminal device, web front-end and security gateway, described security gateway comprises one or more service nodes; First terminal device accesses described front end, by obtaining the authority that connects a certain service node in security gateway after checking; Terminal device uses this authority to be connected with specific service node in cloud computing system subsequently; It is characterized in that: in described security gateway, the interface of all application programs and audio frequency output are converted into Audio and Video stream, and a relative additional information stream, and pass to described terminal device by internet; Described terminal device is competent decoding SD or H.264 video flowing of high definition enough;
Described service node further comprises a processing unit, between application program and system API or network, it is characterized in that: comprise a function address monitoring modular, for detection of the function address of application program in internal memory, an and function address redirection module, the function address of storing for changing the internal memory of application program, is redirected function address; And multiple interfaces, for realizing communicating by letter between computer hardware and API; Described terminal device can be realized in kinds of platform, comprises computer, TV set-top box, mobile terminal device;
Described service node is by the data of jointly accessing on same memory node, and the excessive data that each user is produced stores on single memory node;
Described multiple service node forms a node array, and multiple node arrays are shared the data of a memory node reading out data and recording user generation, and user selects in described multiple node arrays by network delay and band data;
Described processing unit is connected with mixed-media network modules mixed-media with the video/audio compression module on this service node, thus by the interface of application program with play up data transaction and become video/audio stream, be sent to other terminal on internet through mixed-media network modules mixed-media.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110183361.0A CN102857531B (en) | 2011-07-01 | 2011-07-01 | A kind of long-distance interactive system based on cloud computing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110183361.0A CN102857531B (en) | 2011-07-01 | 2011-07-01 | A kind of long-distance interactive system based on cloud computing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102857531A CN102857531A (en) | 2013-01-02 |
| CN102857531B true CN102857531B (en) | 2016-05-18 |
Family
ID=47403723
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201110183361.0A Active CN102857531B (en) | 2011-07-01 | 2011-07-01 | A kind of long-distance interactive system based on cloud computing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102857531B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102857535B (en) * | 2011-07-01 | 2015-09-09 | 云联（北京）信息技术有限公司 | A kind of computer processing unit, computer gateway, interactive system |
| CN105897653A (en) * | 2014-11-28 | 2016-08-24 | 韩成阳 | High-performance mobile device framework |
| CN104516969A (en) * | 2014-12-25 | 2015-04-15 | 祝峰 | Data and compute-intensive processing system of cloud computing platform |
| CN106022007B (en) * | 2016-06-14 | 2019-03-26 | 中国科学院北京基因组研究所 | The cloud platform system and method learning big data and calculating is organized towards biology |
| CN109257350A (en) * | 2018-09-14 | 2019-01-22 | 国云科技股份有限公司 | A kind of authorization method that limitation video redirects |
| CN109304031B (en) * | 2018-09-19 | 2021-11-02 | 电子科技大学 | Virtual cloud game platform based on heterogeneous intelligent terminal |
| CN111225243B (en) * | 2020-01-20 | 2021-02-02 | 中南大学 | Video block scheduling method and system |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101179868A (en) * | 2006-11-09 | 2008-05-14 | 中国电信股份有限公司 | Method to access to remote monitoring system through mobile terminal and remote monitoring system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| AU2002225810A1 (en) * | 2000-11-02 | 2002-05-15 | Sony Electronics Inc. | Content and application download based on a home network system configuration profile |
-
2011
- 2011-07-01 CN CN201110183361.0A patent/CN102857531B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101179868A (en) * | 2006-11-09 | 2008-05-14 | 中国电信股份有限公司 | Method to access to remote monitoring system through mobile terminal and remote monitoring system |
Non-Patent Citations (2)
| Title |
|---|
| 基于多层结构的网络游戏平台的研究与应用;匡华;《中国优秀博硕士学位论文全文数据库 (硕士)》;20070415;对比文件1第4章，图4.1，4.2 * |
| 面向图形应用的远程设备交互技术;吕回;《中国优秀硕士学位论文全文数据库》;20091215;对比文件2第二章第2.2-2.4节，第三章，图3.3 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102857531A (en) | 2013-01-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102855133B (en) | A kind of computer processing unit interactive system | |
| CN102857531B (en) | A kind of long-distance interactive system based on cloud computing | |
| CN102857533B (en) | A kind of long-distance interactive system based on cloud computing | |
| CN106713485B (en) | Cloud computing mobile terminal | |
| CN105263050B (en) | Mobile terminal real-time rendering system and method based on cloud platform | |
| CN105635751B (en) | A kind of video cloud platform plays the method and device of video | |
| CN103631634A (en) | Graphics processor virtualization achieving method and device | |
| WO2021185302A1 (en) | Live streaming and configuration method based on cloud phones, and related apparatus and system | |
| CN105072190A (en) | Method and system for realizing real-time desktop cloud | |
| CN109304031A (en) | A kind of virtualization cloud game platform based on isomery intelligent terminal | |
| CN103838779A (en) | Idle computing resource multiplexing type cloud transcoding method and system and distributed file device | |
| CN102279752B (en) | Device and method for rendering ultra-large scene in real time based on Web three-dimension (3D) | |
| CN102868908A (en) | High-efficiency streaming media playing method and device | |
| CN108449633A (en) | Set-top box cloud frame based on NFV and service implementation method | |
| CN104040995B (en) | Multimedia redirection method, multimedia server, and computer system | |
| US11689749B1 (en) | Centralized streaming video composition | |
| US20230405455A1 (en) | Method and apparatus for processing cloud gaming resource data, computer device, and storage medium | |
| CN110025957A (en) | A kind of cloud game service device end-rack structure, client and system | |
| WO2024037110A1 (en) | Data processing method and apparatus, device, and medium | |
| CN113209632A (en) | Cloud game processing method, device, equipment and storage medium | |
| CN106131550A (en) | Play the method and device of multimedia file | |
| CN102857535B (en) | A kind of computer processing unit, computer gateway, interactive system | |
| CN102857534B (en) | Remote interaction method on basis of cloud computing | |
| CN105491021A (en) | Android cloud application server and Android cloud application server system | |
| CN115221446A (en) | Asynchronous on-demand Web3D scene animation loading optimization method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20170727 Address after: 100083 Beijing City, Haidian District Xueyuan Road District No. 30 Fangxing building room 702 Patentee after: Beijing billion intelligent technology Limited by Share Ltd. Address before: 100027, room 2, No. 2026, East Third Ring Road North, Beijing, Chaoyang District Patentee before: Cloud Alliance (Beijing) Information Technology Co.,Ltd. |
|
| TR01 | Transfer of patent right | ||
| CP03 | Change of name, title or address |
Address after: 100083 205-6, 2nd floor, building 5, yard 1, Shangdi East Road, Haidian District, Beijing Patentee after: Beijing Taiyi cloud Technology Co.,Ltd. Address before: 100083 Beijing City, Haidian District Xueyuan Road District No. 30 Fangxing building room 702 Patentee before: Beijing billion intelligent technology Limited by Share Ltd. |
|
| CP03 | Change of name, title or address | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20211019 Address after: 100000 room 801-a40, 8th floor, building 2, No. A5, Rongchang East Street, Beijing Economic and Technological Development Zone, Daxing District, Beijing (Yizhuang group, high-end industrial area of Beijing Pilot Free Trade Zone) Patentee after: CLOUD ALLIANCE (BEIJING) INFORMATION TECHNOLOGY Co.,Ltd. Address before: 100083 205-6, 2nd floor, building 5, yard 1, Shangdi East Road, Haidian District, Beijing Patentee before: Beijing Taiyi cloud Technology Co.,Ltd. |
|
| TR01 | Transfer of patent right | ||
| CP01 | Change in the name or title of a patent holder |
Address after: 100000 room 801-a40, 8th floor, building 2, No. A5, Rongchang East Street, Beijing Economic and Technological Development Zone, Daxing District, Beijing (Yizhuang group, high-end industrial area of Beijing Pilot Free Trade Zone) Patentee after: Beijing Millennium Information Technology Co.,Ltd. Address before: 100000 room 801-a40, 8th floor, building 2, No. A5, Rongchang East Street, Beijing Economic and Technological Development Zone, Daxing District, Beijing (Yizhuang group, high-end industrial area of Beijing Pilot Free Trade Zone) Patentee before: Cloud Alliance (Beijing) Information Technology Co.,Ltd. |
|
| CP01 | Change in the name or title of a patent holder | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20231213 Address after: 100000 C5-05, F1, Building 19, No. 10, Langjiayuan, Jianguomenwai, Chaoyang District, Beijing Patentee after: Taiyi Yunjing Technology Co.,Ltd. Address before: 100000 room 801-a40, 8th floor, building 2, No. A5, Rongchang East Street, Beijing Economic and Technological Development Zone, Daxing District, Beijing (Yizhuang group, high-end industrial area of Beijing Pilot Free Trade Zone) Patentee before: Beijing Millennium Information Technology Co.,Ltd. |
|
| TR01 | Transfer of patent right |