CN114390355A

CN114390355A - Playback method of protocol data and electronic equipment

Info

Publication number: CN114390355A
Application number: CN202111509756.5A
Authority: CN
Inventors: 马合军
Original assignee: Alibaba China Co Ltd
Current assignee: Alibaba China Co Ltd
Priority date: 2021-12-10
Filing date: 2021-12-10
Publication date: 2022-04-22

Abstract

The embodiment of the disclosure discloses a protocol data playback method and electronic equipment, wherein the protocol data playback method comprises the steps of responding to a received playback instruction for target protocol data, and sending a request message for requesting to download the target protocol data to a server, wherein the target protocol data comprises client original control data and server instance original response data generated when a client remotely controls a server instance; receiving the target protocol data sent by the server; and performing visual picture playback and audio playback on the target protocol data based on the target protocol data.

Description

Playback method of protocol data and electronic equipment

Technical Field

The present disclosure relates to the field of data processing technologies, and in particular, to a method for playing back protocol data and an electronic device.

Background

RDP (Remote Desktop Protocol)/SSH (Secure Shell Protocol) is a communication Protocol through which a user can remotely access and operate a server instance. When a user remotely operates a server, illegal operations such as unauthorized access, abuse and misoperation sometimes occur to cause potential safety hazards or loss, so that an auditing system is needed to audit operation data of the user to analyze the root cause of problems, and in order to facilitate more intuitive positioning of auditors, RDP/SSH protocol data needs to be played back, and an interface presented by a user client can be played back. Visual playback can help the auditor visually audit whether illegal operation exists in the operation process, and can help the auditor very conveniently analyze and locate the reasons of accidents or faults. Players for playing RDP/SSH protocol data in the market at present can only play quiet operation pictures, and cannot play audio during operation.

Disclosure of Invention

In order to solve the problems in the related art, embodiments of the present disclosure provide a method, an apparatus, an electronic device, a medium, and a program product for playback of protocol data.

In a first aspect, a method for playing back protocol data is provided in the embodiments of the present disclosure.

Specifically, the playback method of the protocol data includes:

in response to receiving a playback instruction for target protocol data, sending a request message for requesting downloading of the target protocol data to a server, wherein the target protocol data comprises client original control data and server instance original response data generated when a client remotely controls a server instance;

receiving the target protocol data sent by the server;

and performing visual picture playback and audio playback on the target protocol data based on the target protocol data.

With reference to the first aspect, in a first implementation manner of the first aspect, the receiving the target protocol data sent by the server includes:

acquiring subsequent target protocol data to be loaded from the server in advance;

and storing the subsequent target protocol data to be loaded to a play cache region so as to acquire the corresponding target protocol data from the play cache region for playback.

With reference to the first aspect and the first implementation manner of the first aspect, in a second implementation manner of the first aspect, the obtaining, in advance, target protocol data to be subsequently loaded from the server includes:

determining the early loading time quantum of the subsequent target protocol data to be loaded based on the subsequent unplayed data quantum cached in the play cache region;

and acquiring the subsequent target protocol data to be loaded from the server in advance according to the amount of the advanced loading time.

With reference to the first aspect and the foregoing various implementation manners of the first aspect, in a third implementation manner of the first aspect, the performing, based on the target protocol data, visual image playback and audio playback on the target protocol data includes:

and desensitizing and then playing back desensitized data which accord with a preset desensitization rule in the target protocol data.

With reference to the first aspect and the foregoing various implementation manners of the first aspect, in a fourth implementation manner of the first aspect, the performing, based on the target protocol data, visual image playback and audio playback on the target protocol data includes:

acquiring preset data to be added;

and performing visual picture playback and audio playback on the target protocol data and the data to be added.

With reference to the first aspect and the foregoing various implementations of the first aspect, in a fifth implementation of the first aspect, an embodiment of the present disclosure further includes:

when a jump playing instruction is received, responding to first target protocol data between a playing time point before jump and the jump playing time point indicated by the jump playing instruction stored in a playing cache region, and playing back a picture and an audio corresponding to the jump playing time point based on a playback picture before jump and the first target protocol data.

With reference to the first aspect and the various implementation manners of the first aspect, in a sixth implementation manner of the first aspect, in the embodiment of the present disclosure, when a skip play instruction is received, in response to that target protocol data of a skip play time point indicated by the skip play instruction is not stored in a cache, a skip loading request is sent to the server, where the skip loading request carries the skip play time point;

receiving a first key frame image corresponding to the skip playing time point and second target protocol data between the time point of the first key frame image and the skip playing time point, which are sent by the server;

and playing back the picture and the audio corresponding to the jumping play time point based on the first key frame image and the second target protocol data.

With reference to the first aspect and the foregoing various implementations of the first aspect, in a seventh implementation of the first aspect, an embodiment of the present disclosure further includes:

and responding to that first target protocol data are stored in the play cache region, and clearing the data in the play cache region except the first target protocol data.

With reference to the first aspect and the foregoing various implementations of the first aspect, in an eighth implementation of the first aspect, an embodiment of the present disclosure further includes:

and in response to that the target protocol data of the jump playing time point is not stored in the playing cache region, clearing all data in the playing cache region.

With reference to the first aspect and the foregoing various implementation manners of the first aspect, in a ninth implementation manner of the first aspect, the target protocol includes a remote desktop protocol RDP or a secure shell protocol SSH, the target protocol data includes audio data with a timestamp, and when the target protocol includes RDP, the target protocol data further includes picture layer data with a timestamp; when the target protocol comprises SSH, the target protocol data further comprises time-stamped character data, the method further comprising:

and clearing the data to be cleared in the play cache region, wherein the time stamp of the data to be cleared is before the time stamp of the data corresponding to the current playback picture, and the time difference between the time stamp of the data corresponding to the current playback picture and the time stamp of the data corresponding to the current playback picture exceeds a first preset time length.

With reference to the first aspect and the foregoing various implementations of the first aspect, in a tenth implementation of the first aspect, an embodiment of the present disclosure further includes:

responding to the received speed-multiplying playing instruction, and obtaining the playing time difference between the subsequent frame data corresponding to the playing speed indicated by the speed-multiplying playing instruction;

and playing the subsequent frame data according to the playing time difference.

With reference to the first aspect and the foregoing various implementations of the first aspect, in an eleventh implementation of the first aspect, an embodiment of the present disclosure further includes:

obtaining data of each frame to be rendered based on the target protocol data;

and in response to the condition that the playback is not smooth is met, discarding at least one frame of data, the similarity of which to the target frame of data exceeds a preset threshold, of the frames of data to be rendered.

With reference to the first aspect and the foregoing various implementations of the first aspect, in a twelfth implementation of the first aspect, an embodiment of the present disclosure further includes:

and sending a retransmission request to the server in response to the detection that a target data packet corresponding to the target protocol data is lost, wherein the retransmission request is used for requesting the server to resend the target data packet.

With reference to the first aspect and the foregoing various implementations of the first aspect, in a thirteenth implementation of the first aspect, an embodiment of the present disclosure further includes:

and outputting prompt information in response to that the target protocol data sent by the server is not received after the second preset time length is exceeded, wherein the prompt information is used for prompting to reload data.

With reference to the first aspect and the foregoing various implementations of the first aspect, in a fourteenth implementation of the first aspect, an embodiment of the present disclosure further includes:

and responding to the disconnection of the server, reestablishing the connection with the server, and continuing the playback from the current playback picture after the connection is established.

With reference to the first aspect and the foregoing various implementations of the first aspect, in a fifteenth implementation of the first aspect, an embodiment of the present disclosure is a method further including:

receiving a second key frame sent by the server;

and when the initial playing interface of the target protocol data is entered, using the second key frame as a playing cover.

With reference to the first aspect and the foregoing various implementations of the first aspect, in a sixteenth implementation of the first aspect, an embodiment of the present disclosure further includes:

and in response to receiving the screen capture instruction, capturing the current playback picture as a screen capture image, and storing the screen capture image.

With reference to the first aspect and the foregoing various implementations of the first aspect, in a seventeenth implementation of the first aspect, an embodiment of the present disclosure further includes:

and responding to the received interface adjusting instruction, and adjusting the current playback interface according to the instruction of the interface adjusting instruction, wherein the interface adjusting instruction is used for instructing to adjust at least one of the size, the position and the interface display content of the current playback interface.

In a second aspect, the present disclosure provides a playback apparatus for protocol data.

Specifically, the playback apparatus of protocol data includes:

the server comprises a first sending module, a second sending module and a third sending module, wherein the first sending module is used for responding to a playback instruction of target protocol data and sending a request message for requesting downloading of the target protocol data to the server, and the target protocol data comprises client original control data and server instance original response data generated when a client remotely controls a server instance;

a first receiving module configured to receive the target protocol data sent by the server;

a playback module configured to perform visual picture playback and audio playback on the target protocol data based on the target protocol data.

With reference to the second aspect, in a first implementation manner of the second aspect, the first receiving module is configured to:

With reference to the second aspect and the first implementation manner of the second aspect, in a second implementation manner of the second aspect, the portion of the first receiving module that obtains, in advance, the target protocol data to be subsequently loaded from the server is configured to:

With reference to the second aspect and the first implementation manner of the second aspect, in a third implementation manner of the second aspect, the playback module is configured to:

With reference to the second aspect and the first implementation manner of the second aspect, in a fourth implementation manner of the second aspect, the playback module is configured to:

acquiring preset data to be added;

With reference to the second aspect and the first implementation manner of the second aspect, in a fifth implementation manner of the second aspect, the apparatus further includes:

the first skip playing module is configured to respond to first target protocol data between a play time point before skip and a skip playing time point indicated by a skip playing instruction stored in a playing cache area when the skip playing instruction is received, and play back a picture and an audio corresponding to the skip playing time point based on a playback picture before skip and the first target protocol data.

With reference to the second aspect and the first implementation manner of the second aspect, in a sixth implementation manner of the second aspect, the apparatus further includes:

the second sending module is configured to respond to that target protocol data of a jump playing time point indicated by the jump playing instruction is not stored in a cache when the jump playing instruction is received, and send a jump loading request to the server, wherein the jump loading request carries the jump playing time point;

a second receiving module, configured to receive a first key frame image corresponding to the skip playing time point and second target protocol data between a time point of the first key frame image and the skip playing time point, where the first key frame image corresponds to the skip playing time point and the second key frame image is sent by the server;

and the second jumping play module is configured to play back the pictures and the audios corresponding to the jumping play time points based on the first key frame images and the second target protocol data.

With reference to the second aspect and the first implementation manner of the second aspect, in a seventh implementation manner of the second aspect, the apparatus further includes:

the first clearing module is configured to respond to the fact that first target protocol data are stored in the playing buffer, and clear data except the first target protocol data in the playing buffer.

With reference to the second aspect and the first implementation manner of the second aspect, in an eighth implementation manner of the second aspect, the apparatus further includes:

and the second clearing module is configured to respond to the situation that the target protocol data of the jump playing time point is not stored in the playing cache region, and clear the data in the playing cache region.

With reference to the second aspect and the first implementation manner of the second aspect, in a ninth implementation manner of the second aspect, the target protocol includes a remote desktop protocol RDP or a secure shell protocol SSH, the target protocol data includes audio data with a timestamp, and when the target protocol includes RDP, the target protocol data further includes picture layer data with a timestamp; when the target protocol comprises SSH, the target protocol data further comprises time-stamped character data, the apparatus further comprising:

and the third clearing module is configured to clear the data to be cleared in the play buffer area, wherein the timestamp of the data to be cleared is before the timestamp of the data corresponding to the current playback picture, and the time difference between the timestamp of the data corresponding to the current playback picture and the timestamp of the data corresponding to the current playback picture exceeds a first preset time length.

With reference to the second aspect and the first implementation manner of the second aspect, in a tenth implementation manner of the second aspect, the apparatus further includes:

the first acquisition module is configured to respond to a received speed-doubling playing instruction, and obtain a playing time difference between subsequent frame data corresponding to a playing speed indicated by the speed-doubling playing instruction;

and the double-speed playing module is configured to play the subsequent frame data according to the playing time difference.

With reference to the second aspect and the first implementation manner of the second aspect, in an eleventh implementation manner of the second aspect, the apparatus further includes:

the second acquisition module is configured to obtain data of each frame to be rendered based on the target protocol data;

the discarding module is configured to discard at least one frame data of the frame data to be rendered, the similarity of which to the target frame data exceeds a preset threshold value, in response to the playback non-fluency condition being met.

With reference to the second aspect and the first implementation manner of the second aspect, in a twelfth implementation manner of the second aspect, the apparatus further includes:

a third sending module, configured to send a retransmission request to the server in response to detecting that a target data packet corresponding to the target protocol data is lost, where the retransmission request is used to request the server to re-issue the target data packet.

With reference to the second aspect and the first implementation manner of the second aspect, in a thirteenth implementation manner of the second aspect, the apparatus further includes:

and the output module is configured to respond to the situation that the target protocol data sent by the server is not received for more than a second preset time length, and output prompt information, wherein the prompt information is used for prompting to reload data.

With reference to the second aspect and the first implementation manner of the second aspect, in a fourteenth implementation manner of the second aspect, the apparatus further includes:

and the reconnection module is configured to respond to disconnection from the server, reestablish connection with the server and continue playback from the current playback picture after connection is established.

With reference to the second aspect and the first implementation manner of the second aspect, in a fifteenth implementation manner of the second aspect, the apparatus further includes:

a third receiving module configured to receive a second key frame sent by the server;

and the cover display module is configured to display the second key frame as a playing cover when the initial playing interface of the target protocol data is entered.

With reference to the second aspect and the first implementation manner of the second aspect, in a sixteenth implementation manner of the second aspect, the apparatus further includes:

and the screen capture module is configured to respond to the screen capture instruction, capture the current playback picture as a screen capture image and store the screen capture image.

With reference to the second aspect and the first implementation manner of the second aspect, in a seventeenth implementation manner of the second aspect, the apparatus further includes:

the adjustment module is configured to adjust the current playback interface according to the indication of the interface adjustment instruction in response to receiving the interface adjustment instruction, and the interface adjustment instruction is used for indicating to adjust at least one of the size, the position and the interface display content of the current playback interface.

In a third aspect, the disclosed embodiments provide an electronic device comprising a memory and a processor, wherein the memory is configured to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method according to any one of the first aspect.

In a fourth aspect, the disclosed embodiments provide a computer-readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the method according to the first aspect.

In a fifth aspect, the disclosed embodiments provide a computer program product comprising computer instructions which, when executed by a processor, implement the method steps as described in the first aspect.

According to the technical scheme provided by the embodiment of the disclosure, a request message for requesting downloading of target protocol data can be sent to a server in response to receiving a playback instruction for the target protocol data; receiving the target protocol data sent by the server, wherein the target protocol data is original protocol data and comprises at least one of the following data with a time stamp: audio data, picture layer data and character data; and performing visual picture playback and audio playback on the target protocol data based on the target protocol data. Therefore, the remote control picture of the client can be played back, and the audio played by the remote control of the client can be played back, so that a more real remote control condition of the client is provided for a user, and the user can watch the remote control picture conveniently; in addition, the target protocol data is original data, data fusion and other processing are not performed, data addition or processing on each data is more convenient for the playback terminal, and personalized playback is realized.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

Other features, objects, and advantages of the present disclosure will become more apparent from the following detailed description of non-limiting embodiments when taken in conjunction with the accompanying drawings. In the drawings:

fig. 1 shows a flow chart of a playback method of protocol data according to an embodiment of the present disclosure;

fig. 2 illustrates an overall flowchart of a playback method of protocol data according to an embodiment of the present disclosure;

fig. 3 shows a block diagram of a structure of a playback apparatus of protocol data according to an embodiment of the present disclosure;

FIG. 4 shows a block diagram of an electronic device according to an embodiment of the present disclosure;

FIG. 5 shows a schematic block diagram of a computer system suitable for use in implementing a method according to an embodiment of the present disclosure.

Detailed Description

Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. Also, for the sake of clarity, parts not relevant to the description of the exemplary embodiments are omitted in the drawings.

In the present disclosure, it is to be understood that terms such as "including" or "having," etc., are intended to indicate the presence of the disclosed features, numbers, steps, behaviors, components, parts, or combinations thereof, and are not intended to preclude the possibility that one or more other features, numbers, steps, behaviors, components, parts, or combinations thereof may be present or added.

It should be further noted that the embodiments and features of the embodiments in the present disclosure may be combined with each other without conflict. The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.

In the present disclosure, the acquisition of the user information or the user data is an operation that is authorized, confirmed, or actively selected by the user.

As mentioned above, RDP (Remote Desktop Protocol)/SSH (Secure Shell Protocol) is a communication Protocol through which a user can remotely access and operate a server instance. When a user remotely operates a server, illegal operations such as unauthorized access, abuse and misoperation sometimes occur to cause potential safety hazards or loss, so that an auditing system is needed to audit operation data of the user to analyze the root cause of problems, and in order to facilitate more intuitive positioning of auditors, RDP/SSH protocol data needs to be played back, and an interface presented by a user client can be played back. Visual playback can help the auditor visually audit whether illegal operation exists in the operation process, and can help the auditor very conveniently analyze and locate the reasons of accidents or faults. Playback devices which are used for playing RDP/SSH protocol data in the market at present can only play quiet operation pictures and cannot play audio during operation.

In view of the above, the present disclosure provides a playback scheme of protocol data, which may send a request message requesting downloading of target protocol data to a server in response to receiving a playback instruction for the target protocol data; receiving the target protocol data sent by the server, wherein the target protocol data is original protocol data and comprises at least one of the following data with a time stamp: audio data, picture layer data and character data; and performing visual picture playback and audio playback on the target protocol data based on the target protocol data. Therefore, the remote control picture of the client can be played back, and the audio played by the remote control of the client can be played back, so that a more real remote control condition of the client is provided for a user, and the user can watch the remote control picture conveniently; in addition, the target protocol data is original data, data fusion and other processing are not performed, data addition or processing on each data is more convenient for the playback terminal, and personalized playback is realized.

Fig. 1 illustrates a flowchart of a playback method of protocol data according to an embodiment of the present disclosure. As shown in fig. 1, the playback method of the protocol data includes the following steps S101 to S103:

in step S101, in response to receiving a playback instruction for target protocol data, sending a request message requesting downloading of the target protocol data to a server, where the target protocol data includes client original manipulation data and server instance original response data generated when a client remotely manipulates a server instance;

in step S102, receiving the target protocol data sent by the server;

in step S103, based on the target protocol data, visual picture playback and audio playback are performed on the target protocol data.

In an embodiment of the present disclosure, the method for playing back the protocol data may be applied to a computer, a computing device, an electronic device, a mobile terminal, and the like that play back the protocol data.

In an embodiment of the present disclosure, the target protocol data includes client raw manipulation data and server instance raw response data generated when the client remotely manipulates the server instance. When the client remotely controls the server instance through the remote communication protocol, the backend server records and stores protocol data such as original control data of the client and original response data of the server, for example, the protocol data may be stored in an OSS (Object Storage Service). The protocol data may be stored as a file of a particular format in chronological order as it is saved, with each protocol session being stored as a separate file. When each file is stored, the user name and the service instance corresponding to the file are recorded so as to identify the protocol data stored in the file, wherein the protocol data is used for operating the corresponding server instance by the user corresponding to the user name.

In an embodiment of the present disclosure, the target protocol data refers to protocol data selected for playback. For example, in a scenario where an auditor audits an operation behavior of each user to locate a fault or an accident, an audit interface may be logged in, an icon of a file where protocol data is located may be displayed on the audit interface, where the protocol data is generated when a user corresponding to each user name within an audit authority of the auditor remotely operates each server instance each time, and the auditor may select, as needed, target protocol data generated when the user audits a server instance of a user of a certain user name operates at a certain time, for example, click on a target file icon corresponding to the target protocol data, so that a playback instruction for the target protocol data may be input, and the target protocol data may be played back.

In an embodiment of the present disclosure, the playback terminal may send, in response to receiving a playback instruction for target protocol data, a request message requesting downloading of the target protocol data to the server, and after receiving the request message, the server may send the target protocol data acquired from the OSS to the playback terminal for playback.

In an embodiment of the present disclosure, the target protocol data is originally recorded protocol data that is not processed by any process such as fusion.

In an embodiment of the present disclosure, the target protocol data includes audio data, where the audio data refers to audio data that is issued by a server instance to a client in response to a play request of the client in a process of remotely operating a server instance by the client, and is provided for the client to play. Of course, it should be noted here that if the client plays audio for the user in the process of remote operation of the client, the target protocol data only includes the audio data, and if the client does not play audio for the user, the target protocol data does not include the audio data.

In an embodiment of the present disclosure, the remote communication protocol may be a character protocol, such as an SSH protocol, or a graphic protocol, such as an RDP protocol, when the remote communication protocol is a character protocol, the target protocol data includes character data, the character data includes character data input by an input device of the client, such as a keyboard, and character data returned by the server; when the remote communication protocol is a graphic protocol, the target protocol data includes picture layer data including change image data that changes a screen display.

In an embodiment of the present disclosure, the target protocol data may further include any one of the following data: cursor layer data, size data and media channel data. The cursor layer data refers to data of a cursor layer corresponding to a mouse of a client, the size data comprises size data of each layer and size data of an overall operation interface remotely controlled by the client, the media channel data refers to a corresponding media channel when various types of protocol data are rendered, and different media channels are used for rendering different types of protocol data.

In an embodiment of the present disclosure, the target protocol data is recorded in a time sequence of the client operation, and also needs to be played back in the time sequence during playback, so that these data all have time stamps, and corresponding target protocol data needs to be played back in the order of the time stamps during playback.

In an embodiment of the present disclosure, the playback terminal may not only play back the operation screen of the client, but also play back the audio that is remotely played at that time. The playback terminal can restore the operation picture of the client according to the recorded visual target protocol data such as picture layer data, character data, cursor layer data and size data to perform visual playback, and simultaneously can play corresponding audio at corresponding video time points based on the audio data.

The playback method provided by the embodiment can not only play back the remote control picture of the client, but also play back the audio played by the client through remote control, so that a more real and complete client remote control condition is provided for a user, and the user can conveniently check the remote control condition; in addition, the target protocol data is original data, data fusion and other processing are not performed, data addition or processing on each data is more convenient for the playback terminal, and personalized playback is realized.

In a possible implementation manner, the receiving the target protocol data sent by the server in step S102 in the playback method may further include the following steps:

In this embodiment, in the process of playback, in order to make playback smoother and not jammed, target protocol data to be played back subsequently may be loaded in advance from the server, the target protocol data acquired in advance may be cached in the play buffer, and the playback terminal may directly acquire the corresponding target protocol data from the play buffer for playback when performing playback.

For example, when the playback terminal receives a playback instruction for target protocol data and enters a play interface of the target protocol data, a predetermined number of target protocol data to be played back subsequently, such as about 1M, may be received from the server first, and cached in the play cache region, and after the user clicks a play key of the play interface, the target protocol data in the play cache region may be played back according to a time stamp sequence, and as the playback progresses, the target protocol data not to be played back in the play cache region gradually decreases, at this time, the target protocol data to be played back subsequently may be continuously loaded from the server in advance, and it is ensured that more than a certain number of target protocol data not to be played back is continuously cached in the play cache region, so that when the subsequent network state is bad, and the loaded data is slow, because the played back is the target protocol data in the play cache region, so that the jamming can be avoided.

In a possible implementation manner, the step of obtaining the target protocol data to be played back subsequently from the server in advance in the above playback method may be implemented as the following steps:

determining the advanced loading time of the subsequent target protocol data to be loaded based on the subsequent unplayed data amount cached in the play cache region;

and acquiring the subsequent target protocol data to be loaded from the server in advance according to the advanced loading time.

In this embodiment, in order to ensure that the amount of unplayed data buffered in the play buffer is sufficient, no playback jams occur, meanwhile, the method avoids too much data in the play buffer area, reduces the memory occupation of the playback terminal, optimizes the terminal performance, the loading time of the subsequent target protocol data to be loaded can be controlled by playing the subsequent unplayed data amount cached in the cache region, the subsequent unplayed data amount cached in the play cache region is large, which indicates that the available bandwidth is high, the speed for acquiring the target protocol data from the server is high, at the moment, the target protocol data to be subsequently loaded from the server does not need to be advanced for too much time, the determined advanced loading time amount is short, and therefore, playback jamming can not occur due to high loading speed, and the subsequent target protocol data to be loaded is loaded in advance in a short time, so that the subsequent unplayed data amount cached in the play cache region can be reduced. When the amount of the subsequent unplayed data cached in the play cache region is small, it indicates that the available bandwidth is low, and the speed for acquiring the target protocol data from the server is low, at this time, to avoid the stuttering, it is necessary to advance a lot of time to subsequently load the target protocol data from the server, and the determined ahead loading time amount is long, so that although the loading speed is low, the amount of the subsequent unplayed data cached in the play cache region can be increased due to the fact that the ahead loading is performed for a long time, and the stuttering is avoided.

For example, the amount of the subsequent unplayed data may be represented by a percentage of the total storage capacity of the play buffer occupied by the amount of the subsequent unplayed data, the amount of the load ahead time may be increased when the percentage is small, such as less than 30%, the amount of the load ahead time may be decreased when the percentage is large, such as greater than 80%, and a default amount of the load ahead time may be used when the percentage is greater than 30% and less than 80%.

In a possible implementation manner, step S103 in the above playback method, that is, performing visual picture playback and audio playback on the target protocol data based on the target protocol data, may include the following steps:

In this embodiment, since the target protocol data is originally recorded protocol data in which processing such as fusion is not performed on these pieces of data, each piece of data in the target protocol data can be processed individually. The security level of protocol data in the target protocol data is higher, and only a user with high authority can browse the protocol data, so that in order to protect the privacy of the user and the safety of the user data, desensitization data which cannot be checked by a current playback user in the target protocol data can be played back after desensitization according to a preset desensitization rule.

In this embodiment, the preset desensitization rule may be that users with different authority levels correspond to different desensitization data, which refers to data that cannot be viewed by users with the authority levels, such as privacy/confidential data. Desensitizing the desensitization data refers to modifying or blurring the desensitization data to hide the real desensitization data.

acquiring preset data to be added;

In this embodiment, the data to be added may be the data defined by the user or may be data default for the playback terminal, so that when performing playback, the data to be added may be added and displayed on the visual picture corresponding to the target protocol data for playback.

For example, the data to be added may be an ID number of a user or a user ID layer, so that the ID number of the user is displayed on a played back visual picture, which can prevent the user from taking a picture with a peripheral device and uploading the picture, and avoid data outflow.

In a possible implementation manner, the playback method may further include the following steps:

In the embodiment, in the process of watching playback, in order to quickly check, a user drags a play progress bar on a play interface and inputs a jump play instruction, wherein the jump can be forward jump or backward jump; the jump playing instruction indicates a jump playing time point. Illustratively, when a user drags a playing progress bar on a playing interface to drag the playing progress bar backwards from the 10 th time to the 30 th time, the jump playing time point refers to the 30 th time, the play time point before the jump refers to the 10 th time, and first target protocol data between the 10 th time and the 30 th time needs to be acquired.

In this embodiment, in response to receiving a skip play instruction, the playback terminal may check whether a first target protocol data between a play time point before skip and a skip play time point indicated by the skip play instruction is stored in the play buffer, where the first target protocol data is a change data subsequent to a play picture before skip, and if the first target protocol data is stored, the playback terminal may perform corresponding change on the play picture before skip, such as adding some picture layers and the like, based on the first target protocol data, play back a picture corresponding to the skip play time point, and play an audio corresponding to the skip play time point.

when a jump playing instruction is received, responding to that target protocol data of a jump playing time point indicated by the jump playing instruction is not stored in the playing cache region, and sending a jump loading request to the server, wherein the jump loading request carries the jump playing time point;

In this embodiment, when receiving a skip play instruction, the playback terminal checks and finds that the target protocol data at the skip play time point indicated by the skip play instruction is not stored in the play cache region, and at this time, the playback terminal cannot directly play back the target protocol data at the skip play time point, and needs to send a skip loading request to the server, and request the server to send data required for playing back the client side picture and audio corresponding to the skip play time point.

In this embodiment, in order to facilitate jump playback at any position, during the process of recording the target protocol data, screenshot images at some time points are also recorded as key frame images, and the selection rules of these time points may not be too close or too far, and a certain interval range is required. Therefore, after receiving the skip loading request, the server obtains a skip playing time point, at this time, the server can search for a first key frame image corresponding to the skip playing time point, wherein the first key frame image is a key frame image of which the time point is before the skip playing time point and is closest to the skip playing time point. The second target protocol data refers to the change data from the time point to the jump play time point.

In this embodiment, after the playback terminal acquires the first key frame image and the second target protocol data, it may perform corresponding changes, such as adding some image layers, to the first key frame image based on the second target protocol data, play back the image corresponding to the skipped playing time point, and play back the audio corresponding to the skipped playing time point.

The skip scheme provided by the embodiment does not need to load data from the head, and as the key frame images are arranged, only the corresponding first key frame image and the second target protocol data between the time point of the first key frame image and the skip playing time point need to be recorded, so that the skip is quicker.

In a possible embodiment, the method can be further implemented as the following steps:

and in response to the first target protocol data stored in the play cache region, clearing the data in the play cache region except the first target protocol data.

In this embodiment, in order to reduce the memory usage of the playback terminal and optimize the performance of the terminal, the data cached in the play cache region needs to be cleaned, and when the first target protocol data is stored in the play cache region, since the first target protocol data is only needed for the picture and the audio corresponding to the skip play time point, the data in the play cache region except for the first target protocol data can be cleaned.

In this embodiment, in order to reduce the memory usage of the playback terminal and optimize the performance of the terminal, the data cached in the play cache region needs to be cleaned, and when the target protocol data of the jump play time point is not stored in the play cache region, the first key frame image and the second target protocol data, which are needed by the picture and the audio corresponding to the jump play time point, need to be acquired from the server, so all the data in the play cache region can be cleaned.

In this embodiment, in order to reduce the memory usage of the playback terminal and optimize the terminal performance, the data cached in the play buffer area needs to be cleared, and the data to be cleared in the play buffer area may be cleared, for example, assuming that the first preset duration is 3s, and the timestamp of the data corresponding to the current playback picture is t1, the data to be cleared is the data whose timestamp is before t1-3 s.

and playing the subsequent frame data according to the playing time difference.

In this embodiment, the playback terminal may obtain a frame of image data to be rendered based on the target protocol data, and when receiving the multiple speed play instruction, may obtain a play rate indicated by the multiple speed play instruction, where the play rate indicated by the multiple speed play instruction may be represented by a multiple of an original play rate, such as 0.5, 1, 1.25, 1.5, and 2 times of the original play rate, and based on the play rate indicated by the multiple speed play instruction, may calculate a play time difference between the subsequent frames of data, and play the subsequent frames of data according to the play time difference, so as to implement the multiple speed play function. For example, assuming that the playback speed indicated by the multiple-speed playback command is 0.5 multiple speed, the calculated playback time difference is 2 times the original playback time difference, and the original playback time difference is the next frame data played after the interval of 0.1s, when the multiple-speed playback command indicates 0.5 multiple speed playback, the next frame data may be played after the interval of 0.2 s.

In a possible embodiment, step S103 of the above playback method, namely, based on the target protocol data, the portion for performing visual picture playback on the target protocol data may further include the following steps:

obtaining data of each frame to be rendered based on the target protocol data;

In this embodiment, each frame of data to be rendered may be obtained based on the visualization data in the target protocol data, such as each picture layer data, cursor layer data, and the like. When the resource use condition of the playback terminal is too high, such as CPU (Central processing Unit) and rendering time is too long, such as exceeding a preset time length, and a playback non-smoothness condition is met, in order to ensure the playback smoothness, picture similarity can be utilized to eliminate interframe redundancy, similar pictures are actively discarded, at least one frame of data with the similarity of target frame data exceeding a preset threshold value in each frame of data to be rendered is discarded, the target frame of data can be any frame of data to be rendered, only the left data is rendered, and due to the fact that the lost data and the left data have higher similarity, video display cannot be influenced even if the lost data and the left data are discarded, and only the left data are rendered, so that the rendered data volume is reduced, and the playback smoothness is ensured.

and sending a retransmission request to the server in response to the detection that a target data packet corresponding to the target protocol data is lost, wherein the retransmission request is used for indicating the server to resend the target data packet.

In this embodiment, the server may issue each data packet corresponding to the target protocol data to the playback terminal according to a packet sequence number or a time sequence, and if the playback terminal detects that the received data packets are discontinuous, it may be determined that the target data packets between the discontinuous data packets are lost, at this time, the playback terminal may actively send a retransmission request to the server to request the server to issue the target data packets again.

The packet loss retransmission in the embodiment is retransmission on demand, is more suitable for the application scene of video transmission, and does not increase too much extra bandwidth.

In this embodiment, the playback method further supports a screen capture function, a screen capture button may be disposed on the playback interface, and after the user clicks the screen capture button, the playback terminal may capture the current playback picture as a screen capture image and directly download the screen capture image to a local preset storage location.

and responding to the received interface adjusting instruction, and adjusting the current playback interface according to the instruction of the interface adjusting instruction.

In this embodiment, the interface adjustment instruction is used to instruct to adjust at least one of a size, a position, and interface display content of the current playback interface. For example, when the screen adjustment instruction indicates full-screen playing, the playback terminal may adjust the current playback interface to display a playback picture, and the playback picture occupies a full screen of the playback terminal; there are, of course, instructions to maximize display, minimize display, etc., and reference may be made to existing video players, not to mention here.

receiving a second key frame sent by the server;

In this embodiment, the playback terminal may, in response to receiving a playback instruction for the target protocol data, send a request message to the server, at the same time, the playback terminal will draw the initial playing interface of the target protocol data, when the server issues the target protocol data to the playback terminal, a second key frame is also sent downwards, usually the second key frame is the first key frame in the key frames corresponding to the target protocol data stored in the server, and of course, other key frames can also be sent downwards, without limitation, when the playback terminal enters the initial playing interface of the target protocol data and does not start playing back the target protocol data, the second key frame may be used as a play cover to be displayed, and when the user clicks a play button on the initial play interface to start playing, the target protocol data is played back on the play interface.

In this embodiment, the playback terminal may further display, on the initial play interface, basic video information, such as a user name of a remote operation user corresponding to the video, an instance of a remote operation server, a total video length, and the like, which is issued by the server and stored in the playback terminal.

In this embodiment, if the playback terminal does not receive the target protocol data sent by the server for more than a second preset time period, which indicates that a problem occurs in the network between the playback terminal and the server, resulting in a loading failure or slow loading, at this time, the playback terminal may pop up a prompt box, where the prompt box may display the prompt information "whether to reload" and the options "yes" and "no", and the user selects the option "yes", and the playback terminal may reload the corresponding data from the server.

In this embodiment, if the connection between the playback terminal and the server is disconnected, the playback terminal may implicitly reconnect to the server, continue to acquire subsequent data from the server after the connection is established, and continue playback from the current playback screen, instead of starting playback from the beginning.

In this embodiment, continuing playback from the current playback screen requires acquiring, from a server, a third key frame image corresponding to the time point of the current playback screen and third target protocol data between the time point of the third key frame image and the time point of the current playback screen; and playing back the picture and the audio corresponding to the current played back picture based on the third key frame image and the third target protocol data, and continuing playing back the subsequent picture and audio through the loaded subsequent data.

By way of example, fig. 2 shows an overall flowchart of a playback method of protocol data according to an embodiment of the present disclosure; it is assumed that the client uses the RDP protocol when remotely operating the server instance. As shown in fig. 2, after the playback terminal enters the play interface of the target protocol data, a play page is drawn, the size of the play interface is reset, basic video information, such as a user name of a remote operation user corresponding to the video, a server instance of the remote operation, total video length and the like, is loaded, part of the target protocol data and a second key frame image are loaded from the server, and if the loading is successful, the rendering playback can be started, and the second key frame is displayed as a play cover. If the play key of the play interface is started, the progress bar timer of the player is started, and if the pause key of the play interface is started, the progress bar timer of the player is closed. The time stamp of the target protocol data to be played can be updated according to the progress bar timer of the player, the target protocol data of the corresponding time stamp, such as audio data, picture layer data, size data, cursor layer data, media channel data and the like, can be obtained by processing, the data of the next frame to be rendered and played can be rendered and played according to the playing speed indicated by the double-speed playing instruction, the data of the next frame can be played, the next frame data can be played, a playback picture can be obtained, the current playback picture can be intercepted as a screen capture image in response to the received screen capture instruction, and the screen capture image can be stored. And updating the position of the playing progress bar after playing and rendering the next frame data, updating a progress bar timer of the player, and acquiring and playing the next frame data.

As shown in fig. 2, in the process of playback, a play progress bar may also be dragged to jump, and when a jump play instruction is received, in response to that first target protocol data between a play time point before the jump and a jump play time point indicated by the jump play instruction is stored in a play cache region, data in the play cache region except for the first target protocol data is cleared, and then direct jump is performed. Responding to that target protocol data of a jump playing time point indicated by the jump playing instruction is not stored in the playing cache region, clearing the data in the playing cache region, sending a jump loading request to the server, requesting the server to send a first key frame image corresponding to the jump playing time point and second target protocol data between the time point of the first key frame image and the jump playing time point, and directly jumping after data is obtained. When jumping, rendering and playing back pictures and audios corresponding to the jumping playing time points based on the played pictures before jumping and the first target protocol data, wherein the pictures and the audios are provided with data, and rendering and playing back the pictures and the audios corresponding to the jumping playing time points based on the first key frame images and the second target protocol data. And then starting or closing a progress bar timer of the player according to whether the playing state before jumping is to start or pause the playing, and further continuing to play the next frame of data according to the timing of the timer.

As shown in fig. 2, when the target protocol data is initially loaded from the server or loaded after being skipped, a prompt message may be output due to loading failure or loading timeout caused by poor network, and the prompt message is used to prompt reloading of the data, and when the prompt message displays a prompt box indicating whether to reload the data and options "reload" and "cancel" and the user selects the option "reload", the playback terminal draws a playing page, resets the size of the playing interface, loads basic video information, and reloads the target protocol data required during initial loading or the target protocol data to be loaded after being skipped.

It should be noted here that, assuming that the SSH protocol is used when the client remotely operates the server instance, the SSH protocol can be played back using xterm.js, the server needs to record protocol data including character data, that is, character data input and returned by the user during operation, and data related to pictures does not need to be produced, and compared with a conventional video player using picture drawing, the server has a very small volume, and only operation character data used by the user is needed during playing, and accordingly, operations such as jumping, loading, playing and the like can be performed faster; like the example of fig. 2, the RDP protocol also has the same functions of screenshot playback, speed doubling, cache cleaning and the like.

Fig. 3 shows a block diagram of a structure of a playback apparatus of protocol data according to an embodiment of the present disclosure. The apparatus may be implemented as part or all of an electronic device through software, hardware, or a combination of both.

As shown in fig. 3, the playback apparatus 300 of the protocol data includes a first transmitting module 301, a first receiving module 302, and a playback module 303.

A first sending module 301, configured to send, in response to receiving a playback instruction for target protocol data, a request message requesting downloading of the target protocol data to a server, where the target protocol data includes client original manipulation data and server instance original response data generated when a client remotely manipulates a server instance;

a first receiving module 302 configured to receive the target protocol data sent by the server;

a playback module 303 configured to perform visual picture playback and audio playback on the target protocol data based on the target protocol data.

In one possible implementation, the first receiving module 302 is configured to:

In a possible implementation manner, the portion of the first receiving module 302 that obtains the target protocol data to be loaded from the server in advance is configured to:

In one possible implementation, the playback module 303 is configured to:

acquiring preset data to be added;

In a possible embodiment, the apparatus further comprises:

In one possible implementation, the target protocol includes a remote desktop protocol RDP or a secure shell protocol SSH, the target protocol data includes audio data with a timestamp, and when the target protocol includes RDP, the target protocol data further includes picture layer data with a timestamp; when the target protocol comprises SSH, the target protocol data further comprises time-stamped character data, the apparatus further comprising:

In a possible embodiment, the apparatus further comprises:

In this embodiment, the playback device of the protocol data corresponds to and is consistent with the playback method of the protocol data, and specific details may refer to the description of the playback method of the protocol data, which is not described herein again.

The present disclosure also discloses an electronic device, and fig. 4 shows a block diagram of the electronic device according to an embodiment of the present disclosure.

As shown in fig. 4, the electronic device 400 includes a memory 401 and a processor 402, wherein the memory 401 is configured to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 402 to implement a method according to an embodiment of the disclosure.

As shown in fig. 5, the computer system 500 includes a processing unit 501 that can execute various processes in the above-described embodiments according to a program stored in a Read Only Memory (ROM)502 or a program loaded from a storage section 508 into a Random Access Memory (RAM) 503. In the RAM 503, various programs and data necessary for the operation of the system 500 are also stored. The processing unit 501, the ROM 502, and the RAM 503 are connected to each other by a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.

The following components are connected to the I/O interface 505: an input portion 506 including a keyboard, a mouse, and the like; an output portion 507 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 508 including a hard disk and the like; and a communication section 509 including a network interface card such as a LAN card, a modem, or the like. The communication section 509 performs communication processing via a network such as the internet. The driver 510 is also connected to the I/O interface 505 as necessary. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 510 as necessary, so that a computer program read out therefrom is mounted into the storage section 508 as necessary. The processing unit 501 may be implemented as a CPU, a GPU, a TPU, an FPGA, an NPU, or other processing units.

In particular, the above described methods may be implemented as computer software programs according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising computer instructions that, when executed by a processor, implement the method steps described above. In such an embodiment, the computer program product may be downloaded and installed from a network through the communication section 509, and/or installed from the removable medium 511.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units or modules described in the embodiments of the present disclosure may be implemented by software or by programmable hardware. The units or modules described may also be provided in a processor, and the names of the units or modules do not in some cases constitute a limitation of the units or modules themselves.

As another aspect, the present disclosure also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the electronic device or the computer system in the above embodiments; or it may be a separate computer readable storage medium not incorporated into the device. The computer readable storage medium stores one or more programs for use by one or more processors in performing the methods described in the present disclosure.

The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention in the present disclosure is not limited to the specific combination of the above-mentioned features, but also encompasses other embodiments in which any combination of the above-mentioned features or their equivalents is possible without departing from the inventive concept. For example, the above features and (but not limited to) the features disclosed in this disclosure having similar functions are replaced with each other to form the technical solution.

Claims

1. A method of playback of protocol data, comprising:

receiving the target protocol data sent by the server;

2. The method of claim 1, wherein the receiving the target protocol data sent by the server comprises:

3. The method of claim 2, wherein the obtaining target protocol data to be loaded subsequently from the server in advance comprises:

4. The method of claim 1, wherein the performing visual and audio playback of the target protocol data based on the target protocol data comprises:

5. The method of claim 1, wherein the performing visual and audio playback of the target protocol data based on the target protocol data comprises:

acquiring preset data to be added;

6. The method of claim 1, wherein the method further comprises:

7. The method of claim 1, wherein,

when a jump playing instruction is received, responding to that target protocol data of a jump playing time point indicated by the jump playing instruction is not stored in a cache, and sending a jump loading request to the server, wherein the jump loading request carries the jump playing time point;

8. The method of claim 6, wherein the method further comprises:

9. The method of claim 7, wherein the method further comprises:

10. The method of claim 1, wherein the target protocol comprises a Remote Desktop Protocol (RDP) or a secure Shell protocol (SSH), the target protocol data comprises time-stamped audio data, and when the target protocol comprises RDP, the target protocol data further comprises time-stamped picture layer data; when the target protocol comprises SSH, the target protocol data further comprises time-stamped character data, the method further comprising:

11. The method of claim 1, wherein the method further comprises:

and playing the subsequent frame data according to the playing time difference.

12. The method of claim 1, wherein the method further comprises:

obtaining data of each frame to be rendered based on the target protocol data;

13. The method of claim 1, wherein the method further comprises:

14. An electronic device comprising a memory and a processor; wherein the memory is to store one or more computer instructions, wherein the one or more computer instructions are to be executed by the processor to implement the method steps of any of claims 1 to 13.