音视频的处理方法、装置及存储介质Audio and video processing method, device and storage medium
本申请要求于2020年11月18日提交中国专利局、申请号为202011292718.4、申请名称为“音视频的处理方法、装置及存储介质”的中国专利申请的优先权,其全部内容通过引用结合在本申请中。This application claims the priority of the Chinese patent application with the application number 202011292718.4 and the application title "Audio and video processing method, device and storage medium", which was submitted to the China Patent Office on November 18, 2020, the entire contents of which are incorporated by reference in in this application.
技术领域technical field
本申请涉及多媒体技术领域,尤其涉及一种音视频的处理方法、装置及存储介质。The present application relates to the field of multimedia technologies, and in particular, to an audio and video processing method, device, and storage medium.
背景技术Background technique
目前,音视频播放一般采用基于音视频分片的下载模式。例如,通过互联网向终端设备提供音视频的下载方式中,服务器将同一个音视频的不同码率的版本切分为预设长度的分片,对每个分片进行封装。At present, audio and video playback generally adopts a download mode based on audio and video segments. For example, in the download method of providing audio and video to terminal devices through the Internet, the server divides versions of the same audio and video with different bit rates into fragments of preset lengths, and encapsulates each fragment.
在终端设备需要对音视频进行播放时,向服务器发送连接请求,服务器在接收到连接请求后,通过与终端设备的多次交互进行带宽探测,从而确定合适的网络通信初始窗口大小。服务器在接收到终端设备发送的音视频播放请求后,根据确定出的网络通信初始窗口大小传输音视频的初始分片。When the terminal device needs to play audio and video, it sends a connection request to the server. After receiving the connection request, the server performs bandwidth detection through multiple interactions with the terminal device to determine an appropriate initial window size for network communication. After receiving the audio and video playback request sent by the terminal device, the server transmits the initial fragmentation of the audio and video according to the determined initial window size of the network communication.
但是在上述方法中,服务器在客户端发送连接请求后,需要经过多个网络通信往返时延(Round trip time,RTT)进行带宽探测,导致音视频的起播时延较长,音视频播放效果不佳。However, in the above method, after the server sends a connection request from the client, it needs to go through multiple network communication round trip delays (Round trip time, RTT) for bandwidth detection, resulting in a long delay in the start of audio and video, and the effect of audio and video playback. not good.
发明内容SUMMARY OF THE INVENTION
有鉴于此,提出了一种音视频的处理方法、装置及存储介质,在终端设备向服务器发送音视频播放请求时,主动上报用于指示终端设备当前的接入网络的网络质量情况的网络质量信息,避免了相关技术中服务器与终端设备之间多次交互进行网络带宽探测的情况,从而缩短了音视频的起播时延,保证了音视频的播放效果。In view of this, an audio and video processing method, device and storage medium are proposed. When a terminal device sends an audio and video playback request to a server, it actively reports the network quality used to indicate the current network quality of the terminal device's access network. information, avoids multiple interactions between the server and the terminal device for network bandwidth detection in the related art, thereby shortening the playback delay of audio and video, and ensuring the playback effect of audio and video.
第一方面,本申请的实施例提供了一种音视频的处理方法,用于终端设备中,该方法包括:In a first aspect, an embodiment of the present application provides a method for processing audio and video, which is used in a terminal device, and the method includes:
接收播放指令,播放指令用于指示开始播放目标音视频;Receive a playback instruction, and the playback instruction is used to instruct to start playing the target audio and video;
根据播放指令,向服务器发送携带有网络质量信息的音视频播放请求,网络质量信息用于指示终端设备当前的接入网络的网络质量情况,音视频播放请求用于指示服务器返回目标音视频的音视频数据;According to the playback instruction, an audio and video playback request carrying network quality information is sent to the server. The network quality information is used to indicate the current network quality of the terminal device accessing the network, and the audio and video playback request is used to instruct the server to return the audio and video of the target audio and video. video data;
下载并播放目标音视频的音视频数据。Download and play the audio and video data of the target audio and video.
在该实现方式中,终端设备接收到目标音视频的播放指令后,向服务器发送携带有网络质量信息的音视频播放请求,音视频播放请求用于指示服务器返回目标音视频的音视频数据;终端设备下载并播放服务器返回的目标音视频的音视频数据;即在终端设备向服务器发送音视频播放请求时,主动上报终端设备当前的接入网络的网络质量信息,避免了相关技术中服务器与终端设备之间多次交互进行网络带宽探测的情况, 从而缩短了音视频的起播时延,保证了音视频的播放效果。In this implementation, after receiving the target audio and video playback instruction, the terminal device sends an audio and video playback request carrying network quality information to the server, and the audio and video playback request is used to instruct the server to return the target audio and video audio and video data; the terminal The device downloads and plays the audio and video data of the target audio and video returned by the server; that is, when the terminal device sends an audio and video playback request to the server, it actively reports the network quality information of the current access network of the terminal device, avoiding the need for the server and the terminal in the related art. In the case of multiple interactions between devices for network bandwidth detection, the playback delay of audio and video is shortened and the playback effect of audio and video is guaranteed.
在一种可能的实现方式中,网络质量信息包括终端设备当前的接入网络的网络带宽。In a possible implementation manner, the network quality information includes the network bandwidth of the current access network of the terminal device.
在该实现方式中,终端设备主动上报终端设备当前的接入网络的网络带宽,避免了相关技术中网络带宽需要服务器经过多个RTT探测才能获取的情况。In this implementation manner, the terminal device actively reports the network bandwidth of the current access network of the terminal device, which avoids the situation in the related art that the network bandwidth needs to be acquired by the server through multiple RTT detections.
在另一种可能的实现方式中,音视频数据包括至少一个音视频分片,该方法还包括:In another possible implementation manner, the audio and video data includes at least one audio and video segment, and the method further includes:
在接收到音视频分片后,向服务器发送多个确认应答(Acknowledge,ACK)消息,ACK消息用于指示终端设备已成功接收到音视频分片。After receiving the audio and video fragments, multiple acknowledgment (Acknowledge, ACK) messages are sent to the server, and the ACK messages are used to indicate that the terminal device has successfully received the audio and video fragments.
在该实现方式中,为了防止ACK消息丢包,终端设备在收到音视频分片后,通过发送多个ACK消息,来避免服务器由于未收到ACK消息而进行降速的情况,只有在多个ACK消息同时丢包的情况下,服务器才会主动下降播放码率,这种方法极大地降低了音视频播放过程发生卡顿的情况。In this implementation, in order to prevent the ACK message from being lost, the terminal device sends multiple ACK messages after receiving the audio and video segments to avoid the situation that the server slows down due to not receiving the ACK message. When all ACK messages are lost at the same time, the server will actively reduce the playback bit rate. This method greatly reduces the situation that the audio and video playback process is stuck.
在另一种可能的实现方式中,在接收到音视频分片后,向服务器发送多个ACK消息,包括:In another possible implementation, after receiving the audio and video segments, multiple ACK messages are sent to the server, including:
在接收到音视频分片后,当接入网络当前的信号强度和/或网络时延满足预设条件时,向服务器发送多个ACK消息;After receiving the audio and video segments, when the current signal strength and/or network delay of the access network meet the preset conditions, send multiple ACK messages to the server;
其中,预设条件包括信号强度小于预设强度阈值,和/或网络时延大于预设时延阈值。Wherein, the preset condition includes that the signal strength is less than the preset strength threshold, and/or the network delay is greater than the preset delay threshold.
在该实现方式中,当接入网络当前的信号强度小于预设强度阈值,和/或网络时延大于预设时延阈值时,终端设备向服务器发送多个ACK消息,即终端设备能够根据当前的网络质量情况灵活地控制ACK消息的发送方式,在终端设备当前的接入网络的网络质量较差的情况下才会发送多个ACK消息,进一步提高了终端设备的智能性和灵活性。In this implementation manner, when the current signal strength of the access network is less than the preset strength threshold, and/or the network delay is greater than the preset delay threshold, the terminal device sends multiple ACK messages to the server, that is, the terminal device can It can flexibly control the sending mode of the ACK message according to the network quality situation of the terminal device, and only send multiple ACK messages when the network quality of the current access network of the terminal device is poor, which further improves the intelligence and flexibility of the terminal device.
在另一种可能的实现方式中,ACK消息的发送个数与接入网络的信号强度呈负相关关系。In another possible implementation manner, the number of sent ACK messages is negatively correlated with the signal strength of the access network.
在该实现方式中,ACK消息的发送个数与接入网络的信号强度呈负相关关系,即接入网络的信号强度越高,ACK消息的冗余量越少。In this implementation manner, the number of ACK messages sent is negatively correlated with the signal strength of the access network, that is, the higher the signal strength of the access network, the less redundancy of the ACK message.
第二方面,本申请的实施例提供了一种音视频的处理方法,用于服务器中,该方法包括:In a second aspect, the embodiments of the present application provide a method for processing audio and video, which is used in a server, and the method includes:
接收终端设备发送的携带有网络质量信息的音视频播放请求,网络质量信息用于指示终端设备当前的接入网络的网络质量情况;Receive an audio and video playback request that carries network quality information sent by the terminal device, where the network quality information is used to indicate the current network quality of the terminal device's access network;
根据音视频播放请求,向终端设备返回目标音视频的音视频数据。Returns the audio and video data of the target audio and video to the terminal device according to the audio and video playback request.
在一种可能的实现方式中,网络质量信息包括接入网络的网络带宽,根据音视频播放请求,向终端设备返回目标音视频的音视频数据,包括:In a possible implementation manner, the network quality information includes the network bandwidth of the access network, and according to the audio and video playback request, the audio and video data of the target audio and video is returned to the terminal device, including:
根据接入网络的网络带宽和服务器的网络带宽,确定通信窗口大小和播放码率;According to the network bandwidth of the access network and the network bandwidth of the server, determine the size of the communication window and the playback code rate;
根据通信窗口大小和播放码率,向终端设备返回目标音视频的音视频数据。According to the size of the communication window and the playing bit rate, the audio and video data of the target audio and video are returned to the terminal device.
在该实现方式中,服务器根据终端设备主动上报的网络质量信息确定通信窗口大小和播放码率,根据通信窗口大小和播放码率返回目标音视频的音视频数据,尽量保 障目标音视频的第一个音视频分片尽快发送至终端设备,跳过了相关技术中的TCP协议的慢启动过程,实现了音视频播放的快速启动。In this implementation, the server determines the size of the communication window and the playback code rate according to the network quality information actively reported by the terminal device, and returns the audio and video data of the target audio and video according to the size of the communication window and the playback code rate, so as to ensure the first quality of the target audio and video as much as possible. Each audio and video segment is sent to the terminal device as soon as possible, skipping the slow start process of the TCP protocol in the related art, and realizing the fast start of audio and video playback.
在另一种可能的实现方式中,音视频数据包括至少一个音视频分片,该方法还包括:In another possible implementation manner, the audio and video data includes at least one audio and video segment, and the method further includes:
接收终端设备发送的音视频分片对应的多个ACK消息,ACK消息用于指示终端设备已成功接收到音视频分片。Receive multiple ACK messages corresponding to the audio and video segments sent by the terminal device, where the ACK messages are used to indicate that the terminal device has successfully received the audio and video segments.
在另一种可能的实现方式中,该方法还包括:In another possible implementation, the method further includes:
在接收到同一个音视频分片对应的多个ACK消息的情况下,去除重复的ACK消息。In the case of receiving multiple ACK messages corresponding to the same audio and video segment, the duplicate ACK messages are removed.
在该实现方式中,服务器在接收到同一个音视频分片对应的多个ACK消息的情况下,去除重复的ACK消息,实现了ACK消息的去冗余处理。In this implementation manner, in the case of receiving multiple ACK messages corresponding to the same audio and video segment, the server removes the duplicate ACK messages, thereby realizing the de-redundancy processing of the ACK messages.
第三方面,本申请的实施例提供了一种音视频的处理装置,该装置包括至少一个单元,至少一个单元用于实现上述第一方面或第一方面中的任意一种可能的实现方式所提供的方法。In a third aspect, embodiments of the present application provide an apparatus for processing audio and video, the apparatus includes at least one unit, and the at least one unit is configured to implement the first aspect or any one of the possible implementation manners of the first aspect. provided method.
第四方面,本申请的实施例提供了一种音视频的处理装置,该装置包括至少一个单元,至少一个单元用于实现上述第二方面或第二方面中的任意一种可能的实现方式所提供的方法。In a fourth aspect, an embodiment of the present application provides an audio and video processing device, the device includes at least one unit, and the at least one unit is configured to implement the second aspect or any one of the possible implementation manners of the second aspect. provided method.
第五方面,本申请的实施例提供了一种音视频的处理装置,装置包括:处理器;用于存储处理器可执行指令的存储器;其中,处理器被配置为执行指令时实现上述第一方面或第一方面中的任意一种可能的实现方式所提供的方法。In a fifth aspect, embodiments of the present application provide an apparatus for processing audio and video, the apparatus comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above-mentioned first step when executing the instructions A method provided by any one of the possible implementations of the aspect or the first aspect.
第六方面,本申请的实施例提供了一种音视频的处理装置,装置包括:处理器;用于存储处理器可执行指令的存储器;其中,处理器被配置为执行指令时实现上述第二方面或第二方面中的任意一种可能的实现方式所提供的方法。In a sixth aspect, embodiments of the present application provide an apparatus for processing audio and video, the apparatus comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above-mentioned second when executing the instructions A method provided by any one of the possible implementations of the aspect or the second aspect.
第七方面,本申请的实施例提供了一种计算机程序产品,包括计算机可读代码,或者承载有计算机可读代码的非易失性计算机可读存储介质,当计算机可读代码在电子设备中运行时,电子设备中的处理器执行上述第一方面或者第一方面中的任意一种可能的实现方式所提供的方法。In a seventh aspect, embodiments of the present application provide a computer program product, comprising computer-readable codes, or a non-volatile computer-readable storage medium carrying computer-readable codes, when the computer-readable codes are stored in an electronic device When running, the processor in the electronic device executes the method provided by the first aspect or any one of the possible implementation manners of the first aspect.
第八方面,本申请的实施例提供了一种计算机程序产品,包括计算机可读代码,或者承载有计算机可读代码的非易失性计算机可读存储介质,当计算机可读代码在电子设备中运行时,电子设备中的处理器执行上述第二方面或者第二方面中的任意一种可能的实现方式所提供的方法。In an eighth aspect, embodiments of the present application provide a computer program product, comprising computer-readable codes, or a non-volatile computer-readable storage medium carrying computer-readable codes, when the computer-readable codes are stored in an electronic device When running, the processor in the electronic device executes the method provided by the second aspect or any one of the possible implementation manners of the second aspect.
第九方面,本申请的实施例提供了一种非易失性计算机可读存储介质,其上存储有计算机程序指令,计算机程序指令被处理器执行时实现上述第一方面或者第一方面中的任意一种可能的实现方式所提供的方法。In a ninth aspect, embodiments of the present application provide a non-volatile computer-readable storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the first aspect or the first aspect is implemented. A method provided by any of the possible implementations.
第十方面,本申请的实施例提供了一种非易失性计算机可读存储介质,其上存储有计算机程序指令,计算机程序指令被处理器执行时实现上述第二方面或第二方面中的任意一种可能的实现方式所提供的方法。In a tenth aspect, embodiments of the present application provide a non-volatile computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, implement the second aspect or the second aspect above. A method provided by any of the possible implementations.
附图说明Description of drawings
包含在说明书中并且构成说明书的一部分的附图与说明书一起示出了本申请的示例性实施例、特征和方面,并且用于解释本申请的原理。The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features and aspects of the application and together with the description, serve to explain the principles of the application.
图1示出相关技术中不同的影响因素在视频播放过程中的位置示意图。FIG. 1 shows a schematic diagram of the positions of different influencing factors in the video playback process in the related art.
图2示出相关技术中RTMP的执行流程的流程示意图。FIG. 2 shows a schematic flow chart of an execution flow of RTMP in the related art.
图3示出了本申请一个示例性实施例提供的音视频的处理***的结构示意图。FIG. 3 shows a schematic structural diagram of an audio and video processing system provided by an exemplary embodiment of the present application.
图4示出了本申请一个示例性实施例提供的音视频的处理方法的流程图。FIG. 4 shows a flowchart of a method for processing audio and video provided by an exemplary embodiment of the present application.
图5示出相关技术中音视频的处理方法的流程图。FIG. 5 shows a flowchart of a method for processing audio and video in the related art.
图6示出了本申请另一个示例性实施例提供的音视频的处理方法的流程图。FIG. 6 shows a flowchart of an audio and video processing method provided by another exemplary embodiment of the present application.
图7示出了本申请另一个示例性实施例提供的音视频的处理方法的流程图。FIG. 7 shows a flowchart of a method for processing audio and video provided by another exemplary embodiment of the present application.
图8示出了本申请一个示例性实施例提供的音视频的处理装置的框图。FIG. 8 shows a block diagram of an apparatus for processing audio and video provided by an exemplary embodiment of the present application.
图9示出了本申请一个示例性实施例提供的音视频的处理装置的框图。FIG. 9 shows a block diagram of an audio and video processing apparatus provided by an exemplary embodiment of the present application.
图10示出了本申请一个实施例提供的终端设备的结构示意图。FIG. 10 shows a schematic structural diagram of a terminal device provided by an embodiment of the present application.
图11示出了本申请一个实施例提供的服务器的结构示意图。FIG. 11 shows a schematic structural diagram of a server provided by an embodiment of the present application.
具体实施方式Detailed ways
以下将参考附图详细说明本申请的各种示例性实施例、特征和方面。附图中相同的附图标记表示功能相同或相似的元件。尽管在附图中示出了实施例的各种方面,但是除非特别指出,不必按比例绘制附图。Various exemplary embodiments, features and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numbers in the figures denote elements that have the same or similar functions. While various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.
在这里专用的词“示例性”意为“用作例子、实施例或说明性”。这里作为“示例性”所说明的任何实施例不必解释为优于或好于其它实施例。The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.
另外,为了更好的说明本申请,在下文的具体实施方式中给出了众多的具体细节。本领域技术人员应当理解,没有某些具体细节,本申请同样可以实施。在一些实例中,对于本领域技术人员熟知的方法、手段、元件和电路未作详细描述,以便于凸显本申请的主旨。In addition, in order to better illustrate the present application, numerous specific details are given in the following detailed description. It should be understood by those skilled in the art that the present application may be practiced without certain specific details. In some instances, methods, means, components and circuits well known to those skilled in the art have not been described in detail so as not to obscure the subject matter of the present application.
在音视频传输技术中,如何反馈网络质量情况,并根据网络质量情况对音视频的播放码率进行调整,从而获取更佳的流畅度和更短的起播时延,是音视频传输技术中的关键。以视频传输为例,在视频播放过程中,影响播放效果的多种影响因素包括但不限于:请求的建链时间、起始分片的返回时间、分片大小及分片下载策略等。不同的影响因素在视频播放过程中的位置示意图如图1所示。其中,在预先缓冲过中,终端设备发送播放请求后开始下载分片,下载数据量达到预先缓冲阈值后终端设备开始播放视频,该过程影响首次缓冲时间。在分片下载过中,一个视频分为不同品质的多个分片,分布在不同的服务器上调度,以分片为单位进行下载和播放,该过程影响卡顿事件发生。若终端设备采用“尽快下载完”当前播放分片的策略,充分利用当前实际可用带宽,边播边下,该过程影响流量模型。为降低服务器压力,终端设备采用“间歇下载”的策略即不会一直下载尚未播放的分片,若剩余播放时间大于暂停缓冲阈值则暂停下载,若剩余播放时间小于暂停缓冲阈值则重新下载,该过程影响卡顿事件发生。In the audio and video transmission technology, how to feedback the network quality and adjust the playback bit rate of the audio and video according to the network quality, so as to obtain better fluency and shorter start-up delay, is an important issue in the audio and video transmission technology. key. Taking video transmission as an example, in the process of video playback, various influencing factors that affect the playback effect include but are not limited to: the requested chain establishment time, the return time of the initial segment, the segment size, and the segment download strategy. A schematic diagram of the positions of different influencing factors during video playback is shown in Figure 1. Wherein, in the pre-buffering process, the terminal device starts to download segments after sending a play request, and the terminal device starts to play the video after the amount of downloaded data reaches the pre-buffering threshold, which affects the first buffering time. In the fragmented download process, a video is divided into multiple fragments of different qualities, which are distributed and scheduled on different servers, and downloaded and played in units of fragments. This process affects the occurrence of stuck events. If the terminal device adopts the strategy of "downloading the currently playing segment as soon as possible", making full use of the currently available bandwidth, and downloading while playing, this process affects the traffic model. In order to reduce the pressure on the server, the terminal device adopts the "intermittent download" strategy, that is, it will not download the unplayed segments all the time. If the remaining playback time is greater than the pause buffer threshold, the download will be paused, and if the remaining playback time is less than the pause buffer threshold, the download will be re-downloaded. Process affects the occurrence of Caton events.
为了解决上述技术问题,相关技术中通常采用内容分发网络(Content Delivery Network,CDN)、实时消息传输协议(Real-Time messaging protocol,RTMP)、基于超文本传输协议(Hypertext Transfer Protocol,HTTP)的动态自适应流媒体(Dynamic Adaptive Streaming over HTTP,DASH)协议等技术。其中,RTMP为直播场景下音视频播放的传输协议。DASH协议为点播场景下音视频播放的主要协议。服务器存储多种清晰度的编码,根据网络情况动态选择。在一个示意性的例子中,如图2所示,RTMP的执行流程包括但不限于以下几个步骤。步骤201,终端设备发送连接请求至服务器;步骤202,服务器根据自身的带宽大小确定发送内容消息的通信窗口大小,发送通信窗口大小发送至终端设备;步骤203,服务器发送设置的带宽信息至终端设备;步骤204,终端设备根据当前接入网络的网络带宽和服务器发送的带宽信息确定协商的通信窗口大小,发送协商的通信窗口大小至服务器,从而协商完成两者之间传输数据的通信窗口大小;步骤205,终端设备发起音视频播放请求至服务器;步骤206,服务器发送内容块大小的确认消息至终端设备;步骤207,服务器发送播放操作执行成功消息至终端设备;步骤208,服务器发送音视频数据至终端设备。In order to solve the above-mentioned technical problems, related technologies usually adopt Content Delivery Network (CDN), Real-Time Messaging Protocol (RTMP), dynamic based on Hypertext Transfer Protocol (HTTP). Adaptive streaming media (Dynamic Adaptive Streaming over HTTP, DASH) protocol and other technologies. Among them, RTMP is a transmission protocol for audio and video playback in a live broadcast scenario. The DASH protocol is the main protocol for audio and video playback in on-demand scenarios. The server stores codes of various resolutions and dynamically selects them according to network conditions. In an illustrative example, as shown in FIG. 2 , the execution flow of RTMP includes but is not limited to the following steps. Step 201, the terminal device sends a connection request to the server; Step 202, the server determines the size of the communication window for sending content messages according to its own bandwidth, and sends the size of the communication window to the terminal device; Step 203, the server sends the set bandwidth information to the terminal device Step 204, the terminal device determines the negotiated communication window size according to the network bandwidth of the current access network and the bandwidth information sent by the server, and sends the negotiated communication window size to the server, thereby negotiating and completing the communication window size of the transmission data between the two; Step 205, the terminal device initiates an audio and video playback request to the server; Step 206, the server sends a confirmation message of the size of the content block to the terminal device; Step 207, the server sends a playback operation execution success message to the terminal device; Step 208, The server sends audio and video data to the terminal device.
上述的音视频传输技术中,服务器在终端设备发起连接请求后,通常需要至少2~3个RTT进行网络带宽探测。In the above audio and video transmission technology, after the terminal device initiates a connection request, the server usually needs at least 2 to 3 RTTs for network bandwidth detection.
而本申请实施例提供了一种音视频的处理方法、装置及存储介质,在终端设备向服务器发送音视频播放请求时,主动上报用于指示终端设备当前的接入网络的网络质量情况的网络质量信息,避免了相关技术中服务器与终端设备之间多次交互进行网络带宽探测的情况,从而缩短了音视频的起播时延,保证了音视频的播放效果。However, the embodiments of the present application provide an audio and video processing method, device, and storage medium. When a terminal device sends an audio and video playback request to a server, it actively reports a network for indicating the network quality of the current access network of the terminal device. The quality information avoids the situation of multiple interactions between the server and the terminal device for network bandwidth detection in the related art, thereby shortening the playback delay of audio and video, and ensuring the playback effect of audio and video.
首先,对本申请涉及的应用场景进行介绍。First, the application scenarios involved in this application are introduced.
请参考图3,其示出了本申请一个示例性实施例提供的音视频的处理***的结构示意图。该***包括终端设备120和服务器140。Please refer to FIG. 3 , which shows a schematic structural diagram of an audio and video processing system provided by an exemplary embodiment of the present application. The system includes a terminal device 120 and a server 140 .
终端设备120中运行有音视频客户端。音视频客户端是用于播放音视频的软件应用,用户可以通过音视频客户端播放音视频。The terminal device 120 runs an audio and video client. The audio and video client is a software application for playing audio and video. Users can play audio and video through the audio and video client.
终端设备120用于通过音视频客户端向服务器140发送音视频播放请求,接收服务器140返回的音视频数据,在音视频客户端内完成对音视频的播放。比如,终端设备120为手机,车载终端,平板电脑,电子书阅读器,动态影像专家压缩标准音频层面3(Moving Picture Experts Group Audio Layer III,MP3)播放器,动态影像专家压缩标准音频层面4(Moving Picture Experts Group Audio Layer IV,MP4)播放器,笔记本电脑,膝上型便携计算机和台式计算机等。本申请实施例对终端设备120的类型不加以限定。The terminal device 120 is configured to send an audio and video playback request to the server 140 through the audio and video client, receive the audio and video data returned by the server 140, and complete the audio and video playback in the audio and video client. For example, the terminal device 120 is a mobile phone, a car terminal, a tablet computer, an e-book reader, a moving picture expert compression standard audio layer 3 (Moving Picture Experts Group Audio Layer III, MP3) player, and a moving picture expert compression standard audio layer 4 ( Moving Picture Experts Group Audio Layer IV, MP4) Players, Notebooks, Laptops and Desktops, etc. This embodiment of the present application does not limit the type of the terminal device 120 .
可选的,终端设备120和服务器140之间建立有通信连接,该通信连接可以是有线网络或无线网络。Optionally, a communication connection is established between the terminal device 120 and the server 140, and the communication connection may be a wired network or a wireless network.
该服务器140也称媒体内容服务器,用于在接收到终端设备120发送的音视频播放请求后,返回音视频数据。比如,该服务器140为CDN服务器。The server 140 is also called a media content server, and is configured to return audio and video data after receiving the audio and video playback request sent by the terminal device 120 . For example, the server 140 is a CDN server.
可选地,服务器140为音频服务器或者视频服务器。Optionally, the server 140 is an audio server or a video server.
在本申请实施例中,终端设备120用于接收播放指令,播放指令用于指示开始播放目标音视频;根据播放指令,向服务器140发送携带有网络质量信息的音视频播放 请求,网络质量信息用于指示终端设备120当前的接入网络的网络质量情况,音视频播放请求用于指示服务器140返回目标音视频的音视频数据;下载并播放目标音视频的音视频数据。In the embodiment of the present application, the terminal device 120 is used to receive a playback instruction, and the playback instruction is used to instruct to start playing the target audio and video; In order to indicate the network quality of the current access network of the terminal device 120, the audio and video playback request is used to instruct the server 140 to return the audio and video data of the target audio and video; download and play the audio and video data of the target audio and video.
下面,采用几个示例性实施例对本申请实施例提供的音视频的处理方法进行介绍。Hereinafter, the audio and video processing methods provided by the embodiments of the present application will be introduced by using several exemplary embodiments.
请参考图4,其示出了本申请一个示例性实施例提供的音视频的处理方法的流程图,本实施例以该方法用于图3所示的终端设备中来举例说明。该方法包括但不限于以下几个步骤。Please refer to FIG. 4 , which shows a flowchart of an audio and video processing method provided by an exemplary embodiment of the present application. This embodiment is illustrated by using the method in the terminal device shown in FIG. 3 . The method includes but is not limited to the following steps.
步骤401,终端设备接收播放指令,播放指令用于指示开始播放目标音视频。In step 401, the terminal device receives a play instruction, where the play instruction is used to instruct to start playing the target audio and video.
可选的,终端设备显示音视频客户端的用户界面,用户界面中包括目标音视频的播放控件。终端设备在接收到作用于播放控件上的用户操作信号即接收到播放指令的情况下,执行步骤402。Optionally, the terminal device displays a user interface of the audio and video client, and the user interface includes playback controls of the target audio and video. When the terminal device receives a user operation signal acting on the playback control, that is, receives a playback instruction, step 402 is performed.
可选的,音视频客户端的用户界面中显示有多个音视频各自对应的播放控件。目标音视频为多个音视频中的任意一个。Optionally, the user interface of the audio and video client displays a plurality of audio and video playback controls corresponding to each other. The target audio and video is any one of multiple audio and video.
步骤402,终端设备根据播放指令,向服务器发送携带有网络质量信息的音视频播放请求。Step 402: The terminal device sends an audio and video playback request carrying network quality information to the server according to the playback instruction.
其中,网络质量信息用于指示终端设备当前的接入网络的网络质量情况,音视频播放请求用于指示服务器返回目标音视频的音视频数据。The network quality information is used to indicate the current network quality of the terminal device accessing the network, and the audio and video playback request is used to instruct the server to return the audio and video data of the target audio and video.
可选的,终端设备在接收到播放指令之前,获取终端设备当前的接入网络的网络质量信息。终端设备在接收到播放指令后向服务器发送音视频播放请求,该音视频播放请求携带有网络质量信息。即在终端设备向服务器发送音视频播放请求的时候,根据***信息和历史信息主动上报网络质量信息。Optionally, before receiving the play instruction, the terminal device acquires network quality information of the current access network of the terminal device. After receiving the playback instruction, the terminal device sends an audio and video playback request to the server, where the audio and video playback request carries network quality information. That is, when the terminal device sends an audio and video playback request to the server, it actively reports network quality information according to system information and historical information.
其中,网络质量信息用于指示终端设备当前的接入网络的网络质量情况。网络质量信息可以包括终端设备当前的接入网络的网络带宽。网络质量信息还可以包括接入网络当前的信号强度和/或网络时延。Wherein, the network quality information is used to indicate the network quality of the current access network of the terminal device. The network quality information may include the network bandwidth of the current access network of the terminal device. The network quality information may also include the current signal strength and/or network delay of the access network.
可选的,音视频播放请求携带有目标音视频的标识,音视频播放请求用于指示服务器返回与目标音视频的标识对应的音视频数据。Optionally, the audio and video playback request carries the identification of the target audio and video, and the audio and video playback request is used to instruct the server to return audio and video data corresponding to the identification of the target audio and video.
步骤403,服务器接收终端设备发送的携带有网络质量信息的音视频播放请求。Step 403: The server receives an audio and video playback request that carries network quality information and is sent by the terminal device.
其中,网络质量信息用于指示终端设备当前的接入网络的网络质量情况。Wherein, the network quality information is used to indicate the network quality of the current access network of the terminal device.
服务器接收到终端设备发送的音视频播放请求后,获取音视频播放请求中携带的网络质量信息。After receiving the audio and video playback request sent by the terminal device, the server obtains the network quality information carried in the audio and video playback request.
步骤404,服务器根据音视频播放请求,向终端设备返回目标音视频的音视频数据。Step 404, the server returns the audio and video data of the target audio and video to the terminal device according to the audio and video playback request.
可选地,服务器根据网络质量信息确定通信窗口大小和播放码率;根据通信窗口大小和播放码率,向终端设备返回目标音视频的音视频数据。Optionally, the server determines the size of the communication window and the playback code rate according to the network quality information; and returns the audio and video data of the target audio and video to the terminal device according to the size of the communication window and the playback code rate.
可选的,音视频数据包括至少一个音视频分片,服务器向终端设备依次返回目标音视频的至少一个音视频分片。Optionally, the audio and video data includes at least one audio and video segment, and the server sequentially returns at least one audio and video segment of the target audio and video to the terminal device.
步骤405,终端设备下载并播放目标音视频的音视频数据。 Step 405, the terminal device downloads and plays the audio and video data of the target audio and video.
可选地,终端设备接收服务器依次返回的目标音视频的至少一个音视频分片。对 于每个音视频分片,终端设备接收到服务器返回的音视频分片后下载并播放该音视频分片。Optionally, the terminal device receives at least one audio and video segment of the target audio and video sequentially returned by the server. For each audio and video fragment, the terminal device downloads and plays the audio and video fragment after receiving the audio and video fragment returned by the server.
综上所述,本申请实施例通过终端设备接收到目标音视频的播放指令后,向服务器发送携带有网络质量信息的音视频播放请求,音视频播放请求用于指示服务器返回目标音视频的音视频数据;终端设备下载并播放服务器返回的目标音视频的音视频数据;即在终端设备向服务器发送音视频播放请求时,主动上报用于指示终端设备当前的接入网络的网络质量情况的网络质量信息,避免了相关技术中服务器与终端设备之间多次交互进行网络带宽探测的情况,从而缩短了音视频的起播时延,保证了音视频的播放效果。To sum up, in this embodiment of the present application, after receiving the target audio and video playback instruction through the terminal device, an audio and video playback request carrying network quality information is sent to the server, and the audio and video playback request is used to instruct the server to return the target audio and video audio. Video data; the terminal device downloads and plays the audio and video data of the target audio and video returned by the server; that is, when the terminal device sends an audio and video playback request to the server, it actively reports the network used to indicate the network quality of the current access network of the terminal device. The quality information avoids the situation of multiple interactions between the server and the terminal device for network bandwidth detection in the related art, thereby shortening the playback delay of audio and video, and ensuring the playback effect of audio and video.
相关技术中,如图5所示,发送数据的控制端在服务器侧,但是网络瓶颈往往在终端设备侧,这导致如下两个方面的问题:一方面,终端设备侧的网络质量情况反馈慢。比如,步骤501,服务器在接收到终端设备发送的连接请求后,向终端设备发送初始报文;步骤502,终端设备反馈ACK消息;步骤503,服务器向终端设备发送第二报文,第二报文的报文大小大于初始报文的报文大小;步骤504,终端设备反馈ACK消息。按照上述方法服务器进行网络带宽探测,通常需要至少2~3个RTT才能探测到网络带宽。另一方面,终端设备与服务器之间的链路不对称。比如,步骤505,服务器在探测到终端设备当前的接入网络的网络带宽后,根据网络带宽调整通信窗口大小和音视频的播放码率。步骤506,服务器下发音视频分片。终端设备实际收到音视频分片后会返回ACK消息至服务器。步骤507,若服务器没有收到终端设备发送的ACK消息,则认为网络受损,立刻降低音视频的播放码率,按照降低后的播放码率发送后续的音视频分片。若服务器没有收到的ACK消息超过预设个数,则终止发送数据。从而导致音视频的播放界面画质下降或者音视频播放过程中出现卡顿。In the related art, as shown in FIG. 5 , the control end that sends data is on the server side, but the network bottleneck is often on the terminal device side, which leads to the following two problems: On the one hand, the feedback of the network quality on the terminal device side is slow. For example, in step 501, the server sends an initial message to the terminal device after receiving the connection request sent by the terminal device; in step 502, the terminal device feeds back an ACK message; in step 503, the server sends a second message to the terminal device, the second message The packet size of the message is larger than that of the initial packet; in step 504, the terminal device feeds back an ACK message. According to the above method, when the server detects the network bandwidth, it usually takes at least 2 to 3 RTTs to detect the network bandwidth. On the other hand, the link between the end device and the server is not symmetrical. For example, in step 505, after detecting the network bandwidth of the current access network of the terminal device, the server adjusts the size of the communication window and the playing bit rate of audio and video according to the network bandwidth. Step 506, the server downloads the pronunciation video segment. After the terminal device actually receives the audio and video segments, it will return an ACK message to the server. Step 507: If the server does not receive the ACK message sent by the terminal device, it considers that the network is damaged, immediately reduces the audio and video playback bit rate, and sends subsequent audio and video segments according to the reduced playback bit rate. If the server does not receive more than the preset number of ACK messages, it will stop sending data. As a result, the image quality of the audio and video playback interface is degraded or the audio and video playback process freezes.
为了解决音视频传输流程中,网络带宽探测效率低下以及网络出现抖动情况下音视频的播放界面画质下降、卡顿的问题,本申请实施例提供了一种音视频的处理方法,包括但不限于如下两个阶段,如图6所示:第一个阶段即初始建链阶段,步骤601,终端设备通过音视频客户端发送连接请求至服务器;步骤602,服务器返回ACK消息至终端设备。步骤603,终端设备通过音视频客户端发送携带有网络质量信息的音视频播放请求至服务器,该网络质量信息包括终端设备当前的接入网络的网络带宽;步骤604,服务器根据音视频播放请求中的网络带宽,确定合适的通信窗口大小和播放码率。从而避免了相关技术中服务器与终端设备之间多次交互进行网络带宽探测的情况。第二阶段即数据传输阶段,步骤605,服务器根据确定出的通信窗口大小和播放码率,依次向终端设备返回目标音视频的至少一个音视频分片。步骤606,终端设备在接收到音视频分片后,向服务器发送多个ACK消息,确保被服务器接收到,避免相关技术中服务器因未接收到ACK消息而降低播放码率,使得画质下降,甚至降低通信窗口大小,导致卡顿的情况。步骤607,服务器在接收到同一个音视频分片对应的多个ACK消息的情况下,去除重复的ACK消息。In order to solve the problems of low network bandwidth detection efficiency and network jitter in the audio and video transmission process, the audio and video playback interface is degraded and stuck, the embodiment of the present application provides an audio and video processing method, including but not limited to It is limited to the following two stages, as shown in Figure 6: the first stage is the initial link establishment stage, step 601, the terminal device sends a connection request to the server through the audio and video client; step 602, the server returns an ACK message to the terminal device. Step 603, the terminal device sends an audio and video playback request carrying network quality information to the server through the audio and video client, and the network quality information includes the network bandwidth of the terminal device's current access network; Step 604, the server according to the audio and video playback request. the network bandwidth, determine the appropriate communication window size and playback bit rate. Thus, the situation in the related art in which the server and the terminal device are repeatedly interacted to perform network bandwidth detection is avoided. The second stage is the data transmission stage. In step 605, the server sequentially returns at least one audio and video segment of the target audio and video to the terminal device according to the determined communication window size and playback bit rate. Step 606, after receiving the audio and video segments, the terminal device sends multiple ACK messages to the server to ensure that they are received by the server, so as to prevent the server from reducing the playback bit rate due to the failure to receive the ACK message in the related art, so that the picture quality is degraded, Even reduce the size of the communication window, resulting in a stuck situation. Step 607: In the case of receiving multiple ACK messages corresponding to the same audio and video segment, the server removes the duplicate ACK messages.
请参考图7,其示出了本申请另一个示例性实施例提供的音视频的处理方法的流程图,本实施例以该方法用于图3所示的终端设备中来举例说明。该方法包括但不限 于以下几个步骤。Please refer to FIG. 7 , which shows a flowchart of an audio and video processing method provided by another exemplary embodiment of the present application. This embodiment is illustrated by using the method in the terminal device shown in FIG. 3 . The method includes but is not limited to the following steps.
步骤701,终端设备获取当前的接入网络的网络质量信息。Step 701, the terminal device acquires the network quality information of the current access network.
终端设备在接收到播放指令之前,提取获取当前的接入网络的网络质量信息。Before receiving the play instruction, the terminal device extracts and acquires the network quality information of the current access network.
可选的,当音视频客户端处于前台运行时,终端设备实时或者每隔预设时间段或者在接收到预设触发信号时获取网络质量信息。Optionally, when the audio and video client is running in the foreground, the terminal device acquires the network quality information in real time or every preset time period or when a preset trigger signal is received.
示意性的,预设时间间隔是默认设置的,或者是自定义设置的。Illustratively, the preset time interval is set by default, or is custom set.
示意性的,预设触发信号为作用于音视频客户端的用户界面中的用户操作信号。预设触发信号可以为用户界面切换信号,还可以为用户界面刷新信号。预设触发信号包括点击操作信号、滑动操作信号、按压操作信号、长按操作信号中的任意一种或多种的组合。本申请实施例对此不加以限定。Illustratively, the preset trigger signal is a user operation signal acting on the user interface of the audio and video client. The preset trigger signal can be a user interface switching signal, and can also be a user interface refresh signal. The preset trigger signal includes any one or a combination of a click operation signal, a sliding operation signal, a pressing operation signal, and a long-pressing operation signal. This embodiment of the present application does not limit this.
步骤702,终端设备根据接收到的播放指令,向服务器发送携带有网络质量信息的音视频播放请求。Step 702: The terminal device sends an audio and video playback request carrying network quality information to the server according to the received playback instruction.
终端设备在接收到播放指令后,向服务器发送携带有网络质量信息的音视频播放请求。After receiving the playback instruction, the terminal device sends an audio and video playback request carrying network quality information to the server.
需要说明的是,相关细节可参考上述实施例中的相关描述,在此不再赘述。It should be noted that, for relevant details, reference may be made to the relevant descriptions in the foregoing embodiments, which will not be repeated here.
步骤703,服务器根据接入网络的网络带宽和服务器的网络带宽,确定通信窗口大小和播放码率。 Step 703, the server determines the size of the communication window and the playing bit rate according to the network bandwidth of the access network and the network bandwidth of the server.
服务器接收终端设备发送的音视频播放请求,获取音视频播放请求中携带的网络质量信息。其中,网络质量信息包括终端设备当前的接入网络的网络带宽。可选地,网络质量信息还包括接入网络当前的信号强度和/或网络时延。The server receives the audio and video playback request sent by the terminal device, and obtains the network quality information carried in the audio and video playback request. Wherein, the network quality information includes the network bandwidth of the current access network of the terminal device. Optionally, the network quality information further includes the current signal strength and/or network delay of the access network.
可选地,服务器根据接入网络的网络带宽和服务器的网络带宽,确定目标网络带宽;根据预设对应关系,确定与该目标网络带宽对应的通信窗口大小和播放码率,预设对应关系包括网络带宽、通信窗口大小和播放码率之间的对应关系。本申请实施例对根据网络带宽确定通信窗口大小和播放码率的方式不加以限定。Optionally, the server determines the target network bandwidth according to the network bandwidth of the access network and the network bandwidth of the server; according to the preset correspondence, determines the communication window size and playback code rate corresponding to the target network bandwidth, and the preset correspondence includes: Correspondence between network bandwidth, communication window size and playback bit rate. The embodiments of the present application do not limit the manner in which the size of the communication window and the playing bit rate are determined according to the network bandwidth.
可选地,服务器根据接入网络的网络带宽和服务器的网络带宽,确定初始的通信窗口大小和播放码率,以便保证目标音视频的第一个音视频分片尽快发送至终端设备。Optionally, the server determines the initial communication window size and playback bit rate according to the network bandwidth of the access network and the network bandwidth of the server, so as to ensure that the first audio and video segment of the target audio and video is sent to the terminal device as soon as possible.
步骤704,服务器根据通信窗口大小和播放码率,向终端设备返回目标音视频的音视频分片。Step 704, the server returns the audio and video segments of the target audio and video to the terminal device according to the size of the communication window and the playing bit rate.
服务器按照确定出的通信窗口大小和播放码率,依次向终端设备返回目标音视频的至少一个音视频分片。The server sequentially returns at least one audio and video segment of the target audio and video to the terminal device according to the determined communication window size and playback bit rate.
可选地,服务器按照确定出的初始的通信窗口大小和播放码率,向终端设备返回目标音视频的第一个音视频分片。服务器在接收到终端设备返回的该音视频分片对应的ACK消息后,继续向终端设备返回目标音视频的第二个音视频分片。以此类推,不再赘述。Optionally, the server returns the first audio and video segment of the target audio and video to the terminal device according to the determined initial communication window size and playback code rate. After receiving the ACK message corresponding to the audio and video fragment returned by the terminal device, the server continues to return the second audio and video fragment of the target audio and video to the terminal device. And so on, and will not repeat them.
步骤705,终端设备在接收到音视频分片后,向服务器发送多个ACK消息。Step 705: After receiving the audio and video segments, the terminal device sends multiple ACK messages to the server.
其中,ACK消息用于指示终端设备已成功接收到音视频分片。The ACK message is used to indicate that the terminal device has successfully received the audio and video segments.
为防止ACK消息因为网络抖动导致服务器未收到,终端设备在接收到音视频分片后,向服务器发送多个ACK消息,其中音视频分片为目标音视频的至少一个音视频分片中的任意一个。In order to prevent the server from not receiving the ACK message due to network jitter, the terminal device sends multiple ACK messages to the server after receiving the audio and video segments, where the audio and video segments are at least one of the target audio and video segments. anyone.
可选地,终端设备在接收到音视频分片后,根据采集的接入网络当前的信号强度和/或网络时延,返回一个或多个ACK消息。其中,多个ACK消息为至少两个ACK消息。Optionally, after receiving the audio and video segments, the terminal device returns one or more ACK messages according to the collected current signal strength and/or network delay of the access network. The multiple ACK messages are at least two ACK messages.
可选地,当接入网络当前的信号强度和/或网络时延满足预设条件时,终端设备向服务器发送多个ACK消息;其中,预设条件包括信号强度小于预设强度阈值,和/或网络时延大于预设时延阈值。Optionally, when the current signal strength and/or network delay of the access network meet a preset condition, the terminal device sends multiple ACK messages to the server; wherein the preset condition includes that the signal strength is less than a preset strength threshold, and/ Or the network delay is greater than the preset delay threshold.
接入网络的信号强度和/或网络时延用于指示接入网络的网络质量。接入网络的信号强度与网络质量呈正相关关系,即接入网络的信号强度越强,表示接入网络的网络质量越好。接入网络的网络时延与网络质量呈负相关关系,即接入网络的网络时延越大,表示接入网络的网络质量越差。其中,接入网络的网络时延也称RTT。The signal strength and/or network delay of the access network is used to indicate the network quality of the access network. The signal strength of the access network is positively correlated with the network quality, that is, the stronger the signal strength of the access network, the better the network quality of the access network. The network delay of the access network is negatively correlated with the network quality, that is, the greater the network delay of the access network, the worse the network quality of the access network. Among them, the network delay of the access network is also called RTT.
其中,预设强度阈值或者预设时延阈值为默认设置的,或者是自定义设置的。本申请实施例对此不加以限定。The preset intensity threshold or the preset delay threshold is a default setting or a custom setting. This embodiment of the present application does not limit this.
在一种可能的实现方式中,终端设备在接收到音视频分片后,判断接入网络当前的信号强度和/或网络时延是否满足预设条件,若接入网络当前的信号强度和/或网络时延满足预设条件则确定待发送的ACK消息的发送个数和发送间隔,按照发送间隔依次发送多个ACK消息;若不满足预设条件则向服务器发送一个ACK消息。In a possible implementation manner, after receiving the audio and video segments, the terminal device determines whether the current signal strength and/or network delay of the access network meets the preset conditions, if the current signal strength and/or network delay of the access network Or if the network delay satisfies the preset condition, the number of ACK messages to be sent and the sending interval are determined, and multiple ACK messages are sent in sequence according to the sending interval; if the preset condition is not met, an ACK message is sent to the server.
可选地,ACK消息的发送个数与接入网络的信号强度呈负相关关系。即终端设备当前的接入网络的信号强度越弱,则ACK消息的发送个数越大。示意性的,ACK消息的发送个数设置有最小值和最大值。比如,最小值为1,最大值为3。本申请实施例对此不加以限定。Optionally, the number of ACK messages sent is negatively correlated with the signal strength of the access network. That is, the weaker the signal strength of the current access network of the terminal device, the greater the number of ACK messages sent. Illustratively, the number of sent ACK messages is set to a minimum value and a maximum value. For example, the minimum value is 1 and the maximum value is 3. This embodiment of the present application does not limit this.
可选地,发送间隔为依次发送的两个ACK消息之间的时间间隔,多个发送间隔可以是相同的,也可以是不同的。本申请实施例对此不加以限定。Optionally, the sending interval is a time interval between two ACK messages sent in sequence, and multiple sending intervals may be the same or different. This embodiment of the present application does not limit this.
在另一种可能的实现方式中,终端设备在接收到音视频分片后,向服务器发送ACK消息,并判断接入网络当前的信号强度和/或网络时延是否满足预设条件,若接入网络当前的信号强度和/或网络时延满足预设条件,则终端设备确定待发送的ACK消息的发送个数。终端设备实时检测接入网络当前的信号强度,当信号强度大于预设强度阈值时再次发送ACK消息。In another possible implementation manner, after receiving the audio and video segments, the terminal device sends an ACK message to the server, and judges whether the current signal strength and/or network delay of the access network meets the preset conditions. If the current signal strength and/or network delay of the incoming network meets the preset condition, the terminal device determines the number of ACK messages to be sent. The terminal device detects the current signal strength of the access network in real time, and sends an ACK message again when the signal strength is greater than a preset strength threshold.
需要说明的是,本申请实施例对多个ACK消息的发送方式不加以限定。It should be noted that, the embodiment of the present application does not limit the sending manner of multiple ACK messages.
步骤706,服务器在接收到同一个音视频分片对应的多个ACK消息的情况下,去除重复的ACK消息。Step 706: The server removes duplicate ACK messages in the case of receiving multiple ACK messages corresponding to the same audio and video segment.
为了避免误操作,服务器在接收到同一个音视频分片对应的多个ACK消息的情况下,去除重复的ACK消息,重新估算网络时延。服务器在接收到音视频分片对应的ACK消息后,保持通信窗口大小和播放码率不变,继续执行根据通信窗口大小和播放码率,向终端设备返回目标音视频的音视频分片的步骤。In order to avoid misoperation, when the server receives multiple ACK messages corresponding to the same audio and video segment, it removes the duplicate ACK messages and re-estimates the network delay. After receiving the ACK message corresponding to the audio and video segments, the server keeps the size of the communication window and the playback bit rate unchanged, and continues to perform the step of returning the audio and video segments of the target audio and video to the terminal device according to the size of the communication window and the playback bit rate .
在一个示意性的例子中,以终端设备为手机,服务器为视频服务器为例,手机通过无线上网(英文:Wi-Fi)方式接入家庭网络A,启动K视频客户端,根据当前接入的家庭网络A,探测到当前家庭网络A的带宽是50Mbps,在发起视频“XX烦恼”对应的视频播放请求时,通知视频服务器当前手机所接入的家庭网络A的带宽是50Mbps。视频服务器根据自身的带宽和手机当前的家庭网络A的带宽,选择最佳的视频播放码 率为1080p,按照15Mbps进行播放。由于视频是分片播放,首分片假设是2s时长,约30Mb大小,视频服务器到手机的网络时延RTT约为100ms,则手机在接收到“播放”按钮的点击操作至开始播放视频的时长(即起播时延)约为100ms,即100ms后就可以开始播放视频“XX烦恼”。而在相关技术中,由于前期至少2~3个RTT进行网络带宽探测,已经消耗200ms,加上传输控制协议(Transmission Control Protocol,TCP)慢启动本身的特性,还需要额外3~5个RTT才完成正常内容块大小的确认,此时起播时延至少在500ms以上。In a schematic example, taking the terminal device as a mobile phone and the server as a video server as an example, the mobile phone accesses the home network A through wireless Internet access (English: Wi-Fi), and starts the K video client. The home network A detects that the bandwidth of the current home network A is 50 Mbps, and when initiating a video playback request corresponding to the video "XX trouble", it notifies the video server that the bandwidth of the home network A currently connected by the mobile phone is 50 Mbps. The video server selects the best video playback bit rate of 1080p according to its own bandwidth and the bandwidth of the current home network A of the mobile phone, and plays it at 15Mbps. Since the video is played in segments, the first segment is assumed to be 2s long, about 30Mb in size, and the network delay RTT from the video server to the mobile phone is about 100ms, then the mobile phone receives the click operation of the "Play" button until the video starts to play. (i.e. the start-up delay) is about 100ms, that is, the video "XX trouble" can be played after 100ms. In the related art, due to at least 2-3 RTTs in the early stage for network bandwidth detection, it has already consumed 200ms, plus the transmission control protocol (Transmission Control Protocol, TCP) slow-start feature itself, additional 3-5 RTTs are required to The confirmation of the normal content block size is completed, and the start-up delay is at least 500ms.
在视频播放过程中,若用户携带手机走动,走到信号弱的位置,考虑障碍物对Wi-Fi的影响,可能会出现数据包丢包的情况,导致视频服务器向终端设备推送视频分片后,未接收到终端设备的ACK消息,但若用户继续走动到信号较强的位置,此时,该视频片段尚未播放完,终端设备检测到信号强度高于预设强度阈值则再次向视频服务器发送该视频片段对应的ACK消息,视频服务器获取到该ACK消息后,正常推送后续的1080p播放码率的视频分片。而在相关技术中,视频服务器在未收到ACK消息后直接降低视频的播放码率,导致画质下降,并且由于播放码率变化,可能还会导致时间延迟,出现视频播放卡顿的现象。During the video playback process, if the user walks around with the mobile phone to a location with weak signal, considering the impact of obstacles on Wi-Fi, data packets may be lost, causing the video server to push video fragments to the terminal device. , the ACK message from the terminal device has not been received, but if the user continues to move to a position with a strong signal, at this time, the video clip has not been played, and the terminal device detects that the signal strength is higher than the preset strength threshold and sends it to the video server again. The ACK message corresponding to the video clip, after the video server obtains the ACK message, normally pushes the subsequent video clips of the 1080p playback bit rate. In the related art, the video server directly reduces the playback bit rate of the video after not receiving the ACK message, resulting in a decrease in image quality, and due to the change in the playback bit rate, time delay may occur, and video playback may become stuck.
基于上述的例子可以看出,本申请实施例提供的音视频的处理方法,对音视频播放场景中的关键指标进行了优化:Based on the above example, it can be seen that the audio and video processing method provided by the embodiment of the present application optimizes the key indicators in the audio and video playback scene:
1、启播时延,即从接收到播放指令到音视频开始播放的时长。1. Startup delay, that is, the time from receiving the playback instruction to the start of audio and video playback.
该指标与终端设备当前的接入网络的网络带宽、网络时延和音视频的播放码率呈相关关系。比如,待播放视频为1080p视频,网络带宽约需要15Mbps,视频的第一个视频分片为2s,该视频分片的大小为30Mb;用户家庭网络带宽为50Mb,时延为100ms。This indicator is related to the current network bandwidth, network delay and audio and video playback bit rate of the terminal device's current access network. For example, if the video to be played is 1080p video, the network bandwidth is about 15Mbps, the first video segment of the video is 2s, and the size of the video segment is 30Mb; the user's home network bandwidth is 50Mb, and the delay is 100ms.
在相关技术中,采用RTMP/HTTP DASH机制,即利用TCP协议进行传输:由于慢启动的原因,需要500ms才能加载第一个片段:第一个RTT只能传输15KB,第二个RTT可以传输30KB,第三个RTT可以传输60KB,第四个RTT可以传输120KB;第5个RTT可以传输240KB,即:15+30+60+120+240=465KB=3.7MKb。In the related art, the RTMP/HTTP DASH mechanism is used, that is, the TCP protocol is used for transmission: due to slow start, it takes 500ms to load the first segment: the first RTT can only transmit 15KB, and the second RTT can transmit 30KB , the third RTT can transmit 60KB, the fourth RTT can transmit 120KB; the fifth RTT can transmit 240KB, namely: 15+30+60+120+240=465KB=3.7MKb.
而在本申请实施例中,终端设备主动上报接入网络的网络带宽为50Mb,服务器通过30Mb进行传输,在一个RTT就可以在终端设备上下载并播放音视频的第一个音视频分片。即将起播时延从500ms降到100ms。In the embodiment of the present application, the terminal device actively reports that the network bandwidth of the access network is 50Mb, the server transmits through 30Mb, and the terminal device can download and play the first audio and video segment of the audio and video in one RTT. The start-up delay will be reduced from 500ms to 100ms.
2、网络丢包率,即所丢失的数据包数量占所发送数据包的比率。2. The network packet loss rate, that is, the ratio of the number of lost packets to the transmitted packets.
丢包会影响音视频播放的连续性,若网络出现丢包,则音视频播放过程中可能卡顿。比如,网络丢包率为5%。终端设备采用单ACK消息进行响应的丢包率为5%,丢包后,服务器降低音视频的播放码率,影响音视频的传输吞吐,即音视频播放过程中发生卡顿的概率约为5%。Packet loss will affect the continuity of audio and video playback. If packet loss occurs on the network, the audio and video playback process may freeze. For example, the network packet loss rate is 5%. The packet loss rate of the terminal device responding with a single ACK message is 5%. After the packet loss, the server reduces the playback bit rate of the audio and video, which affects the transmission throughput of the audio and video. That is, the probability of a freeze during the audio and video playback process is about 5%. %.
而在本申请实施例中,终端设备采用多个ACK消息进行响应,比如采用两个ACK消息进行响应的丢包率为0.25%,即仅有0.25%的概率会导致服务器降低音视频的播放码率,即音视频播放过程中发生卡顿的概率仅为0.25%。However, in the embodiment of the present application, the terminal device responds with multiple ACK messages. For example, the packet loss rate of using two ACK messages to respond is 0.25%, that is, only a 0.25% probability will cause the server to reduce the playback code of audio and video. rate, that is, the probability of stuttering during audio and video playback is only 0.25%.
综上所述,本申请实施例提供的音视频的处理方法,在一方面,终端设备主动上报网络质量信息:终端设备向服务器发送音视频播放请求,该音视频播放请求携带有终端设备当前的接入网络的网络质量信息,网络质量信息包括网络带宽,避免了相关 技术中网络质量信息需要服务器经过多个RTT探测才能获取的情况。在另一方面,实现了音视频播放的快速启动:服务器根据终端设备主动上报的网络质量信息确定通信窗口大小和播放码率,根据通信窗口大小和播放码率返回目标音视频的音视频数据,尽量保障目标音视频的第一个音视频分片尽快发送至终端设备,跳过了相关技术中的TCP协议的慢启动过程。在另一方面,实现了接入网络拥塞处理:为了防止ACK消息丢包,终端设备在收到音视频分片后,通过发送多个ACK消息,来避免服务器由于未收到ACK消息而进行降速的情况,其中ACK消息的发送个数与接入网络的信号强度呈负相关关系,即接入网络的信号强度越高,ACK消息的冗余量越少;只有在多个ACK消息同时丢包的情况下,服务器才会主动下降播放码率,这种方法极大地降低了音视频播放过程发生卡顿的情况;在另一方面,实现了ACK消息的去冗余处理:服务器在接收到同一个音视频分片对应的多个ACK消息的情况下,去除重复的ACK消息,重新估算RTT,只要服务器收到某一个ACK消息,则说明即便上行链路出现了某些问题,当前的下行链路是正常的,这种情况下服务器无需降低音视频的播放码率,可以仍然按照当前的通信窗口大小和播放码率向终端设备返回下一个音视频分片,而只有在多个ACK消息均没有收到的情况下,服务器才会降低音视频的播放码率,按照降低后的播放码率发送后续的音视频分片。To sum up, in the audio and video processing method provided by the embodiments of the present application, on the one hand, the terminal device actively reports network quality information: the terminal device sends an audio and video playback request to the server, and the audio and video playback request carries the current information of the terminal device. The network quality information of the access network, including the network bandwidth, avoids the situation in the related art that the network quality information needs to be acquired by the server through multiple RTT detections. On the other hand, the quick start of audio and video playback is realized: the server determines the size of the communication window and the playback code rate according to the network quality information actively reported by the terminal device, and returns the audio and video data of the target audio and video according to the size of the communication window and the playback code rate, Try to ensure that the first audio and video segment of the target audio and video is sent to the terminal device as soon as possible, skipping the slow start process of the TCP protocol in the related art. On the other hand, the access network congestion processing is implemented: in order to prevent the loss of ACK messages, the terminal device sends multiple ACK messages after receiving the audio and video segments to avoid the server from downgrading due to not receiving the ACK message. In the case of high speed, the number of ACK messages sent is negatively correlated with the signal strength of the access network, that is, the higher the signal strength of the access network, the less redundancy of ACK messages; only when multiple ACK messages are lost at the same time In the case of packets, the server will actively reduce the playback bit rate. This method greatly reduces the situation that the audio and video playback process is stuck; on the other hand, the de-redundancy processing of ACK messages is realized: the server receives In the case of multiple ACK messages corresponding to the same audio and video segment, remove the duplicate ACK messages and re-estimate the RTT. As long as the server receives a certain ACK message, it means that even if there are some problems in the uplink, the current downlink The link is normal. In this case, the server does not need to reduce the audio and video playback bit rate, and can still return the next audio and video fragment to the terminal device according to the current communication window size and playback bit rate. If neither is received, the server will reduce the playback bit rate of the audio and video, and send the subsequent audio and video segments according to the reduced playback bit rate.
请参考图8,其示出了本申请一个示例性实施例提供的音视频的处理装置的框图。该音视频的处理装置可以通过软件、硬件或者两者的结合实现成为图3所示的终端设备的全部或者一部分。该音视频的处理装置可以包括:接收单元810、发送单元820和处理单元830。Please refer to FIG. 8 , which shows a block diagram of an apparatus for processing audio and video provided by an exemplary embodiment of the present application. The audio and video processing apparatus can be implemented by software, hardware or a combination of the two to become all or a part of the terminal device shown in FIG. 3 . The audio and video processing apparatus may include: a receiving unit 810 , a sending unit 820 and a processing unit 830 .
接收单元810,用于接收播放指令,播放指令用于指示开始播放目标音视频;The receiving unit 810 is used to receive a play instruction, and the play instruction is used to instruct to start playing the target audio and video;
发送单元820,用于根据播放指令,向服务器发送携带有网络质量信息的音视频播放请求,网络质量信息用于指示终端设备当前的接入网络的网络质量情况,音视频播放请求用于指示服务器返回目标音视频的音视频数据;The sending unit 820 is configured to send an audio and video playback request carrying network quality information to the server according to the playback instruction, where the network quality information is used to indicate the network quality of the current access network of the terminal device, and the audio and video playback request is used to indicate the server. Return the audio and video data of the target audio and video;
处理单元830,下载并播放目标音视频的音视频数据。The processing unit 830 downloads and plays the audio and video data of the target audio and video.
在一种可能的实现方式中,网络质量信息包括终端设备当前的接入网络的网络带宽。In a possible implementation manner, the network quality information includes the network bandwidth of the current access network of the terminal device.
在另一种可能的实现方式中,音视频数据包括至少一个音视频分片,该装置还包括:In another possible implementation manner, the audio and video data includes at least one audio and video segment, and the device further includes:
发送单元820,还用于在接收到音视频分片后,向服务器发送多个ACK消息,ACK消息用于指示终端设备已成功接收到音视频分片。The sending unit 820 is further configured to send multiple ACK messages to the server after receiving the audio and video segments, where the ACK messages are used to indicate that the terminal device has successfully received the audio and video segments.
在另一种可能的实现方式中,发送单元820,还用于在接收到音视频分片后,当接入网络当前的信号强度和/或网络时延满足预设条件时,向服务器发送多个ACK消息;In another possible implementation manner, the sending unit 820 is further configured to, after receiving the audio and video segments, when the current signal strength and/or network delay of the access network meets a preset condition, send the multiple ACK messages;
其中,预设条件包括信号强度小于预设强度阈值,和/或网络时延大于预设时延阈值。Wherein, the preset condition includes that the signal strength is less than the preset strength threshold, and/or the network delay is greater than the preset delay threshold.
在另一种可能的实现方式中,ACK消息的发送个数与接入网络的信号强度呈负相关关系。In another possible implementation manner, the number of sent ACK messages is negatively correlated with the signal strength of the access network.
需要说明的是,上述实施例提供的装置在实现其功能时,仅以上述各个功能模块的划分进行举例说明,实际应用中,可以根据实际需要而将上述功能分配由不同的功能模块完成,即将设备的内容结构划分成不同的功能模块,以完成以上描述的全部或者部分功能。It should be noted that when the device provided in the above embodiment realizes its functions, only the division of the above functional modules is used as an example for illustration. In practical applications, the above functions can be allocated to different functional modules according to actual needs. The content structure of the device is divided into different functional modules to complete all or part of the functions described above.
关于上述实施例中的装置,其中各个模块执行操作的具体方式已经在有关该方法的实施例中进行了详细描述,此处将不做详细阐述说明。Regarding the apparatus in the above-mentioned embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be described in detail here.
请参考图9,其示出了本申请一个示例性实施例提供的音视频的处理装置的框图。该音视频的处理装置可以通过软件、硬件或者两者的结合实现成为图3所示的服务器的全部或者一部分。该音视频的处理装置可以包括:接收单元910和发送单元920。Please refer to FIG. 9 , which shows a block diagram of an apparatus for processing audio and video provided by an exemplary embodiment of the present application. The audio and video processing apparatus can be implemented by software, hardware or a combination of the two to become all or a part of the server shown in FIG. 3 . The audio and video processing apparatus may include: a receiving unit 910 and a sending unit 920 .
接收单元910,用于接收终端设备发送的携带有网络质量信息的音视频播放请求,网络质量信息用于指示终端设备当前的接入网络的网络质量情况;A receiving unit 910, configured to receive an audio and video playback request that carries network quality information sent by the terminal device, where the network quality information is used to indicate the current network quality of the terminal device's access network;
发送单元920,用于根据音视频播放请求,向终端设备返回目标音视频的音视频数据。The sending unit 920 is configured to return the audio and video data of the target audio and video to the terminal device according to the audio and video playback request.
在一种可能的实现方式中,网络质量信息包括接入网络的网络带宽,发送单元920,还用于:In a possible implementation manner, the network quality information includes the network bandwidth of the access network, and the sending unit 920 is further configured to:
根据接入网络的网络带宽和服务器的网络带宽,确定通信窗口大小和播放码率;According to the network bandwidth of the access network and the network bandwidth of the server, determine the size of the communication window and the playback code rate;
根据通信窗口大小和播放码率,向终端设备返回目标音视频的音视频数据。According to the size of the communication window and the playing bit rate, the audio and video data of the target audio and video are returned to the terminal device.
在另一种可能的实现方式中,音视频数据包括至少一个音视频分片,该装置还包括:In another possible implementation manner, the audio and video data includes at least one audio and video segment, and the device further includes:
接收单元910,还用于接收终端设备发送的音视频分片对应的多个ACK消息,ACK消息用于指示终端设备已成功接收到音视频分片。The receiving unit 910 is further configured to receive multiple ACK messages corresponding to the audio and video segments sent by the terminal device, where the ACK messages are used to indicate that the terminal device has successfully received the audio and video segments.
在另一种可能的实现方式中,该装置还包括:处理单元;In another possible implementation manner, the apparatus further includes: a processing unit;
处理单元,用于在接收到同一个音视频分片对应的多个ACK消息的情况下,去除重复的ACK消息。The processing unit is configured to remove duplicate ACK messages in the case of receiving multiple ACK messages corresponding to the same audio and video segment.
需要说明的是,上述实施例提供的装置在实现其功能时,仅以上述各个功能模块的划分进行举例说明,实际应用中,可以根据实际需要而将上述功能分配由不同的功能模块完成,即将设备的内容结构划分成不同的功能模块,以完成以上描述的全部或者部分功能。It should be noted that when the device provided in the above embodiment realizes its functions, only the division of the above functional modules is used as an example for illustration. In practical applications, the above functions can be allocated to different functional modules according to actual needs. The content structure of the device is divided into different functional modules to complete all or part of the functions described above.
关于上述实施例中的装置,其中各个模块执行操作的具体方式已经在有关该方法的实施例中进行了详细描述,此处将不做详细阐述说明。Regarding the apparatus in the above-mentioned embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be described in detail here.
请参考图10,其示出了本申请一个实施例提供的终端设备的结构示意图。该终端设备包括中央处理器(Central Processing Unit,CPU)1010、存储器1020、网络接口1030。Please refer to FIG. 10 , which shows a schematic structural diagram of a terminal device provided by an embodiment of the present application. The terminal device includes a central processing unit (Central Processing Unit, CPU) 1010, a memory 1020, and a network interface 1030.
中央处理器1010包括一个或者一个以上处理核心。中央处理器1010用于执行终端设备的各种功能应用以及用于进行数据处理。The central processing unit 1010 includes one or more processing cores. The central processing unit 1010 is used to execute various functional applications of the terminal device and to perform data processing.
终端设备通常包括多个网络接口1030。Terminal devices typically include multiple network interfaces 1030 .
存储器1020通过总线与中央处理器1010相连。存储器1020用于存储指令,处理 器1010通过执行存储器1020中存储的指令实现上述由终端设备执行的音视频的处理方法。The memory 1020 is connected to the central processing unit 1010 through a bus. The memory 1020 is used to store instructions, and the processor 1010 implements the above-mentioned audio and video processing method executed by the terminal device by executing the instructions stored in the memory 1020.
存储器1020可存储操作***1021和至少一个功能所需的应用程序模块1022。操作***1021包括LINUX操作***、Unix操作***、Windows操作***中的至少一种。The memory 1020 may store an operating system 1021 and an application program module 1022 required for at least one function. The operating system 1021 includes at least one of a LINUX operating system, a Unix operating system, and a Windows operating system.
可选地,应用程序模块1022包括接收单元、发送单元和处理单元以及其他用于实现上述音视频的处理方法的单元等。Optionally, the application module 1022 includes a receiving unit, a sending unit, a processing unit, and other units for implementing the above-mentioned audio and video processing methods, and the like.
接收单元,用于接收播放指令,播放指令用于指示开始播放目标音视频;a receiving unit, for receiving a playback instruction, and the playback instruction is used for instructing to start playing the target audio and video;
发送单元,用于根据播放指令,向服务器发送携带有网络质量信息的音视频播放请求,网络质量信息用于指示终端设备当前的接入网络的网络质量情况,音视频播放请求用于指示服务器返回目标音视频的音视频数据;The sending unit is used to send an audio and video playback request carrying network quality information to the server according to the playback instruction, where the network quality information is used to indicate the network quality of the current access network of the terminal device, and the audio and video playback request is used to instruct the server to return The audio and video data of the target audio and video;
处理单元,下载并播放目标音视频的音视频数据。The processing unit downloads and plays the audio and video data of the target audio and video.
可选地,存储器1020可以由任何类型的易失性或非易失性存储设备或者它们的组合实现,如静态随机存取存储器(SRAM),电可擦除可编程只读存储器(EEPROM),可擦除可编程只读存储器(EPROM),可编程只读存储器(PROM),只读存储器(ROM),磁存储器,快闪存储器,磁盘或光盘。Alternatively, memory 1020 may be implemented by any type of volatile or non-volatile storage device or combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic or Optical Disk.
请参考图11,其示出了本申请一个实施例提供的服务器的结构示意图。该服务器包括CPU1110、存储器1120、网络接口1130。Please refer to FIG. 11 , which shows a schematic structural diagram of a server provided by an embodiment of the present application. The server includes a CPU 1110 , a memory 1120 , and a network interface 1130 .
中央处理器1110包括一个或者一个以上处理核心。中央处理器1110用于执行服务器的各种功能应用以及用于进行数据处理。The central processing unit 1110 includes one or more processing cores. The central processing unit 1110 is used to execute various functional applications of the server and to perform data processing.
服务器通常包括多个网络接口1130。The server typically includes multiple network interfaces 1130 .
存储器1120通过总线与中央处理器1110相连。存储器1120用于存储指令,处理器1110通过执行存储器1120中存储的指令实现上述由服务器执行的音视频的处理方法。The memory 1120 is connected to the central processing unit 1110 through a bus. The memory 1120 is used for storing instructions, and the processor 1110 implements the above-mentioned audio and video processing method executed by the server by executing the instructions stored in the memory 1120 .
存储器1120可存储操作***1121和至少一个功能所需的应用程序模块1122。操作***1121包括LINUX操作***、Unix操作***、Windows操作***中的至少一种。The memory 1120 may store an operating system 1121 and an application module 1122 required for at least one function. The operating system 1121 includes at least one of a LINUX operating system, a Unix operating system, and a Windows operating system.
可选地,应用程序模块1122包括接收单元、发送单元以及其他用于实现上述音视频的处理方法的单元等。Optionally, the application module 1122 includes a receiving unit, a sending unit, and other units for implementing the above-mentioned audio and video processing methods, and the like.
接收单元,用于接收终端设备发送的携带有网络质量信息的音视频播放请求,网络质量信息用于指示终端设备当前的接入网络的网络质量情况;a receiving unit, configured to receive an audio and video playback request that carries network quality information sent by the terminal device, and the network quality information is used to indicate the current network quality of the terminal device's access network;
发送单元,用于根据音视频播放请求,向终端设备返回目标音视频的音视频数据。The sending unit is configured to return the audio and video data of the target audio and video to the terminal device according to the audio and video playback request.
可选地,存储器1120可以由任何类型的易失性或非易失性存储设备或者它们的组合实现,如静态随机存取存储器(SRAM),电可擦除可编程只读存储器(EEPROM),可擦除可编程只读存储器(EPROM),可编程只读存储器(PROM),只读存储器(ROM),磁存储器,快闪存储器,磁盘或光盘。Alternatively, memory 1120 may be implemented by any type of volatile or non-volatile storage device or combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic or Optical Disk.
本申请的实施例提供了一种终端设备,该终端设备包括:处理器以及用于存储处理器可执行指令的存储器;其中,处理器被配置为执行指令时实现上述终端设备侧执行的方法。An embodiment of the present application provides a terminal device, the terminal device includes: a processor and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above-mentioned method executed on the terminal device side when executing the instructions.
本申请的实施例提供了一种服务器,该服务器包括:处理器以及用于存储处理器可执行指令的存储器;其中,处理器被配置为执行指令时实现上述服务器侧执行的方法。An embodiment of the present application provides a server, which includes: a processor and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above server-side execution method when executing the instructions.
本申请的实施例提供了一种计算机程序产品,包括计算机可读代码,或者承载有计算机可读代码的非易失性计算机可读存储介质,当计算机可读代码在电子设备的处理器中运行时,电子设备中的处理器执行上述方法。Embodiments of the present application provide a computer program product, including computer-readable codes, or a non-volatile computer-readable storage medium carrying computer-readable codes, when the computer-readable codes are executed in a processor of an electronic device , the processor in the electronic device executes the above method.
本申请的实施例提供了一种非易失性计算机可读存储介质,其上存储有计算机程序指令,计算机程序指令被处理器执行时实现上述方法。Embodiments of the present application provide a non-volatile computer-readable storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the foregoing method is implemented.
计算机可读存储介质可以是保持和存储由指令执行设备使用的指令的有形设备。比如,计算机可读存储介质例包括但不限于:电存储设备、磁存储设备、光存储设备、电磁存储设备、半导体存储设备或者上述的任意合适的组合。计算机可读存储介质的更具体的例子(非穷举的列表)包括:便携式计算机盘、硬盘、随机存取存储器(Random Access Memory,RAM)、只读存储器(Read Only Memory,ROM)、可擦式可编程只读存储器(Electrically Programmable Read-Only-Memory,EPROM或闪存)、静态随机存取存储器(Static Random-Access Memory,SRAM)、便携式压缩盘只读存储器(Compact Disc Read-Only Memory,CD-ROM)、数字多功能盘(Digital Video Disc,DVD)、记忆棒、软盘、机械编码设备、例如其上存储有指令的打孔卡或凹槽内凸起结构、以及上述的任意合适的组合。A computer-readable storage medium may be a tangible device that retains and stores instructions for use by the instruction execution device. For example, examples of computer-readable storage media include, but are not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read-only memory (Electrically Programmable Read-Only-Memory, EPROM or flash memory), static random access memory (Static Random-Access Memory, SRAM), portable compact disk read-only memory (Compact Disc Read-Only Memory, CD - ROM), Digital Video Disc (DVD), memory sticks, floppy disks, mechanically encoded devices, such as punch cards or raised structures in grooves on which instructions are stored, and any suitable combination of the foregoing .
这里所描述的计算机可读程序指令或代码可以从计算机可读存储介质下载到各个计算/处理设备,或者通过网络、例如因特网、局域网、广域网和/或无线网下载到外部计算机或外部存储设备。网络可以包括铜传输电缆、光纤传输、无线传输、路由器、防火墙、交换机、网关计算机和/或边缘服务器。每个计算/处理设备中的网络适配卡或者网络接口从网络接收计算机可读程序指令,并转发该计算机可读程序指令,以供存储在各个计算/处理设备中的计算机可读存储介质中。Computer readable program instructions or code described herein may be downloaded to various computing/processing devices from a computer readable storage medium, or to an external computer or external storage device over a network such as the Internet, a local area network, a wide area network and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer-readable program instructions from a network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .
用于执行本申请操作的计算机程序指令可以是汇编指令、指令集架构(Instruction Set Architecture,ISA)指令、机器指令、机器相关指令、微代码、固件指令、状态设置数据、或者以一种或多种编程语言的任意组合编写的源代码或目标代码,所述编程语言包括面向对象的编程语言—诸如Smalltalk、C++等,以及常规的过程式编程语言—诸如“C”语言或类似的编程语言。计算机可读程序指令可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中,远程计算机可以通过任意种类的网络—包括局域网(Local Area Network,LAN)或广域网(Wide Area Network,WAN)—连接到用户计算机,或者,可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。在一些实施例中,通过利用计算机可读程序指令的状态信息来个性化定制电子电路,例如可编程逻辑电路、现场可编程门阵列(Field-Programmable Gate Array,FPGA)或可编程逻辑阵列(Programmable Logic Array,PLA),该电子电路可以执行计算机可读程序指令,从而实现本申请的各个方面。The computer program instructions used to perform the operations of the present application may be assembly instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or in one or more source or object code written in any combination of programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or, may be connected to an external computer (eg, use an internet service provider to connect via the internet). In some embodiments, electronic circuits, such as programmable logic circuits, Field-Programmable Gate Arrays (FPGA), or Programmable Logic Arrays (Programmable Logic Arrays), are personalized by utilizing state information of computer-readable program instructions. Logic Array, PLA), the electronic circuit can execute computer readable program instructions to implement various aspects of the present application.
这里参照根据本申请实施例的方法、装置(***)和计算机程序产品的流程图和/ 或框图描述了本申请的各个方面。应当理解,流程图和/或框图的每个方框以及流程图和/或框图中各方框的组合,都可以由计算机可读程序指令实现。Aspects of the present application are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present application. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.
这些计算机可读程序指令可以提供给通用计算机、专用计算机或其它可编程数据处理装置的处理器,从而生产出一种机器,使得这些指令在通过计算机或其它可编程数据处理装置的处理器执行时,产生了实现流程图和/或框图中的一个或多个方框中规定的功能/动作的装置。也可以把这些计算机可读程序指令存储在计算机可读存储介质中,这些指令使得计算机、可编程数据处理装置和/或其他设备以特定方式工作,从而,存储有指令的计算机可读介质则包括一个制造品,其包括实现流程图和/或框图中的一个或多个方框中规定的功能/动作的各个方面的指令。These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer or other programmable data processing apparatus to produce a machine that causes the instructions when executed by the processor of the computer or other programmable data processing apparatus , resulting in means for implementing the functions/acts specified in one or more blocks of the flowchart and/or block diagrams. These computer readable program instructions can also be stored in a computer readable storage medium, these instructions cause a computer, programmable data processing apparatus and/or other equipment to operate in a specific manner, so that the computer readable medium on which the instructions are stored includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.
也可以把计算机可读程序指令加载到计算机、其它可编程数据处理装置、或其它设备上,使得在计算机、其它可编程数据处理装置或其它设备上执行一系列操作步骤,以产生计算机实现的过程,从而使得在计算机、其它可编程数据处理装置、或其它设备上执行的指令实现流程图和/或框图中的一个或多个方框中规定的功能/动作。Computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other equipment to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented process , thereby causing instructions executing on a computer, other programmable data processing apparatus, or other device to implement the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.
附图中的流程图和框图显示了根据本申请的多个实施例的装置、***、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上,流程图或框图中的每个方框可以代表一个模块、程序段或指令的一部分,所述模块、程序段或指令的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。在有些作为替换的实现中,方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如,两个连续的方框实际上可以基本并行地执行,它们有时也可以按相反的顺序执行,这依所涉及的功能而定。The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of apparatuses, systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more functions for implementing the specified logical function(s) executable instructions. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.
也要注意的是,框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合,可以用执行相应的功能或动作的硬件(例如电路或ASIC(Application Specific Integrated Circuit,专用集成电路))来实现,或者可以用硬件和软件的组合,如固件等来实现。It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in hardware (eg, circuits or ASICs (Application) that perform the corresponding functions or actions. Specific Integrated Circuit, application-specific integrated circuit)), or can be implemented by a combination of hardware and software, such as firmware.
尽管在此结合各实施例对本申请进行了描述,然而,在实施所要求保护的本申请过程中,本领域技术人员通过查看所述附图、公开内容、以及所附权利要求书,可理解并实现所述公开实施例的其它变化。在权利要求中,“包括”(comprising)一词不排除其他组成部分或步骤,“一”或“一个”不排除多个的情况。单个处理器或其它单元可以实现权利要求中列举的若干项功能。相互不同的从属权利要求中记载了某些措施,但这并不表示这些措施不能组合起来产生良好的效果。Although the application is described herein in conjunction with the various embodiments, those skilled in the art will understand and understand from a review of the drawings, the disclosure, and the appended claims in practicing the claimed application. Other variations of the disclosed embodiments are implemented. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that these measures cannot be combined to advantage.
以上已经描述了本申请的各实施例,上述说明是示例性的,并非穷尽性的,并且也不限于所披露的各实施例。在不偏离所说明的各实施例的范围和精神的情况下,对于本技术领域的普通技术人员来说许多修改和变更都是显而易见的。本文中所用术语的选择,旨在最好地解释各实施例的原理、实际应用或对市场中的技术的改进,或者使本技术领域的其它普通技术人员能理解本文披露的各实施例。Various embodiments of the present application have been described above, and the foregoing descriptions are exemplary, not exhaustive, and not limiting of the disclosed embodiments. Numerous modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the various embodiments, the practical application or improvement over the technology in the marketplace, or to enable others of ordinary skill in the art to understand the various embodiments disclosed herein.